Install¶

$ pip install pyroid

To stay on the unrelated 0.7.x toolkit, pin pyroid<1 (Migrating from 0.7).

pyroid has no runtime dependencies. Extras:

Extras combine: pip install "pyroid[cli,fasttext]". pyroid.ftfy, pyroid.filters, pyroid.dedup, pyroid.translit, pyroid.metrics, pyroid.json_repair, pyroid.trafilatura, pyroid.nltk, pyroid.jieba, pyroid.warc's reader and pyroid.patch() need none of them.

Some dependencies of the datatrove, hf and warc extras (orjson, tokenizers, PyYAML and faust-cchardet) publish no wheels for CPython 3.15 and 3.15t yet, and orjson none for 3.14t. There pip builds them from source: faust-cchardet needs a C++ compiler, and orjson and tokenizers a Rust toolchain. pyroid[all] skips those three extras on 3.15; name them to build them anyway. pyroid's own wheels cover every interpreter below.

Optional libraries, needed only by the features that use them:

• pyarrow, polars, pandas, Hugging Face datasets or any other Arrow library: pass their string columns straight in (Inputs and outputs).

• The incumbents, only for the few features pyroid hands back to them (Fallbacks):

  • ftfy 6.3.1;
  • datatrove 0.10.1 and a language's segmenter (Stanza, Kiwi, ...) for python_tokenizer=True;
  • trafilatura 2.3.1 (or 1.11.0 with compat="1.11.0") for the delegated extraction options (other output formats, metadata, links, ...);
  • nltk 3.10.4 for Punkt training and the other pyroid.nltk features it hands back;
  • jieba 0.42.1 for pyroid.jieba.posseg, pyroid.jieba.analyse and paddle mode;
  • rouge-score 0.1.2 and sacrebleu 2.6.0 for the delegated metrics features;
  • json_repair 0.64.5 for schema=;
  • datasketch 2.0.0 for its classes pyroid does not port (WeightedMinHash, HyperLogLog, ...) and for Redis or Cassandra LSH storage.

  Without them, those calls raise pyroid.PyroidError. - NLTK's punkt_tab data for pyroid.nltk's Punkt and ROUGE's split_summaries=True (python -m nltk.downloader punkt_tab; pyroid does not bundle it, details). - numpy for pyroid.metrics.scoring.BootstrapAggregator, which runs rouge-score's numpy code. - tldextract for pyroid.filters.URLFilter. Its suffix lists are read from a tldextract.TLDExtract when the filter is built; the matching itself runs in Rust. - sudachidict-core (20261015 or newer) for native Japanese in pyroid.filters and pyroid.dedup: pip install sudachidict-core. pyroid does not ship the dictionary, whose system.dic is 202 MB. It reads the installed one, or the file that PYROID_SUDACHI_DICT names, without importing SudachiPy. Without it the Japanese codes need python_tokenizer=True (Languages). - pythainlp for native Thai in pyroid.filters and pyroid.dedup: pip install pythainlp. pyroid reads its word list and sentence model (or the corpus directory that PYROID_PYTHAINLP_CORPUS names) without importing it. With both, 4,049 of datatrove's 4,343 language codes are native; without either, 4,025 (Languages).

Supported platforms¶

Each release has three wheels per platform, which cover every supported interpreter:

That is 24 wheels plus an sdist per release, and no compiler is needed. Before a release is published, the release workflow installs each wheel in a clean virtual environment and runs pyroid's test suite with it, on CPython 3.11, 3.14t, 3.15 and 3.15t, on every platform above. PyPy and GraalPy are not supported.

On free-threaded builds, importing pyroid does not re-enable the GIL. See Threads and free-threading.

Check the install¶

$ python -c "import pyroid; print(pyroid.__version__)"

It prints the installed version as pip shows it (PEP 440, e.g. 1.0.0).

If the native extension is missing or was built for another Python, import pyroid fails with an ImportError that says so. pyroid has no pure-Python fallback.

Build from source¶

Where no wheel fits (or with pip install --no-binary pyroid pyroid), pip builds the sdist. That needs no Rust install: if cargo is not on PATH, maturin first downloads a Rust toolchain (about 580 MB, into ~/.cache/puccinialin), so the build needs network access and takes a few minutes.

To build a checkout, you need a Rust toolchain and maturin 1.16 or newer:

$ git clone https://github.com/ao/pyroid && cd pyroid
$ uv venv && source .venv/bin/activate
$ uv pip install maturin
$ maturin develop --uv --release      # builds pyroid._pyroid into the venv

Without --release you get a much slower debug build. CONTRIBUTING.md covers the test and lint setup.