API reference¶

Generated from pyroid's type stubs (the .pyi files shipped in the wheel), which are also what your type checker and IDE read. The integrations have no stubs; their pages come from the Python source.

Module	Contents
`pyroid`	`PyroidError`, `InputError`, `FallbackWarning`, `Column`, `patch`, `unpatch`
`pyroid.ftfy`	ftfy 6.3.1's API, plus `fix_text_batch`
`pyroid.fasttext`	`load_model`, `Model`, `ModelError`, `tokenize`
`pyroid.filters`	datatrove's Gopher, C4 and FineWeb filters, `apply`, PII, C4 bad words, symbol lines, `URLFilter`, `languages`
`pyroid.dedup`	`minhash_dedup`, `MinHashDedup`, `sentence_dedup`, `SentenceDedup`, `exact_dedup`, `ExactDedup`, `minhash_signatures`, `simplify_text`, datatrove's config classes
`pyroid.translit`	`anyascii`, `unidecode`, `is_ascii`, `UnidecodeError`
`pyroid.metrics`	`rouge_scorer`, `scoring`, `tokenizers`, `tokenize` (rouge-score's modules) and `sacrebleu`
`pyroid.json_repair`	`repair_json`, `loads`, `load`, `from_file`, `repair_json_batch`, `NestingError`
`pyroid.datasketch`	`MinHash`, `LeanMinHash`, `MinHashLSH`, `MinHashLSHForest`, `MinHashLSHEnsemble` (datasketch's modules), `minhash_batch`
`pyroid.jieba`	jieba 0.42.1's API: `cut`, `lcut`, `cut_for_search`, `tokenize`, `add_word`, `load_userdict`, `Tokenizer`, ...
`pyroid.trafilatura`	`extract`, `bare_extraction`, `extract_with_metadata`, `extract_batch`, `Document`, `LRUCache`, `reset_caches`, the `compat=` modes 2.3.1 and 1.11.0
`pyroid.nltk`	NLTK 3.10.4's `sent_tokenize`, `word_tokenize`, `PunktSentenceTokenizer`, `PunktTokenizer`, `TreebankWordTokenizer`, `NLTKWordTokenizer`, `TreebankWordDetokenizer`
`pyroid.warc`	warcio 1.8.2's `ArchiveIterator`, `WARCIterator`, `ARCIterator`, `StatusAndHeaders`, `ArcWarcRecord`, `ArchiveLoadFailed`; datatrove's WARC loop: `iter_documents`, `iter_batches`
`pyroid.integrations`	the datatrove steps (`WarcReader` included), the Hugging Face helpers, the FineWeb recipes, `patch` and friends
`pyroid.text`	`normalize_whitespace`, `min_chars`: small examples of pyroid's conventions

Batch functions are listed with one signature per input kind (str, list, tuple, Arrow); their parameter tables give each parameter's types across all of them. The conventions they share are in Inputs and outputs, Threads and Errors.