Skip to content

Compatibility

pyroid aims for the same output as the libraries it replaces, not output close to it. This page is the evidence: what each module is compared against, by which rule, and how many comparisons have been made. Every known difference is listed in the module's guide.

How parity is tested

Each module is tested differentially: the same input goes to pyroid and to the incumbent, and the results are compared exactly. For fastText, probabilities may also differ by 1e-6, though none do on Linux (on macOS they can differ more). The metrics are compared as the strings they print, with no tolerance. The inputs are:

  • Pinned real documents: the first 10,000 FineWeb documents, 8 FineWeb-2 languages, and a seeded adversarial set of 3,232 documents (mojibake, near-duplicates, edge cases), all fixed by sha256. The metrics use pinned evaluation data instead: CNN/DailyMail, SummEval and the WMT21 system outputs. Trafilatura uses 7,192 pinned Common Crawl pages from 3 crawls, as datatrove's WARC reader and URL filter pass them on.
  • The incumbent's own test corpus, where it has one. For ftfy that is its 161 golden cases, including its 10 known failures, which pyroid reproduces too. The test suites of json_repair and Trafilatura are replayed call by call.
  • Property-based fuzzing with Hypothesis, with strategies that reach lone surrogates, every option value and every documented edge case. Hypothesis never generates surrogates by default; these strategies do.

For invalid input, the exception type must match too (messages may differ).

The suites run against the installed wheel, on regular and free-threaded CPython. Deliberately broken builds (for example ftfy without surrogate-pair joining) fail them.

Results

The counts below are from each module's test runs between 2026-11-05 and 2026-11-09, in a Docker/OrbStack VM on linux/aarch64 unless a row says otherwise. The native runs on AWS EC2 are under Platforms. CI runs the parity/ suites on Linux x86_64 and aarch64 on every pull request and push to main; their counts are in each job's summary and parity-report.json artifact, not on this page.

pyroid.ftfy

Reference: ftfy 6.3.1. Rule: byte-identical output, same exception type.

Check Compared Identical Different
Golden cases (ftfy's test_cases.json and re-garbled variants, doctests), 8 call shapes, replayed on Unicode 14.0 to 17.0 74,950 74,950 0
Fuzz: random corner-case mix, half with lone surrogates, random options, 5 Unicode versions 500,000 500,000 0
Realistic mojibake (1 to 2 layers, CESU-8, partial, lossy), 5 Unicode versions 1,151,345 1,151,345 0
Combining sequences, jamo, Indic scripts, new characters, all 4 normalization forms 2,199,550 2,199,550 0
Real documents: FineWeb dev 10k, FineWeb-2 (8 languages), adversarial set, 5 Unicode versions 76,160 76,160 0
Real documents: FineWeb 100k on Unicode 15.0 and 17.0 200,000 200,000 0
Replay after the final rebase, fresh seeds 649,606 649,606 0
Python Hypothesis, deep profile, on CPython 3.12 and 3.15t 120,000 120,000 0
Total 4,971,611 4,971,611 0

Source: the Rust oracle replay (crates/pyroid-core/tests/ftfy_oracle.rs) and python/tests/test_ftfy_parity.py, run on 2026-11-05.

pyroid.fasttext

Reference: fasttext-predict 0.9.2.4 (fastText's C++ predict path). Rule: same labels, probabilities within 1e-6 (measured: bit-identical).

Check Compared Identical Different
Pinned lid.176.bin and lid.176.ftz predictions, top-5, FineWeb dev 10k and adversarial set (str, list and Arrow paths) 26,464 26,464 0
Synthetic models vs the live oracle: 52 configurations (4 losses, dense, quantized, pruned, word n-grams) 63,024 63,024 0
Dense softmax and hierarchical softmax, dimensions 1 to 33 26,664 26,664 0
Total 116,152 116,152 0

Source: python/tests/test_fasttext_parity.py and the parity/ fastText suite, run on 2026-11-05.

pyroid.filters

Reference: datatrove 0.10.1 with spaCy 3.8.18 (and its other tokenizers: IndicNLP, whitespace, SudachiPy 0.7.0 with sudachidict-core 20261015, pyvi 0.1.1, PyThaiNLP 5.4.0 with python-crfsuite 0.9.12), tldextract 5.4.0, numpy 2.5.4. Rule: same keep/drop decision, same filter_reason, same rewritten text, same exception.

Check Compared Identical Different
Harness parity suite, 90 checks: Gopher repetition, Gopher quality, C4 (default and FineWeb settings) and FineWeb quality on 13,232 documents; the same filters with each language's settings on 8 FineWeb-2 slices; PII, C4 bad words, symbol lines and the URL filter 194,784 194,784 0
English tokenizer oracle replay (words and sentence counts): FineWeb, FineWeb-2, adversarial, fuzz and Unicode probes, CPython 3.12 to 3.14 (5 IS_ALPHA rows skipped: a spaCy history-dependent bug, see the filters guide) 964,148 964,148 0
Per-language tokenizer goldens: 57 spaCy rule-based languages, the whitespace tokenizer, IndicNLP (hi, ur, bn) and English 43,146 43,146 0
Chinese, native (spaCy's ChineseTokenizer over pyroid.jieba): FineWeb-2 cmn_Hani, the 8 filter cases against live datatrove, 2026-11-07 2,000 2,000 0
Japanese, native (datatrove's JapaneseTokenizer over the Rust port of SudachiPy 0.7.0): FineWeb-2 jpn_Jpan, the 8 filter cases against live datatrove, 2026-11-09 2,000 2,000 0
Japanese tokenizer, Rust replay of the committed recordings: SudachiPy morphemes for 4,567 texts (special cases, FineWeb-2 lines, fuzz, a code-point sample), datatrove's word_tokenize, span_tokenize and sent_tokenize for 4,205 texts (54 documents over 40,000 bytes, with chosen chunk boundaries), 1,911 chunk_text_on_bytes cases and the spaCy tag of all 1,558 POS (every tag in spaCy's TAG_MAP), 2026-11-09 12,241 12,241 0
Japanese tokenizer replay, not committed: every code point (69,504 rows each for morphemes and datatrove), the FineWeb-2 jpn_Jpan test slice (7,809 lines, 250 documents) and the first 20,000 jpn_Jpan training documents (32.5 M characters: 610,624 lines with 18,879,619 morphemes; datatrove's words and spans of the 20,000 documents, 134 over 40,000 bytes), 2026-11-09 777,691 777,691 0
Vietnamese and Thai, native (spaCy's VietnameseTokenizer over the Rust port of pyvi 0.1.1; datatrove's ThaiTokenizer over the Rust port of PyThaiNLP 5.4.0): FineWeb-2 vie_Latn and tha_Thai, the 8 filter cases each against live datatrove, 2026-11-09 4,000 4,000 0
Vietnamese tokenizer, Rust replay of the committed recordings: syllables, CRF labels, pyvi_tokenize and datatrove's word_tokenize, span_tokenize and sent_tokenize for 4,697 texts (special cases, FineWeb-2 documents, fuzz, a code-point sample), 2026-11-09 4,697 4,697 0
Vietnamese tokenizer replay, not committed: every code point (69,504 rows), the FineWeb-2 vie_Latn test slice (250 documents), all cases and fuzz, and the first 20,000 vie_Latn training documents (90.9 M characters); 151.5 M compared syllables, labels, words, spans and sentences, 2026-11-09 94,392 94,392 0
Thai tokenizer, Rust replay of the committed recordings: character clusters, newmm with and without safe mode, crfcut's words and raw labels, and datatrove's word_tokenize, span_tokenize and sent_tokenize for 3,165 texts (special cases, FineWeb-2 documents, fuzz, long texts, a code-point sample), 2026-11-09 3,165 3,165 0
Thai tokenizer replay, not committed: every code point (69,504 rows), the FineWeb-2 tha_Thai test slice (250 documents), all cases, fuzz and 45 long texts, the first 20,000 tha_Thai training documents (60.1 M characters), and 29,806 more rows (random text, 6 documents of 100k to 200k characters, 6,000 rows over synthetic dictionaries where newmm's 50-edge cutoff fires, and digits new in Unicode 16.0 and 17.0 under CPython 3.13 to 3.15), 2026-11-09 122,667 122,667 0
Per-language tokenizer replay: whole FineWeb-2 slices (ar, de, fr, ru, es, hi, with C4 lines), 10k fuzz rows per tokenizer, a 57-language FineWeb-2 smoke set 662,530 662,530 0
IndicNLP tokenizer replay: goldens and 8k fuzz rows per CPython 3.11 to 3.15 118,690 118,690 0
Whitespace tokenizer replay: goldens and extra rows 118,990 118,990 0
Total 3,125,141 3,125,141 0
  • Hypothesis against live datatrove: 92 tests x 300 examples over script-specific text in 22 native languages and the jieba, SudachiPy, pyvi and PyThaiNLP fallback paths (python_tokenizer=True).
  • Japanese, native: 12 Hypothesis properties against live datatrove (python/tests/test_filters_japanese.py, 628 examples per run; 11 of them also at 300 examples each).
  • Vietnamese and Thai, native: 80 tests against live datatrove (python/tests/test_filters_vithai.py: every filter, FineWeb-2's chains, words, sentence dedup and MinHash, on generated and FineWeb-2 text; 1,046 Hypothesis examples per run, 5,230 with the ci profile).
  • The newer steps, input by input: PII on 13,232 documents x 4 configurations, C4 bad words x 5 (per-document languages, seeded keep_fraction), symbol lines x 3, the URL filter on 10,000 FineWeb URLs x 7 configurations and 9,695 URL-like strings, plus Hypothesis against the reference and live datatrove.

Source: the parity/ filters suite (after the last rebase, 2026-11-06; vie_Latn and tha_Thai 2026-11-09), crates/pyroid-core/tests/filters_tokenizer.rs, crates/pyroid-core/src/{sudachi,pyvi,pythainlp}/replay.rs and the multilingual replays, run on 2026-11-05, 2026-11-06 and 2026-11-09.

pyroid.dedup

Reference: datatrove 0.10.1 MinHash (FineWeb: sha1, 64-bit, 14 buckets x 8 hashes, 5-grams; FineWeb-2: xxhash, 64-bit) and sentence dedup. Rule: identical signatures, removed set, cluster ids and sizes; sentence dedup: byte-identical signature, duplicate and index files.

Check Compared Identical Different
Harness parity suite, 25 checks: MinHash signatures and removal decisions on 13,232 documents (445 removed); FineWeb-2's MinHash on 8 language slices with injected near-duplicates; sentence dedup, 7 configurations on 13,232 documents and 9 on an 8,300-document corpus with injected duplicates 52,934 52,934 0
simplify_text vs datatrove on the corpus documents 13,232 13,232 0
Chinese, native: FineWeb-2 cmn_Hani MinHash signatures and removed set (300 documents with near-duplicates) against live datatrove, 2026-11-07 600 600 0
Japanese, native: FineWeb-2 jpn_Jpan MinHash signatures and removed set (300 documents with near-duplicates) against live datatrove, 2026-11-09 600 600 0
Vietnamese and Thai, native: FineWeb-2 vie_Latn and tha_Thai MinHash signatures and removed set (300 documents each, with near-duplicates) against live datatrove, 2026-11-09 1,200 1,200 0
Total 68,566 68,566 0
  • datatrove's stages 1 to 3 re-run and compared from raw text: every bucket's pairs, the removed ids, cluster ids and sizes are identical for sha1/64 (445 removed) and xxhash/32 (439 removed, 3,146 pairs).
  • Live-datatrove Hypothesis families (case, whitespace, punctuation, edits, swaps; 1-, 2- and 5-grams; 4 hash configurations) and a Python transcription of stages 1 to 3: identical.
  • simplify_text: CRC digests over all 1,112,064 code points x 13 configurations x 5 Unicode versions identical.
  • Sentence dedup: datatrove's three stages re-run on pyroid's files give the same documents and exclusion documents; Hypothesis 300 examples against the reference and 100 three-stage runs against live datatrove.

Source: the parity/ dedup suite (after the last rebase, 2026-11-06) and python/tests/test_dedup_parity.py, run on 2026-11-05 and 2026-11-06.

pyroid.translit

Reference: anyascii 0.3.3 (for anyascii()); Unidecode 1.4.0 as a black box (for unidecode()'s errors= semantics only). Rule: anyascii(): byte-identical; unidecode(): Unidecode's errors=, exceptions and warnings, on anyascii's data.

Check Compared Identical Different
anyascii() vs the anyascii package, every code point U+0000 to U+10FFFF (lists, single strings, chunked, Arrow) 1,114,112 1,114,112 0
Documents: FineWeb dev 10k, FineWeb-2 8 x 250 and FineWeb 100k (list, Arrow, unidecode), on CPython 3.12, 3.15 and 3.15t 112,000 112,000 0
Total 1,226,112 1,226,112 0
  • The generated table matches the anyascii package on all 1,112,064 non-surrogate code points, and the kernel matches the any_ascii crate on every scalar in all 4 errors= modes.
  • unidecode() vs Unidecode 1.4.0, black box: 194 tests over the alphabets both libraries map (every errors= mode, replace_str edge cases, invalid errors, messages, args, index, warning order, batches) pass. The transliterations themselves differ by design; see the translit guide.

Source: python/tests/test_translit_parity.py, test_translit_unidecode.py and the module's agreement run, 2026-11-05.

pyroid.metrics

Reference: rouge-score 0.1.2 (with NLTK 3.10.4's Porter stemmer) and sacrebleu 2.6.0. Rule: identical repr, str, format(), JSON and signature strings of every score: no tolerance.

Check Compared Identical Different
Parity suite on CPython 3.12 (GIL), 47 checks: 6 ROUGE configurations, 22 corpus BLEU/chrF configurations x 141 WMT21 systems, 4 sentence-level ones x 163,000 segments, per-call and batch APIs, live Hypothesis against both libraries 1,396,827 1,396,827 0
Parity suite on free-threaded CPython 3.15t, 44 checks (live sacrebleu skipped: importing it re-enables the GIL) 1,388,827 1,388,827 0
Parity suite on x86_64 (AWS EC2 c7i.4xlarge), CPython 3.12, 47 checks, live sacrebleu included, against the x86_64 reference re-pinned there (identical to the aarch64 one) 1,396,827 1,396,827 0
Parity suite on x86_64 (AWS EC2 c7i.4xlarge), free-threaded CPython 3.15t, 44 checks (live sacrebleu skipped) 1,388,827 1,388,827 0
Total 5,571,308 5,571,308 0
  • Data, pinned by sha256: CNN/DailyMail 3.0.0 test (11,490 lead-3 summaries and 2,000 whole articles against the highlights), SummEval (1,600 summaries from 16 systems x 11 references, score_multi), WMT21 newstest2021 (8 language pairs, 141 systems, 163,000 segments, up to 3 references). The reference file holds 694,757 results.
  • The Porter stemmer matches NLTK on all 512,136 distinct tokens of FineWeb 100k; 28 rouge_types edge cases and 49 sacrebleu error cases match in exception type, message and order.

Source: the parity/ metrics suite, run on 2026-11-06 (the x86_64 rows: bench/results/aws-c7i.4xlarge/2026-11-06/parity).

pyroid.json_repair

Reference: json_repair 0.64.5. Rule: the same returned string or object, the same repair log, the same exception.

Check Compared Identical Different
Parity suite on CPython 3.12, 33 checks: json_repair's own test suite replayed as 672 calls, 20,000 seeded broken LLM outputs x 8 option sets, Hypothesis grammar fuzzing with lone surrogates x 9 option sets, load and from_file 184,403 184,403 0
Parity suite on free-threaded CPython 3.15t (the 149 schema= replays skipped: no jsonschema) 184,254 184,254 0
Total 368,657 368,657 0
  • Skipped by design: 4 calls whose result depends on the hash seed and 3 that depend on the recursion limit.
  • The Rust kernel also replays 451 distinct upstream inputs x 4 option sets, and its Python-semantics tables are checked per interpreter (3.11 to 3.15t): str.isspace/isalpha/isdigit/isalnum on all 1,112,064 code points, json.loads 84k cases, json.dumps 45k, float.repr 145k.

Source: the parity/ json_repair suite, run on 2026-11-06.

pyroid.datasketch

Reference: datasketch 2.0.0 (live, with numpy 2.5.4 and scipy 1.18.1). Rule: identical hash values and dtypes, identical LeanMinHash bytes, identical (b, r), identical query result sets.

Check Compared Identical Different
MinHash.bulk signatures, 128 permutations, for affine32, affine64 and legacy: 100,000 FineWeb documents plus the 3,232-document adversarial set, word 5-gram shingles 309,696 309,696 0
minhash_batch(texts, ngram=5) rows, same corpus and schemes 309,696 309,696 0
LeanMinHash.serialize bytes (and deserialize back), one byte order per scheme 309,696 309,696 0
MinHashLSH (b, r): 20 thresholds x 4 num_perm x 3 weight pairs 240 240 0
Every document's MinHashLSH query result set: threshold 0.9, 0.5, params (32, 4), legacy at 0.8; query and query_batch 825,856 825,856 0
Result sets after removing every 10th key, and key membership 104,232 104,232 0
Two half indexes merged (every 10th document queried); MinHashLSHForest(l=8) top-10 key sets (every 10th document) 20,648 20,648 0
MinHashLSHEnsemble(threshold=0.8, num_part=16) containment result sets (20,000 documents indexed, every 10th queried) 2,000 2,000 0
Total 1,882,064 1,882,064 0
  • The (b, r) search ports scipy 1.18.1's QUADPACK: all 715,040 LSH and 258,300 LSH Ensemble probabilities it evaluates are bitwise identical to scipy's on aarch64, and all 9,156 grid argmins and 126 ensemble constructions match.
  • datasketch's own test suite (test_minhash, test_lean_minhash, test_minhash_schemes, test_lsh, test_lshforest, test_lshensemble) passes against pyroid's classes, with Hypothesis and seeded workloads compared against live datasketch: 1,020 tests on CPython 3.12, 3.14t, 3.15 and 3.15t.

Source: the parity/ datasketch suite (CPython 3.13, release wheel), run on 2026-11-07.

pyroid.jieba

Reference: jieba 0.42.1. Rule: the same words in the same order (empty strings included), the same offsets, FREQ values, total and exceptions.

Check Compared Identical Different
Parity suite on CPython 3.13, 61 checks: FineWeb-2 cmn_Hani and the adversarial set in 9 modes (cut with and without the HMM, full mode, search, tokenize), the batch path, user dictionaries, FREQ 387,557 387,557 0
Rust replay of recorded jieba and datatrove outputs: jieba's test sentences, 2,400 FineWeb-2 excerpts, mixed scripts and whitespace, 2,000 fuzz rows, 16 user-dictionary scenarios, dict.txt.small 73,236 73,236 0
20,000 FineWeb-2 cmn_Hani training documents (29.5 M characters), 5 modes and the batch path, documents x modes 120,000 120,000 0
Total 580,793 580,793 0
  • python/tests: Hypothesis properties against live jieba for every mode, add_word / del_word / suggest_freq / load_userdict sequences, broken and unusual dictionary files and every code point; 656 tests on CPython 3.13, 657 on 3.15t, also under pytest-run-parallel.
  • The route takes math.log of integers: bit-identical on CPython 3.12 to 3.15t for every dictionary frequency, and jieba's recorded outputs are byte-identical on 3.12, 3.13, 3.14 and 3.15.

Source: the parity/ jieba suite, crates/pyroid-core/src/jieba/replay.rs and a 20,000-document replay, run on 2026-11-07.

pyroid.trafilatura

Reference: trafilatura 2.3.1 with lxml 6.1.3 (libxml2 2.14.6) and jusText 3.0.2; for compat="1.11.0", trafilatura 1.11.0 with the releases a fresh datatrove[processing]==0.10.1 install resolves (lxml 6.1.3, jusText 3.0.2, lxml-html-clean 0.4.6, charset-normalizer 3.5.2, courlan 1.4.0, htmldate 1.11.0). Rule: extract text byte for byte (None distinct from ""), bare_extraction field by field, the same exceptions (type and message) and warnings.

Check Compared Identical Different
2.3.1: extract on the 7,192 pinned Common Crawl pages of 3 crawls (CC-MAIN-2026-39, 2023-50, 2019-35), 7 option sets (datatrove's call and the stateful dedup sets run in page order) 50,344 50,344 0
2.3.1: bare_extraction on the same pages, field by field, 2 option sets 14,384 14,384 0
2.3.1: Hypothesis pages from a tag grammar against live trafilatura, 8 option sets x 210 examples 1,680 1,680 0
2.3.1: Trafilatura's own test suite replayed call by call, 2,669 native and 1,884 delegated calls 4,553 4,553 0
2.3.1 on x86_64 (AWS EC2 c7i.4xlarge): 27,585 pages of CC-MAIN-2026-43 (21,977) and CC-MAIN-2025-08 (5,608), 8 option sets 220,680 220,680 0
1.11.0: extract on the same 7,192 pinned pages, 7 option sets 50,344 50,344 0
1.11.0: bare_extraction on the same pages, 2 option sets 14,384 14,384 0
1.11.0: Hypothesis pages against live trafilatura 1.11.0, 8 option sets x 210 examples 1,680 1,680 0
1.11.0: Trafilatura 1.11.0's own tests replayed, 108 native and 525 delegated calls 633 633 0
1.11.0 on x86_64 (AWS EC2 c7i.4xlarge): the same 27,585 pages, 8 option sets 220,680 220,680 0
Total 579,362 579,362 0
  • The suite also checks every delegated option (trafilatura's result and exactly one FallbackWarning), trafilatura's exact signatures plus compat=, extract_batch against the loop, and pyroid.patch("trafilatura")'s signatures for the installed release: 109 tests passed for 2.3.1 and 111 for 1.11.0.
  • The Rust core's own check, before the parity suite: 4,677 pages of the CC-MAIN-2026-39 development sample in 10 option sets, extract and bare_extraction, on CPython 3.13 (the dev split also on 3.12 and 3.15), and Unicode 14.0 to 17.0 vectors; 151 extra edge pages for 1.11.0, including a family with control characters. Identical in both modes.
  • python/tests/test_trafilatura*.py: 544 tests against trafilatura 2.3.1, and 395 against trafilatura 1.11.0 in its own environment, including the datatrove Trafilatura step against datatrove's own step, statistics included.

Source: the parity/ trafilatura suite in both modes (CPython 3.13, run on 2026-11-09), and parity/pyroid_parity/trafilatura_scale.py on AWS EC2 c7i.4xlarge (bench/results/aws-c7i.4xlarge/2026-11-09-warc/scale).

pyroid.nltk

Reference: NLTK 3.10.4 (with regex 2026.9.29). Rule: the same sentences, tokens and offsets, str for str, and the same exceptions (type and message).

Check Compared Identical Different
Parity suite on CPython 3.13, 1,045 checks, every document: Punkt in all 19 punkt_tab models on FineWeb (dev 10k), FineWeb-2 (the 4 pinned 250-document slices, and 5,000 documents in each of 14 more languages) and the adversarial set, realign on and off, spans, word_tokenize, both Treebank tokenizers with and without PTB brackets, their spans, detokenize round trips, the regexp tokenizers, the batch paths 14,174,241 14,174,241 0
All 19 punkt_tab models on 39,618 FineWeb and FineWeb-2 documents (848,037 sentences, 208,349 decisions), Rust replay 178,282 178,282 0
Rust replay of recorded Punkt outputs: NLTK's examples, adversarial text, fuzz, FineWeb excerpts with parameters PunktTrainer learned from FineWeb, and the case rules of CPython 3.12 to 3.15 37,936 37,936 0
Rust replay of recorded Treebank, NLTKWordTokenizer, detokenizer and align_tokens outputs: NLTK's examples, adversarial text, fuzz 39,599 39,599 0
The same functions on every sentence of 5,000 FineWeb documents and up to 1,000 documents of each of 27 FineWeb-2 files (600,372 texts), Rust replay 5,713,962 5,713,962 0
Every code point (surrogates excepted) in 13 Treebank templates, 6 of them also on Unicode 15.0, 16.0 and 17.0, and in the regexp tokenizers' templates, Rust replay 149,016,576 149,016,576 0
Total 169,160,596 169,160,596 0
  • python/tests: Hypothesis properties and NLTK's doctest examples against live NLTK for every ported function, every code point alone and in templates, batches (list, tuple, Arrow), parameters and their in-place edits, model loading with and without NLTK, fallbacks and pyroid.patch("nltk"); 1,088 tests on CPython 3.13 and 3.15t, also under pytest-run-parallel, with the GIL off on 3.15t.
  • Character classes: regex 2026.9.29's \w, \d and \s and CPython 3.11 to 3.15's isupper / islower are dumped code point by code point; the rules hard-coded in Rust (regex's \s, both engines' (?i) letters) are checked on each interpreter.

Source: the parity/ nltk suite, crates/pyroid-core/src/nltk/testdata/ replays and their large uncommitted recordings, run on 2026-11-09.

pyroid.nemo and pyroid.integrations.nemo_curator

Reference: NeMo Curator 1.4.0 on CPython 3.12 (pandas 3.1.1, regex 2026.11.9, ftfy 6.3.1, jieba 0.42.1, fasttext 0.9.3 built with g++ 12). Rule: the same score (repr and type), the same exception type and the same modified text; for the stages, the same DataFrame (values, dtypes, index).

Check Compared Identical Different
Parity suite on CPython 3.12 against live NeMo, every document of FineWeb dev 10k and the adversarial set: the scores, exceptions and modifier outputs of 74 filter, modifier and fastText cases, one str call per document 875,286 875,286 0
The same cases as list and Arrow batches (float64 or int64 arrays; the exception of the first document that raises) 1,630,796 1,630,796 0
pyroid.integrations.nemo_curator's filters and modifiers per document (score_document, keep_document, modify_document) 888,556 888,556 0
NeMo's ScoreFilter, Score, Filter and Modify with pyroid's and NeMo's classes mixed, against NeMo's stages 606 606 0
Hypothesis against live NeMo, 200 examples per filter and modifier, for pyroid.nemo and for the integration's classes 116,646 116,646 0
Lone surrogates: pyroid.nemo raises UnicodeEncodeError (pyroid's rule); the integration's classes and stages give NeMo's result 175,542 175,542 0
NeMo's own pipelines (heuristic_filter_english_pipeline.yaml; fastText language ID with the modifier benchmark's stages) in its Xenna executor, stock against YAML targets rewritten and against pyroid.patch("nemo_curator"): every output row, column and dtype 47,056 47,056 0
The YAML pipeline on 100,000 FineWeb documents at 1 and 8 CPUs: the same 95,774 kept documents (id and text) 191,548 191,548 0
Total 3,926,036 3,926,036 0
  • NeMo's RepeatingTopNGramsFilter breaks ties between equally frequent n-grams in Python set order, which follows the process's string hash seed, so its own scores change from run to run unless PYTHONHASHSEED is fixed (the references use 0); pyroid makes the choice NeMo makes in the same process.
  • The pinned checks also pass on CPython 3.14t and 3.15t with the GIL off; NeMo does not install there, so the live checks skip.
  • python/tests: Hypothesis properties of every score function and modifier against NeMo's code (copied, so they run without NeMo) and against live NeMo; the stages on Arrow, string[pyarrow], str and object columns; the batch paths never call NeMo's per-row methods; pyroid.patch("nemo_curator") with hydra configs. 517 tests on CPython 3.12 with NeMo, 167 on 3.15t, also under pytest-run-parallel.
  • Not ported: NeMo Curator 1.4.0's exact and fuzzy deduplication run on GPUs only (cuDF), and it has no CPU PII stage.

Source: the parity/ nemo suite (parity-report of 2026-11-10, linux/aarch64), parity/pyroid_parity/nemo_pipeline.py and bench/nemo_pipeline_bench.py.

Integrations: datatrove steps, pyroid clean, pyroid.patch()

Reference: datatrove 0.10.1's stock FineWeb pipeline (and the libraries pyroid.patch() stands in for). Rule: the same per-document digest of id, text, language, score, keep, reason and duplicate flag; for patch(), the same call-site output.

Check Compared Identical Different
pyroid's datatrove steps in datatrove's LocalPipelineExecutor (8 workers), FineWeb dev 10k: digest ef314b8d... 10,000 10,000 0
The same on the parity corpus (FineWeb dev and the adversarial set): digest c0c6b746... 13,232 13,232 0
pyroid clean --recipe fineweb on FineWeb dev, 3 runs (Parquet to Parquet; JSONL to JSONL.gz at 1 and 8 threads) 30,000 30,000 0
pyroid clean on the parity corpus, 2,500-row batches 13,232 13,232 0
pyroid.patch(): open_clip 3.3.0's ViT-B-32 tokenizer, token ids patched against unpatched 592 592 0
pyroid.patch(): fast-langdetect 1.1.0, lite model 600 600 0
pyroid.patch(): datatrove's FT176LID(k=3) against fasttext-predict 592 592 0
pyroid clean on the parity corpus split into 5 files: one process, --workers 2, and --shard I/3 with the four cross-shard MinHash stages (signatures, buckets, cluster, filter): digest c0c6b746..., 3 runs 39,696 39,696 0
pyroid clean on the FineWeb 100k slice in 16 files, 4 shards x 2 workers and the four stages: digest da897999..., as datatrove's chain 100,000 100,000 0
datatrove's FineWeb pipeline from 8 WARC files of CC-MAIN-2026-39 (171,595 HTML records) on AWS EC2 c7i.4xlarge: pyroid's native chain, with the pages stock datatrove lost at 8 workers left out, against stock datatrove at 8 workers, final documents after MinHash and after PII 23,503 23,503 0
The same against stock datatrove at 1 worker, with that run's lost pages left out 23,501 23,501 0
Total 254,948 254,948 0
  • Per-step tests against live datatrove (67 tests, 313 documents): every step's result, reason, rewritten text and metadata at several batch sizes and thread counts; MinhashDedupSignature's files are byte-identical, and datatrove's stages 2 to 4 on them remove the same documents.
  • The FineWeb recipe in Python (recipes.FineWeb, whole corpus and 777-document batches) and hf.fineweb (also with num_proc=2) give digest ef314b8d... on FineWeb dev too.
  • From WARC, without leaving any pages out: pyroid's steps in datatrove's executor and the native chain keep the same 23,505 documents at 8 and at 1 worker (digest d0e2ad48...). That is 2 more than stock datatrove at 8 workers, both pages stock datatrove gave up on, with 0 text differences. Stock datatrove stops extracting a page after 1 s and drops it, and which pages hit that limit changes between runs, so its own two runs kept different sets (23,503 and 23,501). pyroid has no timeout and gave the same documents in every run. The 5 pages on which trafilatura 1.11.0 itself raises are dropped by both (datatrove's child process dies on them; pyroid's step counts them as broken_process).
  • Cross-shard MinHash files: pyroid's .minhash.sig, .dups, .remove, .clusters and .sizes files are byte-identical to datatrove 0.10.1's (4 configurations, empty ranks, spills, 1 and 2 workers per bucket), and datatrove's own stages 1 to 3 on the same cleaned 100k files write the same bytes.
  • On AWS EC2 c7i.4xlarge, 4,188,430 FineWeb documents (sample/10BT, files 000 to 003) gave the same kept (3,547,373) and excluded documents in one process, with --workers 4, with 4 shards and the four stages, and when resumed after a Ctrl-C.
  • The newer drop-ins (PIIFormatter, C4BadWordsFilter, SymbolLinesFormatter, URLFilter, sentence dedup) are counted under pyroid.filters and pyroid.dedup above.

Source: python/tests/test_integrations_*.py, test_cli.py, test_cli_scale.py, test_dedup_files.py and examples/datatrove/verify.py (2026-11-06 to 2026-11-09), and bench/fineweb_warc.py on AWS EC2 c7i.4xlarge (bench/results/aws-c7i.4xlarge/2026-11-09-warc, 2026-11-09).

pyroid.warc

Reference: warcio 1.8.2 and datatrove 0.10.1's WarcReader (with faust-cchardet 3.2.0, python-magic 0.4.27 and brotlicffi 1.2.1.0), live. Rule: event by event, the same records, headers, payloads, offsets, fh.read() and fh.tell() calls, digest results, stderr writes, log warnings and exceptions (type and message), and the same documents out of datatrove's reader; for pyroid.patch("warcio"), the same traces from warcio's own names and datatrove's own reader.

The parity/ warc suite (pytest -m warc --impl pyroid.warc --parity-limit 0) compares 31,291 record traces per implementation, under up to 29 iterator options, read plans and openers, over 34 Common Crawl slices of 16 crawls from 2013 to 2026 (WARC, WET, WAT and a 2012 ARC file; 86.9 MB), warcio's 16 test files, 1,085 seeded synthetic files in 13 families (framing, truncation, bit flips, lengths, headers, HTTP, chunked, encodings, record types, digests, ARC, charsets, big records), 120 variants of the slices and Hypothesis-generated WARC streams, plus datatrove's process_record, WarcReader and executor runs: no differences, in a Docker VM (aarch64, CPython 3.13, and 3.14t with the GIL off) and natively on AWS EC2 c7i.4xlarge (x86_64). On that machine, pyroid_parity.warc_scale compared every record and document of 15 whole files (8 CC-MAIN-2026-39 WARC files; WARC files of 2016-40 and 2019-35, WET files of 2013-20 and 2023-50, WAT files of 2018-30 and 2026-43, and the 2012 ARC file): 1,169,108 records and 366,199 documents in 74 jobs, all the same.

  • CPython's gzip.GzipFile read path, as datatrove reads .warc.gz files, is emulated per interpreter: it gave CPython's bytes and errors on all 4,910 cases on CPython 3.11 to 3.15.
  • The known differences are in the guide: digest algorithm names and damaged brotli streams (41 of 2,296 corruption cases).

Known differences

Each guide lists its module's deliberate differences. Most are edge cases (inputs on which the incumbent hangs, crashes or raises a different exception type); the rest are design choices, such as returning an unchanged string as the same object.

Interpreters and Unicode versions

Results that depend on the Python version are matched per interpreter. That covers ftfy's and datatrove's use of \w, \d and isalpha, json_repair's str.isspace and friends, and Unicode normalization. pyroid carries Unicode tables for 14.0, 15.0, 15.1, 16.0 and 17.0 (CPython 3.11 to 3.15) and picks the one matching unicodedata.unidata_version.

The ftfy suites passed on CPython 3.11, 3.12, 3.13, 3.14, 3.14t, 3.15 and 3.15t. The other suites ran on 3.12 or 3.13 and on free-threaded 3.15t, as their rows say. A few references can't be loaded on a free-threaded build without re-enabling the GIL (sacrebleu imports lxml), so their live checks run on GIL builds only.

Languages

The filters and dedup suites cover English and 8 FineWeb-2 languages end to end, and the tokenizer replays cover every natively supported datatrove tokenizer: 4,013 of datatrove's 4,343 language codes, Chinese included (its parity is pyroid.jieba's) and Japanese (the SudachiPy and datatrove replays under pyroid.filters). The rest go through datatrove's own Python tokenizers (python_tokenizer=True), so they are datatrove's results by construction. See Languages.

Platforms

Every suite on this page ran on Linux aarch64, in a Docker/OrbStack VM on Apple Silicon. The Trafilatura scale runs and the WARC pipeline comparison ran on x86_64, on an AWS EC2 c7i.4xlarge, on 2026-11-09 (their rows under pyroid.trafilatura and Integrations).

Linux x86_64 and aarch64 on AWS EC2

On 2026-11-06 the parity suites ran natively, with no emulation, on two AWS EC2 instances running Amazon Linux 2023:

  • x86_64: c7i.4xlarge, Intel Xeon Platinum 8488C (Sapphire Rapids), with AVX2 and FMA;
  • aarch64: c8g.4xlarge, AWS Graviton4.

Each machine built its own release wheels and ran every suite on CPython 3.12 (the cp311-abi3 wheel, with the live oracles) and on free-threaded 3.15t (the abi3t wheel). Both machines gave the same counts, with 0 differences in every check that ran:

Suite 3.12: checks / identical 3.15t: checks / identical
ftfy 13 / 23,736 13 / 23,736
fasttext 7 / 30,474 5 / 26,466
filters 60 / 142,320 50 / 76,160
dedup 24 / 52,804 12 / 29,464
metrics 47 / 1,396,827 43 (1 skipped: live sacrebleu) / 1,388,827
json_repair 31 (2 skipped) / 184,403 30 (3 skipped) / 184,254

On 3.12, 6 live-oracle filter tests did not run: 4 for C4 bad words and 2 for the URL filter. datatrove's C4BadWordsFilter and URLFilter need optional dependencies that parity's lock file lacked at the time, so the tests failed on import before comparing anything. The lock file has them now. They have not been re-run on EC2, but CI's parity jobs run them on x86_64 and aarch64.

Both machines also regenerated the parity references. On the c8g, all 8 match the linux-aarch64 pins byte for byte. On the c7i, the fastText reference is byte-identical to the x86_64 pin made earlier under emulation. The metrics reference was not: 165 of its 694,757 results (all BLEU scores) differ by 1–2 ulps from the emulated pin. Regenerated on the c7i, the x86_64 metrics reference is identical to the aarch64 one, and it has been re-pinned.

Per-platform references

fastText's own output differs between architectures for the quantized lid.176.ftz: on 10,837 of 13,232 documents, by at most 5.96e-7, with the same labels. Its aarch64 build fuses multiply-adds that baseline x86-64 can't. pyroid reproduces each architecture's arithmetic, so the fastText reference is pinned per platform, and pyroid's x86_64 output is bit-identical to x86_64 fasttext-predict 0.9.2.4 on all 26,464 pinned predictions (max |Δp| = 0).

The C library matters as well. pyroid calls the platform's exp, expf and log, as fastText does, and glibc, musl, Apple's libm and Windows' UCRT round some of them differently: on GitHub's runners (2026-11-10), 39 of the 54 test-zoo models gave other digests on macOS arm64, 16 on macOS x86_64 and 14 on Windows x64, with the same model files. So the zoo digests are pinned per OS, C library and architecture; on each platform they were the same for every Python version and for the PGO release wheels. On Windows x64, pyroid matched fasttext-predict to the last bit over 316 (model, k) pairs. On macOS the largest difference was 1.91e-6, on a test model with weights scaled by 20, where fasttext-predict's clang build rounds its dot products differently.

The metrics reference is pinned per platform the same way, because sacrebleu's BLEU calls the C library's exp and log. Its difference came only from Rosetta 2's emulated CPU, which has SSE4.2, FMA and BMI2 but no AVX or AVX2, so glibc took other exp/log paths there. On the c7i, with AVX2 and FMA, the x86_64 and aarch64 files are the same. GitHub's x86_64 runners have AVX2 too, and CI's parity-data job regenerates the references on both architectures (when its cache misses) and checks them against their pins. pyroid and the reference share the C library, so they would agree with each other either way.

Earlier runs under emulation

Before the EC2 runs, Linux x86_64 was checked under emulation (OrbStack with Rosetta 2) on CPython 3.12 and 3.15t, with the release wheels:

  • python/tests: 1,850 passed on 3.12 and 1,678 on 3.15t (GIL off), 0 failed.
  • The ftfy, fastText, filters and dedup parity suites: 0 differences (filters: 66,160 decisions; dedup: 13,232 signatures and the removed set). The live datatrove oracle job: 44 passed.
  • The metrics suite, against the x86_64 reference pinned under emulation: 1,307,887 checks, 0 differences.

macOS, Windows and musl

The parity/ suites, with the pinned FineWeb, Common Crawl and evaluation data, run on Linux only. On macOS (x86_64 and arm64), Windows (x64 and ARM64) and musl Linux (x86_64 and aarch64), the release workflow installs each wheel in a clean virtual environment and runs pyroid's test suite (python/tests) with it, on CPython 3.11, 3.14t, 3.15 and 3.15t, before the release is published. That suite compares pyroid against the installed ftfy, anyascii, fasttext-predict, rouge-score, NLTK, json_repair, datasketch, jieba and warcio, where each installs, and checks the fastText test-zoo digests pinned per platform (above; none are pinned for Windows ARM64).

pyroid fuses fastText's multiply-adds on every aarch64 target, as Linux builds of fastText do, and orders exact ties between labels as libstdc++ orders them. pyroid's x86 builds stay at baseline x86-64 (no -C target-cpu), as bit-identity with baseline fastText builds requires.

Hardening

Beyond parity, the native code was checked for crashes and undefined behaviour on 2026-11-06, in Docker:

  • Fuzzing with cargo-fuzz: 7 targets (ftfy, filters, the tokenizer, translit, dedup, fastText model loading and inference, the Arrow importer), about 4.4 hours and 19.8 million executions, each target for at least 25 minutes. There were no crashes in pyroid's code. One resource bug was fixed: a tiny crafted fastText model could make every vector call allocate 8 GiB. pyroid now rejects it, where fastText loads it. The fuzzers' results over their final corpora also matched ftfy, datatrove, spaCy and anyascii.
  • Miri (Stacked and Tree Borrows, strict provenance, symbolic alignment, 16 seeds on the thread tests) on the Arrow FFI code: no undefined behaviour. AddressSanitizer and LeakSanitizer: clean. The pure-Rust core forbids unsafe.
  • MSRV: Rust 1.85, checked with a real 1.85 build.
  • Supply chain: cargo deny advisories over the Rust dependencies, and pip-audit over the locked test, parity and benchmark dependency sets. No findings.