Benchmarks¶

Most numbers on this page come from two AWS EC2 instances, measured on 2026-11-06 at commit f35bd9e4: a c7i.4xlarge (Intel Xeon Platinum 8488C, Sapphire Rapids) and a c8g.4xlarge (AWS Graviton4). Both have 16 vCPUs and run Amazon Linux 2023. They are VMs on real hardware, not bare metal. Every timed run was pinned to 8 CPUs; on the c7i that is one thread per core. The runs from WARC files are from two other c7i.4xlarge instances, on 2026-11-09 and 2026-11-10.

datatrove 0.10.1 and the other incumbents ran on CPython 3.13.18, and pyroid on free-threaded CPython 3.15.2t, from release wheels built on each machine (the WARC run used CPython 3.13.18 for both). The section on other modules and the earlier runs are from a Docker VM on a Mac and are labelled as such.

Summary¶

From raw Common Crawl WARC files to deduplicated documents, datatrove's FineWeb pipeline ran 9.5× faster with pyroid's steps in datatrove's own executor, pyroid's WarcReader included: 89.4 s against 848.3 s for stock datatrove with 8 workers. As one command, pyroid clean --recipe fineweb-warc took 79.6 s (10.7×). That is 8 WARC files of CC-MAIN-2026-39 (171,595 HTML records) on the c7i.4xlarge. With datatrove's reader, the first run had given 8.4× at 8 workers and 9.0× at one. Text extraction is most of the difference. Both kept the same documents, once the pages stock datatrove drops at its 1 s extraction timeout are left out of pyroid's run too.

On 100,000 FineWeb documents on the c7i.4xlarge, pyroid's FineWeb chain took 10.2 s with 8 threads, against 272.1 s for datatrove with 8 processes: about 27× faster. On one thread it took 43.4 s against 2,444.4 s, about 56× faster. The output is the same document for document. On the c8g.4xlarge the same comparison gives 29× and 50×.

On the 10,000-document dev slice the ratios are 18× (8 threads against 8 processes) and 38× (one thread against one process) on the c7i, and 20× and 45× on the c8g.

Stage by stage, on one thread against one process on the c7i, pyroid is 1.6× (fastText with lid.176.bin) to 108× (ftfy) faster than the library it replaces. The quality filters are 35–69× faster and MinHash signatures 34×.

The 100k datatrove runs and the runs of pyroid's stages in datatrove's executor were timed once each, without a warm-up, because a single-process datatrove run takes about 40 minutes. The other rows are medians of several runs.

From WARC files: datatrove's FineWeb pipeline¶

bench/fineweb_warc.py runs datatrove 0.10.1's examples/fineweb.py from WARC files to the final documents: WarcReader, URLFilter, Trafilatura(favour_precision=True), LanguageFilter (English), the Gopher, C4 and FineWeb quality filters with their exclusion writers, MinHash dedup in its four stages, and PIIFormatter. It ran three ways, each with 8 workers and with 1:

• stock datatrove, as a fresh pip install 'datatrove[io,processing]==0.10.1' resolves it (trafilatura 1.11.0), in datatrove's LocalPipelineExecutor;
• the same pipeline and executor with pyroid's drop-in steps for Trafilatura (which follows the installed trafilatura 1.11.0), URLFilter, LanguageFilter, the four quality filters, the MinHash signatures and PIIFormatter; the reader, the writers and the other three MinHash stages stay datatrove's;
• a native pyroid chain without the executor: per WARC file, in a pool of processes, pyroid.trafilatura.extract_batch(compat="1.11.0"), language ID and the filters; then MinHash dedup and PII over all files in one process.

Every mode runs one task per WARC file, so its output does not depend on the number of workers.

AWS EC2 c7i.4xlarge (Intel Xeon Platinum 8488C, 16 vCPU), Amazon Linux 2023, every run on CPUs 0 to 7; CPython 3.13.18, datatrove 0.10.1 with trafilatura 1.11.0, and a pyroid release wheel built on the instance. 8 WARC files of CC-MAIN-2026-39 (7.5 GB, 171,595 HTML records), one task per file. Each run was timed once; fastest first. CPU s is user+sys including the worker processes. The Trafilatura column is the time datatrove's statistics give that step, summed over tasks.

In every run, 5 more pages are lost because trafilatura 1.11.0 itself raises on them: datatrove's child process dies, and pyroid's step counts them as broken_process and drops them the same way.

Source: bench/fineweb_warc.py and bench/ec2/run_warc.sh, run on 2026-11-09 (bench/results/aws-c7i.4xlarge/2026-11-09-warc: runs//summary.json and compare.txt).

Stock datatrove spends most of its time in Trafilatura: 5,543 of its 6,759 CPU seconds at 8 workers. With pyroid's steps that step takes 514 s.

Stock datatrove's results vary from run to run, and pyroid's do not. datatrove extracts each page in a child process and gives up on it after 1 s (timeout=1), dropping the page and starting a new child with an empty dedup cache. Which pages hit that limit depends on the load on the machine, so the two stock runs dropped different pages (69 and 80) and kept different documents (23,503 and 23,501). pyroid's step extracts in the task's process and has no timeout, so it kept the same 23,505 documents in all four runs, with the same digest after MinHash and after PII.

To compare the documents exactly, the native chain was re-run with the pages each stock run dropped left out the same way (native --timeouts-of). Its final documents then equal stock datatrove's byte for byte, after MinHash and after PII, for both stock runs. Without that step, pyroid's steps at 8 workers keep 2 documents more than stock datatrove at 8 workers, both from pages stock datatrove timed out on, and no document's text differs.

Trafilatura is still the largest step with pyroid (514 of 792 CPU seconds at 8 workers). The next was datatrove's WarcReader, which parses the WARC files with warcio in Python: 154 s.

With pyroid's WARC reader¶

On 2026-11-10 the pipeline ran again over the same 8 files on another c7i.4xlarge, with pyroid's WarcReader drop-in (pyroid.warc) among pyroid's steps, and as one command, pyroid clean --recipe fineweb-warc --workers 8 (command line). Stock datatrove ran from the first run's exact package versions.

AWS EC2 c7i.4xlarge (Intel Xeon Platinum 8488C, 16 vCPU), Amazon Linux 2023, every run on CPUs 0 to 7; CPython 3.13.18, datatrove 0.10.1 with trafilatura 1.11.0 (the package versions of the 2026-11-09 run, installed from its freeze), and a pyroid release wheel built on the instance. The same 8 WARC files of CC-MAIN-2026-39 (7.5 GB, 171,595 HTML records), one task per file. Each run was timed once; fastest first. CPU s is user+sys including the worker processes. The step columns are the time datatrove's statistics, or the native chain's own timers, give each step, summed over tasks; the command line reports no per-step times.

In every run, 5 more pages are lost because trafilatura 1.11.0 itself raises on them, as in the 2026-11-09 run. At 1 worker, datatrove with pyroid's steps took 640.5 s and the native chain 607.8 s, with the same 23,505 documents; stock datatrove was not re-run at 1 worker.

Source: bench/fineweb_warc.py and bench/ec2/run_warc.sh, run on 2026-11-10 (bench/results/aws-c7i.4xlarge/2026-11-10-warc: runs//summary.json and compare.txt).

The reader step went from 170 CPU seconds in stock datatrove (154 s in the first run with pyroid's other steps) to 71 s. Stock datatrove kept the same 23,503 documents as its first run at 8 workers, with the same digests; every pyroid run kept the same 23,505, and with the 73 pages stock datatrove lost left out the same way, the native chain's documents equal stock datatrove's byte for byte.

The reader alone, on one core, from bench/fineweb_warc.py reader:

The same machine and files, one process on CPU 0: each reader over each file, timed on a pass that only counts the documents, then checked on a second pass that digests them (id, url, date, text). Every reader gave the same 172,376 documents: the pages process_record returns, before datatrove skips the 781 with no text.

Part of the time is the same for every reader: faust-cchardet's guess for the pages that are not valid UTF-8, and making the Python strings.

Source: bench/fineweb_warc.py reader, run on 2026-11-10 (bench/results/aws-c7i.4xlarge/2026-11-10-warc/runs/reader/summary.json).

The results, per-step times and digests are in bench/results/aws-c7i.4xlarge/2026-11-09-warc/ and 2026-11-10-warc/; bench/README.md has the commands.

End to end: the FineWeb chain¶

datatrove 0.10.1's FineWeb-style chain: fix_text → LanguageFilter (English, lid.176.bin) → Gopher repetition → Gopher quality → C4 quality (FineWeb settings) → FineWeb quality → MinHash dedup. It ran three ways, each with 1 and 8 workers:

• datatrove in its own LocalPipelineExecutor, on CPython 3.13;
• the same pipeline in datatrove's executor with every stage swapped for pyroid's, on CPython 3.13;
• pyroid's native chain: the same steps as pyroid calls in one process, on Arrow input, with threads=1 and threads=8, on free-threaded CPython 3.15t (bench/chain.py).

The output is checked with a per-document digest of each document's text, language, score, keep/drop decision, reason and duplicate flag. On the dev slice, datatrove and pyroid's stages in its executor give the same digest (ef314b8d…) with 1 and 8 processes, and so does the native chain with 8 threads. On the 100k corpus, the 8-process runs and the native chain with 8 threads give da897999…. Both machines give the same digests, and pyroid's output never depends on threads.

Each run is a whole process under hyperfine, so interpreter start-up, imports, model loading and reading the input are included. The 8-process rows give each process its own copy of the fastText model, and the peak RSS column is for one process, not the total.

100,000 documents¶

The pinned 100,000-document FineWeb slice.

c7i.4xlarge, 100,000 documents. Wall seconds for the whole chain, fastest first.

Source: bench/results/aws-c7i.4xlarge/2026-11-06/headline.

c8g.4xlarge, 100,000 documents. Wall seconds for the whole chain, fastest first.

Source: bench/results/aws-c8g.4xlarge/2026-11-06/headline.

10,000 documents¶

The dev slice: the first 10,000 documents of the 100k slice. datatrove and pyroid's stages in its executor are the median of 3 runs, the native chain the median of 5, each after a warm-up.

c7i.4xlarge, 10,000 documents (dev slice). Wall seconds for the whole chain, fastest first.

Source: bench/results/aws-c7i.4xlarge/2026-11-06/dev.

c8g.4xlarge, 10,000 documents (dev slice). Wall seconds for the whole chain, fastest first.

Source: bench/results/aws-c8g.4xlarge/2026-11-06/dev.

Per stage¶

Each stage alone, on the dev slice. The incumbents ran in one process on CPython 3.13, as a per-document loop over their largest measured batch: 10,000 documents for fix_text, language ID and translit, and 1,000 for the filters and MinHash. pyroid made one call over the same documents as a list[str], with a pool of 1 or 8 threads. Every row first passed its output gate: identical ftfy and anyascii output, the same fastText labels with |Δp| ≤ 1e-6, datatrove's filter decisions and text, and datatrove's MinHash signatures.

The same list is reused between timed calls, so CPython's cached UTF-8 form of each str is warm. The earlier translit run rebuilt the strings every time.

c7i.4xlarge, 10,000 documents (dev slice). µs per document; the speedup is pyroid on one thread against the incumbent in one process.

Source: bench/results/aws-c7i.4xlarge/2026-11-06/dev.

c8g.4xlarge, 10,000 documents (dev slice). µs per document; the speedup is pyroid on one thread against the incumbent in one process.

Source: bench/results/aws-c8g.4xlarge/2026-11-06/dev.

On the c7i, with 10,000 documents per call, the filters and MinHash took 92.2–173 µs per document on one thread and 12.7–23.9 on 8: larger batches use the pool better. With Arrow input, fix_text_batch took 6.63 µs per document on one thread and 1.31 on 8, and translit.anyascii 1.79 and 0.60.

Other MinHash libraries use other hash families, so they don't reproduce datatrove's signatures or decisions; they were gated on near-duplicate precision and recall ≥ 0.90 instead. On the c7i, rensa 0.5.0 took 144 µs per document (RMinHash.update, 112 permutations) and 161 (RMinHash.from_token_sets), and datasketch 2.0.0 took 436 (MinHash.bulk, num_perm=112).

Each run's CONSOLIDATED.md has every row with its interquartile range, pyroid on CPython 3.13, and each incumbent stage alone in datatrove's executor at 8 processes: c7i, c8g.

Other modules, languages and integrations¶

These runs are not in the EC2 benchmark yet. They come from each module's own benchmark in a Docker VM (OrbStack, Linux aarch64, 12 vCPUs on an Apple Silicon Mac) that other work was also using, so treat them as indicative. Each was gated on identical output before timing. Their conditions differ from table to table (interpreters, batch sizes, how loaded the VM was), so compare rows within a table, not across tables.

pyroid.metrics: ROUGE, BLEU and chrF¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. Milliseconds for the whole workload, median of the runs (a single run for the reference libraries on 100k pairs). Every row was first checked identical to the 1-process reference. The reference libraries ran in 1 process and in 8 forked processes, pyroid in one process with threads=1 and threads=8. CPython 3.13, release wheel; other work kept the load average between 4 and 19. ROUGE scores CNN/DailyMail lead-3 against the highlights with rouge1/2/L/Lsum (the 100k pairs are the 11,490 pairs repeated). BLEU and chrF use WMT21 segments, the first reference and default settings. Guide: metrics.

Source: bench/results/orbstack-aarch64-12cpu/2026-11-06-metrics (bench/metrics.py).

pyroid.json_repair¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. Single documents, best time per call, json_repair 0.64.5 on CPython 3.12 (3.15t was within ±10%). Release wheel; other work kept the load average between 16 and 24. Guide: json_repair.

Source: bench/json_repair_bench.py, run on 2026-11-06.

pyroid.json_repair: batches¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. Documents per second over 10,000 broken documents of about 1.2 KB, on free-threaded CPython 3.15t with the GIL off. The shared, loaded VM limits the 8-thread rows. Guide: json_repair.

Source: bench/json_repair_bench.py, run on 2026-11-06.

pyroid.datasketch: MinHash signatures¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. Seconds for the first 10,000 FineWeb dev documents (5.1 million word 5-gram shingles), 128 permutations, best of 3, on free-threaded CPython 3.15t with the GIL off. datasketch 2.0.0 with numpy 2.5.4; its 8-thread rows split the documents over a thread pool. Every pyroid result was first checked for identical signatures. Load average 7 to 10. Guide: datasketch.

Source: bench/results/orbstack-aarch64-12cpu/2026-11-07-datasketch (bench/datasketch_bench.py).

pyroid.datasketch: MinHashLSH¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. Milliseconds for 10,000 affine32 signatures, 128 permutations, same run. The loop rows call insert or query per document with LeanMinHash objects; the batch rows pass the signature matrix. Every query result set was first checked against datasketch's. Guide: datasketch.

Source: bench/results/orbstack-aarch64-12cpu/2026-11-07-datasketch (bench/datasketch_bench.py).

pyroid.jieba¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. µs per document over 1,000 FineWeb-2 cmn_Hani documents (the slice four times, about 1,450 characters each), on free-threaded CPython 3.15t with the GIL off, release wheel, median of 5 runs (2 for jieba), load average 1.5 to 3. Every document's words were first checked against jieba 0.42.1's. jieba's 8 processes are a warmed fork Pool(8); its 8 threads run in parallel on 3.15t. Guide: jieba.

pyroid builds the Python lists of words on the calling thread (about 35 µs per document), which limits the 8-thread rows. A fresh interpreter reaches its first segmented sentence in 27 ms with pyroid (import included) and about 410 ms with jieba.

Source: bench/results/orbstack-aarch64-12cpu/2026-11-07-jieba (bench/jieba_bench.py).

pyroid.nltk¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. µs per document over the first 1,000 documents of the FineWeb dev slice (3,018 characters each on average), English model, release wheels, median of 5 runs (2 for NLTK), load average 3 to 5. Every document's output was first checked against NLTK 3.10.4's. NLTK's 8 processes are a warmed fork Pool(8); its 8 threads run in parallel only on 3.15t. Guide: nltk.

Speedups are against NLTK on one thread. pyroid builds the Python lists of sentences and tokens on the calling thread, which limits the 8-thread rows; against NLTK on 8 processes they are 35× (sent_tokenize) and 14× (word_tokenize) on 3.13. The first sentence in a fresh interpreter takes 177 ms with pyroid and 203 ms with NLTK on 3.13: pyroid imports NLTK to find the model when NLTK is installed.

Source: bench/results/orbstack-aarch64-12cpu/2026-11-09-nltk (bench/nltk_bench.py).

pyroid.trafilatura¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. Milliseconds per page over 1,000 pages of the CC-MAIN-2026-39 development sample (157,000 characters each on average), CPython 3.13, release wheel, load average 2 to 4. Each mode against its own trafilatura release, with lxml 6.1.3 and jusText 3.0.2, and every page's text first checked identical. datatrove's call is favor_precision=True, include_comments=False, deduplicate=True; with dedup, pages are extracted in page order, so threads only parse ahead. Guide: trafilatura.

Source: the trafilatura port's timing runs on 2026-11-09, whose scripts are not in the repository.

Languages: FineWeb-2 filters and MinHash¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. µs per document over 1,000 documents per language (each FineWeb-2 slice four times), median of 5 runs (2 for datatrove). Every timed document was first checked for the same keep/drop, reason and signatures. The filters are FineWeb-2's chain (Gopher repetition, FineWeb quality, Gopher quality) with each language's settings; MinHash computes FineWeb-2's signatures. datatrove's 8 processes are a warmed fork Pool(8). CPython 3.13, release wheel, load average 6 to 16. Guide: languages.

* An outlier: in this run pyroid was slower with 8 threads than with 1 while the load rose. It needs a rerun before it can be quoted. † Rerun on 2026-11-07 with the native Chinese tokenizer (pyroid.jieba), load average 1 to 2; through python_tokenizer=True it had run at 1.2× / 0.9× on one thread and 0.4× with 8. ‡ Rerun on 2026-11-09 with the native Japanese tokenizer (a port of SudachiPy 0.7.0, sudachidict-core 20261015), load average 0.6 to 1.9; through python_tokenizer=True it had run at 1.2× / 0.9× on one thread and 0.3× / 0.2× with 8. SudachiPy's lattice dominates both sides: pyroid's whole MinHash (1,151 µs per document, one thread) takes less than SudachiPy's own tokenize() on the same chunks (1,583 µs). § Measured on 2026-11-09 with the native Vietnamese tokenizer (a port of pyvi 0.1.1 through spaCy) and the native Thai one (a port of PyThaiNLP 5.4.0, reading the installed pythainlp's word list and model), load average 2.1 at the start and 7.4 after the 8-worker runs. Through python_tokenizer=True, before the ports, they ran at 1.5× / 1.5× (Vietnamese) and 1.5× / 1.6× (Thai) on one thread, and 0.4× / 0.2× and 0.6× / 0.7× with 8; that run overlapped other builds on the VM (load average 3.5 to 23), so only its ratios are given.

Source: bench/results/orbstack-aarch64-12cpu/2026-11-06-multilingual, 2026-11-07-jieba, 2026-11-09-japanese and 2026-11-09-vithai (bench/multilingual.py).

More datatrove steps¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. Each datatrove step alone on the FineWeb dev slice, in one process and in a warmed multiprocessing.Pool(8), against pyroid's native call at 1 and 8 threads. Medians of 5 runs for pyroid and 2 for datatrove; release wheel, CPython 3.13, taskset -c 0-7, load average about 8 (about 22 for the URL filter). Where datatrove wasn't measured at 8 processes, the 8-thread speedup is against one process. Guide: datatrove.

Source: a one-off run on 2026-11-06, whose scripts are not in the repository.

datatrove's FineWeb pipeline with pyroid's steps swapped in¶

Docker/OrbStack VM on Apple Silicon (12 vCPU), indicative. Whole runs on the FineWeb dev slice (10,000 documents unless noted): wall seconds, and user+sys CPU seconds including children, from bash's time. CPython 3.13, release wheel, taskset -c 0-7. The VM was heavily shared (load average 18 to 30), which slows stock datatrove the most, so each ratio comes from a pair of runs in the same minute (stock, then pyroid). pyroid's runs include datatrove's executor phases, worker start-up, Parquet IO and loading the 125 MiB model; pyroid clean includes interpreter start-up, IO and writing both outputs. Every output has datatrove's per-document digest. Guide: datatrove.

Source: examples/datatrove/verify.py and the pyroid CLI, timed with bash's time on 2026-11-06.

Method¶

The benchmark harness is in bench/. It enforces these rules in code:

1. Same output before any timing. Every timed row first passes an equivalence gate against pinned reference outputs: ftfy byte-identical; language ID with the same top-k labels and |Δp| ≤ 1e-6; filters with the same keep/drop, reason and rewritten text; MinHash with identical signatures. A mismatch aborts the run. The end-to-end runs must also give datatrove's per-document digest.
2. Pinned data: FineWeb slices of 10k and 100k documents, 8 FineWeb-2 languages, lid.176.bin and lid.176.ftz, and a seeded adversarial set of 3,232 documents, each pinned by sha256 in bench/manifest.json.
3. Stage rows use pyperf. One value is one call over the batch, reported as the median [interquartile range] per document over all of a row's values. Rows with more than one thread are pinned to 8 CPUs.
4. End-to-end rows use hyperfine 2.0 over whole processes pinned to 8 CPUs: datatrove's own LocalPipelineExecutor(tasks=N, workers=N) with N = 1 and 8, and bench/chain.py for pyroid's native chain.
5. CPU seconds and peak memory are recorded for every row.
6. Charts are horizontal bars on a linear axis, fastest on top, with the incumbents and close rivals named.

To reproduce on Linux (or in a Linux container), install the pyroid wheel into a CPython 3.13 environment that has the bench dependencies and into a free-threaded CPython 3.15t one (BENCH_PY313, BENCH_PY315T), then:

$ cd bench
$ uv run --frozen python fetch.py                     # the pinned inputs
$ BENCH_GIT_SHA=$(git rev-parse --short HEAD) bash run_consolidated.sh

This writes CONSOLIDATED.md, consolidated.json (the source of the tables above) and the charts into bench/results/<machine>/<date>-consolidated/. bench/README.md documents every option, including BENCH_RIGOROUS=1 for a quiet machine, and bench/ec2/ has the scripts for the EC2 runs.

Where the CPU goes¶

How datatrove's time splits between the stages, from the harness's first baseline run in the Docker VM (CPython 3.13, one process, the first 10k FineWeb documents, 30.5 M characters). The share of the chain is the stage's single-process time over the sum of all stages.

The four filters take about 79% of the CPU and MinHash about 18%; ftfy and language ID together take about 3%. Making fix_text 10× faster on its own would make this pipeline only about 1.02× faster. datatrove tokenizes English with spaCy's blank("en") tokenizer for both the filters and MinHash, and in MinHash spaCy is about 70% of the cost. The same baseline run measured a peak of 2.36 GiB of anonymous memory at 8 processes.

On the c7i.4xlarge, datatrove ran the whole chain on the dev slice in 212.8 s in one process and 35.4 s with 8 (End to end).

Thread scaling¶

One process on free-threaded CPython 3.15t (GIL off), one call over 10,000 FineWeb documents, on the c7i.4xlarge (µs per document):

The earlier module runs in the Docker VM measured these speedups at 8 threads over 1:

Arrow input scales best. With a list on a free-threaded build, pyroid builds each string's UTF-8 form on the pool (for batches of at least 128 documents and 1 Mi code points), but it still builds the output strings one after another on the calling thread. That limits transforms that change most documents. A different global allocator (mimalloc) was measured and not adopted.

Earlier runs¶

These ran in the Docker VM described above, before the EC2 runs, and cover what the EC2 run doesn't: other languages, other calls and options, whole pipelines and memory. They are indicative.

The consolidated run of 2026-11-05 in that VM gave 26× (8 threads against 8 processes) and 49× (one thread against one process) on the dev slice; its CONSOLIDATED.md has the details. The EC2 numbers above replace it.

pyroid.ftfy¶

fix_text called once per document (the consolidated runs time fix_text_batch), on one thread, best of 3, the same documents for both:

An identical earlier run of the FineWeb rows gave 71× (3.12) and 93× (3.15).

pyroid.fasttext¶

The harness's stages.py language-ID rows at k=5, against fasttext-predict 0.9.2.4: median ms per document, 1,000 FineWeb documents with newlines replaced by spaces, CPython 3.13, gated on identical output.

load_model in a fresh process took 50–60 ms (.bin) and 4–7 ms (.ftz) for pyroid, against 115–205 ms and 38–68 ms.

Threads in one process (free-threaded CPython 3.15t, 10k documents, best of 5, ms):

Single-threaded fastText (fasttext-predict, CPython 3.13) took 3,426 ms (.bin) and 6,354 ms (.ftz) for the same 10k documents.

With eight threads in one process, the model and predictions added 132 MiB (.bin) or 1 MiB (.ftz), against about 1.04 GiB for eight per-process copies of the model. The whole process peaked at 473 MiB, 320 MiB of which was the corpus itself: about 25% of the 1.87 GiB that datatrove's 8-process language-ID stage holds.

pyroid.filters¶

FineWeb's four filters chained through apply, on one core, CPython 3.13, the first 2,000 FineWeb documents, median of 5 (datatrove: median of 2), every result checked equal to datatrove's before timing: 23,875 µs per document for datatrove against 277 µs for pyroid on a list (363 on Arrow), about 86×.

datatrove's chain costs less than the sum of its filters because it caches tokenization between them, and apply shares it too. A separate check of three separate filter calls measured 0.43 ms per document on one thread, about 52× datatrove, and 0.097 ms per document with 8 threads.

pyroid.dedup¶

The whole pipeline, MinHashDedup.fit (sha1/64, free-threaded CPython 3.15t, median of 3 fresh processes, wall seconds):

The 1-process datatrove figure was measured earlier, under heavier load. With xxhash/64 instead of sha1/64, signatures took 0.085–0.091 ms per document (1,000 FineWeb documents, CPython 3.13, one thread).

For exact duplicates (100k documents, CPython 3.15t), Python's own set over a list[str] beat pyroid.dedup.exact_dedup: 0.020 s against 0.071 s on one thread. exact_dedup is for Arrow input, normalized comparison and streaming state.

pyroid.translit¶

One thread, CPython 3.12, best of at least 5 runs. The str objects were rebuilt before every call, because CPython caches a string's UTF-8 form and that makes repeated runs 1.1–2.9× faster. pyroid's output was first checked equal to the anyascii package's on every document. µs per document:

On CPython 3.15 (GIL build) the speedups were 11–34× against anyascii and 19–68× against Unidecode. pyroid.translit.unidecode's output is not Unidecode's; see how different it is.

What's pending¶

• The metrics, json_repair, datasketch, jieba, nltk and trafilatura modules, the FineWeb-2 languages and the newer datatrove steps in the gated harness on EC2. Their numbers above come from each module's own run in the Docker VM; only trafilatura has also run on EC2, inside the WARC pipeline.
• Runs with more than 8 threads. Every EC2 run used 8 of the 16 vCPUs.