Languages¶

pyroid's filters and dedup reproduce datatrove 0.10.1 beyond English. datatrove picks a word tokenizer for each of its 4,343 language codes (its tokenizer_assignment.csv), and every word count, ratio, n-gram and shingle depends on that tokenizer's words. pyroid ports the rule-based tokenizers, jieba for Chinese, SudachiPy for Japanese, pyvi for Vietnamese and PyThaiNLP's tokenizers for Thai to Rust. That makes 4,049 of the 4,343 codes native, 98.5% of FineWeb-2's bytes, with datatrove's decisions (parity), in Rust on every thread. Japanese needs a SudachiDict dictionary installed (Japanese) and Thai needs PyThaiNLP (Thai); without either, 4,025 codes (89.9%) are native. The other 294 codes (318 without either) use statistical segmenters that pyroid does not port. For them, python_tokenizer=True calls datatrove's own tokenizer and runs the rest in Rust.

The largest languages are native: Russian, German, Spanish, French and Arabic together are 48% of FineWeb-2's bytes, Chinese another 11.6%, Japanese 7.2%, Vietnamese 1.5% and Thai 1.3%.

The language table¶

datatrove 0.10.1's tokenizer for each code, and the share of FineWeb-2's training data (1,913 subsets, 22.9 TB) in those codes:

• Native codes are tokenized in Rust. For the spaCy languages pyroid ports spaCy 3.8.18's prefix, suffix, infix and URL rules, each language's tokenizer exceptions, the special-case matcher and the sentencizer, extracted from spaCy's own objects (48,032 rules). For the Indic languages it ports indic-nlp-library's trivial_tokenize and sentence_split, and for the whitespace codes datatrove's WhitespaceTokenizer. Chinese is spaCy's ChineseTokenizer over pyroid.jieba, the port of jieba 0.42.1, with jieba's default dictionary. Japanese is datatrove's JapaneseTokenizer (SudachiPy in split mode A) and the sentencizer, over a port of SudachiPy 0.7.0's tokenizer that reads an installed SudachiDict dictionary. Vietnamese is spaCy's VietnameseTokenizer with pyvi 0.1.1: its syllable pattern, its CRF (pyvi's model and n-gram list are bundled) and its rules for joining syllables into words. Thai is PyThaiNLP 5.4.0's newmm (maximal matching over its word list, within Thai character clusters) and crfcut (a CRF over newmm's words), reading the installed PyThaiNLP's word list and sentence model. Both CRFs run on a Rust reader of CRFsuite's model files. Words and sentences match datatrove's.
• The python_tokenizer=True codes need a segmenter pyroid does not port: Stanza's neural models, Kiwi, khmer-nltk, LaoNLP, botok and pyidaungsu. Without python_tokenizer=True these codes raise NotImplementedError, and so do the Japanese codes when no dictionary is installed and the Thai codes without PyThaiNLP.
• Codes datatrove does not know raise datatrove's ValueError when the call starts.

How to choose¶

Look the code up in pyroid.filters.languages() (the same function is pyroid.dedup.languages). Its keys are what language= accepts, in datatrove's spellings ("deu_Latn", "deu", "de", ...). Each value says which tokenizer datatrove uses and whether pyroid's is native (here with a Japanese dictionary and PyThaiNLP installed; without the dictionary the 4 Japanese codes are native=False, and without PyThaiNLP the 20 Thai codes):

import pyroid.filters as F

langs = F.languages()  # {code: Language(tokenizer, arg, native)}, sorted by code
langs["deu_Latn"]  # Language(tokenizer='SpaCyTokenizer', arg='de', native=True)
langs["hin_Deva"]  # Language(tokenizer='IndicNLPTokenizer', arg='hi', native=True)
langs["kat_Geor"]  # Language(tokenizer='WhitespaceTokenizer', arg='', native=True)
langs["cmn_Hani"]  # Language(tokenizer='SpaCyTokenizer', arg='zh', native=True)
langs["jpn_Jpan"]  # Language(tokenizer='SpaCyTokenizer', arg='ja', native=True)
langs["vie_Latn"]  # Language(tokenizer='SpaCyTokenizer', arg='vi', native=True)
langs["tha_Thai"]  # Language(tokenizer='ThaiTokenizer', arg='', native=True)
langs["kaz_Cyrl"]  # Language(tokenizer='StanzaTokenizer', arg='kk', native=False)
sum(lang.native for lang in langs.values())  # 4049 (of 4343)
F.english_languages()[:4]  # ['en', 'en_Latn', 'eng', 'eng_Latn']: SpaCyTokenizer("en")

Then:

1. If native=True, pass language=. Nothing else is needed (no datatrove, no spaCy), and the work runs in Rust on every thread.
2. If native=False, pass language= and python_tokenizer=True, with datatrove 0.10.1 and the segmenter installed. The NotImplementedError you get without them gives the pip install line. (For the Japanese codes, pip install sudachidict-core makes them native instead, and for the Thai codes pip install pythainlp.) Each document is tokenized by datatrove's own tokenizer, in Python, on the calling thread, so the call runs at that tokenizer's speed and does not scale with threads=. On 8 cores datatrove's 8 processes can beat it (Speed). For a large corpus in such a language, run several processes, or keep datatrove for those steps.
3. If the code is not in the table, the call raises datatrove's ValueError. Pass a code datatrove knows, or a tokenizer object (below).

language= may also be a word tokenizer object, as datatrove allows: a datatrove WordTokenizer instance, or any object with word_tokenize(text) (and span_tokenize(text) for C4). It is called in Python, as on the python_tokenizer=True path and with the same FallbackWarning, but needs no python_tokenizer=True.

What takes language=¶

What python_tokenizer=True does, from the modules' documentation (pyroid/filters.pyi, pyroid/dedup.pyi):

• Each document is tokenized by datatrove's own tokenizer for language, in Python, on the calling thread. It needs datatrove 0.10.1 and the segmenter installed, and pyroid.PyroidError says what is missing.
• The rest runs in Rust on those words, so the decisions are datatrove's. That is the filter math, or simplify_text, shingling, MinHash, bucketing and clustering. threads= applies to the Rust part only.
• The filters only tokenize the documents that reach a word-based check.
• A C4Quality step (which splits sentences with the tokenizer) runs as datatrove's own C4QualityFilter. In apply, later steps then see (and tokenize again) its rewritten text, as in a datatrove pipeline, and consecutive steps with the same tokenizer share one tokenization per document.
• Each function emits one pyroid.FallbackWarning per language (C4 steps their own; MinHashDedup once, when the index is made).
• It also works for native codes, where it uses datatrove's tokenizer. That is slower and no more exact, so it is only needed for the others.

Sentence dedup has no Python-tokenizer path. A code whose datatrove tokenizer pyroid does not port raises NotImplementedError, unless no tokenizer is needed (split_sentences=False and min_doc_words, min_num_sentences and min_words_to_remove_span all <= 0). The datatrove sentence-dedup steps in pyroid.integrations.datatrove run datatrove's own code for such codes, with a FallbackWarning.

Native languages: German and French¶

FineWeb-2 sets thresholds and stop words per language; pass them as you would to datatrove:

import pyroid.dedup as D
import pyroid.filters as F

de = [
    "Die Stadt liegt am Ufer des Flusses und war seit dem Mittelalter wichtig für Kaufleute aus "
    "vielen Ländern. Heute leben hier etwa achtzigtausend Menschen. Die Altstadt mit ihren "
    "engen Gassen und alten Häusern zieht jedes Jahr viele Besucher an. Im Sommer finden auf "
    "dem Marktplatz Konzerte statt, und im Winter gibt es einen kleinen Weihnachtsmarkt. Der "
    "Bahnhof liegt nicht weit vom Zentrum entfernt, und die Züge fahren regelmäßig in die "
    "größeren Städte der Region. Wer die Umgebung erkunden möchte, kann Fahrräder mieten und "
    "den Wegen am Fluss folgen.",
    "Jetzt kaufen! Jetzt kaufen! Jetzt kaufen!",
]
stop_words = ["der", "die", "das", "und", "mit", "nicht", "es", "in", "zu", "den"]
F.gopher_quality_reasons(de, language="deu_Latn", stop_words=stop_words)
# [None, 'gopher_short_doc']

fr = [
    "Le village se trouve au pied de la montagne, à quelques kilomètres de la frontière. "
    "On y vient pour les randonnées en été et pour le ski en hiver. L'église date du "
    "douzième siècle et le marché a lieu chaque samedi matin devant la mairie. "
    "Les habitants parlent volontiers de l'histoire de leur région, qui a longtemps vécu "
    "de l'agriculture et de l'élevage. Aujourd'hui, le tourisme est devenu la principale "
    "source de revenus, mais les fermes n'ont pas disparu.",
]
result = F.apply(
    fr,
    [
        F.GopherRepetition(language="fra_Latn"),
        F.FineWebQuality(language="fra_Latn"),
        F.GopherQuality(language="fra_Latn", stop_words=["le", "la", "de", "et", "les"]),
    ],
)
result.keep  # [True]

near = [de[0], de[0].replace("achtzigtausend", "neunzigtausend"), fr[0]]
D.minhash_dedup(near, language="deu_Latn")  # [True, False, True]

datatrove's GopherQualityFilter(language="deu_Latn", stop_words=stop_words) gives the same two reasons, and the same French chain keeps the document. The MinHash signatures are datatrove's.

Chinese¶

cmn_Hani and the other SpaCyTokenizer("zh") codes are native. datatrove segments them with jieba through spaCy, and pyroid runs the same segmentation in Rust, so neither datatrove nor jieba is needed:

import pyroid.dedup as D
import pyroid.filters as F

zh = [
    "这座城市位于河边，自古以来就是重要的商业中心。今天，这里大约有八万人口。"
    "老城区的街道很窄，房子也很古老，每年都吸引很多游客。夏天，广场上有音乐会；"
    "冬天，有一个小小的圣诞市场。火车站离市中心不远，火车定期开往附近的大城市。"
    "想要了解周边地区的人，可以租一辆自行车，沿着河边的小路骑行。",
]
zh.append(zh[0].replace("八万", "九万"))  # a near-duplicate

F.fineweb_quality(zh, language="cmn_Hani")  # [True, True]
D.minhash_dedup(zh, language="cmn_Hani")  # [True, False]

The words are those of jieba 0.42.1 with its default dictionary, as datatrove gets them from an unmodified jieba. Words added with pyroid.jieba.add_word or load_userdict do not change them. To tokenize with a customised jieba, pass python_tokenizer=True and change jieba itself.

Japanese¶

jpn_Jpan and the other SpaCyTokenizer("ja") codes (ja, ja_Jpan, jpn) are native when a SudachiDict dictionary is installed. datatrove segments them with SudachiPy in split mode A, through spaCy. pyroid ports SudachiPy 0.7.0's tokenizer and datatrove's handling around it (40,000-byte chunks, the alignment of morphemes with the text, the sentencizer) to Rust, so neither datatrove, spaCy nor SudachiPy is needed, only the dictionary:

$ pip install sudachidict-core

import pyroid.dedup as D
import pyroid.filters as F

F.languages()["jpn_Jpan"]  # Language(tokenizer='SpaCyTokenizer', arg='ja', native=True)

ja = [
    "この町は川のほとりにあり、昔から商業の中心として栄えてきました。今では約八万人が暮らしています。"
    "旧市街の通りは狭く、古い家が並び、毎年多くの観光客が訪れます。夏には広場で音楽会が開かれ、"
    "冬には小さなクリスマスマーケットが立ちます。駅は中心部から近く、近くの大きな町へ電車が定期的に走っています。"
    "周辺を見て回りたい人は、自転車を借りて川沿いの小道を走ることができます。",
]
ja.append(ja[0].replace("八万", "九万"))  # a near-duplicate

F.fineweb_quality(ja, language="jpn_Jpan")  # [True, True]
D.minhash_dedup(ja, language="jpn_Jpan")  # [True, False]

pyroid does not ship the dictionary: SudachiDict core's system.dic is 202 MB (the sudachidict-core wheel is 77 MB). pyroid reads the system.dic of the installed sudachidict-core, version 20261015 or newer (the SudachiPy 0.7 format), and finds the package without importing it or SudachiPy. If PYROID_SUDACHI_DICT is set, the file it names is read instead. The dictionary is read once per process, into memory (about 200 MB), the first time a filter pipeline, MinHash or sentence dedup is built for a Japanese code. A dictionary in an older format raises ValueError, and a path that cannot be read OSError. languages() does not open the file, so with PYROID_SUDACHI_DICT naming a file that cannot be read, the Japanese codes still show native=True and the datatrove steps do not fall back to datatrove's code. Parity was checked with sudachidict-core 20261015 (Parity).

One error differs in type: where SudachiPy's input limit stops datatrove (a 40,000-byte chunk whose normalization passes 65,535 bytes), datatrove raises sudachipy.errors.SudachiError and pyroid raises ValueError with the same message.

Without a dictionary these codes are not native: they raise NotImplementedError, whose message says to install sudachidict-core, unless you pass python_tokenizer=True as in the Python tokenizer section.

Vietnamese¶

vie_Latn and the other SpaCyTokenizer("vi") codes are native, with nothing to install. datatrove segments them with spaCy's VietnameseTokenizer, which labels syllables with pyvi 0.1.1's CRF and joins them into words ("Việt Nam", "Hồ Chí Minh"). pyroid ports the syllable pattern, pyvi's features and spaCy's joining rules to Rust and bundles pyvi's model and n-gram list (MIT, 1 MB), so neither datatrove, spaCy, pyvi nor scikit-learn is needed:

import pyroid.dedup as D
import pyroid.filters as F

vi = [
    "Thị trấn nằm bên bờ sông và từ thời Trung cổ đã là nơi buôn bán quan trọng của các thương "
    "nhân đến từ nhiều nước. Ngày nay ở đây có khoảng tám mươi nghìn người sinh sống. Khu phố cổ "
    "với những con đường hẹp và những ngôi nhà cũ thu hút rất nhiều du khách mỗi năm. Vào mùa hè, "
    "trên quảng trường có các buổi hòa nhạc, còn vào mùa đông có một chợ Giáng sinh nhỏ. Nhà ga "
    "không xa trung tâm, và tàu hỏa chạy thường xuyên đến các thành phố lớn trong vùng. Ai muốn "
    "khám phá vùng xung quanh có thể thuê xe đạp và đi theo những con đường nhỏ dọc bờ sông.",
]
vi.append(vi[0].replace("dọc bờ sông", "dọc bờ hồ"))  # a near-duplicate

F.fineweb_quality(vi, language="vie_Latn")  # [True, True]
D.minhash_dedup(vi, language="vie_Latn")  # [True, False]

Case tests and lowercasing in pyvi's features follow the running interpreter's Unicode version, as they do in Python.

Thai¶

tha_Thai and the other ThaiTokenizer codes are native when PyThaiNLP is installed. datatrove splits words with PyThaiNLP's newmm-safe (maximal matching over its word list, within Thai character clusters, in chunks for long texts) and sentences with crfcut (a CRF over newmm's words). pyroid ports both, and the CRFsuite tagger crfcut uses, to Rust. It does not ship PyThaiNLP's word list (words_th.txt, CC0) or sentence model (sentenceseg_crfcut.model, CC-BY-4.0): it reads those of the installed pythainlp, which datatrove's Thai tokenizer needs anyway, so the words follow the word list of the PyThaiNLP you have:

$ pip install pythainlp

import pyroid.dedup as D
import pyroid.filters as F

th = [
    "เมืองนี้ตั้งอยู่ริมแม่น้ำ และเป็นศูนย์กลางการค้าที่สำคัญมาตั้งแต่สมัยโบราณ "
    "ปัจจุบันมีผู้คนอาศัยอยู่ราวแปดหมื่นคน ย่านเมืองเก่ามีถนนแคบและบ้านเรือนเก่าแก่ "
    "ซึ่งดึงดูดนักท่องเที่ยวจำนวนมากทุกปี ในฤดูร้อนมีการแสดงดนตรีที่ลานกลางเมือง "
    "ส่วนในฤดูหนาวมีตลาดคริสต์มาสเล็ก ๆ สถานีรถไฟอยู่ไม่ไกลจากใจกลางเมือง "
    "และมีรถไฟวิ่งไปยังเมืองใหญ่ในภูมิภาคเป็นประจำ "
    "ผู้ที่อยากสำรวจพื้นที่โดยรอบสามารถเช่าจักรยานและปั่นไปตามทางเลียบแม่น้ำได้",
]
th.append(th[0].replace("แปดหมื่น", "เก้าหมื่น"))  # a near-duplicate

# Thai lines rarely end in punctuation; FineWeb-2 turns that check off for Thai.
F.fineweb_quality(th, language="tha_Thai", line_punct_thr=0)  # [True, True]
D.minhash_dedup(th, language="tha_Thai")  # [True, False]

pyroid finds the package without importing it and reads corpus/words_th.txt and corpus/sentenceseg_crfcut.model once per process (5.6 MB), the first time a filter pipeline, MinHash or sentence dedup is built for a Thai code. If PYROID_PYTHAINLP_CORPUS is set, the files of the directory it names are read instead. A file that cannot be read raises OSError, and a model that is not a CRFsuite model ValueError. The algorithms are PyThaiNLP 5.4.0's, and parity was checked with PyThaiNLP 5.4.0 (Parity); newer releases change the word list, which pyroid then reads.

Without PyThaiNLP these codes are not native: they raise NotImplementedError, whose message says to install pythainlp. (datatrove's own Thai tokenizer also needs python-crfsuite for its sentences; pyroid does not.)

The Python tokenizer: Khmer with khmer-nltk¶

khm_Khmr is KhmerTokenizer(), which segments with khmer-nltk's CRF. pyroid does not port it. Without python_tokenizer=True, the call says what to do:

NotImplementedError: pyroid has no native word tokenizer for language 'khm_Khmr': datatrove
tokenizes it with KhmerTokenizer() (khmer-nltk). Pass python_tokenizer=True to tokenize it with
datatrove's own tokenizer in Python, on the calling thread (needs `pip install datatrove==0.10.1
regex khmer-nltk`); the filter math still runs in Rust.

With datatrove 0.10.1 and khmer-nltk 1.6 installed:

km = [
    "ភ្នំពេញជារាជធានីនៃប្រទេសកម្ពុជា។ ទីក្រុងនេះស្ថិតនៅលើច្រាំងទន្លេមេគង្គ និងទន្លេសាប។ "
    "ប្រជាជនជាច្រើនរស់នៅទីនេះ ហើយភ្ញៀវទេសចរមកលេងរៀងរាល់ឆ្នាំ។",
]
km.append(km[0].replace("រៀងរាល់ឆ្នាំ", "ជារៀងរាល់ឆ្នាំ"))  # a near-duplicate

F.fineweb_quality(km, language="khm_Khmr", python_tokenizer=True)  # [True, True]
D.minhash_dedup(km, language="khm_Khmr", python_tokenizer=True)  # [True, False]

Each call warns once:

FallbackWarning: pyroid.filters.fineweb_quality: tokenizing language 'khm_Khmr' with datatrove's
KhmerTokenizer() (khmer-nltk) in Python, one document at a time on the calling thread; the filter
math runs in Rust. Expect the tokenizer's speed, not pyroid's. This warning is shown once.

An unknown code raises datatrove's error before any document is read:

F.gopher_quality(km, language="und_Zzzz")
# ValueError: Language 'und_Zzzz' doesn't have a tokenizer assigned. Pass in a WordTokenizer
# directly or update tokenizer_assignment.csv

Parity¶

Every check below found 0 differences against datatrove 0.10.1 (spaCy 3.8.18; SudachiPy 0.7.0 with sudachidict-core 20261015 for Japanese; pyvi 0.1.1 for Vietnamese; PyThaiNLP 5.4.0 with python-crfsuite 0.9.12 for Thai; CPython 3.13 for the live checks):

Japanese also has 12 Hypothesis properties against live datatrove (python/tests/test_filters_japanese.py): 628 examples per run, and 11 of them also at 300 examples each. Vietnamese and Thai have 80 tests against live datatrove (python/tests/test_filters_vithai.py): every filter, FineWeb-2's chains, words, sentence dedup and MinHash, with 1,046 Hypothesis examples per run (5,230 in CI).

Sentence dedup has FineWeb-2 parity cases on six slices (deu_Latn, fra_Latn, rus_Cyrl, spa_Latn, arb_Arab, hin_Deva), for sentences and lines, and Vietnamese and Thai are checked against datatrove's three stages in test_filters_vithai.py. The Stanza, Kiwi, Lao, Tibetan and Burmese segmenters were not installed in these runs, so their python_tokenizer=True path is untested; Khmer's ran for the example above. See Compatibility.

Vietnamese words follow spaCy's quirk with a token that is exactly IS_ALPHA as the other spaCy tokenizers do: pyroid gives what datatrove gives the first time in a process (see the filters guide).

Quirks reproduced as datatrove has them:

• In C4 for Hindi and Bengali, indic-nlp re-joins sentences with " ", so datatrove's text.index raises ValueError on some documents (9 of 250 real hin_Deva documents). pyroid raises the same.
• Urdu sentences (C4 only) follow datatrove[multilingual], where indic-nlp imports urduhack. With plain datatrove and indic-nlp-library, datatrove raises ModuleNotFoundError for every Urdu sentence call.

spaCy is quadratic on long glued runs of characters (9.6 s for one 4.7 kB text); pyroid is linear there.

Speed¶

These timings come from a Docker/OrbStack VM on Apple Silicon, indicative. They cover FineWeb-2's filter chain (Gopher repetition, FineWeb quality and Gopher quality, with each language's settings) and its MinHash signatures, on 1,000 documents per language (each FineWeb-2 slice four times). "8" is 8 threads for pyroid and a warmed 8-process Pool for datatrove. Medians of 5 runs (datatrove: 2), with every timed document checked first for the same decisions and signatures. Release wheel, CPython 3.13, load average 6 to 16:

* An outlier: pyroid with 8 threads was slower than with 1 thread in this run (246 µs per document) while the load rose. It needs a rerun before it is quoted.

† Measured on 2026-11-07, after the jieba port, at load average 1 to 2. Before it, cmn_Hani went through python_tokenizer=True, at 1.2× / 0.9× on one thread and 0.4× with 8.

‡ Measured on 2026-11-09, after the SudachiPy port, at load average 0.6 to 1.9. Before it, jpn_Jpan went through python_tokenizer=True, at 1.2× / 0.9× on one thread and 0.3× / 0.2× with 8. Japanese gains less than the other native languages because SudachiPy's lattice dominates on both sides: pyroid's whole MinHash (1,151 µs per document, one thread) takes less time than SudachiPy's own tokenize() call on the same chunks (1,583 µs).

§ Measured on 2026-11-09, after the pyvi and PyThaiNLP ports, at load average 2.1 at the start (7.4 after the 8-worker runs). Before them, both went through python_tokenizer=True, at 1.5× / 1.5× (Vietnamese) and 1.5× / 1.6× (Thai) on one thread and 0.4× / 0.2× and 0.6× / 0.7× with 8; that run overlapped other builds on the VM (load average 3.5 to 23), so only its ratios are given. The FineWeb-2 Vietnamese documents are long (4,278 characters on average); tokenizing them takes about 320 µs per document of pyroid's 839.

The native languages run the filters 36–86× faster than datatrove on one thread (Vietnamese: 22×, Hindi: 20×, Thai: 13×, Japanese: 7.8×). The Python-tokenizer path runs at about the reference tokenizer's speed (1.2–1.5×, measured for Chinese, Japanese, Vietnamese and Thai before their ports) and does not scale with threads, so datatrove's 8 processes beat it (0.2–0.7×). Adding the languages did not slow English down: old and new builds measured within noise on 5,000 FineWeb documents. See Benchmarks.

Binary size¶

The language tables add 1.11 MB to the release extension module (fat LTO, stripped, aarch64). It grew from 3.29 to 4.40 MB, or from 1.20 to 1.46 MB gzipped. Chinese adds pyroid.jieba's dictionary and HMM tables: 3.0 MB more (5.58 to 8.60 MB), and 1.5 MB more in the wheel. Japanese adds the SudachiPy port and SudachiPy's resource files (char.def, unk.def, rewrite.def, Apache-2.0), but not the dictionary: the release extension module (aarch64, abi3) grew from 9.19 to 9.32 MB, and the wheel by 65 KB. Vietnamese and Thai add the CRFsuite reader, the two ports and pyvi's bundled model and word list (0.99 MB of it), but not PyThaiNLP's data: the release extension module grew from 9.32 to 10.44 MB (+1.12 MB), and the wheel from 4.02 to 4.45 MB.

API: pyroid.filters reference, pyroid.dedup reference.