pyroid.nltk: drop-in NLTK tokenizers¶

pyroid.nltk is a Rust port of the sentence and word tokenizers of NLTK 3.10.4 (Apache-2.0): Punkt (sent_tokenize, PunktSentenceTokenizer, PunktTokenizer), word_tokenize, the Treebank tokenizers and detokenizer, and the common regular-expression tokenizers. It returns the same sentences, tokens and offsets as NLTK 3.10.4 for every input (parity), several times faster on one thread, and batches run in parallel.

Change one import¶

-import nltk
+import pyroid.nltk as nltk

 nltk.sent_tokenize("Mr. Smith went to Washington. He liked it.")
 # ['Mr. Smith went to Washington.', 'He liked it.']
 nltk.word_tokenize("They'll pay $3.88 (roughly).")
 # ['They', "'ll", 'pay', '$', '3.88', '(', 'roughly', ')', '.']

Or keep import nltk and call pyroid.patch("nltk") once: NLTK's own nltk.sent_tokenize, nltk.word_tokenize and the tokenizer classes are then pyroid's, and pyroid.unpatch("nltk") puts them back.

The Punkt models¶

Punkt needs NLTK's pretrained punkt_tab models. pyroid does not bundle them, because nltk_data publishes them without a licence. Install them as for NLTK:

python -m nltk.downloader punkt_tab     # or, in Python: nltk.download("punkt_tab")

With NLTK installed, pyroid finds and reads the model with NLTK's own nltk.data.find and load_punkt_params, so nltk.data.path, NLTK_DATA and NLTK's checks apply as usual. NLTK is not needed otherwise. Without it, pyroid looks where NLTK looks by default (NLTK_DATA, ~/nltk_data, and the nltk_data directories under sys.prefix, /usr/share, /usr/local/share, /usr/lib and /usr/local/lib) for an unpacked tokenizers/punkt_tab/<language>/ or for tokenizers/punkt_tab.zip. A missing model raises NLTK's LookupError.

All 19 models of punkt_tab work: czech, danish, dutch, english, estonian, finnish, french, german, greek, italian, malayalam, norwegian, polish, portuguese, russian, slovene, spanish, swedish and turkish.

Sentences¶

import pyroid.nltk as nltk

nltk.sent_tokenize("Dr. Müller kam um 9 Uhr. Er war müde.", language="german")
# ['Dr. Müller kam um 9 Uhr.', 'Er war müde.']

tok = nltk.PunktTokenizer("english")
list(tok.span_tokenize("One. Two."))  # [(0, 4), (5, 9)]
tok.tokenize('"Go." He went.')  # ['"Go."', 'He went.']: the quote stays with its sentence
tok.tokenize('"Go." He went.', realign_boundaries=False)  # ['"Go.', '" He went.']

PunktSentenceTokenizer takes a PunktParameters, as in NLTK. Its sets can be changed in place, and the tokenizer sees the change at its next call:

tok = nltk.PunktTokenizer()
tok.tokenize("Sales rose approx. 4%. The dept. agreed.")
# ['Sales rose approx.', '4%.', 'The dept.', 'agreed.']
tok._params.abbrev_types.update({"approx", "dept"})
tok.tokenize("Sales rose approx. 4%. The dept. agreed.")
# ['Sales rose approx. 4%.', 'The dept. agreed.']

span_tokenize, sentences_from_text, text_contains_sentbreak, sentences_from_tokens, sentences_from_text_legacy and debug_decisions are there too. A PunktLanguageVars subclass that changes sent_end_chars (for example to add "…") still runs in Rust.

ROUGE in pyroid.metrics uses this port for split_summaries=True, so rougeLsum's sentence split runs in Rust too.

Words¶

word_tokenize splits the Punkt sentences with NLTKWordTokenizer, NLTK's improved Treebank rules. With preserve_line=True it skips the sentence split, and needs no model.

import pyroid.nltk as nltk

nltk.word_tokenize("I can't go to «Paris» -- it's 3:30.")
# ['I', 'ca', "n't", 'go', 'to', '«', 'Paris', '»', '--', 'it', "'s", '3:30', '.']
nltk.TreebankWordTokenizer().tokenize("Good muffins cost $3.88\nin New York.")
# ['Good', 'muffins', 'cost', '$', '3.88', 'in', 'New', 'York', '.']
list(nltk.NLTKWordTokenizer().span_tokenize("Hello, world."))  # [(0, 5), (5, 6), (7, 12), (12, 13)]
nltk.TreebankWordDetokenizer().detokenize(["I", "ca", "n't", "go", "."])  # "I can't go."

convert_parentheses=True gives the Penn Treebank symbols (-LRB-, -RRB-, ...), as in NLTK. align_tokens(tokens, sentence) gives the offsets of tokens found one after another.

Regular-expression tokenizers¶

RegexpTokenizer, WhitespaceTokenizer, BlanklineTokenizer, WordPunctTokenizer, regexp_tokenize, wordpunct_tokenize and blankline_tokenize are NLTK's, with gaps, discard_empty and span_tokenize:

import pyroid.nltk as nltk

nltk.wordpunct_tokenize("Good muffins cost $3.88\nin New York.")
# ['Good', 'muffins', 'cost', '$', '3', '.', '88', 'in', 'New', 'York', '.']
nltk.RegexpTokenizer(r"\w+|\$[\d\.]+|\S+").tokenize("Good muffins cost $3.88\nin New York.")
# ['Good', 'muffins', 'cost', '$3.88', 'in', 'New', 'York', '.']

These patterns run in Rust when spelled exactly so, with no flags beyond NLTK's default re.UNICODE | re.MULTILINE | re.DOTALL: \w+, \w+|[^\w\s]+, \w+|\$[\d\.]+|\S+, [\w']+, \S+, \s+, [a-zA-Z]+ and BlanklineTokenizer's. Any other pattern runs in NLTK.

Many documents¶

A list or tuple of str, or an Arrow string column, gives one list per document, computed in parallel on pyroid's shared pool. Arrow nulls give None.

import pyroid.nltk as nltk

docs = ["One. Two.", "Three."]
nltk.sent_tokenize(docs)  # [['One.', 'Two.'], ['Three.']]
nltk.word_tokenize(docs, threads=8)  # at most 8 tasks; threads=1 runs on the calling thread
nltk.PunktTokenizer("french").span_tokenize(docs)  # [[(0, 4), (5, 9)], [(0, 6)]]

threads= is keyword-only and follows the threading conventions. On free-threaded CPython the GIL stays off.

What NLTK still does¶

These are handed to NLTK 3.10.4, which must then be installed. The first call emits a pyroid.FallbackWarning; without NLTK it raises pyroid.PyroidError.

• Training: PunktTrainer, and PunktSentenceTokenizer(train_text) with a str. The trained parameters then tokenize in Rust.
• A PunktSentenceTokenizer with a custom token_cls, or with a PunktLanguageVars that overrides more than sent_end_chars; parameters that are not sets of str.
• dump, PunktTokenizer.save_params and save_dir.
• A Treebank tokenizer or detokenizer whose pattern lists were edited or overridden.
• A RegexpTokenizer pattern that is not one of the native ones.

The other NLTK tokenizers (TweetTokenizer, ToktokTokenizer, ...) are not in pyroid.nltk; keep using NLTK's.

Differences from NLTK¶

• A str with a lone surrogate ("\ud800") raises UnicodeEncodeError, as everywhere in pyroid.
• NLTK runs every pattern under a 5-second timeout (nltk.redos) and raises TimeoutError when a match takes longer; pyroid has no such limit.
• NLTK's patterns run on the regex module. pyroid reproduces regex 2026.9.29's character classes. With another regex release, NLTK can differ on characters whose Unicode properties changed between the two releases.
• Tokenizing does not add a 0 entry to PunktParameters.ortho_context for each unseen word, as NLTK's defaultdict does.
• An NLTK PunktParameters given to a tokenizer is copied, so edit it through tok._params; later edits of NLTK's object are not seen.
• NLTK compiles a PunktLanguageVars object's patterns at first use and keeps the old sent_end_chars in part of the algorithm after they change; pyroid uses the new ones throughout.

Parity¶

Every check found 0 differences from NLTK 3.10.4:

Hypothesis properties also compare every function against live NLTK, on CPython 3.13 and 3.15t, as do NLTK's doctest examples and every code point alone. See Compatibility.

Speed¶

Docker/OrbStack VM on Apple Silicon (12 vCPU, load 3 to 5), indicative. These are the first 1,000 documents of the FineWeb dev slice (3,018 characters each on average), with release wheels and the English model. Medians of 5 runs (NLTK: 2), with every document's output checked against NLTK's first. µs per document:

On 8 threads the batch gains about 2× over one thread: building the Python lists of sentences and tokens runs on the calling thread. Against NLTK on 8 processes, pyroid on 8 threads is 35× faster for sent_tokenize and 14× for word_tokenize on CPython 3.13. The first sentence in a fresh interpreter takes about as long as NLTK's (177 ms against 203 ms on 3.13), because pyroid imports NLTK to find the model when NLTK is installed. Source: bench/results/orbstack-aarch64-12cpu/2026-11-09-nltk/.

API: pyroid.nltk reference.