pyroid.filters¶

GopherQualityFilter: True for each document datatrove keeps.

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column (string, large_string or string_view).

• config (GopherQuality | None, default: None ) –

  A GopherQuality (or a datatrove GopherQualityFilter); keyword arguments override its fields.

• threads (int | None, default: None ) –

  None uses pyroid's shared thread pool; n runs at most n tasks at once; 1 runs on the calling thread. On the Python-tokenizer path it applies to the Rust part only.

• python_tokenizer (bool, default: False ) –

  Tokenize with datatrove's own tokenizer for language, in Python on the calling thread, and run the filter math in Rust on its words (see "Words and languages" above). Needed for the languages pyroid does not tokenize natively; emits pyroid.FallbackWarning once per function and language.

• **kwargs (Unpack[_GopherQualityKwargs], default: {} ) –

  GopherQualityFilter's parameters (language: a datatrove language code or a word tokenizer object).

Returns:

• bool | list[bool] | tuple[bool, ...] | Column[bool] –

  bool, list[bool], tuple[bool, ...], or a pyroid.Column of bools (nulls stay null), matching texts.

Raises:

• InputError –

  texts is not one of the accepted kinds.

• TypeError –

  A parameter has the wrong type.

• ValueError –

  datatrove has no tokenizer for language (datatrove's message), or a document of 2**30 characters or more (spaCy's limit).

• NotImplementedError –

  pyroid does not tokenize language natively and python_tokenizer is not set (the message names datatrove's tokenizer and what to install).

• PyroidError –

  The Python-tokenizer path needs datatrove or the tokenizer's package, and it is not installed.

• ZeroDivisionError –

  As datatrove, e.g. min_doc_words=None and a document without words.

• UnicodeEncodeError –

  A str contains a lone surrogate.

• UnicodeDecodeError –

  An Arrow row is not valid UTF-8.

GopherQualityFilter: datatrove's drop reason, None when kept.

Reasons: gopher_short_doc, gopher_long_doc, gopher_below_avg_threshold, gopher_above_avg_threshold, gopher_too_many_hashes, gopher_too_many_ellipsis, gopher_too_many_bullets, gopher_too_many_end_ellipsis, gopher_below_alpha_threshold, gopher_enough_stop_words. Arguments and errors as for gopher_quality; Arrow input gives a string pyroid.Column (null when kept or null).

GopherRepetitionFilter: True for each document datatrove keeps.

Arguments, returns and errors as for gopher_quality, with GopherRepetition / GopherRepetitionFilter parameters.

GopherRepetitionFilter: datatrove's drop reason, None when kept.

Reasons: empty, dup_para_frac, dup_para_char_frac, dup_line_frac, dup_line_char_frac, top_{n}_gram, duplicated_{n}_n_grams.

C4QualityFilter's document decision: True for each document kept.

Arguments, returns and errors as for gopher_quality, with C4Quality / C4QualityFilter parameters. The text after line removal is c4_lines.

C4QualityFilter: datatrove's drop reason, None when kept.

Reasons: lorem_ipsum, curly_bracket, too_few_sentences.

C4QualityFilter's line removal: doc.text after the filter.

For a kept document, the lines (or sentences) C4 keeps, stripped and re-joined, citations removed; for a dropped one, the input (datatrove leaves it untouched). Unchanged documents are the original objects; Arrow nulls stay null.

Returns:

• str | list[str] | tuple[str, ...] | Column[str] –

  str, list, tuple, or a pyroid.Column of strings (large_string for large_string input), matching texts.

FineWebQualityFilter: True for each document datatrove keeps.

Arguments, returns and errors as for gopher_quality, with FineWebQuality / FineWebQualityFilter parameters.

FineWebQualityFilter: datatrove's drop reason, None when kept.

Reasons: empty, line_punct_ratio, short_line_ratio, char_dup_ratio, list_ratio.

Run several filters in order, in one pass over each document.

The same as running the datatrove filters one after another in a pipeline: a document dropped by one filter is not seen by the next, and a C4Quality step's rewritten text is what later steps filter. Steps that see the same text tokenize it once.

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  As for gopher_quality.

• filters (Iterable[GopherQuality | GopherRepetition | C4Quality | FineWebQuality | Any] | GopherQuality | GopherRepetition | C4Quality | FineWebQuality) –

  Config objects, or datatrove filter instances (or one of either) (GopherQualityFilter etc.) whose settings are read. Each has its own language.

• threads (int | None, default: None ) –

  As for gopher_quality.

• python_tokenizer (bool, default: False ) –

  As for gopher_quality, for every step. Steps are then run in datatrove's order: consecutive steps with the same tokenizer share one tokenization per document, and a C4 step runs as datatrove's C4QualityFilter, whose rewritten text the later steps tokenize again.

Returns:

• FilterResult –

  FilterResult(keep, reason, text), each in the caller's kind.

Raises the same exceptions as gopher_quality.

Every language code datatrove 0.10.1 knows, with its tokenizer.

The keys are what language= accepts: datatrove's tokenizer_assignment.csv as load_word_tokenizer reads it ("deu_Latn", "deu", "de", ...). pyroid.dedup.languages is the same function.

Example:

>>> pyroid.filters.languages()["cmn_Hani"]
Language(tokenizer='SpaCyTokenizer', arg='zh', native=False)

Returns:

• dict[str, Language] –

  {code: Language(tokenizer, arg, native)}, sorted by code.

The datatrove language codes whose tokenizer is SpaCyTokenizer("en").

PIIFormatter: each document's text with emails and IP addresses replaced.

A fresh formatter per call, so the replacement counters start at 0: one call over a whole corpus gives datatrove's output. To process a corpus in batches, use one PIIReplacer.

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• config (PII | None, default: None ) –

  A PII; keyword arguments override its fields.

• threads (int | None, default: None ) –

  As for PIIReplacer.__call__.

• **kwargs (Unpack[_PIIKwargs], default: {} ) –

  PIIFormatter's parameters.

Returns:

• str | list[str] | tuple[str, ...] | Column[str] –

  As PIIReplacer.__call__.

Datatrove's bad-word set for language.

The lists are the LDNOOBW "List of Dirty, Naughty, Obscene, and Otherwise Bad Words" (https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words, CC BY 4.0), at the two commits datatrove pins. pyroid does not ship them: like datatrove, it downloads a list the first time it is needed, into cache_dir (default $PYROID_CACHE_DIR, else $XDG_CACHE_HOME/pyroid, else ~/.cache/pyroid) under c4_badwords/, and checks its sha256. The set is built as datatrove builds it: each line strip()-ed, minus every language's allowlist (sm, グロ, 女の子, 性).

Parameters:

• language (str) –

  One of C4_BADWORDS_LANGUAGES.

• cache_dir (str | PathLike[str] | None, default: None ) –

  Where the lists are kept.

• download (bool, default: True ) –

  Fetch a missing list (else FileNotFoundError).

Raises:

• ValueError –

  No list for language (datatrove's message).

• OSError –

  The download failed, or the file's sha256 is not the pinned one.

C4BadWordsFilter: True for each document datatrove keeps.

A fresh BadWordsFilter per call, so the draws start from default_rng(seed): one call over a whole corpus gives datatrove's decisions. For batches, use one BadWordsFilter.

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• config (C4BadWords | None, default: None ) –

  A C4BadWords; keyword arguments override its fields.

• languages (_Languages, default: None ) –

  As for BadWordsFilter.__call__.

• words (Mapping[str, Iterable[str]] | None, default: None ) –

  As for BadWordsFilter.

• threads (int | None, default: None ) –

  As for BadWordsFilter.__call__.

• **kwargs (Unpack[_C4BadWordsKwargs], default: {} ) –

  C4BadWordsFilter's parameters.

Returns:

• bool | list[bool] | tuple[bool, ...] | Column[bool] –

  A keep mask in the caller's kind.

C4BadWordsFilter: datatrove's drop reason per document, None when kept.

As c4_badwords, returning "document_removed_with_badwords" or None.

Extract datatrove's integrated URL block lists as datatrove does; return their paths.

datatrove 0.10.1 ships them in its package (datatrove/assets): url_filterblacklistsv0_3_0.tar.gz (domains: 4,558,940 lines; urls: 19,587), banned_words.txt, banned_subwords.txt and soft_banned_words.txt. The archive is extracted once into datatrove's own cache (huggingface_hub.cached_assets_path("datatrove", "filters", "url_filter")), under datatrove's lock and .completed marker, so datatrove and pyroid share one copy. pyroid does not ship these lists: they are not under pyroid's licence.

Raises:

• PyroidError –

  datatrove (or huggingface_hub) is not installed.

• ImportError –

  fasteners is not installed (datatrove's lock needs it).

URLFilter: True for each URL datatrove keeps.

config is a URLFilterConfig (keyword arguments override its fields) or a built URLFilter. The last two filters built from configs are cached (configs whose lists are lists, tuples, sets or strings), so repeated calls do not reload the block lists.

Returns:

• bool | list[bool] | tuple[bool, ...] | Column[bool] –

  As URLFilter.keep.