pyroid.nemo¶

Score each document as NeMo's NonAlphaNumericFilter does.

The share of characters outside [a-zA-Z0-9\n?!,.]: (n - k) / n for n characters (code points), 1.0 for an empty document. NeMo keeps a document when score <= max_non_alpha_numeric_to_text_ratio (0.25).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column (string, large_string or string_view).

• threads (int | None, default: None ) –

  None uses pyroid's shared thread pool; n runs at most n tasks at once; 1 runs on the calling thread.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ValueError –

  An Arrow element is null.

Score each document as NeMo's SymbolsToWordsFilter does.

The share of words (of text.strip()) that are an ellipsis mark (..., [...], …, (...), […], -», read more.., read more) or whose runs of # number more than half their length. NeMo keeps a document when score <= max_symbol_to_word_ratio (0.1).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• lang (str, default: 'en' ) –

  NeMo's lang, compared as lang.lower(): "zh" splits words with jieba 0.42.1's default dictionary (jieba.cut, through pyroid.jieba); any other code but "ja" splits on whitespace (str.split()).

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ZeroDivisionError –

  The document has no words, or (lang="zh") a word that strips to "", as in NeMo.

• NotImplementedError –

  lang is "ja" (NeMo's MeCab splitter is not ported).

• ValueError –

  An Arrow element is null.

Score each document as NeMo's NumbersFilter does.

The share of characters that are ASCII digits (NeMo's [[:digit:]]), 1.0 for an empty document. NeMo keeps a document when score <= max_number_to_text_ratio (0.15).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ValueError –

  An Arrow element is null.

Score each document as NeMo's UrlsFilter does, with its default URL pattern.

The share of characters inside matches of NeMo's regex_url (https?:// and a run of URL characters or %XX escapes), 1.0 for an empty document. NeMo keeps a document when score <= max_url_to_text_ratio (0.2). A custom url_regex is not supported.

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ValueError –

  An Arrow element is null.

Score each document as NeMo's WhiteSpaceFilter does.

The share of characters that are \t, \n, \r, \b or a space, 1.0 for an empty document. NeMo keeps a document when score <= max_white_space_ratio (0.25).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ValueError –

  An Arrow element is null.

Score each document as NeMo's ParenthesesFilter does.

The share of characters in { } ⟨ ⟩ [ ] ( ), 1.0 for an empty document. NeMo keeps a document when score <= max_parentheses_ratio (0.1).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ValueError –

  An Arrow element is null.

Score each document as NeMo's BoilerPlateStringFilter does.

1.0 if a paragraph (text.split("\n\n")) contains "lorem ipsum" in any case; otherwise the share of paragraphs whose strip().lower() contains one of NeMo's policy strings ("terms of use", "privacy policy", "use cookies", ...). NeMo keeps a document when score <= max_boilerplate_string_ratio (0.4).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ValueError –

  An Arrow element is null.

Score each document as NeMo's RepeatedLinesFilter does.

Distinct lines over lines, where the lines are the non-blank pieces of text.split("\n"), unstripped. NeMo keeps a document when score >= max_repeated_line_fraction (0.7).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ZeroDivisionError –

  The document has no non-blank line.

• ValueError –

  An Arrow element is null.

Score each document as NeMo's RepeatedParagraphsFilter does.

Distinct paragraphs over paragraphs (text.split("\n\n"), empty ones included). NeMo keeps a document when score >= max_repeated_paragraphs_ratio (0.7).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ValueError –

  An Arrow element is null.

Score each document as NeMo's RepeatedLinesByCharFilter does.

The characters of the distinct lines over the characters of all lines (lines as for repeated_lines). NeMo keeps a document when score >= max_repeated_lines_char_ratio (0.8).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ZeroDivisionError –

  The document has no non-blank line.

• ValueError –

  An Arrow element is null.

Score each document as NeMo's RepeatedParagraphsByCharFilter does.

The characters of the distinct paragraphs over the characters of all paragraphs. NeMo keeps a document when score >= max_repeated_paragraphs_char_ratio (0.8).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ZeroDivisionError –

  Every paragraph is empty ("", "\n\n").

• ValueError –

  An Arrow element is null.

Score each document as NeMo's WordCountFilter does: its number of words.

NeMo keeps a document when min_words <= score <= max_words (50 and 100000).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• lang (str, default: 'en' ) –

  As for symbols_to_words.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• int | _I64 –

  An int for a str; an int64 numpy array for a batch.

Raises:

• NotImplementedError –

  lang is "ja".

• ValueError –

  An Arrow element is null.

Score each document as NeMo's PunctuationFilter does.

The share of lines (non-blank pieces of text.split("\n")) whose stripped text does not end with ., ?, !, " or '. NeMo keeps a document when score <= max_num_sentences_without_endmark_ratio (0.85).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ZeroDivisionError –

  The document has no non-blank line.

• ValueError –

  An Arrow element is null.

Score each document as NeMo's WordsWithoutAlphabetsFilter does.

The share of words with at least one character in [[:alpha:]] as the regex module defines it (its Alphabetic property, Unicode 18.0.0 in regex 2026.9.29 and 2026.11.9; not str.isalpha()). NeMo keeps a document when score >= min_words_with_alphabets (0.8).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• lang (str, default: 'en' ) –

  As for symbols_to_words.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ZeroDivisionError –

  The document has no words.

• NotImplementedError –

  lang is "ja".

• ValueError –

  An Arrow element is null.

Score each document as NeMo's CommonEnglishWordsFilter does.

The number of words of text.split() that are exactly "the", "be", "to", "of", "and", "that", "have" or "with". With stop_at_false, counting stops as soon as the count reaches min_num_common_words (checked after every word, as NeMo does). NeMo keeps a document when score >= min_num_common_words.

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• min_num_common_words (int, default: 2 ) –

  NeMo's min_num_common_words.

• stop_at_false (bool, default: True ) –

  NeMo's stop_at_false.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• int | _I64 –

  An int for a str; an int64 numpy array for a batch.

Raises:

• ValueError –

  An Arrow element is null.

Score each document as NeMo's MeanWordLengthFilter does: the mean word length.

Lengths are in characters (code points); empty words are left out. NeMo keeps a document when min_mean_word_length <= score <= max_mean_word_length (3 and 10).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• lang (str, default: 'en' ) –

  As for symbols_to_words.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ZeroDivisionError –

  The document has no words.

• NotImplementedError –

  lang is "ja".

• ValueError –

  An Arrow element is null.

Score each document as NeMo's LongWordFilter does: its longest word's length.

NeMo keeps a document when score <= max_word_length (1000).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• lang (str, default: 'en' ) –

  As for symbols_to_words.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• int | _I64 –

  An int for a str; an int64 numpy array for a batch.

Raises:

• ValueError –

  The document has no words (max() of nothing, as in NeMo), or an Arrow element is null.

• NotImplementedError –

  lang is "ja".

Score each document as NeMo's EllipsisFilter does.

The share of lines (non-blank pieces of text.split("\n")) whose strip().lower() ends with an ellipsis mark (see symbols_to_words). NeMo keeps a document when score <= max_num_lines_ending_with_ellipsis_ratio (0.3).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ZeroDivisionError –

  The document has no non-blank line.

• ValueError –

  An Arrow element is null.

Score each document as NeMo's RepeatingTopNGramsFilter(n) does.

The share of characters that removing the most frequent word n-gram (its words joined with single spaces, every non-overlapping occurrence in the text) would remove; 1.0 for a document with fewer than n words. NeMo keeps a document when score <= max_repeating_ngram_ratio (0.2).

When several n-grams share the top count, NeMo takes the first of them in the iteration order of a Python set, which depends on the process's string hash seed (PYTHONHASHSEED). The result is the one NeMo gives in the same process; with a fixed PYTHONHASHSEED it is the same in every run. On 64-bit CPython (siphash13) pyroid reads the process's hash key and reproduces the set's order in Rust, after checking its hashes and set order against the interpreter once; elsewhere it builds the same set in Python.

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• n (int, default: 2 ) –

  The n-gram length. Below 1 there are no n-grams and NeMo raises ValueError.

• lang (str, default: 'en' ) –

  As for symbols_to_words.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ValueError –

  n < 1 (max() of nothing, as in NeMo), or an Arrow element is null.

• NotImplementedError –

  lang is "ja".

Score each document as NeMo's RepeatingDuplicateNGramsFilter(n) does.

The share of characters in word n-grams seen earlier in the document (overlapping repeats counted once, with the spaces between their words); 1.0 below n words. For n < 1 there are no n-grams: 0.0 (1.0 for an empty document). NeMo keeps a document when score <= max_repeating_duplicate_ngram_ratio (0.2).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• n (int, default: 2 ) –

  The n-gram length.

• lang (str, default: 'en' ) –

  As for symbols_to_words.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• NotImplementedError –

  lang is "ja".

• ValueError –

  An Arrow element is null.

Score each document as NeMo's BulletsFilter does.

The share of lines (non-blank pieces of text.split("\n")) whose stripped text starts with one of NeMo's twelve bullets (U+2022 •, U+2023, U+2043, U+204C, U+204D, U+2219, U+25CB, U+25CF, U+25D8, U+25E6, U+29BE, U+29BF). NeMo keeps a document when score <= max_bullet_lines_ratio (0.9).

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• ZeroDivisionError –

  The document has no non-blank line.

• ValueError –

  An Arrow element is null.

Score each document as NeMo's PornographicUrlsFilter does, with its URL pattern.

1 if a URL (as for urls) contains "porn" (case-sensitive), else 0. NeMo keeps a document when score != 1.

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• int | _I64 –

  An int for a str; an int64 numpy array for a batch.

Raises:

• ValueError –

  An Arrow element is null.

Score each document as NeMo's SubstringFilter(substring, position) does.

1 if the document starts with ("prefix"), ends with ("suffix") or contains ("any") substring, else 0. NeMo keeps a document when score == 1.

Parameters:

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• substring (str) –

  The substring to look for.

• position (Literal['prefix', 'suffix', 'any']) –

  "prefix", "suffix" or "any".

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• int | _I64 –

  An int for a str; an int64 numpy array for a batch.

Raises:

• ValueError –

  position is not one of the three (NeMo's message, before any document is read), or an Arrow element is null.

Score each document as NeMo's FastTextLangId does, as a probability and a language.

The line text.strip().replace("\n", " ") goes to model's top-1 prediction; the score is its probability (fastText's float32, widened) and the language is its label without the __label__ prefix. NeMo stores str([score, lang]) (lang_id_strings) and keeps a document when score >= min_langid_score (0.3) and, with a lang filter, the language matches (case-insensitively; "en" matches any "en_..." script).

Parameters:

• model (Model) –

  A pyroid.fasttext.Model (for example lid.176.bin).

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• tuple[float, str] | tuple[_F64, list[str]] –

  (score, lang) for a str; for a batch, a float64 numpy array of scores and a list of languages.

Raises:

• IndexError –

  The model predicts no label for a line (no known token), as NeMo's score[0][0] does.

• ValueError –

  The model is not supervised, or an Arrow element is null.

• RuntimeError –

  The model's weights give NaN.

• UnicodeDecodeError –

  The predicted label is not valid UTF-8.

Score each document as NeMo's FastTextQualityFilter does.

The line text.replace("\n", " ").replace("__label__", " ") goes to model's top-1 prediction with probability p (fastText's float32, widened); the score is p if its label is label, else 1 - p. NeMo keeps a document when np.random.pareto(alpha) > 1 - score, a random draw (alpha 3).

Parameters:

• model (Model) –

  A pyroid.fasttext.Model.

• texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

  A str, a list or tuple of str, or an Arrow string array/column.

• label (str, default: '__label__hq' ) –

  The label that means high quality.

• threads (int | None, default: None ) –

  As for non_alpha_numeric.

Returns:

• float | _F64 –

  A float for a str; a float64 numpy array for a batch.

Raises:

• IndexError –

  The model predicts no label for a line.

• ValueError –

  The model is not supervised, or an Arrow element is null.

• RuntimeError –

  The model's weights give NaN.

• UnicodeDecodeError –

  The predicted label is not valid UTF-8.

Format fasttext_lang_id's results as NeMo's FastTextLangId scores.

Each is str([score, lang]), e.g. "[0.98, 'en']": Python's repr of the float and of the str.

Parameters:

• scores (Sequence[float] | NDArray[floating[Any]]) –

  The probabilities (a float64 array, or any sequence of floats).

• langs (Sequence[str]) –

  The languages, as many as scores.

Returns:

• list[str] –

  One str per score.

Raises:

• ValueError –

  scores and langs differ in length.

• TypeError –

  An item of langs is not a str.

Drop boilerplate paragraphs, as BoilerPlateStringModifier does.

Paragraphs are text.split("\n\n"). A paragraph containing "lorem ipsum" (any case) makes the document empty. A paragraph with one of NeMo's policy strings ("terms of use", "privacy policy", "uses cookies", ...) is boilerplate: with remove_if_at_top_or_bottom=False it makes the document empty; with True the boilerplate paragraphs are removed when they form one block at the top or the bottom of the document, and the document is left as it is otherwise (also when every paragraph is boilerplate).

Parameters:

• x (str | list[str] | tuple[str, ...] | ArrowStrings) –

  The documents.

• remove_if_at_top_or_bottom (bool, default: True ) –

  As for BoilerPlateStringModifier.

• threads (int | None, default: None ) –

  None uses pyroid's shared thread pool; n runs at most n tasks at once; 1 runs on the calling thread.

Returns:

• str | list[str] | tuple[str, ...] | Column[str] –

  The documents, the same kind as x.

Raises:

• InputError –

  x is not one of the accepted kinds.

Remove Markdown emphasis and links, as MarkdownRemover does.

Line by line, **a**, then *a*, then _a_ become a, and [a](b) becomes b (NeMo's four re.sub calls with lazy groups, in that order).

Parameters:

• x (str | list[str] | tuple[str, ...] | ArrowStrings) –

  The documents.

• threads (int | None, default: None ) –

  As for remove_boilerplate_strings.

Returns:

• str | list[str] | tuple[str, ...] | Column[str] –

  The documents, the same kind as x.

Shorten newline runs, as NewlineNormalizer does.

Three or more \n in a row become \n\n; then three or more \r\n in a row become \r\n\r\n.

Parameters:

• x (str | list[str] | tuple[str, ...] | ArrowStrings) –

  The documents.

• threads (int | None, default: None ) –

  As for remove_boilerplate_strings.

Returns:

• str | list[str] | tuple[str, ...] | Column[str] –

  The documents, the same kind as x.

Remove the quotes around a quoted document, as QuotationRemover does.

A document that starts and ends with " and has more than 2 characters after strip() loses its first and last character, unless it contains a newline and its first line ends with ".

Parameters:

• x (str | list[str] | tuple[str, ...] | ArrowStrings) –

  The documents.

• threads (int | None, default: None ) –

  As for remove_boilerplate_strings.

Returns:

• str | list[str] | tuple[str, ...] | Column[str] –

  The documents, the same kind as x.

Delete URLs, as UrlRemover does.

The same as re.sub(r"https?://\S+|www\.\S+", "", text, flags=re.IGNORECASE), including re's case rules (U+017F LATIN SMALL LETTER LONG S matches s).

Parameters:

• x (str | list[str] | tuple[str, ...] | ArrowStrings) –

  The documents.

• threads (int | None, default: None ) –

  As for remove_boilerplate_strings.

Returns:

• str | list[str] | tuple[str, ...] | Column[str] –

  The documents, the same kind as x.

Drop the lines equal to one of patterns, as LineRemover(patterns) does.

Lines are text.split("\n"); a line is dropped when it equals a pattern exactly.

Parameters:

• x (str | list[str] | tuple[str, ...] | ArrowStrings) –

  The documents.

• patterns (Sequence[str]) –

  The lines to drop.

• threads (int | None, default: None ) –

  As for remove_boilerplate_strings.

Returns:

• str | list[str] | tuple[str, ...] | Column[str] –

  The documents, the same kind as x.

Slice each document, as Slicer(left, right, include_left, include_right, strip) does.

An int bound is a Python slice index (in characters). A str left starts the slice at its first occurrence (after it when include_left is false), a str right ends it at its last occurrence (after it when include_right is true); the result is "" when either string is missing. None means the start or the end. strip strips the result.

Parameters:

• x (str | list[str] | tuple[str, ...] | ArrowStrings) –

  The documents.

• left (int | str | None, default: 0 ) –

  The start bound.

• right (int | str | None, default: None ) –

  The end bound.

• include_left (bool, default: True ) –

  Keep a str left in the result.

• include_right (bool, default: True ) –

  Keep a str right in the result.

• strip (bool, default: True ) –

  Strip whitespace from the result.

• threads (int | None, default: None ) –

  As for remove_boilerplate_strings.

Returns:

• str | list[str] | tuple[str, ...] | Column[str] –

  The documents, the same kind as x.

Prefix each document with a fastText label, as FastTextLabelModifier(label) does.

The result is f"{label} {text}" after replacing \n and __label__ in the text by spaces. Every document changes.

Parameters:

• x (str | list[str] | tuple[str, ...] | ArrowStrings) –

  The documents.

• label (object) –

  The label (formatted with format(label, ""), as an f-string does).

• threads (int | None, default: None ) –

  As for remove_boilerplate_strings.

Returns:

• str | list[str] | tuple[str, ...] | Column[str] –

  The documents, the same kind as x.