Skip to content

pyroid.dedup

MinHash near-duplicate and exact deduplication (pyroid.dedup).

The MinHash pipeline makes the same removals as datatrove 0.10.1's MinhashDedupSignature -> MinhashDedupBuckets -> MinhashDedupCluster -> MinhashDedupFilter, in memory and multi-threaded:

  1. Signatures. Each document is normalised with datatrove's simplify_text and split into words by the word tokenizer of language (datatrove's word_tokenize; spaCy English by default). Every n_grams-word window is hashed (hash_fc). The signature comes from num_buckets * hashes_per_bucket permutations (a * x + b) mod (2**61 - 1), with a and b drawn from numpy's RandomState(seed) exactly as datatrove draws them. Documents with fewer than n_grams words have no signature and are always kept.
  2. Buckets. Two documents are a duplicate pair when all hashes_per_bucket values of one bucket are equal.
  3. Clusters. Pairs are merged with datatrove's union-find (union by size). The document kept from each cluster is the union-find root: the root of the larger side wins each merge, and ties go to the earlier document of the pair. That is usually, but not always, the cluster's first document; MinHashDedup.clusters lists the kept one first.
  4. Filter. Every other member of a cluster is removed.

Documents are numbered in the order they are added, as datatrove numbers the documents of one task. Configurations are datatrove's: pass a MinhashConfig, datatrove's own MinhashConfig, a dict, or keyword arguments (hash_fc= and precision= set hash_config). The default is datatrove's default (xxhash/64); FineWeb used MinhashConfig(hash_config=HashConfig(precision=64, hash_fc="sha1")).

Languages

language= is MinhashDedupSignature(language=...): any of datatrove's language codes (languages), as in pyroid.filters. Up to 4,049 of the 4,343 codes are tokenized natively, in Rust, on every thread: those datatrove tokenizes with a rule-based spacy.blank(code) (57 spaCy languages, xx included), spaCy's Chinese tokenizer over jieba (pyroid.jieba), spaCy's Vietnamese tokenizer with pyvi, IndicNLPTokenizer or WhitespaceTokenizer; Japanese (SpaCyTokenizer("ja"), SudachiPy ported to Rust) when a SudachiDict dictionary is installed (pip install sudachidict-core, or the system.dic that PYROID_SUDACHI_DICT names); and Thai (ThaiTokenizer, PyThaiNLP's newmm and crfcut ported to Rust) when PyThaiNLP is installed (pip install pythainlp, or the corpus directory PYROID_PYTHAINLP_CORPUS names); see pyroid.filters. Without them, Japanese and Thai are among the codes below. Where SudachiPy's input limit stops datatrove, pyroid raises ValueError with SudachiPy's message, not sudachipy.errors.SudachiError.

The other 294 (Stanza, Kiwi, Khmer, ...) raise NotImplementedError unless python_tokenizer=True. With it, pyroid's simplify_text runs in Rust, datatrove's own tokenizer splits each document in Python on the calling thread, and shingling, MinHash, bucketing and clustering run in Rust on those words. Each function emits one pyroid.FallbackWarning per language. That path runs at the reference tokenizer's speed. language may also be a word tokenizer object (a datatrove WordTokenizer, or anything with word_tokenize(text)), called the same way.

Known differences from datatrove

  • A str with a lone surrogate raises UnicodeEncodeError (pyroid's policy). datatrove raises it too, unless the document has fewer than n_grams words.
  • n_grams, num_buckets or hashes_per_bucket below 1 raise ValueError (datatrove silently produces no signatures or fails later).
  • Index files (index_folder, create_index_name, only_dedup_in_index) are not supported.
  • An unknown language raises datatrove's ValueError when the call starts, as MinhashDedupSignature does when it is built.

DATATROVE_VERSION module-attribute 

DATATROVE_VERSION: Final[str]

Language

Bases: NamedTuple

How datatrove 0.10.1 tokenizes one language code (an entry of languages).

tokenizer instance-attribute 

tokenizer: str

datatrove's tokenizer class: "SpaCyTokenizer", "StanzaTokenizer", "IndicNLPTokenizer", "WhitespaceTokenizer", "KiwiTokenizer", ...

arg instance-attribute 

arg: str

The argument it is built with (a spaCy, Stanza or Indic NLP language code), or "".

native instance-attribute 

native: bool

pyroid tokenizes this code natively; if not, it needs python_tokenizer=True.

HashConfig dataclass 

HashConfig(precision: int = 64, hash_fc: str = 'xxhash')

datatrove's HashConfig: the shingle hash function and its width.

Attributes:

  • precision (int) –

    64 or 32 bits.

  • hash_fc (str) –

    "xxhash" (xxHash64/xxHash32, seed 0) or "sha1" (the first 8 or 4 bytes of SHA-1, little-endian).

precision class-attribute instance-attribute 

precision: int = 64

hash_fc class-attribute instance-attribute 

hash_fc: str = 'xxhash'

TextNormConfig dataclass 

TextNormConfig(
    lowercase: bool = True,
    norm_whitespace: bool = True,
    remove_punctuation: bool = True,
    norm_unicode_diacritics: bool = True,
    norm_numbers: bool = True,
    norm_weekdays: bool = False,
    norm_monthnames: bool = False,
)

datatrove's TextNormConfig: what simplify_text does, in order.

Attributes:

lowercase class-attribute instance-attribute 

lowercase: bool = True

norm_whitespace class-attribute instance-attribute 

norm_whitespace: bool = True

remove_punctuation class-attribute instance-attribute 

remove_punctuation: bool = True

norm_unicode_diacritics class-attribute instance-attribute 

norm_unicode_diacritics: bool = True

norm_numbers class-attribute instance-attribute 

norm_numbers: bool = True

norm_weekdays class-attribute instance-attribute 

norm_weekdays: bool = False

norm_monthnames class-attribute instance-attribute 

norm_monthnames: bool = False

MinhashConfig dataclass 

MinhashConfig(
    n_grams: int = 5,
    num_buckets: int = 14,
    hashes_per_bucket: int = 8,
    seed: int = 1,
    norm_config: TextNormConfig = ...,
    hash_config: HashConfig = ...,
)

datatrove's MinhashConfig, with the same fields and defaults.

Attributes:

n_grams class-attribute instance-attribute 

n_grams: int = 5

num_buckets class-attribute instance-attribute 

num_buckets: int = 14

hashes_per_bucket class-attribute instance-attribute 

hashes_per_bucket: int = 8

seed class-attribute instance-attribute 

seed: int = 1

norm_config class-attribute instance-attribute 

norm_config: TextNormConfig = ...

hash_config class-attribute instance-attribute 

hash_config: HashConfig = ...

MinHashDedup 

MinHashDedup(
    config: Any = None,
    *,
    threads: int | None = None,
    language: str | _WordTokenizer = "eng",
    python_tokenizer: bool = False,
    **kwargs: Any,
)

An in-memory MinHash index that makes datatrove's removals.

Add documents in batches with add (only their signatures are kept, 8 * num_buckets * hashes_per_bucket bytes each, in Rust), then read the decisions for everything added so far. Decisions are global: a later batch can change which document of a cluster is kept, so masks are only final once every batch is in. fit does one corpus in a single call. Methods may be called from several threads.

Example:

d = MinHashDedup(hash_fc="sha1")       # FineWeb's configuration
for batch in batches:
    d.add(batch)
keep = d.keep_mask()                   # one bool per document added

Parameters:

config property 

config: MinhashConfig

The configuration (a MinhashConfig).

language property 

language: str | _WordTokenizer

The language the documents are tokenized for (a code or a tokenizer object).

add 

add(
    texts: str | list[str] | tuple[str, ...] | ArrowStrings,
    *,
    threads: int | None = None,
) -> range

Add a batch of documents.

Returns:

  • range –

    The indices given to the batch's documents (Arrow nulls get one too, and are always kept).

fit 

fit(texts: str, *, threads: int | None = None) -> bool

fit(
    texts: list[str], *, threads: int | None = None
) -> list[bool]

fit(
    texts: tuple[str, ...], *, threads: int | None = None
) -> tuple[bool, ...]

fit(
    texts: ArrowStrings, *, threads: int | None = None
) -> Column[bool]

Replace the index with texts and return their keep mask.

Returns:

  • bool | list[bool] | tuple[bool, ...] | Column[bool] –

    A mask in the same kind as texts (see minhash_dedup).

keep_mask 

keep_mask() -> list[bool]

True for each document to keep, over every document added.

duplicates 

duplicates() -> list[int]

Indices of the documents to remove, ascending (datatrove's .remove file).

clusters 

clusters() -> list[list[int]]

Every cluster of two or more near-duplicates.

Each cluster lists the kept document first, then the removed ones in ascending order. Clusters are ordered by their smallest member, so clusters()[k] is datatrove's cluster id k.

cluster_ids 

cluster_ids() -> list[int]

Per-document minhash_cluster_id, as datatrove writes it (-1: in no cluster).

cluster_sizes 

cluster_sizes() -> list[int]

Per-document minhash_cluster_size, as datatrove writes it (1: no cluster).

representatives 

representatives() -> list[int]

For each document, the index of its cluster's kept document (itself if kept).

signature 

signature(index: int) -> list[int] | None

The stored signature of document index (None: no shingles).

Raises:

  • IndexError –

    index is not a document added so far.

reset 

reset() -> None

Forget every document.

ExactDedup 

ExactDedup(*, normalize: Any = False)

Streaming exact dedup: keep the first copy of each text, across batches.

Holds one 128-bit hash per distinct text (see exact_dedup), so each batch's mask is final as soon as add returns.

Parameters:

add 

add(texts: str, *, threads: int | None = None) -> bool

add(
    texts: list[str], *, threads: int | None = None
) -> list[bool]

add(
    texts: tuple[str, ...], *, threads: int | None = None
) -> tuple[bool, ...]

add(
    texts: ArrowStrings, *, threads: int | None = None
) -> Column[bool]

Keep mask for one more batch: True for each first copy so far.

Returns:

  • bool | list[bool] | tuple[bool, ...] | Column[bool] –

    A mask in the same kind as texts (Arrow nulls stay null).

reset 

reset() -> None

Forget every text seen.

SentDedupConfig dataclass 

SentDedupConfig(
    n_sentences: int = 3,
    split_sentences: bool = True,
    only_dedup_in_index: bool = True,
    min_doc_words: int = 50,
    min_num_sentences: int = 3,
    min_words_to_remove_span: int = 0,
    norm_config: TextNormConfig = ...,
    hash_config: HashConfig = ...,
)

datatrove's SentDedupConfig, with the same fields and defaults.

Attributes:

n_sentences class-attribute instance-attribute 

n_sentences: int = 3

split_sentences class-attribute instance-attribute 

split_sentences: bool = True

only_dedup_in_index class-attribute instance-attribute 

only_dedup_in_index: bool = True

min_doc_words class-attribute instance-attribute 

min_doc_words: int = 50

min_num_sentences class-attribute instance-attribute 

min_num_sentences: int = 3

min_words_to_remove_span class-attribute instance-attribute 

min_words_to_remove_span: int = 0

norm_config class-attribute instance-attribute 

norm_config: TextNormConfig = ...

hash_config class-attribute instance-attribute 

hash_config: HashConfig = ...

SentenceDedupResult

Bases: NamedTuple, Generic[_SentT, _SentM]

What sentence_dedup and the SentenceDedup methods return.

Both fields are in the caller's container kind (str/bool, list, tuple, or a pyroid.Column; Arrow nulls stay null).

text instance-attribute 

text: _SentT

Each document's text with its duplicate windows removed (the original object when nothing was removed). Dropped documents get their text too.

keep instance-attribute 

keep: _SentM

True for each document datatrove keeps.

SentenceDedup 

SentenceDedup(
    config: Any = None,
    *,
    language: str = "eng",
    threads: int | None = None,
    **kwargs: Any,
)

datatrove's sentence-level dedup, in memory and multi-threaded.

C4's rule ("we discarded all but one of any three-sentence span occurring more than once"), with the same removals and keep decisions as datatrove 0.10.1's SentenceDedupSignature -> SentenceFindDedups -> SentenceDedupFilter run as one task:

  1. Signatures (add). A document's sentences (datatrove's sent_tokenize: spaCy's sentences, stripped, empty ones dropped; or, with split_sentences=False, its lines) each go through simplify_text; every window of n_sentences consecutive ones, joined by a space, is hashed (hash_fc) unless it is blank. A document with fewer than n_sentences sentences has none.
  2. Duplicates (duplicates). For each hash seen more than once, every window but the first (lowest document, then window) is removed.
  3. Filter (filter). The n_sentences sentences of each removed window are cut out. Sentences are now spaCy's span_tokenize spans, each with the whitespace before it, which (unlike step 1) include a whitespace-only last sentence; the result is left-stripped (lines are rejoined with newlines). A document whose text changed is dropped if it has fewer than min_doc_words words or min_num_sentences sentences; an empty text is always dropped, even with nothing removed, as in datatrove.

datatrove's quirks are kept: min_words_to_remove_span only counts, and only restores, the first sentence of a removed run.

Documents are numbered in the order they are added, as datatrove numbers the documents of one task. Add every batch first, then filter them (pass start= for batches after the first); fit does one corpus in a single call. Only the signatures are kept (16 bytes per window, in Rust). Methods may be called from several threads.

Example:

d = SentenceDedup(min_doc_words=50)          # datatrove's example config
ranges = [d.add(batch) for batch in batches]
for batch, r in zip(batches, ranges):
    texts, keep = d.filter(batch, start=r.start)

Known differences from datatrove:

  • A str with a lone surrogate raises UnicodeEncodeError (so does datatrove, except with split_sentences=False on a document with fewer than n_sentences lines).
  • n_sentences below 1 raises ValueError.
  • A hashed window index past 65,535 raises datatrove's OverflowError (numpy's <u2) when its batch is added, and the batch is not added (datatrove fails when it saves the task's signatures).
  • Index files: use the datatrove drop-ins (pyroid.integrations.datatrove).

Parameters:

  • config (Any, default: None ) –

    As for sentence_dedup.

  • language (str, default: 'eng' ) –

    As for sentence_dedup; any code is accepted when no tokenizer is needed (split_sentences=False and the three word/sentence thresholds <= 0).

  • threads (int | None, default: None ) –

    Default threads= for add, filter and fit.

  • **kwargs (Any, default: {} ) –

    As for sentence_dedup.

config property 

config: SentDedupConfig

The configuration (a SentDedupConfig).

language property 

language: str

The datatrove language code of the word tokenizer.

add 

add(
    texts: str | list[str] | tuple[str, ...] | ArrowStrings,
    *,
    threads: int | None = None,
) -> range

Stage 1 for a batch of documents.

Returns:

  • range –

    The indices given to the batch's documents (Arrow nulls get one, and no signatures).

Raises:

  • OverflowError –

    A window index past 65,535 (nothing is added).

duplicates 

duplicates() -> list[tuple[int, int]]

Stage 2: the (document, window) pairs to remove, sorted.

These are the records of datatrove's .c4_dup files for the task.

filter 

filter(
    texts: str,
    *,
    start: int = 0,
    threads: int | None = None,
) -> SentenceDedupResult[str, bool]

filter(
    texts: list[str],
    *,
    start: int = 0,
    threads: int | None = None,
) -> SentenceDedupResult[list[str], list[bool]]

filter(
    texts: tuple[str, ...],
    *,
    start: int = 0,
    threads: int | None = None,
) -> SentenceDedupResult[tuple[str, ...], tuple[bool, ...]]

filter(
    texts: ArrowStrings,
    *,
    start: int = 0,
    threads: int | None = None,
) -> SentenceDedupResult[Column[str], Column[bool]]

Stage 3 for documents start, start + 1, ... (as added).

Pass the same documents, in the same order, as to add. Decisions are global, so filter once every batch has been added.

Returns:

  • SentenceDedupResult[str, bool] | SentenceDedupResult[list[str], list[bool]] | SentenceDedupResult[tuple[str, ...], tuple[bool, ...]] | SentenceDedupResult[Column[str], Column[bool]] –

    A SentenceDedupResult (text, keep) in the caller's kind.

fit 

fit(
    texts: str, *, threads: int | None = None
) -> SentenceDedupResult[str, bool]

fit(
    texts: list[str], *, threads: int | None = None
) -> SentenceDedupResult[list[str], list[bool]]

fit(
    texts: tuple[str, ...], *, threads: int | None = None
) -> SentenceDedupResult[tuple[str, ...], tuple[bool, ...]]

fit(
    texts: ArrowStrings, *, threads: int | None = None
) -> SentenceDedupResult[Column[str], Column[bool]]

Replace the documents with texts and run the three stages on them.

Returns:

  • SentenceDedupResult[str, bool] | SentenceDedupResult[list[str], list[bool]] | SentenceDedupResult[tuple[str, ...], tuple[bool, ...]] | SentenceDedupResult[Column[str], Column[bool]] –

    A SentenceDedupResult (text, keep) (see sentence_dedup).

reset 

reset() -> None

Forget every document.

languages 

languages() -> dict[str, Language]

Every language code datatrove 0.10.1 knows, with its tokenizer.

The keys are what language= accepts: datatrove's tokenizer_assignment.csv as load_word_tokenizer reads it ("deu_Latn", "deu", "de", ...). pyroid.dedup.languages is the same function.

Example:

>>> pyroid.filters.languages()["cmn_Hani"]
Language(tokenizer='SpaCyTokenizer', arg='zh', native=False)

Returns:

  • dict[str, Language] –

    {code: Language(tokenizer, arg, native)}, sorted by code.

minhash_signatures 

minhash_signatures(
    texts: str,
    config: Any = None,
    *,
    threads: int | None = None,
    language: str | _WordTokenizer = "eng",
    python_tokenizer: bool = False,
    **kwargs: Any,
) -> list[int] | None

minhash_signatures(
    texts: list[str] | tuple[str, ...] | ArrowStrings,
    config: Any = None,
    *,
    threads: int | None = None,
    language: str | _WordTokenizer = "eng",
    python_tokenizer: bool = False,
    **kwargs: Any,
) -> list[list[int] | None]

MinHash signature of each document, identical to datatrove's.

Each signature is num_buckets * hashes_per_bucket ints, bucket by bucket (datatrove's get_signature flattened); None for a document with fewer than n_grams words, which datatrove gives no signature. Arrow nulls give None too.

Parameters:

  • texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

    A str, a list or tuple of str, or an Arrow string array/column.

  • config (Any, default: None ) –

    A MinhashConfig, datatrove's MinhashConfig, a dict of its fields, or None for the defaults.

  • threads (int | None, default: None ) –

    None uses pyroid's shared thread pool; n runs at most n tasks at once; 1 runs on the calling thread.

  • language (str | _WordTokenizer, default: 'eng' ) –

    datatrove's MinhashDedupSignature(language=...): the language code whose word tokenizer splits the simplified text (see "Languages" above), or a word tokenizer object.

  • python_tokenizer (bool, default: False ) –

    Tokenize with datatrove's own tokenizer, in Python on the calling thread (needed for the languages pyroid does not tokenize natively; emits pyroid.FallbackWarning once per function and language). simplify_text, shingling and MinHash still run in Rust.

  • **kwargs (Any, default: {} ) –

    MinhashConfig fields to override, plus hash_fc and precision.

Returns:

  • list[int] | None | list[list[int] | None] –

    One signature (str input) or a list with one entry per document.

Raises:

  • InputError –

    texts is not one of the accepted kinds.

  • TypeError –

    An unknown or mistyped config field, or a language that is neither a str nor a word tokenizer.

  • ValueError –

    A config value out of range, or a language datatrove has no tokenizer for (datatrove's message).

  • NotImplementedError –

    pyroid does not tokenize language natively and python_tokenizer is not set.

  • PyroidError –

    The Python-tokenizer path needs datatrove or the tokenizer's package, and it is not installed.

  • UnicodeEncodeError –

    A str contains a lone surrogate.

  • UnicodeDecodeError –

    An Arrow row is not valid UTF-8.

minhash_dedup 

minhash_dedup(
    texts: str,
    config: Any = None,
    *,
    threads: int | None = None,
    language: str | _WordTokenizer = "eng",
    python_tokenizer: bool = False,
    **kwargs: Any,
) -> bool

minhash_dedup(
    texts: list[str],
    config: Any = None,
    *,
    threads: int | None = None,
    language: str | _WordTokenizer = "eng",
    python_tokenizer: bool = False,
    **kwargs: Any,
) -> list[bool]

minhash_dedup(
    texts: tuple[str, ...],
    config: Any = None,
    *,
    threads: int | None = None,
    language: str | _WordTokenizer = "eng",
    python_tokenizer: bool = False,
    **kwargs: Any,
) -> tuple[bool, ...]

minhash_dedup(
    texts: ArrowStrings,
    config: Any = None,
    *,
    threads: int | None = None,
    language: str | _WordTokenizer = "eng",
    python_tokenizer: bool = False,
    **kwargs: Any,
) -> Column[bool]

Keep mask of datatrove's MinHash dedup over texts (one corpus).

minhash_dedup(texts, cfg) is MinHashDedup(cfg).fit(texts): False exactly for the documents datatrove's four stages remove.

Parameters:

  • texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

    A str, a list or tuple of str, or an Arrow string array/column.

  • config (Any, default: None ) –

    As for minhash_signatures.

  • threads (int | None, default: None ) –

    As for minhash_signatures.

  • language (str | _WordTokenizer, default: 'eng' ) –

    As for minhash_signatures.

  • python_tokenizer (bool, default: False ) –

    As for minhash_signatures.

  • **kwargs (Any, default: {} ) –

    As for minhash_signatures.

Returns:

  • bool | list[bool] | tuple[bool, ...] | Column[bool] –

    A mask in the same kind as texts: bool, list, tuple, or a pyroid.Column of bools (Arrow nulls stay null).

Raises the same exceptions as minhash_signatures.

simplify_text 

simplify_text(
    texts: str,
    config: Any = None,
    *,
    threads: int | None = None,
) -> str

simplify_text(
    texts: list[str],
    config: Any = None,
    *,
    threads: int | None = None,
) -> list[str]

simplify_text(
    texts: tuple[str, ...],
    config: Any = None,
    *,
    threads: int | None = None,
) -> tuple[str, ...]

simplify_text(
    texts: ArrowStrings,
    config: Any = None,
    *,
    threads: int | None = None,
) -> Column[str]

Normalise text with datatrove's simplify_text (what MinHash shingles see).

Byte-identical to datatrove.utils.text.simplify_text on the same CPython version (its unicodedata) with the regex module's digit class. Unchanged documents are returned as the original objects.

Parameters:

  • texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

    A str, a list or tuple of str, or an Arrow string array/column.

  • config (Any, default: None ) –

    A TextNormConfig, datatrove's TextNormConfig, a dict of its fields, or None for the defaults.

  • threads (int | None, default: None ) –

    As for minhash_signatures.

Returns:

  • str | list[str] | tuple[str, ...] | Column[str] –

    The same kind as texts: str, list, tuple, or a pyroid.Column of strings.

Raises:

exact_dedup 

exact_dedup(
    texts: str,
    *,
    normalize: Any = False,
    threads: int | None = None,
) -> bool

exact_dedup(
    texts: list[str],
    *,
    normalize: Any = False,
    threads: int | None = None,
) -> list[bool]

exact_dedup(
    texts: tuple[str, ...],
    *,
    normalize: Any = False,
    threads: int | None = None,
) -> tuple[bool, ...]

exact_dedup(
    texts: ArrowStrings,
    *,
    normalize: Any = False,
    threads: int | None = None,
) -> Column[bool]

Keep mask of exact duplicates: True for the first copy of each text.

Texts are compared by a 128-bit XXH3 hash (for 10⁹ documents the chance of any false match is about 10⁻²¹). Arrow nulls stay null and match nothing.

Parameters:

  • texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

    A str, a list or tuple of str, or an Arrow string array/column.

  • normalize (Any, default: False ) –

    False compares the raw text; True compares simplify_text with the default TextNormConfig; a TextNormConfig (pyroid's or datatrove's) or dict uses that.

  • threads (int | None, default: None ) –

    As for minhash_signatures.

Returns:

  • bool | list[bool] | tuple[bool, ...] | Column[bool] –

    A mask in the same kind as texts.

Raises:

sentence_dedup 

sentence_dedup(
    texts: str,
    config: Any = None,
    *,
    language: str = "eng",
    threads: int | None = None,
    **kwargs: Any,
) -> SentenceDedupResult[str, bool]

sentence_dedup(
    texts: list[str],
    config: Any = None,
    *,
    language: str = "eng",
    threads: int | None = None,
    **kwargs: Any,
) -> SentenceDedupResult[list[str], list[bool]]

sentence_dedup(
    texts: tuple[str, ...],
    config: Any = None,
    *,
    language: str = "eng",
    threads: int | None = None,
    **kwargs: Any,
) -> SentenceDedupResult[tuple[str, ...], tuple[bool, ...]]

sentence_dedup(
    texts: ArrowStrings,
    config: Any = None,
    *,
    language: str = "eng",
    threads: int | None = None,
    **kwargs: Any,
) -> SentenceDedupResult[Column[str], Column[bool]]

The three sentence-dedup stages of datatrove over one corpus.

sentence_dedup(texts, cfg) is SentenceDedup(cfg).fit(texts): the texts and keep decisions of datatrove's SentenceDedupSignature -> SentenceFindDedups -> SentenceDedupFilter run as one task (see SentenceDedup). The input is read once.

Parameters:

  • texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

    A str, a list or tuple of str, or an Arrow string array/column.

  • config (Any, default: None ) –

    A SentDedupConfig, datatrove's SentDedupConfig, a dict of its fields, or None for the defaults.

  • language (str, default: 'eng' ) –

    datatrove's language code (as its language=), for the sentence and word tokenizer: any code pyroid.filters.languages() marks native (spaCy rule-based languages, Chinese with jieba, Vietnamese with pyvi, Japanese with an installed SudachiDict, Thai with an installed PyThaiNLP, the whitespace and IndicNLP tokenizers). Codes whose datatrove tokenizer pyroid does not port (Stanza, Kiwi, ...) raise NotImplementedError; use the datatrove drop-ins.

  • threads (int | None, default: None ) –

    None uses pyroid's shared thread pool; n runs at most n tasks at once; 1 runs on the calling thread.

  • **kwargs (Any, default: {} ) –

    SentDedupConfig fields to override, plus hash_fc and precision.

Returns:

  • SentenceDedupResult[str, bool] | SentenceDedupResult[list[str], list[bool]] | SentenceDedupResult[tuple[str, ...], tuple[bool, ...]] | SentenceDedupResult[Column[str], Column[bool]] –

    A SentenceDedupResult (text, keep).

Raises:

  • InputError –

    texts is not one of the accepted kinds.

  • TypeError –

    An unknown or mistyped config field.

  • ValueError –

    A config value out of range, language is not a datatrove code (datatrove's message), or a text of 2**30 characters or more.

  • NotImplementedError –

    No native word tokenizer for language.

  • OverflowError –

    A document has more than 65,536 sentence windows (datatrove's <u2 window index).

  • UnicodeEncodeError –

    A str contains a lone surrogate.

  • UnicodeDecodeError –

    An Arrow row is not valid UTF-8.