pyroid.nltk¶

Stores data used to perform sentence boundary detection with Punkt.

The attributes are NLTK's. Change them in place (params.abbrev_types.add("dr")) or assign new ones; tokenizers using these parameters see the change at their next call.

abbrev_types: set[str]

collocations: set[tuple[str, str]]

sent_starters: set[str]

ortho_context: defaultdict[str, int]

clear_abbrevs() -> None

clear_collocations() -> None

clear_sent_starters() -> None

clear_ortho_context() -> None

add_ortho_context(typ: str, flag: int) -> None

NLTK's language-dependent variables of the Punkt algorithm.

Subclass it and change sent_end_chars (a tuple of one-character strings) to break sentences at other characters; that still runs in Rust. Overriding anything else (the patterns, word_tokenize) hands the tokenizer to NLTK, with a pyroid.FallbackWarning.

sent_end_chars: tuple[str, ...]

internal_punctuation: str

word_tokenize(s: str) -> list[str]

period_context_re() -> Any

NLTK's PunktSentenceTokenizer, running in Rust.

Parameters:

• train_text (Any, default: None ) –

  Training text (trained with NLTK's PunktTrainer: a pyroid.FallbackWarning), or a PunktParameters (pyroid's or NLTK's) to use. None or empty: empty parameters.

• verbose (bool, default: False ) –

  Passed to the trainer.

• lang_vars (Any, default: None ) –

  A PunktLanguageVars (pyroid's or NLTK's).

• token_cls (Any, default: ... ) –

  NLTK's PunktToken or a subclass; anything but the default runs in NLTK.

PUNCTUATION: tuple[str, ...]

train(train_text: Any, verbose: bool = False) -> Any

tokenize(
    text: str,
    realign_boundaries: bool = True,
    *,
    threads: int | None = None,
) -> list[str]

tokenize(
    text: _Batch,
    realign_boundaries: bool = True,
    *,
    threads: int | None = None,
) -> list[list[str]]

tokenize(
    text: ArrowStrings,
    realign_boundaries: bool = True,
    *,
    threads: int | None = None,
) -> list[list[str] | None]

sentences_from_text(
    text: str,
    realign_boundaries: bool = True,
    *,
    threads: int | None = None,
) -> list[str]

sentences_from_text(
    text: _Batch,
    realign_boundaries: bool = True,
    *,
    threads: int | None = None,
) -> list[list[str]]

sentences_from_text(
    text: ArrowStrings,
    realign_boundaries: bool = True,
    *,
    threads: int | None = None,
) -> list[list[str] | None]

span_tokenize(
    text: str,
    realign_boundaries: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[tuple[int, int]]

span_tokenize(
    text: _Batch,
    realign_boundaries: bool = True,
    *,
    threads: int | None = None,
) -> list[list[tuple[int, int]]]

span_tokenize(
    text: ArrowStrings,
    realign_boundaries: bool = True,
    *,
    threads: int | None = None,
) -> list[list[tuple[int, int]] | None]

tokenize_sents(
    strings: Sequence[str], *, threads: int | None = None
) -> list[list[str]]

span_tokenize_sents(
    strings: Sequence[str], *, threads: int | None = None
) -> Iterator[list[tuple[int, int]]]

text_contains_sentbreak(text: str) -> bool

sentences_from_text_legacy(text: str) -> Iterator[str]

sentences_from_tokens(
    tokens: Iterable[str],
) -> Iterator[list[str]]

debug_decisions(text: str) -> Iterator[dict[str, Any]]

dump(tokens: Any) -> str

Bases: PunktSentenceTokenizer

A PunktSentenceTokenizer with the parameters of an installed punkt_tab model.

Parameters:

• lang (str, default: 'english' ) –

  The model name ("english", "german", ...).

Raises:

• TypeError –

  lang is not a str.

• ValueError –

  lang cannot name one model.

• LookupError –

  The model is not installed (nltk.download('punkt_tab')).

save_dir: str

load_lang(lang: str = 'english') -> None

save_params() -> str

List of contractions adapted from Robert MacIntyre's tokenizer (NLTK's).

CONTRACTIONS2: list[str]

CONTRACTIONS3: list[str]

CONTRACTIONS4: list[str]

Bases: _WordTokenizer

NLTK's TreebankWordTokenizer: the Penn Treebank tokenizer's rules, in Rust.

Splits contractions (don't -> do n't), most punctuation, and a period at the end of the text. Changing its pattern lists (on the class, a subclass or an instance) hands the call to NLTK with those patterns.

Bases: _WordTokenizer

NLTK's NLTKWordTokenizer, the improved Treebank rules word_tokenize uses.

It also splits Unicode quotes and guillemets, dashes, ellipses and some clitics.

NLTK's TreebankWordDetokenizer: undo the Treebank tokenizer's padding, in Rust.

CONTRACTIONS2: Any

CONTRACTIONS3: Any

ENDING_QUOTES: Any

DOUBLE_DASHES: Any

CONVERT_PARENTHESES: Any

PARENS_BRACKETS: Any

PUNCTUATION: Any

STARTING_QUOTES: Any

tokenize(
    tokens: Sequence[str], convert_parentheses: bool = False
) -> str

detokenize(
    tokens: Sequence[str], convert_parentheses: bool = False
) -> str

NLTK's RegexpTokenizer: the matches of a pattern, or the text between them.

These patterns run in Rust when spelled exactly so and compiled with no flags beyond the default re.UNICODE | re.MULTILINE | re.DOTALL: \w+, \w+|[^\w\s]+, \w+|\$[\d\.]+|\S+, [\w']+, \S+, \s+, [a-zA-Z]+ and BlanklineTokenizer's. Any other pattern runs in NLTK, with one pyroid.FallbackWarning.

Parameters:

• pattern (Any) –

  The pattern (a str, or a compiled pattern whose .pattern is used).

• gaps (bool, default: False ) –

  The pattern matches the separators, not the tokens.

• discard_empty (bool, default: True ) –

  With gaps, drop empty tokens.

• flags (int, default: ... ) –

  The flags the pattern is compiled with.

tokenize(
    text: str, *, threads: int | None = None
) -> list[str]

tokenize(
    text: _Batch, *, threads: int | None = None
) -> list[list[str]]

tokenize(
    text: ArrowStrings, *, threads: int | None = None
) -> list[list[str] | None]

span_tokenize(text: str) -> Iterator[tuple[int, int]]

tokenize_sents(
    strings: Sequence[str], *, threads: int | None = None
) -> list[list[str]]

span_tokenize_sents(
    strings: Sequence[str],
) -> Iterator[list[tuple[int, int]]]

Bases: RegexpTokenizer

Split text at runs of whitespace (\s+ with gaps=True, regex's \s).

Bases: RegexpTokenizer

Split text at blank lines (lines of nothing but whitespace).

Bases: RegexpTokenizer

Split text into runs of word characters and of other non-space characters.

The pattern is \w+|[^\w\s]+.