`pyroid.fasttext`¶

fastText inference in Rust (pyroid.fasttext).

ModelError ¶

Bases: PyroidError, ValueError

A fastText model file is malformed, truncated, or not a fastText model.

A ValueError, like the errors fastText's own loader raises.

Model ¶

A loaded fastText model.

The inference methods of fastText's _FastText, with the same results. predict and get_sentence_vector also accept a batch: a list or tuple of str, or an Arrow string array/column, processed on pyroid's thread pool without the GIL.

words `property` ¶

words: list[str]

The dictionary's words (get_words(), computed once).

labels `property` ¶

labels: list[str]

The model's labels (get_labels(), computed once).

predict ¶

predict(
    text: str,
    k: int = 1,
    threshold: float = 0.0,
    on_unicode_error: str = "strict",
    *,
    threads: int | None = None,
) -> tuple[tuple[str, ...], _F64]

predict(
    text: _Batch,
    k: int = 1,
    threshold: float = 0.0,
    on_unicode_error: str = "strict",
    *,
    threads: int | None = None,
) -> tuple[list[list[str]], list[_F32]]

Predict the most likely labels of each line of text.

As fastText: the text is split on spaces, tabs, vertical tabs, form feeds, carriage returns and NULs; tokens with the label prefix are ignored; the end-of-line token </s> is always added (so an empty line still has predictions). Probabilities are fastText's, which add 1e-5 and so can slightly exceed 1.

Parameters:

text (str | _Batch) –

One line (no "\n"), or a batch of lines: a list or tuple of str, or an Arrow string array/column.
k (int, default: 1 ) –

Number of labels to return, best first; -1 for all of them.
threshold (float, default: 0.0 ) –

Only return labels with at least this probability.
on_unicode_error (str, default: 'strict' ) –

Error handler for decoding labels that are not valid UTF-8 ("strict", "replace", ...).
threads (int | None, default: None ) –

None uses pyroid's shared thread pool; n runs at most n tasks at once; 1 runs on the calling thread.

Returns:

tuple[tuple[str, ...], _F64] | tuple[list[list[str]], list[_F32]] –

For a str: (labels, probabilities), a tuple of str and a float64 array. For a batch: (list of label lists, list of float32 arrays), one entry per line (None for Arrow nulls).

Raises:

ValueError –

A line contains "\n"; k is 0 or below -1; the model is unsupervised.
TypeError –

A line contains a lone surrogate, or k does not fit in a 32-bit int (as fastText). A "\n" anywhere takes precedence (ValueError), as in fastText.
InputError –

text is not a str or a batch of str.
RuntimeError –

The model's weights produce NaN ("Encountered NaN.").

get_sentence_vector ¶

get_sentence_vector(
    text: str, *, threads: int | None = None
) -> _F32

get_sentence_vector(
    text: _Batch, *, threads: int | None = None
) -> _F32

The sentence vector of a line (float32, get_dimension() values).

Supervised models average the input rows predict uses (the hidden layer); unsupervised models average the L2-normalized vectors of the whitespace-separated words.

Parameters:

text (str | _Batch) –

One line (no "\n"), or a batch of lines.
threads (int | None, default: None ) –

As for predict.

Returns:

_F32 –

A 1-D float32 array for a str; a 2-D (len(batch), dim) float32 array for a batch (rows of NaN for Arrow nulls).

Raises:

ValueError –

A line contains "\n".
TypeError –

A line contains a lone surrogate.

get_word_vector ¶

get_word_vector(word: str) -> _F32

The vector of word: the mean of its subword rows (float32).

Works for any word: unknown words use their character n-grams.

get_input_vector ¶

get_input_vector(ind: int) -> _F32

Row ind of the input matrix (float32).

Raises:

IndexError –

ind is outside the input matrix.

get_words ¶

get_words(
    include_freq: Literal[False] = False,
    on_unicode_error: str = "strict",
) -> list[str]

get_words(
    include_freq: Literal[True],
    on_unicode_error: str = "strict",
) -> tuple[list[str], _I64]

The words of the dictionary (no labels, no subwords).

Parameters:

include_freq (Literal[False] | Literal[True], default: False ) –

Also return each word's training count.
on_unicode_error (str, default: 'strict' ) –

Error handler for words that are not valid UTF-8.

Returns:

list[str] | tuple[list[str], _I64] –

The words, or (words, counts) with an int64 array of counts.

get_labels ¶

get_labels(
    include_freq: Literal[False] = False,
    on_unicode_error: str = "strict",
) -> list[str]

get_labels(
    include_freq: Literal[True],
    on_unicode_error: str = "strict",
) -> tuple[list[str], _I64]

The labels, prefix included. Unsupervised models return their words.

Parameters:

include_freq (Literal[False] | Literal[True], default: False ) –

Also return each label's training count.
on_unicode_error (str, default: 'strict' ) –

Error handler for labels that are not valid UTF-8.

Returns:

list[str] | tuple[list[str], _I64] –

The labels, or (labels, counts) with an int64 array of counts.

get_line ¶

get_line(
    text: str, on_unicode_error: str = "strict"
) -> tuple[list[str], list[str]]

get_line(
    text: _Batch, on_unicode_error: str = "strict"
) -> tuple[list[list[str]], list[list[str]]]

Split a line into its word tokens (</s> included) and known labels.

Parameters:

text (str | _Batch) –

One line (no "\n"), or a batch of lines.
on_unicode_error (str, default: 'strict' ) –

Accepted for compatibility (tokens of a str are always valid UTF-8).

Returns:

tuple[list[str], list[str]] | tuple[list[list[str]], list[list[str]]] –

(words, labels) for a str; (list of word lists, list of label lists) for a batch.

Raises:

ValueError –

A line contains "\n".

get_subwords ¶

get_subwords(
    word: str, on_unicode_error: str = "strict"
) -> tuple[list[str], _I64]

The word (if known) and its character n-grams, with their input rows.

As in fastText, a pruned (quantized) model lists n-grams that have no row, so the two results can differ in length.

get_word_id ¶

get_word_id(word: str) -> int

Dictionary id of word (labels included), or -1.

get_label_id ¶

get_label_id(label: str) -> int

Label index of label, or -1.

fastText's arithmetic: a known word gives a negative id other than -1.

get_subword_id ¶

get_subword_id(subword: str) -> int

The input-matrix row a character n-gram hashes to (no pruning).

Raises:

ValueError –

The model has no n-gram buckets.

get_dimension ¶

get_dimension() -> int

Dimension of the word and sentence vectors.

is_quantized ¶

is_quantized() -> bool

True for a quantized (.ftz) model.

get_input_matrix ¶

get_input_matrix() -> _F32

A copy of the input matrix (float32, rows x dim).

Raises:

ValueError –

The model is quantized ("Can't get quantized Matrix").

get_output_matrix ¶

get_output_matrix() -> _F32

A copy of the output matrix (float32, rows x dim).

Raises:

ValueError –

The model is quantized ("Can't get quantized Matrix").

load_model ¶

load_model(path: str | PathLike[str]) -> Model

Load a fastText model: a .bin file, or a quantized .ftz.

The whole file is read and validated in one pass (the GIL is released meanwhile). The model is immutable: share it across threads.

Parameters:

path (str | PathLike[str]) –

The model file.

Returns:

Model –

The loaded Model.

Raises:

ModelError –

The file is not a fastText model ("... has wrong file format!"), is truncated, or is internally inconsistent (sizes or ids that fastText itself would read out of bounds).
OSError –

The file cannot be opened or read (FileNotFoundError, PermissionError, ...; fastText raises ValueError here).
MemoryError –

A matrix of a valid file does not fit in memory (as fastText's std::bad_alloc).

tokenize ¶

tokenize(text: str) -> list[str]

Split text into fastText tokens, as fasttext.tokenize.

Tokens are separated by spaces, \t, \v, \f, \r, \n and NUL; each \n also yields the end-of-line token "</s>".

Raises:

TypeError –

text is not a str, or contains a lone surrogate.

pyroid.fasttext¶

ModelError ¶

Model ¶

words property ¶

labels property ¶

predict ¶

get_sentence_vector ¶

get_word_vector ¶

get_input_vector ¶

get_words ¶

get_labels ¶

get_line ¶

get_subwords ¶

get_word_id ¶

get_label_id ¶

get_subword_id ¶

get_dimension ¶

is_quantized ¶

get_input_matrix ¶

get_output_matrix ¶

load_model ¶

tokenize ¶

`pyroid.fasttext`¶

words `property` ¶

labels `property` ¶