`pyroid.jieba`¶

Chinese word segmentation with jieba 0.42.1's API and results. Guide: pyroid.jieba.

A drop-in, Rust-backed replacement for jieba 0.42.1's word segmentation.

import pyroid.jieba as jieba

jieba.lcut("我来到北京清华大学")  # ['我', '来到', '北京', '清华大学']
jieba.lcut("我来到北京清华大学", cut_all=True)  # every dictionary word
jieba.lcut_for_search("小明硕士毕业于中国科学院计算所")
list(jieba.tokenize("永和服装饰品有限公司"))  # [('永和', 0, 2), ...]
jieba.load_userdict("userdict.txt")
jieba.add_word("石墨烯")
jieba.lcut(docs, threads=8)  # one list of words per document, in parallel

The words, offsets, frequencies and exceptions are jieba 0.42.1's for every input: the blocks of re_han_default / re_skip_default, the prefix-dictionary DAG, the maximum-probability route (the same floating-point operations and tie-breaking), and the BMES HMM for unknown words (HMM=True) are ported step for step, as are add_word, del_word, suggest_freq, load_userdict's line format and Tokenizer with a custom dictionary file. The default dictionary (jieba's dict.txt, MIT) is built into pyroid as a prebuilt trie, so loading it takes about 3 ms (jieba: about 0.4 s, with or without its cache file) and writes no cache file; jieba itself is not needed.

Delegated to the real jieba (it must be installed; otherwise pyroid.PyroidError), with one pyroid.FallbackWarning per feature: pyroid.jieba.posseg (part-of-speech tags), pyroid.jieba.analyse (TF-IDF and TextRank keywords), paddle mode (enable_paddle, use_paddle=True) and get_dict_file for the default dictionary. Before each delegated call, the words added through pyroid are replayed onto jieba's tokenizer, so those features see them.

pyroid extensions: the segmentation functions also take a list or tuple of str, or an Arrow string column, and return one list of words per document (None for an Arrow null), segmented in parallel on pyroid's shared pool (threads=: None for the whole pool, n for at most n tasks, 1 for the calling thread).

Differences from jieba:

Lone surrogates ("\ud800") raise UnicodeEncodeError, as everywhere in pyroid; jieba treats them as characters.
Frequencies must fit in a signed 64-bit integer (OverflowError otherwise); jieba takes any Python int.
cut is a generator, as in jieba, but it segments the whole input on the first next().
FREQ is a read-only mapping (change it with add_word); iterating it lists the dictionary's keys in UTF-8 byte order, not file order.
enable_parallel uses threads instead of processes and also works on Windows.
enable_paddle does not pip install paddlepaddle-tiny for you.
Wrong argument types raise TypeError (pyroid.InputError for the text) where jieba raises AttributeError.
There is no finalseg module; del_word and add_word(word, 0) still update the process-wide set of words the HMM must split, as jieba's Force_Split_Words.
jieba's log messages ("Building prefix dict ...") are not printed.

JIEBA_VERSION `module-attribute` ¶

JIEBA_VERSION: Final[str]

version `module-attribute` ¶

__version__: Final[str]

DEFAULT_DICT `module-attribute` ¶

DEFAULT_DICT: None

DEFAULT_DICT_NAME `module-attribute` ¶

DEFAULT_DICT_NAME: str

re_userdict `module-attribute` ¶

re_userdict: Pattern[str]

re_eng `module-attribute` ¶

re_eng: Pattern[str]

re_han_default `module-attribute` ¶

re_han_default: Pattern[str]

re_skip_default `module-attribute` ¶

re_skip_default: Pattern[str]

text_type `module-attribute` ¶

text_type: type[str]

string_types `module-attribute` ¶

string_types: tuple[type[str]]

default_logger `module-attribute` ¶

default_logger: Any

dt `module-attribute` ¶

dt: Tokenizer

user_word_tag_tab `module-attribute` ¶

user_word_tag_tab: dict[str, str]

Tokenizer ¶

Tokenizer(dictionary: str | None = None)

A jieba tokenizer: a main dictionary plus the words added to it.

Tokenizer() uses jieba's default dictionary (built in). Tokenizer(dictionary=path) reads a jieba-format dictionary file on first use: one word freq [tag] per line, UTF-8; for a repeated word the last frequency wins, and total adds up every line, as in jieba.

Parameters:

dictionary (str | None, default: None ) –

None for the default dictionary, or a file path.

lock `instance-attribute` ¶

lock: Any

dictionary `instance-attribute` ¶

dictionary: str | None

user_word_tag_tab `instance-attribute` ¶

user_word_tag_tab: dict[str, str]

initialized `instance-attribute` ¶

initialized: bool

tmp_dir `instance-attribute` ¶

tmp_dir: str | None

cache_file `instance-attribute` ¶

cache_file: str | None

FREQ `property` ¶

FREQ: Mapping[str, int]

Every word and word prefix with its frequency (0 for a prefix); empty until loaded.

total `property` ¶

total: int

The sum of the frequencies (0 until initialized).

gen_pfdict `staticmethod` ¶

gen_pfdict(f: IO[bytes]) -> tuple[dict[str, int], int]

Read a binary dictionary file into jieba's (FREQ, total) (a plain dict).

Raises:

ValueError –

A line is not word freq ... with an int freq (invalid dictionary entry in <file> at Line <n>: <line>).

initialize ¶

initialize(dictionary: str | None = None) -> None

Load the dictionary now (otherwise it loads on first use).

Parameters:

dictionary (str | None, default: None ) –

Switch to this dictionary file first.

Raises:

ValueError –

The dictionary file has an invalid line.
OSError –

The dictionary file cannot be read.

check_initialized ¶

check_initialized() -> None

Load the dictionary if it is not loaded yet.

calc ¶

calc(
    sentence: str,
    DAG: dict[int, list[int]],
    route: dict[int, Any],
) -> None

Fill route with the maximum-probability route over DAG (jieba's calc).

get_DAG ¶

get_DAG(sentence: str) -> dict[int, list[int]]

For each character index of sentence, the ends of the dictionary words there.

An index where no dictionary word starts maps to itself.

cut ¶

cut(
    sentence: _Sentence,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> Iterator[str]

cut(
    sentence: _Batch,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> Iterator[list[str]]

cut(
    sentence: ArrowStrings,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> Iterator[list[str] | None]

Segment sentence into words; yields them.

Parameters:

sentence (_Sentence | _Batch | ArrowStrings) –

A str (bytes are decoded as UTF-8, else GBK), or a batch: a list or tuple of str, or an Arrow string array/column.
cut_all (bool, default: False ) –

Full mode: every dictionary word in the text, overlapping (empty strings included, as jieba yields them).
HMM (bool, default: True ) –

Find words that are not in the dictionary with the HMM.
use_paddle (bool, default: False ) –

Use paddle mode if enable_paddle enabled it (delegated to jieba); ignored otherwise, as in jieba.
threads (int | None, default: None ) –

For a batch: None for the whole pool, n for at most n tasks at once, 1 for the calling thread.

Yields:

Iterator[str] | Iterator[list[str]] | Iterator[list[str] | None] –

The words, which concatenate to sentence (except in full mode); for a batch, one list of words per document (None for an Arrow null).

Raises:

ValueError –

The dictionary's total is 0, or a word on the route has a negative frequency (math.log's error).
InputError –

A batch holds a non-str item.
UnicodeEncodeError –

A str contains a lone surrogate.

lcut ¶

lcut(
    sentence: _Sentence,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> list[str]

lcut(
    sentence: _Batch,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> list[list[str]]

lcut(
    sentence: ArrowStrings,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> list[list[str] | None]

cut as a list (for a batch, a list of lists).

cut_for_search ¶

cut_for_search(
    sentence: _Sentence,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[str]

cut_for_search(
    sentence: _Batch,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[list[str]]

cut_for_search(
    sentence: ArrowStrings,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[list[str] | None]

Segment for search engines: cut, plus the dictionary words inside long ones.

Each word longer than 2 characters is preceded by the 2-character dictionary words inside it, and one longer than 3 also by the 3-character ones.

Parameters:

sentence (_Sentence | _Batch | ArrowStrings) –

A str, or a batch as for cut.
HMM (bool, default: True ) –

Find words that are not in the dictionary with the HMM.
threads (int | None, default: None ) –

For a batch, as for cut.

Raises:

ValueError –

As for cut.

lcut_for_search ¶

lcut_for_search(
    sentence: _Sentence,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> list[str]

lcut_for_search(
    sentence: _Batch,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> list[list[str]]

lcut_for_search(
    sentence: ArrowStrings,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> list[list[str] | None]

cut_for_search as a list (for a batch, a list of lists).

get_dict_file ¶

get_dict_file() -> IO[bytes]

The main dictionary file, opened in binary mode.

For the default dictionary this is jieba's own dict.txt (pyroid stores the dictionary in another form), so it is delegated to jieba.

Raises:

PyroidError –

The default dictionary and jieba is not installed.

load_userdict ¶

load_userdict(f: str | IO[bytes] | IO[str]) -> None

Add the words of a user dictionary, as add_word does for each line.

One entry per line: word [freq] [tag], separated by spaces (jieba's re_userdict: a word may contain spaces; freq is ASCII digits; tag lowercase ASCII letters). Blank lines are skipped.

Parameters:

f (str | IO[bytes] | IO[str]) –

A path, or a file opened in binary mode (UTF-8, an initial BOM is ignored) or in text mode.

Raises:

ValueError –

A file given by path or opened in binary mode is not UTF-8.

add_word ¶

add_word(
    word: str | bytes,
    freq: Any = None,
    tag: str | None = None,
) -> None

Add word to the dictionary, or change its frequency.

Parameters:

word (str | bytes) –

The word.
freq (Any, default: None ) –

Its frequency (anything int() takes); None uses suggest_freq, which makes the word cut whole.
tag (str | None, default: None ) –

A part-of-speech tag, kept in user_word_tag_tab.

Raises:

ValueError –

int(freq) fails.
OverflowError –

freq or the new total does not fit in 64 bits.

del_word ¶

del_word(word: str | bytes) -> None

Remove word: set its frequency to 0 and stop the HMM from producing it.

suggest_freq ¶

suggest_freq(
    segment: str | tuple[str, ...] | list[str],
    tune: bool = False,
) -> int

The frequency that makes segment cut as given.

Parameters:

segment (str | tuple[str, ...] | list[str]) –

A str, to keep it whole; or a tuple of str, to cut their concatenation into them.
tune (bool, default: False ) –

Also set the word to that frequency (add_word).

Returns:

int –

jieba's value. The HMM can still change how the word is cut; if so, cut with HMM=False.

Raises:

ValueError –

The dictionary's total is 0 (math.log's error).
ZeroDivisionError –

The dictionary's total is 0.

tokenize ¶

tokenize(
    unicode_sentence: str,
    mode: str = "default",
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[_Token]

tokenize(
    unicode_sentence: _Batch,
    mode: str = "default",
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[list[_Token]]

tokenize(
    unicode_sentence: ArrowStrings,
    mode: str = "default",
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[list[_Token] | None]

Yield (word, start, end) for each word, with character offsets.

Parameters:

unicode_sentence (str | _Batch | ArrowStrings) –

A str, or a batch as for cut (then one list of tuples per document).
mode (str, default: 'default' ) –

"default" for cut's words; anything else for cut_for_search's.
HMM (bool, default: True ) –

Find words that are not in the dictionary with the HMM.
threads (int | None, default: None ) –

For a batch, as for cut.

Raises:

ValueError –

unicode_sentence is not a str or a batch (jieba: the input parameter should be unicode.), or as for cut.

set_dictionary ¶

set_dictionary(dictionary_path: str) -> None

Use another main dictionary file from now on (loaded on next use, without added words).

Raises:

Exception –

The file does not exist (jieba: file does not exist: <path>).

setLogLevel ¶

setLogLevel(log_level: int | str) -> None

Set the level of this module's logger (jieba's setLogLevel).

strdecode ¶

strdecode(sentence: Any) -> Any

Decode bytes as jieba does: UTF-8, else GBK ignoring errors; anything else as is.

resolve_filename ¶

resolve_filename(f: Any) -> str

f.name, or repr(f) for an object without one.

enable_paddle ¶

enable_paddle() -> None

Enable jieba's paddle mode (delegated to jieba, with a FallbackWarning).

Raises:

PyroidError –

jieba or paddlepaddle-tiny is not installed.

get_FREQ ¶

get_FREQ(k: str, d: Any = None) -> Any

dt.FREQ.get(k, d).

cut ¶

cut(
    sentence: _Sentence,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> Iterator[str]

cut(
    sentence: _Batch,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> Iterator[list[str]]

cut(
    sentence: ArrowStrings,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> Iterator[list[str] | None]

Tokenizer.cut on the default tokenizer.

lcut ¶

lcut(
    sentence: _Sentence,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> list[str]

lcut(
    sentence: _Batch,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> list[list[str]]

lcut(
    sentence: ArrowStrings,
    cut_all: bool = False,
    HMM: bool = True,
    use_paddle: bool = False,
    *,
    threads: int | None = None,
) -> list[list[str] | None]

Tokenizer.lcut on the default tokenizer.

cut_for_search ¶

cut_for_search(
    sentence: _Sentence,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[str]

cut_for_search(
    sentence: _Batch,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[list[str]]

cut_for_search(
    sentence: ArrowStrings,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[list[str] | None]

Tokenizer.cut_for_search on the default tokenizer.

lcut_for_search ¶

lcut_for_search(
    sentence: _Sentence,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> list[str]

lcut_for_search(
    sentence: _Batch,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> list[list[str]]

lcut_for_search(
    sentence: ArrowStrings,
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> list[list[str] | None]

Tokenizer.lcut_for_search on the default tokenizer.

tokenize ¶

tokenize(
    unicode_sentence: str,
    mode: str = "default",
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[_Token]

tokenize(
    unicode_sentence: _Batch,
    mode: str = "default",
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[list[_Token]]

tokenize(
    unicode_sentence: ArrowStrings,
    mode: str = "default",
    HMM: bool = True,
    *,
    threads: int | None = None,
) -> Iterator[list[_Token] | None]

Tokenizer.tokenize on the default tokenizer.

add_word ¶

add_word(
    word: str | bytes,
    freq: Any = None,
    tag: str | None = None,
) -> None

Tokenizer.add_word on the default tokenizer.

del_word ¶

del_word(word: str | bytes) -> None

Tokenizer.del_word on the default tokenizer.

suggest_freq ¶

suggest_freq(
    segment: str | tuple[str, ...] | list[str],
    tune: bool = False,
) -> int

Tokenizer.suggest_freq on the default tokenizer.

load_userdict ¶

load_userdict(f: str | IO[bytes] | IO[str]) -> None

Tokenizer.load_userdict on the default tokenizer.

calc ¶

calc(
    sentence: str,
    DAG: dict[int, list[int]],
    route: dict[int, Any],
) -> None

Tokenizer.calc on the default tokenizer.

get_DAG ¶

get_DAG(sentence: str) -> dict[int, list[int]]

Tokenizer.get_DAG on the default tokenizer.

get_dict_file ¶

get_dict_file() -> IO[bytes]

Tokenizer.get_dict_file on the default tokenizer.

initialize ¶

initialize(dictionary: str | None = None) -> None

Tokenizer.initialize on the default tokenizer.

set_dictionary ¶

set_dictionary(dictionary_path: str) -> None

Tokenizer.set_dictionary on the default tokenizer.

enable_parallel ¶

enable_parallel(processnum: int | None = None) -> None

Make the module-level cut and cut_for_search cut lines in parallel.

As in jieba, only those two functions change (on dt). pyroid uses threads, at most processnum (the whole pool by default), and works on every platform; the words are the same.

Parameters:

processnum (int | None, default: None ) –

The most threads to cut with; None for the whole pool.

Raises:

ValueError –

processnum is below 1 ("Number of processes must be at least 1", as jieba's multiprocessing.Pool).
TypeError –

processnum is not an int (as jieba's Pool).

disable_parallel ¶

disable_parallel() -> None

Undo enable_parallel.

pyroid.jieba¶

JIEBA_VERSION module-attribute ¶

__version__ module-attribute ¶

DEFAULT_DICT module-attribute ¶

DEFAULT_DICT_NAME module-attribute ¶

re_userdict module-attribute ¶

re_eng module-attribute ¶

re_han_default module-attribute ¶

re_skip_default module-attribute ¶

text_type module-attribute ¶

string_types module-attribute ¶

default_logger module-attribute ¶

dt module-attribute ¶

user_word_tag_tab module-attribute ¶

Tokenizer ¶

lock instance-attribute ¶

dictionary instance-attribute ¶

user_word_tag_tab instance-attribute ¶

initialized instance-attribute ¶

tmp_dir instance-attribute ¶

cache_file instance-attribute ¶

FREQ property ¶

total property ¶

gen_pfdict staticmethod ¶

initialize ¶

check_initialized ¶

calc ¶

get_DAG ¶

cut ¶

lcut ¶

cut_for_search ¶

lcut_for_search ¶

get_dict_file ¶

load_userdict ¶

add_word ¶

del_word ¶

suggest_freq ¶

tokenize ¶

set_dictionary ¶

setLogLevel ¶

strdecode ¶

resolve_filename ¶

enable_paddle ¶

get_FREQ ¶

cut ¶

lcut ¶

cut_for_search ¶

lcut_for_search ¶

tokenize ¶

add_word ¶

del_word ¶

suggest_freq ¶

load_userdict ¶

calc ¶

get_DAG ¶

get_dict_file ¶

initialize ¶

set_dictionary ¶

enable_parallel ¶

disable_parallel ¶

`pyroid.jieba`¶

JIEBA_VERSION `module-attribute` ¶

version `module-attribute` ¶

DEFAULT_DICT `module-attribute` ¶

DEFAULT_DICT_NAME `module-attribute` ¶

re_userdict `module-attribute` ¶

re_eng `module-attribute` ¶

re_han_default `module-attribute` ¶

re_skip_default `module-attribute` ¶

text_type `module-attribute` ¶

string_types `module-attribute` ¶

default_logger `module-attribute` ¶

dt `module-attribute` ¶

user_word_tag_tab `module-attribute` ¶

lock `instance-attribute` ¶

dictionary `instance-attribute` ¶

user_word_tag_tab `instance-attribute` ¶

initialized `instance-attribute` ¶

tmp_dir `instance-attribute` ¶

cache_file `instance-attribute` ¶

FREQ `property` ¶

total `property` ¶

gen_pfdict `staticmethod` ¶