`pyroid.text`¶

Whitespace normalisation and length filtering (pyroid.text).

normalize_whitespace ¶

normalize_whitespace(
    x: str, /, *, threads: int | None = None
) -> str

normalize_whitespace(
    x: list[str], /, *, threads: int | None = None
) -> list[str]

normalize_whitespace(
    x: tuple[str, ...], /, *, threads: int | None = None
) -> tuple[str, ...]

normalize_whitespace(
    x: ArrowStrings, /, *, threads: int | None = None
) -> Column[str]

Collapse every run of whitespace to one space and strip both ends.

Same result as " ".join(s.split()) for every document (whitespace is exactly what str.isspace() accepts). Unchanged documents are returned as the original objects; Arrow nulls stay null.

Parameters:

x (str | list[str] | tuple[str, ...] | ArrowStrings) –

A str, a list or tuple of str, or an Arrow string array/column (string, large_string or string_view).
threads (int | None, default: None ) –

None uses pyroid's shared thread pool; n runs at most n tasks at once; 1 runs on the calling thread.

Returns:

str | list[str] | tuple[str, ...] | Column[str] –

The same kind as x: str, list, tuple, or a pyroid.Column of strings (large_string for large_string input, else string).

Raises:

InputError –

x is not one of the accepted kinds, or holds a non-str item or a non-string Arrow type.
UnicodeEncodeError –

A str contains a lone surrogate.
UnicodeDecodeError –

An Arrow row is not valid UTF-8.

min_chars ¶

min_chars(
    x: str, /, n: int, *, threads: int | None = None
) -> bool

min_chars(
    x: list[str], /, n: int, *, threads: int | None = None
) -> list[bool]

min_chars(
    x: tuple[str, ...],
    /,
    n: int,
    *,
    threads: int | None = None,
) -> tuple[bool, ...]

min_chars(
    x: ArrowStrings,
    /,
    n: int,
    *,
    threads: int | None = None,
) -> Column[bool]

True for each document with at least n characters (len(s) >= n).

A predicate: it never builds new strings, it returns a mask.