Skip to content

pyroid.ftfy

A drop-in, Rust-backed replacement for ftfy 6.3.1 (import pyroid.ftfy as ftfy).

Results are byte-identical to ftfy 6.3.1 on the same CPython version. apply_plan, guess_bytes, fix_file and explain_unicode, and option values the Rust port does not handle (an unknown normalization form, a non-integer max_decode_length), are delegated to the real ftfy with a one-time pyroid.FallbackWarning, or raise pyroid.PyroidError if it is not installed.

Differences from ftfy

  • max_decode_length below 1 raises ValueError. ftfy loops forever on such a value.
  • fix_text on a list or tuple raises TypeError (ftfy raises AttributeError). For other non-str inputs both raise TypeError, with different messages.
  • TextFixerConfig, ExplanationStep and ExplainedText are pyroid's own classes. They have the same fields and defaults and compare equal to ftfy's (they are tuples), but are not instances of ftfy's classes. ftfy's TextFixerConfig objects are accepted wherever a config is.

FTFY_VERSION module-attribute 

FTFY_VERSION: Final[str]

DELEGATED module-attribute 

DELEGATED: Final[tuple[str, ...]]

BYTES_ERROR_TEXT module-attribute 

BYTES_ERROR_TEXT: Final[str]

ExplanationStep

Bases: NamedTuple

One step of an ExplainedText explanation, as in ftfy.

Attributes:

  • action (str) –

    "encode", "decode", "transcode", "apply" or "normalize".

  • parameter (str) –

    The encoding, or the name of the fixer or normalization form.

action instance-attribute 

action: str

parameter instance-attribute 

parameter: str

ExplainedText

Bases: NamedTuple

Fixed text and the steps that fixed it, as in ftfy.

Attributes:

text instance-attribute 

text: str

explanation instance-attribute 

explanation: list[ExplanationStep] | None

TextFixerConfig

Bases: NamedTuple

ftfy's options, with the same fields and defaults.

Pass one to any fixer, or give fields as keyword arguments. Instances of ftfy.TextFixerConfig are accepted too.

Attributes:

  • unescape_html (str | bool) –

    Decode HTML entities such as &amp;. "auto" does it unless the text contains < (then it is probably HTML); True always, False never.

  • remove_terminal_escapes (bool) –

    Remove ANSI terminal escape sequences, such as colour codes.

  • fix_encoding (bool) –

    Fix mojibake: text decoded with the wrong encoding.

  • restore_byte_a0 (bool) –

    When fixing mojibake, allow a space to stand for a lost non-breaking space (byte 0xA0).

  • replace_lossy_sequences (bool) –

    Replace mojibake whose bytes were partly lost (to � or ?) with �.

  • decode_inconsistent_utf8 (bool) –

    Fix UTF-8 mojibake even when the rest of the text does not decode the same way.

  • fix_c1_controls (bool) –

    Replace C1 control characters (U+0080 to U+009F) with the Windows-1252 characters they stand for.

  • fix_latin_ligatures (bool) –

    Replace Latin ligatures such as fi with their letters.

  • fix_character_width (bool) –

    Replace fullwidth Latin characters and halfwidth Katakana with their standard forms.

  • uncurl_quotes (bool) –

    Replace curly quotes with straight ones.

  • fix_line_breaks (bool) –

    Replace \r\n, \r and Unicode line breaks with \n.

  • fix_surrogates (bool) –

    Join UTF-16 surrogate pairs into the characters they encode, and replace lone surrogates with �.

  • remove_control_chars (bool) –

    Remove control characters that have no visible effect on text.

  • normalization (Literal['NFC', 'NFD', 'NFKC', 'NFKD'] | None) –

    The Unicode normalization form applied last, or None for none.

  • max_decode_length (int) –

    The longest segment, in code points, that fix_text fixes at once. Longer lines are split.

  • explain (bool) –

    Record the steps taken. Functions that return no explanation default to False.

unescape_html class-attribute instance-attribute 

unescape_html: str | bool = 'auto'

remove_terminal_escapes class-attribute instance-attribute 

remove_terminal_escapes: bool = True

fix_encoding class-attribute instance-attribute 

fix_encoding: bool = True

restore_byte_a0 class-attribute instance-attribute 

restore_byte_a0: bool = True

replace_lossy_sequences class-attribute instance-attribute 

replace_lossy_sequences: bool = True

decode_inconsistent_utf8 class-attribute instance-attribute 

decode_inconsistent_utf8: bool = True

fix_c1_controls class-attribute instance-attribute 

fix_c1_controls: bool = True

fix_latin_ligatures class-attribute instance-attribute 

fix_latin_ligatures: bool = True

fix_character_width class-attribute instance-attribute 

fix_character_width: bool = True

uncurl_quotes class-attribute instance-attribute 

uncurl_quotes: bool = True

fix_line_breaks class-attribute instance-attribute 

fix_line_breaks: bool = True

fix_surrogates class-attribute instance-attribute 

fix_surrogates: bool = True

remove_control_chars class-attribute instance-attribute 

remove_control_chars: bool = True

normalization class-attribute instance-attribute 

normalization: (
    Literal["NFC", "NFD", "NFKC", "NFKD"] | None
) = "NFC"

max_decode_length class-attribute instance-attribute 

max_decode_length: int = 1000000

explain class-attribute instance-attribute 

explain: bool = True

fix_text 

fix_text(
    text: str,
    config: TextFixerConfig | None = None,
    **kwargs: Any,
) -> str

Fix mojibake and other problems in text, as ftfy.fix_text does.

The text is fixed line by line, in segments of at most max_decode_length code points, each as fix_text_segment fixes it. With unescape_html="auto", HTML entities are decoded until the first line that contains <.

fix_text("naïve résumé") 'naïve résumé'

Parameters:

  • text (str) –

    The text to fix.

  • config (TextFixerConfig | None, default: None ) –

    The options (default: TextFixerConfig(explain=False)).

  • **kwargs (Any, default: {} ) –

    TextFixerConfig fields that override config.

Returns:

  • str –

    The fixed text.

Raises:

ftfy 

ftfy(
    text: str,
    config: TextFixerConfig | None = None,
    **kwargs: Any,
) -> str

Fix text: ftfy's alias for fix_text.

fix_text_batch 

fix_text_batch(
    texts: str,
    config: TextFixerConfig | None = None,
    *,
    threads: int | None = None,
    **kwargs: Any,
) -> str

fix_text_batch(
    texts: list[str],
    config: TextFixerConfig | None = None,
    *,
    threads: int | None = None,
    **kwargs: Any,
) -> list[str]

fix_text_batch(
    texts: tuple[str, ...],
    config: TextFixerConfig | None = None,
    *,
    threads: int | None = None,
    **kwargs: Any,
) -> tuple[str, ...]

fix_text_batch(
    texts: ArrowStrings,
    config: TextFixerConfig | None = None,
    *,
    threads: int | None = None,
    **kwargs: Any,
) -> Column[str]

fix_text over many documents at once, on pyroid's thread pool.

Not part of ftfy. Each result equals ftfy.fix_text(doc, config, **kwargs); documents that need no fixing come back as the original objects, and Arrow nulls stay null. Options the Rust port does not handle are delegated to ftfy for str/list/tuple input and raise pyroid.PyroidError for Arrow.

Parameters:

  • texts (str | list[str] | tuple[str, ...] | ArrowStrings) –

    A str, a list or tuple of str, or an Arrow string array/column (string, large_string or string_view). Lone surrogates in a str are fixed as ftfy fixes them.

  • config (TextFixerConfig | None, default: None ) –

    As for fix_text (default: explain=False).

  • threads (int | None, default: None ) –

    None uses pyroid's shared thread pool; n runs at most n tasks at once; 1 runs on the calling thread.

  • **kwargs (Any, default: {} ) –

    Config fields, as for fix_text.

Returns:

  • str | list[str] | tuple[str, ...] | Column[str] –

    The same kind as texts: str, list, tuple, or a pyroid.Column of strings (large_string for large_string input, else string).

Raises:

  • InputError –

    texts is not one of the accepted kinds, or holds a non-str item or a non-string Arrow type.

  • UnicodeDecodeError –

    An Arrow row is not valid UTF-8.

fix_and_explain 

fix_and_explain(
    text: str,
    config: TextFixerConfig | None = None,
    **kwargs: Any,
) -> ExplainedText

Fix text as one segment and say how, as ftfy.fix_and_explain does.

Unlike fix_text, the text is not split into lines.

Parameters:

  • text (str) –

    The text to fix.

  • config (TextFixerConfig | None, default: None ) –

    The options (default: TextFixerConfig(), with explain=True).

  • **kwargs (Any, default: {} ) –

    TextFixerConfig fields that override config.

Returns:

  • ExplainedText –

    The fixed text and the steps taken (None with explain=False).

Raises:

fix_encoding_and_explain 

fix_encoding_and_explain(
    text: str,
    config: TextFixerConfig | None = None,
    **kwargs: Any,
) -> ExplainedText

Fix only the mojibake in text and say how, as ftfy.fix_encoding_and_explain does.

Of the options, only fix_encoding and those that shape it apply (restore_byte_a0, replace_lossy_sequences, decode_inconsistent_utf8, fix_c1_controls). With fix_encoding=False the text comes back unchanged, with no steps.

Parameters:

  • text (str) –

    The text to fix.

  • config (TextFixerConfig | None, default: None ) –

    The options (default: TextFixerConfig()).

  • **kwargs (Any, default: {} ) –

    TextFixerConfig fields that override config.

Returns:

  • ExplainedText –

    The fixed text and the steps taken (None with explain=False).

Raises:

fix_encoding 

fix_encoding(
    text: str,
    config: TextFixerConfig | None = None,
    **kwargs: Any,
) -> str

Fix only the mojibake in text, as ftfy.fix_encoding does.

fix_encoding("ó") 'ó'

This is fix_encoding_and_explain without the steps.

fix_text_segment 

fix_text_segment(
    text: str,
    config: TextFixerConfig | None = None,
    **kwargs: Any,
) -> str

Fix text as one segment, as ftfy.fix_text_segment does.

This is fix_and_explain without the steps.

apply_plan 

apply_plan(text: str, plan: list[tuple[str, str]]) -> str

Replay plan, an explanation's steps, on text, as ftfy.apply_plan does.

Delegated to ftfy, with a one-time pyroid.FallbackWarning.

Raises:

guess_bytes 

guess_bytes(bstring: bytes) -> tuple[str, str]

Guess the encoding of bstring and decode it, as ftfy.guess_bytes does.

Delegated to ftfy, with a one-time pyroid.FallbackWarning.

Returns:

  • tuple[str, str] –

    The decoded text and the name of the encoding.

Raises:

explain_unicode 

explain_unicode(text: str) -> None

Print each code point of text with its name, as ftfy.explain_unicode does.

Delegated to ftfy, with a one-time pyroid.FallbackWarning.

Raises:

fix_file 

fix_file(
    input_file: TextIO | BinaryIO,
    encoding: str | None = None,
    config: TextFixerConfig | None = None,
    **kwargs: Any,
) -> Iterator[str]

Fix a file line by line, as ftfy.fix_file does.

Delegated to ftfy, with a one-time pyroid.FallbackWarning.

Parameters:

  • input_file (TextIO | BinaryIO) –

    A file open in text or binary mode.

  • encoding (str | None, default: None ) –

    The encoding of a binary file (default: guessed from its first line with guess_bytes).

  • config (TextFixerConfig | None, default: None ) –

    The options (default: TextFixerConfig()).

  • **kwargs (Any, default: {} ) –

    TextFixerConfig fields that override config.

Yields:

  • str –

    The fixed lines.

Raises: