`pyroid.trafilatura`¶

Trafilatura's text extraction, releases 2.3.1 (the default) and 1.11.0, with trafilatura's API and results. Guide: pyroid.trafilatura.

A drop-in, Rust-backed replacement for trafilatura's text extraction (2.3.1 and 1.11.0).

extract and bare_extraction take trafilatura 2.3.1's arguments, with its defaults, and return the same text, byte for byte. The whole path is ported to Rust: trafilatura's load_html and HTML repair, libxml2 2.14.6's HTML parser as lxml 6.1.3 drives it, the tree cleaning, the main extractor, the readability and jusText 3.0.2 comparisons, the baseline and recall fallbacks, deduplication and the text output. Nothing is imported from trafilatura or lxml on that path. compat="1.11.0" reproduces trafilatura 1.11.0 instead, the release datatrove 0.10.1 installs (see below).

Native: a str page (lone surrogates included), or bytes that are valid UTF-8 and not compressed; output_format="txt" ("python" for bare_extraction); fast / no_fallback, favor_precision, favor_recall, include_comments, include_tables, deduplicate (True, or an LRUCache), url, record_id, tei_validation, date_extraction_params (no effect without metadata), max_tree_size (the same ValueError), and config / settingsfile (the size settings are read as trafilatura reads them).

Delegated to the installed trafilatura 2.3.1 with a one-time pyroid.FallbackWarning (per function and reason): the other output formats, include_formatting, include_links, include_images, metadata (with_metadata, only_with_metadata, url_blacklist, author_blacklist, extract_with_metadata), target_language, prune_xpath, options=Extractor(...), deduplicate=trafilatura.deduplication.LRUCache(...), an lxml tree or a response object as input, bytes that are compressed or not UTF-8, a negative size setting, and sys.set_int_max_str_digits() limits other than 4300. Without trafilatura, or with another version of it, these raise pyroid.PyroidError and say why.

trafilatura 1.11.0 (`compat="1.11.0"`)¶

The same options run natively, with 1.11.0's behaviour; the delegated ones go to an installed trafilatura 1.11.0 (PyroidError without it, or with another version). Where 1.11.0 differs:

It has no fast: extract ignores it (1.11.0's extract takes **kwargs) and bare_extraction raises its TypeError for a true value. Its no_fallback is the same switch, without a DeprecationWarning.
max_tree_size works (its len(body) check) instead of raising; author_blacklist does not need metadata, so it has no effect here.
deduplicate is a flag: any true value, an LRUCache included, uses the module cache LRU_TEST, as 1.11.0 always uses its own.
config wins over settingsfile (which is read only with config=None, and then on its own); the default config stands for 1.11.0's settings.cfg (DEFAULT_CONFIGS["1.11.0"]).
bare_extraction raises 1.11.0's AttributeError ('NoneType' object has no attribute 'source') for a page that does not load, as 1.11.0 does when called directly; its commentsbody is None without include_comments.
There is no extract_with_metadata (compat="1.11.0" raises ValueError).
bare_extraction's as_dict defaults to True, as in 1.11.0. extract ignores keywords it does not know, and raises 1.11.0's NameError for the old csv_output / json_output / tei_output / xml_output, as 1.11.0's extract(**kwargs) does (its signature is shown as 2.3.1's; in 2.3.1 mode unknown keywords raise TypeError).
Positional arguments follow this module's (2.3.1's) order: pass 1.11.0's options by keyword. pyroid.patch("trafilatura") with trafilatura 1.11.0 installed gives functions with 1.11.0's exact signatures.

Differences from trafilatura¶

Trafilatura's log messages (logging.getLogger("trafilatura")) are not emitted.
config defaults to this module's DEFAULT_CONFIG, not trafilatura.settings.DEFAULT_CONFIG: changing trafilatura's object changes nothing here.
LRUCache keys are str without lone surrogates; its cache is a read-only copy.
bare_extraction returns a Document whose body and commentsbody become lxml elements when first read (the XML strings when lxml is not installed).

TRAFILATURA_VERSION `module-attribute` ¶

TRAFILATURA_VERSION: Final[str]

The trafilatura release reproduced by default ("2.3.1", the default of compat=).

TRAFILATURA_VERSIONS `module-attribute` ¶

TRAFILATURA_VERSIONS: Final[tuple[str, ...]]

The trafilatura releases pyroid reproduces, one compat= mode each (2.3.1 and 1.11.0).

REPRODUCED_VERSIONS `module-attribute` ¶

REPRODUCED_VERSIONS: Final[Mapping[str, Mapping[str, str]]]

For each mode, the exact library versions it reproduces together.

"2.3.1": trafilatura 2.3.1, lxml 6.1.3, libxml2 2.14.6 (bundled in lxml 6.1.3's wheels), justext 3.0.2. "1.11.0": trafilatura 1.11.0, lxml 6.1.3, libxml2 2.14.6, justext 3.0.2, and lxml-html-clean 0.4.6, charset-normalizer 3.5.2, courlan 1.4.0 and htmldate 1.11.0, as pip install datatrove[processing]==0.10.1 resolves them (these four do not change the text).

DELEGATED `module-attribute` ¶

DELEGATED: Final[tuple[str, ...]]

Functions that always run the installed trafilatura: ("extract_with_metadata",).

LRU_SIZE `module-attribute` ¶

LRU_SIZE: Final[int]

trafilatura.settings.LRU_SIZE (4096): the size of LRU_TEST.

DEFAULT_CONFIG `module-attribute` ¶

DEFAULT_CONFIG: Final[ConfigParser]

trafilatura.settings.DEFAULT_CONFIG: trafilatura 2.3.1's settings.cfg values.

DEFAULT_CONFIGS `module-attribute` ¶

DEFAULT_CONFIGS: Final[Mapping[str, ConfigParser]]

Each mode's trafilatura.settings.DEFAULT_CONFIG (DEFAULT_CONFIGS["2.3.1"] is DEFAULT_CONFIG). With compat="1.11.0" the default config stands for 1.11.0's.

LRU_TEST `module-attribute` ¶

LRU_TEST: Final[LRUCache]

The cache deduplicate=True reads and updates (trafilatura.deduplication.LRU_TEST).

LRUCache ¶

Least Recently Used cache of string counts: trafilatura.deduplication.LRUCache in Rust.

Pass one as deduplicate= to scope deduplication to it. It is safe to share between threads; an extraction holds it for the whole page, so concurrent extractions through one cache run one at a time. Instances pickle and copy with their entries.

maxsize `property` ¶

maxsize: int

The most entries the cache keeps.

cache `property` ¶

cache: MappingProxyType[str, int]

A read-only copy of the entries, oldest first (trafilatura's .cache).

get ¶

get(key: str) -> int

Retrieve a value from the cache, or -1 if the key is absent.

put ¶

put(key: str, value: int) -> None

Store a given key in the cache, evicting the oldest entry if full.

increment ¶

increment(key: str) -> int

Increment the stored count, return the previous count or -1.

clear ¶

clear() -> None

Delete all cache content.

Document ¶

Document(
    *,
    title: str | None = None,
    author: str | None = None,
    url: str | None = None,
    hostname: str | None = None,
    description: str | None = None,
    sitename: str | None = None,
    date: str | None = None,
    categories: list[str] | None = None,
    tags: list[str] | None = None,
    fingerprint: str | None = None,
    idval: str | None = None,
    license_val: str | None = None,
    body: Any = None,
    comments: str | None = None,
    commentsbody: Any = None,
    raw_text: str | None = None,
    text: str | None = None,
    language: str | None = None,
    image: str | None = None,
    pagetype: str | None = None,
    filedate: str | None = None,
)

What bare_extraction returns: trafilatura.settings.Document's fields.

A subclass of trafilatura.settings.Document when trafilatura 2.3.1 is installed (it is imported the first time this class is used), else a class with the same fields, constructor, from_dict and as_dict. The extraction fills text, raw_text (the same str), comments (with include_comments), body and commentsbody; the metadata fields are None, as in trafilatura without metadata extraction. body and commentsbody are parsed into lxml elements when first read (they are the strings etree.tostring(element, encoding="unicode") gives when lxml is not installed).

With compat="1.11.0" the result is a subclass of trafilatura 1.11.0's Document (when 1.11.0 is installed) or of a class with its fields: its constructor takes no arguments, setting an unknown attribute does nothing, and commentsbody is None without include_comments. A tree that holds control characters (1.11.0 keeps them, and lxml cannot build such a tree from Python) stays its XML string.

title `instance-attribute` ¶

title: str | None

author `instance-attribute` ¶

author: str | None

url `instance-attribute` ¶

url: str | None

hostname `instance-attribute` ¶

hostname: str | None

description `instance-attribute` ¶

description: str | None

sitename `instance-attribute` ¶

sitename: str | None

date `instance-attribute` ¶

date: str | None

categories `instance-attribute` ¶

categories: list[str] | None

tags `instance-attribute` ¶

tags: list[str] | None

fingerprint `instance-attribute` ¶

fingerprint: str | None

id `instance-attribute` ¶

id: str | None

license `instance-attribute` ¶

license: str | None

body `instance-attribute` ¶

body: Any

comments `instance-attribute` ¶

comments: str | None

commentsbody `instance-attribute` ¶

commentsbody: Any

raw_text `instance-attribute` ¶

raw_text: str | None

text `instance-attribute` ¶

text: str | None

language `instance-attribute` ¶

language: str | None

image `instance-attribute` ¶

image: str | None

pagetype `instance-attribute` ¶

pagetype: str | None

filedate `instance-attribute` ¶

filedate: str | None

from_dict `classmethod` ¶

from_dict(data: dict[str, Any]) -> Document

Set a series of attributes using a dictionary.

as_dict ¶

as_dict() -> dict[str, Any]

Convert the document to a dictionary (body and commentsbody as lxml elements).

reset_caches ¶

reset_caches() -> None

Empty the module-level dedup cache LRU_TEST (trafilatura.meta.reset_caches).

extract ¶

extract(
    filecontent: str | bytes | Any,
    url: str | None = None,
    record_id: str | None = None,
    fast: bool = False,
    no_fallback: bool = False,
    favor_precision: bool = False,
    favor_recall: bool = False,
    include_comments: bool = True,
    output_format: str = "txt",
    tei_validation: bool = False,
    target_language: str | None = None,
    include_tables: bool = True,
    include_images: bool = False,
    include_formatting: bool | None = None,
    include_links: bool = False,
    deduplicate: bool | LRUCache | Any = False,
    date_extraction_params: dict[str, Any] | None = None,
    with_metadata: bool = False,
    only_with_metadata: bool = False,
    max_tree_size: int | None = None,
    url_blacklist: set[str] | None = None,
    author_blacklist: set[str] | None = None,
    settingsfile: str | None = None,
    prune_xpath: str | list[str] | None = None,
    config: ConfigParser = ...,
    options: Any = None,
    *,
    compat: str = "2.3.1",
) -> str | None

Extract the main text of an HTML page: trafilatura.extract.

page = "
" + "Some words. " * 30 + "
" extract(page)[:22] 'Some words. Some words'

The arguments, defaults and results are trafilatura 2.3.1's; compat is pyroid's only addition. The text is NFC-normalized, as trafilatura's.

Parameters:

filecontent (str | bytes | Any) –

The page: a str, or bytes (UTF-8, else delegated). lxml trees and response objects are delegated; other types give None, as in trafilatura.
url (str | None, default: None ) –

The page's URL (only metadata and links use it).
record_id (str | None, default: None ) –

An ID for the metadata (not part of the txt output).
fast (bool, default: False ) –

Skip the readability and jusText comparison.
no_fallback (bool, default: False ) –

Deprecated alias of fast (trafilatura's DeprecationWarning).
favor_precision (bool, default: False ) –

Prefer less text but correct extraction.
favor_recall (bool, default: False ) –

Prefer more text when unsure (it wins over favor_precision).
include_comments (bool, default: True ) –

Extract comments along with the main text.
output_format (str, default: 'txt' ) –

"txt" runs natively; the others are delegated.
tei_validation (bool, default: False ) –

Only used for "xmltei" (delegated).
target_language (str | None, default: None ) –

Delegated (language detection).
include_tables (bool, default: True ) –

Take <table> content into account.
include_images (bool, default: False ) –

Delegated.
include_formatting (bool | None, default: None ) –

True is delegated.
include_links (bool, default: False ) –

Delegated.
deduplicate (bool | LRUCache | Any, default: False ) –

True deduplicates through the module cache LRU_TEST; an LRUCache scopes it to that cache (a trafilatura LRUCache is delegated).
date_extraction_params (dict[str, Any] | None, default: None ) –

htmldate parameters (only used with metadata).
with_metadata (bool, default: False ) –

Delegated.
only_with_metadata (bool, default: False ) –

Delegated.
max_tree_size (int | None, default: None ) –

Deprecated: a truthy value raises trafilatura's ValueError.
url_blacklist (set[str] | None, default: None ) –

A non-empty set is delegated (it needs metadata).
author_blacklist (set[str] | None, default: None ) –

A non-empty set is delegated (it needs metadata).
settingsfile (str | None, default: None ) –

A settings.cfg file overriding the defaults.
prune_xpath (str | list[str] | None, default: None ) –

Delegated.
config (ConfigParser, default: ... ) –

A ConfigParser with trafilatura's settings (default DEFAULT_CONFIG).
options (Any, default: None ) –

A trafilatura.settings.Extractor is delegated; other values are ignored, as in trafilatura.
compat (str, default: '2.3.1' ) –

The trafilatura release to reproduce, one of TRAFILATURA_VERSIONS ("1.11.0": see the module docstring for how its arguments differ).

Returns:

str | None –

The text, or None where trafilatura returns None (nothing extracted, too short, a duplicate, not HTML).

Raises:

ValueError –

max_tree_size; a page with a &# reference of more than 4300 digits (CPython's int() limit, as in trafilatura); a compat pyroid does not reproduce.
AttributeError –

An unknown output_format (from trafilatura, delegated).
PyroidError –

The call needs trafilatura (see the module docstring), and it is not installed, or is another version than compat.

bare_extraction ¶

bare_extraction(
    filecontent: str | bytes | Any,
    url: str | None = None,
    fast: bool = False,
    no_fallback: bool = False,
    favor_precision: bool = False,
    favor_recall: bool = False,
    include_comments: bool = True,
    output_format: str = "python",
    target_language: str | None = None,
    include_tables: bool = True,
    include_images: bool = False,
    include_formatting: bool | None = None,
    include_links: bool = False,
    deduplicate: bool | LRUCache | Any = False,
    date_extraction_params: dict[str, Any] | None = None,
    with_metadata: bool = False,
    only_with_metadata: bool = False,
    max_tree_size: int | None = None,
    url_blacklist: set[str] | None = None,
    author_blacklist: set[str] | None = None,
    as_dict: Literal[False] = False,
    prune_xpath: str | list[str] | None = None,
    config: ConfigParser = ...,
    options: Any = None,
    *,
    compat: str = "2.3.1",
) -> Document | None

bare_extraction(
    filecontent: str | bytes | Any,
    url: str | None = None,
    fast: bool = False,
    no_fallback: bool = False,
    favor_precision: bool = False,
    favor_recall: bool = False,
    include_comments: bool = True,
    output_format: str = "python",
    target_language: str | None = None,
    include_tables: bool = True,
    include_images: bool = False,
    include_formatting: bool | None = None,
    include_links: bool = False,
    deduplicate: bool | LRUCache | Any = False,
    date_extraction_params: dict[str, Any] | None = None,
    with_metadata: bool = False,
    only_with_metadata: bool = False,
    max_tree_size: int | None = None,
    url_blacklist: set[str] | None = None,
    author_blacklist: set[str] | None = None,
    as_dict: bool = False,
    prune_xpath: str | list[str] | None = None,
    config: ConfigParser = ...,
    options: Any = None,
    *,
    compat: str = "2.3.1",
) -> Document | dict[str, Any] | None

Extract a page's text and trees: trafilatura.bare_extraction.

The arguments are extract's, with output_format="python" (native; the others are delegated) and the deprecated as_dict (DeprecationWarning, then Document.as_dict()).

Returns:

Document | None | Document | dict[str, Any] | None –

A Document (text is not NFC-normalized, as in trafilatura), its as_dict() with as_dict=True, or None.

Raises:

ValueError –

As for extract.
PyroidError –

As for extract.

extract_with_metadata ¶

extract_with_metadata(
    filecontent: str | bytes | Any,
    url: str | None = None,
    record_id: str | None = None,
    fast: bool = False,
    favor_precision: bool = False,
    favor_recall: bool = False,
    include_comments: bool = True,
    output_format: str = "txt",
    tei_validation: bool = False,
    target_language: str | None = None,
    include_tables: bool = True,
    include_images: bool = False,
    include_formatting: bool | None = None,
    include_links: bool = False,
    deduplicate: bool | LRUCache | Any = False,
    date_extraction_params: dict[str, Any] | None = None,
    url_blacklist: set[str] | None = None,
    author_blacklist: set[str] | None = None,
    settingsfile: str | None = None,
    prune_xpath: str | list[str] | None = None,
    config: ConfigParser = ...,
    options: Any = None,
    *,
    compat: str = "2.3.1",
) -> Any

trafilatura.extract_with_metadata: always delegated (metadata extraction).

Returns trafilatura's Document. Needs trafilatura 2.3.1; a pyroid LRUCache passed as deduplicate is lent to it and updated.

Raises:

PyroidError –

trafilatura is not installed, or is another version.
ValueError –

compat="1.11.0" (trafilatura 1.11.0 has no extract_with_metadata).

extract_batch ¶

extract_batch(
    htmls: str,
    *,
    threads: int | None = None,
    compat: str = "2.3.1",
    **kwargs: Any,
) -> str | None

extract_batch(
    htmls: list[str] | tuple[str, ...] | ArrowStrings,
    *,
    threads: int | None = None,
    compat: str = "2.3.1",
    **kwargs: Any,
) -> list[str | None]

extract over many pages, in order, on pyroid's thread pool without the GIL.

Not part of trafilatura. extract_batch(htmls, **kwargs) equals [extract(html, **kwargs) for html in htmls], including with deduplicate: with deduplicate=False the pages are extracted in parallel; with True or an LRUCache they are parsed in parallel and extracted one after the other in order through the cache, so the result (and the cache afterwards) is the loop's. A batch that holds a non-str item (bytes, as extract takes them), or uses a delegated option or no_fallback, runs that loop.

Parameters:

htmls (str | list[str] | tuple[str, ...] | ArrowStrings) –

A list or tuple of pages (str), an Arrow string array/column (pyarrow, polars, pyroid.Column, ...), or one page (str).
threads (int | None, default: None ) –

None uses pyroid's whole thread pool; n at most n tasks at once; 1 the calling thread only.
compat (str, default: '2.3.1' ) –

As for extract.
**kwargs (Any, default: {} ) –

extract's keyword arguments.

Returns:

str | None | list[str | None] –

A list (also for a tuple or Arrow input) with one text or None per page, in order; None for each Arrow null. For a str, extract(htmls, **kwargs).

Raises:

ValueError –

As for extract (the note names the page); threads <= 0.
TypeError –

An argument extract does not take.
InputError –

htmls is not a str, list, tuple or Arrow string column, or a list or tuple holds None (the message gives its index).

pyroid.trafilatura¶

trafilatura 1.11.0 (compat="1.11.0")¶

Differences from trafilatura¶

TRAFILATURA_VERSION module-attribute ¶

TRAFILATURA_VERSIONS module-attribute ¶

REPRODUCED_VERSIONS module-attribute ¶

DELEGATED module-attribute ¶

LRU_SIZE module-attribute ¶

DEFAULT_CONFIG module-attribute ¶

DEFAULT_CONFIGS module-attribute ¶

LRU_TEST module-attribute ¶

LRUCache ¶

maxsize property ¶

cache property ¶

get ¶

put ¶

increment ¶

clear ¶

Document ¶

title instance-attribute ¶

author instance-attribute ¶

url instance-attribute ¶

hostname instance-attribute ¶

description instance-attribute ¶

sitename instance-attribute ¶

date instance-attribute ¶

categories instance-attribute ¶

tags instance-attribute ¶

fingerprint instance-attribute ¶

id instance-attribute ¶

license instance-attribute ¶

body instance-attribute ¶

comments instance-attribute ¶

commentsbody instance-attribute ¶

raw_text instance-attribute ¶

text instance-attribute ¶

language instance-attribute ¶

image instance-attribute ¶

pagetype instance-attribute ¶

filedate instance-attribute ¶

from_dict classmethod ¶

as_dict ¶

reset_caches ¶

extract ¶

bare_extraction ¶

extract_with_metadata ¶

extract_batch ¶

`pyroid.trafilatura`¶

trafilatura 1.11.0 (`compat="1.11.0"`)¶

TRAFILATURA_VERSION `module-attribute` ¶

TRAFILATURA_VERSIONS `module-attribute` ¶

REPRODUCED_VERSIONS `module-attribute` ¶

DELEGATED `module-attribute` ¶

LRU_SIZE `module-attribute` ¶

DEFAULT_CONFIG `module-attribute` ¶

DEFAULT_CONFIGS `module-attribute` ¶

LRU_TEST `module-attribute` ¶

maxsize `property` ¶

cache `property` ¶

title `instance-attribute` ¶

author `instance-attribute` ¶

url `instance-attribute` ¶

hostname `instance-attribute` ¶

description `instance-attribute` ¶

sitename `instance-attribute` ¶

date `instance-attribute` ¶

categories `instance-attribute` ¶

tags `instance-attribute` ¶

fingerprint `instance-attribute` ¶

id `instance-attribute` ¶

license `instance-attribute` ¶

body `instance-attribute` ¶

comments `instance-attribute` ¶

commentsbody `instance-attribute` ¶

raw_text `instance-attribute` ¶

text `instance-attribute` ¶

language `instance-attribute` ¶

image `instance-attribute` ¶

pagetype `instance-attribute` ¶

filedate `instance-attribute` ¶

from_dict `classmethod` ¶