pyroid.warc¶

Bases: Exception

The archive could not be read (warcio's warcio.exceptions.ArchiveLoadFailed).

Raised for an unknown format ("Unknown archive format, first line: ..."), a WARC or ARC record that does not start as one, a gzip file that is not one member per record, and, with check_digests="raise", a digest mismatch.

msg: str

Bases: Exception

An HTTP status line that does not start with a known protocol (verify_http=True).

Attributes:

• statusline (Any) –

  The line, as read.

statusline: Any

A status line and an ordered list of (name, value) headers.

warcio's StatusAndHeaders: rec_headers (the WARC or ARC headers) and http_headers of a record. Lookups by name ignore case; headers[...], get, in and item assignment act like a case-insensitive dict.

Parameters:

• statusline (str) –

  The status line after the protocol ("200 OK"; empty for WARC headers).

• headers (Any) –

  (name, value) pairs (or a mapping); bytes are decoded as latin-1.

• protocol (str, default: '' ) –

  "WARC/1.0", "HTTP/1.1", "GET", ...

• total_len (int, default: 0 ) –

  The characters read for the block, line ends included.

• is_http_request (bool, default: False ) –

  Split the protocol (the verb) off statusline.

ENCODE_HEADER_RX: Pattern[str]

statusline: str

headers: list[tuple[str, str]]

protocol: str

total_len: int

headers_buff: bytes | None

get_header(name: str, default_value: Any = None) -> Any

add_header(name: str, value: str) -> None

replace_header(name: str, value: str) -> str | None

remove_header(name: str) -> bool

get_statuscode() -> str

validate_statusline(valid_statusline: str) -> bool

add_range(
    start: int, part_len: int, total_len: int
) -> StatusAndHeaders

compute_headers_buffer(
    header_filter: Callable[..., Any] | None = None,
) -> None

to_str(
    filter_func: Callable[..., Any] | None = None,
) -> str

to_bytes(
    filter_func: Callable[..., Any] | None = None,
    encoding: str = "utf-8",
) -> bytes

to_ascii_bytes(
    filter_func: Callable[..., Any] | None = None,
) -> bytes

percent_encode_non_ascii_headers(
    encoding: str = "UTF-8",
) -> None

get(name: str, default_value: Any = None) -> Any

The digest checks of one record (warcio's DigestChecker).

record.digest_checker is kept up to date as the record's stream is read.

Parameters:

• kind (Any, default: None ) –

  ArchiveIterator's check_digests: "raise" raises ArchiveLoadFailed at the first problem, "log" also writes each problem to sys.stderr.

kind: Any

passed: bool | None

problems: list[str]

problem(value: str, passed: bool = False) -> None

A record's bytes after its headers (record.raw_stream).

warcio's LimitReader over the iterator's reader: it stops at the record's Content-Length. Reading an old record reads the file where the iterator is now, as in warcio.

limit: int | None

read(length: int | None = None) -> bytes

readline(length: int | None = None) -> bytes

tell() -> int

close() -> None

A record's payload, de-chunked and decompressed (record.content_stream()).

read(length: int | None = None) -> bytes

readline(length: int | None = None) -> bytes

tell() -> int

close() -> None

A WARC or ARC record (warcio's ArcWarcRecord).

Attributes:

• format (str) –

  "warc" or "arc" ("warc" for ARC records read with arc2warc).

• rec_type (str | None) –

  WARC-Type ("response", "request", "warcinfo", ...); "response" or "arc_header" for ARC records.

• rec_headers (StatusAndHeaders) –

  The WARC (or ARC) headers.

• raw_stream (RawStream) –

  The bytes after the headers (the HTTP headers already read).

• http_headers (StatusAndHeaders | None) –

  The HTTP headers of response, request and revisit records with an http: or https: URI, else None.

• content_type (str | None) –

  The record's Content-Type.

• length (int | None) –

  Content-Length as an int (0 when it does not parse), None without one.

• payload_length (int) –

  The bytes after the HTTP headers (-1 when they were not parsed).

• digest_checker (DigestChecker) –

  The record's DigestChecker.

format: str

rec_type: str | None

rec_headers: StatusAndHeaders

raw_stream: RawStream

http_headers: StatusAndHeaders | None

content_type: str | None

length: int | None

payload_length: int

digest_checker: DigestChecker

content_stream() -> RawStream | ContentStream

Iterate over the records of a WARC or ARC file (warcio 1.8.2's ArchiveIterator).

The file may be uncompressed or gzip-compressed with one member per record, as warcio reads them.

Parameters:

• fileobj (SupportsRead[bytes] | StrPath) –

  A binary file object, or a path to open (read with the GIL released).

• no_record_parse (bool, default: False ) –

  Do not parse the HTTP headers.

• verify_http (bool, default: False ) –

  Raise StatusAndHeadersParserException for an HTTP status line that does not start with a known protocol (or verb, for requests).

• arc2warc (bool, default: False ) –

  Present ARC records as WARC records.

• ensure_http_headers (bool, default: False ) –

  Give records without HTTP headers a 200 OK set.

• block_size (int, default: 16384 ) –

  The bytes asked for in each fh.read().

• check_digests (bool | Literal['log', 'raise'], default: False ) –

  Check WARC-Block-Digest and WARC-Payload-Digest: True records the results in record.digest_checker, "log" also writes problems to stderr, "raise" raises ArchiveLoadFailed at the first.

Raises:

• ArchiveLoadFailed –

  The file is not a WARC or ARC file, a record is malformed, or a gzip member holds several records.

• EOFError –

  A gzip member or the file ends inside a record's headers.

Subclass to change the classes it creates (what pyroid.patch("warcio") does): the class attributes record_cls (called as warcio's ArcWarcRecord(format, rec_type, rec_headers, raw_stream, http_headers, content_type, length, payload_length=..., digest_checker=...)), headers_cls (as StatusAndHeaders(statusline=..., headers=..., protocol=..., total_len=...)), digest_checker_cls, archive_load_failed, parser_exception and logger (warcio.recordloader's by default).

GZIP_ERR_MSG: str

INC_RECORD: str

record_cls: Any

headers_cls: Any

digest_checker_cls: Any

archive_load_failed: Any

parser_exception: Any

logger: Logger

no_record_parse: bool

ensure_http_headers: bool

mixed_arc_warc: bool

check_digests: Any

the_iter: Iterator[ArcWarcRecord]

offset: int

err_count: int

member_info: tuple[int, int] | None

known_format: Literal['warc', 'arc'] | None

record: ArcWarcRecord | None

fh: Any

close() -> None

read_to_end(record: object = None) -> None

get_record_offset() -> int

get_record_length() -> int

Bases: ArchiveIterator

ArchiveIterator that reads WARC records only.

Bases: ArchiveIterator

ArchiveIterator that reads ARC records only.

Bases: TypedDict

A document from iter_documents (datatrove's process_record dict).

text: str

id: str | None

url: str

date: str

Documents of a WARC file as an Arrow record batch (no Arrow library needed).

Columns: id (WARC-Record-ID), url, date (WARC-Target-URI and WARC-Date, with datatrove's ARC fallbacks), mime_type (WARC-Identified-Payload-Type), content_type (the HTTP Content-Type), all large_string; html (large_binary: the decoded payload); and text (large_string: datatrove's decoding of it).

pyarrow.record_batch(b), pyarrow.table(b) and polars.DataFrame(b) take it without a copy, through the Arrow PyCapsule interface.

num_rows: int

num_columns: int

column_names: list[str]

column(key: str | int) -> Column[Any]

to_pydict() -> dict[str, list[Any]]