Skip to content

pyroid.metrics

ROUGE as a drop-in for rouge-score 0.1.2's modules, and BLEU and chrF as a drop-in for sacrebleu 2.6.0. Guide: pyroid.metrics.

pyroid.metrics.rouge_scorer

A drop-in, Rust-backed replacement for rouge_score.rouge_scorer (rouge-score 0.1.2).

from pyroid.metrics import rouge_scorer, scoring replaces from rouge_score import rouge_scorer, scoring. RougeScorer takes rouge-score's arguments and returns the same dict[str, Score]: the same floats bit for bit, int zeros where rouge-score has them, the same key order, the same ValueError messages. ROUGE-N (rouge1..rouge9), ROUGE-L and ROUGE-Lsum are supported, with or without nltk 3.10.4's Porter stemmer (ported to Rust: nltk is not needed).

Tokenizing, stemming and scoring run in Rust. What needs the interpreter runs in Python, exactly where rouge-score runs it:

  • a custom tokenizer is called from Python (for the target, the prediction, and each rougeLsum sentence); its tokens are scored in Rust. Tokens that are not lists of str are scored by the real rouge-score, with a one-time pyroid.FallbackWarning (pyroid.PyroidError if rouge-score is not installed);
  • split_summaries=True splits rougeLsum sentences with pyroid.nltk.sent_tokenize, the Rust port of nltk.sent_tokenize (the same sentences; it reads NLTK's punkt_tab model, and a missing model raises LookupError as in rouge-score).

RougeScorer.score_batch (a pyroid extension) scores many pairs at once on pyroid's thread pool.

Example 
from pyroid.metrics import rouge_scorer

scorer = rouge_scorer.RougeScorer(["rouge1", "rougeL"], use_stemmer=True)
scorer.score("The quick brown fox jumps over the lazy dog",
             "The quick brown dog jumps on the log.")
# {'rouge1': Score(precision=0.75, recall=0.666..., fmeasure=0.705...),
#  'rougeL': Score(precision=0.625, recall=0.555..., fmeasure=0.588...)}
scorer.score_batch(targets, predictions, threads=8)  # list of such dicts

Differences from rouge-score

  • Lone surrogates (U+D800..U+DFFF) in a str target or prediction raise UnicodeEncodeError (with a note) when the default tokenizer is used. rouge-score treats them as separators. Remove them first, e.g. s.encode("utf-8", "replace").decode("utf-8"). A custom tokenizer's tokens may hold them (those are scored by rouge-score).
  • RougeScorer is pyroid's class, a pyroid.metrics.scoring.BaseScorer; its scores are pyroid.metrics.scoring.Score namedtuples, equal to rouge-score's but not instances of rouge_score.scoring.Score.
  • rouge-score's private helpers (_score_lcs, _lcs_table, _summary_level_lcs, ...) and lcs_ind are not provided.
  • score_multi picks the first maximal f-measure as np.argmax does, without numpy (its error for no targets is numpy's ValueError("attempt to get argmax of an empty sequence")).
  • "Using default tokenizer." is logged at INFO level with logging (logger pyroid.metrics.rouge_scorer), not with absl.
  • nltk is not needed. With split_summaries=True it is imported, if installed, to find the punkt_tab model; rouge-score always imports nltk, absl, numpy and six.

ROUGE_SCORE_VERSION module-attribute 

ROUGE_SCORE_VERSION: Final[str]

RougeScorer 

RougeScorer(
    rouge_types: Sequence[str],
    use_stemmer: bool = False,
    split_summaries: bool = False,
    tokenizer: _Tokenizer | None = None,
)

Bases: BaseScorer

Calculate rouge scores between two blobs of text (rouge-score's RougeScorer).

Sample usage 
scorer = RougeScorer(["rouge1", "rougeL"], use_stemmer=True)
scores = scorer.score("The quick brown fox jumps over the lazy dog",
                      "The quick brown dog jumps on the log.")

Initializes a new RougeScorer.

Valid rouge types that can be computed are:

  • rouge1 .. rouge9: n-gram based scoring.
  • rougeL: longest common subsequence based scoring.
  • rougeLsum: summary-level LCS over sentences (lines, or Punkt sentences with split_summaries).

Types are checked when scoring, as in rouge-score.

Parameters:

  • rouge_types (Sequence[str]) –

    A list of rouge types to calculate.

  • use_stemmer (bool, default: False ) –

    Bool indicating whether Porter stemmer should be used to strip word suffixes to improve matching. This arg is used in the DefaultTokenizer, but other tokenizers might or might not choose to use this.

  • split_summaries (bool, default: False ) –

    whether to add newlines between sentences for rougeLsum (sentences are split with pyroid.nltk.sent_tokenize).

  • tokenizer (_Tokenizer | None, default: None ) –

    Tokenizer object which has a tokenize() method. A falsy value means the default tokenizer.

rouge_types instance-attribute 

rouge_types: Sequence[str]

The rouge types to calculate, read at every call (as in rouge-score).

score_multi 

score_multi(
    targets: Iterable[str], prediction: str
) -> dict[str, Score]

Calculates rouge scores between targets and prediction.

The target with the maximum f-measure is used for the final score for each score type (the first one on ties).

Parameters:

  • targets (Iterable[str]) –

    list of texts containing the targets

  • prediction (str) –

    Text containing the predicted text.

Returns:

  • dict[str, Score] –

    A dict mapping each rouge type to a Score object.

Raises:

  • ValueError –

    If an invalid rouge type is encountered, or targets is empty.

score 

score(target: str, prediction: str) -> dict[str, Score]

Calculates rouge scores between the target and prediction.

Parameters:

  • target (str) –

    Text containing the target (ground truth) text.

  • prediction (str) –

    Text containing the predicted text.

Returns:

  • dict[str, Score] –

    A dict mapping each rouge type to a Score object, in rouge_types order.

Raises:

  • ValueError –

    If an invalid rouge type is encountered ("Invalid rouge type: ...", or "rougen requires positive n: ..." for rouge0).

  • UnicodeEncodeError –

    target or prediction holds a lone surrogate (default tokenizer).

score_batch 

score_batch(
    targets: Sequence[str],
    predictions: Sequence[str],
    *,
    threads: int | None = None,
) -> list[dict[str, Score]]

score_batch(
    targets: Sequence[str] | ArrowStrings,
    predictions: Sequence[str] | ArrowStrings,
    *,
    threads: int | None = None,
) -> list[dict[str, Score] | None]

Score many pairs at once, on pyroid's thread pool (not part of rouge-score).

Element i equals self.score(targets[i], predictions[i]). With the default tokenizer and a list/tuple of rouge type strs, every pair is tokenized and scored in Rust in parallel (with split_summaries=True, sentences are split in parallel too). Otherwise (a custom tokenizer, list items that are not str) the pairs are scored one by one with score.

Parameters:

  • targets (Sequence[str] | Sequence[str] | ArrowStrings) –

    The targets: a list or tuple of str, or an Arrow string array/column (string, large_string or string_view; pyarrow, polars, pyroid.Column, ...). Not a single str: use score for one pair.

  • predictions (Sequence[str] | Sequence[str] | ArrowStrings) –

    The predictions, as for targets, of the same length.

  • threads (int | None, default: None ) –

    None uses pyroid's shared thread pool; n runs at most n tasks at once; 1 runs on the calling thread.

Returns:

  • list[dict[str, scoring.Score]] | list[dict[str, scoring.Score] | None] –

    A list (for any input kind) with one dict[str, Score] per pair, in order, and None for a pair with an Arrow null on either side.

Raises:

  • ValueError –

    targets and predictions differ in length, or a rouge type is invalid (raised before any pair is scored, even for an empty batch).

  • InputError –

    targets or predictions is not a list, tuple or Arrow string array (a single str is rejected), a list or tuple holds None (the message gives its index), or threads is not an int.

  • UnicodeEncodeError –

    A str holds a lone surrogate.

  • UnicodeDecodeError –

    An Arrow value is not valid UTF-8.

pyroid.metrics.scoring

rouge-score's scoring classes (rouge_score.scoring), as a drop-in.

from pyroid.metrics import scoring replaces from rouge_score import scoring. The classes are ports of rouge-score 0.1.2's, line for line:

  • Score and AggregateScore are namedtuples with the same fields and reprs (Score(precision=1.0, recall=0.5, fmeasure=0.666...)); they compare equal to rouge-score's.
  • BootstrapAggregator resamples with numpy's global random state, exactly as rouge-score does: after the same np.random.seed(...) it returns identical intervals.

Differences from rouge-score

  • numpy is imported when BootstrapAggregator.aggregate is first called (it is not a pyroid dependency). Without it, aggregate raises ImportError with an install hint. rouge-score imports numpy at import.
  • The classes are pyroid's own: isinstance(score, rouge_score.scoring.Score) is false for a pyroid Score (and vice versa), though they compare equal.

Score

Bases: NamedTuple

Tuple containing precision, recall, and f-measure values.

rougeL and rougeLsum return Score(precision=0, recall=0, fmeasure=0) with int zeros when the target or prediction has no tokens, as rouge-score does; every other value is a float.

precision instance-attribute 

precision: float

recall instance-attribute 

recall: float

fmeasure instance-attribute 

fmeasure: float

BaseScorer

Base class for Scorer objects.

score abstractmethod 

score(target: str, prediction: str) -> dict[str, Score]

Calculates score between the target and prediction.

Parameters:

  • target (str) –

    Text containing the target (ground truth) text.

  • prediction (str) –

    Text containing the predicted text.

Returns:

  • dict[str, Score] –

    A dict mapping each score_type (string) to Score object.

AggregateScore

Bases: NamedTuple

Tuple containing confidence intervals for scores.

low instance-attribute 

low: Score

mid instance-attribute 

mid: Score

high instance-attribute 

high: Score

BootstrapAggregator 

BootstrapAggregator(
    confidence_interval: float = 0.95, n_samples: int = 1000
)

Aggregates scores to provide confidence intervals (bootstrap resampling).

Example 
from pyroid.metrics import rouge_scorer, scoring

scorer = rouge_scorer.RougeScorer(["rouge1", "rougeL"])
aggregator = scoring.BootstrapAggregator()
aggregator.add_scores(scorer.score("one two three", "one two"))
aggregator.add_scores(scorer.score("one two five six", "seven eight"))
result = aggregator.aggregate()
# {'rouge1': AggregateScore(low=..., mid=..., high=...), 'rougeL': ...}

Initializes a BootstrapAggregator object.

Parameters:

  • confidence_interval (float, default: 0.95 ) –

    Confidence interval to compute on the mean as a decimal.

  • n_samples (int, default: 1000 ) –

    Number of samples to use for bootstrap resampling.

Raises:

add_scores 

add_scores(scores: Mapping[str, Any]) -> None

Adds a sample for future aggregation.

Parameters:

  • scores (Mapping[str, Any]) –

    Dict mapping score_type strings to a namedtuple object/class representing a score.

aggregate 

aggregate() -> dict[str, AggregateScore]

Aggregates scores previously added using add_scores.

The low and high bounds are the percentiles of n_samples bootstrap means given by confidence_interval; mid is the median of those means. Resampling uses numpy.random (the global state).

Returns:

Raises:

fmeasure 

fmeasure(precision: float, recall: float) -> float

Computes f-measure given precision and recall values.

2 * precision * recall / (precision + recall), or 0.0 when both are 0.

pyroid.metrics.tokenizers

rouge-score's tokenizer classes (rouge_score.tokenizers), Rust-backed.

from pyroid.metrics import tokenizers replaces from rouge_score import tokenizers. DefaultTokenizer tokenizes exactly as rouge-score 0.1.2's: lowercase, keep runs of a-z0-9, and with use_stemmer=True stem tokens longer than 3 characters with nltk 3.10.4's Porter stemmer (ported to Rust; nltk is not needed).

Pass a Tokenizer subclass (or any object with a tokenize method) as RougeScorer(tokenizer=...) to tokenize differently; it is called from Python where rouge-score calls it.

Differences from rouge-score

  • DefaultTokenizer has no _stemmer attribute (rouge-score keeps an nltk PorterStemmer there). Stemming runs in Rust.
  • A str with lone surrogates (U+D800..U+DFFF) raises UnicodeEncodeError (rouge-score drops them like any other non-alphanumeric character).

Tokenizer

Bases: ABC

Abstract base class for a tokenizer.

Subclasses of Tokenizer must implement the tokenize() method.

tokenize abstractmethod 

tokenize(text: str) -> list[str]

Split text into tokens.

DefaultTokenizer 

DefaultTokenizer(use_stemmer: bool = False)

Bases: Tokenizer

Default tokenizer which tokenizes on whitespace (rouge-score's).

Example 
from pyroid.metrics import tokenizers

tokenizers.DefaultTokenizer(use_stemmer=True).tokenize("the friends had a meeting")
# ['the', 'friend', 'had', 'a', 'meet']

Constructor for DefaultTokenizer.

Parameters:

  • use_stemmer (bool, default: False ) –

    boolean, indicating whether Porter stemmer should be used to strip word suffixes to improve matching.

tokenize 

tokenize(text: str) -> list[str]

rouge-score's tokens of text.

text.lower(), then every run of characters other than a-z0-9 separates tokens (Unicode lowercasing included: "İ" gives "i", the Kelvin sign gives "k"). With use_stemmer, tokens longer than 3 are Porter-stemmed. bytes and other objects with a lower() method are handled as rouge-score handles them.

Parameters:

  • text (str) –

    The text to tokenize.

Returns:

  • list[str] –

    The tokens, in order.

Raises:

pyroid.metrics.tokenize

rouge-score's tokenizing function (rouge_score.tokenize), Rust-backed.

from pyroid.metrics import tokenize replaces from rouge_score import tokenize: tokenize and the module's regex constants.

Differences from rouge-score

  • A str with lone surrogates (U+D800..U+DFFF) raises UnicodeEncodeError.
  • A stemmer passed here is called from Python for every token longer than 3, as in rouge-score. pyroid.metrics.tokenizers.DefaultTokenizer with use_stemmer=True stems in Rust instead.

NON_ALPHANUM_PATTERN module-attribute 

NON_ALPHANUM_PATTERN: Final[str]

NON_ALPHANUM_RE module-attribute 

NON_ALPHANUM_RE: Final[Pattern[str]]

SPACES_PATTERN module-attribute 

SPACES_PATTERN: Final[str]

SPACES_RE module-attribute 

SPACES_RE: Final[Pattern[str]]

VALID_TOKEN_PATTERN module-attribute 

VALID_TOKEN_PATTERN: Final[str]

VALID_TOKEN_RE module-attribute 

VALID_TOKEN_RE: Final[Pattern[str]]

tokenize 

tokenize(text: str, stemmer: _Stemmer | None) -> list[str]

Tokenize input text into a list of tokens, as rouge-score does.

This approach aims to replicate the approach taken by Chin-Yew Lin in the original ROUGE implementation.

Parameters:

  • text (str) –

    A text blob to tokenize.

  • stemmer (_Stemmer | None) –

    An optional stemmer (e.g. nltk.stem.porter.PorterStemmer()), applied to tokens longer than 3 characters.

Returns:

  • list[str] –

    A list of string tokens extracted from input text.

pyroid.metrics.sacrebleu

A drop-in, Rust-backed replacement for sacrebleu 2.6.0's BLEU and chrF API.

import pyroid.metrics.sacrebleu as sacrebleu

bleu = sacrebleu.corpus_bleu(hyps, [refs])  # BLEUScore, exactly sacrebleu's
print(bleu)  # the same line sacrebleu prints: "BLEU = ... (BP = ... ratio = ...)"
chrf = sacrebleu.corpus_chrf(hyps, [refs], word_order=2)  # chrF2++
sacrebleu.sentence_bleu_batch(hyps, [refs], threads=8)  # one BLEUScore per pair

Every score, statistic, str()/repr(), format() output, signature string, exception and sacrebleu log warning is sacrebleu 2.6.0's. __version__, like the version: field of signatures, is the sacrebleu API version reproduced. The integer n-gram statistics are computed in Rust, in parallel on pyroid's thread pool. The score formulas (BLEU.compute_bleu, CHRF._compute_f_score), result classes and argument checks are sacrebleu's own Python code, so scores are bit-for-bit sacrebleu's on the same interpreter. To pick the best of several references, chrF uses a step-for-step Rust port of the f-score, with the same operations in the same order.

pyroid handles these natively:

  • the tokenizers 13a (the default), intl, zh (also via trg_lang="zh"), char and none;
  • every smoothing method and value, lowercase (Python's str.lower()), force, effective_order and max_ngram_order;
  • several references per segment, including None holes (nrefs:var);
  • chrF and chrF++ with any char_order, word_order, beta, whitespace and eps_smoothing;
  • pre-computed references (BLEU(references=...), CHRF(references=...)).

Degenerate settings (e.g. max_ngram_order=0, a non-int order, char_order + word_order == 0) run sacrebleu's own code in pure Python, with sacrebleu's results and errors.

These are delegated to the real sacrebleu, with one pyroid.FallbackWarning per function and reason: the ja-mecab, ko-mecab, spm, flores101, flores200 and spBLEU-1K tokenizers (and trg_lang "ja"/"ko"), corpus_score(..., n_bootstrap > 1), TER, corpus_ter, sentence_ter, DATASETS and the test set helpers (DELEGATED). They need sacrebleu installed; without it they raise pyroid.PyroidError with an install hint. Importing this module never imports sacrebleu.

pyroid extensions (keyword-only): threads= on corpus_bleu, raw_corpus_bleu, corpus_chrf and corpus_score (None: the whole pool; n: at most n tasks; 1: the calling thread), and sentence_bleu_batch / sentence_chrf_batch.

Differences from sacrebleu:

  • Lone surrogates ("\ud800") in a hypothesis or reference raise UnicodeEncodeError (with a note naming the item), as everywhere in pyroid. sacrebleu treats them as code points. Remove them first, e.g. s.encode("utf-8", "replace").decode("utf-8").
  • Wrong item types that sacrebleu's own argument checks let through (a non-str hypothesis after the first, a non-str, non-None reference) raise pyroid.InputError (a TypeError) naming the item. sacrebleu raises an AttributeError from deep inside. sacrebleu's checks themselves (_check_*_args) raise the same TypeErrors with the same messages.
  • BLEU.tokenizer is a pyroid tokenizer object for the native tokenizers (signature() and __call__(line), the same output as sacrebleu's), and sacrebleu's own tokenizer object for the delegated ones.
  • Pre-computed references are held in a native object. Changing max_ngram_order, char_order or word_order of a metric after constructing it with references= raises ValueError (sacrebleu would mix the old reference n-grams with new hypothesis ones).
  • sentence_bleu_batch logs sacrebleu's effective_order recommendation once per call, not once per sentence.

__version__ module-attribute 

__version__: Final[str]

The sacrebleu release whose API and results this module reproduces: "2.6.0".

DELEGATED module-attribute 

DELEGATED: Final[tuple[str, ...]]

Names that are forwarded to the real sacrebleu (it must be installed).

SACREBLEU_DIR module-attribute 

SACREBLEU_DIR: str

Where sacrebleu stores test sets: $SACREBLEU, else ~/.sacrebleu (as sacrebleu).

DATASETS module-attribute 

DATASETS: Mapping[str, Any]

sacrebleu's test set registry (read-only; delegated to sacrebleu on first use).

Score 

Score(name: str, score: float)

The result of a metric (sacrebleu's Score).

str(score) and repr(score) are score.format().

Parameters:

  • name (str) –

    The name of the underlying metric.

  • score (float) –

    A floating point number for the final metric.

name instance-attribute 

name: str

score instance-attribute 

score: float

format 

format(
    width: int = 2,
    score_only: bool = False,
    signature: str = "",
    is_json: bool = False,
) -> str

A pretty representation of the score, as sacrebleu prints it.

Parameters:

  • width (int, default: 2 ) –

    Floating point decimal precision width.

  • score_only (bool, default: False ) –

    If True, and the format is not json, returns a single score string.

  • signature (str, default: '' ) –

    A string representation of the metric's signature (str(metric.get_signature())).

  • is_json (bool, default: False ) –

    If True, will output the score in JSON string.

Returns:

  • str –

    A plain or JSON-formatted string, e.g. "BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0 = ...".

estimate_ci 

estimate_ci(scores: list[Score]) -> None

Store the mean and 95% confidence interval of bootstrap resampled scores.

Parameters:

  • scores (list[Score]) –

    Scores of bootstrap resamples of the corpus.

Signature 

Signature(args: dict[str, Any])

A sacreBLEU reproducibility signature; str() is format().

Parameters:

  • args (dict[str, Any]) –

    The metric's __dict__ (must hold num_refs: evaluate the metric first).

Raises:

  • ValueError –

    The metric has not been evaluated yet.

info instance-attribute 

info: dict[str, Any]

format 

format(short: bool = False) -> str

The signature as a string, e.g. nrefs:1|case:mixed|...|version:2.6.0.

Parameters:

  • short (bool, default: False ) –

    If True, the abbreviated keys (#:1|c:mixed|...|v:2.6.0).

Returns:

  • str –

    The signature string.

update 

update(key: str, value: Any) -> None

Add a new item or update an existing one.

Parameters:

  • key (str) –

    The key to use in the dictionary.

  • value (Any) –

    The associated value for the key.

BLEUSignature 

BLEUSignature(args: dict[str, Any])

Bases: Signature

The signature of a BLEU metric (case, eff, tok, smooth fields).

CHRFSignature 

CHRFSignature(args: dict[str, Any])

Bases: Signature

The signature of a chrF metric (case, eff, nc, nw, space fields).

BLEUScore 

BLEUScore(
    score: float,
    counts: list[int],
    totals: list[int],
    precisions: list[float],
    bp: float,
    sys_len: int,
    ref_len: int,
)

Bases: Score

A BLEU score with its statistics.

Parameters:

  • score (float) –

    The BLEU score.

  • counts (list[int]) –

    Counts of correct n-grams, 1 <= n <= max_ngram_order (with add-k smoothing, including the added k, as in sacrebleu).

  • totals (list[int]) –

    Counts of total n-grams, 1 <= n <= max_ngram_order.

  • precisions (list[float]) –

    n-gram precisions (percentages).

  • bp (float) –

    The brevity penalty.

  • sys_len (int) –

    The cumulative system length.

  • ref_len (int) –

    The cumulative reference length.

bp instance-attribute 

bp: float

counts instance-attribute 

counts: list[int]

totals instance-attribute 

totals: list[int]

sys_len instance-attribute 

sys_len: int

ref_len instance-attribute 

ref_len: int

precisions instance-attribute 

precisions: list[float]

prec_str instance-attribute 

prec_str: str

ratio instance-attribute 

ratio: float

CHRFScore 

CHRFScore(
    score: float,
    char_order: int,
    word_order: int,
    beta: int,
)

Bases: Score

A chrF score (named chrF2, chrF2++, ... after beta and word_order).

Parameters:

  • score (float) –

    The chrF (chrF++) score.

  • char_order (int) –

    The character n-gram order.

  • word_order (int) –

    The word n-gram order. If equals to 2, the metric is referred to as chrF++.

  • beta (int) –

    Determine the importance of recall w.r.t precision.

beta instance-attribute 

beta: int

char_order instance-attribute 

char_order: int

word_order instance-attribute 

word_order: int

Metric 

Metric()

The base class of BLEU and CHRF (sacrebleu's Metric).

n_bootstrap instance-attribute 

n_bootstrap: int | None

seed instance-attribute 

seed: str | None

num_refs instance-attribute 

num_refs: int

References per segment of the last evaluation (or the cache); -1 if they vary.

sentence_score 

sentence_score(
    hypothesis: str, references: Sequence[str | None]
) -> Score

Score one hypothesis against its reference(s).

Parameters:

  • hypothesis (str) –

    A single hypothesis string.

  • references (Sequence[str | None]) –

    Its reference strings (None entries are skipped).

Returns:

Raises:

  • TypeError –

    sacrebleu's argument checks (e.g. hypothesis is not a str).

corpus_score 

corpus_score(
    hypotheses: Sequence[str],
    references: Sequence[Sequence[str | None]] | None,
    n_bootstrap: int = 1,
    *,
    threads: int | None = None,
) -> Score

Score a corpus against one or more reference streams.

Hypotheses and reference segments are zipped, as in sacrebleu: extra items of the longer one are ignored.

Parameters:

  • hypotheses (Sequence[str]) –

    A sequence of hypothesis strings.

  • references (Sequence[Sequence[str | None]] | None) –

    Reference streams: references[k][i] is the k-th reference of hypothesis i (None for a missing one). None (or empty) uses the references the metric was constructed with.

  • n_bootstrap (int, default: 1 ) –

    If > 1, also estimate a 95% confidence interval by bootstrap resampling with that many samples (delegated to sacrebleu).

  • threads (int | None, default: None ) –

    pyroid extension: None uses the whole thread pool, n at most n tasks at once, 1 the calling thread. Results are the same.

Returns:

  • Score –

    The corpus score.

Raises:

get_signature 

get_signature() -> Signature

The reproducibility signature (available once the metric has been evaluated).

Raises:

  • ValueError –

    The metric has not been evaluated yet.

BLEU 

BLEU(
    lowercase: bool = False,
    force: bool = False,
    tokenize: str | None = None,
    smooth_method: str = "exp",
    smooth_value: float | None = None,
    max_ngram_order: int = 4,
    effective_order: bool = False,
    trg_lang: str = "",
    references: Sequence[Sequence[str | None]]
    | None = None,
)

Bases: Metric

The BLEU metric (sacrebleu.BLEU).

Parameters:

  • lowercase (bool, default: False ) –

    If True, lowercased BLEU is computed.

  • force (bool, default: False ) –

    Ignore data that looks already tokenized (otherwise 100 hypotheses ending in " ." log a warning on the sacrebleu logger).

  • tokenize (str | None, default: None ) –

    The tokenizer (one of TOKENIZERS). If None, "13a", or the language's tokenizer for trg_lang "zh", "ja", "ko".

  • smooth_method (str, default: 'exp' ) –

    "floor", "add-k", "exp" or "none".

  • smooth_value (float | None, default: None ) –

    The value for floor and add-k (None: 0.1 and 1).

  • max_ngram_order (int, default: 4 ) –

    The maximum n-gram order (default 4).

  • effective_order (bool, default: False ) –

    Stop including n-gram orders for which precision is 0 (use it for sentence-level BLEU).

  • trg_lang (str, default: '' ) –

    An optional language code to pick (or warn about) the tokenizer.

  • references (Sequence[Sequence[str | None]] | None, default: None ) –

    Reference streams to pre-compute for scoring many systems.

Raises:

SMOOTH_DEFAULTS class-attribute 

SMOOTH_DEFAULTS: dict[str, float | None]

TOKENIZERS class-attribute 

TOKENIZERS: KeysView[str]

TOKENIZER_DEFAULT class-attribute 

TOKENIZER_DEFAULT: str

trg_lang instance-attribute 

trg_lang: str

lowercase instance-attribute 

lowercase: bool

smooth_value instance-attribute 

smooth_value: float | None

smooth_method instance-attribute 

smooth_method: str

max_ngram_order instance-attribute 

max_ngram_order: int

effective_order instance-attribute 

effective_order: bool

tokenizer instance-attribute 

tokenizer: _Tokenizer

tokenizer_signature instance-attribute 

tokenizer_signature: str

compute_bleu staticmethod 

compute_bleu(
    correct: list[int],
    total: list[int],
    sys_len: int,
    ref_len: int,
    smooth_method: str = "none",
    smooth_value: float | None = None,
    effective_order: bool = False,
    max_ngram_order: int = 4,
) -> BLEUScore

BLEU from its sufficient statistics, with smoothing (sacrebleu's own code).

Parameters:

  • correct (list[int]) –

    Counts of correct n-grams, 1 <= n <= max_ngram_order (modified in place by add-k).

  • total (list[int]) –

    Counts of total n-grams, 1 <= n <= max_ngram_order.

  • sys_len (int) –

    The cumulative system length.

  • ref_len (int) –

    The cumulative reference length.

  • smooth_method (str, default: 'none' ) –

    "floor", "add-k", "exp" or "none".

  • smooth_value (float | None, default: None ) –

    The value for floor and add-k (None: the default).

  • effective_order (bool, default: False ) –

    Stop at the first order without n-grams.

  • max_ngram_order (int, default: 4 ) –

    The maximum n-gram order.

Returns:

sentence_score 

sentence_score(
    hypothesis: str, references: Sequence[str | None]
) -> BLEUScore

Sentence-level BLEU (logs a recommendation unless effective_order).

Parameters:

  • hypothesis (str) –

    A single hypothesis string.

  • references (Sequence[str | None]) –

    Its reference strings (None entries are skipped).

Returns:

corpus_score 

corpus_score(
    hypotheses: Sequence[str],
    references: Sequence[Sequence[str | None]] | None,
    n_bootstrap: int = 1,
    *,
    threads: int | None = None,
) -> BLEUScore

Corpus-level BLEU; see Metric.corpus_score.

Parameters:

  • hypotheses (Sequence[str]) –

    A sequence of hypothesis strings.

  • references (Sequence[Sequence[str | None]] | None) –

    Reference streams (references[k][i]), or None for the cached ones.

  • n_bootstrap (int, default: 1 ) –

    Bootstrap resamples for a confidence interval (delegated if > 1).

  • threads (int | None, default: None ) –

    pyroid's thread pool setting.

Returns:

get_signature 

get_signature() -> BLEUSignature

The signature, e.g. nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0.

CHRF 

CHRF(
    char_order: int = 6,
    word_order: int = 0,
    beta: int = 2,
    lowercase: bool = False,
    whitespace: bool = False,
    eps_smoothing: bool = False,
    references: Sequence[Sequence[str | None]]
    | None = None,
)

Bases: Metric

The chrF and chrF++ metrics (sacrebleu.CHRF).

Parameters:

  • char_order (int, default: 6 ) –

    Character n-gram order.

  • word_order (int, default: 0 ) –

    Word n-gram order. If equals to 2, the metric is referred to as chrF++.

  • beta (int, default: 2 ) –

    Determine the importance of recall w.r.t precision.

  • lowercase (bool, default: False ) –

    Enable case-insensitivity.

  • whitespace (bool, default: False ) –

    If True, include whitespaces when extracting character n-grams.

  • eps_smoothing (bool, default: False ) –

    If True, applies epsilon smoothing similar to reference chrF++.py, NLTK and Moses implementations. Otherwise, it takes into account effective match order similar to sacreBLEU < 2.0.0.

  • references (Sequence[Sequence[str | None]] | None, default: None ) –

    Reference streams to pre-compute for scoring many systems.

CHAR_ORDER class-attribute 

CHAR_ORDER: int

WORD_ORDER class-attribute 

WORD_ORDER: int

BETA class-attribute 

BETA: int

beta instance-attribute 

beta: int

char_order instance-attribute 

char_order: int

word_order instance-attribute 

word_order: int

order instance-attribute 

order: int

lowercase instance-attribute 

lowercase: bool

whitespace instance-attribute 

whitespace: bool

eps_smoothing instance-attribute 

eps_smoothing: bool

sentence_score 

sentence_score(
    hypothesis: str, references: Sequence[str | None]
) -> CHRFScore

Sentence-level chrF against the best of the references.

Parameters:

  • hypothesis (str) –

    A single hypothesis string.

  • references (Sequence[str | None]) –

    Its reference strings (None entries are skipped).

Returns:

corpus_score 

corpus_score(
    hypotheses: Sequence[str],
    references: Sequence[Sequence[str | None]] | None,
    n_bootstrap: int = 1,
    *,
    threads: int | None = None,
) -> CHRFScore

Corpus-level chrF; see Metric.corpus_score.

Parameters:

  • hypotheses (Sequence[str]) –

    A sequence of hypothesis strings.

  • references (Sequence[Sequence[str | None]] | None) –

    Reference streams (references[k][i]), or None for the cached ones.

  • n_bootstrap (int, default: 1 ) –

    Bootstrap resamples for a confidence interval (delegated if > 1).

  • threads (int | None, default: None ) –

    pyroid's thread pool setting.

Returns:

get_signature 

get_signature() -> CHRFSignature

The signature, e.g. nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.6.0.

TER 

TER(
    normalized: bool = False,
    no_punct: bool = False,
    asian_support: bool = False,
    case_sensitive: bool = False,
    references: Sequence[Sequence[str]] | None = None,
)

sacrebleu's TER metric (delegated: every call runs a real sacrebleu.TER).

Parameters:

  • normalized (bool, default: False ) –

    Enable character normalization.

  • no_punct (bool, default: False ) –

    Remove punctuation.

  • asian_support (bool, default: False ) –

    Enable special treatment of Asian characters.

  • case_sensitive (bool, default: False ) –

    Enables case-sensitivity.

  • references (Sequence[Sequence[str]] | None, default: None ) –

    Reference streams to pre-compute.

Raises:

corpus_score 

corpus_score(
    hypotheses: Sequence[str],
    references: Sequence[Sequence[str]] | None,
    n_bootstrap: int = 1,
) -> Any

sacrebleu.TER.corpus_score (returns sacrebleu's TERScore).

Parameters:

  • hypotheses (Sequence[str]) –

    A sequence of hypothesis strings.

  • references (Sequence[Sequence[str]] | None) –

    Reference streams, or None for the cached ones.

  • n_bootstrap (int, default: 1 ) –

    Bootstrap resamples for a confidence interval.

sentence_score 

sentence_score(
    hypothesis: str, references: Sequence[str]
) -> Any

sacrebleu.TER.sentence_score (returns sacrebleu's TERScore).

Parameters:

  • hypothesis (str) –

    A single hypothesis string.

  • references (Sequence[str]) –

    Its reference strings.

get_signature 

get_signature() -> Any

sacrebleu.TER.get_signature.

corpus_bleu 

corpus_bleu(
    hypotheses: Sequence[str],
    references: Sequence[Sequence[str | None]],
    smooth_method: str = "exp",
    smooth_value: float | None = None,
    force: bool = False,
    lowercase: bool = False,
    tokenize: str = "13a",
    use_effective_order: bool = False,
    *,
    threads: int | None = None,
) -> BLEUScore

BLEU of a corpus against one or more reference streams, like sacrebleu.corpus_bleu.

>>> corpus_bleu(["The cat sat on the mat."], [["The cat sat on a mat."]]).score
48.892302243490086

Parameters:

  • hypotheses (Sequence[str]) –

    A sequence of hypothesis strings.

  • references (Sequence[Sequence[str | None]]) –

    Reference streams: references[k][i] is the k-th reference of hypothesis i (None for a missing one).

  • smooth_method (str, default: 'exp' ) –

    "floor", "add-k", "exp" or "none".

  • smooth_value (float | None, default: None ) –

    The value for floor and add-k (None: the default).

  • force (bool, default: False ) –

    Ignore data that looks already tokenized.

  • lowercase (bool, default: False ) –

    Lowercase the data.

  • tokenize (str, default: '13a' ) –

    The tokenizer to use.

  • use_effective_order (bool, default: False ) –

    Don't take into account n-gram orders without any match.

  • threads (int | None, default: None ) –

    pyroid extension: None uses the whole thread pool, n at most n tasks at once, 1 the calling thread.

Returns:

raw_corpus_bleu 

raw_corpus_bleu(
    hypotheses: Sequence[str],
    references: Sequence[Sequence[str | None]],
    smooth_value: float | None = 0.1,
    *,
    threads: int | None = None,
) -> BLEUScore

BLEU of the raw corpus: no tokenization, floor smoothing, effective order.

Parameters:

  • hypotheses (Sequence[str]) –

    A sequence of hypothesis strings.

  • references (Sequence[Sequence[str | None]]) –

    Reference streams (references[k][i]).

  • smooth_value (float | None, default: 0.1 ) –

    The floor smoothing value.

  • threads (int | None, default: None ) –

    pyroid's thread pool setting (see corpus_bleu).

Returns:

sentence_bleu 

sentence_bleu(
    hypothesis: str,
    references: Sequence[str | None],
    smooth_method: str = "exp",
    smooth_value: float | None = None,
    lowercase: bool = False,
    tokenize: str = "13a",
    use_effective_order: bool = True,
) -> BLEUScore

BLEU of one hypothesis against its reference(s), like sacrebleu.sentence_bleu.

Parameters:

  • hypothesis (str) –

    A single hypothesis string.

  • references (Sequence[str | None]) –

    Its reference strings (None entries are skipped).

  • smooth_method (str, default: 'exp' ) –

    "floor", "add-k", "exp" or "none".

  • smooth_value (float | None, default: None ) –

    The value for floor and add-k (None: the default).

  • lowercase (bool, default: False ) –

    Lowercase the data.

  • tokenize (str, default: '13a' ) –

    The tokenizer to use.

  • use_effective_order (bool, default: True ) –

    Don't take into account n-gram orders without any match.

Returns:

sentence_bleu_batch 

sentence_bleu_batch(
    hypotheses: Sequence[str],
    references: Sequence[
        Sequence[str | None] | ArrowStrings
    ],
    smooth_method: str = "exp",
    smooth_value: float | None = None,
    lowercase: bool = False,
    tokenize: str = "13a",
    use_effective_order: bool = True,
    *,
    threads: int | None = None,
) -> list[BLEUScore]

sentence_bleu_batch(
    hypotheses: ArrowStrings,
    references: Sequence[
        Sequence[str | None] | ArrowStrings
    ],
    smooth_method: str = "exp",
    smooth_value: float | None = None,
    lowercase: bool = False,
    tokenize: str = "13a",
    use_effective_order: bool = True,
    *,
    threads: int | None = None,
) -> list[BLEUScore | None]

sentence_bleu of every hypothesis at once, on pyroid's thread pool.

Element i equals sentence_bleu(hypotheses[i], [r[i] for r in references], ...).

Parameters:

  • hypotheses (Sequence[str] | ArrowStrings) –

    The hypothesis strings: a list or tuple of str, or an Arrow string array/column (pyarrow, polars, pyroid.Column, ...). Not a single str: use sentence_bleu for one hypothesis.

  • references (Sequence[Sequence[str | None] | ArrowStrings]) –

    Reference streams, as for corpus_bleu: references[k][i] is the k-th reference of hypothesis i (None for a missing one). Every stream must have one entry per hypothesis. A stream may be an Arrow string column, whose nulls are missing references.

  • smooth_method (str, default: 'exp' ) –

    "floor", "add-k", "exp" or "none".

  • smooth_value (float | None, default: None ) –

    The value for floor and add-k (None: the default).

  • lowercase (bool, default: False ) –

    Lowercase the data.

  • tokenize (str, default: '13a' ) –

    The tokenizer to use.

  • use_effective_order (bool, default: True ) –

    Don't take into account n-gram orders without any match.

  • threads (int | None, default: None ) –

    None uses the whole thread pool, n at most n tasks at once, 1 the calling thread.

Returns:

  • list[BLEUScore] | list[BLEUScore | None] –

    A list (for any input kind) with one score per hypothesis, and None for a null in an Arrow hypotheses column.

Raises:

  • InputError –

    hypotheses is not a sequence of str or an Arrow string column, or a list or tuple of hypotheses holds None (the message gives its index), or references is not a sequence of streams (e.g. a flat list of str: pass [refs]).

  • ValueError –

    No streams, or a stream's length differs from the hypotheses'.

corpus_chrf 

corpus_chrf(
    hypotheses: Sequence[str],
    references: Sequence[Sequence[str | None]],
    char_order: int = 6,
    word_order: int = 0,
    beta: int = 2,
    remove_whitespace: bool = True,
    eps_smoothing: bool = False,
    *,
    threads: int | None = None,
) -> CHRFScore

Corpus-level chrF (word_order=2: chrF++), like sacrebleu.corpus_chrf.

Parameters:

  • hypotheses (Sequence[str]) –

    A sequence of hypothesis strings.

  • references (Sequence[Sequence[str | None]]) –

    Reference streams (references[k][i]; None for a missing one).

  • char_order (int, default: 6 ) –

    Character n-gram order.

  • word_order (int, default: 0 ) –

    Word n-gram order. If equals to 2, the metric is referred to as chrF++.

  • beta (int, default: 2 ) –

    Determine the importance of recall w.r.t precision.

  • remove_whitespace (bool, default: True ) –

    If True, removes whitespaces prior to character n-gram extraction.

  • eps_smoothing (bool, default: False ) –

    If True, applies epsilon smoothing similar to reference chrF++.py, NLTK and Moses implementations. Otherwise, it takes into account effective match order similar to sacreBLEU < 2.0.0.

  • threads (int | None, default: None ) –

    pyroid's thread pool setting (see corpus_bleu).

Returns:

sentence_chrf 

sentence_chrf(
    hypothesis: str,
    references: Sequence[str | None],
    char_order: int = 6,
    word_order: int = 0,
    beta: int = 2,
    remove_whitespace: bool = True,
    eps_smoothing: bool = False,
) -> CHRFScore

Sentence-level chrF against the best reference, like sacrebleu.sentence_chrf.

Parameters:

  • hypothesis (str) –

    A single hypothesis string.

  • references (Sequence[str | None]) –

    Its reference strings (None entries are skipped).

  • char_order (int, default: 6 ) –

    Character n-gram order.

  • word_order (int, default: 0 ) –

    Word n-gram order. If equals to 2, the metric is referred to as chrF++.

  • beta (int, default: 2 ) –

    Determine the importance of recall w.r.t precision.

  • remove_whitespace (bool, default: True ) –

    If True, removes whitespaces prior to character n-gram extraction.

  • eps_smoothing (bool, default: False ) –

    Epsilon smoothing instead of effective-order smoothing.

Returns:

sentence_chrf_batch 

sentence_chrf_batch(
    hypotheses: Sequence[str],
    references: Sequence[
        Sequence[str | None] | ArrowStrings
    ],
    char_order: int = 6,
    word_order: int = 0,
    beta: int = 2,
    remove_whitespace: bool = True,
    eps_smoothing: bool = False,
    *,
    threads: int | None = None,
) -> list[CHRFScore]

sentence_chrf_batch(
    hypotheses: ArrowStrings,
    references: Sequence[
        Sequence[str | None] | ArrowStrings
    ],
    char_order: int = 6,
    word_order: int = 0,
    beta: int = 2,
    remove_whitespace: bool = True,
    eps_smoothing: bool = False,
    *,
    threads: int | None = None,
) -> list[CHRFScore | None]

sentence_chrf of every hypothesis at once, on pyroid's thread pool.

Element i equals sentence_chrf(hypotheses[i], [r[i] for r in references], ...).

Parameters:

  • hypotheses (Sequence[str] | ArrowStrings) –

    The hypothesis strings, as for sentence_bleu_batch.

  • references (Sequence[Sequence[str | None] | ArrowStrings]) –

    Reference streams (see sentence_bleu_batch).

  • char_order (int, default: 6 ) –

    Character n-gram order.

  • word_order (int, default: 0 ) –

    Word n-gram order. If equals to 2, the metric is referred to as chrF++.

  • beta (int, default: 2 ) –

    Determine the importance of recall w.r.t precision.

  • remove_whitespace (bool, default: True ) –

    If True, removes whitespaces prior to character n-gram extraction.

  • eps_smoothing (bool, default: False ) –

    Epsilon smoothing instead of effective-order smoothing.

  • threads (int | None, default: None ) –

    None uses the whole thread pool, n at most n tasks at once, 1 the calling thread.

Returns:

  • list[CHRFScore] | list[CHRFScore | None] –

    A list (for any input kind) with one score per hypothesis, and None for a null in an Arrow hypotheses column.

Raises:

extract_word_ngrams 

extract_word_ngrams(
    tokens: list[str], n: int
) -> Counter[str]

Counts of the word n-grams of order n of tokens (joined with spaces).

Parameters:

  • tokens (list[str]) –

    A list of tokens.

  • n (int) –

    The order of n-grams.

Returns:

extract_char_ngrams 

extract_char_ngrams(
    line: str, n: int, include_whitespace: bool = False
) -> Counter[str]

Counts of the character n-grams of order n of line.

Parameters:

  • line (str) –

    A segment containing a sequence of words.

  • n (int) –

    The order of the n-grams.

  • include_whitespace (bool, default: False ) –

    If given, will not strip whitespaces from the line.

Returns:

smart_open 

smart_open(
    file: str, mode: str = "rt", encoding: str = "utf-8"
) -> IO[Any]

Open a plain or gzip-compressed (.gz) text file, like sacrebleu.smart_open.

Parameters:

  • file (str) –

    The file to open.

  • mode (str, default: 'rt' ) –

    The file mode.

  • encoding (str, default: 'utf-8' ) –

    The text encoding.

Returns:

  • IO[Any] –

    The open file.

corpus_ter 

corpus_ter(
    hypotheses: Sequence[str],
    references: Sequence[Sequence[str]],
    normalized: bool = False,
    no_punct: bool = False,
    asian_support: bool = False,
    case_sensitive: bool = False,
) -> Any

sacrebleu.corpus_ter (delegated; returns sacrebleu's TERScore).

Parameters:

  • hypotheses (Sequence[str]) –

    A sequence of hypothesis strings.

  • references (Sequence[Sequence[str]]) –

    Reference streams.

  • normalized (bool, default: False ) –

    Enable character normalization.

  • no_punct (bool, default: False ) –

    Remove punctuation.

  • asian_support (bool, default: False ) –

    Enable special treatment of Asian characters.

  • case_sensitive (bool, default: False ) –

    Enables case-sensitivity.

sentence_ter 

sentence_ter(
    hypothesis: str,
    references: Sequence[str],
    normalized: bool = False,
    no_punct: bool = False,
    asian_support: bool = False,
    case_sensitive: bool = False,
) -> Any

sacrebleu.sentence_ter (delegated; returns sacrebleu's TERScore).

Parameters:

  • hypothesis (str) –

    A single hypothesis string.

  • references (Sequence[str]) –

    Its reference strings.

  • normalized (bool, default: False ) –

    Enable character normalization.

  • no_punct (bool, default: False ) –

    Remove punctuation.

  • asian_support (bool, default: False ) –

    Enable special treatment of Asian characters.

  • case_sensitive (bool, default: False ) –

    Enables case-sensitivity.

download_test_set 

download_test_set(
    test_set: str, langpair: str | None = None
) -> Any

sacrebleu.download_test_set (delegated).

Parameters:

  • test_set (str) –

    The test set to download.

  • langpair (str | None, default: None ) –

    The language pair (all of them if None).

get_source_file 

get_source_file(test_set: str, langpair: str) -> str

sacrebleu.get_source_file (delegated).

Parameters:

  • test_set (str) –

    The test set.

  • langpair (str) –

    The language pair.

get_reference_files 

get_reference_files(
    test_set: str, langpair: str
) -> list[str]

sacrebleu.get_reference_files (delegated).

Parameters:

  • test_set (str) –

    The test set.

  • langpair (str) –

    The language pair.

get_available_testsets 

get_available_testsets() -> list[str]

sacrebleu.get_available_testsets (delegated).

get_langpairs_for_testset 

get_langpairs_for_testset(testset: str) -> list[str]

sacrebleu.get_langpairs_for_testset (delegated).

Parameters:

  • testset (str) –

    The test set.