Scoring & Metrics¶

This guide covers how the evaluation harness scores model outputs against gold-standard references, including the scorer pipeline, metrics, reductions, and aggregations.

Overview¶

After the model produces responses and any filters are applied, the scoring pipeline turns those responses into final metric values:

flowchart LR
    A["Filtered Responses"] --> B["Score"]
    B --> C["Reduce"]
    C --> D["Aggregate"]

    B -. "per-sample<br>metric values" .-> C
    C -. "per-doc<br>reduced values" .-> D
    D -. "single<br>score" .-> E["Results"]

1. Score — Compare each response to the gold reference (e.g., does the output match the correct answer?)
2. Reduce — If repeats > 1, collapse multiple scores per document into one (e.g., take the first, majority vote, mean)
3. Aggregate — Combine per-document scores into a single number (e.g., mean accuracy across all documents)

Default behavior¶

For most tasks, you don't need to configure scoring explicitly:

• Multiple-choice tasks (output_type: multiple_choice): Automatically scored with acc (accuracy) and acc_norm (length-normalized accuracy)
• Generation tasks (output_type: generate_until): No default metrics — you must specify metric_list
• Loglikelihood tasks (output_type: loglikelihood_rolling): Automatically scored with perplexity, word_perplexity, byte_perplexity, and bits_per_byte

Configuring metrics in YAML¶

Basic metric_list¶

List the metrics you want to compute:

metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
  - metric: bleu

Each entry is a MetricConfig with these fields:

Passing arguments to metrics¶

Extra fields in a metric config entry are treated as kwargs:

metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true           # Passed as kwarg to exact_match()
    ignore_punctuation: false
    regexes_to_ignore:
      - ","
      - "\\$"

Built-in metrics¶

Metrics¶

All metrics supported by HuggingFace Evaluate can also be used — if a metric name isn't recognized as a built-in, the harness will attempt to load it from HF Evaluate.

Aggregation functions¶

Reduction functions (for repeats > 1)¶

Scorers¶

Scorers encapsulate the full scoring pipeline — which filters to apply, which metrics to compute, and how to aggregate. For most tasks, metric_list and filter_list are sufficient and scorers are configured implicitly. Use explicit scorer config when you need custom scoring logic like LLM-as-judge, code execution, or reusable scoring patterns across tasks. See Custom Scorers for implementation details.

Built-in scorers¶

Configuring scorers in YAML¶

Scorers are configured via the scorer field or are auto-configured by formats:

# Explicit scorer configuration
scorer:
  type: regex_extraction
  kwargs:
    regex_pattern: "The answer is (\\w+)"

In practice, most tasks use the implicit scorer that is set up by the filter_list and metric_list fields, or by the formats field if using prompt formats.

Precedence rules¶

When the harness resolves scoring configuration, the precedence is:

1. Explicit config — filter_list / metric_list in the YAML or passed as overrides
2. Scorer class defaults — default_filter_cfg / default_metric_cfg ClassVars on the scorer
3. Fallback — noop filter / output-type-appropriate default metrics

Repeats and reduction¶

When repeats > 1, the model runs on each document multiple times (useful for self-consistency or sampling-based evaluation):

repeats: 64

Each repeat produces a separate response, and the reduction function collapses them:

metric_list:
  - metric: exact_match
    reduction: mean          # Average across 64 repeats

The default reduction is take_first — only the first repeat is scored. For self-consistency patterns, combine repeats with filter pipelines that do majority voting (see Filters).

Per-pipeline metrics¶

When using multiple filter pipelines, each pipeline can have its own metrics:

filter_list:
  - name: "strict-match"
    filter:
      - function: "regex"
        regex_pattern: "The answer is (\\d+)"
      - function: "take_first"
    metric_list:
      - metric: exact_match

  - name: "flexible-match"
    filter:
      - function: "take_first"
    metric_list:
      - metric: exact_match
        ignore_case: true

If a pipeline doesn't specify metric_list, it inherits the task-level metric_list.

Custom metrics¶

For metrics not built into the harness, you can use !function to reference a Python function:

metric_list:
  - metric: !function utils.my_custom_metric
    aggregation: mean
    higher_is_better: true

For more details on implementing custom metrics, including the Metric[_T, _K] generic class and registration, see Custom Metrics & Filters.