TaskConfig¶

Unique task identifier used for registration and CLI selection (e.g. --tasks arc_easy). Append @<formats> to select a prompt formats at runtime (e.g. "arc_easy@cloze").

Optional display name shown in result tables instead of task.

Prompt formats to apply. Can be a registered formats name (e.g. "cloze", "mcq"), or an inline dict of [FormatConfig][lm_eval.config.task.TaskConfig.formats.formats.FormatConfig] fields. When set, the format's template overrides doc_to_text, doc_to_target, output_type, etc. If None and task contains @, the suffix is used as the formats name.

The type of model request to construct for each document.

• "generate_until": open-ended text generation (default).
• "loglikelihood": score the likelihood of a target string.
• "loglikelihood_rolling": score a full sequence without a context split.
• "multiple_choice": rank answer choices by loglikelihood.

Tags for categorizing this task. Users can select all tasks sharing a tag via --tasks <tag_name> (e.g. tag: [math, reasoning] lets users run --tasks math to include this task). Distinct from explicit group configs (see GroupConfig).

A callable that returns a HuggingFace DatasetDict. Should accept arbitrary kwargs. Typically set in YAML via !function utils.xxx Use this when you need custom loading logic instead of datasets.load_dataset. At runtime, receives metadata (from this config) and model_args (if using evaluate) as keyword arguments.

HuggingFace dataset path passed to datasets.load_dataset(). Can be a Hub identifier (e.g. "allenai/ai2_arc") or a local path.

HuggingFace dataset config/subset name (e.g. "ARC-Easy").

Extra keyword arguments forwarded to datasets.load_dataset() (e.g. {"data_dir": "path/to/data"} or {"data_files": "data.json"}).

Name of the training split in the dataset (e.g. "train").

Name of the validation split. Used as the evaluation split when test_split is not set.

Name of the test split. When set, this is the primary split evaluated.

Name of the split from which few-shot examples are drawn. Passed as the default for fewshot_config.split; overridden if fewshot_config explicitly sets its own split.

A callable applied to a dataset split before evaluation. Use this to filter, transform, or resample documents (e.g. renaming columns, expanding multi-answer rows). Typically set in YAML via !function utils.xxx

A Jinja2 template or plain string prepended to every prompt. Useful for task-level instructions, e.g. ``"The following are questions (with answers) about {{subject}}.

". When a chat template is applied, this is combined withsystem_instructionand sent as thesystem`` message.

Converts a document dict into the prompt text shown to the model. Can be a Jinja2 template string (e.g. "{{question}}"), a column name, or a callable. For loglikelihood tasks this is the context preceding the target.

Defines the set of answer choices for multiple-choice tasks. Can be a Jinja2 template (e.g. "{{choices.text}}"), a column name, a static list of strings, or a callable returning a list of strings.

The gold-standard target for each document. Can be the column name, a Jinja2 template, or a callable. For multiple-choice tasks this is typically the integer index into doc_to_choice (e.g. "{{answer}}"). For generation tasks, it is the expected answer string.

A string or Jinja2 template appended after the prompt (and choices, if any) but before the model generates or the target is scored. With a chat template, this is appended after the <|assistant|> token; without one it is appended to the end of the prompt. Useful for answer cues like "The answer is".

Extracts an image from the document for multimodal models. Can be a column name or a callable returning image data.

Extracts audio from the document for multimodal models. Can be a column name or a callable returning audio data.

Custom post-processing of model outputs for metric computation. Receives (doc, results) and returns a dict mapping metric names to lists of values. Typically set in YAML via !function utils.xxx.

String inserted between the input (prompt/choices) and the target output for each example (both few-shot and the test document).

String inserted between consecutive few-shot examples. Also used as the default until stop sequence for generation.

Advanced few-shot configuration. Accepts a dict or FewshotConfig to override how few-shot examples are sampled and formatted (e.g. separate doc_to_text for examples, custom sampler, fixed indices). When None or a dict, it is converted to FewshotConfig in __post_init__.

Number of few-shot examples to prepend to each prompt. When None, the value is determined at runtime (typically by CLI --num_fewshot).

Keyword arguments for text generation (e.g. temperature, until, max_gen_toks, do_sample). Only relevant when output_type is "generate_until". If empty, greedy defaults are applied.

List of metrics to compute on model outputs. Each entry specifies a metric name, optional aggregation function, and whether higher is better (e.g. [{"metric": "exact_match", "higher_is_better": true}]).

List of named filter pipelines applied to model outputs before scoring. Each pipeline is a sequence of filter steps (e.g. regex extraction, stripping) and can carry its own metric_list. Pipelines run independently on the same model outputs, allowing multiple scoring strategies from a single evaluation run (e.g. "strict-match" and "maj@64" on GSM8k).

A registered scorer name or inline scorer config. When set, scoring is delegated to this scorer instead of the default metric pipeline.

Accepts a string (e.g. "first_token") which is normalised to {"type": "first_token"} in __post_init__, or a full ScorerConfig dict with extra kwargs forwarded to the scorer constructor.

Number of times to repeat each instance. Only used for generation tasks. Useful for sampling diversity (e.g. pass@k, self-consistency).

Set to True to enable execution of untrusted code (e.g. for code-execution benchmarks). Must be explicitly opted in.

Name of a registered prompt template to apply (e.g. "promptsource:GPT-3 Style"). When set, overrides doc_to_text, doc_to_target, and doc_to_choice.

Only for multiple_choice tasks. When True, doc_to_text returns a list of strings (one per choice) and doc_to_choice returns a 1 elememnt list. Each choice produces a different context scored via loglikelihood (e.g. Winogrande, where each option fills a blank).

When True, doc_to_target may return a list of acceptable answers. Scoring considers any match a success.

Whether to run decontamination checks against training data.

Jinja2 template or callable that extracts the decontamination query string from a document. Used when should_decontaminate is True. Falls back to doc_to_text if left as None.

Metadata dict stored alongside results. Most tasks should include a version key. The num_fewshot key overrides the displayed n-shot column in result tables. Also passed to custom_dataset at runtime to pass arbitrary kwargs

Sampling strategy for selecting few-shot examples (e.g. "default", "first_n"). "default" samples randomly.

Dataset split to draw few-shot examples from. Inherited from TaskConfig.fewshot_split if not set directly. Takes precedence over samples when both are provided.

Optional callable to transform the few-shot split before sampling. Inherited from TaskConfig.process_docs if not set.

Explicit list of document indices to use as few-shot examples. When set, overrides random sampling with a fixed selection.

Hardcoded few-shot examples as a list of dicts, or a callable returning such a list. Used when examples don't come from a dataset split. Ignored if split is also set.

Override doc_to_text for formatting few-shot examples differently from the test example. Inherited from TaskConfig.doc_to_text.

Override doc_to_choice for few-shot examples. Inherited from TaskConfig.doc_to_choice.

Override doc_to_target for few-shot examples. Inherited from TaskConfig.doc_to_target.

Override gen_prefix for few-shot examples. Inherited from TaskConfig.gen_prefix.

Override the delimiter between few-shot examples. Inherited from TaskConfig.fewshot_delimiter.

Override the delimiter between prompt and target in few-shot examples. Inherited from TaskConfig.target_delimiter.

Name of a registered metric (e.g. "acc", "exact_match", "bleu") or a callable. See lm_eval/api/metrics.py for built-in metrics.

How per-document metric values are combined into a single score (e.g. "mean", "median"). Can be a registered aggregation or a callable. Defaults to the metric's registered aggregation if not set.

How per-instance repeated values are reduced before aggregation (e.g. when repeats > 1). Can be a registered reduction or a callable. Defaults to the metric's registered reduction if not set.

Whether a higher metric value indicates better performance. Defaults to the metric's registered value if not set.

Extra keyword arguments forwarded to the metric function (e.g. {"ignore_case": true} for exact_match) Extreneous fields are also. treated as kwargs.

Identifier for this pipeline, used as a prefix in result keys (e.g. "strict-match", "maj@64").

Ordered sequence of filter steps applied to model outputs. Steps run in order; each step's output feeds into the next.

Optional per-pipeline metrics. When set, these metrics are scored only on this pipeline's filtered outputs, instead of the task-level metric_list.

Name of a registered filter (e.g. "regex", "custom", "majority_vote"). Custom filters can be registered with @register_filter.

Keyword arguments passed to the filter function (e.g. {"regex_pattern": "The answer is (\d+)"} for "regex", or {"filter_fn": "!function utils.my_filter"} for "custom").

Name of a registered scorer (e.g. "first_token", "regex", "choice_match"). Resolved via lm_eval.api.registry.get_scorer.

Extra keyword arguments forwarded to the scorer constructor. Scorer subclasses declare these as dataclass fields (e.g. judge_model on an AIJudgeScorer).