CLI Reference¶

The lm-eval CLI is organized into subcommands:

Run the library via the lm-eval entrypoint or python -m lm_eval.

Use -h or --help to see available options:

lm-eval -h              # Show all subcommands
lm-eval run -h          # Show options for run command
lm-eval ls -h           # Show options for list command

Legacy Compatibility: The original single-command interface still works. Running lm-eval --model hf --tasks hellaswag automatically inserts the run subcommand.

lm-eval run¶

Run evaluations on language models.

lm-eval run --model <model> --tasks <task> [options]

Examples¶

# Basic evaluation with HuggingFace model
lm-eval run --model hf --model_args pretrained=gpt2 dtype=float32 --tasks hellaswag

# Multiple tasks with few-shot examples
lm-eval run --model vllm --model_args pretrained=EleutherAI/gpt-j-6B --tasks arc_easy arc_challenge --num_fewshot 5

# Custom generation parameters
lm-eval run --model hf --model_args pretrained=gpt2 --tasks lambada --gen_kwargs temperature=0.8 top_p=0.95

# Use a YAML configuration file
lm-eval run --config my_config.yaml --tasks mmlu

Task selection¶

Tasks can be specified by name, group, tag, or path. You can also apply formats and navigate nested groups using special syntax:

# Apply a format at runtime with @
lm-eval run --tasks my_task@mcqa --model hf --model_args pretrained=gpt2

# Try different prompt formats on the same dataset
lm-eval run --tasks hellaswag@generate --model hf --model_args pretrained=gpt2
lm-eval run --tasks hellaswag@cloze --model hf --model_args pretrained=gpt2

# Address a specific subtask within a group with ::
lm-eval run --tasks mmlu::mmlu_anatomy --model hf --model_args pretrained=gpt2

The @format suffix tells lm-eval to apply a prompt format (e.g., mcqa, cloze, generate, cot) at runtime without modifying the task YAML.

The :: path syntax navigates nested groups: group::subgroup::task.

Model and Tasks¶

Evaluation Settings¶

Data and Output¶

Caching and Performance¶

Prompt Formatting¶

Task Management¶

Logging and Tracking¶

Advanced Options¶

Configuration File¶

HuggingFace Hub Logging¶

The --hf_hub_log_args argument accepts these keys:

lm-eval ls¶

List available tasks, groups, subtasks, or tags.

lm-eval ls [tasks|groups|subtasks|tags] [--include_path DIR]

Arguments¶

Task Organization¶

• Groups: Collections of related tasks with aggregated metrics across subtasks (e.g., mmlu contains 57 subtasks)
• Subtasks: Individual evaluation tasks (e.g., mmlu_anatomy, hellaswag)
• Tags: Categories for filtering tasks without aggregated metrics (e.g., reasoning, language)

Examples¶

# List all tasks
lm-eval ls tasks

# List only task groups
lm-eval ls groups

# Filter tasks by pattern
lm-eval ls tasks --pattern "arc*"

# Include external tasks
lm-eval ls tasks --include_path /path/to/external/tasks

lm-eval validate¶

Validate task configurations before running evaluations.

lm-eval validate --tasks <task1,task2> [--include_path DIR]

Arguments¶

Validation Checks¶

The validate command performs:

• Task existence: Verifies all specified tasks are available
• Configuration syntax: Checks YAML/JSON configuration files
• Dataset access: Validates dataset paths and configurations
• Required fields: Ensures all mandatory task parameters are present
• Metric definitions: Verifies metric functions and aggregation methods
• Filter pipelines: Validates filter chains and their parameters
• Template rendering: Tests prompt templates with sample data

Examples¶

# Validate a single task
lm-eval validate --tasks hellaswag

# Validate multiple tasks
lm-eval validate --tasks arc_easy,arc_challenge,hellaswag

# Validate a task group
lm-eval validate --tasks mmlu

# Validate external tasks
lm-eval validate --tasks my_custom_task --include_path ./custom_tasks

Environment Variables¶