> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veri.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# veri evals

> Define and run evaluations from the terminal — datasets, benchmarks, and live run progress

The `veri evals` command group defines evals (a dataset + scorers, or a benchmark task), runs them against serving deployments, and inspects per-sample results. `veri run <config>.toml` with `kind = "eval"` dispatches here too.

## veri evals benchmarks

List the curated benchmark catalog.

```bash theme={null}
veri evals benchmarks
```

```
 task       name       tier            num_samples  description
 gsm8k      GSM8K      inference_only  1319         Grade-school math word problems...
 humaneval  HumanEval  code_exec       164          Python function synthesis...
```

## veri evals create

Three ways to create (and implicitly start) an eval:

### Benchmark — one line, no dataset

```bash theme={null}
veri evals create --benchmark gsm8k --model deployment:dep_abc123 --n 100 --follow

# Use the pinned lm-eval adapter instead of the native evaluator
veri evals create --benchmark gsm8k --model deployment:dep_abc123 \
  --runner lm_eval --runner-task gsm8k --n 100 --follow
```

| Flag                   | Meaning                                                                   |
| ---------------------- | ------------------------------------------------------------------------- |
| `--benchmark <task>`   | Catalog task (`gsm8k`, `humaneval`). Mutually exclusive with `--dataset`. |
| `--model <target>`     | Required. A serving deployment: `deployment:<id>`.                        |
| `--n <int>`            | Sample limit (default 100).                                               |
| `--runner <type>`      | `native` (default), `lm_eval`, or `inspect`.                              |
| `--runner-task <task>` | Allowlisted harness task for an external runner.                          |
| `--follow`, `-f`       | Poll the run, stream progress, print scores at the end.                   |

### Quick — inline dataset eval

```bash theme={null}
veri evals create --quick --dataset ds_abc123 --model deployment:dep_abc123 --scorer exact_match --n 200
```

### Config — reusable TOML

```bash theme={null}
veri evals create configs/eval.toml --follow
# or equivalently:
veri run configs/eval.toml --follow
```

```toml theme={null}
kind = "eval"

[eval]
name = "gsm8k-baseline"
# dataset_id = "ds_abc123"   # only for dataset evals; benchmarks omit it

[[scorers]]
type = "benchmark"
name = "gsm8k"
task = "gsm8k"
sample_limit = 100

[run]                         # optional — starts a run on create
model = "deployment:dep_abc123"
```

Override any config value from the CLI with `--set`, e.g. `--set run.model=deployment:dep_xyz`.

## veri evals list / get / delete

```bash theme={null}
veri evals list
veri evals get eval_9f2c01
veri evals delete eval_9f2c01
```

## veri evals runs

```bash theme={null}
veri evals runs list eval_9f2c01                 # all runs for an eval
veri evals runs get eval_9f2c01 run_5b7d22       # status, progress, results
veri evals runs items eval_9f2c01 run_5b7d22     # per-sample outputs + scores
veri evals runs cancel eval_9f2c01 run_5b7d22    # stop an active run
veri evals runs rescore eval_9f2c01 run_5b7d22 --scorer exact_match --name revised
veri evals runs calibrate eval_9f2c01 run_5b7d22 --scorer-name quality --threshold 0.7
veri evals runs compare eval_9f2c01 run_5b7d22 --baseline run_baseline --junit eval.xml
```

All commands support `--format json` for machine output and `--quiet` for pipe-friendly IDs. `runs compare --junit <path>` writes JUnit XML. Gate-aware commands use exit code `0` for pass, `1` for gate failure, `2` for invalid gate configuration, and `3` for run/infrastructure failure.

## Following a run

`--follow` on `create` (or `veri run`) polls until the run is terminal:

```
  [   1.2s] running/generating  0/100
  [   9.8s] running/scoring  100/100
  [  14.1s] completed/completed  100/100
  gsm8k: mean=0.8100  n=100  errors=0
```

Exits non-zero if the run fails, so it composes with CI:

```bash theme={null}
veri evals create --benchmark gsm8k --model "deployment:$DEP" --n 100 --follow --quiet
```