veri evals

The veri evals command group defines evals (a dataset + scorers, or a benchmark task), runs them against serving deployments, and inspects per-sample results. veri run <config>.toml with kind = "eval" dispatches here too.

veri evals benchmarks

List the curated benchmark catalog.

veri evals benchmarks

 task       name       tier            num_samples  description
 gsm8k      GSM8K      inference_only  1319         Grade-school math word problems...
 humaneval  HumanEval  code_exec       164          Python function synthesis...

veri evals create

Three ways to create (and implicitly start) an eval:

Benchmark — one line, no dataset

veri evals create --benchmark gsm8k --model deployment:dep_abc123 --n 100 --follow

# Use the pinned lm-eval adapter instead of the native evaluator
veri evals create --benchmark gsm8k --model deployment:dep_abc123 \
  --runner lm_eval --runner-task gsm8k --n 100 --follow

Flag	Meaning
`--benchmark <task>`	Catalog task (`gsm8k`, `humaneval`). Mutually exclusive with `--dataset`.
`--model <target>`	Required. A serving deployment: `deployment:<id>`.
`--n <int>`	Sample limit (default 100).
`--runner <type>`	`native` (default), `lm_eval`, or `inspect`.
`--runner-task <task>`	Allowlisted harness task for an external runner.
`--follow`, `-f`	Poll the run, stream progress, print scores at the end.

Quick — inline dataset eval

veri evals create --quick --dataset ds_abc123 --model deployment:dep_abc123 --scorer exact_match --n 200

Config — reusable TOML

veri evals create configs/eval.toml --follow
# or equivalently:
veri run configs/eval.toml --follow

kind = "eval"

[eval]
name = "gsm8k-baseline"
# dataset_id = "ds_abc123"   # only for dataset evals; benchmarks omit it

[[scorers]]
type = "benchmark"
name = "gsm8k"
task = "gsm8k"
sample_limit = 100

[run]                         # optional — starts a run on create
model = "deployment:dep_abc123"

Override any config value from the CLI with --set, e.g. --set run.model=deployment:dep_xyz.

veri evals list / get / delete

veri evals list
veri evals get eval_9f2c01
veri evals delete eval_9f2c01

veri evals runs

veri evals runs list eval_9f2c01                 # all runs for an eval
veri evals runs get eval_9f2c01 run_5b7d22       # status, progress, results
veri evals runs items eval_9f2c01 run_5b7d22     # per-sample outputs + scores
veri evals runs cancel eval_9f2c01 run_5b7d22    # stop an active run
veri evals runs rescore eval_9f2c01 run_5b7d22 --scorer exact_match --name revised
veri evals runs calibrate eval_9f2c01 run_5b7d22 --scorer-name quality --threshold 0.7
veri evals runs compare eval_9f2c01 run_5b7d22 --baseline run_baseline --junit eval.xml

All commands support --format json for machine output and --quiet for pipe-friendly IDs. runs compare --junit <path> writes JUnit XML. Gate-aware commands use exit code 0 for pass, 1 for gate failure, 2 for invalid gate configuration, and 3 for run/infrastructure failure.

Following a run

--follow on create (or veri run) polls until the run is terminal:

  [   1.2s] running/generating  0/100
  [   9.8s] running/scoring  100/100
  [  14.1s] completed/completed  100/100
  gsm8k: mean=0.8100  n=100  errors=0

Exits non-zero if the run fails, so it composes with CI:

veri evals create --benchmark gsm8k --model "deployment:$DEP" --n 100 --follow --quiet

Get started

Training

Deployments

Evaluations

CLI

Volumes

veri evals benchmarks

veri evals create

Benchmark — one line, no dataset

Quick — inline dataset eval

Config — reusable TOML

veri evals list / get / delete

veri evals runs

Following a run

​veri evals benchmarks

​veri evals create

​Benchmark — one line, no dataset

​Quick — inline dataset eval

​Config — reusable TOML

​veri evals list / get / delete

​veri evals runs

​Following a run

veri evals benchmarks

veri evals create

Benchmark — one line, no dataset

Quick — inline dataset eval

Config — reusable TOML

veri evals list / get / delete

veri evals runs

Following a run