mean, median, std, min, max, count, error_count).
Scorers are declared as a JSON array on the eval. Every scorer has a type and a name (the key its scores appear under in results).
exact_match
Case-insensitive (by default) string comparison of the model output against a reference field on the dataset row.
Score:
1.0 on match, 0.0 otherwise. Both sides are whitespace-trimmed.
contains
Substring or regex match against a reference field.
Score:
1.0 when the output contains the reference (or matches the regex), 0.0 otherwise. An invalid regex scores as an error, not a failure.
llm_judge
Grades persisted candidate outputs with a serving deployment. Judge responses must be JSON withscore or pass; explanations and per-dimension scores are retained with each item.
0..1. Judge token usage is reported separately from candidate usage. Use veri evals runs rescore to apply a new judge without repeating candidate inference.
benchmark
Runs a curated task from the benchmark catalog. The task supplies its own rows, prompting, and scoring — so a benchmark eval has no dataset, and the benchmark scorer cannot be combined with other scorers.Errors vs. failures
A score of0.0 means the model got the sample wrong. A null score means the sample errored — inference failed, the row was malformed, or (for code benchmarks) sandbox infrastructure failed. Errors are reported in error_count and excluded from the aggregate stats, so an infra blip doesn’t masquerade as a model regression.
Planned
reward_function (reuse an uploaded reward function) and python (custom scoring code) remain declared but are not executable yet. Samples scored with them land in error_count rather than producing scores.