> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veri.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# Scorers

> Configuration reference for deterministic, benchmark, and LLM judge eval scorers

A scorer turns a model's output for a single sample into a numeric score. An eval applies its scorers to every sample, then aggregates per-scorer stats (`mean`, `median`, `std`, `min`, `max`, `count`, `error_count`).

Scorers are declared as a JSON array on the eval. Every scorer has a `type` and a `name` (the key its scores appear under in results).

## exact\_match

Case-insensitive (by default) string comparison of the model output against a reference field on the dataset row.

```json theme={null}
{
  "type": "exact_match",
  "name": "answer_accuracy",
  "reference_field": "label",
  "case_sensitive": false
}
```

| Field             | Default   | Meaning                                       |
| ----------------- | --------- | --------------------------------------------- |
| `reference_field` | `"label"` | Dataset row field holding the expected output |
| `case_sensitive`  | `false`   | Exact-case comparison when `true`             |

Score: `1.0` on match, `0.0` otherwise. Both sides are whitespace-trimmed.

## contains

Substring or regex match against a reference field.

```json theme={null}
{
  "type": "contains",
  "name": "mentions_refund",
  "reference_field": "label",
  "regex": false
}
```

| Field             | Default   | Meaning                                           |
| ----------------- | --------- | ------------------------------------------------- |
| `reference_field` | `"label"` | Dataset row field holding the pattern             |
| `regex`           | `false`   | Treat the reference value as a regular expression |

Score: `1.0` when the output contains the reference (or matches the regex), `0.0` otherwise. An invalid regex scores as an error, not a failure.

## llm\_judge

Grades persisted candidate outputs with a serving deployment. Judge responses must be JSON with `score` or `pass`; explanations and per-dimension scores are retained with each item.

```json theme={null}
{
  "type": "llm_judge",
  "name": "quality",
  "judge_model": "deployment:dep_judge123",
  "judge_prompt": "Evaluate the candidate response.",
  "rubric": "The response must be correct, relevant, and concise.",
  "dimensions": ["correctness", "relevance"],
  "min_score": 1,
  "max_score": 5,
  "pass_threshold": 0.75
}
```

Judge scores are normalized to `0..1`. Judge token usage is reported separately from candidate usage. Use `veri evals runs rescore` to apply a new judge without repeating candidate inference.

## benchmark

Runs a curated task from the [benchmark catalog](/evaluations/benchmarks). The task supplies its own rows, prompting, and scoring — so a benchmark eval has **no dataset**, and the benchmark scorer cannot be combined with other scorers.

```json theme={null}
{
  "type": "benchmark",
  "name": "gsm8k",
  "task": "gsm8k",
  "sample_limit": 100
}
```

| Field          | Default  | Meaning                                |
| -------------- | -------- | -------------------------------------- |
| `task`         | —        | Catalog task id (`gsm8k`, `humaneval`) |
| `sample_limit` | all rows | Cap on how many task rows run          |

## Errors vs. failures

A score of `0.0` means the model got the sample wrong. A `null` score means the sample **errored** — inference failed, the row was malformed, or (for code benchmarks) sandbox infrastructure failed. Errors are reported in `error_count` and excluded from the aggregate stats, so an infra blip doesn't masquerade as a model regression.

## Planned

`reward_function` (reuse an uploaded reward function) and `python` (custom scoring code) remain declared but are not executable yet. Samples scored with them land in `error_count` rather than producing scores.