Skip to main content
An evaluator is a named rule that scores a model output. Every evaluator has three independent choices: Names are 1-64 lowercase letters, digits, ., _ or -, unique per workspace, and usable in place of the evl_... id everywhere.

Output types and pass rules

The output config says what a score may be and, optionally, when it passes. Experiment verdicts and gate thresholds count passes. An evaluator without a pass rule is informational in an experiment’s verdict (its mean or label counts are reported); the gate’s non-regression rule still compares a numeric evaluator’s mean with production’s.

Variables and bindings

Evaluators read five built-in variables from the item being scored: Bindings map a variable to a JSONPath inside the item, so an evaluator can read a nested field or define a new variable for its prompt:
The path grammar is $ followed by .name, ['name'], or [index]. Wildcards, slices, and filters are rejected: a binding selects exactly one value. An experiment can layer its own bindings over every evaluator’s. Set requires_reference: true when the evaluator compares against expected: items without an expected value are then recorded as skipped instead of being scored.

LLM judge

The prompt is a template: {{input}}, {{output}}, and any other variable name (built-in or bound) are replaced with the item’s values. Strings are inserted as-is and other JSON values as compact JSON. A placeholder that names no variable is an error at scoring time; a known variable with no value renders empty and is reported. How the judge runs:
  • The judge is a deployment in your workspace (--judge-deployment; --judge-model picks a model on it, or an adapter on a multi-adapter fleet (beta)). An experiment pins the judge’s served model when it starts.
  • Veri adds a fixed system instruction (grade strictly on the evidence shown), asks for a JSON verdict {"reasoning", "score"} with structured outputs at temperature 0, and validates the reply against your output type.
  • A reply that does not parse or is out of range becomes an error score, never a pass.

Decision mode

Set "mode": "decision" in method_config and the judge answers with exactly one label instead of a reasoned verdict: yes or no for a boolean, one of your labels for a categorical, or one integer for a numeric scale. Veri constrains the reply to those labels and stores the chosen label’s probability (read from the first token’s log probabilities and renormalized over the labels) as the score’s confidence, from 0 to 1. A one-label answer is faster, cheaper, and more repeatable than a reasoned verdict; its score has no reasoning. There is no CLI flag for the mode: set it in a spec file or the API request.
on-topic.json
A numeric scale in decision mode needs integer min and max at most 10 apart: 1 to 5 works, while 0 to 100 or 0.5 to 5 is refused with 400. Because the confidence is read off the first token, decision mode also refuses (with 400) a label longer than 32 characters, a label that is the start of another (1 and 10, so 1 to 10 is refused too: use 0 to 9 or 1 to 5), and two labels that start with the same word (not helpful and not harmful). Versions stored before these rules keep running. Use the default mode, reasoned, for wider or fractional scales. confidence is empty when the judge’s reply carries no usable log probabilities.

Jev judge

A jev evaluator asks TypeSafe’s Jev one typed question about the item. Jev returns a typed answer with its probabilities instead of generated text, so there is no judge deployment to run and no verdict to parse. The question type fixes the output type:
  • The options become the labels and the levels the range (0 to levels - 1), so output_config carries only the pass rule: pass_when (noul), pass_labels (choice), or pass_threshold and higher_is_better (score). To fix the order Jev reads a choice’s options in, also send output_config.labels listing every option in that order.
  • Jev reads a state built from the item: by default input and output (single-turn) or transcript (thread), message lists reduced to role, text, and tool calls. Set method_config.state to name your own fields: {"ticket": "$.metadata.ticket", "answer": "$.output"}. Put the whole question in instructions; refer to state fields by name.
  • Long conversations are cut to fit TypeSafe’s 32k-token budget, keeping the first turn and the most recent turns.
  • Each score keeps the answer’s probabilities (noul {"true": p, "false": 1 - p}, choice per option, score per level index) and its confidence. Jev writes no reasoning; use an llm_judge evaluator when you need an explanation.
  • Jev reads text only and works best in English. Rate limits and overloads are retried; a question TypeSafe rejects becomes an error score.
What leaves Veri: the state Jev reads (the trace or conversation text the evaluator selects) and your question are sent to TypeSafe, a sub-processor. A workspace can turn third-party judges off with PATCH /v1/workspace/settings ({"third_party_judges": false}) or in the dashboard’s workspace settings: new jev evaluators are then refused and queued Jev work is recorded as skipped, with nothing sent.

Code evaluators

A code evaluator is one function that receives {input, output, expected, metadata, transcript} (after bindings) and returns {"score": ..., "reasoning": ...}. The score is a boolean, a finite number, or a label string; reasoning is optional.
The language comes from the file extension (.py, .js/.mjs, .ts/.mts) unless you pass --language. Where the code runs is the runtime: Both runtimes enforce the same contract: a 5 second wall clock, source up to 256 KB, arguments up to 5 MB, a result up to 256 KB, labels up to 256 characters, and reasoning truncated to 8 KB. Use the standard library only: a local evaluator that imports a package you have installed would score differently (or fail) anywhere it is missing. A local evaluator’s score must carry the SHA-256 of the evaluator version’s source; the SDK computes it, and a score from different code is refused (409 source_mismatch). Python code evaluators are local-only: Python in the cloud sandbox would not match local results.

Human evaluators

Reviewers score human evaluators in annotation queues. Experiments and the promotion gate take llm_judge, jev, and code evaluators.

Spec files

Keep evaluators in version control as JSON (or YAML, with PyYAML installed) and create them with --file. method_config.source_file is read relative to the spec file:
evaluator.json

Versions

Every edit is a new immutable version; nothing that pinned an earlier version changes meaning. Send only what changes, and method flags are merged into the latest version’s config:
Pin a version as helpful@2 in experiments, gates, and queues; a bare name means the latest. A version that changes scope (for example single_turn to thread) is refused with 409 while any monitor tracks the evaluator’s latest version, because it would change what the monitor scores. The error names those monitors: pin them to a version (evaluator_version) or delete them first.

Scores

A score is one evaluator result on one target. Scores are immutable: a second score for the same evaluator version and target is refused with 409 score_exists. Your own harness can post verdicts on production traffic, and they show up on the trace in the dashboard and in veri datasets import-traces --score-evaluator:
Each score records its source (api, sdk, human, experiment, or monitor), its author, passed (the version’s pass rule applied to the value), and the model_version_id that answered the scored request, when known. status is ok, error (with error), or skipped. A monitor’s scores also carry its monitor_id. Judge scores also carry confidence (0 to 1) and, from Jev, probabilities. Filter scores by their typed value as well as by passed:
The same filters narrow GET /v1/observability/conversations (evaluator, label, min_score, max_score, min_confidence, max_confidence). The dashboard’s Traces page shows scores but can’t filter by them yet. A monitor with review_below_confidence and review_queue_id sends every target whose verdict falls below that confidence to an annotation queue for a person to check.

Threads

A thread is the requests of one conversation. Send the same X-Veri-Thread-Id header (1-128 characters of A-Z a-z 0-9 . _ : -) on every request of the conversation; the SDK’s session_id argument sends the equivalent X-Veri-Session-Id, which is accepted as an alias. The thread id is stored on each request and trace, and thread evaluators score the whole conversation up to its latest request.

Where to go next

Experiments

Run evaluators over a dataset snapshot against a target.

Human review

Score human evaluators and calibrate judges.