Names are 1-64 lowercase letters, digits,
., _ or -, unique per workspace, and usable in place of the evl_... id everywhere.
Output types and pass rules
The output config says what a score may be and, optionally, when it passes. Experiment verdicts and gate thresholds count passes. An evaluator without a pass rule is informational in an experiment’s verdict (its mean or label counts are reported); the gate’s non-regression rule still compares a numeric evaluator’s mean with production’s.Variables and bindings
Evaluators read five built-in variables from the item being scored:
Bindings map a variable to a JSONPath inside the item, so an evaluator can read a nested field or define a new variable for its prompt:
$ followed by .name, ['name'], or [index]. Wildcards, slices, and filters are rejected: a binding selects exactly one value. An experiment can layer its own bindings over every evaluator’s.
Set requires_reference: true when the evaluator compares against expected: items without an expected value are then recorded as skipped instead of being scored.
LLM judge
{{input}}, {{output}}, and any other variable name (built-in or bound) are replaced with the item’s values. Strings are inserted as-is and other JSON values as compact JSON. A placeholder that names no variable is an error at scoring time; a known variable with no value renders empty and is reported.
How the judge runs:
- The judge is a deployment in your workspace (
--judge-deployment;--judge-modelpicks a model on it, or an adapter on a multi-adapter fleet (beta)). An experiment pins the judge’s served model when it starts. - Veri adds a fixed system instruction (grade strictly on the evidence shown), asks for a JSON verdict
{"reasoning", "score"}with structured outputs at temperature 0, and validates the reply against your output type. - A reply that does not parse or is out of range becomes an error score, never a pass.
Decision mode
Set"mode": "decision" in method_config and the judge answers with exactly one label instead of a reasoned verdict: yes or no for a boolean, one of your labels for a categorical, or one integer for a numeric scale. Veri constrains the reply to those labels and stores the chosen label’s probability (read from the first token’s log probabilities and renormalized over the labels) as the score’s confidence, from 0 to 1. A one-label answer is faster, cheaper, and more repeatable than a reasoned verdict; its score has no reasoning. There is no CLI flag for the mode: set it in a spec file or the API request.
on-topic.json
min and max at most 10 apart: 1 to 5 works, while 0 to 100 or 0.5 to 5 is refused with 400. Because the confidence is read off the first token, decision mode also refuses (with 400) a label longer than 32 characters, a label that is the start of another (1 and 10, so 1 to 10 is refused too: use 0 to 9 or 1 to 5), and two labels that start with the same word (not helpful and not harmful). Versions stored before these rules keep running. Use the default mode, reasoned, for wider or fractional scales. confidence is empty when the judge’s reply carries no usable log probabilities.
Jev judge
Ajev evaluator asks TypeSafe’s Jev one typed question about the item. Jev returns a typed answer with its probabilities instead of generated text, so there is no judge deployment to run and no verdict to parse. The question type fixes the output type:
- The options become the labels and the levels the range (0 to levels - 1), so
output_configcarries only the pass rule:pass_when(noul),pass_labels(choice), orpass_thresholdandhigher_is_better(score). To fix the order Jev reads a choice’s options in, also sendoutput_config.labelslisting every option in that order. - Jev reads a state built from the item: by default
inputandoutput(single-turn) ortranscript(thread), message lists reduced to role, text, and tool calls. Setmethod_config.stateto name your own fields:{"ticket": "$.metadata.ticket", "answer": "$.output"}. Put the whole question ininstructions; refer to state fields by name. - Long conversations are cut to fit TypeSafe’s 32k-token budget, keeping the first turn and the most recent turns.
- Each score keeps the answer’s
probabilities(noul{"true": p, "false": 1 - p}, choice per option, score per level index) and itsconfidence. Jev writes no reasoning; use anllm_judgeevaluator when you need an explanation. - Jev reads text only and works best in English. Rate limits and overloads are retried; a question TypeSafe rejects becomes an error score.
PATCH /v1/workspace/settings ({"third_party_judges": false}) or in the dashboard’s workspace settings: new jev evaluators are then refused and queued Jev work is recorded as skipped, with nothing sent.
Code evaluators
A code evaluator is one function that receives{input, output, expected, metadata, transcript} (after bindings) and returns {"score": ..., "reasoning": ...}. The score is a boolean, a finite number, or a label string; reasoning is optional.
.py, .js/.mjs, .ts/.mts) unless you pass --language. Where the code runs is the runtime:
Both runtimes enforce the same contract: a 5 second wall clock, source up to 256 KB, arguments up to 5 MB, a result up to 256 KB, labels up to 256 characters, and reasoning truncated to 8 KB. Use the standard library only: a local evaluator that imports a package you have installed would score differently (or fail) anywhere it is missing.
A local evaluator’s score must carry the SHA-256 of the evaluator version’s source; the SDK computes it, and a score from different code is refused (
409 source_mismatch). Python code evaluators are local-only: Python in the cloud sandbox would not match local results.
Human evaluators
llm_judge, jev, and code evaluators.
Spec files
Keep evaluators in version control as JSON (or YAML, with PyYAML installed) and create them with--file. method_config.source_file is read relative to the spec file:
evaluator.json
Versions
Every edit is a new immutable version; nothing that pinned an earlier version changes meaning. Send only what changes, and method flags are merged into the latest version’s config:helpful@2 in experiments, gates, and queues; a bare name means the latest.
A version that changes scope (for example single_turn to thread) is refused with 409 while any monitor tracks the evaluator’s latest version, because it would change what the monitor scores. The error names those monitors: pin them to a version (evaluator_version) or delete them first.
Scores
A score is one evaluator result on one target. Scores are immutable: a second score for the same evaluator version and target is refused with409 score_exists.
Your own harness can post verdicts on production traffic, and they show up on the trace in the dashboard and in
veri datasets import-traces --score-evaluator:
source (api, sdk, human, experiment, or monitor), its author, passed (the version’s pass rule applied to the value), and the model_version_id that answered the scored request, when known. status is ok, error (with error), or skipped. A monitor’s scores also carry its monitor_id. Judge scores also carry confidence (0 to 1) and, from Jev, probabilities.
Filter scores by their typed value as well as by passed:
GET /v1/observability/conversations (evaluator, label, min_score, max_score, min_confidence, max_confidence). The dashboard’s Traces page shows scores but can’t filter by them yet. A monitor with review_below_confidence and review_queue_id sends every target whose verdict falls below that confidence to an annotation queue for a person to check.
Threads
A thread is the requests of one conversation. Send the sameX-Veri-Thread-Id header (1-128 characters of A-Z a-z 0-9 . _ : -) on every request of the conversation; the SDK’s session_id argument sends the equivalent X-Veri-Session-Id, which is accepted as an alias. The thread id is stored on each request and trace, and thread evaluators score the whole conversation up to its latest request.
Where to go next
Experiments
Run evaluators over a dataset snapshot against a target.
Human review
Score human evaluators and calibrate judges.

