production promotion that does not meet your bar.
Target and judge calls are ordinary requests to your own deployments, through the same serving path your users hit. They cost what those deployments cost; there is no second inference stack.
The pieces
The loop
1
Capture traffic
Deploy with request traces on. Send an
X-Veri-Thread-Id header on every request of a conversation so its turns group into a thread.2
Turn traffic into data
Import a window of traces into a dataset stream, optionally only the ones scored as failing (by reviewers or your own harness), and cut a snapshot.
3
Train and register
Train on the snapshot and register the job as the lineage’s next version.
4
Evaluate
Run the gate’s experiment against the new version:
veri models evaluate acme-bot@v7 --wait.5
Promote
veri models promote acme-bot v7 -m "...". With the gate on, the move succeeds only if the evidence meets every rule; otherwise it is refused with the reasons.Try it in five commands
A golden set, a code evaluator that runs on your machine, and an experiment against a serving deployment:exact.py defines one function:
veri experiments run waits, prints a per-evaluator table, and exits 1 when the run fails, which makes it a CI check.
Monitors
A monitor runs one evaluator on a sampled share of one deployment’s live traffic, so production keeps getting scored between experiments. Monitors are managed through the API (there is no CLI or SDK command yet):- The deployment must store trace bodies, because the judge reads them: create it with
--trace-bodiesor turn capture on (see request traces). Otherwise the call returns409 trace_bodies_required. - The evaluator must be an
llm_judge, ajev, or acloudcode evaluator that does not setrequires_reference(live traffic has no expected answer). Human and local code evaluators are refused with400. - A
single_turnevaluator scores sampled requests shortly after their traces are stored. Athreadevaluator scores a conversation once it has had no request foridle_minutes, and again if it resumes and goes idle again.
A name another monitor of the workspace already has is refused with
409 on create or rename (the error names that monitor); deleting a monitor frees its name. One evaluator version writes at most one monitor score per target, so two monitors running the same evaluator version on the same deployment compete: the second one’s tasks are skipped and its stats stay zero. Give it its own evaluator or version.
List monitors with GET /v1/monitors (?deployment_id= for one deployment), and read, edit, or delete one with GET, PATCH, or DELETE /v1/monitors/<id>. Each monitor reports stats: scores and flagged scores (those that failed the pass rule) today and over 7 days, last_scored_at, and human_agreement once human reviews share targets with it. Its scores carry source: monitor and its monitor_id (GET /v1/scores?monitor_id=<id>) and are kept when the monitor is deleted. An evaluation that errors sends the score.failed webhook. Judge calls are ordinary requests to the judge deployment.
Limitations
What evaluations do not do today:- No span or tool-span targets. Evaluators score a whole request (a trace), a whole conversation (a thread), or an experiment item. Individual tool calls or steps inside a request are not scoring targets.
- No pairwise or composite evaluators. Each evaluator scores one output on its own; there is no A-vs-B judge and no evaluator built from other evaluators. Compare versions with an experiment’s baseline diff instead.
- Only Veri-served models. Experiment targets and LLM judges are deployments in your workspace. The exceptions:
target: nonescores the outputs already recorded in your dataset, and local code evaluators run on your machine. - No templates. Veri ships no built-in evaluators, rubrics, or benchmark suites; you write every prompt and function.
- Text only. No multimodal inputs or outputs.
- Cloud code evaluators are JavaScript or TypeScript only. Python code evaluators run locally through the SDK or CLI.
- Monitors need traced traffic and a reference-free evaluator. A monitor scores a sampled share of one deployment’s successful live traffic (a failed call is never sampled), so the deployment must capture bodies (
trace_bodies), and onlyllm_judge,jev, and cloud code evaluators that need no reference answer can run on it. Monitors are API only: there is no CLI or SDK command for them yet.
Where to go next
Evaluators
Judge prompts, code functions, human rubrics, and how variables bind.
Experiments
Datasets, targets, thread replay, the local flow, and baseline diffs.
Promotion gate
Require passing evidence before
production moves.Human review
Annotation queues, corrections, and judge calibration.

