Skip to main content
Evaluations close Veri’s self-improvement loop: serve a model, capture its traffic, turn traffic into data, train, and only promote the new version when it measurably holds up. You write the evaluators (a judge prompt, a code function, or a human rubric); Veri runs them against your own deployments, stores every score, and can refuse a production promotion that does not meet your bar. Target and judge calls are ordinary requests to your own deployments, through the same serving path your users hit. They cost what those deployments cost; there is no second inference stack.

The pieces

The loop

1

Capture traffic

Deploy with request traces on. Send an X-Veri-Thread-Id header on every request of a conversation so its turns group into a thread.
2

Turn traffic into data

Import a window of traces into a dataset stream, optionally only the ones scored as failing (by reviewers or your own harness), and cut a snapshot.
3

Train and register

Train on the snapshot and register the job as the lineage’s next version.
4

Evaluate

Run the gate’s experiment against the new version: veri models evaluate acme-bot@v7 --wait.
5

Promote

veri models promote acme-bot v7 -m "...". With the gate on, the move succeeds only if the evidence meets every rule; otherwise it is refused with the reasons.

Try it in five commands

A golden set, a code evaluator that runs on your machine, and an experiment against a serving deployment:
exact.py defines one function:
veri experiments run waits, prints a per-evaluator table, and exits 1 when the run fails, which makes it a CI check.

Monitors

A monitor runs one evaluator on a sampled share of one deployment’s live traffic, so production keeps getting scored between experiments. Monitors are managed through the API (there is no CLI or SDK command yet):
  • The deployment must store trace bodies, because the judge reads them: create it with --trace-bodies or turn capture on (see request traces). Otherwise the call returns 409 trace_bodies_required.
  • The evaluator must be an llm_judge, a jev, or a cloud code evaluator that does not set requires_reference (live traffic has no expected answer). Human and local code evaluators are refused with 400.
  • A single_turn evaluator scores sampled requests shortly after their traces are stored. A thread evaluator scores a conversation once it has had no request for idle_minutes, and again if it resumes and goes idle again.
A name another monitor of the workspace already has is refused with 409 on create or rename (the error names that monitor); deleting a monitor frees its name. One evaluator version writes at most one monitor score per target, so two monitors running the same evaluator version on the same deployment compete: the second one’s tasks are skipped and its stats stay zero. Give it its own evaluator or version. List monitors with GET /v1/monitors (?deployment_id= for one deployment), and read, edit, or delete one with GET, PATCH, or DELETE /v1/monitors/<id>. Each monitor reports stats: scores and flagged scores (those that failed the pass rule) today and over 7 days, last_scored_at, and human_agreement once human reviews share targets with it. Its scores carry source: monitor and its monitor_id (GET /v1/scores?monitor_id=<id>) and are kept when the monitor is deleted. An evaluation that errors sends the score.failed webhook. Judge calls are ordinary requests to the judge deployment.

Limitations

What evaluations do not do today:
  • No span or tool-span targets. Evaluators score a whole request (a trace), a whole conversation (a thread), or an experiment item. Individual tool calls or steps inside a request are not scoring targets.
  • No pairwise or composite evaluators. Each evaluator scores one output on its own; there is no A-vs-B judge and no evaluator built from other evaluators. Compare versions with an experiment’s baseline diff instead.
  • Only Veri-served models. Experiment targets and LLM judges are deployments in your workspace. The exceptions: target: none scores the outputs already recorded in your dataset, and local code evaluators run on your machine.
  • No templates. Veri ships no built-in evaluators, rubrics, or benchmark suites; you write every prompt and function.
  • Text only. No multimodal inputs or outputs.
  • Cloud code evaluators are JavaScript or TypeScript only. Python code evaluators run locally through the SDK or CLI.
  • Monitors need traced traffic and a reference-free evaluator. A monitor scores a sampled share of one deployment’s successful live traffic (a failed call is never sampled), so the deployment must capture bodies (trace_bodies), and only llm_judge, jev, and cloud code evaluators that need no reference answer can run on it. Monitors are API only: there is no CLI or SDK command for them yet.

Where to go next

Evaluators

Judge prompts, code functions, human rubrics, and how variables bind.

Experiments

Datasets, targets, thread replay, the local flow, and baseline diffs.

Promotion gate

Require passing evidence before production moves.

Human review

Annotation queues, corrections, and judge calibration.