Skip to main content
Harness-in-the-loop training is experimental. The loop (policy endpoint, trajectory capture, GRPO update) is exercised end-to-end in CI, but the GPU path is new — expect rough edges and please report them.

​
How it works

For each training step, on one GPU box:
  1. Policy-as-endpoint. The current policy is served by vLLM (OpenAI-compatible, plus an Anthropic route) with weight sync — after each update the endpoint immediately reflects the new policy.
  2. Rollouts. Your harness is invoked rollouts_per_prompt times per task, sandboxed (gVisor) with network egress. Each invocation is one rollout; the N rollouts per task form the GRPO comparison group.
  3. Capture. A proxy between your harness and the policy records every request’s token ids and logprobs. Tool results and user turns are masked out of the loss — only model-generated tokens are trained.
  4. Reward. Your reward function (attached to the job inline) scores each finished trajectory (e.g. “did the agent extract the right URL?”). Then the GRPO update runs and the loop repeats.
Every rollout is archived: the Trajectories archive sits below the logs on the job page, faceted by policy step, so you can watch behavior evolve as training progresses.

​
The harness contract

Your harness is a code bundle plus an entrypoint — the same shape as a custom script. It is invoked once per rollout with these environment variables: You don’t add tracing, callbacks, or logging hooks. If your harness already reads OPENAI_BASE_URL or ANTHROPIC_BASE_URL — the standard way to point those SDKs at a custom endpoint — it works unchanged.
Both protocols train with full token fidelity: the OpenAI route (harness_protocol = "openai", the default) and the Anthropic route ("anthropic") capture token ids + logprobs on every request, which is what GRPO trains on. The Anthropic route supports native tool_use end to end — the tools array rides the policy’s chat template, tool calls come back as tool_use blocks, streaming and count_tokens are served — so Anthropic-SDK tool loops train unmodified. Rollouts whose spans lack token ids are rejected from training (fail-safe, never silent drift). Image and document content blocks are rejected with a 400 (no token-native path). The stock Claude Agent SDK runtime (claude -p ...) runs on the veri/base-claude-agent base image, currently rolling out.
The runtime that implements this loop — the capture proxy, the protocol adapters, episode rendering, and the rejection rules — is open source at Veri-Studios/veri-runner. If a rollout is rejected, the repo’s debugging guide walks the exact rules against your downloaded trajectory; handing the repo and the trajectory to a coding agent works well too.

​
Quickstart

The full end-to-end walkthrough lives in the demos section: Train a URL-extraction agent in your own harness — dataset, reward function, unmodified OpenAI-SDK agent, veri run-harness, the trajectory archive, and the Claude Agent SDK variant (plus the TOML config-file form).

​
Hyperparameters

grpo_harness takes all GRPO knobs plus:

​
GPU layout

One node, split devices: the trainer takes GPU 0 and the vLLM policy server takes the next GPUs with tensor parallel set to the largest power of two that fits in the remainder (vLLM requires TP to divide the model’s attention-head count, so odd remainders are clamped down and any leftover GPUs idle). TRL’s weight-sync communicator requires the trainer and the server to sit on different CUDA devices, so at least 2 GPUs are required — 1-GPU submits are rejected with a 400. Recommended shapes: Idle GPUs are reclaimable with data parallelism when the model fits on fewer cards: vllm_data_parallel_size = 3 on L4-24GB x4 runs three TP=1 replicas (zero idle, ~3x rollout throughput for a 4B policy). Rollout generation is usually the training bottleneck, so extra replicas translate directly into faster steps. Each replica holds a full model copy, so pick TP large enough for the model to fit and spend the rest on DP. The trainer side is a single-GPU full fine-tune today, which caps the trainable policy at roughly 7-8B regardless of the server’s size. Set lora_rank for larger policies.

​
Current constraints

  • NVIDIA, single node (2–8 GPUs — see GPU layout above). AMD, 1-GPU, and multi-node are rejected at submit.
  • Streaming is forced off on policy requests during training (spans must be capturable); responses arrive buffered.
  • Requires a worker TRL build with rollout_func support; jobs on older images fail fast with a clear message.
  • One policy model per rollout — multi-agent setups with several distinct in-training models are out of scope for now.
  • Harness web egress is live user traffic from Veri IPs: keep fetch targets task-relevant.
Docs: Demo: URL-extraction agent · Managed training · Custom scripts · Reward functions