Skip to main content
Harness-in-the-loop training is experimental. The loop (policy endpoint, trajectory capture, GRPO update) is exercised end-to-end in CI, but the GPU path is new — expect rough edges and please report them.

​
How it works

For each training step, on one GPU box:
  1. Policy-as-endpoint. The current policy is served by vLLM (OpenAI-compatible, plus an Anthropic route) with weight sync — after each update the endpoint immediately reflects the new policy.
  2. Rollouts. Your harness is invoked rollouts_per_prompt times per task, sandboxed (gVisor) with network egress. Each invocation is one rollout; the N rollouts per task form the GRPO comparison group.
  3. Capture. A proxy between your harness and the policy records every request’s token ids and logprobs. Tool results and user turns are masked out of the loss — only model-generated tokens are trained.
  4. Reward. Your uploaded reward function scores each finished trajectory (e.g. “did the agent extract the right URL?”). Then the GRPO update runs and the loop repeats.
Every rollout is archived: the Trajectories archive sits below the logs on the job page, faceted by policy step, so you can watch behavior evolve as training progresses.

​
The harness contract

Your harness is a code bundle plus an entrypoint — the same shape as a custom script. It is invoked once per rollout with these environment variables: You don’t add tracing, callbacks, or logging hooks. If your harness already reads OPENAI_BASE_URL or ANTHROPIC_BASE_URL — the standard way to point those SDKs at a custom endpoint — it works unchanged.
Both protocols train with full token fidelity: the OpenAI route (harness_protocol = "openai", the default) and the Anthropic route ("anthropic", plain-text messages) capture token ids + logprobs on every request, which is what GRPO trains on. One Anthropic caveat: native tool_use blocks are not translated yet — requests carrying tools get a clear 400, so use a text-protocol loop (or the OpenAI route) for tool-calling agents. Rollouts whose spans lack token ids are rejected from training (fail-safe, never silent drift).
The runtime that implements this loop — the capture proxy, the protocol adapters, episode rendering, and the rejection rules — is open source at Veri-Studios/veri-runner. If a rollout is rejected, the repo’s debugging guide walks the exact rules against your downloaded trajectory; handing the repo and the trajectory to a coding agent works well too.

​
Quickstart

The full end-to-end walkthrough lives in the demos section: Train a URL-extraction agent in your own harness — dataset, reward function, unmodified OpenAI-SDK agent, veri run-harness, the trajectory archive, and the Claude Agent SDK variant (plus the TOML config-file form).

​
Hyperparameters

grpo_harness takes all GRPO knobs plus:

​
GPU layout

One node, split devices: the trainer takes GPU 0 and the vLLM policy server takes the remaining GPUs (tensor parallel across them). TRL’s weight-sync communicator requires the trainer and the server to sit on different CUDA devices, so at least 2 GPUs are required — 1-GPU submits are rejected with a 400. Recommended shapes:

​
Current constraints

  • NVIDIA, single node (2–8 GPUs — see GPU layout above). AMD, 1-GPU, and multi-node are rejected at submit.
  • Streaming is forced off on policy requests during training (spans must be capturable); responses arrive buffered.
  • Requires a worker TRL build with rollout_func support; jobs on older images fail fast with a clear message.
  • One policy model per rollout — multi-agent setups with several distinct in-training models are out of scope for now.
  • Harness web egress is live user traffic from Veri IPs: keep fetch targets task-relevant.
Docs: Demo: URL-extraction agent · Managed training · Custom scripts · Reward functions