> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veri.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Harness-in-the-loop RL
> RL-train a model inside your own agent harness (Claude Agent SDK, LangChain, or OpenAI Agents SDK, etc)
Harness-in-the-loop training is **experimental**. The loop (policy endpoint, trajectory capture, GRPO update) is exercised end-to-end in CI, but the GPU path is new — expect rough edges and please report them.
## How it works
For each training step, on one GPU box:
1. **Policy-as-endpoint.** The current policy is served by vLLM (OpenAI-compatible, plus an Anthropic route) with weight sync — after each update the endpoint immediately reflects the new policy.
2. **Rollouts.** Your harness is invoked `rollouts_per_prompt` times per task, sandboxed (gVisor) with network egress. Each invocation is one **rollout**; the N rollouts per task form the GRPO comparison group.
3. **Capture.** A proxy between your harness and the policy records every request's token ids and logprobs. Tool results and user turns are masked out of the loss — only model-generated tokens are trained.
4. **Reward.** Your uploaded reward function scores each finished trajectory (e.g. "did the agent extract the right URL?"). Then the GRPO update runs and the loop repeats.
Every rollout is archived: the **Trajectories** archive sits below the logs on the job page, faceted by policy step, so you can watch behavior evolve as training progresses.
## The harness contract
Your harness is a code bundle plus an entrypoint — the same shape as a [custom script](/training/custom-script). It is invoked **once per rollout** with these environment variables:
| Variable | Meaning |
| ---------------------- | ----------------------------------------------------------------------------------------------------- |
| `VERI_TASK_INPUT` | The task: the row's `prompt` column as JSON (string or messages list). |
| `VERI_TASK` | The full dataset row as JSON, including extra columns. |
| `OPENAI_BASE_URL` | The policy endpoint (OpenAI-compatible). Point your OpenAI SDK / LangChain client here. |
| `VERI_POLICY_BASE_URL` | Same endpoint, SDK-agnostic name. |
| `VERI_POLICY_MODEL` | The model name the endpoint serves (the base model id). vLLM 404s any other name — send exactly this. |
| `ANTHROPIC_BASE_URL` | The policy endpoint's root for the Claude Agent SDK / Anthropic SDK. |
| `ANTHROPIC_AUTH_TOKEN` | Placeholder token (the endpoint doesn't auth; the SDK requires one to boot). |
| `VERI_MAX_TURNS` | Turn budget hint (default 40). |
| `VERI_DATA_DIR` | Scratch dir shared across rollouts. |
You don't add tracing, callbacks, or logging hooks. If your harness already reads `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` — the standard way to point those SDKs at a custom endpoint — it works unchanged.
Both protocols train with full token fidelity: the OpenAI route (`harness_protocol = "openai"`, the default) and the Anthropic route (`"anthropic"`, plain-text messages) capture token ids + logprobs on every request, which is what GRPO trains on. One Anthropic caveat: native `tool_use` blocks are not translated yet — requests carrying `tools` get a clear 400, so use a text-protocol loop (or the OpenAI route) for tool-calling agents. Rollouts whose spans lack token ids are **rejected from training** (fail-safe, never silent drift).
The runtime that implements this loop — the capture proxy, the protocol adapters, episode rendering, and the rejection rules — is open source at [Veri-Studios/veri-runner](https://github.com/Veri-Studios/veri-runner). If a rollout is rejected, the repo's [debugging guide](https://github.com/Veri-Studios/veri-runner#debugging-token-fidelity-rejections) walks the exact rules against your downloaded trajectory; handing the repo and the trajectory to a coding agent works well too.
## Quickstart
The full end-to-end walkthrough lives in the demos section: [Train a URL-extraction agent in your own harness](/demos/harness-url-extraction-agent) — dataset, reward function, unmodified OpenAI-SDK agent, `veri run-harness`, the trajectory archive, and the Claude Agent SDK variant (plus the TOML config-file form).
## Hyperparameters
`grpo_harness` takes all [GRPO knobs](/cli/training) plus:
| Field | Default | Meaning |
| ----------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `harness_protocol` | `"openai"` | `"openai"` or `"anthropic"` — which SDK family the harness uses. |
| `max_turns` | 40 | Turn budget hint (`VERI_MAX_TURNS`). |
| `harness_timeout_s` | 900 | Wall-clock budget for one rollout. |
| `max_parallel_rollouts` | 4 | Concurrent harness invocations per step. |
| `vllm_gpu_memory_utilization` | 0.85 | Policy-server memory fraction on its dedicated GPUs (see GPU layout below). |
| `max_model_len` | unset | Cap the policy server's context length. |
| `trajectory_retention_days` | keep forever | Auto-delete raw trajectories after N days. |
| `chat_template_kwargs` | `{"enable_thinking": false}` | Forwarded to the policy server's chat template. Thinking stays off by default: reasoning models (Qwen3) strip `` blocks from prior turns on re-render, which breaks multi-turn token capture. |
## GPU layout
One node, split devices: the trainer takes GPU 0 and the vLLM policy server takes the remaining GPUs (tensor parallel across them). TRL's weight-sync communicator requires the trainer and the server to sit on **different** CUDA devices, so **at least 2 GPUs** are required — 1-GPU submits are rejected with a 400.
Recommended shapes:
| Shape | Node | Fits |
| --------------------------- | ------------------------- | ----------------------------------------------- |
| `L4-24GB` x4 on `aws` | g6.12xlarge (192 GiB RAM) | \~4B policies (1 trainer GPU + 3-way TP server) |
| `H100-80GB` x2/x4 on `vast` | marketplace chunk | 7B-70B policies |
## Current constraints
* **NVIDIA, single node** (2–8 GPUs — see GPU layout above). AMD, 1-GPU, and multi-node are rejected at submit.
* **Streaming is forced off** on policy requests during training (spans must be capturable); responses arrive buffered.
* Requires a worker TRL build with `rollout_func` support; jobs on older images fail fast with a clear message.
* One policy model per rollout — multi-agent setups with several distinct in-training models are out of scope for now.
* Harness web egress is live user traffic from Veri IPs: keep fetch targets task-relevant.
Docs: [Demo: URL-extraction agent](/demos/harness-url-extraction-agent) · [Managed training](/training/managed) · [Custom scripts](/training/custom-script) · [Reward functions](/training/reward-functions)