> ## Documentation Index > Fetch the complete documentation index at: https://docs.veri.studio/llms.txt > Use this file to discover all available pages before exploring further. # Train a URL-extraction agent in your own harness > End-to-end harness-in-the-loop RL: improve Qwen3.5-4B on arXiv paper → GitHub/project-page URL extraction, trained inside the same agent harness it deploys in. The end-to-end [harness-in-the-loop](/training/harness) flow on one concrete task: **improve Qwen3.5-4B at finding the GitHub and project-page URLs of a given arXiv paper**, trained inside the same agent harness it will be deployed in (the train/serve skew argument in one task). The dataset is public ([`Mithilss/neurips-2025-paperswithcode-artifacts`](https://huggingface.co/datasets/Mithilss/neurips-2025-paperswithcode-artifacts) — 3.4k NeurIPS 2025 papers, \~1k with official repositories), the reward is verifiable (URL match vs. ground truth), and the harness is a plain agent loop with nothing Veri-specific in it. Harness training is **experimental** — expect rough edges on the GPU path and please report them. This spends real GPU credit; check your balance with `veri billing balance` first. ## Prerequisites * Install the SDK: `pip install veri-sdk` * Authenticate: `veri login` (or set `VERI_API_KEY`) * Read the concept page once: [Harness-in-the-loop RL](/training/harness) (the env contract below is what makes the "unmodified harness" part work) * The complete runnable bundle (all harness variants, prep script, reward, smoke script) lives in the repo at `tests/documentation_tests/Documentation/Demos/Harness URL extraction agent/` ## 1. Dataset prep The HF dataset has `arxiv_id` / `title` as inputs and `repositories` / `project_pages` lists as ground truth — but no `prompt` column, so we synthesize it. We keep rows with an **official** repository (cleanest ground truth), and each row gets one of five **phrasing variants** so the policy doesn't overfit a single template. This is the whole "port": JSONL in, JSONL out, no environment spec: ```python prep_dataset.py theme={null} from datasets import load_dataset import json DATASET = "Mithilss/neurips-2025-paperswithcode-artifacts" PROMPT_VARIANTS = [ "Find the GitHub repository URL for the arXiv paper {arxiv_id}{title_part}. " "If the paper also has a project page, include it. Answer with just the URL(s).", "What's the official GitHub repo of arXiv:{arxiv_id}{title_part}? " "Include the project page if there is one. URLs only.", "I need the code repository for the paper {arxiv_id}{title_part}. " "Track down the GitHub URL (and the project page, if any) and reply with just the links.", "Look up arXiv paper {arxiv_id}{title_part} and give me its GitHub URL. " "Project page too if it exists. Just the URLs, nothing else.", "Locate the source code for {arxiv_id}{title_part}. " "Return the repository URL, plus the project page URL if the paper has one.", ] def first_official(entries): for e in entries or []: if e.get("is_official") and e.get("url"): return e["url"] return "" ds = load_dataset(DATASET, split="train") with open("tasks.jsonl", "w") as f: written = 0 for row in ds: github_url = first_official(row.get("repositories")) if written >= 128 or not github_url: continue title = (row.get("title") or "").strip() f.write(json.dumps({ "prompt": PROMPT_VARIANTS[written % len(PROMPT_VARIANTS)].format( arxiv_id=row["arxiv_id"], title_part=f' ("{title}")' if title else "", ), "answer_github": github_url, "answer_project": first_official(row.get("project_pages")), }) + "\n") written += 1 ``` A task is only trainable if the harness's fetch can actually surface the ground-truth URL — an unwinnable task scores 0 on every rollout, so its whole GRPO group contributes zero advantage. Filter before uploading (the bundle's `prep_dataset.py --verify` runs the same lookup the harness performs and keeps the winnable rows — about 5 in 6 papers pass): ```bash theme={null} python prep_dataset.py --verify ``` Upload it: `veri datasets upload tasks.jsonl --name paper-url-extraction` → note the dataset id. ## 2. Reward function: score the finished trajectory A TRL-format reward with partial credit — 0.7 for the GitHub URL, +0.3 for the project page when one is labeled. `completions` is the trajectory's **final assistant message**; extra dataset columns arrive as lists: ```python reward.py theme={null} def _norm(url): return (url or "").strip().rstrip("./").lower() def reward(prompts, completions, answer_github, answer_project=None, **kwargs): rewards = [] for i, completion in enumerate(completions): text = _norm(completion) score = 0.0 if answer_github[i] and _norm(answer_github[i]) in text: score += 0.7 labeled_project = answer_project[i] if answer_project else "" if labeled_project and _norm(labeled_project) in text: score += 0.3 elif not labeled_project and answer_github[i] and _norm(answer_github[i]) in text: score += 0.3 # no project page labeled: GitHub alone is full credit rewards.append(score) return rewards ``` Upload it: `veri rewards upload reward.py --name url-match` → note the reward id. ## 3. The harness: your agent, unchanged Nothing Veri-specific in any of these — the rollout redirects the SDK with env vars alone (`ANTHROPIC_BASE_URL` / `OPENAI_BASE_URL`, plus the task in `VERI_TASK_INPUT` and the served model name in `VERI_POLICY_MODEL`). Pick the stack your production harness already uses: `agent.py` (the bundle's default) — native `tool_use` blocks over the Anthropic messages API. The model calls `web_fetch`, the harness executes it and returns `tool_result`: ```python agent.py theme={null} import anthropic client = anthropic.Anthropic() # ANTHROPIC_BASE_URL / ANTHROPIC_AUTH_TOKEN injected per rollout MODEL = os.environ.get("VERI_POLICY_MODEL", "Qwen/Qwen3-4B") TOOLS = [{ "name": "web_fetch", "description": "Fetch a web page and return its visible text (truncated).", "input_schema": { "type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"], }, }] messages = [{"role": "user", "content": task}] for turn in range(MAX_TURNS): resp = client.messages.create(model=MODEL, max_tokens=1024, system=SYSTEM, tools=TOOLS, messages=messages) messages.append({"role": "assistant", "content": resp.content}) tool_results = [ {"type": "tool_result", "tool_use_id": b.id, "content": web_fetch(b.input["url"])} for b in resp.content if b.type == "tool_use" and b.name == "web_fetch" ] if not tool_results: break # final answer messages.append({"role": "user", "content": tool_results}) ``` Submit with `--harness-protocol anthropic`. This is the same harness shape as the Claude Agent SDK (its Bash / Read / WebFetch loop) over the identical `/v1/messages` protocol — see the caveats below. `agent_openai.py` — a plain ReAct-style loop against `OPENAI_BASE_URL` (the model replies `FETCH: ` to read a page, `ANSWER: ` when done; no native tool-calling needed, so it works on any vLLM config): ```python agent_openai.py theme={null} import json import os import re import time import urllib.error import urllib.request from openai import OpenAI client = OpenAI() # reads OPENAI_BASE_URL / OPENAI_API_KEY from the env MODEL = os.environ.get("VERI_POLICY_MODEL", "Qwen/Qwen3-4B") MAX_TURNS = int(os.environ.get("VERI_MAX_TURNS", "8")) SYSTEM = ( "You find GitHub repository URLs for arXiv papers. On each turn reply with " "EXACTLY ONE of:\n" " FETCH: to look up the paper's page (e.g. FETCH: 2305.10601)\n" " ANSWER: when you know the GitHub URL\n" "No other text." ) def fetch_arxiv(arxiv_id: str) -> str: # The HF paper page carries the paper's linked code repositories (the # arXiv abstract page usually does not); links come from the raw HTML # hrefs so they survive the text truncation. arxiv_id = arxiv_id.strip().rstrip(".") for url in (f"https://huggingface.co/papers/{arxiv_id}", f"https://arxiv.org/abs/{arxiv_id}"): html = None for delay in (0, 5, 15): # parallel rollouts can trip HF's 429 rate limit time.sleep(delay) try: req = urllib.request.Request(url, headers={"User-Agent": "veri-demo-agent/1.0"}) with urllib.request.urlopen(req, timeout=30) as resp: html = resp.read().decode("utf-8", errors="replace") break except urllib.error.HTTPError as e: if e.code != 429: break # hard miss (e.g. 404): fall through to the next source except Exception: break if html is None: continue links = sorted(set(re.findall(r"https?://(?:www\.)?github\.com/[\w.\-]+/[\w.\-]+", html))) text = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", html))[:2500] if links: text += "\nLINKS ON PAGE: " + " ".join(links[:10]) return text return f"fetch failed: no page found for {arxiv_id}" task = json.loads(os.environ["VERI_TASK_INPUT"]) messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": task}] for turn in range(MAX_TURNS): resp = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.7, max_tokens=512) content = (resp.choices[0].message.content or "").strip() messages.append({"role": "assistant", "content": content}) if content.upper().startswith("ANSWER:"): break match = re.match(r"(?i)^FETCH:\s*(\S+)", content) observation = fetch_arxiv(match.group(1)) if match else "Reply with FETCH: or ANSWER: ." messages.append({"role": "user", "content": f"OBSERVATION:\n{observation}"}) ``` Token-id capture is confirmed on this route (as it is on the Anthropic text route). `agent_langchain.py` — the same loop with `ChatOpenAI` + message types (`langchain-openai` reads `OPENAI_BASE_URL` from the env): ```python agent_langchain.py theme={null} from langchain_core.messages import AIMessage, HumanMessage, SystemMessage from langchain_openai import ChatOpenAI llm = ChatOpenAI( model=os.environ.get("VERI_POLICY_MODEL", "Qwen/Qwen3-4B"), temperature=0.7, max_tokens=512, ) # reads OPENAI_BASE_URL / OPENAI_API_KEY from the env messages = [SystemMessage(content=SYSTEM), HumanMessage(content=task)] for turn in range(MAX_TURNS): content = (llm.invoke(messages).content or "").strip() messages.append(AIMessage(content=content)) if content.upper().startswith("ANSWER:"): break match = re.match(r"(?i)^FETCH:\s*(\S+)", content) observation = fetch_arxiv(match.group(1)) if match else "Reply with FETCH: or ANSWER: ." messages.append(HumanMessage(content=f"OBSERVATION:\n{observation}")) ``` Trains over the same verified OpenAI route — `ChatOpenAI` speaks the OpenAI protocol under the hood. ```text requirements.txt theme={null} openai>=1.0 anthropic>=0.40 langchain-openai>=0.2 ``` **Both routes are verified training paths for text-protocol agents** (token-id capture confirmed live on OpenAI and Anthropic `/v1/messages`). The remaining Anthropic caveat is native `tool_use`: requests carrying `tools` get a clear 400 until block translation ships, so tool-calling agents should use a text protocol like `agent_anthropic.py` (or the OpenAI route). Spans without token ids are rejected from training (fail-safe, never silent drift). The Claude Agent SDK proper wraps the `claude-code` CLI (Node.js), which the Python rollout images don't ship yet. The complete, runnable bundle (all harness variants, the dataset prep script, 6 curated smoke tasks, reward, and a local smoke script that runs any variant against any endpoint) lives in the repo at `tests/documentation_tests/Documentation/Demos/Harness URL extraction agent/`. ## 4. Submit The bundle's default `agent.py` is the Anthropic tool\_use harness — submit it with the matching protocol: ```bash theme={null} veri run-harness ./harness \ --entrypoint "python agent.py" \ --harness-protocol anthropic \ --base-model Qwen/Qwen3-4B \ --dataset-id \ --reward-id \ --gpu-type L4-24GB --gpu-count 4 \ --rollouts-per-prompt 8 --max-steps 50 \ --requirements ./harness/requirements.txt ``` `--gpu-count 4` is not optional: the trainer and the vLLM policy server must sit on different CUDA devices (see [GPU layout](/training/harness#gpu-layout)), so harness jobs need at least 2 GPUs on one node. On AWS the smallest multi-L4 node is the 4-GPU g6.12xlarge; 2x/4x H100 chunks are available with `--provider vast`. For the OpenAI route, swap the entrypoint: `--entrypoint "python agent_openai.py"` and drop `--harness-protocol` (openai is the default). LangChain: `--entrypoint "python agent_langchain.py"`. The Anthropic text route shown above trains with the same token fidelity. One node hosts everything: the trainer on GPU 0, the policy server tensor-parallel across GPUs 1–3, and the rollouts. A 4B policy fits comfortably on the 4x L4 node. ## 5. Watch it learn Open the job in the dashboard — the **Trajectories** archive sits right below the logs on harness jobs. Pick a policy step, pick a task, compare the N trials: full message history of every turn, tool calls, per-turn token counts, and the reward. Flip between step 0 and the latest step to see behavior change. Trajectory archive trial view showing a task card, outcome card with reward 0.70 and token counts, run card with model and job cost, and the numbered system, user, and assistant messages of the rollout Download any trajectory as raw JSONL — or programmatically: ```python theme={null} rows = client.training_jobs.trajectories(job_id, policy_step=10) url = client.training_jobs.trajectory_download_url(job_id, rows[0]["id"]) ``` The JSONL schema (rollout record, per-request spans, rendered episode) is documented in the open-source runtime: [trajectory-format.md](https://github.com/Veri-Studios/veri-runner/blob/main/docs/trajectory-format.md), with worked examples of rejected and fallback trajectories in the same repo. ## 6. Deploy the result The trained checkpoint is on the job like any managed run — **Save as model**, then `veri deploy`, and point the same harness at the deployment endpoint. ## Config-file form `veri run` accepts the same job as TOML. Upload the harness directory first (any `veri run-script ...` upload or `client.code_artifacts.upload(dir)` returns an artifact id), then: ```toml configs/harness.toml theme={null} kind = "train" [model] base = "Qwen/Qwen3-4B" [dataset] id = "" [reward] id = "" [resources] gpu_type = "L4-24GB" gpu_count = 4 [method] type = "grpo_harness" rollouts_per_prompt = 8 max_steps = 50 harness_protocol = "openai" max_turns = 40 [harness] base_image = "veri/base" code_artifact_id = "" entrypoint = "python agent.py" ``` Docs: [Harness-in-the-loop RL (reference)](/training/harness) · [Managed training](/training/managed) · [Reward functions](/training/reward-functions)