Train a URL-extraction agent in your own harness

The end-to-end harness-in-the-loop flow on one concrete task: improve Qwen3-4B at finding the GitHub and project-page URLs of a given arXiv paper, trained inside the same agent harness it will be deployed in (the train/serve skew argument in one task). The dataset is public (Mithilss/neurips-2025-paperswithcode-artifacts: 3.4k NeurIPS 2025 papers, about 900 with an official repository), the reward is verifiable (URL match vs. ground truth), and the harness is a plain agent loop with nothing Veri-specific in it.

Harness training is experimental — expect rough edges on the GPU path and please report them. This spends real GPU credit; check your balance with veri billing balance first.

Prerequisites

Install the SDK: pip install 'veri-sdk[mcp]' in a Python 3.10+ virtualenv (the stock macOS python3 is too old), and pip install datasets for the dataset prep script
Authenticate: veri login (or set VERI_API_KEY)
Read the concept page once: Harness-in-the-loop RL (the env contract below is what makes the “unmodified harness” part work)

1. Dataset prep

The HF dataset has arxiv_id / title as inputs and repositories / project_pages lists as ground truth — but no prompt column, so we synthesize it. We keep rows with an official repository (cleanest ground truth), and each row gets one of five phrasing variants so the policy doesn’t overfit a single template. This is the whole “port”: JSONL in, JSONL out, no environment spec:

prep_dataset.py

from datasets import load_dataset
import json

DATASET = "Mithilss/neurips-2025-paperswithcode-artifacts"

PROMPT_VARIANTS = [
    "Find the GitHub repository URL for the arXiv paper {arxiv_id}{title_part}. "
    "If the paper also has a project page, include it. Answer with just the URL(s).",
    "What's the official GitHub repo of arXiv:{arxiv_id}{title_part}? "
    "Include the project page if there is one. URLs only.",
    "I need the code repository for the paper {arxiv_id}{title_part}. "
    "Track down the GitHub URL (and the project page, if any) and reply with just the links.",
    "Look up arXiv paper {arxiv_id}{title_part} and give me its GitHub URL. "
    "Project page too if it exists. Just the URLs, nothing else.",
    "Locate the source code for {arxiv_id}{title_part}. "
    "Return the repository URL, plus the project page URL if the paper has one.",
]

def first_official(entries):
    for e in entries or []:
        if e.get("is_official") and e.get("url"):
            return e["url"]
    return ""

ds = load_dataset(DATASET, split="train")
with open("tasks.jsonl", "w") as f:
    written = 0
    for row in ds:
        github_url = first_official(row.get("repositories"))
        if written >= 128 or not github_url:
            continue
        title = (row.get("title") or "").strip()
        f.write(json.dumps({
            "prompt": PROMPT_VARIANTS[written % len(PROMPT_VARIANTS)].format(
                arxiv_id=row["arxiv_id"],
                title_part=f' ("{title}")' if title else "",
            ),
            "answer_github": github_url,
            "answer_project": first_official(row.get("project_pages")),
        }) + "\n")
        written += 1

A task is only trainable if the harness’s fetch can actually surface the ground-truth URL: an unwinnable task scores 0 on every rollout, so its whole GRPO group contributes zero advantage. Filter before uploading: verify_tasks.py runs the same lookup the harness performs and keeps the winnable rows (about 9 in 10 papers pass). Run it after prep_dataset.py:

verify_tasks.py

import json
import re
import time
import urllib.request


def observation(arxiv_id):
    # The same lookup the harness performs: HF paper page first, arXiv as fallback.
    for url in (f"https://huggingface.co/papers/{arxiv_id}", f"https://arxiv.org/abs/{arxiv_id}"):
        try:
            req = urllib.request.Request(url, headers={"User-Agent": "veri-demo-agent/1.0"})
            with urllib.request.urlopen(req, timeout=30) as resp:
                html = resp.read().decode("utf-8", errors="replace")
        except Exception:
            continue
        links = sorted(set(re.findall(r"https?://(?:www\.)?github\.com/[\w.\-]+/[\w.\-]+", html)))
        text = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", html))[:2500]
        return (text + " " + " ".join(links[:10])).lower()
    return ""


rows = [json.loads(line) for line in open("tasks.jsonl")]
kept = []
for row in rows:
    arxiv_id = re.search(r"\b(\d{4}\.\d{4,5})\b", row["prompt"]).group(1)
    truth = row["answer_github"].strip().rstrip("./").lower()
    if truth in observation(arxiv_id):
        kept.append(row)
    time.sleep(0.3)  # be polite to the paper hosts
with open("tasks.jsonl", "w") as f:
    f.writelines(json.dumps(row) + "\n" for row in kept)
print(f"kept {len(kept)}/{len(rows)} winnable tasks")

Upload it: veri datasets upload tasks.jsonl --name paper-url-extraction → note the dataset id.

2. Reward function: score the finished trajectory

A TRL-format reward with partial credit — 0.7 for the GitHub URL, +0.3 for the project page when one is labeled. completions is the trajectory’s final assistant message; extra dataset columns arrive as lists:

reward.py

def _norm(url):
    return (url or "").strip().rstrip("./").lower()

def reward(prompts, completions, answer_github, answer_project=None, **kwargs):
    rewards = []
    for i, completion in enumerate(completions):
        text = _norm(completion)
        score = 0.0
        if answer_github[i] and _norm(answer_github[i]) in text:
            score += 0.7
        labeled_project = answer_project[i] if answer_project else ""
        if labeled_project and _norm(labeled_project) in text:
            score += 0.3
        elif not labeled_project and answer_github[i] and _norm(answer_github[i]) in text:
            score += 0.3  # no project page labeled: GitHub alone is full credit
        rewards.append(score)
    return rewards

Nothing to upload here: the file rides the submit itself (--reward reward.py in step 4).

3. The harness: your agent, unchanged

Nothing Veri-specific in any of these — the rollout redirects the SDK with env vars alone (ANTHROPIC_BASE_URL / OPENAI_BASE_URL, plus the task in VERI_TASK_INPUT and the served model name in VERI_POLICY_MODEL). Pick the stack your production harness already uses:

Anthropic (Claude-style tools)
OpenAI SDK
LangChain

agent.py (the default) uses native tool_use blocks over the Anthropic messages API. The model calls web_fetch, the harness executes it and returns tool_result:

agent.py

import json
import os
import re
import time
import urllib.error
import urllib.request

import anthropic

client = anthropic.Anthropic()  # ANTHROPIC_BASE_URL / ANTHROPIC_AUTH_TOKEN injected per rollout
MODEL = os.environ.get("VERI_POLICY_MODEL", "Qwen/Qwen3-4B")
MAX_TURNS = int(os.environ.get("VERI_MAX_TURNS", "10"))

SYSTEM = (
    "You find GitHub repository (and project page) URLs for arXiv papers. "
    "Use the web_fetch tool to look up the paper: "
    "https://huggingface.co/papers/<id> lists the paper's linked code "
    "repositories (fall back to https://arxiv.org/abs/<id>), and any "
    "candidate repository page. When you are confident, reply with "
    "'ANSWER: <github url>' (plus 'PROJECT: <project page url>' if one "
    "exists) and nothing else."
)

TOOLS = [{
    "name": "web_fetch",
    "description": "Fetch a web page and return its visible text (truncated).",
    "input_schema": {
        "type": "object",
        "properties": {"url": {"type": "string"}},
        "required": ["url"],
    },
}]


def web_fetch(url: str) -> str:
    html = None
    for delay in (0, 5, 15):  # parallel rollouts can trip HF's 429 rate limit
        time.sleep(delay)
        try:
            req = urllib.request.Request(url, headers={"User-Agent": "veri-demo-agent/1.0"})
            with urllib.request.urlopen(req, timeout=30) as resp:
                html = resp.read().decode("utf-8", errors="replace")
            break
        except urllib.error.HTTPError as e:
            if e.code != 429:
                raise
    if html is None:
        raise RuntimeError(f"rate-limited fetching {url}")
    # Repo links come from the raw HTML hrefs so they survive the truncation.
    links = sorted(set(re.findall(r"https?://(?:www\.)?github\.com/[\w.\-]+/[\w.\-]+", html)))
    text = re.sub(r"<script\b[^>]*>.*?</script\b[^>]*>", " ", html, flags=re.DOTALL | re.IGNORECASE)
    text = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", text))[:3500]
    if links:
        text += "\nLINKS ON PAGE: " + " ".join(links[:10])
    return text


task = json.loads(os.environ["VERI_TASK_INPUT"])
messages = [{"role": "user", "content": task}]
for turn in range(MAX_TURNS):
    resp = client.messages.create(model=MODEL, max_tokens=1024, system=SYSTEM, tools=TOOLS, messages=messages)
    messages.append({"role": "assistant", "content": resp.content})
    tool_results = []
    for b in resp.content:
        if b.type == "tool_use" and b.name == "web_fetch":
            try:
                content = web_fetch(b.input["url"])
            except Exception as e:
                content = f"fetch failed: {e}"
            tool_results.append({"type": "tool_result", "tool_use_id": b.id, "content": content})
    if not tool_results:
        break  # no tool call: that was the final answer
    messages.append({"role": "user", "content": tool_results})

Submit with --harness-protocol anthropic. This is the same harness shape as the Claude Agent SDK (its Bash / Read / WebFetch loop) over the identical /v1/messages protocol — see the caveats below.

agent_openai.py — a plain ReAct-style loop against OPENAI_BASE_URL (the model replies FETCH: <arxiv_id> to read a page, ANSWER: <url> when done; no native tool-calling needed, so it works on any vLLM config):

agent_openai.py

import json
import os
import re
import time
import urllib.error
import urllib.request

from openai import OpenAI

client = OpenAI()  # reads OPENAI_BASE_URL / OPENAI_API_KEY from the env
MODEL = os.environ.get("VERI_POLICY_MODEL", "Qwen/Qwen3-4B")
MAX_TURNS = int(os.environ.get("VERI_MAX_TURNS", "8"))

SYSTEM = (
    "You find GitHub repository URLs for arXiv papers. On each turn reply with "
    "EXACTLY ONE of:\n"
    "  FETCH: <arxiv_id>   to look up the paper's page (e.g. FETCH: 2305.10601)\n"
    "  ANSWER: <url>       when you know the GitHub URL\n"
    "No other text."
)

def fetch_arxiv(arxiv_id: str) -> str:
    # The HF paper page carries the paper's linked code repositories (the
    # arXiv abstract page usually does not); links come from the raw HTML
    # hrefs so they survive the text truncation.
    arxiv_id = arxiv_id.strip().rstrip(".")
    for url in (f"https://huggingface.co/papers/{arxiv_id}", f"https://arxiv.org/abs/{arxiv_id}"):
        html = None
        for delay in (0, 5, 15):  # parallel rollouts can trip HF's 429 rate limit
            time.sleep(delay)
            try:
                req = urllib.request.Request(url, headers={"User-Agent": "veri-demo-agent/1.0"})
                with urllib.request.urlopen(req, timeout=30) as resp:
                    html = resp.read().decode("utf-8", errors="replace")
                break
            except urllib.error.HTTPError as e:
                if e.code != 429:
                    break  # hard miss (e.g. 404): fall through to the next source
            except Exception:
                break
        if html is None:
            continue
        links = sorted(set(re.findall(r"https?://(?:www\.)?github\.com/[\w.\-]+/[\w.\-]+", html)))
        text = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", html))[:2500]
        if links:
            text += "\nLINKS ON PAGE: " + " ".join(links[:10])
        return text
    return f"fetch failed: no page found for {arxiv_id}"

if __name__ == "__main__":  # lets agent_langchain.py import the helpers above
    task = json.loads(os.environ["VERI_TASK_INPUT"])
    messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": task}]
    for turn in range(MAX_TURNS):
        resp = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.7, max_tokens=512)
        content = (resp.choices[0].message.content or "").strip()
        messages.append({"role": "assistant", "content": content})
        if content.upper().startswith("ANSWER:"):
            break
        match = re.match(r"(?i)^FETCH:\s*(\S+)", content)
        observation = fetch_arxiv(match.group(1)) if match else "Reply with FETCH: <arxiv_id> or ANSWER: <url>."
        messages.append({"role": "user", "content": f"OBSERVATION:\n{observation}"})

Token-id capture is confirmed on this route (as it is on the Anthropic text route).

agent_langchain.py runs the same loop with ChatOpenAI and message types (langchain-openai reads OPENAI_BASE_URL from the env). It reuses the prompt and fetch helper from agent_openai.py (OpenAI tab), so keep both files in ./harness:

agent_langchain.py

import json
import os
import re

from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
from langchain_openai import ChatOpenAI

from agent_openai import MAX_TURNS, SYSTEM, fetch_arxiv  # same prompt and lookup as the OpenAI tab

llm = ChatOpenAI(
    model=os.environ.get("VERI_POLICY_MODEL", "Qwen/Qwen3-4B"),
    temperature=0.7,
    max_tokens=512,
)  # reads OPENAI_BASE_URL / OPENAI_API_KEY from the env

task = json.loads(os.environ["VERI_TASK_INPUT"])
messages = [SystemMessage(content=SYSTEM), HumanMessage(content=task)]
for turn in range(MAX_TURNS):
    content = (llm.invoke(messages).content or "").strip()
    messages.append(AIMessage(content=content))
    if content.upper().startswith("ANSWER:"):
        break
    match = re.match(r"(?i)^FETCH:\s*(\S+)", content)
    observation = fetch_arxiv(match.group(1)) if match else "Reply with FETCH: <arxiv_id> or ANSWER: <url>."
    messages.append(HumanMessage(content=f"OBSERVATION:\n{observation}"))

Trains over the same verified OpenAI route — ChatOpenAI speaks the OpenAI protocol under the hood.

requirements.txt

openai>=1.0
anthropic>=0.40
langchain-openai>=0.2

Both routes are verified training paths (token-id capture confirmed live on OpenAI and Anthropic /v1/messages). The Anthropic route supports native tool_use end to end: the tools array rides the policy’s chat template, tool calls come back as tool_use blocks, and tool_result blocks round-trip, so agent.py is a verified live path. Spans without token ids are rejected from training (fail-safe, never silent drift). The stock Claude Agent SDK runtime (claude -p ...) is also verified live; image and document content blocks are rejected with a 400 (no token-native path).

4. Submit

agent.py (the Anthropic tab) is the tool_use harness. Submit it with the matching protocol, from the directory that holds reward.py and harness/:

veri run-harness ./harness \
  --entrypoint "python agent.py" \
  --harness-protocol anthropic \
  --base-model Qwen/Qwen3-4B \
  --dataset-id <dataset id> \
  --reward reward.py \
  --gpu-type L4-24GB --gpu-count 4 \
  --rollouts-per-prompt 8 --max-steps 50 \
  --requirements ./harness/requirements.txt

Harness jobs need 2 to 8 GPUs on one node: the trainer and the vLLM policy server must sit on different CUDA devices (see GPU layout). --gpu-count 4 is the veri run-harness default (veri-sdk 0.2.45 or later; older CLIs default to 1, which the API rejects), and the CLI refuses a count outside 2 to 8 before it uploads ./harness. On AWS the smallest multi-L4 node is the 4-GPU g6.12xlarge; 2x/4x H100 chunks are available with --provider vast.

For the OpenAI route, swap the entrypoint: --entrypoint "python agent_openai.py" and drop --harness-protocol (openai is the default). LangChain: --entrypoint "python agent_langchain.py". The Anthropic text route shown above trains with the same token fidelity. One node hosts everything: the trainer on GPU 0, the policy server tensor-parallel across GPUs 1–3, and the rollouts. A 4B policy fits comfortably on the 4x L4 node.

5. Watch it learn

Open the job in the dashboard — the Trajectories archive sits right below the logs on harness jobs. Pick a policy step, pick a task, compare the N trials: full message history of every turn, tool calls, per-turn token counts, and the reward. Flip between step 0 and the latest step to see behavior change.

Trajectory archive trial view showing a task card, outcome card with reward 0.70 and token counts, run card with model and job cost, and the numbered system, user, and assistant messages of the rollout — A per-trial trajectory page: task, outcome (reward, turns, tokens), and the full message-by-message conversation.

Download any trajectory as raw JSONL — or programmatically:

rows = client.training_jobs.trajectories(job_id, policy_step=10)
url = client.training_jobs.trajectory_download_url(job_id, rows[0]["id"])

The JSONL schema (rollout record, per-request spans, rendered episode) is documented in the open-source runtime: trajectory-format.md, with worked examples of rejected and fallback trajectories in the same repo.

6. Deploy the result

The trained checkpoint is on the job like any managed run. Save it as a model, deploy that model, and point the same harness at the deployment endpoint:

veri models save <job id> --name url-agent              # prints the model id
veri deployments create --from-model <model id> --name url-agent \
  --gpu-type L4-24GB --gpu-count 1

veri deploy only deploys Hugging Face repos, so use veri deployments create --from-model for a saved model.

Config-file form

veri run accepts the same job as TOML. Upload the harness directory first (client.code_artifacts.upload("./harness") returns an artifact id), then run veri run configs/harness.toml:

configs/harness.toml

kind = "train"

[model]
base = "Qwen/Qwen3-4B"

[dataset]
id = "<dataset id>"

[reward]
file = "../reward.py"  # relative to this config file

[resources]
gpu_type = "L4-24GB"
gpu_count = 4

[method]
type = "grpo_harness"
rollouts_per_prompt = 8
max_steps = 50
harness_protocol = "anthropic"
max_turns = 40

[harness]
base_image = "veri/base"
code_artifact_id = "<artifact id>"
entrypoint = "python agent.py"

[harness.deps]  # same as --requirements ./harness/requirements.txt
kind = "requirements"
content = "openai>=1.0\nanthropic>=0.40\nlangchain-openai>=0.2\n"

Docs: Harness-in-the-loop RL (reference) · Managed training · Reward functions

Managed training

Harness training

Custom training scripts

Deployments

Train a URL-extraction agent in your own harness

Prerequisites

1. Dataset prep

2. Reward function: score the finished trajectory

3. The harness: your agent, unchanged

4. Submit

5. Watch it learn

6. Deploy the result

Config-file form

​Prerequisites

​1. Dataset prep

​2. Reward function: score the finished trajectory

​3. The harness: your agent, unchanged

​4. Submit

​5. Watch it learn

​6. Deploy the result

​Config-file form

Prerequisites

1. Dataset prep

2. Reward function: score the finished trajectory

3. The harness: your agent, unchanged

4. Submit

5. Watch it learn

6. Deploy the result

Config-file form