> ## Documentation Index > Fetch the complete documentation index at: https://docs.veri.studio/llms.txt > Use this file to discover all available pages before exploring further. # Agent skill (SKILL.md) > A copy-paste SKILL.md that teaches coding agents (Claude Code, Cursor, Devin, Codex) how to drive Veri: training, deployments, datasets, volumes, and the API. Give your coding agent working knowledge of Veri. Copy the SKILL.md below into your agent's skills directory and it will know how to authenticate, submit training jobs, create deployments, and call the API — with the platform's current constraints baked in. | Agent | Where to put it | | ------------------------------------------------------------ | ------------------------------ | | Claude Code | `.claude/skills/veri/SKILL.md` | | Devin CLI | `.devin/skills/veri/SKILL.md` | | Cursor / others ([agentskills spec](https://agentskills.io)) | `.agents/skills/veri/SKILL.md` | Every command and constraint below is verified against the live API and `veri-sdk` 0.2.35 (the latest release). When docs and this file disagree, trust `veri --help` and [the docs](https://docs.veri.studio). ````markdown SKILL.md theme={null} --- name: veri description: Train, deploy, and manage language models on Veri, the AI compute platform (api.veri.studio). Covers the veri CLI and REST API for fine-tuning (SFT, DPO, GRPO), harness-in-the-loop RL (train the model inside your own agent), custom training scripts, GPU inference deployments with OpenAI-compatible endpoints, datasets, inline reward functions, volumes, billing, lifecycle webhooks, Prometheus metrics export, and resource cleanup. Use when the user mentions Veri, veri.toml, the veri CLI, api.veri.studio, or wants to fine-tune / RL-train / serve a model on rented GPUs — including RL-training the model that powers an existing agent or harness. --- # Veri ## Setup — do this first ```bash uv tool install veri-sdk # or: pipx install veri-sdk / pip install veri-sdk veri version # need sdk_version >= 0.2.35 (this skill's baseline); Python >= 3.10 veri login # browser flow; or: veri login --token vk_... (headless/CI) veri whoami # verify: status must NOT be invalid_or_expired_token veri doctor # diagnose install/network if anything fails ``` If `sdk_version` is older than 0.2.35, upgrade before continuing — older CLIs are missing commands this skill uses: `uv tool upgrade veri-sdk` (or `pipx upgrade veri-sdk` / `pip install -U veri-sdk`). 0.2.35 is a hard floor for two reasons: in 0.2.33 and earlier `--reward` took the id of a stored reward resource (it now takes a path to a local `.py`, so reward flows fail against an old CLI), and in 0.2.34 and earlier the `veri-sdk[mcp]` extra installs a broken MCP dependency, so `veri mcp serve` crashes. Veri rents GPUs for model post-training and serving. One CLI (`veri`), one REST API (`https://api.veri.studio`, Bearer auth with `vk_...` keys), TOML configs. - API keys: https://www.veri.studio/settings/api-keys - Env vars `VERI_API_KEY` / `VERI_API_URL` override the config file. - Config file: `~/.config/veri/config.toml` (Linux), `~/Library/Application Support/veri/config.toml` (macOS). `veri whoami` prints the active path. - Global output flags on every command: `--format json|jsonl|csv|table`, `--quiet`. Docs: https://docs.veri.studio/quickstart · https://docs.veri.studio/cli ## Account limits — plan around these Concurrency is capped per account. Defaults: **4 in-flight training jobs** and **2 active deployment replicas**. Both are counted weighted, so a 4-node job consumes 4 job slots and a 2-replica deployment consumes the whole replica budget. - In-flight jobs = `queued` + `provisioning` + `configuring` + `running`. Terminal jobs never count. - Active replicas = `queued` + `provisioning` + `serving` + `unhealthy` + `waking`. A `scaled_to_zero` deployment does NOT count, but it starts counting the moment it wakes. - Enforced only at CREATE. Scaling an existing deployment up with `veri deployments update` is not capped. Hitting a cap returns **HTTP 429 with a `code`** — `CONCURRENT_JOB_LIMIT` or `ACTIVE_DEPLOYMENT_LIMIT`: ```json {"error":{"code":"CONCURRENT_JOB_LIMIT","message":"This workspace has 5 in-flight training jobs (limit 4). Wait for one to finish, or cancel one with `veri jobs cancel `, ..."}} ``` **These 429s are NOT transient — do not retry-loop.** Retrying without first freeing capacity fails every time. Cancel or stop something, then resubmit. Distinguish them from the request rate limiter, which also returns 429 but with a plain-text body and no `code` field (120 requests / 60s per key; back off and retry that one). "Workspace" in these messages is the ownership boundary. For a normal single-user account your personal workspace IS your account, and there are no workspace CLI commands or public API routes — nothing to create, switch, or pass. Raising a limit is a "contact us" lever. ## Project conventions `veri init ` scaffolds: `veri.toml` (project defaults), `configs/{train,deploy}.toml`, `reward.py`, `AGENTS.md`. Every config starts with `kind = "train"|"deploy"`; `veri run ` dispatches on it. Override any key from the CLI: `--set method.learning_rate=2e-6` (TOML-typed; `--set-string` forces string). Docs (full config schemas + `--set` rules): https://docs.veri.studio/cli/run ## Datasets JSONL rows with a `prompt` column (string or OpenAI-style message list) and typically `answer`: ```json {"prompt": [{"role": "user", "content": "What is 12 + 7?"}], "answer": "19"} ``` Chat/ShareGPT/Alpaca/plain-text/prompt-completion/preference (`prompt`,`chosen`,`rejected` for DPO) formats are auto-detected. ```bash veri datasets check ./train.jsonl # validate locally, no upload veri datasets upload ./train.jsonl --name my-train # -> ds_... veri datasets connect-hf openai/gsm8k --config main --split train --map question=prompt ``` Private/gated HF datasets are not supported. Datasets are deletable (API + dashboard). Docs (formats, column mapping, volumes as sources): https://docs.veri.studio/training/datasets · https://docs.veri.studio/cli/datasets ## Reward functions (GRPO) TRL signature, attached to the job inline as Python source text (no separate upload, no reward ids). No network egress inside the sandbox. ```python def reward(completions, answer, **kwargs) -> list[float]: # completions: list of strings OR list of message-dict lists ... ``` CLI: `--reward ./reward.py` on `veri jobs create` / `veri run-harness`, or `[reward] file = "./reward.py"` in train.toml (paths resolve against the config file's directory). The CLI AST-lints the file before submit (parses as Python + defines a function). API/SDK: `reward_source` = the source text; multiple rewards: `reward_sources` + optional `reward_weights` (equal lengths). Limits: max 8 sources, 256 KiB each; violations are 400 at submit. Read a job's submitted source back with `client.training_jobs.reward_source(job_id, index=0)` or on the job's dashboard page. Docs (contract, sandbox limits, debugging): https://docs.veri.studio/training/reward-functions ## Training Methods: `grpo` (needs a reward source), `sft_text`, `dpo`, `sft_video_gen` (these three must omit the reward), `grpo_harness` (needs a reward source; submit via `veri run-harness`, see the harness section below). `grpo_agentic` is NOT runnable (400, being rebuilt). ```toml # configs/train.toml kind = "train" [job] # name, output_model [model] # base = "Qwen/Qwen3-4B" [dataset] # id = "ds_..." [reward] # file = "./reward.py" (grpo only) [method] # type = "grpo"; learning_rate = 1e-6; max_steps = 100; rollouts_per_prompt = 4 [resources] # gpu_type = "H100-80GB"; gpu_count = 1 (optional: provider, region) ``` ```bash veri jobs create configs/train.toml --dry-run # validate + cost estimate, no billing veri jobs create configs/train.toml --follow # submit + stream logs (Ctrl+C detaches) veri jobs logs -f # tail later veri jobs get && veri jobs download --output-dir ./checkpoints veri jobs cancel ``` Key hyperparameter defaults — grpo: lr 1e-6, rollouts_per_prompt 8, kl_coef 0.001, max_response_length 2048; sft_text: lr 2e-5, num_epochs 1; dpo: lr 5e-6, beta 0.1. LoRA via `lora_rank`/`lora_alpha`; QLoRA via `load_in_4bit` (NVIDIA only). Optional: push the trained artifact to the user's Hugging Face account when the job finishes — requires `veri hf set` once (see the Hugging Face section), then add to the train config: ```toml [huggingface] repo = "namespace/name" artifact = "merged" # or "adapter" (needs method.lora_rank); required private = true # default ``` The finished job carries `hf_repo_url`. Submit fails fast (400) if the account isn't connected or `artifact = "adapter"` lacks `lora_rank` — fix, don't retry. Docs (per-method guides + full hyperparameter tables): https://docs.veri.studio/training · https://docs.veri.studio/training/grpo · https://docs.veri.studio/training/sft · https://docs.veri.studio/training/dpo · https://docs.veri.studio/cli/training ### GPUs and providers (training) | Provider | GPUs | Status | | --- | --- | --- | | `aws` (default) | L4-24GB, A10G-24GB (1x); A100-80GB, H100-80GB (whole 8-GPU nodes — you pay for all 8) | live | | `hotaisle` | MI300X-192GB (1/2/4x, AMD ROCm — no unsloth/bitsandbytes/QLoRA) | coming soon (submits 400) | | `vast` | A100/H100-80GB (1/2/4/8x marketplace) | live | | `digitalocean` | H100-80GB (1x/8x droplets; capacity flaps — failed submits are unbilled, retry) | live | | `gcp` | — | gated off: submit returns "coming soon" 400 | Check live prices/availability: `veri gpu list` (add `--provider`/`--region`/`--gpu-type` to switch from the static catalog to live availability), `veri gpu describe ` for specs and node size, `veri gpu compare --gpu-type H100-80GB`, `veri regions list`. Docs (per-provider constraints): https://docs.veri.studio/training/providers ### Custom scripts ```bash veri run-script ./my-run \ --entrypoint "torchrun --nproc_per_node=\$VERI_NUM_GPUS train.py" \ --base-image veri/base --gpu-type H100-80GB --gpu-count 1 \ --requirements ./my-run/requirements.txt ``` Base images: `veri/base`, `veri/base-vllm`, `veri/base-sglang`. Env injected: `VERI_DATA_DIR`, `VERI_OUTPUT_DIR` (write checkpoints here), `VERI_METRICS_FILE`, `VERI_NUM_GPUS`. NVIDIA only (AMD gpu_type → 400). Multi-node (`--num-nodes` ≥ 2) is currently gated off ("coming soon" 400). Exit 0 = completed. Docs (env contract, images, worked examples): https://docs.veri.studio/training/custom-script ### Harness-in-the-loop RL (train the model inside the user's own agent) GRPO-trains a policy while the user's EXISTING agent code drives the multi-turn tool-use rollouts, unmodified. Works with any agent that reads the standard env vars — OpenAI SDK, LangChain, Anthropic SDK. No Veri code inside the agent; each rollout the platform injects: - `OPENAI_BASE_URL` / `ANTHROPIC_BASE_URL` → the in-training policy's endpoint - `VERI_TASK_INPUT` → one dataset row's `prompt` (JSON string); the agent solves it - `VERI_POLICY_MODEL` → model name to send in requests (other names 404) The agent runs its normal loop and exits; the reward function (attached to the job inline) scores the trajectory's FINAL assistant message (extra dataset columns arrive as lists). ```bash veri run-harness ./harness \ --entrypoint "python agent.py" \ --base-model Qwen/Qwen3-4B \ --dataset-id ds_... --reward ./reward.py \ --gpu-type H100-80GB --gpu-count 2 --provider vast \ --rollouts-per-prompt 4 --max-steps 20 \ --requirements ./harness/requirements.txt ``` Constraints that WILL bite: 1. `--gpu-count` must be >= 2 on one node (trainer and policy server need disjoint GPUs). The policy server runs tensor-parallel at gpu_count-1, and vLLM needs the model's attention heads divisible by that — for 32-head models (Qwen3-4B) valid counts are 2, 3, or 5. `H100-80GB x2 --provider vast` is the proven shape. 2. Protocols: OpenAI (default) and Anthropic plain-text (`--harness-protocol anthropic`) both train with exact token fidelity. Anthropic native tool_use blocks are rejected with a 400 — use a text-protocol tool loop instead. 3. Policy ceiling today: ~7-8B full fine-tune (the trainer runs on one GPU). 4. Every dataset task must be WINNABLE through the agent's own tools: if the tools can never surface the ground-truth answer, every rollout scores 0 and that task contributes zero learning signal at full GPU price. Before spending, verify a sample — run the agent's fetch/tool path offline and check the answer appears. 5. Rollouts run sandboxed with network egress; per-rollout deps come from `--requirements`. Inspect results: the job page shows a per-step/task/trial trajectory archive (full message history, tool calls, tokens, reward); programmatic: `client.training_jobs.trajectories(job_id)` + `trajectory_download_url(job_id, tid)`. Docs (env contract + full worked demo incl. dataset prep and reward): https://docs.veri.studio/training/harness · https://docs.veri.studio/demos/harness-url-extraction-agent ## Deployments (inference) ```bash veri deploy Qwen/Qwen3-4B --gpu-type L4-24GB --no-cache # one-step HF deploy veri deployments create --from-hf Qwen/Qwen3-4B --name qwen-dev \ --gpu-type L4-24GB --min-replicas 0 --max-replicas 2 veri deployments create --model ... # serve a finished training job ``` Always pass `--cache` or `--no-cache` to `veri deploy`: without it the command prompts interactively on a TTY and will hang an agent session. (`--cache` reuses/populates a server-side copy of the HF snapshot so repeat deploys of the same repo skip the download.) ALWAYS pass `--gpu-type` explicitly and right-size it (`veri gpu list` for prices; L4-24GB handles ≤4B models). The default is `A100-80GB`, which on AWS is a whole 8-GPU node — an expensive shape for small models. - Engine: `vllm` only (`max` returns 400). Extra vLLM flags: repeat `--vllm-arg=` or `deployment.vllm_extra_args` list in the config. - Serving providers live: `aws` (default), `gcp` (single-GPU A100). Gated off (400): `hotaisle`, `digitalocean`. - Defaults: gpu_count 1, replicas 1, concurrency_target 8, scale_to_zero_window 3600s (min 300). - Mutable on a live deployment: replica bounds, concurrency target, idle windows. Model/GPU/provider/engine are immutable. - Replica budget is 2 by default (see Account limits), so `--max-replicas` above 2 is not servable. - BILLING: a deployment with min_replicas ≥ 1 bills per-minute until YOU stop it. Use `--min-replicas 0` (scale-to-zero) or `veri deployments stop ` when done. Always list deployments after a work session and stop strays. - `veri deployments stop` is IMMEDIATE, not graceful: every box is terminated and in-flight requests are cut. To retire one replica without dropping requests, drain it (below). ### OpenAI-compatible endpoint ```python from openai import OpenAI oai = OpenAI(base_url=f"https://api.veri.studio/v1/deployments/{dep_id}", api_key="vk_...") # client appends /chat/completions resp = oai.chat.completions.create(model="", messages=[...], stream=True) ``` - Streaming works (SSE passthrough). All other params (`top_p`, `stop`, `seed`, `response_format`, `tools`, ...) pass through to vLLM untouched; the `model` field is echoed, not validated. - Tool calling requires the deployment's vLLM to be launched with tool parsing (`--vllm-arg=--enable-auto-tool-choice --vllm-arg=--tool-call-parser --vllm-arg=hermes`). - A scaled-to-zero deployment returns 503 + `Retry-After` (`deployment_waking`) on first request; retry until warm or pre-warm with `veri deployments wake `. - Chat completions only — no embeddings endpoint. Diagnostics: `veri deployments chat|metrics|requests|bench `. ### Replicas and draining (API only, no CLI) `GET /v1/deployments/{id}/replicas` lists every replica with live per-box telemetry: `status`, `requests_running`, `requests_waiting`, `kv_cache_usage`, GPU utilization/memory, `last_heartbeat_at`. Replica statuses: `queued`, `provisioning`, `serving`, `unhealthy`, `draining`, `stopped`, `failed`. `POST /v1/deployments/{id}/replicas/{replica_id}/drain` retires ONE replica gracefully: it stops receiving new requests immediately, then terminates once in-flight requests finish (minimum 60s, hard timeout 600s). Returns 202. Add `?replace=true` to recycle the box without shrinking the fleet — required when it is the last serving replica, or when draining would breach `min_replicas` (both are 409 otherwise). A draining replica is still billed until it terminates, so a drain can cost up to 10 extra minutes of box time. Docs: https://docs.veri.studio/deployments (params + sizing) · https://docs.veri.studio/deployments/openai-compat (interop) · https://docs.veri.studio/deployments/cost (billing states) · https://docs.veri.studio/cli/deployments (scaling) ## Deleting resources ```bash veri jobs delete # terminal jobs only; also deletes stored checkpoints veri datasets delete veri deployments delete # stopped deployments only; also deletes request history veri models delete veri volumes delete --yes # --yes is REQUIRED for agents (see below) veri volumes rm # one file inside a volume ``` Deletes are permanent and there is no `--force` / undo. Two things to know before an agent runs these: 1. `veri volumes delete` is the ONLY one that prompts (it asks you to retype the volume name). Without `--yes` it will hang an agent session. Every other delete above executes immediately with no confirmation, so confirm with the user yourself first. 2. Ordering constraints are enforced server-side, not by the CLI. A job that a deployment still references will not delete; a model with an active deployment returns 409; a deployment only deletes once it is stopped AND its final billing settlement has landed (a few seconds after stopping). Stop first, then delete. ## Webhooks (API only, no CLI) Subscribe to lifecycle events instead of polling. `POST /v1/webhooks` with `{"url": ..., "event_types": [...]}` — omit `event_types` to receive all. The only valid values are: ``` deployment.serving deployment.failed deployment.scaled_to_zero job.completed job.failed ``` Cancelling a job emits nothing (only `completed`/`failed` fire). URLs must be public HTTPS; private/internal addresses are rejected at registration and again at every delivery. Verify every delivery. Each POST carries `webhook-id`, `webhook-timestamp`, and `webhook-signature: v1,` (Standard Webhooks). Sign the exact string `{webhook-id}.{webhook-timestamp}.{raw body}` using the secret with its `whsec_` prefix stripped and the remainder base64-DECODED to raw key bytes. Compare against the value after `v1,` in constant time, and use the raw received body — re-serializing the JSON changes the signature. `webhook-id` is stable across retries, so use it as the idempotency key. Retries run 5s, 60s, 10m, 1h, 6h, then every 12h, giving up 72h after the event. 50 consecutive failures auto-disable the endpoint (re-enable with `PATCH {"enabled": true}`). `POST /v1/webhooks/{id}/test` sends a `webhook.test` event; `GET /v1/webhooks/{id}/deliveries` returns the 50 most recent attempts with response codes and errors for debugging. ## Metrics export (API only, no CLI) `GET /v1/metrics/export` returns your deployments in Prometheus text exposition format (v0.0.4), authenticated with the same `vk_...` bearer key — point a Prometheus, Grafana Agent, or Datadog OpenMetrics scrape straight at it. Gauges cover deployment status and desired/ready replicas; per-replica series cover `requests_running`, `requests_waiting`, `kv_cache_usage_ratio`, GPU utilization and memory, token counters, and `veri_deployment_replica_metrics_age_seconds` (alert on this to catch a stalled replica). Only live replicas emit series, and an idle account returns an empty body — that is normal, not an error. Scrapes share the account's 120-requests-per-60s budget, so keep the interval at 15s or slower. ## Volumes, models, billing, W&B, Hugging Face ```bash veri volumes create my-corpus && veri volumes upload my-corpus ./big.jsonl # >5GiB auto-multipart veri volumes ls my-corpus # then: veri datasets ... source volume veri models save --name my-model # reusable model library; deploy with --from-model veri models register --model acme-bot # versioned lineage (acme-bot@v6); first version = production veri models promote acme-bot v6 -m "reason" # move production (--alias staging); rollback / pin / retention exist veri datasets create acme-feedback --format preference # append-only stream; snapshot, then train on acme-feedback@latest veri billing balance | veri billing burn-rate | veri cost summary veri wandb set --api-key --project

# training jobs auto-log to W&B veri hf set # connect a write-scope HF token (prompted, hidden) veri models push-hf --repo ns/name # push a ready library model to HF (--public, --no-wait) ``` Hugging Face pushes run server-side (nothing downloads locally): connect once with `veri hf set`, then either push at end of training (`[huggingface]` in the train config, see Training) or push any ready library model with `veri models push-hf`. Repos default private. One export per model/repo at a time, two concurrent per account (409 = wait, don't retry-loop). Docs: https://docs.veri.studio/volumes · https://docs.veri.studio/deployments/custom-models · https://docs.veri.studio/deployments/model-versions · https://docs.veri.studio/training/dataset-streams · https://docs.veri.studio/training/wandb · https://docs.veri.studio/training/huggingface ## REST API essentials Base `https://api.veri.studio`, header `Authorization: Bearer vk_...`. Public spec: `GET /openapi.json`. Main resources: `/v1/me` (identity), `/v1/training_jobs` (+ `/logs`, `/events`, `/metrics`, `/cancel`, `/model`, `/trajectories`), `/v1/deployments` (+ `/chat/completions`, `/stop`, `/wake`, `/replicas`, `/replicas/{id}/drain`), `/v1/datasets`, `/v1/volumes`, `/v1/models` (+ `/{id}/push-to-hf`, poll `/v1/hf_exports/{id}`), `/v1/webhooks` (+ `/secret`, `/test`, `/deliveries`), `/v1/metrics/export`, `/v1/settings/integrations/{wandb,huggingface}`, `/v1/billing/*`, `/v1/gpu/catalog`, `/v1/gpu/availability`, `/v1/regions`. DELETE exists for training jobs, datasets, deployments, models, volumes (and volume files). Errors: 400 invalid/gated, 401 bad key, 402 insufficient credit, 404 not found/not owned, 409 still referenced/wrong state, 422 schema, 429 concurrency cap (has `code`) or rate limit (plain text). Pagination: `limit` + `after` cursor, `has_more` in response. MCP alternative: hosted server at `https://api.veri.studio/mcp` (read-only: `/readonly/mcp`), or local `veri mcp serve` (needs `veri-sdk[mcp]`). Docs: https://docs.veri.studio/api-reference/introduction · https://docs.veri.studio/cli/mcp ## Ground rules for agents 1. Run `veri whoami` before anything; fix auth first if the token is invalid. 2. Estimate before spending: `veri jobs create --dry-run` / `veri cost estimate`. GPU time bills real money (402 = out of credit). 3. Never leave deployments running: stop or scale-to-zero after use. 4. Prefer `--format json --quiet` when parsing output. 5. On a "coming soon" 400, the provider/feature is gated off — pick a live alternative, don't retry. 6. On a 429 with `CONCURRENT_JOB_LIMIT` / `ACTIVE_DEPLOYMENT_LIMIT`, free capacity first — retrying alone never clears it. 7. Deletes are permanent and mostly unprompted. Confirm with the user before deleting anything you did not create in this session. ## When stuck 1. `veri --help` — authoritative for the installed CLI version; `veri doctor` for install/auth/network. 2. The docs: https://docs.veri.studio — key pages: [quickstart](https://docs.veri.studio/quickstart), [CLI overview](https://docs.veri.studio/cli), [full CLI reference + exit codes + env vars](https://docs.veri.studio/cli/reference), [training](https://docs.veri.studio/training), [deployments](https://docs.veri.studio/deployments), [API reference](https://docs.veri.studio/api-reference/introduction). 3. Live API surface: `curl -s https://api.veri.studio/openapi.json` (public, no auth). 4. Read the error body — Veri 400s explain themselves (gated features say "coming soon"; validation lists supported values). ```` ## Keeping it current Feature gates (providers and engines) change server-side without an SDK release. The skill tells agents to trust live signals — `veri gpu list`, `--dry-run`, and HTTP error messages — over any static table, including its own.