Custom training scripts

Veri’s managed methods (GRPO, SFT, DPO) own the training loop for you. When you want whole control (your own framework, your own loop), use a custom script: you bring the code, Veri provisions the GPU, runs your command, and shows you exactly what is happening (live logs, GPU utilization, checkpoints, reward curves) next to your managed jobs and deployments. Use this for OpenRLHF, slime, verl, a plain torchrun loop, or anything you can launch with a command.

How it works

You supply four things:

Input	What it is
base image	One of the curated images Veri prebakes (see below).
code	Your local directory. The CLI tars it and uploads it.
entrypoint	The command Veri runs, e.g. `torchrun --nproc_per_node=$VERI_NUM_GPUS train.py`.
dependencies	An optional `requirements.txt` layered on top of the base image.

Veri runs your entrypoint inside the base image on the GPU, with a set of VERI_* environment variables injected:

Variable	Meaning
`VERI_DATA_DIR`	Your dataset is materialized here (if you attach one).
`VERI_OUTPUT_DIR`	Write checkpoints and artifacts here. Everything in this dir is captured to your account.
`VERI_METRICS_FILE`	Append metrics here (or use `veri.log_metrics`, below).
`VERI_NUM_GPUS`	GPUs on this node.
`VERI_NODE_RANK`	This node’s 0-based rank in the gang. Single node: `0`.
`VERI_NUM_NODES`	Number of nodes in the gang (`N`). Single node: `1`.
`VERI_HEAD_IP`	Private IP of rank 0, the rendezvous host. Set only for a multi-node gang.
`VERI_NODE_IPS`	Comma-joined ordered private IPs of every node (index 0 = rank 0). Single node: one IP.
`VERI_JOB_ID`	The rendezvous id for the gang. Set only for a multi-node gang.

Exit code 0 marks the job completed; a non-zero exit marks it failed.

Base images

Pick the base that matches your stack. Each prebakes the slow, compiled pieces so cold start stays fast; you layer light dependencies on top.

Image	Contains	Good for
`veri/base`	CUDA + PyTorch + the Hugging Face stack (transformers, datasets, accelerate, peft)	Most custom training and SFT-style loops
`veri/base-vllm`	`veri/base` + vLLM	OpenRLHF-style scripts, any vLLM-rollout RL
`veri/base-sglang`	`veri/base` + SGLang	slime and SGLang-based stacks

Example: a minimal training run

A small project directory:

my-run/
  train.py
  requirements.txt

train.py reads the Veri contract, trains, logs metrics, and writes a checkpoint:

train.py

import os
from pathlib import Path

import veri_sdk

output_dir = Path(os.environ["VERI_OUTPUT_DIR"])

# ... your training setup (model, dataset from VERI_DATA_DIR, optimizer) ...

for step in range(100):
    # ... one training step, producing `reward` and `loss` ...
    reward, loss = train_one_step()

    # Reward/loss curves render natively in the Veri UI.
    veri_sdk.log_metrics(step=step, reward=reward, loss=loss)

# Anything written under VERI_OUTPUT_DIR is captured as your checkpoint.
save_checkpoint(output_dir / "final")

requirements.txt layers your extra dependencies on top of the base image:

requirements.txt

trl==0.22.2
veri-sdk

Submit it with the CLI:

veri run-script ./my-run \
  --entrypoint "torchrun --nproc_per_node=$VERI_NUM_GPUS train.py" \
  --base-image veri/base \
  --gpu-type H100-80GB \
  --gpu-count 1 \
  --requirements ./my-run/requirements.txt

The CLI uploads your code, submits the job, and prints the job id and dashboard link.

Multi-node

For runs that do not fit on one box, launch a multi-node gang with --num-nodes. Veri provisions all nodes together as a single unit, wires the topology into the VERI_* env above, and runs your entrypoint on every node. Your script drives the distributed launch (torchrun, with c10d rendezvous on rank 0). Constraints:

Whole-box only: each node is an 8-GPU box, so set --gpu-count 8 with --gpu-type A100-80GB or --gpu-type H100-80GB. Other shapes are rejected.
AWS only: multi-node runs on AWS.
Custom script only: managed methods (GRPO, SFT, DPO) are single-node.
Exact node count or fail: --num-nodes must be one of 1, 2, 4, 8, 16. The gang launches at exactly that size or the submit fails fast (no silent cap-down).

Point torchrun at the injected topology. Rank 0 is the rendezvous host (VERI_HEAD_IP), every node reads its own VERI_NODE_RANK, and VERI_JOB_ID is the rendezvous id:

train.py

import os
from pathlib import Path

import torch
import torch.distributed as dist
import veri_sdk

output_dir = Path(os.environ["VERI_OUTPUT_DIR"])

# torchrun sets RANK / LOCAL_RANK / WORLD_SIZE from the flags below.
dist.init_process_group(backend="nccl")
rank = dist.get_rank()
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))

# ... build model, wrap in DDP/FSDP, train ...
for step in range(100):
    reward, loss = train_one_step()
    if rank == 0:
        veri_sdk.log_metrics(step=step, reward=reward, loss=loss)

# Only rank 0 writes the checkpoint so the N nodes do not clobber each other.
if rank == 0:
    save_checkpoint(output_dir / "final")

dist.barrier()
dist.destroy_process_group()

Your entrypoint is the torchrun launch line, consuming the injected topology directly:

torchrun --nnodes=$VERI_NUM_NODES --node-rank=$VERI_NODE_RANK \
         --rdzv-backend=c10d --rdzv-endpoint=$VERI_HEAD_IP:29400 \
         --rdzv-id=$VERI_JOB_ID --nproc-per-node=$VERI_NUM_GPUS \
         train.py

Submit a 2-node H100 gang (16 GPUs total):

veri run-script ./my-run \
  --entrypoint "torchrun --nnodes=\$VERI_NUM_NODES --node-rank=\$VERI_NODE_RANK --rdzv-backend=c10d --rdzv-endpoint=\$VERI_HEAD_IP:29400 --rdzv-id=\$VERI_JOB_ID --nproc-per-node=\$VERI_NUM_GPUS train.py" \
  --base-image veri/base \
  --gpu-type H100-80GB \
  --gpu-count 8 \
  --num-nodes 2

Checkpoints across nodes

Everything written under VERI_OUTPUT_DIR is captured, the same as single node. With N nodes you must avoid each rank writing the same path:

Rank 0 saves (shown above): the simplest pattern. Gate the write on rank == 0 and use a dist.barrier() so other ranks wait for the save to finish.
Sharded checkpoints: for large models, torch.distributed.checkpoint (DCP) writes one shard per rank under VERI_OUTPUT_DIR, so no two ranks collide and you reload the shards on resume.

Each node writes to its own VERI_OUTPUT_DIR; a shared filesystem across nodes is not provided yet.

Observability

Everything shows up in the Veri UI for the job, alongside your managed jobs and deployments:

Live logs: your script’s stdout and stderr stream in real time.
GPU metrics: utilization, memory, and power.
Checkpoints: whatever you write under VERI_OUTPUT_DIR.
Reward curves: two ways, pick either.

For reward and loss curves you have two options:

veri_sdk.log_metrics (shown above): append {"step": n, "reward": ..., "loss": ...} records and Veri renders them natively. Add veri-sdk to your requirements.txt.
Weights & Biases: connect your W&B account once (veri wandb set) and Veri injects WANDB_API_KEY, WANDB_PROJECT, and WANDB_RUN_ID into the container, so your framework’s existing wandb logging flows to your own W&B account with no code change (OpenRLHF, verl, slime, and TRL all support it out of the box). Veri deep-links the run from the job page. Any WANDB_* value you set yourself in env takes precedence.

Limits

Node counts are exact: 1, 2, 4, 8, or 16. A gang launches at the requested size or the submit fails fast (no cap-down to fewer nodes).
Multi-node is whole-box, AWS, custom script only: --gpu-count 8 with --gpu-type A100-80GB or H100-80GB on AWS. Managed methods (GRPO, SFT, DPO) are single-node.
No shared filesystem across nodes yet: each node writes to its own VERI_OUTPUT_DIR. Have rank 0 save, or shard with torch.distributed.checkpoint.
Three base images (above). If your stack does not fit, a bring-your-own container image is the escape hatch (coming soon).

Get started

Training

Deployments

Evaluations

CLI

Volumes

How it works

Base images

Example: a minimal training run

Multi-node

Checkpoints across nodes

Observability

Limits

​How it works

​Base images

​Example: a minimal training run

​Multi-node

​Checkpoints across nodes

​Observability

​Limits

How it works

Base images

Example: a minimal training run

Multi-node

Checkpoints across nodes

Observability

Limits