> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veri.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom training scripts

> Run your own training script or framework (torchrun, OpenRLHF, slime, verl) on a GPU Veri provisions, with live logs, GPU metrics, checkpoints, and reward curves on one platform.

Veri's managed methods (GRPO, SFT, DPO) own the training loop for you. When you want **whole control** (your own framework, your own loop), use a custom script: you bring the code, Veri provisions the GPU, runs your command, and shows you exactly what is happening (live logs, GPU utilization, checkpoints, reward curves) next to your managed jobs and deployments.

Use this for OpenRLHF, slime, verl, a plain `torchrun` loop, or anything you can launch with a command.

## How it works

You supply four things:

| Input            | What it is                                                                       |
| ---------------- | -------------------------------------------------------------------------------- |
| **base image**   | One of the curated images Veri prebakes (see below).                             |
| **code**         | Your local directory. The CLI tars it and uploads it.                            |
| **entrypoint**   | The command Veri runs, e.g. `torchrun --nproc_per_node=$VERI_NUM_GPUS train.py`. |
| **dependencies** | An optional `requirements.txt` layered on top of the base image.                 |

Veri runs your entrypoint inside the base image on the GPU, with a set of `VERI_*` environment variables injected:

| Variable            | Meaning                                                                                   |
| ------------------- | ----------------------------------------------------------------------------------------- |
| `VERI_DATA_DIR`     | Your dataset is materialized here (if you attach one).                                    |
| `VERI_OUTPUT_DIR`   | Write checkpoints and artifacts here. Everything in this dir is captured to your account. |
| `VERI_METRICS_FILE` | Append metrics here (or use `veri.log_metrics`, below).                                   |
| `VERI_NUM_GPUS`     | GPUs on this node.                                                                        |
| `VERI_NODE_RANK`    | This node's 0-based rank in the gang. Single node: `0`.                                   |
| `VERI_NUM_NODES`    | Number of nodes in the gang (`N`). Single node: `1`.                                      |
| `VERI_HEAD_IP`      | Private IP of rank 0, the rendezvous host. Set only for a multi-node gang.                |
| `VERI_NODE_IPS`     | Comma-joined ordered private IPs of every node (index 0 = rank 0). Single node: one IP.   |
| `VERI_JOB_ID`       | The rendezvous id for the gang. Set only for a multi-node gang.                           |

Exit code `0` marks the job completed; a non-zero exit marks it failed.

## Base images

Pick the base that matches your stack. Each prebakes the slow, compiled pieces so cold start stays fast; you layer light dependencies on top.

| Image              | Contains                                                                           | Good for                                    |
| ------------------ | ---------------------------------------------------------------------------------- | ------------------------------------------- |
| `veri/base`        | CUDA + PyTorch + the Hugging Face stack (transformers, datasets, accelerate, peft) | Most custom training and SFT-style loops    |
| `veri/base-vllm`   | `veri/base` + vLLM                                                                 | OpenRLHF-style scripts, any vLLM-rollout RL |
| `veri/base-sglang` | `veri/base` + SGLang                                                               | slime and SGLang-based stacks               |

## Example: a minimal training run

A small project directory:

```text theme={null}
my-run/
  train.py
  requirements.txt
```

`train.py` reads the Veri contract, trains, logs metrics, and writes a checkpoint:

```python train.py theme={null}
import os
from pathlib import Path

import veri_sdk

output_dir = Path(os.environ["VERI_OUTPUT_DIR"])

# ... your training setup (model, dataset from VERI_DATA_DIR, optimizer) ...

for step in range(100):
    # ... one training step, producing `reward` and `loss` ...
    reward, loss = train_one_step()

    # Reward/loss curves render natively in the Veri UI.
    veri_sdk.log_metrics(step=step, reward=reward, loss=loss)

# Anything written under VERI_OUTPUT_DIR is captured as your checkpoint.
save_checkpoint(output_dir / "final")
```

`requirements.txt` layers your extra dependencies on top of the base image:

```text requirements.txt theme={null}
trl==0.22.2
veri-sdk
```

Submit it with the CLI:

```bash theme={null}
veri run-script ./my-run \
  --entrypoint "torchrun --nproc_per_node=$VERI_NUM_GPUS train.py" \
  --base-image veri/base \
  --gpu-type H100-80GB \
  --gpu-count 1 \
  --requirements ./my-run/requirements.txt
```

The CLI uploads your code, submits the job, and prints the job id and dashboard link.

## Multi-node

For runs that do not fit on one box, launch a multi-node gang with `--num-nodes`. Veri provisions all nodes together as a single unit, wires the topology into the `VERI_*` env above, and runs your entrypoint on every node. Your script drives the distributed launch (`torchrun`, with c10d rendezvous on rank 0).

Constraints:

* **Whole-box only**: each node is an 8-GPU box, so set `--gpu-count 8` with `--gpu-type A100-80GB` or `--gpu-type H100-80GB`. Other shapes are rejected.
* **AWS only**: multi-node runs on AWS.
* **Custom script only**: managed methods (GRPO, SFT, DPO) are single-node.
* **Exact node count or fail**: `--num-nodes` must be one of `1`, `2`, `4`, `8`, `16`. The gang launches at exactly that size or the submit fails fast (no silent cap-down).

Point `torchrun` at the injected topology. Rank 0 is the rendezvous host (`VERI_HEAD_IP`), every node reads its own `VERI_NODE_RANK`, and `VERI_JOB_ID` is the rendezvous id:

```python train.py theme={null}
import os
from pathlib import Path

import torch
import torch.distributed as dist
import veri_sdk

output_dir = Path(os.environ["VERI_OUTPUT_DIR"])

# torchrun sets RANK / LOCAL_RANK / WORLD_SIZE from the flags below.
dist.init_process_group(backend="nccl")
rank = dist.get_rank()
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))

# ... build model, wrap in DDP/FSDP, train ...
for step in range(100):
    reward, loss = train_one_step()
    if rank == 0:
        veri_sdk.log_metrics(step=step, reward=reward, loss=loss)

# Only rank 0 writes the checkpoint so the N nodes do not clobber each other.
if rank == 0:
    save_checkpoint(output_dir / "final")

dist.barrier()
dist.destroy_process_group()
```

Your entrypoint is the `torchrun` launch line, consuming the injected topology directly:

```bash theme={null}
torchrun --nnodes=$VERI_NUM_NODES --node-rank=$VERI_NODE_RANK \
         --rdzv-backend=c10d --rdzv-endpoint=$VERI_HEAD_IP:29400 \
         --rdzv-id=$VERI_JOB_ID --nproc-per-node=$VERI_NUM_GPUS \
         train.py
```

Submit a 2-node H100 gang (16 GPUs total):

```bash theme={null}
veri run-script ./my-run \
  --entrypoint "torchrun --nnodes=\$VERI_NUM_NODES --node-rank=\$VERI_NODE_RANK --rdzv-backend=c10d --rdzv-endpoint=\$VERI_HEAD_IP:29400 --rdzv-id=\$VERI_JOB_ID --nproc-per-node=\$VERI_NUM_GPUS train.py" \
  --base-image veri/base \
  --gpu-type H100-80GB \
  --gpu-count 8 \
  --num-nodes 2
```

### Checkpoints across nodes

Everything written under `VERI_OUTPUT_DIR` is captured, the same as single node. With N nodes you must avoid each rank writing the same path:

* **Rank 0 saves** (shown above): the simplest pattern. Gate the write on `rank == 0` and use a `dist.barrier()` so other ranks wait for the save to finish.
* **Sharded checkpoints**: for large models, `torch.distributed.checkpoint` (DCP) writes one shard per rank under `VERI_OUTPUT_DIR`, so no two ranks collide and you reload the shards on resume.

Each node writes to its own `VERI_OUTPUT_DIR`; a shared filesystem across nodes is not provided yet.

## Observability

Everything shows up in the Veri UI for the job, alongside your managed jobs and deployments:

* **Live logs**: your script's stdout and stderr stream in real time.
* **GPU metrics**: utilization, memory, and power.
* **Checkpoints**: whatever you write under `VERI_OUTPUT_DIR`.
* **Reward curves**: two ways, pick either.

For reward and loss curves you have two options:

1. **`veri_sdk.log_metrics`** (shown above): append `{"step": n, "reward": ..., "loss": ...}` records and Veri renders them natively. Add `veri-sdk` to your `requirements.txt`.
2. **Weights & Biases**: [connect your W\&B account](/training/wandb) once (`veri wandb set`) and Veri injects `WANDB_API_KEY`, `WANDB_PROJECT`, and `WANDB_RUN_ID` into the container, so your framework's existing `wandb` logging flows to your own W\&B account with no code change (OpenRLHF, verl, slime, and TRL all support it out of the box). Veri deep-links the run from the job page. Any `WANDB_*` value you set yourself in `env` takes precedence.

## Limits

* **Node counts** are exact: `1`, `2`, `4`, `8`, or `16`. A gang launches at the requested size or the submit fails fast (no cap-down to fewer nodes).
* **Multi-node is whole-box, AWS, custom script only**: `--gpu-count 8` with `--gpu-type A100-80GB` or `H100-80GB` on AWS. Managed methods (GRPO, SFT, DPO) are single-node.
* **No shared filesystem** across nodes yet: each node writes to its own `VERI_OUTPUT_DIR`. Have rank 0 save, or shard with `torch.distributed.checkpoint`.
* **Three base images** (above). If your stack does not fit, a bring-your-own container image is the escape hatch (coming soon).