Custom training scripts

​

How it works

​

When your script fails

Your script exited with status 1. Last lines:
Traceback (most recent call last):
  File "/workspace/code/train.py", line 12, in <module>
KeyError: 'prompt'

​

Keeping the job alive

• Progress: append one JSON line per step to $VERI_METRICS_FILE, for example {"step": 40, "loss": 1.21}, or call veri_sdk.log_metrics. The job’s current step and its curves come only from this file; without it the job shows no step.
• Checkpoints: write checkpoint-<step>/ directories into $VERI_CHECKPOINT_DIR during the run (see Checkpoints). $VERI_OUTPUT_DIR is captured only when your script exits with status 0, so a run that dies midway keeps only the checkpoints it already wrote to $VERI_CHECKPOINT_DIR.

Worker stopped reporting for 960 s (last status: training, step 40). The worker reports at least once a minute while it is alive, so after more than 900 s of silence the control plane failed the job and terminated its instance.

​

Base images

​

Example: a minimal training run

my-run/
  train.py
  requirements.txt

import os
from pathlib import Path

import veri_sdk

output_dir = Path(os.environ["VERI_OUTPUT_DIR"])

# ... your training setup (model, dataset from VERI_DATA_DIR, optimizer) ...

for step in range(100):
    # ... one training step, producing `reward` and `loss` ...
    reward, loss = train_one_step()

    # Reward/loss curves render natively in the Veri UI.
    veri_sdk.log_metrics(step=step, reward=reward, loss=loss)

# Anything written under VERI_OUTPUT_DIR is captured as your checkpoint.
save_checkpoint(output_dir / "final")

trl==0.22.2
veri-sdk

veri run-script ./my-run \
  --entrypoint 'torchrun --nproc_per_node=$VERI_NUM_GPUS train.py' \
  --base-image veri/base \
  --gpu-type L4-24GB \
  --gpu-count 1 \
  --requirements ./my-run/requirements.txt

​

Multi-node

• Whole-box only: each node is an 8-GPU box, so set --gpu-count 8 with --gpu-type A100-80GB or --gpu-type H100-80GB. Other shapes are rejected.
• AWS only: multi-node runs on AWS.
• Custom script only: managed methods (GRPO, SFT, DPO) are single-node.
• Exact node count or fail: --num-nodes must be one of 1, 2, 4, 8, 16. The gang launches at exactly that size or the submit fails fast (no silent cap-down).

import os
from pathlib import Path

import torch
import torch.distributed as dist
import veri_sdk

output_dir = Path(os.environ["VERI_OUTPUT_DIR"])

# torchrun sets RANK / LOCAL_RANK / WORLD_SIZE from the flags below.
dist.init_process_group(backend="nccl")
rank = dist.get_rank()
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))

# ... build model, wrap in DDP/FSDP, train ...
for step in range(100):
    reward, loss = train_one_step()
    if rank == 0:
        veri_sdk.log_metrics(step=step, reward=reward, loss=loss)

# Only rank 0 writes the checkpoint so the N nodes do not clobber each other.
if rank == 0:
    save_checkpoint(output_dir / "final")

dist.barrier()
dist.destroy_process_group()

torchrun --nnodes=$VERI_NUM_NODES --node-rank=$VERI_NODE_RANK \
         --rdzv-backend=c10d --rdzv-endpoint=$VERI_HEAD_IP:29400 \
         --rdzv-id=$VERI_JOB_ID --nproc-per-node=$VERI_NUM_GPUS \
         train.py

veri run-script ./my-run \
  --entrypoint "torchrun --nnodes=\$VERI_NUM_NODES --node-rank=\$VERI_NODE_RANK --rdzv-backend=c10d --rdzv-endpoint=\$VERI_HEAD_IP:29400 --rdzv-id=\$VERI_JOB_ID --nproc-per-node=\$VERI_NUM_GPUS train.py" \
  --base-image veri/base \
  --gpu-type H100-80GB \
  --gpu-count 8 \
  --num-nodes 2

​

Checkpoints across nodes

• Rank 0 saves (shown above): the simplest pattern. Gate the write on rank == 0 and use a dist.barrier() so other ranks wait for the save to finish.
• Sharded checkpoints: for large models, torch.distributed.checkpoint (DCP) writes one shard per rank under VERI_OUTPUT_DIR, so no two ranks collide and you reload the shards on resume.

​

Observability

• Live logs: your script’s stdout and stderr stream in real time.
• GPU metrics: utilization, memory, and power.
• Checkpoints: whatever you write under VERI_OUTPUT_DIR.
• Reward curves: two ways, pick either.

1. veri_sdk.log_metrics (shown above): append {"step": n, "reward": ..., "loss": ...} records and Veri renders them natively. Add veri-sdk to your requirements.txt.
2. Weights & Biases: connect your W&B account once (veri wandb set) and Veri injects WANDB_API_KEY, WANDB_PROJECT, and WANDB_RUN_ID into the container, so your framework’s existing wandb logging flows to your own W&B account with no code change (OpenRLHF, verl, slime, and TRL all support it out of the box). Veri deep-links the run from the job page. Any WANDB_* value you set yourself in env takes precedence.

​

Limits

• Node counts are exact: 1, 2, 4, 8, or 16. A gang launches at the requested size or the submit fails fast (no cap-down to fewer nodes).
• Multi-node is whole-box, AWS, custom script only: --gpu-count 8 with --gpu-type A100-80GB or H100-80GB on AWS. Managed methods (GRPO, SFT, DPO) are single-node.
• No shared filesystem across nodes yet: each node writes to its own VERI_OUTPUT_DIR. Have rank 0 save, or shard with torch.distributed.checkpoint.
• Four base images (above). If your stack does not fit, a bring-your-own container image is the escape hatch (coming soon).