Managed Training

Managed training on Veri is dataset + base model → trained checkpoint. You submit a training job; Veri provisions the GPU, runs the loop, streams metrics, and uploads the result. You either download the checkpoint or deploy it directly. The training method you pick decides how outputs are scored. GRPO takes a reward function; SFT and DPO learn directly from examples or preference pairs. For the smallest end-to-end run, see the Managed GRPO quickstart — one decorated reward and one command.

Methods

Method	Summary	Backend
`grpo`	Single-turn RL for LLMs: the model generates one completion and your reward function scores it.	TRL GRPOTrainer (Unsloth-accelerated when enabled)
`grpo_agentic`	Multi-turn agentic RL for LLMs: the model acts in an environment across turns and reward comes from the environment outcome. Not currently available while the agentic engine is rebuilt.	Rebuilding
`sft_text`	Supervised finetuning for LLMs on text or chat data, learning directly from your labeled completions.	TRL SFTTrainer
`dpo`	Preference tuning for LLMs on chosen/rejected pairs, with no reward function.	TRL DPOTrainer
`sft_video_gen`	LoRA finetuning for video-gen models (CogVideoX, Wan2.1, LTX, Mochi) from the dataset’s video pairs.	diffusers + LoRA via accelerate

GRPO is the default. All methods support LoRA and QLoRA.

grpo_agentic is being rebuilt and is not currently runnable as a managed method. To run multi-turn, environment-based RL today, use a custom script with your own framework (for example verl) and environment.

The training loop

What happens at each transition:

queued — job row written, balance pre-checked (HTTP 402 if insufficient credit). No spend yet.
provisioning — Veri leases a GPU and the instance boots and self-configures the worker. started_at is still null. Once the instance boots, the GPU is rented and cost accrues from this point, even before the first training step. A job that never secures a GPU (for example CAPACITY_UNAVAILABLE) rents nothing and is not billed.
running — worker reports its first step. started_at populates. Metrics start streaming.
completed | failed | cancelled — pod auto-terminates. Cost is attributed (cost_attributed_at). Stripe credit is deducted (billing_deducted_at). The full stdout is uploaded to S3 and available via GET /v1/training_jobs/{id}/logs/full.

Cancellation is immediate: the instance is terminated as soon as the request is accepted. The worker does not get a chance to flush anything, so whatever checkpoints already reached storage are what you keep — see Checkpoints.

Failure modes

When a job ends in failed, error.code carries one of:

`error.code`	Common cause	What to do
`oom`	Model too large for the chosen GPU	Drop to a smaller base, raise `gpu_count`, or shrink `max_response_length`
`CAPACITY_UNAVAILABLE`	No GPU capacity for the requested type in any availability zone, after trying every capable zone (`max_launch_attempts`)	Retry later, pick a different `gpu_type`, or launch in another region (`veri regions list`, then `region="..."`). Capacity is provider-side and not pre-checkable
`provisioning_error`	The instance booted but the worker did not come up	Re-submit; if persistent, try a different `gpu_type`
`provisioning_timeout`	The instance never reached a running worker within the provisioning deadline	Re-submit; capacity or boot was too slow
`configuring_timeout`	The worker booted but made no training progress within the configuring deadline	Re-submit; if persistent, check the full log via `GET /v1/training_jobs/{id}/logs/full`
`heartbeat_timeout`	Worker stopped reporting status for longer than `heartbeat_timeout_seconds` (default 300s; likely OOM-killed or network dropped)	Re-submit; if persistent, raise `gpu_count`
`training_error`	Worker crash, reward function error, or dataset loading error (generic catch-all)	Read `error.message` + the full log via `GET /v1/training_jobs/{id}/logs/full`

The last 200 log lines from the worker are surfaced in error.log_tail for fast triage; the full stdout is uploaded to S3 and accessible via GET /v1/training_jobs/{id}/logs/full.

Checkpoints

Every managed job writes a final artifact when training finishes. It also saves intermediate checkpoints while training runs, so a job that fails at hour 19 of 20 still leaves usable weights behind.

Intermediate checkpoints are rolling out behind a feature flag. Until it is enabled for your workspace, jobs write only the final artifact and the veri checkpoints commands report the feature as unavailable.

Two hyperparameters control them:

Knob	Default	Meaning
`checkpoint_every`	`0.1`	How often to save. A value between 0 and 1 is a fraction of the run (`0.1` = ten checkpoints, whatever the run length). A value of 1 or more is a literal step count and must be a whole number. `0` turns intermediate checkpoints off.
`checkpoint_keep`	`3`	How many checkpoints to retain. Older ones are deleted as new ones are written. `0` keeps every checkpoint.

import veri

job = veri.train(
    base_model="Qwen/Qwen3-4B",
    dataset="gsm8k-train",
    method="grpo",
    checkpoint_every=0.1,
    checkpoint_keep=3,
)

Working with checkpoints

veri checkpoints list job_abc123
veri checkpoints get job_abc123 job_abc123@400
veri checkpoints download job_abc123 job_abc123@400 --output ./ckpt

Anywhere a checkpoint ID is accepted, <job_id>@<step> works too, so you can act on a step number you read off a loss curve. download fetches the model and tokenizer files by default — enough to load with from_pretrained. Optimizer state is roughly two thirds of a checkpoint’s size and from_pretrained never reads it, so pass --artifact when you need more:

`--artifact`	Contents
`weights` (default)	Model + tokenizer. Loads with `from_pretrained`.
`resumable`	Everything, including optimizer, scheduler, and RNG state.
`adapter`	LoRA adapter files only.

Checkpoints from a custom script

Custom script jobs get a directory instead of a hyperparameter. Write a checkpoint-<step> subdirectory into $VERI_CHECKPOINT_DIR and Veri uploads it:

import os

checkpoint_dir = os.environ.get("VERI_CHECKPOINT_DIR")
if checkpoint_dir:
    model.save_pretrained(f"{checkpoint_dir}/checkpoint-{step}")
    tokenizer.save_pretrained(f"{checkpoint_dir}/checkpoint-{step}")

Veri uploads a checkpoint directory once it stops changing, and sweeps for any remaining ones after your script exits. The variable is unset when the feature is not enabled, so guard on it as shown above.

A checkpoint that was still being written when a script crashed cannot be told apart from a complete one, so Veri uploads it rather than discard it. Checkpoints from a custom script are reported as weights and are not guaranteed to be resumable.

Submitting a job

from veri_sdk import Client
client = Client()

job = client.training_jobs.create(
    base_model="Qwen/Qwen2.5-0.5B-Instruct",
    dataset_id=dataset.id,
    reward_source=open("math_reward.py").read(),
    output_name="qwen-0.5b-math",
    method="grpo",
    hyperparameters={
        "learning_rate": 1e-6,
        "rollouts_per_prompt": 4,
        "max_response_length": 512,
        "kl_coef": 0.001,
        "max_steps": 50,
    },
    gpu_type="L4-24GB",
    gpu_count=1,
)

job.wait(poll_interval=15)
job.download(output_dir="./checkpoints")

wait() streams the job’s logs to your terminal while it polls: provisioning, installs, and training steps appear live with no extra flags. Streaming is best-effort: if the log stream drops, wait() keeps polling silently and still returns the finished job. Pass stream=False for quiet polling only. You can also tail any job from another terminal:

veri jobs logs job_abc       # print everything so far
veri jobs logs job_abc -f    # follow until the job finishes (Ctrl+C detaches)

download() fetches the complete Hugging Face checkpoint — weights plus adapter config and tokenizer files — into ./checkpoints/<job_id>/, a directory you can load directly with from_pretrained(...) or push to the Hugging Face Hub.

Choosing a launch region

Capacity is per-region. If your job fails with CAPACITY_UNAVAILABLE, another region may have the GPU you need right now:

veri regions list        # offered regions; default marked, coming-soon flagged
veri gpu list            # live availability per provider and region

Pass region="us-west-2" to client.training_jobs.create, the @veri.train decorator, or veri train --region us-west-2. Omit it to launch in the default region. Unknown or not-yet-available regions are rejected at submit with the list of valid options.

Choosing a GPU provider

Veri supports multiple GPU providers — AWS (NVIDIA, default), Hot Aisle (AMD MI300X), and Vast.ai (NVIDIA marketplace). Pass provider at submit time to choose where your job runs:

job = client.training_jobs.create(
    base_model="Qwen/Qwen2.5-0.5B-Instruct",
    dataset_id=dataset.id,
    reward_source=open("math_reward.py").read(),
    gpu_type="MI300X-192GB",
    gpu_count=1,
    provider="hotaisle",
)

See GPU providers for the full provider list, GPU types, pricing, and availability checks.

This snippet uses the low-level client and assumes you have already uploaded a dataset (the dataset it references) and have a reward file on disk; the reward source is attached to the job inline. For a runnable end-to-end version with @training_job + veri train, see the Managed GRPO quickstart.

Same shape from the CLI:

veri run configs/grpo.toml

with kind = "train" in the TOML config. See the CLI training reference for the full schema.

Where to go next

Managed GRPO quickstart

End-to-end on a small model: one decorator, one command.

Deploy your checkpoint

Skip downloading — serve from Veri with an OpenAI-compatible API.

Track it with W&B

Live loss and reward charts for your run.

CLI training commands

veri run, veri jobs, reward files.

Get started

Training

Deployments

CLI

Volumes

Webhooks

Methods

The training loop

Failure modes

Checkpoints

Working with checkpoints

Checkpoints from a custom script

Submitting a job

Choosing a launch region

Choosing a GPU provider

Where to go next

Managed GRPO quickstart

Deploy your checkpoint

Track it with W&B

CLI training commands

​Methods

​The training loop

​Failure modes

​Checkpoints

​Working with checkpoints

​Checkpoints from a custom script

​Submitting a job

​Choosing a launch region

​Choosing a GPU provider

​Where to go next

Managed GRPO quickstart

Deploy your checkpoint

Track it with W&B

CLI training commands

Methods

The training loop

Failure modes

Checkpoints

Working with checkpoints

Checkpoints from a custom script

Submitting a job

Choosing a launch region

Choosing a GPU provider

Where to go next