Managed Training

Managed training on Veri is dataset + base model → trained checkpoint. You submit a training job; Veri provisions the GPU, runs the loop, streams metrics, and uploads the result. You either download the checkpoint or deploy it directly. The training method you pick decides how outputs are scored. GRPO takes a reward function; SFT and DPO learn directly from examples or preference pairs. For the smallest end-to-end run, see the Managed GRPO quickstart — one decorated reward and one command.

Methods

Method	Summary	Backend
`grpo`	Single-turn RL for LLMs: the model generates one completion and your reward function scores it.	TRL GRPOTrainer (Unsloth-accelerated when enabled)
`grpo_agentic`	Multi-turn agentic RL for LLMs: the model acts in an environment across turns and reward comes from the environment outcome. Not currently available while the agentic engine is rebuilt.	Rebuilding
`sft_text`	Supervised finetuning for LLMs on text or chat data, learning directly from your labeled completions.	TRL SFTTrainer
`dpo`	Preference tuning for LLMs on chosen/rejected pairs, with no reward function.	TRL DPOTrainer

GRPO is the default. All methods support LoRA and QLoRA.

grpo_agentic is being rebuilt and is not currently runnable as a managed method. To run multi-turn, environment-based RL today, use a custom script with your own framework (for example verl) and environment.

The training loop

What happens at each transition:

queued: job row written (HTTP 402 at submit if the workspace has no billing account). No spend yet. Veri tries to acquire a GPU instance from here; if no zone has capacity, the job goes straight to failed with CAPACITY_UNAVAILABLE. On GCP the job stays queued for up to about 15 minutes while Veri waits for Flex-start capacity, and longer when the one GCP GPU is in use (see GCP).
provisioning: Veri has leased a GPU and the instance boots. started_at is still null. Once the instance is acquired, the GPU is rented and cost accrues from this point, even before the first training step. If you cancel the job before it reaches running, that cost is waived and the job is not charged (see Training job charges). A job that never secures a GPU (for example CAPACITY_UNAVAILABLE) rents nothing and is not billed.
configuring: the instance is up and the worker is setting up its environment.
running: the worker has started the training process (dataset and model load, then training steps). started_at populates. Metrics start streaming with the first step.
completed | failed | cancelled: pod auto-terminates. Cost is attributed (cost_attributed_at). Stripe credit is deducted (billing_deducted_at). The full stdout is uploaded to S3 and available via GET /v1/training_jobs/{id}/logs/full.

Cancellation is immediate: the instance is terminated as soon as the request is accepted. The worker does not get a chance to flush anything, and a job saves its weights only at completion (see Checkpoints), so a cancelled job leaves no trained weights.

Failure modes

When a job ends in failed, error.code carries one of:

`error.code`	Common cause	What to do
`oom`	Model too large for the chosen GPU	Drop to a smaller base, raise `gpu_count`, or shrink `max_response_length`
`CAPACITY_UNAVAILABLE`	No GPU capacity for the requested type in any availability zone, after trying every capable zone (`max_launch_attempts`)	Retry later, pick a different `gpu_type`, or launch in another region (`veri regions list`, then `region="..."`). Capacity is provider-side and not pre-checkable
`QUOTA_EXCEEDED`	Veri’s own account limit on the provider is full: on AWS the vCPU limit that 4-GPU L4 and A10G boxes share, on DigitalOcean the GPU Droplet limit. A platform limit, not your account (`error.attribution` is `platform`). Nothing booted and nothing is billed	Retry in a few minutes, or resubmit with a shape the message names after `or use:` (always another provider, since the limit covers the whole provider)
`PROVIDER_OUT_OF_FUNDS`	The provider refused the launch because Veri’s own prepaid balance with it (Vast) ran out. A platform problem, not your account (`error.attribution` is `platform`). Nothing booted and nothing is billed	Retry later, or resubmit with a shape the message names after `or use:` (always another provider)
`launch_timeout`	The provider’s GPU launch did not finish within its time limit (10 min on AWS, 15 min on DigitalOcean and Vast, 18 min on GCP) and was stopped. Not billed	Re-submit; if it keeps happening, try another provider or `gpu_type`
`provisioning_error`	The instance booted but the worker did not come up	Re-submit; if persistent, try a different `gpu_type`
`provisioning_timeout`	The instance never reached a running worker within the provisioning deadline	Re-submit; capacity or boot was too slow
`configuring_timeout`	The worker booted but made no training progress within the configuring deadline	Re-submit; if persistent, check the full log via `GET /v1/training_jobs/{id}/logs/full`
`heartbeat_timeout`	Worker stopped reporting status for longer than `heartbeat_timeout_seconds` (default 300s; likely OOM-killed or network dropped)	Re-submit; if persistent, raise `gpu_count`
`worker_failed`	The worker on the instance stopped reporting for more than 15 minutes (900 s), or the instance ended before the worker reported a result (the machine ran out of memory, the worker was killed, or the host went away). For a silent worker the message names how long it was silent and the last status and step it reported	Re-submit, or resume from the newest checkpoint with `resume_from` (`veri jobs resume <id>`); if it repeats, report the job id
`training_error`	Worker crash, reward function error, or dataset loading error (generic catch-all)	Read `error.message` + the full log via `GET /v1/training_jobs/{id}/logs/full`
`distributed_error`	Multi-GPU job: the GPUs stopped talking to each other (NCCL error, collective timeout, rendezvous failure)	Re-submit once; if it repeats on the same configuration, report the job id. See Multi-GPU training
`training_stalled`	Multi-GPU job: no progress line for the longer of 10 minutes and three times the median step time	Check the log tail (it carries each GPU’s stack); lower `micro_batch_size` or the sequence length
`hardware_error`	The box failed a hardware check (missing GPU, memory errors)	Re-submit; this one is on the platform
`upload_failed`	Training finished, but files of the final model could not be uploaded to Veri storage. The job fails rather than completing with missing weights; `error.message` names each file and why	Re-submit; this one is on the platform, and the run is reviewed for a credit
`provider_reclaimed`	GCP deleted or stopped the VM (host error, host maintenance, preemption) before the job finished. Credited back automatically	Resume from the newest checkpoint with `resume_from` (`veri jobs resume <id>`)
`max_run_duration_reached`	GCP ended the VM at its 24-hour maximum run duration	Lower `max_steps` or split the run; resume from the newest checkpoint with `resume_from`

error carries code, message and attribution (there is no log_tail field). For triage, read the last worker lines in the dashboard’s Logs tab or fetch the full stdout, uploaded every 10 seconds and retained after the job ends, via GET /v1/training_jobs/{id}/logs/full.

Who the failure belongs to

A failed job also carries error.attribution:

`error.attribution`	Meaning	Billing
`customer`	The failure came from the job’s own configuration, data or code (`oom`, `config_error`, `dependency_error`, `training_stalled`, `heartbeat_timeout`, `max_run_duration_reached` and similar)	Charged as usual
`platform`	Veri’s hardware or provisioning failed (`hardware_error`, `provisioning_error`, `provisioning_timeout`, `configuring_timeout`), the provider ended the VM (`provider_reclaimed`), the worker on the instance went silent (`worker_failed`), or Veri’s account limit on the provider was full (`QUOTA_EXCEEDED`)	The job’s cost is credited back to the workspace automatically. A `QUOTA_EXCEEDED` job never booted, so it cost nothing. A `worker_failed` job, and any custom-script job, is reviewed by Veri before the credit
`pending`	The cause is not clear from the evidence (for example `distributed_error` or `training_error` on a multi-GPU job). An operator reviews it	Charged; credited if the review finds a platform fault

Credits appear on the job as refund with status (approved, credited, pending or denied) and amount_usd, and as a job_refunded entry in the job’s events. Multi-GPU jobs run a hardware check before any of your code and after a failure; the result is what makes a hardware_error verdict, never text in the log.

Checkpoints

Every managed job saves its trained weights as a final artifact when training completes. Managed jobs can also save intermediate checkpoints while they run, so a job that fails or is cancelled at hour 19 of 20 still leaves weights (and the optimizer state) to download or to continue training from with resume_from. See Checkpoints and resume for the knobs (checkpoint_every, checkpoint_keep), what a checkpoint holds, retention, and how to resume.

Submitting a job

from veri_sdk import Client
client = Client()

job = client.training_jobs.create(
    base_model="Qwen/Qwen2.5-0.5B-Instruct",
    dataset_id=dataset.id,
    reward_source=open("math_reward.py").read(),
    output_name="qwen-0.5b-math",
    method="grpo",
    hyperparameters={
        "learning_rate": 1e-6,
        "rollouts_per_prompt": 4,
        "max_response_length": 512,
        "kl_coef": 0.001,
        "max_steps": 50,
    },
    gpu_type="L4-24GB",
    gpu_count=1,
)

job.wait(poll_interval=15)
job.download(output_dir="./checkpoints")

wait() streams the job’s logs to your terminal while it polls: provisioning, installs, and training steps appear live with no extra flags. Streaming is best-effort: if the log stream drops, wait() keeps polling silently and still returns the finished job. Pass stream=False for quiet polling only. You can also tail any job from another terminal:

veri jobs logs $JOB_ID       # print everything so far
veri jobs logs $JOB_ID -f    # follow until the job finishes (Ctrl+C detaches)

A job that has not written any log yet prints [no logs available yet] and exits 0. A finished job whose log was never uploaded fails instead, with log was never uploaded for this job and a non-zero exit, since no log will ever arrive for it. download() fetches the complete Hugging Face checkpoint — weights plus adapter config and tokenizer files — into ./checkpoints/<job_id>/, a directory you can load directly with from_pretrained(...) or push to the Hugging Face Hub.

Choosing a launch region

Capacity is per-region. If your job fails with CAPACITY_UNAVAILABLE, another region may have the GPU you need right now:

veri regions list              # offered regions; default marked, coming-soon flagged
veri gpu list --provider aws   # live availability per region (bare `veri gpu list` shows the catalog and prices)

Pass region="us-west-2" to client.training_jobs.create or the @training_job decorator, or run veri train --region us-west-2. Omit it to launch in the default region. Unknown or not-yet-available regions are rejected at submit with the list of valid options.

Choosing a GPU provider

Veri supports multiple GPU providers: AWS (NVIDIA, default), Vast.ai (NVIDIA marketplace), DigitalOcean (H100 droplets), and GCP (a single A100 80GB or H100 80GB). Pass provider at submit time to choose where your job runs:

job = client.training_jobs.create(
    base_model="Qwen/Qwen2.5-0.5B-Instruct",
    dataset_id=dataset.id,
    reward_source=open("math_reward.py").read(),
    gpu_type="H100-80GB",
    gpu_count=1,
    provider="digitalocean",
)

See GPU providers for the full provider list, GPU types, pricing, and availability checks.

This snippet uses the low-level client and assumes you have already uploaded a dataset (the dataset it references) and have a reward file on disk; the reward source is attached to the job inline. For a runnable end-to-end version with @training_job + veri train, see the Managed GRPO quickstart.

Same shape from the CLI:

veri run configs/grpo.toml

with kind = "train" in the TOML config. See the CLI training reference for the full schema.

Where to go next

Managed GRPO quickstart

End-to-end on a small model: one decorator, one command.

Deploy your checkpoint

Skip downloading — serve from Veri with an OpenAI-compatible API.

Track it with W&B

Live loss and reward charts for your run.

CLI training commands

veri run, veri jobs, reward files.

Get started

Training

Deployments

Evaluations

Agents

CLI

Volumes

Webhooks

Secrets

Billing

Methods

The training loop

Failure modes

Who the failure belongs to

Checkpoints

Submitting a job

Choosing a launch region

Choosing a GPU provider

Where to go next

Managed GRPO quickstart

Deploy your checkpoint

Track it with W&B

CLI training commands

​Methods

​The training loop

​Failure modes

​Who the failure belongs to

​Checkpoints

​Submitting a job

​Choosing a launch region

​Choosing a GPU provider

​Where to go next

Managed GRPO quickstart

Deploy your checkpoint

Track it with W&B

CLI training commands

Methods

The training loop

Failure modes

Who the failure belongs to

Checkpoints

Submitting a job

Choosing a launch region

Choosing a GPU provider

Where to go next