Training jobs API

Reference for /v1/training_jobs/*. All endpoints require Bearer token auth (Authorization: Bearer vk_...) unless flagged worker-only.

Endpoints

Method	Path	Purpose
`GET`	`/v1/training_jobs`	List jobs (paginated, filter by `status`, `method`)
`POST`	`/v1/training_jobs`	Create a job
`GET`	`/v1/training_jobs/{id}`	Get one
`GET`	`/v1/training_jobs/{id}/logs`	Stream the worker’s log as server-sent events
`GET`	`/v1/training_jobs/{id}/logs/full`	The full log as plain text (complete once the job ends)
`GET`	`/v1/training_jobs/{id}/events`	Lifecycle events: setup stages, checkpoints, warnings, the outcome, billing
`GET`	`/v1/training_jobs/{id}/metrics`	Sampled training metrics
`POST`	`/v1/training_jobs/{id}/cancel`	Cancel a non-terminal job
`DELETE`	`/v1/training_jobs/{id}`	Hard-delete a finished job (S3 + DB)
`GET`	`/v1/training_jobs/{id}/model`	Presigned download URLs for the checkpoint files
`GET`	`/v1/training_jobs/{id}/rewards/{index}/source`	The reward source a job was submitted with (Reward functions)
`GET`	`/v1/training_jobs/{id}/files`	Every file a job was submitted with: reward sources, code archive, dependency file, entrypoint
`GET`	`/v1/training_jobs/{id}/trajectories`	Trajectory archive of a `grpo_harness` job (Harness training)
`POST`, `DELETE`	`/v1/training_jobs/{id}/publish`	Publish or unpublish a run (Sharing runs)

Worker-only endpoints (used by the runner agent, authenticated via per-job callback_token): POST /status, POST /presigned-upload, POST /logs/presigned-upload. Documented in the worker-callbacks appendix at the bottom.

POST /v1/training_jobs

Create a job. Returns immediately with status queued. Request schema (TrainingJobCreate). This body is the L4 GSM8K example as a raw request: GRPO with LoRA on a single L4-24GB.

{
  "base_model": "Qwen/Qwen2.5-1.5B-Instruct",
  "dataset_id": "9f2c81ab04de",
  "reward_source": "def reward(completions, answer, **kwargs):\n    ...",
  "method": "grpo",
  "hyperparameters": {
    "max_steps": 200,
    "rollouts_per_prompt": 8,
    "max_response_length": 512,
    "lora_rank": 16,
    "lora_alpha": 32,
    "learning_rate": 2e-5,
    "system_prompt": "You are a math tutor. Think step by step, then put ONLY the final numeric answer inside <answer></answer> tags, e.g. <answer>42</answer>."
  },
  "output_name": "qwen-1.5b-gsm8k",
  "gpu": {"gpu_type": "L4-24GB", "gpu_count": 1},
  "provider": "aws",
  "checkpoint_destination": {"type": "veri"}
}

Required: base_model, dataset_id, output_name, gpu (a custom_script job needs script instead of base_model and dataset_id). Required for method=grpo and grpo_harness: reward_source (or reward_sources). Request fields:

Field	Default	Meaning
`job_type`	`"managed"`	`"managed"` runs a managed `method`. `"custom_script"` runs your own code instead: send `script` and omit `base_model`, `method`, and the reward fields (Custom training scripts).
`base_model`	none	Hugging Face model id. Required for managed jobs.
`dataset_id`	none	Dataset id or name (see below). Required for managed jobs, optional for `custom_script`.
`method`	`"grpo"`	`grpo`, `grpo_harness`, `sft_text`, or `dpo`.
`hyperparameters`	`{}`	Method knobs; see Hyperparameter schemas. Keys the method does not read are accepted and listed in an `ignored_hyperparameter_keys` warning event on the job; a key it reads with a value of the wrong type (for example `"lora_rank": "8"`) is a 400.
`output_name`	none	Name of the trained model. Required.
`gpu`	none	`{"gpu_type", "gpu_count"}`; see GPU config schema. Required.
`provider`	`"aws"`	`aws`, `gcp`, `vast`, or `digitalocean`; see GPU providers.
`region`	default region	Launch region, AWS only (`GET /v1/regions` lists them). Other providers choose placement themselves and reject a `region`.
`reward_source`, `reward_sources`, `reward_weights`	none	Inline reward functions (below).
`script`	none	`custom_script` and `grpo_harness` code bundle: `{"base_image", "code_artifact_id", "entrypoint", "deps", "env"}`.
`hf_push`	none	Push the result to Hugging Face when the job finishes: `{"repo", "artifact", "private"}` (Hugging Face).
`num_nodes`	`1`	Gang size for multi-node `custom_script` jobs. Multi-node is not enabled yet: `2` or more returns a 400.
`requires_heartbeat`	`true`	When `true`, a running job whose client stops sending heartbeats (`POST /v1/training_jobs/{id}/heartbeat`, which `veri train` sends while it waits) for 300 seconds is stopped with `heartbeat_timeout`. A job that never received a heartbeat is not affected.
`checkpoint_destination`	`{"type": "veri"}`	Where the checkpoint is stored. Only `{"type": "veri"}` (or omitting the field) is accepted; any other value is a 400. See Checkpoint destination schema.

dataset_id also accepts dataset stream references: a stream’s name or id with @latest or @snap-N (for example acme-feedback@latest). The job resolves and stores the pinned snapshot as dataset_snapshot_id in its response, so later appends never change what the job trains on. Reward fields: reward_source is the Python source text of a TRL-signature reward function. For multiple weighted rewards, pass reward_sources (a list) plus optional reward_weights (same length). Limits: at most 8 sources, 256 KiB each, and each source must define a function; violations are HTTP 400 at submit. See Reward functions. Validation rules:

method=grpo or grpo_harness → reward required (reward_source or reward_sources). grpo_agentic is being rebuilt and sft_video_gen is not available: both are a 400.
method=sft_text or dpo, and job_type=custom_script → reward fields must be omitted (these learn from the dataset, or own their loop)
custom_script is a job_type, not a method: "method": "custom_script" is a 400 (Unknown training method). job_type must be managed or custom_script; anything else is a 400.
environments is not supported: a non-empty value is a 400 (no available method reads it).
A job-level volume (a whole-volume read-only mount at /mnt/<name>) is coming soon: until it is enabled, it is a 400 whose message points at from_volume. A dataset_id whose dataset was connected with datasets.from_volume trains: the file is downloaded to the training machine at job start. See Volumes.
checkpoint_destination other than {"type": "veri"} (another type, a uri, or credentials) is a 400; see Checkpoint destination schema.
hyperparameters.load_in_4bit: true without lora_rank is a 400 (QLoRA trains adapters on a frozen 4-bit base).
hyperparameters.lora_rank of 0 or below is a 400: set a positive rank (e.g. 16) for LoRA, or omit it (or send null) for a full fine-tune.
job_type=custom_script with provider: "vast" is a 400: custom scripts run on aws and digitalocean; Vast runs managed methods only.
A hyperparameter value of the wrong type (for example "lora_rank": "8" or "packing": "yes") is a 400 that names the key.
gpu.gpu_type must be one the provider offers for training: L4-24GB, A10G-24GB, A100-80GB, H100-80GB on aws; see GPU config schema
gpu.gpu_count must be a node shape the provider sells for that gpu_type (on aws: 1 or 4 for L4-24GB/A10G-24GB, 8 for A100-80GB/H100-80GB)

Errors: coded errors carry a JSON body {"error": {"code": "...", "message": "..."}}; uncoded ones carry {"error": {"message": "..."}}.

400 Bad Request: the job was rejected (invalid reward source, unknown or unavailable method or job_type, unsupported or unpriced GPU shape, provider or region, invalid or wrongly typed hyperparameters, an unsupported field (environments, volume, or a checkpoint_destination other than Veri storage), or a dataset the method can’t use: a row format it can’t train on, or one that reads from a volume). The message names the problem.
402 Payment Required: no billing account is linked to the workspace yet (see Billing).
404 Not Found: dataset_id doesn’t exist in your workspace.
422 Unprocessable Entity: the body is not valid JSON for this schema (for example a missing output_name). This body is plain text, not JSON.
429 Too Many Requests: code CONCURRENT_JOB_LIMIT when the workspace already has its maximum of in-flight jobs (4 by default; failed, completed and cancelled jobs don’t count). Wait for one to finish or cancel one, then resubmit. Code RATE_LIMITED (with a Retry-After header and retry_after_seconds) when the API key exceeds its request rate.
503 Service Unavailable: code billing_unavailable when the billing account could not be checked; retry after the Retry-After header (retry_after_seconds in the body).

Response (TrainingJobResponse, 201). hyperparameters echoes every field of the method’s schema (see the note under GRPOHyperparameters about echoed defaults):

{
  "object": "training_job",
  "id": "a1b2c3d4e5f6",
  "status": "queued",
  "base_model": "Qwen/Qwen2.5-1.5B-Instruct",
  "dataset_id": "9f2c81ab04de",
  "dataset_snapshot_id": null,
  "reward_count": 1,
  "method": "grpo",
  "hyperparameters": {"learning_rate": 2e-05, "max_steps": 200, "rollouts_per_prompt": 8, "max_response_length": 512, "lora_rank": 16, "lora_alpha": 32, ...},
  "base_image": null,
  "entrypoint": null,
  "output_name": "qwen-1.5b-gsm8k",
  "gpu": null,
  "gpu_requested": {"type": "L4-24GB", "count": 1},
  "provider": "aws",
  "region": null,
  "num_nodes": 1,
  "error": null,
  "dashboard_url": "https://veri.studio/training?job=a1b2c3d4e5f6",
  "download_url": null,
  "wandb_run_url": null,
  "hf_push": null,
  "hf_repo_url": null,
  "current_step": null,
  "total_steps": null,
  "current_loss": null,
  "cost_usd": null,
  "user_rate_per_hour_usd": 1.26,
  "cost_so_far_usd": null,
  "duration_seconds": null,
  "launched_at": null,
  "provision_started_at": null,
  "started_at": null,
  "completed_at": null,
  "terminated_at": null,
  "visibility": "private",
  "published_at": null,
  "created_at": "2026-05-12T10:23:45Z",
  "updated_at": "2026-05-12T10:23:45Z"
}

GET /v1/training_jobs

List your jobs, newest first. Filter by status (queued / provisioning / configuring / running / completed / failed / cancelled) or method (grpo / grpo_harness / sft_text / dpo / custom_script). limit is 1 to 100 (default 20); pass the last job’s id as after for the next page. An unknown status is a 400.

curl "https://api.veri.studio/v1/training_jobs?status=running&limit=10" \
  -H "Authorization: Bearer vk_..."

Response is PaginatedList[TrainingJobResponse]: {"object": "list", "data": [...], "has_more": false}.

GET /v1/training_jobs/{id}

Single job. Same shape as the create response, with the lifecycle timestamps, progress and cost filled in as the job moves. A running job, trimmed to those fields:

{
  "object": "training_job",
  "id": "a1b2c3d4e5f6",
  "status": "running",
  "provider": "aws",
  "current_step": 42,
  "total_steps": 200,
  "current_loss": 0.0123,
  "cost_usd": null,
  "user_rate_per_hour_usd": 1.26,
  "cost_so_far_usd": 0.39,
  "error": null,
  "created_at": "2026-05-12T10:23:45Z",
  "launched_at": "2026-05-12T10:24:02Z",
  "provision_started_at": "2026-05-12T10:24:02Z",
  "started_at": "2026-05-12T10:29:31Z",
  "completed_at": null,
  "terminated_at": null
}

A job is live while its status is queued, provisioning, configuring or running (these count toward the in-flight job limit); completed, failed and cancelled are final. configuring is the worker setting up the machine (dependencies, data, model); the setup events time each stage. Lifecycle and cost fields:

Field	Meaning
`created_at`	Submitted. The job is `queued` until `launched_at`.
`launched_at`	The provider returned the instance and `provisioning` began.
`provision_started_at`	Billing start: `cost_so_far_usd` and the settled `cost_usd` count from here. On AWS, DigitalOcean and GCP it is the launch (equal to `launched_at`); on Vast, which bills a box once it runs, it is set when the job reaches `configuring`.
`started_at`	The first time the job reached `running` (training started).
`completed_at`	When the job reached `completed`, `failed` or `cancelled`.
`terminated_at`	When the instance was released.
`user_rate_per_hour_usd`	The job’s price per hour: every GPU of every node at the rate it is billed. `null` when the shape has no price.
`cost_so_far_usd`	While the job is `provisioning`, `configuring` or `running`: `user_rate_per_hour_usd` times the hours since `provision_started_at`. `null` before billing starts and once the job has finished.
`cost_usd`	The settled cost of a finished job.

Timestamps of jobs created before a field existed stay null. Error and refund. error is null unless the job failed: {"code": "...", "message": "...", "attribution": "customer"}. code is one of the failure codes and attribution says who the failure belongs to (customer, platform or pending; absent on jobs that failed before it existed). A platform-attributed failure also carries refund: {"status": "credited", "amount_usd": 1.84}, with status one of approved, credited, pending or denied. refund is omitted otherwise.

GET /v1/training_jobs/{id}/metrics

Sampled training metrics for charting. Server-side stride downsampling if you ask for more than max_points. Metric names are the trainer’s own log keys (for GRPO: loss, reward, reward_std, entropy, completions/mean_length, and kl when kl_coef is above 0; see Reading the metrics). Query params:

keys: comma-separated metric names (e.g. loss,reward,reward_std). Omit for all available.
from_step, to_step: inclusive bounds
max_points: cap result size, default 500, max 5000

Response:

{
  "metrics": [
    {"step": 1, "values": {"loss": 0.012, "reward": 0.44, "reward_std": 0.31}},
    {"step": 10, "values": {"loss": 0.009, "reward": 0.62, "reward_std": 0.27}}
  ],
  "available_keys": ["completions/mean_length", "entropy", "grad_norm", "learning_rate", "loss", "reward", "reward_std"],
  "latest_step": 100
}

Every step the trainer logs is stored: the worker batches the steps logged since its last status report (it reports every 5 seconds) and sends them all, so the series has no gaps. max_points thins it on read.

Step timing and system keys

Next to the trainer’s own keys, the worker writes these per step. All are numbers; a key is absent when its value is unknown.

Key	Meaning
`time/ts`	Unix time in seconds when the step was logged. Differences between steps give wall time per step and an ETA.
`time/step_s`	Wall seconds for the step.
`time/gen_s`	Generation seconds: GRPO completions, or for `grpo_harness` the whole rollout phase including the sandboxes.
`time/reward_s`	Seconds in the reward functions, summed over all of them.
`time/weight_sync_s`	Seconds pushing the updated weights to the rollout server (`grpo_harness`).
`time/update_s`	The rest of the step: `time/step_s` minus generation, reward and weight sync (never below 0).
`time/checkpoint_save_s`	Seconds writing a checkpoint to local disk, on the steps that save one.
`sys/gpu.{i}.util_pct`	GPU utilization, percent.
`sys/gpu.{i}.mem_used_bytes`, `sys/gpu.{i}.mem_total_bytes`	GPU memory in use and in total.
`sys/gpu.{i}.mem_peak_bytes`	Peak memory PyTorch has allocated on the GPU during the run. On multi-GPU jobs it can be present for GPU 0 only.
`sys/gpu.{i}.power_w`, `sys/gpu.{i}.temp_c`, `sys/gpu.{i}.sm_clock_mhz`	Power draw (W), temperature (C) and SM clock (MHz).
`sys/gpu.{i}.throttle_mask`	Why the GPU’s clocks are held down, as the NVML bitmask: `0x1` idle, `0x2` application clocks, `0x4` software power cap, `0x8` hardware slowdown, `0x10` sync boost, `0x20` software thermal, `0x40` hardware thermal, `0x80` hardware power brake, `0x100` display clocks. `0` means not throttled.
`sys/gpu.{i}.ecc_volatile_uncorrected`	Uncorrected ECC memory errors since the driver loaded.
`sys/host.rss_bytes`, `sys/host.disk_free_bytes`, `sys/host.cpu_pct`	The worker host’s resident memory, free disk and CPU use (percent).

{i} is the GPU’s index on its machine, from 0. The time/ keys come from the managed methods’ trainers: time/gen_s and time/reward_s are GRPO only, and time/weight_sync_s is grpo_harness only. The sys/ values are the latest sample from a sampler that reads the machine every 15 seconds, so neighboring steps can carry the same sample; GPU values come from NVIDIA’s NVML, so hosts without it (AMD GPUs) have no sys/gpu keys.

GET /v1/training_jobs/{id}/events

The job’s lifecycle events: the launch, each setup stage, checkpoints, warnings, the outcome and billing. Each event has id, object ("training_job.event"), job_id, level (info / warning / error), event_type, message, metadata (an object or null), request_id, and created_at. Query params:

order: desc (newest first, the default) or asc (oldest first)
after: an event id. Only events created after that event are returned. An id that is not an event of this job is a 400.
limit: 1 to 500, default 50

has_more is true when more events match than the page holds. To read the whole history, or to poll a live job for new events, page forward with order=asc and the last id you have as after:

curl "https://api.veri.studio/v1/training_jobs/$JOB_ID/events?order=asc&limit=500" \
  -H "Authorization: Bearer vk_..."
# next page, or new events since the last poll:
curl "https://api.veri.studio/v1/training_jobs/$JOB_ID/events?order=asc&after=$LAST_EVENT_ID" \
  -H "Authorization: Bearer vk_..."

Response:

{
  "object": "list",
  "data": [
    {
      "id": "7d0e4b19a2c6",
      "object": "training_job.event",
      "job_id": "a1b2c3d4e5f6",
      "level": "info",
      "event_type": "checkpoint_committed",
      "message": "checkpoint ckpt_4f0b9c2e81a committed at step 100 (14 files, 0.45 GB, 6.1s, 74 MB/s)",
      "metadata": {"checkpoint_id": "ckpt_4f0b9c2e81a", "step": 100, "size_bytes": 452984832, "file_count": 14, "upload_seconds": 6.1, "mb_per_s": 74.26},
      "request_id": null,
      "created_at": "2026-05-12T10:41:12Z"
    }
  ],
  "has_more": false
}

The SDK pages with client.training_jobs.events(job_id, after=..., order="asc", limit=500) and the CLI with veri jobs events.

Event types

Setup, roughly in the order a job goes through it. All are info. waiting_for_provider_gpu, job_submitted, job_configuring and job_running come from Veri; the others come from the worker on the training machine.

`event_type`	When	`metadata`
`waiting_for_provider_gpu`	The job stays `queued` because the provider’s GPUs are all in use	`provider`, `cap`, `held`
`job_submitted`	The instance launched; the job is `provisioning`	`runner_id`, `provider`
`job_configuring`	The instance is up and the worker is setting up	none
`system_inventory`	The worker started: the machine it runs on	`gpus` (each `index`, `uuid`, `name`, `mem_total_bytes`), `driver`, `cuda`, `nccl_version` (or `null`), `host` (`cpu_count`, `ram_bytes`, `disk_free_bytes`), `instance_type` (or `null`)
`deps_installed`	The training dependencies are installed	`duration_s`, `installed` (package names)
`artifacts_downloaded`	The dataset and reward sources are on the machine	`duration_s`, `reward_count`, `dataset_bytes` (or `null`)
`model_download_started`	A managed job starts downloading a Hugging Face base model	`model`
`model_download_finished`	The base model is on disk	`model`, `duration_s`, `bytes` (or `null`), `cached` (`true` when it was already there)
`job_running`	Training started (`started_at`)	none
`model_loaded`	The trainer loaded the model	`duration_s`
`first_step`	Step 1 finished (sent once)	`step_s`

Training and outcome:

`event_type`	`level`	When	`metadata`
`resumed_from_checkpoint`	`info`	At submit, for a job created with `resume_from`	`source_job_id`, `checkpoint_id`, `step`
`ignored_hyperparameter_keys`	`warning`	At submit: hyperparameters the method does not read	`keys`
`checkpoint_saved`	`info`	An intermediate checkpoint was written to local disk	`step`, `local_save_s`
`checkpoint_committed`	`info`	That checkpoint finished uploading and is listed under the job’s checkpoints	`checkpoint_id`, `step`, `size_bytes`, `file_count`, `upload_seconds`, `mb_per_s`
`checkpoint_upload_skipped`	`warning`	A checkpoint was dropped because a newer one was saved while an older one was still uploading	`step`, `in_flight_step`, `newer_step`
`checkpoint_disabled_disk`	`warning`	Intermediate checkpoints were turned off because they do not fit on local disk (the final weights are still saved)	`checkpoint_bytes`, `copies`, `free_bytes`
`final_upload_started`	`info`	Training finished; the final weights started uploading	none
`final_upload_finished`	`info`	The final weights are stored	`duration_s`, `files`, `bytes`
`job_completed`	`info`	The job completed	none
`job_failed`	`error`	The job failed; the message is the error message	varies
`provisioning_timeout`, `configuring_timeout`, `heartbeat_timeout`, `max_runtime_exceeded`	`error`	A deadline stopped the job (see Failure modes)	`elapsed_seconds`, `deadline_seconds`
`job_submission_failed`	`error`	The launch failed	`error`
`job_cancelled`	`warning`	The job was cancelled	none

Billing, after the job ends:

`event_type`	`level`	When	`metadata`
`job_cost_attributed`	`info`	The job’s cost was computed	`duration_seconds`, `user_cost_usd`
`job_billed`	`info`	The cost was deducted from the workspace balance	`user_cost_usd`
`job_not_billed`	`info`	Cancelled before it reached `running`, so not billed	none
`job_refund_pending`	`warning`	A platform-attributed failure’s credit is under review	`amount_usd`
`job_refunded`	`info`	That credit was returned to the workspace balance	`amount_usd`

Unknown event_type values can appear as Veri adds events: treat the list as open.

GET /v1/training_jobs/{id}/logs

Streams the job’s log as server-sent events (text/event-stream): each data: event is one log line, starting with the instance’s boot log and then the worker’s output as it arrives. The stream ends with a final [Job completed], [Job failed], or [Job cancelled] event. A job that has not reached a GPU yet returns 400 (Job has not been submitted yet). For the full log after a job completes, use /logs/full instead.

GET /v1/training_jobs/{id}/logs/full

Returns the instance’s boot log followed by the worker’s stdout, as plain text. The worker uploads its log every 10 seconds while it runs and once more when it exits, so during a run the 200 body is at most about 10 seconds behind, and once the job has ended it is complete.

206 Partial Content before the first upload: the latest lines the worker reported, ending in a [partial ...] marker line, or only the boot log, ending in [provisioning ...].
404 with the plain-text body [no logs available yet] for a live job with no log yet.
404 with the JSON error code log_never_uploaded for a finished job whose log was never uploaded.

Use /logs to follow a live job and /logs/full to read a finished one.

GET /v1/training_jobs/{id}/files

The files a job was submitted with: its inline reward sources and, for custom_script and grpo_harness jobs, the code archive, the dependency file and the entrypoint. The script’s env is never returned. 404 for a job outside your workspace.

{
  "object": "list",
  "data": [
    {"name": "reward_0.py", "kind": "reward", "size_bytes": 412, "content": "def reward(completions, answer, **kwargs):\n    ...", "download_url": "https://...?X-Amz-..."},
    {"name": "code.tar.gz", "kind": "code_artifact", "size_bytes": 18233, "content": null, "download_url": "https://...?X-Amz-..."},
    {"name": "requirements.txt", "kind": "requirements", "size_bytes": 25, "content": "openai>=1.40\nhttpx>=0.27\n", "download_url": null},
    {"name": "entrypoint", "kind": "entrypoint", "size_bytes": 14, "content": "python main.py", "download_url": null}
  ]
}

`kind`	`name`	`content`	`download_url`
`reward`	`reward_0.py`, `reward_1.py`, … (one per reward source)	The source, when it is 64 KiB or smaller	A link valid for 15 minutes
`code_artifact`	`code.tar.gz`	`null`	A link valid for 15 minutes. Code archives expire 30 days after upload; after that `size_bytes` and `download_url` are `null`
`requirements`, `uv_lock`	`requirements.txt`, `uv.lock`	The file	`null`
`entrypoint`	`entrypoint`	The command the job runs	`null`

The SDK returns the list with client.training_jobs.files(job_id).

POST /v1/training_jobs/{id}/cancel

Cancel a non-terminal job. Cancellation is synchronous: Veri stops provisioning, terminates the instance, and returns the job with status: "cancelled". The worker gets no chance to flush, so anything not yet uploaded is lost. Cancelling a job that is already completed, failed, or cancelled is a 400.

DELETE /v1/training_jobs/{id}

Hard-delete the job row + cascading metrics + events + S3 artifacts (logs, status files, checkpoints). Returns {"id": "...", "object": "training_job", "deleted": true}. Constraints:

Job must be in a terminal state (completed, failed, cancelled)
No deployment may reference this job (source="training_job", source_job_id=this_id), whatever the deployment’s status, including stopped ones
No Hugging Face export of a model saved from this job may still be running (launching or uploading)

If a deployment or a running export still references the checkpoint, you get 409 Conflict naming it: delete the deployment, or wait for the export to end, first.

GET /v1/training_jobs/{id}/model

Presigned S3 download URLs for the final HF-format checkpoint, one per file. Only available after status=completed (400 before that).

{
  "download_url": "https://.../jobs/a1b2c3d4e5f6/final/hf/adapter_model.safetensors?X-Amz-...",
  "files": [
    {"path": "adapter_config.json", "url": "https://...?X-Amz-..."},
    {"path": "adapter_model.safetensors", "url": "https://...?X-Amz-..."},
    {"path": "tokenizer.json", "url": "https://...?X-Amz-..."}
  ],
  "artifact_type": "adapter",
  "merged_files": [
    {"path": "config.json", "url": "https://...?X-Amz-..."},
    {"path": "model.safetensors", "url": "https://...?X-Amz-..."},
    {"path": "tokenizer.json", "url": "https://...?X-Amz-..."}
  ]
}

A checkpoint is a directory: download every entry of files, keeping its path, to get something from_pretrained can load (the SDK’s job.download() does this). download_url points at the weights file alone. URLs expire in 1 hour; re-call to get fresh ones. artifact_type says what files holds: adapter for a job trained with lora_rank (a PEFT adapter plus tokenizer, loaded on top of the base model), full for a full fine-tune. For a LoRA job, merged_files lists the merged weights (adapter merged into the base model) that Veri keeps next to the adapter and that a deployment of the job serves; download them the same way for a standalone checkpoint. The field is absent for full fine-tunes and for LoRA jobs trained before merged weights were kept.

Hyperparameter schemas

GRPOHyperparameters (method=grpo)

See GRPO algorithm for the full walkthrough. Fields, with the value a job uses when the field is omitted:

learning_rate: float = 1e-6
num_epochs: int = 1            # not used by GRPO; the run length is max_steps
max_steps: int | None = None   # omitted = 100 steps (one prompt per step)
rollouts_per_prompt: int = 8
kl_coef: float = 0.0           # >= 0; omitted = 0.0: no KL penalty, no reference model (null or negative is a 400)
max_prompt_length: int = 1024  # only with use_unsloth; setting it without use_unsloth is a 400
max_response_length: int = 2048
# global_batch_size: not supported by GRPO; setting it is a 400
seed: int = 42                 # trainer random seed
system_prompt: str | None = None  # wraps each string prompt as a system + user chat
use_unsloth: bool = False
lora_rank: int | None = None   # set to enable LoRA (null = full fine-tune)
lora_alpha: int | None = None  # LoRA scaling; pairs with lora_rank
load_in_4bit: bool = False     # QLoRA: 4-bit base + LoRA (needs lora_rank, else 400; single GPU)

LoRA and QLoRA are supported on GRPO: see GRPO algorithm. The job object echoes kl_coef only when the job set it. It echoes max_prompt_length: 1024 for a job that omitted it; the trainer applies that only with use_unsloth.

AgenticGRPOHyperparameters (method=grpo_agentic)

Inherits all GRPO fields plus:

max_turns: int = 5
max_seq_len: int = 8192

This method is being rebuilt and is not currently available. For environment-based RL today, use a custom script.

Multi-GPU fields (method=sft_text, dpo)

Added to SFTTextHyperparameters and DPOHyperparameters for jobs with gpu_count above 1; see Multi-GPU training for semantics and validation. All optional; absent keys are not echoed back.

zero_stage: "auto" | int = "auto"       # 0 (data parallel), 2 or 3 (FSDP2 sharding); 1 is rejected
data_parallel_size: int | None = None   # must equal gpu_count in this release
tensor_parallel_size: int = 1           # 1 only in this release
context_parallel_size: int = 1          # 1 only in this release
pipeline_parallel_size: int = 1         # 1 only (not available in managed training)
expert_parallel_size: int = 1           # 1 only (not available in managed training)
micro_batch_size: int = 1               # samples per GPU per forward pass (alias: batch_size)
gradient_accumulation_steps: int = 1    # derived from global_batch_size when that is set
global_batch_size: int | None = None    # = micro_batch_size x gradient_accumulation_steps x data_parallel_size
gradient_checkpointing: bool = True
precision: str = "bf16"                 # "bf16" | "bf16_mixed" | "fp32"
cpu_offload: bool = False               # needs zero_stage 2 or 3

micro_batch_size, gradient_accumulation_steps, global_batch_size, gradient_checkpointing and precision are honoured on single-GPU jobs too; the sharding keys need gpu_count above 1.

GPU config schema

gpu = {"gpu_type": "L4-24GB", "gpu_count": 1}

Training shapes by provider (any other gpu_type or gpu_count is rejected at submit with a 400 that lists what the provider offers):

`provider`	`gpu_type`	`gpu_count`
`aws` (default)	`L4-24GB`, `A10G-24GB`	`1` or `4`
`aws` (default)	`A100-80GB`, `H100-80GB`	`8` (whole node)
`vast`	`A100-80GB`, `H100-80GB`	`1`, `2`, `4` or `8`
`digitalocean`	`H100-80GB`	`1` or `8`
`gcp`	`A100-80GB`, `H100-80GB`	`1`

AMD MI300X-192GB is not yet available for training. Run veri gpu list to see current availability and pricing for every shape.

Checkpoint destination schema

checkpoint_destination = {"type": "veri"}  # the default: Veri-managed storage

Checkpoints are always stored in Veri storage, so {"type": "veri"} (or omitting the field) is the only accepted value. Any other value is a 400 at submit: another type (such as s3 or gs), a uri, or credentials. The destination is not stored. Writing checkpoints directly to your own bucket is not supported yet. To get the weights out, download them (GET /v1/training_jobs/{id}/model, veri jobs download <job_id>, or job.download()), or push them to Hugging Face with hf_push.

Worker callbacks (appendix)

These exist for the runner agent and are NOT for users to call directly. Auth is per-job callback_token (a 32-byte random secret stored in the DB on job creation), not your API key.

POST /v1/training_jobs/{id}/status: worker reports progress, metrics, terminal state
POST /v1/training_jobs/{id}/presigned-upload: worker requests a presigned S3 PUT URL for checkpoint files
POST /v1/training_jobs/{id}/logs/presigned-upload: worker requests a presigned S3 PUT URL for the full stdout

These are documented for completeness; the worker SDK handles them.

Datasets

dataset_id setup before submitting.

Reward functions

Writing and attaching reward_source.

Billing

Billing states and balance checks.

Quickstart

Full SDK + CLI walkthrough.

Get started

Training

Deployments

Evaluations

Agents

CLI

Volumes

Webhooks

Secrets

Billing

Endpoints

POST /v1/training_jobs

GET /v1/training_jobs

GET /v1/training_jobs/{id}

GET /v1/training_jobs/{id}/metrics

Step timing and system keys

GET /v1/training_jobs/{id}/events

Event types

GET /v1/training_jobs/{id}/logs

GET /v1/training_jobs/{id}/logs/full

GET /v1/training_jobs/{id}/files

POST /v1/training_jobs/{id}/cancel

DELETE /v1/training_jobs/{id}

GET /v1/training_jobs/{id}/model

Hyperparameter schemas

GRPOHyperparameters (method=grpo)

AgenticGRPOHyperparameters (method=grpo_agentic)

Multi-GPU fields (method=sft_text, dpo)

GPU config schema

Checkpoint destination schema

Worker callbacks (appendix)

Datasets

Reward functions

Billing

Quickstart

​Endpoints

​POST /v1/training_jobs

​GET /v1/training_jobs

​GET /v1/training_jobs/{id}

​GET /v1/training_jobs/{id}/metrics

​Step timing and system keys

​GET /v1/training_jobs/{id}/events

​Event types

​GET /v1/training_jobs/{id}/logs

​GET /v1/training_jobs/{id}/logs/full

​GET /v1/training_jobs/{id}/files

​POST /v1/training_jobs/{id}/cancel

​DELETE /v1/training_jobs/{id}

​GET /v1/training_jobs/{id}/model

​Hyperparameter schemas

​GRPOHyperparameters (method=grpo)

​AgenticGRPOHyperparameters (method=grpo_agentic)

​Multi-GPU fields (method=sft_text, dpo)

​GPU config schema

​Checkpoint destination schema

​Worker callbacks (appendix)

​Related

Datasets

Reward functions

Billing

Quickstart

Endpoints

POST /v1/training_jobs

GET /v1/training_jobs

GET /v1/training_jobs/{id}

GET /v1/training_jobs/{id}/metrics

Step timing and system keys

GET /v1/training_jobs/{id}/events

Event types

GET /v1/training_jobs/{id}/logs

GET /v1/training_jobs/{id}/logs/full

GET /v1/training_jobs/{id}/files

POST /v1/training_jobs/{id}/cancel

DELETE /v1/training_jobs/{id}

GET /v1/training_jobs/{id}/model

Hyperparameter schemas

GRPOHyperparameters (method=grpo)

AgenticGRPOHyperparameters (method=grpo_agentic)

Multi-GPU fields (method=sft_text, dpo)

GPU config schema

Checkpoint destination schema

Worker callbacks (appendix)

Related