Training jobs API

Reference for /v1/training_jobs/*. All endpoints require Bearer token auth (Authorization: Bearer vk_...) unless flagged worker-only.

Endpoints

Method	Path	Purpose
`GET`	`/v1/training_jobs`	List jobs (paginated, filter by `status`, `method`)
`POST`	`/v1/training_jobs`	Create a job
`GET`	`/v1/training_jobs/{id}`	Get one
`GET`	`/v1/training_jobs/{id}/logs`	Stream stdout tail (SSE-shaped)
`GET`	`/v1/training_jobs/{id}/logs/full`	Full stdout from S3 (after job ends)
`GET`	`/v1/training_jobs/{id}/events`	Audit trail
`GET`	`/v1/training_jobs/{id}/metrics`	Sampled training metrics
`POST`	`/v1/training_jobs/{id}/cancel`	Cancel a non-terminal job
`DELETE`	`/v1/training_jobs/{id}`	Hard-delete a finished job (S3 + DB)
`GET`	`/v1/training_jobs/{id}/model`	Presigned download URL for the checkpoint

Worker-only endpoints (used by the runner agent, authenticated via per-job callback_token): POST /status, POST /presigned-upload, POST /logs/presigned-upload. Documented in the worker-callbacks appendix at the bottom.

POST /v1/training_jobs

Create a job. Returns immediately with status queued. Request schema (TrainingJobCreate):

{
  "base_model": "Qwen/Qwen2.5-7B-Instruct",
  "dataset_id": "ds_abc123",
  "reward_source": "def reward(completions, answer, **kwargs):\n    ...",
  "method": "grpo",
  "hyperparameters": {
    "learning_rate": 1e-6,
    "rollouts_per_prompt": 8,
    "max_response_length": 2048,
    "kl_coef": 0.001
  },
  "output_name": "qwen-7b-math",
  "gpu": {"gpu_type": "H100-80GB", "gpu_count": 2},
  "provider": "aws",
  "checkpoint_destination": {"type": "veri"}
}

Required: base_model, dataset_id, output_name, gpu. Required for method=grpo and grpo_harness: reward_source (or reward_sources, or environments). dataset_id also accepts dataset stream references: a stream’s name or id with @latest or @snap-N (for example acme-feedback@latest). The job resolves and stores the pinned snapshot as dataset_snapshot_id in its response, so later appends never change what the job trains on. Reward fields: reward_source is the Python source text of a TRL-signature reward function. For multiple weighted rewards, pass reward_sources (a list) plus optional reward_weights (same length). reward_source/reward_sources and environments are mutually exclusive. Limits: at most 8 sources, 256 KiB each, and each source must define a function; violations are HTTP 400 at submit. See Reward functions. Validation rules:

method=grpo or grpo_harness → reward required (reward_source, reward_sources, or environments). grpo_agentic is being rebuilt and is not currently available.
method=sft_text, dpo, sft_video_gen, or custom_script → reward fields must be omitted (these learn from the dataset)
gpu.gpu_type must be one of: A100-80GB, H100-80GB, L4-24GB, A10G-24GB
gpu.gpu_count in [1, 8]

Errors:

400 Bad Request: invalid reward source (too many, too large, or doesn’t define a function)
402 Payment Required — credit balance too low for the requested resources (see Billing)
404 Not Found — dataset_id doesn’t exist or belongs to another user
422 Unprocessable Entity — schema validation failed

Response (TrainingJobResponse, 201):

{
  "object": "training_job",
  "id": "tj_abc123",
  "status": "queued",
  "base_model": "Qwen/Qwen2.5-7B-Instruct",
  "dataset_id": "ds_abc123",
  "reward_count": 1,
  "method": "grpo",
  "hyperparameters": {...},
  "output_name": "qwen-7b-math",
  "gpu_requested": {"type": "H100-80GB", "count": 2},
  "gpu": null,
  "provider": null,
  "error": null,
  "dashboard_url": null,
  "download_url": null,
  "wandb_run_url": null,
  "current_step": null,
  "total_steps": null,
  "current_loss": null,
  "duration_seconds": null,
  "created_at": "2026-05-12T10:23:45Z",
  "updated_at": "2026-05-12T10:23:45Z",
  "started_at": null,
  "completed_at": null
}

GET /v1/training_jobs

List your jobs. Filter by status (queued / provisioning / running / completed / failed / cancelled) or method (grpo / grpo_harness / sft_text / dpo / sft_video_gen).

curl "https://api.veri.studio/v1/training_jobs?status=running&limit=10" \
  -H "Authorization: Bearer vk_..."

Response is PaginatedList[TrainingJobResponse] with data + has_more.

GET /v1/training_jobs/

Single job. Same shape as the create response, but with timestamps populated as the job progresses.

GET /v1/training_jobs//metrics

Sampled training metrics for charting. Server-side stride downsampling if you ask for more than max_points. Query params:

keys — comma-separated metric names (e.g. loss,reward_mean,kl_div). Omit for all available.
from_step, to_step — inclusive bounds
max_points — cap result size, default 500, max 5000

Response:

{
  "metrics": [
    {"step": 0, "values": {"loss": 2.401, "reward_mean": -0.15}},
    {"step": 10, "values": {"loss": 2.156, "reward_mean": 0.42}}
  ],
  "available_keys": ["loss", "reward_mean", "kl_div", "policy_entropy"],
  "latest_step": 100
}

GET /v1/training_jobs//events

Job lifecycle audit trail. Each event has timestamp, level (info / warn / error), event_type, and metadata.

curl ".../v1/training_jobs/tj_abc/events?limit=50" -H "Authorization: Bearer vk_..."

GET /v1/training_jobs//logs

Tail of recent stdout from the worker. Returns chunked plain text. Falls back to the cached log_tail if the worker hasn’t streamed anything fresh. For the full log after a job completes, use /logs/full instead.

GET /v1/training_jobs//logs/full

Returns the complete stdout as plain text from S3 (uploaded ~30s during the run + a final flush at exit). Returns 206 Partial Content if the upload hasn’t happened yet (early-stage jobs); 404 if neither full log nor tail exists.

POST /v1/training_jobs//cancel

Cancel a non-terminal job. The worker terminates at the next checkpoint boundary, not instantly — status may stay running for a few seconds before flipping to cancelled.

DELETE /v1/training_jobs/

Hard-delete the job row + cascading metrics + events + S3 artifacts (logs, status files, checkpoints). Constraints:

Job must be in a terminal state (completed, failed, cancelled)
No active deployments may reference this job (source="training_job", source_job_id=this_id)

If a deployment still uses the checkpoint, you get 400 Bad Request — delete the deployment first.

GET /v1/training_jobs//model

Presigned S3 download URL for the final HF-format checkpoint. Only available after status=completed.

{ "download_url": "https://veri-artifacts.s3.amazonaws.com/jobs/tj_abc/final/...?X-Amz-..." }

URL expires in 1 hour; re-call to get a fresh one.

Hyperparameter schemas

GRPOHyperparameters (method=grpo)

See GRPO algorithm for the full walkthrough. Schema fields with defaults:

learning_rate: float = 1e-6
num_epochs: int = 1
max_steps: int | None = None
rollouts_per_prompt: int = 8
kl_coef: float = 0.001
max_prompt_length: int = 1024
max_response_length: int = 2048
global_batch_size: int = 64
seed: int = 42
use_unsloth: bool = False
lora_rank: int | None = None   # set to enable LoRA (null = full fine-tune)
lora_alpha: int | None = None  # LoRA scaling; pairs with lora_rank
load_in_4bit: bool = False     # QLoRA: 4-bit base + LoRA (needs use_unsloth + lora_rank)

LoRA and QLoRA are supported on GRPO — see GRPO algorithm.

AgenticGRPOHyperparameters (method=grpo_agentic)

Inherits all GRPO fields plus:

max_turns: int = 5
max_seq_len: int = 8192

This method is being rebuilt and is not currently available. For environment-based RL today, use a custom script.

SFTVideoGenHyperparameters (method=sft_video_gen)

learning_rate: float = 1e-3
num_epochs: int = 30
max_steps: int | None = None
lora_rank: int = 64       # required (LoRA SFT)
lora_alpha: int = 64
resolution_height: int = 480
resolution_width: int = 720
num_frames: int = 49
fps: int = 8
batch_size: int = 1
gradient_accumulation_steps: int = 4
seed: int = 42

GPU config schema

gpu = {"gpu_type": "H100-80GB", "gpu_count": 2}

Supported gpu_type values: A100-80GB, H100-80GB, L4-24GB, A10G-24GB. AMD MI300X-192GB is coming soon. On AWS, A100-80GB and H100-80GB launch as 8-GPU nodes; L4-24GB and A10G-24GB are the single-GPU shapes. On vast, A100-80GB and H100-80GB rent as flexible 1-8 GPU chunks; on digitalocean, H100-80GB comes in fixed 1x and 8x droplets. Run veri gpu list to see current availability and pricing for every shape.

Checkpoint destination schema

checkpoint_destination = {"type": "veri"}  # default — keep on Veri-managed S3

# Or write directly to your bucket:
checkpoint_destination = {
    "type": "s3",
    "uri": "s3://my-bucket/checkpoints/",
}

# Or GCS / Azure:
checkpoint_destination = {"type": "gs", "uri": "gs://..."}
checkpoint_destination = {"type": "az", "uri": "az://..."}

For s3, gs, az, you’ll need credentials configured via account integrations (or pass them inline — schema accepts a credentials field).

Worker callbacks (appendix)

These exist for the runner agent and are NOT for users to call directly. Auth is per-job callback_token (a 32-byte random secret stored in the DB on job creation), not your API key.

POST /v1/training_jobs/{id}/status — worker reports progress, metrics, terminal state
POST /v1/training_jobs/{id}/presigned-upload — worker requests a presigned S3 PUT URL for checkpoint files
POST /v1/training_jobs/{id}/logs/presigned-upload — worker requests a presigned S3 PUT URL for the full stdout

These are documented for completeness; the worker SDK handles them.

Datasets

dataset_id setup before submitting.

Reward functions

Writing and attaching reward_source.

Billing

Billing states and balance checks.

Quickstart

Full SDK + CLI walkthrough.

Get started

Training

Deployments

CLI

Volumes

Webhooks

Endpoints

POST /v1/training_jobs

GET /v1/training_jobs

GET /v1/training_jobs/

GET /v1/training_jobs//metrics

GET /v1/training_jobs//events

GET /v1/training_jobs//logs

GET /v1/training_jobs//logs/full

POST /v1/training_jobs//cancel

DELETE /v1/training_jobs/

GET /v1/training_jobs//model

Hyperparameter schemas

GRPOHyperparameters (method=grpo)

AgenticGRPOHyperparameters (method=grpo_agentic)

SFTVideoGenHyperparameters (method=sft_video_gen)

GPU config schema

Checkpoint destination schema

Worker callbacks (appendix)

Datasets

Reward functions

Billing

Quickstart

​Endpoints

​POST /v1/training_jobs

​GET /v1/training_jobs

​GET /v1/training_jobs/

​GET /v1/training_jobs//metrics

​GET /v1/training_jobs//events

​GET /v1/training_jobs//logs

​GET /v1/training_jobs//logs/full

​POST /v1/training_jobs//cancel

​DELETE /v1/training_jobs/

​GET /v1/training_jobs//model

​Hyperparameter schemas

​GRPOHyperparameters (method=grpo)

​AgenticGRPOHyperparameters (method=grpo_agentic)

​SFTVideoGenHyperparameters (method=sft_video_gen)

​GPU config schema

​Checkpoint destination schema

​Worker callbacks (appendix)

​Related

Datasets

Reward functions

Billing

Quickstart

Endpoints

POST /v1/training_jobs

GET /v1/training_jobs

GET /v1/training_jobs/

GET /v1/training_jobs//metrics

GET /v1/training_jobs//events

GET /v1/training_jobs//logs

GET /v1/training_jobs//logs/full

POST /v1/training_jobs//cancel

DELETE /v1/training_jobs/

GET /v1/training_jobs//model

Hyperparameter schemas

GRPOHyperparameters (method=grpo)

AgenticGRPOHyperparameters (method=grpo_agentic)

SFTVideoGenHyperparameters (method=sft_video_gen)

GPU config schema

Checkpoint destination schema

Worker callbacks (appendix)

Related