> ## Documentation Index > Fetch the complete documentation index at: https://docs.veri.studio/llms.txt > Use this file to discover all available pages before exploring further. # Training jobs API > Every endpoint for creating, monitoring, and managing training jobs. Reference for `/v1/training_jobs/*`. All endpoints require Bearer token auth (`Authorization: Bearer vk_...`) unless flagged worker-only. ## Endpoints | Method | Path | Purpose | | -------- | ---------------------------------- | --------------------------------------------------- | | `GET` | `/v1/training_jobs` | List jobs (paginated, filter by `status`, `method`) | | `POST` | `/v1/training_jobs` | Create a job | | `GET` | `/v1/training_jobs/{id}` | Get one | | `GET` | `/v1/training_jobs/{id}/logs` | Stream stdout tail (SSE-shaped) | | `GET` | `/v1/training_jobs/{id}/logs/full` | Full stdout from S3 (after job ends) | | `GET` | `/v1/training_jobs/{id}/events` | Audit trail | | `GET` | `/v1/training_jobs/{id}/metrics` | Sampled training metrics | | `POST` | `/v1/training_jobs/{id}/cancel` | Cancel a non-terminal job | | `DELETE` | `/v1/training_jobs/{id}` | Hard-delete a finished job (S3 + DB) | | `GET` | `/v1/training_jobs/{id}/model` | Presigned download URL for the checkpoint | Worker-only endpoints (used by the runner agent, authenticated via per-job `callback_token`): `POST /status`, `POST /presigned-upload`, `POST /logs/presigned-upload`. Documented in the worker-callbacks appendix at the bottom. ## POST /v1/training\_jobs Create a job. Returns immediately with status `queued`. **Request schema** (`TrainingJobCreate`): ```json theme={null} { "base_model": "Qwen/Qwen2.5-7B-Instruct", "dataset_id": "ds_abc123", "reward_function_id": "rf_xyz789", "method": "grpo", "hyperparameters": { "learning_rate": 1e-6, "rollouts_per_prompt": 8, "max_response_length": 2048, "kl_coef": 0.001 }, "output_name": "qwen-7b-math", "gpu": {"gpu_type": "H100-80GB", "gpu_count": 2}, "provider": "aws", "checkpoint_destination": {"type": "veri"} } ``` **Required:** `base_model`, `dataset_id`, `output_name`, `gpu`. **Required for `method=grpo`:** `reward_function_id`. **Validation rules:** * `method=grpo` → `reward_function_id` (required). `grpo_agentic` is being rebuilt and is not currently available. * `method=sft_text`, `dpo`, or `sft_video_gen` → `reward_function_id` must be `null` (these learn from the dataset) * `gpu.gpu_type` must be one of: `A100-80GB`, `H100-80GB`, `L4-24GB`, `A10G-24GB` * `gpu.gpu_count` in `[1, 8]` **Errors:** * `402 Payment Required` — credit balance too low for the requested resources (see [Billing](/deployments/cost)) * `404 Not Found` — `dataset_id` or `reward_function_id` doesn't exist or belongs to another user * `422 Unprocessable Entity` — schema validation failed **Response** (`TrainingJobResponse`, `201`): ```json theme={null} { "object": "training_job", "id": "tj_abc123", "status": "queued", "base_model": "Qwen/Qwen2.5-7B-Instruct", "dataset_id": "ds_abc123", "reward_function_id": "rf_xyz789", "method": "grpo", "hyperparameters": {...}, "output_name": "qwen-7b-math", "gpu_requested": {"type": "H100-80GB", "count": 2}, "gpu": null, "provider": null, "error": null, "dashboard_url": null, "download_url": null, "wandb_run_url": null, "current_step": null, "total_steps": null, "current_loss": null, "duration_seconds": null, "created_at": "2026-05-12T10:23:45Z", "updated_at": "2026-05-12T10:23:45Z", "started_at": null, "completed_at": null } ``` ## GET /v1/training\_jobs List your jobs. Filter by `status` (queued / provisioning / running / completed / failed / cancelled) or `method` (grpo / grpo\_agentic / sft\_text / dpo / sft\_video\_gen). ```bash theme={null} curl "https://api.veri.studio/v1/training_jobs?status=running&limit=10" \ -H "Authorization: Bearer vk_..." ``` Response is `PaginatedList[TrainingJobResponse]` with `data` + `has_more`. ## GET /v1/training\_jobs/{id} Single job. Same shape as the create response, but with timestamps populated as the job progresses. ## GET /v1/training\_jobs/{id}/metrics Sampled training metrics for charting. Server-side stride downsampling if you ask for more than `max_points`. **Query params:** * `keys` — comma-separated metric names (e.g. `loss,reward_mean,kl_div`). Omit for all available. * `from_step`, `to_step` — inclusive bounds * `max_points` — cap result size, default `500`, max `5000` **Response:** ```json theme={null} { "metrics": [ {"step": 0, "values": {"loss": 2.401, "reward_mean": -0.15}}, {"step": 10, "values": {"loss": 2.156, "reward_mean": 0.42}} ], "available_keys": ["loss", "reward_mean", "kl_div", "policy_entropy"], "latest_step": 100 } ``` ## GET /v1/training\_jobs/{id}/events Job lifecycle audit trail. Each event has timestamp, level (`info` / `warn` / `error`), event\_type, and metadata. ```bash theme={null} curl ".../v1/training_jobs/tj_abc/events?limit=50" -H "Authorization: Bearer vk_..." ``` ## GET /v1/training\_jobs/{id}/logs Tail of recent stdout from the worker. Returns chunked plain text. Falls back to the cached `log_tail` if the worker hasn't streamed anything fresh. For the **full log** after a job completes, use `/logs/full` instead. ## GET /v1/training\_jobs/{id}/logs/full Returns the complete stdout as plain text from S3 (uploaded \~30s during the run + a final flush at exit). Returns `206 Partial Content` if the upload hasn't happened yet (early-stage jobs); `404` if neither full log nor tail exists. ## POST /v1/training\_jobs/{id}/cancel Cancel a non-terminal job. The worker terminates at the next checkpoint boundary, not instantly — `status` may stay `running` for a few seconds before flipping to `cancelled`. ## DELETE /v1/training\_jobs/{id} Hard-delete the job row + cascading metrics + events + S3 artifacts (logs, status files, checkpoints). **Constraints:** * Job must be in a terminal state (`completed`, `failed`, `cancelled`) * No active deployments may reference this job (`source="training_job"`, `source_job_id=this_id`) If a deployment still uses the checkpoint, you get `400 Bad Request` — delete the deployment first. ## GET /v1/training\_jobs/{id}/model Presigned S3 download URL for the final HF-format checkpoint. Only available after `status=completed`. ```json theme={null} { "download_url": "https://veri-artifacts.s3.amazonaws.com/jobs/tj_abc/final/...?X-Amz-..." } ``` URL expires in 1 hour; re-call to get a fresh one. ## Hyperparameter schemas ### GRPOHyperparameters (method=grpo) See [GRPO algorithm](/training/grpo) for the full walkthrough. Schema fields with defaults: ```python theme={null} learning_rate: float = 1e-6 num_epochs: int = 1 max_steps: int | None = None rollouts_per_prompt: int = 8 kl_coef: float = 0.001 max_prompt_length: int = 1024 max_response_length: int = 2048 global_batch_size: int = 64 seed: int = 42 use_unsloth: bool = False lora_rank: int | None = None # set to enable LoRA (null = full fine-tune) lora_alpha: int | None = None # LoRA scaling; pairs with lora_rank load_in_4bit: bool = False # QLoRA: 4-bit base + LoRA (needs use_unsloth + lora_rank) ``` LoRA and QLoRA are supported on GRPO — see [GRPO algorithm](/training/grpo). ### AgenticGRPOHyperparameters (method=grpo\_agentic) Inherits all GRPO fields plus: ```python theme={null} max_turns: int = 5 max_seq_len: int = 8192 ``` This method is being rebuilt and is not currently available. For environment-based RL today, use a [custom script](/training/custom-script). ### SFTVideoGenHyperparameters (method=sft\_video\_gen) ```python theme={null} learning_rate: float = 1e-3 num_epochs: int = 30 max_steps: int | None = None lora_rank: int = 64 # required (LoRA SFT) lora_alpha: int = 64 resolution_height: int = 480 resolution_width: int = 720 num_frames: int = 49 fps: int = 8 batch_size: int = 1 gradient_accumulation_steps: int = 4 seed: int = 42 ``` ## GPU config schema ```python theme={null} gpu = {"gpu_type": "H100-80GB", "gpu_count": 2} ``` Supported `gpu_type` values: `A100-80GB`, `H100-80GB`, `L4-24GB`, `A10G-24GB`. `MI300X-192GB` (AMD, single GPU, `provider: "hotaisle"`) is coming soon, as are `A100-80GB` and `H100-80GB` in flexible 1-8 GPU shapes on `provider: "vast"`; H200 is roadmap. On AWS, `A100-80GB` and `H100-80GB` launch as 8-GPU nodes; `L4-24GB` and `A10G-24GB` are the single-GPU shapes. Run `veri gpu list` to see current availability and pricing for every shape. ## Checkpoint destination schema ```python theme={null} checkpoint_destination = {"type": "veri"} # default — keep on Veri-managed S3 # Or write directly to your bucket: checkpoint_destination = { "type": "s3", "uri": "s3://my-bucket/checkpoints/", } # Or GCS / Azure: checkpoint_destination = {"type": "gs", "uri": "gs://..."} checkpoint_destination = {"type": "az", "uri": "az://..."} ``` For `s3`, `gs`, `az`, you'll need credentials configured via account integrations (or pass them inline — schema accepts a `credentials` field). ## Worker callbacks (appendix) These exist for the runner agent and are NOT for users to call directly. Auth is per-job `callback_token` (a 32-byte random secret stored in the DB on job creation), not your API key. * `POST /v1/training_jobs/{id}/status` — worker reports progress, metrics, terminal state * `POST /v1/training_jobs/{id}/presigned-upload` — worker requests a presigned S3 PUT URL for checkpoint files * `POST /v1/training_jobs/{id}/logs/presigned-upload` — worker requests a presigned S3 PUT URL for the full stdout These are documented for completeness; the worker SDK handles them. ## Related `dataset_id` setup before submitting. `reward_function_id` setup. Billing states and balance checks. Full SDK + CLI walkthrough.