veri deployments

The veri deployments group creates and manages inference endpoints. Each deployment exposes an OpenAI-compatible /chat/completions API backed by a Veri-trained or HuggingFace model.

Deploy a Hugging Face model in one step

veri deploy is the shortest path from a Hugging Face repo, or a finished training job, to a live endpoint. It accepts a bare repo id, a pasted huggingface.co link, or a completed job id:

veri deploy Qwen/Qwen3-4B
# or
veri deploy https://huggingface.co/Qwen/Qwen3-4B
# or a completed training job (full fine-tune or LoRA)
veri deploy a1b2c3d4e5f6 --gpu-type L4-24GB

A job trained with lora_rank deploys the same way: Veri serves the merged weights it keeps next to the adapter. A job id skips the cache prompt below (the weights already live in Veri storage). In a terminal, a Hugging Face deploy asks whether to cache the model in Veri storage for faster starts next time. Caching saves the downloaded weights to your model library in the background while the deployment serves; once the cached copy is ready, every later deploy of the same repo starts from Veri storage automatically instead of pulling from the Hugging Face hub.

Flag	Purpose
`--name`	Endpoint name. Default: derived from the repo name (`qwen3-4b`).
`--gpu-type`	GPU shape. Default `L4-24GB`.
`--gpu-count`	GPUs per replica. Default `1`.
`--provider`	Compute provider. Default: chosen automatically.
`--engine`	`vllm` (the default and only serving engine).
`--replicas`	GPU-box replicas behind the endpoint. Default `1`.
`--cache` / `--no-cache`	Cache the model in your library (or don’t) without prompting.
`--yes` / `-y`	Skip the prompt. No caching unless `--cache` is passed.
`--vllm-arg`	Extra vLLM CLI token appended to the engine command; repeat per token. Same rules as on `deployments create`.
`--vllm-image`	Explicit vLLM serving image for architectures newer than the platform vLLM, e.g. `vllm/vllm-openai:v0.31.0`.
`--trace-bodies` / `--no-trace-bodies`	Capture request/response bodies as browsable traces. Off by default.
`--trace-retention`	`base` (14 days, default) or `extended` (400 days). Requires `--trace-bodies`.

Outside a terminal (scripts, CI) the prompt is skipped and caching stays off unless --cache is explicit. veri deploy is sugar for veri deployments create --from-hf (a repo) or --model (a job id); every other lifecycle command below applies to the resulting deployment.

Commands

Command	Purpose
`create <config>` or inline	Bring up a deployment.
`list`	List deployments.
`get <id>`	Show one deployment.
`update <id>`	Change replica bounds / scaling settings on a live deployment.
`wake <id>`	Pre-warm a scaled-to-zero deployment.
`stop <id>`	Stop for good (terminal: cannot be resumed). Billing stops.
`delete <id>`	Permanently remove a stopped deployment and its request history.
`chat <id>`	Send a one-shot message for smoke testing.
`bench <id>`	Load-test the endpoint; report TTFB/TTFT/ITL/TPS/e2e at p50/p90/p99.
`metrics <id>`	RPS, latency, GPU utilization.
`requests <id>`	Recent inference requests.
`traces <id>`	Captured request traces (deployments created with `--trace-bodies`).
`trace <id> <trace-id>`	One trace with its full request/response bodies.

Create from a config

veri deployments create configs/deploy.toml
# or, via the hero verb:
veri run configs/deploy.toml

The config must declare kind = "deploy". See the config schema. Flags passed alongside a config override the file’s values the same way --set does (the flag wins). --from-hf <repo> and --from-model <model-id> replace deployment.model and pin deployment.source (huggingface / custom_model); --gpu-count N replaces resources.gpu_count (GPUs per replica, default 1); --name, --gpu-type, and the replica and scaling flags below work the same way.

veri deployments create configs/deploy.toml --from-hf Qwen/Qwen3-4B --gpu-count 4

Create inline

Two mutually exclusive modes — deploy a training job’s output, or deploy a HuggingFace model directly.

# From a finished training job:
veri deployments create \
  --model 3f9c2a7b1d4e \
  --name my-endpoint \
  --gpu-type L4-24GB \
  --gpu-count 1

# From a saved model in your library:
veri deployments create \
  --from-model 6a1f3c9e2b70 \
  --name support-prod \
  --gpu-type L4-24GB \
  --gpu-count 1

# Directly from HuggingFace (no training):
veri deployments create \
  --from-hf Qwen/Qwen3-4B \
  --name qwen-prod \
  --gpu-type L4-24GB \
  --gpu-count 1

Flag	Purpose
`--model <job-id>`	Deploy a training job’s checkpoint.
`--from-model <model-id>`	Deploy a saved custom model.
`--from-hf <repo>`	Deploy a HuggingFace repo.
`--name`	Display name.
`--gpu-type`	Required in inline form.
`--gpu-count`	GPUs per replica. Default `1`.
`--replicas`	GPU-box replicas behind one endpoint (1-8). Default `1`. Shorthand for equal `--min-replicas`/`--max-replicas`.
`--min-replicas`	Lower replica bound. Below `--max-replicas` enables autoscaling; `0` enables scale to zero.
`--max-replicas`	Upper replica bound (1-8).
`--concurrency-target`	In-flight requests per replica the autoscaler aims for. Default `8`.
`--scale-to-zero-window`	Idle seconds before a `--min-replicas 0` deployment parks. Default `3600`, minimum `300`.
`--idle-delete-after`	Auto-stop a parked deployment after N idle days. Default: keep it wakeable forever.
`--self-heal-attempts`	How many times Veri replaces the GPU box when the deployment’s last box dies after it has served, `0` to `5`. Default `3`. See Self-healing. Also available as `resources.self_heal_attempts` in a deploy config.
`--cache` / `--no-cache`	With `--from-hf` only: also save the downloaded model to your library so later deploys of the repo start from Veri storage.
`--vllm-arg`	Extra vLLM CLI token appended verbatim to the engine command; repeat per token (e.g. `--vllm-arg=--kv-cache-dtype --vllm-arg=fp8`). vllm engine only; only the accepted vLLM flags are allowed, anything else returns `400` (`invalid_vllm_extra_args`). Also available as `deployment.vllm_extra_args` (list of strings) in a deploy config.
`--vllm-image`	Explicit vLLM serving image for architectures newer than the platform vLLM, e.g. `vllm/vllm-openai:v0.31.0`. Official upstream images only (`vllm/vllm-openai[-rocm]:<tag>`); vllm engine only. Also available as `deployment.vllm_image` in a deploy config. Default: platform vLLM.
`--trace-bodies` / `--no-trace-bodies`	Capture this deployment’s request/response bodies as browsable traces. Off by default: enabling stores your end users’ prompt/completion content. Also available as `deployment.trace_bodies` in a deploy config.
`--trace-retention`	Trace retention class: `base` (14 days, default) or `extended` (400 days, extended rate). Requires `--trace-bodies`. Also available as `deployment.trace_retention`.

See Replicas, autoscaling, and scale to zero for how the bounds behave. A workspace can run 2 active deployment replicas on the Free plan (4 on Pro, see Plans): a create, wake or update that would boot more fails with 429 (ACTIVE_DEPLOYMENT_LIMIT).

Smoke-test with `chat`

The fastest way to confirm a deployment is healthy:

veri deployments chat 5b0e7c2d9a14 --message "What is 2+2?"

The CLI calls the OpenAI-compatible endpoint with a single user message and prints the assistant’s reply.

Flag	Purpose
`--message` / `-m`	Required. The user message.
`--model`	Override the model name in the request body. Defaults to the deployment’s name.
`--temperature`	Sampling temperature. Default `1.0`.
`--max-tokens`	Cap on response tokens.
`--session-id`	Pin this conversation to one replica for cache-warm affinity (sets the `X-Veri-Session-Id` header). Only affects multi-replica deployments. Experimental.

For programmatic chat, use https://api.veri.studio/v1/deployments/{id} as the OpenAI-compatible base_url with your Veri API key. The deployment’s endpoint_url (visible in veri deployments get) is that same platform URL with /chat/completions already appended, for direct HTTP calls.

Replicas, autoscaling, and scale to zero

One endpoint can be served by several GPU boxes behind a load-balancing router. The replica bounds decide how the fleet behaves:

Bounds	Behavior
`--replicas N` (or equal min/max)	Fixed fleet of N boxes.
`--min-replicas` below `--max-replicas`	Autoscaling: the fleet grows and shrinks with load.
`--min-replicas 0`	Scale to zero: the fleet also parks entirely when idle and wakes on demand.

# Fixed fleet of 2:
veri deployments create \
  --from-hf Qwen/Qwen3-4B --name qwen-prod \
  --gpu-type L4-24GB --gpu-count 1 \
  --replicas 2

# Autoscale 1-4 boxes, park after 30 idle minutes:
veri deployments create \
  --from-hf Qwen/Qwen3-4B --name qwen-dev \
  --gpu-type L4-24GB --gpu-count 1 \
  --min-replicas 0 --max-replicas 4 \
  --scale-to-zero-window 1800

The deployment reports serving once at least one replica is up. list and get show the fleet as ready/desired (for example 2/3 while a scale-up is still provisioning). When some replicas of a fleet die, the deployment keeps serving on the rest at reduced capacity while Veri launches replacements; replacements back off (the first at once, then 10, 20, 40 and 80 seconds after the previous) and pause after 5 in a row until you update the deployment. If the last box dies after the deployment has served, it shows provisioning (with failure_code replica_lost) while Veri replaces it, up to --self-heal-attempts times (default 3); once those are used up, a --min-replicas 0 deployment parks and any other deployment fails. See Self-healing.

Autoscaling

With min-replicas below max-replicas, the fleet targets --concurrency-target in-flight requests per replica (default 8). Scale-up is fast (within about a minute of sustained load); scale-down is deliberately slow (about 15 minutes of sustained low load per step) because a wrong scale-down costs a multi-minute GPU boot. Lower the target for latency-sensitive chat; raise it for batch throughput. When scaling down, replicas that are still loading (not yet serving) are removed first; a serving replica is only drained once there is no loading replica left to cancel.

Scale to zero

With --min-replicas 0, a deployment idle past --scale-to-zero-window (default 1 hour) parks: replicas terminate, billing stops, status becomes scaled_to_zero. The idle window counts from when the deployment starts serving or from its last request (not from create or wake), and a park waits for in-flight requests to finish and for a fresh engine-metrics sample before terminating anything. Turning scale-to-zero on, or changing the window, starts a fresh idle window. The next chat request wakes it. While the GPU boots (a few minutes), the server answers that request with 503, a Retry-After header, and error code deployment_waking. The SDK and CLI absorb this for you: veri deployments chat prints a “waking from zero” note and waits for the answer, and the Python SDK’s chat() retries in the background until the deployment is up (see chat and waking for the knobs). Only raw HTTP clients need to handle the 503 themselves. To hide the cold start from your users, pre-warm on a predictive signal instead:

veri deployments wake 5b0e7c2d9a14   # 202; no-op if already up

Then poll veri deployments get 5b0e7c2d9a14 until it’s serving. Raising the floor also wakes it: veri deployments update 5b0e7c2d9a14 --min-replicas 1 on a parked deployment wakes it immediately. A wake (explicit, request-triggered, or from raising --min-replicas) relaunches min_replicas boxes (at least one; the autoscaler grows the fleet again if traffic returns) and is subject to the workspace active-replica limit, like a create: with no headroom it returns 429 with code ACTIVE_DEPLOYMENT_LIMIT and the deployment stays parked, so stop or scale down something else first. Optionally set --idle-delete-after N to auto-stop a deployment that stays parked with no traffic for N days. A wake that fails (for example no GPU capacity) puts the deployment back to scaled_to_zero with the reason in its error, and the next wake waits out a backoff (1 minute, doubling up to 1 hour). With --min-replicas 0 the next request after the backoff retries it; with --min-replicas 1 or more Veri retries the wake itself once the backoff has passed, so the floor comes back without traffic (and as soon as there is room if the workspace active-replica limit is full). After 3 failed wakes in a row Veri stops retrying on its own and says so in error. veri deployments wake retries immediately whatever the count, and so does veri deployments update with --min-replicas of 1 or more.

Change scaling on a live deployment

veri deployments update 5b0e7c2d9a14 --replicas 2              # fixed fleet of 2
veri deployments update 5b0e7c2d9a14 --min-replicas 1 --max-replicas 6
veri deployments update 5b0e7c2d9a14 --concurrency-target 4
veri deployments update 5b0e7c2d9a14 --min-replicas 1          # parked: wakes it now
veri deployments update 5b0e7c2d9a14 --self-heal-attempts 5    # replace a lost last box up to 5 times
veri deployments update 5b0e7c2d9a14 --clear-scale-to-zero-window  # idle window back to the default (3600s)
veri deployments update 5b0e7c2d9a14 --clear-idle-delete-after  # never auto-stop while parked

update returns immediately with the new desired state; the fleet converges in the background (watch the replicas column approach desired/desired). Model, GPU, and provider are immutable: recreate the deployment to change those. A flag you leave out keeps its current setting; --clear-scale-to-zero-window and --clear-idle-delete-after put those two settings back to their defaults (each cannot be combined with its value flag). Clearing the window on a --min-replicas 0 deployment starts a fresh idle window, like changing it.

Parked (scaled_to_zero): raising --min-replicas to 1 or more wakes the deployment immediately and boots the new floor, like veri deployments wake. It is checked against the workspace active-replica limit first: if the new floor does not fit, update fails with 429 (ACTIVE_DEPLOYMENT_LIMIT) and nothing changes. Other settings are saved and apply from the next wake.
First boot (provisioning): scaling changes apply once the deployment is serving; extra replicas launch after the first one serves. update prints a note saying so.
waking: no second wake; the new settings apply to the boot in progress, or once it serves.
--self-heal-attempts is not a boot setting: it applies at once in every status, the next time the deployment’s last box dies while serving (and any update restarts the replacement count). An update that only changes it prints that instead of the boot note.

Session affinity

With --session-id <key> (or the OpenAI user field, which sets the X-Veri-Session-Id header), a conversation is pinned to one replica so its KV cache stays warm across turns. The same key always lands on the same replica; if that replica is unavailable, the request fails over to another.
Without a session key, requests go to the least-loaded replica.

veri deployments chat 5b0e7c2d9a14 -m "Continue our story" --session-id chat-42

Reuse the same --session-id for every turn of a conversation to keep it on its cache-warm replica.

Stop to end billing

veri deployments stop 5b0e7c2d9a14

stop ends billing and is final: a stopped deployment can’t wake or resume; bring it back by creating a new one. Running stop again on an already stopped (or failed) deployment is a no-op that returns the deployment unchanged. stop is not a pause. For a deployment you’ll come back to, prefer --min-replicas 0 and let it park instead. Once a deployment is stopped (or failed), veri deployments delete <id> removes it and its request history for good. Deletion waits for the final billing settlement, so if you delete within a minute of stopping you may be asked to retry shortly.

start, rollback, and history are Phase-2. Each depends on backend work that is not yet shipped.

Inspect traffic

veri deployments metrics 5b0e7c2d9a14

Returns lifetime counters (total_requests, total_prompt_tokens, total_completion_tokens, uptime_seconds) and past-24-hour stats (avg_latency_ms, latency percentiles, error_rate); eval calls are excluded. JSON output is dict-shaped (suitable for jq):

veri deployments metrics 5b0e7c2d9a14 --format json | jq .avg_latency_ms

Recent requests:

veri deployments requests 5b0e7c2d9a14 --limit 50
veri deployments requests 5b0e7c2d9a14 --after 7c41e09ab2d3 --limit 100   # pagination

Request traces (deployments created with --trace-bodies — see the traces guide):

veri deployments traces 5b0e7c2d9a14 --page 1 --limit 20
veri deployments trace 5b0e7c2d9a14 veri-req000000001    # full request/response bodies

List, get

list shows your deployments, most recent first. Filter and paginate with the flags below.

veri deployments list                              # all of them
veri deployments list --status serving             # only the live ones
veri deployments list --status serving --format json
veri deployments get 5b0e7c2d9a14                  # one deployment, with its status

Flag	Purpose
`--status`	Filter by status: `queued`, `provisioning`, `serving`, `scaled_to_zero`, `waking`, `stopped`, or `failed`.
`--limit`	Maximum number to return. Default `20`.
`--after <id>`	Pagination cursor: return deployments created after this ID.

A deployment holds GPU boxes while it is provisioning or serving. A queued or waking deployment moves to provisioning as soon as its first box launches, and a scaled_to_zero one holds none. To catch everything holding or about to hold a GPU, list without --status and scan the status column. The table (and --format csv) shows id, name, status, replicas, model, endpoint_url, cost_per_hour_usd and self_heal_attempts. With --format json, list and get also return the scaling settings (min_replicas, max_replicas, concurrency_target, scale_to_zero_window_seconds and idle_delete_after_days, where null means the default: a 3600-second idle window, never auto-stop), failure_code, and the wake state of a scale-to-zero deployment: last_wake_at, wake_failure_count and next_wake_not_before (see the deployment object). When wakes have been failing, the table output adds a line under the table:

Wake: 2 failed wake boots in a row (last wake started 2026-10-03T10:00:00Z); a request wakes it again from 2026-10-03T10:04:00Z, `veri deployments wake 5b0e7c2d9a14` retries now.

Exit codes

Code	Meaning
`0`	Success.
`1`	User error (both `--model` and `--from-hf` passed, missing `--gpu-type` inline).
`2`	API error. The server’s message is printed as one `Error:` line on stderr, ending in the error code when the server sent one: `Error: self_heal_attempts must be between 0 and 5 (got 9); 0 turns self-healing off (code: invalid_self_heal_attempts)`.
`4`	Auth error.

Get started

Training

Deployments

Evaluations

Agents

CLI

Volumes

Webhooks

Secrets

Billing

Deploy a Hugging Face model in one step

Commands

Create from a config

Create inline

Smoke-test with `chat`

Replicas, autoscaling, and scale to zero

Autoscaling

Scale to zero

Change scaling on a live deployment

Session affinity

Stop to end billing

Inspect traffic

List, get

Exit codes

What’s next

Deploy quickstart demo

The run verb

​Deploy a Hugging Face model in one step

​Commands

​Create from a config

​Create inline

​Smoke-test with chat

​Replicas, autoscaling, and scale to zero

​Autoscaling

​Scale to zero

​Change scaling on a live deployment

​Session affinity

​Stop to end billing

​Inspect traffic

​List, get

​Exit codes

​What’s next

Deploy quickstart demo

The run verb

Deploy a Hugging Face model in one step

Commands

Create from a config

Create inline

Smoke-test with `chat`

Replicas, autoscaling, and scale to zero

Autoscaling

Scale to zero

Change scaling on a live deployment

Session affinity

Stop to end billing

Inspect traffic

List, get

Exit codes

What’s next