veri deployments

The veri deployments group creates and manages inference endpoints. Each deployment exposes an OpenAI-compatible /chat/completions API backed by a Veri-trained or HuggingFace model.

Commands

Command	Purpose
`create <config>` or inline	Bring up a deployment.
`list`	List deployments.
`get <id>`	Show one deployment.
`update <id>`	Change replica bounds / scaling settings on a live deployment.
`wake <id>`	Pre-warm a scaled-to-zero deployment.
`stop <id>`	Stop for good. Billing stops.
`chat <id>`	Send a one-shot message for smoke testing.
`bench <id>`	Load-test the endpoint; report TTFB/TTFT/ITL/TPS/e2e at p50/p90/p99.
`metrics <id>`	RPS, latency, GPU utilization.
`requests <id>`	Recent inference requests.

Create from a config

veri deployments create configs/deploy.toml
# or, via the hero verb:
veri run configs/deploy.toml

The config must declare kind = "deploy". See the config schema.

Create inline

Two mutually exclusive modes — deploy a training job’s output, or deploy a HuggingFace model directly.

# From a finished training job:
veri deployments create \
  --model job_abc \
  --name my-endpoint \
  --gpu-type A100-80GB \
  --gpu-count 1

# From a saved model in your library:
veri deployments create \
  --from-model mdl_abc \
  --name support-prod \
  --gpu-type A100-80GB \
  --gpu-count 1

# Directly from HuggingFace (no training):
veri deployments create \
  --from-hf Qwen/Qwen3-4B \
  --name qwen-prod \
  --gpu-type A100-80GB \
  --gpu-count 1

Flag	Purpose
`--model <job-id>`	Deploy a training job’s checkpoint.
`--from-model <model-id>`	Deploy a saved custom model.
`--from-hf <repo>`	Deploy a HuggingFace repo.
`--name`	Display name.
`--gpu-type`	Required in inline form.
`--gpu-count`	Default `1`.
`--replicas`	GPU-box replicas behind one endpoint (1-8). Default `1`. Shorthand for equal `--min-replicas`/`--max-replicas`.
`--min-replicas`	Lower replica bound. Below `--max-replicas` enables autoscaling; `0` enables scale to zero.
`--max-replicas`	Upper replica bound (1-8).
`--concurrency-target`	In-flight requests per replica the autoscaler aims for. Default `8`.
`--scale-to-zero-window`	Idle seconds before a `--min-replicas 0` deployment parks. Default `3600`, minimum `300`.
`--idle-delete-after`	Auto-stop a parked deployment after N idle days. Default: keep it wakeable forever.

See Replicas, autoscaling, and scale to zero for how the bounds behave.

Smoke-test with `chat`

The fastest way to confirm a deployment is healthy:

veri deployments chat dep_abc --message "What is 2+2?"

The CLI calls the OpenAI-compatible endpoint with a single user message and prints the assistant’s reply.

Flag	Purpose
`--message` / `-m`	Required. The user message.
`--model`	Override the model name in the request body. Defaults to the deployment’s name.
`--temperature`	Sampling temperature. Default `1.0`.
`--max-tokens`	Cap on response tokens.
`--session-id`	Pin this conversation to one replica for cache-warm affinity (sets the `X-Veri-Session-Id` header). Only affects multi-replica deployments. Experimental.

For programmatic chat, point any OpenAI-compatible client at the deployment’s endpoint_url (visible in veri deployments get).

Replicas, autoscaling, and scale to zero

One endpoint can be served by several GPU boxes behind a load-balancing router. The replica bounds decide how the fleet behaves:

Bounds	Behavior
`--replicas N` (or equal min/max)	Fixed fleet of N boxes.
`--min-replicas` below `--max-replicas`	Autoscaling: the fleet grows and shrinks with load.
`--min-replicas 0`	Scale to zero: the fleet also parks entirely when idle and wakes on demand.

# Fixed fleet of 2:
veri deployments create \
  --from-hf Qwen/Qwen3-4B --name qwen-prod \
  --gpu-type L4-24GB --gpu-count 1 \
  --replicas 2

# Autoscale 1-4 boxes, park after 30 idle minutes:
veri deployments create \
  --from-hf Qwen/Qwen3-4B --name qwen-dev \
  --gpu-type L4-24GB --gpu-count 1 \
  --min-replicas 0 --max-replicas 4 \
  --scale-to-zero-window 1800

The deployment reports serving once at least one replica is up. list and get show the fleet as ready/desired (for example 2/3 while a scale-up is still provisioning). A replica that dies is replaced automatically; the deployment keeps serving at reduced capacity in the meantime.

Autoscaling

With min-replicas below max-replicas, the fleet targets --concurrency-target in-flight requests per replica (default 8). Scale-up is fast (within about a minute of sustained load); scale-down is deliberately slow (about 15 minutes of sustained low load per step) because a wrong scale-down costs a multi-minute GPU boot. Lower the target for latency-sensitive chat; raise it for batch throughput.

Scale to zero

With --min-replicas 0, a deployment idle past --scale-to-zero-window (default 1 hour) parks: replicas terminate, billing stops, status becomes scaled_to_zero. The next chat request wakes it. While the GPU boots (a few minutes), the server answers that request with 503, a Retry-After header, and error code deployment_waking. The SDK and CLI absorb this for you: veri deployments chat prints a “waking from zero” note and waits for the answer, and the Python SDK’s chat() retries in the background until the deployment is up (see chat and waking for the knobs). Only raw HTTP clients need to handle the 503 themselves. To hide the cold start from your users, pre-warm on a predictive signal instead:

veri deployments wake dep_abc     # 202; no-op if already up

Then poll veri deployments get dep_abc until it’s serving. Optionally set --idle-delete-after N to auto-stop a deployment that stays parked with no traffic for N days.

Change scaling on a live deployment

veri deployments update dep_abc --replicas 4                    # fixed fleet of 4
veri deployments update dep_abc --min-replicas 1 --max-replicas 6
veri deployments update dep_abc --concurrency-target 4

update returns immediately with the new desired state; the fleet converges in the background (watch the replicas column approach desired/desired). Model, GPU, and provider are immutable — recreate the deployment to change those.

Session affinity

With --session-id <key> (or the OpenAI user field, which sets the X-Veri-Session-Id header), a conversation is pinned to one replica so its KV cache stays warm across turns. The same key always lands on the same replica; if that replica is unavailable, the request fails over to another.
Without a session key, requests go to the least-loaded replica.

veri deployments chat dep_abc -m "Continue our story" --session-id chat-42

Reuse the same --session-id for every turn of a conversation to keep it on its cache-warm replica.

Stop to end billing

veri deployments stop dep_abc

stop ends billing and is final — a stopped deployment can’t wake; bring it back by creating a new one. For a deployment you’ll come back to, prefer --min-replicas 0 and let it park instead.

start, rollback, and history are Phase-2. Each depends on backend work that is not yet shipped.

Inspect traffic

veri deployments metrics dep_abc

Returns aggregate counters: total_requests, total_prompt_tokens, total_completion_tokens, avg_latency_ms, error_rate, uptime_seconds. JSON output is dict-shaped (suitable for jq):

veri deployments metrics dep_abc --format json | jq .avg_latency_ms

Recent requests:

veri deployments requests dep_abc --limit 50
veri deployments requests dep_abc --after req_xyz --limit 100   # pagination

List, get

list shows your deployments, most recent first. Filter and paginate with the flags below.

veri deployments list                              # all of them
veri deployments list --status serving             # only the live ones
veri deployments list --status serving --format json
veri deployments get dep_abc                        # one deployment, with its status

Flag	Purpose
`--status`	Filter by status: `queued`, `provisioning`, `serving`, `unhealthy`, `scaled_to_zero`, `waking`, `stopped`, or `failed`.
`--limit`	Maximum number to return. Default `20`.
`--after <id>`	Pagination cursor: return deployments created after this ID.

A deployment bills while either serving or unhealthy, so to catch everything holding a GPU, list both (or list without --status and scan the status column).

Exit codes

Code	Meaning
`0`	Success.
`1`	User error (both `--model` and `--from-hf` passed, missing `--gpu-type` inline).
`2`	API error.
`4`	Auth error.

Get started

Training

Deployments

Evaluations

CLI

Volumes

Commands

Create from a config

Create inline

Smoke-test with `chat`

Replicas, autoscaling, and scale to zero

Autoscaling

Scale to zero

Change scaling on a live deployment

Session affinity

Stop to end billing

Inspect traffic

List, get

Exit codes

What’s next

Evaluate the endpoint

The run verb

​Commands

​Create from a config

​Create inline

​Smoke-test with chat

​Replicas, autoscaling, and scale to zero

​Autoscaling

​Scale to zero

​Change scaling on a live deployment

​Session affinity

​Stop to end billing

​Inspect traffic

​List, get

​Exit codes

​What’s next

Evaluate the endpoint

The run verb

Commands

Create from a config

Create inline

Smoke-test with `chat`

Replicas, autoscaling, and scale to zero

Autoscaling

Scale to zero

Change scaling on a live deployment

Session affinity

Stop to end billing

Inspect traffic

List, get

Exit codes

What’s next