> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veri.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# veri deployments

> Create OpenAI-compatible inference endpoints, manage their lifecycle, and inspect traffic.

The `veri deployments` group creates and manages inference endpoints. Each deployment exposes an OpenAI-compatible `/chat/completions` API backed by a Veri-trained or HuggingFace model.

## Commands

| Command                     | Purpose                                                              |
| --------------------------- | -------------------------------------------------------------------- |
| `create <config>` or inline | Bring up a deployment.                                               |
| `list`                      | List deployments.                                                    |
| `get <id>`                  | Show one deployment.                                                 |
| `update <id>`               | Change replica bounds / scaling settings on a live deployment.       |
| `wake <id>`                 | Pre-warm a scaled-to-zero deployment.                                |
| `stop <id>`                 | Stop for good. Billing stops.                                        |
| `chat <id>`                 | Send a one-shot message for smoke testing.                           |
| `bench <id>`                | Load-test the endpoint; report TTFB/TTFT/ITL/TPS/e2e at p50/p90/p99. |
| `metrics <id>`              | RPS, latency, GPU utilization.                                       |
| `requests <id>`             | Recent inference requests.                                           |

## Create from a config

```bash theme={null}
veri deployments create configs/deploy.toml
# or, via the hero verb:
veri run configs/deploy.toml
```

The config must declare `kind = "deploy"`. See the [config schema](/cli/run#configs-deploy-toml).

## Create inline

Two mutually exclusive modes — deploy a training job's output, or deploy a HuggingFace model directly.

```bash theme={null}
# From a finished training job:
veri deployments create \
  --model job_abc \
  --name my-endpoint \
  --gpu-type A100-80GB \
  --gpu-count 1

# From a saved model in your library:
veri deployments create \
  --from-model mdl_abc \
  --name support-prod \
  --gpu-type A100-80GB \
  --gpu-count 1

# Directly from HuggingFace (no training):
veri deployments create \
  --from-hf Qwen/Qwen3-4B \
  --name qwen-prod \
  --gpu-type A100-80GB \
  --gpu-count 1
```

| Flag                      | Purpose                                                                                                         |
| ------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `--model <job-id>`        | Deploy a training job's checkpoint.                                                                             |
| `--from-model <model-id>` | Deploy a saved [custom model](/deployments/custom-models).                                                      |
| `--from-hf <repo>`        | Deploy a HuggingFace repo.                                                                                      |
| `--name`                  | Display name.                                                                                                   |
| `--gpu-type`              | Required in inline form.                                                                                        |
| `--gpu-count`             | Default `1`.                                                                                                    |
| `--replicas`              | GPU-box replicas behind one endpoint (1-8). Default `1`. Shorthand for equal `--min-replicas`/`--max-replicas`. |
| `--min-replicas`          | Lower replica bound. Below `--max-replicas` enables autoscaling; `0` enables scale to zero.                     |
| `--max-replicas`          | Upper replica bound (1-8).                                                                                      |
| `--concurrency-target`    | In-flight requests per replica the autoscaler aims for. Default `8`.                                            |
| `--scale-to-zero-window`  | Idle seconds before a `--min-replicas 0` deployment parks. Default `3600`, minimum `300`.                       |
| `--idle-delete-after`     | Auto-stop a parked deployment after N idle days. Default: keep it wakeable forever.                             |

See [Replicas, autoscaling, and scale to zero](#replicas-autoscaling-and-scale-to-zero) for how the bounds behave.

## Smoke-test with `chat`

The fastest way to confirm a deployment is healthy:

```bash theme={null}
veri deployments chat dep_abc --message "What is 2+2?"
```

The CLI calls the OpenAI-compatible endpoint with a single user message and prints the assistant's reply.

| Flag               | Purpose                                                                                                                                                   |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--message` / `-m` | Required. The user message.                                                                                                                               |
| `--model`          | Override the model name in the request body. Defaults to the deployment's name.                                                                           |
| `--temperature`    | Sampling temperature. Default `1.0`.                                                                                                                      |
| `--max-tokens`     | Cap on response tokens.                                                                                                                                   |
| `--session-id`     | Pin this conversation to one replica for cache-warm affinity (sets the `X-Veri-Session-Id` header). Only affects multi-replica deployments. Experimental. |

For programmatic chat, point any OpenAI-compatible client at the deployment's `endpoint_url` (visible in `veri deployments get`).

## Replicas, autoscaling, and scale to zero

One endpoint can be served by several GPU boxes behind a load-balancing router. The replica bounds decide how the fleet behaves:

| Bounds                                  | Behavior                                                                    |
| --------------------------------------- | --------------------------------------------------------------------------- |
| `--replicas N` (or equal min/max)       | Fixed fleet of N boxes.                                                     |
| `--min-replicas` below `--max-replicas` | Autoscaling: the fleet grows and shrinks with load.                         |
| `--min-replicas 0`                      | Scale to zero: the fleet also parks entirely when idle and wakes on demand. |

```bash theme={null}
# Fixed fleet of 2:
veri deployments create \
  --from-hf Qwen/Qwen3-4B --name qwen-prod \
  --gpu-type L4-24GB --gpu-count 1 \
  --replicas 2

# Autoscale 1-4 boxes, park after 30 idle minutes:
veri deployments create \
  --from-hf Qwen/Qwen3-4B --name qwen-dev \
  --gpu-type L4-24GB --gpu-count 1 \
  --min-replicas 0 --max-replicas 4 \
  --scale-to-zero-window 1800
```

The deployment reports `serving` once at least one replica is up. `list` and `get` show the fleet as `ready/desired` (for example `2/3` while a scale-up is still provisioning). A replica that dies is replaced automatically; the deployment keeps serving at reduced capacity in the meantime.

### Autoscaling

With `min-replicas` below `max-replicas`, the fleet targets `--concurrency-target` in-flight requests per replica (default 8). Scale-up is fast (within about a minute of sustained load); scale-down is deliberately slow (about 15 minutes of sustained low load per step) because a wrong scale-down costs a multi-minute GPU boot. Lower the target for latency-sensitive chat; raise it for batch throughput.

### Scale to zero

With `--min-replicas 0`, a deployment idle past `--scale-to-zero-window` (default 1 hour) parks: replicas terminate, billing stops, status becomes `scaled_to_zero`. The next chat request wakes it. While the GPU boots (a few minutes), the server answers that request with `503`, a `Retry-After` header, and error code `deployment_waking`. The SDK and CLI absorb this for you: `veri deployments chat` prints a "waking from zero" note and waits for the answer, and the Python SDK's `chat()` retries in the background until the deployment is up (see [chat and waking](/deployments/api#post-v1deploymentsidchatcompletions) for the knobs). Only raw HTTP clients need to handle the `503` themselves. To hide the cold start from your users, pre-warm on a predictive signal instead:

```bash theme={null}
veri deployments wake dep_abc     # 202; no-op if already up
```

Then poll `veri deployments get dep_abc` until it's `serving`. Optionally set `--idle-delete-after N` to auto-stop a deployment that stays parked with no traffic for N days.

### Change scaling on a live deployment

```bash theme={null}
veri deployments update dep_abc --replicas 4                    # fixed fleet of 4
veri deployments update dep_abc --min-replicas 1 --max-replicas 6
veri deployments update dep_abc --concurrency-target 4
```

`update` returns immediately with the new desired state; the fleet converges in the background (watch the `replicas` column approach `desired/desired`). Model, GPU, and provider are immutable — recreate the deployment to change those.

### Session affinity

* **With `--session-id <key>`** (or the OpenAI `user` field, which sets the `X-Veri-Session-Id` header), a conversation is pinned to one replica so its KV cache stays warm across turns. The same key always lands on the same replica; if that replica is unavailable, the request fails over to another.
* **Without a session key**, requests go to the least-loaded replica.

```bash theme={null}
veri deployments chat dep_abc -m "Continue our story" --session-id chat-42
```

Reuse the same `--session-id` for every turn of a conversation to keep it on its cache-warm replica.

## Stop to end billing

```bash theme={null}
veri deployments stop dep_abc
```

`stop` ends billing and is final — a stopped deployment can't wake; bring it back by creating a new one. For a deployment you'll come back to, prefer `--min-replicas 0` and let it park instead.

<Note>
  `start`, `rollback`, and `history` are Phase-2. Each depends on backend work that is not yet shipped.
</Note>

## Inspect traffic

```bash theme={null}
veri deployments metrics dep_abc
```

Returns aggregate counters: `total_requests`, `total_prompt_tokens`, `total_completion_tokens`, `avg_latency_ms`, `error_rate`, `uptime_seconds`. JSON output is dict-shaped (suitable for `jq`):

```bash theme={null}
veri deployments metrics dep_abc --format json | jq .avg_latency_ms
```

Recent requests:

```bash theme={null}
veri deployments requests dep_abc --limit 50
veri deployments requests dep_abc --after req_xyz --limit 100   # pagination
```

## List, get

`list` shows your deployments, most recent first. Filter and paginate with the flags below.

```bash theme={null}
veri deployments list                              # all of them
veri deployments list --status serving             # only the live ones
veri deployments list --status serving --format json
veri deployments get dep_abc                        # one deployment, with its status
```

| Flag           | Purpose                                                                                                                 |
| -------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `--status`     | Filter by status: `queued`, `provisioning`, `serving`, `unhealthy`, `scaled_to_zero`, `waking`, `stopped`, or `failed`. |
| `--limit`      | Maximum number to return. Default `20`.                                                                                 |
| `--after <id>` | Pagination cursor: return deployments created after this ID.                                                            |

A deployment bills while either `serving` or `unhealthy`, so to catch everything holding a GPU, list both (or list without `--status` and scan the `status` column).

## Exit codes

| Code | Meaning                                                                          |
| ---- | -------------------------------------------------------------------------------- |
| `0`  | Success.                                                                         |
| `1`  | User error (both `--model` and `--from-hf` passed, missing `--gpu-type` inline). |
| `2`  | API error.                                                                       |
| `4`  | Auth error.                                                                      |

## What's next

<CardGroup cols={2}>
  <Card title="Evaluate the endpoint" icon="chart-line" href="/cli/evals">
    Run scorers against your deployed model.
  </Card>

  <Card title="The run verb" icon="play" href="/cli/run">
    Deploy from a TOML config.
  </Card>
</CardGroup>