> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veri.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Deployments
> Serve any open-source model (base or trained) on our infrastructure with an OpenAI-compatible API.
A deployment is a model running on a GPU you control, exposed at an HTTPS endpoint that speaks the OpenAI Chat Completions API.
The smallest viable example is in the [Deploy a HF model quickstart](/demos/deploy-hf-quickstart). This page is the mental model and the surface map.
## Model sources
| `source` | What you pass as `model` | When to use |
| -------------- | ------------------------------------------------ | --------------------------------------------------------------- |
| `huggingface` | Any HF model name (`Qwen/Qwen2.5-0.5B-Instruct`) | Serving an off-the-shelf model |
| `training_job` | A completed training job ID | Serving your own trained model |
| `custom_model` | A saved model ID | Serving a model from your [library](/deployments/custom-models) |
To name, store, and reuse a trained checkpoint as a deployable model, see [Custom models](/deployments/custom-models).
## Deployment parameters
You set these at create time, whether from the [SDK](/deployments/examples) (`client.deployments.create(...)`), the [CLI](/cli/deployments#create-from-a-config) (`veri deployments create`), or a [`kind = "deploy"` config](/cli/run#configs-deploy-toml).
| Parameter | Required | Default | What it does |
| ------------------------------ | -------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `model` | Yes | — | What to serve: a Hugging Face repo id, a completed training job id, or a [custom model](/deployments/custom-models) id, paired with `source`. |
| `source` | No | `training_job` | Where `model` comes from: `huggingface`, `training_job`, or `custom_model`. See [Model sources](#model-sources). |
| `name` | Yes | — | Display name, also the `model` value clients send in requests. |
| `gpu.gpu_type` | Yes | — | GPU SKU (for example `L4-24GB`, `A100-80GB`). Run `veri gpu list` for offered shapes and [Sizing the GPU](#sizing-the-gpu) for guidance. |
| `gpu.gpu_count` | No | `1` | GPUs per replica, for multi-GPU model parallelism. |
| `engine` | No | `vllm` | Inference engine: `vllm` or `max` (Modular MAX). |
| `provider` | No | auto | Cloud to serve on. Omit to let Veri choose. |
| `replicas` | No | `1` | GPU-box replicas behind one endpoint (1-8). Shorthand for equal `min_replicas`/`max_replicas` (a fixed fleet). |
| `min_replicas` | No | `replicas` | Lower replica bound. Set below `max_replicas` to enable autoscaling; set to `0` to enable scale to zero. |
| `max_replicas` | No | `replicas` | Upper replica bound (1-8). |
| `concurrency_target` | No | `8` | In-flight requests per replica the autoscaler aims for. Lower it for latency-sensitive traffic, raise it for batch throughput. |
| `scale_to_zero_window_seconds` | No | `3600` | Idle seconds before a `min_replicas: 0` deployment parks. Minimum `300`. |
| `idle_delete_after_days` | No | off | Auto-stop a deployment that has been parked at zero with no traffic for this many days. |
`min_replicas`, `max_replicas`, `concurrency_target`, `scale_to_zero_window_seconds`, and `idle_delete_after_days` can also be changed later on a live deployment — see [Scaling](/cli/deployments#replicas-autoscaling-and-scale-to-zero). Model, GPU, and provider are immutable: changing those means a new deployment.
At call time, `chat` also accepts a session key for cache-warm routing across replicas.
| Parameter | Where | What it does |
| ------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `session_id` | `X-Veri-Session-Id` header, or the OpenAI `user` field | Pins a conversation to one replica so its KV cache stays warm across turns. Only affects multi-replica deployments. |
## The deployment lifecycle
```mermaid theme={null}
stateDiagram-v2
[*] --> queued
queued --> provisioning : reconciler picks up
provisioning --> serving : replica ready, endpoint reachable
provisioning --> failed : GPU lease failed / image pull error
serving --> unhealthy : health check fails
unhealthy --> serving : recovers
serving --> scaled_to_zero : idle past window (min_replicas 0 only)
scaled_to_zero --> waking : request arrives or POST /wake
waking --> provisioning : reconciler re-provisions
serving --> stopped : user called stop
unhealthy --> stopped : user called stop
scaled_to_zero --> stopped : user called stop / idle_delete_after_days
serving --> failed : replicas lost
stopped --> [*]
failed --> [*]
```
Deployments consume GPU credit **only while replicas are running** — billing is metered per replica, from each GPU box's launch (warmup included) to its termination. `scaled_to_zero` charges nothing. Once `stopped`, billing stops; the deployment row stays for history.
A deployment with `min_replicas` of 1 or more (the default) **never stops billing on its own** — you asked for an always-warm floor. Either call `stop` when you're done, or create with `min_replicas: 0` so it parks itself after the idle window and wakes on the next request.
## Calling a deployment
Once a deployment is `serving`, it accepts OpenAI-shaped requests at `https://api.veri.studio/v1/deployments/{dep.id}/chat/completions`. Three equivalent ways to call:
```python Veri SDK theme={null}
from veri_sdk import Client
client = Client()
dep = client.deployments.get("dep_...") # the object returned by client.deployments.create(...)
# dep carries its own id, so it routes to the right deployment.
response = dep.chat([
{"role": "user", "content": "Hello"},
])
```
```python OpenAI SDK theme={null}
from openai import OpenAI
oai = OpenAI(
base_url=f"https://api.veri.studio/v1/deployments/{dep.id}",
api_key="vk_your_api_key",
)
response = oai.chat.completions.create(
model=dep.name,
messages=[{"role": "user", "content": "Hello"}],
)
```
```bash curl theme={null}
curl https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/chat/completions \
-H "Authorization: Bearer vk_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"model": "your-deployment-name",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
Anything that already talks OpenAI works without changes — point `base_url` at `https://api.veri.studio/v1/deployments/{id}` (the OpenAI SDK appends `/chat/completions` for you).
Today's chat surface supports basic completions only. **Not yet supported:** streaming (`stream=true` is accepted but ignored — you always get the full response), `tool_calls` / `tools`, `top_p`, `frequency_penalty`, `presence_penalty`, structured outputs, and vision. Tools / streaming are on the roadmap.
## Sizing the GPU
Pick a GPU large enough to hold the weights with headroom. Some rough starting points:
| Model size | Minimum | Comfortable |
| ---------- | ------------------------ | ------------ |
| ≤ 1B | A100-80GB ×1 | A100-80GB ×1 |
| 3–8B | A100-80GB ×1 | H100-80GB ×1 |
| 14–34B | H100-80GB ×2 | H100-80GB ×4 |
| 70B+ | H100-80GB ×4 (quantized) | H100-80GB ×8 |
Serving usually needs **less** GPU than training the same model. A 7B you trained on 8×A100 typically serves on 1×A100.
## Observability
Every deployment records request, token, latency, error, cost, and engine-health telemetry. The [Deployment observability guide](/deployments/observability) includes an end-to-end smoke test plus dashboard, SDK, REST, and MCP workflows.
Lifetime counters and recent requests are available directly from the SDK:
```python theme={null}
dep_metrics = client.deployments.metrics(dep.id)
# request and token totals, latency percentiles, error rate, cost, and uptime
recent = client.deployments.requests(dep.id, limit=50)
# terminal request timing, usage, status, errors, and cancellation state
```
Or via CLI:
```bash theme={null}
veri deployments metrics
veri deployments requests --limit 50 --format jsonl
```
## Billing
Running deployments consume GPU credit until you stop them (or, with `min_replicas: 0`, until they park themselves). See [Billing](/deployments/cost) for states and cadence.
## Where to go next
Spin up + chat + stop in \~10 minutes.
Then deploy it from a `training_job` source.
Score it on a held-out dataset.
`veri deployments create / chat / stop`.