Deployments

A deployment is a model running on a GPU you control, exposed at an HTTPS endpoint that speaks the OpenAI Chat Completions API. The smallest viable example is in the Deploy a HF model quickstart. This page is the mental model and the surface map.

Model sources

`source`	What you pass as `model`	When to use
`huggingface`	Any HF model name (`Qwen/Qwen2.5-0.5B-Instruct`)	Serving an off-the-shelf model
`training_job`	A completed training job ID	Serving your own trained model
`custom_model`	A saved model ID	Serving a model from your library

To name, store, and reuse a trained checkpoint as a deployable model, see Custom models.

Deployment parameters

You set these at create time, whether from the SDK (client.deployments.create(...)), the CLI (veri deployments create), or a kind = "deploy" config.

Parameter	Required	Default	What it does
`model`	Yes	—	What to serve: a Hugging Face repo id, a completed training job id, or a custom model id, paired with `source`.
`source`	No	`training_job`	Where `model` comes from: `huggingface`, `training_job`, or `custom_model`. See Model sources.
`name`	Yes	—	Display name, also the `model` value clients send in requests.
`gpu.gpu_type`	Yes	—	GPU SKU (for example `L4-24GB`, `A100-80GB`). Run `veri gpu list` for offered shapes and Sizing the GPU for guidance.
`gpu.gpu_count`	No	`1`	GPUs per replica, for multi-GPU model parallelism.
`engine`	No	`vllm`	Inference engine: `vllm` or `max` (Modular MAX).
`provider`	No	auto	Cloud to serve on. Omit to let Veri choose.
`replicas`	No	`1`	GPU-box replicas behind one endpoint (1-8). Shorthand for equal `min_replicas`/`max_replicas` (a fixed fleet).
`min_replicas`	No	`replicas`	Lower replica bound. Set below `max_replicas` to enable autoscaling; set to `0` to enable scale to zero.
`max_replicas`	No	`replicas`	Upper replica bound (1-8).
`concurrency_target`	No	`8`	In-flight requests per replica the autoscaler aims for. Lower it for latency-sensitive traffic, raise it for batch throughput.
`scale_to_zero_window_seconds`	No	`3600`	Idle seconds before a `min_replicas: 0` deployment parks. Minimum `300`.
`idle_delete_after_days`	No	off	Auto-stop a deployment that has been parked at zero with no traffic for this many days.

min_replicas, max_replicas, concurrency_target, scale_to_zero_window_seconds, and idle_delete_after_days can also be changed later on a live deployment — see Scaling. Model, GPU, and provider are immutable: changing those means a new deployment. At call time, chat also accepts a session key for cache-warm routing across replicas.

Parameter	Where	What it does
`session_id`	`X-Veri-Session-Id` header, or the OpenAI `user` field	Pins a conversation to one replica so its KV cache stays warm across turns. Only affects multi-replica deployments.

The deployment lifecycle

Deployments consume GPU credit only while replicas are running — billing is metered per replica, from each GPU box’s launch (warmup included) to its termination. scaled_to_zero charges nothing. Once stopped, billing stops; the deployment row stays for history.

A deployment with min_replicas of 1 or more (the default) never stops billing on its own — you asked for an always-warm floor. Either call stop when you’re done, or create with min_replicas: 0 so it parks itself after the idle window and wakes on the next request.

Calling a deployment

Once a deployment is serving, it accepts OpenAI-shaped requests at https://api.veri.studio/v1/deployments/{dep.id}/chat/completions. Three equivalent ways to call:

from veri_sdk import Client

client = Client()
dep = client.deployments.get("dep_...")  # the object returned by client.deployments.create(...)

# dep carries its own id, so it routes to the right deployment.
response = dep.chat([
    {"role": "user", "content": "Hello"},
])

from openai import OpenAI
oai = OpenAI(
    base_url=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
)
response = oai.chat.completions.create(
    model=dep.name,
    messages=[{"role": "user", "content": "Hello"}],
)

curl https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/chat/completions \
  -H "Authorization: Bearer vk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-deployment-name",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Anything that already talks OpenAI works without changes — point base_url at https://api.veri.studio/v1/deployments/{id} (the OpenAI SDK appends /chat/completions for you).

Today’s chat surface supports basic completions only. Not yet supported: streaming (stream=true is accepted but ignored — you always get the full response), tool_calls / tools, top_p, frequency_penalty, presence_penalty, structured outputs, and vision. Tools / streaming are on the roadmap.

Sizing the GPU

Pick a GPU large enough to hold the weights with headroom. Some rough starting points:

Model size	Minimum	Comfortable
≤ 1B	A100-80GB ×1	A100-80GB ×1
3–8B	A100-80GB ×1	H100-80GB ×1
14–34B	H100-80GB ×2	H100-80GB ×4
70B+	H100-80GB ×4 (quantized)	H100-80GB ×8

Serving usually needs less GPU than training the same model. A 7B you trained on 8×A100 typically serves on 1×A100.

Observability

Every deployment records request, token, latency, error, cost, and engine-health telemetry. The Deployment observability guide includes an end-to-end smoke test plus dashboard, SDK, REST, and MCP workflows. Lifetime counters and recent requests are available directly from the SDK:

dep_metrics = client.deployments.metrics(dep.id)
# request and token totals, latency percentiles, error rate, cost, and uptime

recent = client.deployments.requests(dep.id, limit=50)
# terminal request timing, usage, status, errors, and cancellation state

Or via CLI:

veri deployments metrics <id>
veri deployments requests <id> --limit 50 --format jsonl

Billing

Running deployments consume GPU credit until you stop them (or, with min_replicas: 0, until they park themselves). See Billing for states and cadence.

Where to go next

Quickstart: deploy a model

Spin up + chat + stop in ~10 minutes.

Train your own model

Then deploy it from a training_job source.

Evaluate your deployment

Score it on a held-out dataset.

CLI deployment commands

veri deployments create / chat / stop.

Get started

Training

Evaluations

CLI

Volumes

Deployments

Model sources

Deployment parameters

The deployment lifecycle

Calling a deployment

Sizing the GPU

Observability

Billing

Where to go next

Quickstart: deploy a model

Train your own model

Evaluate your deployment

CLI deployment commands

​Model sources

​Deployment parameters

​The deployment lifecycle

​Calling a deployment

​Sizing the GPU

​Observability

​Billing

​Where to go next

Quickstart: deploy a model

Train your own model

Evaluate your deployment

CLI deployment commands

Model sources

Deployment parameters

The deployment lifecycle

Calling a deployment

Sizing the GPU

Observability

Billing

Where to go next