Deployments

A deployment is a model running on a GPU you control, exposed at an HTTPS endpoint that speaks the OpenAI Chat Completions API.

Deployments are in early access. GPU capacity for serving is limited, so creates can hit capacity errors and larger shapes may not be orderable — check veri gpu list for what’s live before planning around a specific GPU. Training is fully supported and is the platform’s primary focus today.

The smallest viable example is in the Deploy a HF model quickstart. This page is the mental model and the surface map.

Model sources

`source`	What you pass as `model`	When to use
`huggingface`	Any HF model name (`Qwen/Qwen2.5-0.5B-Instruct`)	Serving an off-the-shelf model
`training_job`	A completed training job ID	Serving your own trained model
`custom_model`	A saved model ID	Serving a model from your library

To name, store, and reuse a trained checkpoint as a deployable model, see Custom models. A huggingface deployment can also cache the model in your library (cache: true), and it automatically serves from a cached copy when you have one — faster starts, no hub download.

Deployment parameters

You set these at create time, whether from the SDK (client.deployments.create(...)), the CLI (veri deployments create), or a kind = "deploy" config.

Parameter	Required	Default	What it does
`model`	Yes	—	What to serve: a Hugging Face repo id, a completed training job id, or a custom model id, paired with `source`.
`source`	No	`training_job`	Where `model` comes from: `huggingface`, `training_job`, or `custom_model`. See Model sources.
`name`	Yes	—	Display name, also the `model` value clients send in requests.
`gpu.gpu_type`	Yes	—	GPU SKU (for example `L4-24GB`, `A100-80GB`). Run `veri gpu list` for offered shapes and Sizing the GPU for guidance.
`gpu.gpu_count`	No	`1`	GPUs per replica, for multi-GPU model parallelism.
`engine`	No	`vllm`	Inference engine. `vllm` is the supported engine; `max` (Modular MAX) is currently disabled while under validation, and a `max` create returns a 400.
`provider`	No	`aws`	Cloud to serve on — see Serving providers for shapes and status, or run `veri gpu list` for the live catalog.
`replicas`	No	`1`	GPU-box replicas behind one endpoint (1-8). Shorthand for equal `min_replicas`/`max_replicas` (a fixed fleet).
`min_replicas`	No	`replicas`	Lower replica bound. Set below `max_replicas` to enable autoscaling; set to `0` to enable scale to zero.
`max_replicas`	No	`replicas`	Upper replica bound (1-8).
`concurrency_target`	No	`8`	In-flight requests per replica the autoscaler aims for. Lower it for latency-sensitive traffic, raise it for batch throughput.
`scale_to_zero_window_seconds`	No	`3600`	Idle seconds before a `min_replicas: 0` deployment parks. Minimum `300`.
`idle_delete_after_days`	No	off	Auto-stop a deployment that has been parked at zero with no traffic for this many days.
`cache`	No	`false`	`source: huggingface` only: also save the model to your library for faster future starts. Best-effort; never affects this deployment.
`vllm_image`	No	platform vLLM	Explicit vLLM serving image (e.g. `vllm/vllm-openai:v0.25.1`) for architectures newer than the platform’s vLLM. See Serving newer architectures.
`trace_bodies`	No	`false`	Capture this deployment’s request/response bodies as browsable traces. Enabling stores end-user prompt/completion content. First 5,000 traces per workspace each month are free, then metered per 1k from credits.
`trace_retention`	No	`base`	Trace retention class: `base` (14 days) or `extended` (400 days, extended rate). Requires `trace_bodies: true`.

min_replicas, max_replicas, concurrency_target, scale_to_zero_window_seconds, and idle_delete_after_days can also be changed later on a live deployment — see Scaling. Model, GPU, and provider are immutable: changing those means a new deployment. At call time, chat also accepts a session key for cache-warm routing across replicas.

Parameter	Where	What it does
`session_id`	`X-Veri-Session-Id` header, or the OpenAI `user` field	Pins a conversation to one replica so its KV cache stays warm across turns. Only affects multi-replica deployments.

The deployment lifecycle

Deployments consume GPU credit only while replicas are running — billing is metered per replica, from each GPU box’s launch (warmup included) to its termination. scaled_to_zero charges nothing. Once stopped, billing stops; the deployment row stays for history.

A deployment with min_replicas of 1 or more (the default) never stops billing on its own — you asked for an always-warm floor. Either call stop when you’re done, or create with min_replicas: 0 so it parks itself after the idle window and wakes on the next request.

Calling a deployment

Once a deployment is serving, it accepts OpenAI-shaped requests at https://api.veri.studio/v1/deployments/{dep.id}/chat/completions. Three equivalent ways to call:

from veri_sdk import Client

client = Client()
dep = client.deployments.get("dep_...")  # the object returned by client.deployments.create(...)

# dep carries its own id, so it routes to the right deployment.
response = dep.chat([
    {"role": "user", "content": "Hello"},
])

from openai import OpenAI
oai = OpenAI(
    base_url=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
)
response = oai.chat.completions.create(
    model=dep.name,
    messages=[{"role": "user", "content": "Hello"}],
)

curl https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/chat/completions \
  -H "Authorization: Bearer vk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-deployment-name",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Anything that already talks OpenAI works without changes — point base_url at https://api.veri.studio/v1/deployments/{id} (the OpenAI SDK appends /chat/completions for you).

The chat surface is a passthrough to vLLM: streaming (stream=true, real SSE) works, and request parameters beyond temperature / max_tokens (top_p, stop, seed, response_format, tools, …) are forwarded to the engine untouched — vLLM is the schema authority. See OpenAI compatibility for details and caveats (tool calling needs vLLM tool parsing enabled via vllm_extra_args).

Sizing the GPU

Pick a GPU large enough to hold the weights with headroom. Some rough starting points:

Model size	Minimum	Comfortable
≤ 1B	A100-80GB ×1	A100-80GB ×1
3–8B	A100-80GB ×1	H100-80GB ×1
14–34B	H100-80GB ×2	H100-80GB ×4
70B+	H100-80GB ×4 (quantized)	H100-80GB ×8

Serving usually needs less GPU than training the same model. A 7B you trained on 8×A100 typically serves on 1×A100.

Serving newer architectures

Deployments run a platform-managed vLLM version by default. A model architecture released after that version fails to start (vllm_failed_to_start) because vLLM doesn’t recognize it yet. To serve one, pin the deployment to a newer official vLLM image:

veri deployments create \
  --from-hf poolside/Laguna-M.1 \
  --gpu-type A100-80GB --gpu-count 8 \
  --vllm-image vllm/vllm-openai:v0.25.1

Or in a deploy config:

[deployment]
vllm_image = "vllm/vllm-openai:v0.25.1"

How it behaves:

Only official upstream images are accepted: vllm/vllm-openai:<tag> (NVIDIA) or vllm/vllm-openai-rocm:<tag> (AMD). Anything else is rejected at create.
The image runs via Docker on the deployment’s box, on every provider, and is used exactly as written: the platform never rewrites or substitutes it.
The image is pulled from Docker Hub when the box boots. This pull can be too slow or rate limited to provision reliably, and a deployment that hits the provisioning timeout fails. Explicit images are supported as literal overrides; the supported route for a newer architecture is the platform’s next validated vLLM version.
Without vllm_image, the deployment runs the platform-managed vLLM version for its provider, which never pulls at boot.
engine: vllm only.
The image is set at create and immutable, like model and GPU.

Check the vLLM release notes for the minimum version your architecture needs.

A vllm_image deployment keeps its Hugging Face download inside the serving container, so it can’t be combined with cache: true model-library uploads — the cache upload is skipped.

Observability

Every deployment records request, token, latency, error, cost, and engine-health telemetry. The Deployment observability guide includes an end-to-end smoke test plus dashboard, SDK, REST, and MCP workflows. Lifetime counters and recent requests are available directly from the SDK:

dep_metrics = client.deployments.metrics(dep.id)
# request and token totals, latency percentiles, error rate, cost, and uptime

recent = client.deployments.requests(dep.id, limit=50)
# terminal request timing, usage, status, errors, and cancellation state

Or via CLI:

veri deployments metrics <id>
veri deployments requests <id> --limit 50 --format jsonl

Deployments created with trace_bodies: true additionally capture full request/response bodies as traces, browsable from the dashboard, CLI, and API.

Billing

Running deployments consume GPU credit until you stop them (or, with min_replicas: 0, until they park themselves). See Billing for states and cadence.

Where to go next

Quickstart: deploy a model

Spin up + chat + stop in ~10 minutes.

Train your own model

Then deploy it from a training_job source.

Monitor your deployment

Request metrics, GPU telemetry, and logs.

CLI deployment commands

veri deployments create / chat / stop.

Get started

Training

CLI

Volumes

Webhooks

Deployments

Model sources

Deployment parameters

The deployment lifecycle

Calling a deployment

Sizing the GPU

Serving newer architectures

Observability

Billing

Where to go next

Quickstart: deploy a model

Train your own model

Monitor your deployment

CLI deployment commands

​Model sources

​Deployment parameters

​The deployment lifecycle

​Calling a deployment

​Sizing the GPU

​Serving newer architectures

​Observability

​Billing

​Where to go next

Quickstart: deploy a model

Train your own model

Monitor your deployment

CLI deployment commands

Model sources

Deployment parameters

The deployment lifecycle

Calling a deployment

Sizing the GPU

Serving newer architectures

Observability

Billing

Where to go next