Skip to main content
A deployment is a pod on a GPU you control. Picking the GPU matters: too small and the model OOMs at load; too big and you’re burning money for headroom you don’t use.

​
GPU sizing by model size

Rough sizing for inference only (training needs more headroom): Rule of thumb: Serving uses ~half the GPU you trained on. A 7B model trained on 8×A100 typically serves on 1×A100.

​
Serving shapes by provider

Each provider sells fixed node shapes — asking for a shape a provider doesn’t sell returns a 400 at create, before anything boots. The full matrix (with status and per-provider guides) lives on the Serving providers page; the live version, including current prices, is veri gpu list.
The AMD MI300X (Hot Aisle) carries 192 GB of VRAM on a single GPU — enough to serve a 70B model in bf16 on one card, where NVIDIA equivalents need a multi-GPU node.
AWS has no single-GPU A100/H100 instance — the smallest SKU is the whole 8-GPU node, and you pay for all of it. For a single A100/H100, pick a provider with a genuine 1-GPU shape (GCP).

​
Cold start

Time from POST /v1/deployments to serving depends on the provider’s pod-lease time plus the model weight download. Expected ranges: These are operational expectations, not contractual. The slowest part is usually the model weight download from HF. Once a deployment is serving, individual chat requests have sub-second TTFB on small models.

​
Endpoint URL stability

Once a deployment reaches serving, its endpoint_url doesn’t change. Save it to your config — it stays valid until you stop the deployment.

​
OOM at load

If the model fails to load (OOM during weight init, not a request failure), the deployment transitions to failed with error.message populated.
Common fix: bump to a larger GPU (A100-80GB → H100-80GB) or add a second GPU (gpu_count=2).

​
Where to go next

Billing

Billing states and the idle-stop warning.

OpenAI compatibility

What works, what doesn’t.

Deployments API

Endpoint reference for create / chat / stop.