Skip to main content
A deployment is a pod on a GPU you control. Picking the GPU matters: too small and the model OOMs at load; too big and you’re burning money for headroom you don’t use.

​
GPU sizing by model size

Rough sizing for inference only (training needs more headroom): These are AWS shapes (the default provider): L4-24GB and A10G-24GB come as 1 or 4 GPUs, and A100-80GB and H100-80GB only as whole 8-GPU nodes. Any other gpu_count is rejected at create. Rule of thumb: Serving needs less GPU than training. A 7B model trained on an 8×A100 node typically serves on 1 to 4 L4s.

​
Serving shapes by provider

Each provider sells fixed node shapes — asking for a shape a provider doesn’t sell returns a 400 at create, before anything boots. The full matrix (with status and per-provider guides) lives on the Serving providers page; the live version, including current prices, is veri gpu list.
AWS has no single-GPU A100/H100 instance: the smallest SKU is the whole 8-GPU node, and you pay for all of it. Its genuine single-GPU shapes are L4-24GB and A10G-24GB. Single-GPU A100 serving is not available yet.

​
Cold start

Time from POST /v1/deployments to serving depends on the provider’s pod-lease time plus the model weight download. Expected ranges: These are operational expectations, not contractual. The slowest part is usually the model weight download from HF. Once a deployment is serving, individual chat requests have sub-second TTFB on small models.

​
Endpoint URL stability

Once a deployment reaches serving, its endpoint_url is the platform URL https://api.veri.studio/v1/deployments/{id}/chat/completions, authenticated with your Veri API key. The URL is fixed for the deployment’s lifetime, including across scale-to-zero parks and wakes, so you can save it to your config: a request to it while the deployment is parked wakes it. The endpoint_url field itself reads null while the deployment is parked (scaled_to_zero) and returns once a wake serves. After you stop the deployment, requests to the URL return 400.

​
OOM at load

If the model fails to load on a first boot (for example CUDA out of memory during weight init, not a request failure), the deployment transitions to failed. failure_code is usually vllm_failed_to_start (or oom), and error carries the engine’s startup log tail, where the out-of-memory message shows up. A wake boot that fails this way re-parks instead.
Common fix: move to a larger shape (L4-24GB ×1 → L4-24GB ×4, or a whole A100-80GB/H100-80GB ×8 node), or shorten the context with vllm_extra_args: ["--max-model-len", "8192"].

​
Where to go next

Billing

Billing states and the idle-stop warning.

OpenAI compatibility

What works, what doesn’t.

Deployments API

Endpoint reference for create / chat / stop.