Hosting & GPU sizing

A deployment is a pod on a GPU you control. Picking the GPU matters: too small and the model OOMs at load; too big and you’re burning money for headroom you don’t use.

GPU sizing by model size

Rough sizing for inference only (training needs more headroom):

Model size	Minimum	Comfortable	Notes
≤ 1B	A100-80GB ×1	A100-80GB ×1	Small models always fit on a single GPU.
3–8B	A100-80GB ×1	H100-80GB ×1	A100-80GB works; H100-80GB is faster per request.
14–34B	H100-80GB ×2	H100-80GB ×4	Tensor parallel; expect TP2 or TP4.
70B+	H100-80GB ×4 (quantized)	H100-80GB ×8	TP8 for full precision; quantized at TP4.

Rule of thumb: Serving uses ~half the GPU you trained on. A 7B model trained on 8×A100 typically serves on 1×A100.

Supported GPU types

`gpu_type`	Memory	Common use
`L4-24GB`	24 GB	Single-GPU shape for small models (up to ~3B)
`A10G-24GB`	24 GB	Single-GPU shape for small models
`A100-80GB`	80 GB	Default sweet spot; launches as an 8-GPU node on AWS
`H100-80GB`	80 GB	~2x throughput vs A100-80GB; launches as an 8-GPU node on AWS

H200 and MI300X support are on the roadmap.

Provider routing

Deployments run on AWS. The provider field defaults to aws; there are no other providers to select today.

client.deployments.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    source="huggingface",
    name="my-deploy",
    gpu={"gpu_type": "H100-80GB", "gpu_count": 1},
    provider="aws",
)

Cold start

Time from POST /v1/deployments to serving depends on the provider’s pod-lease time plus the model weight download. Expected ranges:

Source	Typical
`huggingface`, small model (≤3B)	~1–3 min
`huggingface`, 7–13B	~3–6 min
`huggingface`, 30B+	~6–15 min (depends on cache hit)
`training_job` (your trained model)	~1–3 min — checkpoint already on Veri-managed S3, no HF download

These are operational expectations, not contractual. The slowest part is usually the model weight download from HF. Once a deployment is serving, individual chat requests have sub-second TTFB on small models.

Endpoint URL stability

Once a deployment reaches serving, its endpoint_url doesn’t change. Save it to your config — it stays valid until you stop the deployment.

OOM at load

If the model fails to load (OOM during weight init, not a request failure), the deployment transitions to failed with error.message populated.

dep = client.deployments.get(deployment_id)
if dep.status == "failed":
    print(dep.error)

Common fix: bump to a larger GPU (A100-80GB → H100-80GB) or add a second GPU (gpu_count=2).

Where to go next

Billing

Billing states and the idle-stop warning.

OpenAI compatibility

What works, what doesn’t.

Deployments API

Endpoint reference for create / chat / stop.

Get started

Training

Deployments

Evaluations

CLI

Volumes

GPU sizing by model size

Supported GPU types

Provider routing

Cold start

Endpoint URL stability

OOM at load

Where to go next

Billing

OpenAI compatibility

Deployments API

​GPU sizing by model size

​Supported GPU types

​Provider routing

​Cold start

​Endpoint URL stability

​OOM at load

​Where to go next

Billing

OpenAI compatibility

Deployments API

GPU sizing by model size

Supported GPU types

Provider routing

Cold start

Endpoint URL stability

OOM at load

Where to go next