> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veri.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# Hosting & GPU sizing

> Pick the right GPU for the model size. Provider routing, cold-start expectations.

A deployment is a pod on a GPU you control. Picking the GPU matters: too small and the model OOMs at load; too big and you're burning money for headroom you don't use.

## GPU sizing by model size

Rough sizing for **inference only** (training needs more headroom):

| Model size | Minimum                  | Comfortable  | Notes                                             |
| ---------- | ------------------------ | ------------ | ------------------------------------------------- |
| ≤ 1B       | A100-80GB ×1             | A100-80GB ×1 | Small models always fit on a single GPU.          |
| 3–8B       | A100-80GB ×1             | H100-80GB ×1 | A100-80GB works; H100-80GB is faster per request. |
| 14–34B     | H100-80GB ×2             | H100-80GB ×4 | Tensor parallel; expect TP2 or TP4.               |
| 70B+       | H100-80GB ×4 (quantized) | H100-80GB ×8 | TP8 for full precision; quantized at TP4.         |

**Rule of thumb:** Serving uses \~half the GPU you trained on. A 7B model trained on 8×A100 typically serves on 1×A100.

## Supported GPU types

| `gpu_type`  | Memory | Common use                                                     |
| ----------- | ------ | -------------------------------------------------------------- |
| `L4-24GB`   | 24 GB  | Single-GPU shape for small models (up to \~3B)                 |
| `A10G-24GB` | 24 GB  | Single-GPU shape for small models                              |
| `A100-80GB` | 80 GB  | Default sweet spot; launches as an 8-GPU node on AWS           |
| `H100-80GB` | 80 GB  | \~2x throughput vs A100-80GB; launches as an 8-GPU node on AWS |

H200 and MI300X support are on the roadmap.

## Provider routing

Deployments run on **AWS**. The `provider` field defaults to `aws`; there are no other providers to select today.

```python theme={null}
client.deployments.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    source="huggingface",
    name="my-deploy",
    gpu={"gpu_type": "H100-80GB", "gpu_count": 1},
    provider="aws",
)
```

## Cold start

Time from `POST /v1/deployments` to `serving` depends on the provider's pod-lease time plus the model weight download. **Expected** ranges:

| Source                              | Typical                                                           |
| ----------------------------------- | ----------------------------------------------------------------- |
| `huggingface`, small model (≤3B)    | \~1–3 min                                                         |
| `huggingface`, 7–13B                | \~3–6 min                                                         |
| `huggingface`, 30B+                 | \~6–15 min (depends on cache hit)                                 |
| `training_job` (your trained model) | \~1–3 min — checkpoint already on Veri-managed S3, no HF download |

These are operational expectations, not contractual. The slowest part is usually the model weight download from HF. Once a deployment is `serving`, individual chat requests have sub-second TTFB on small models.

## Endpoint URL stability

Once a deployment reaches `serving`, its `endpoint_url` doesn't change. Save it to your config — it stays valid until you `stop` the deployment.

## OOM at load

If the model fails to load (OOM during weight init, not a request failure), the deployment transitions to `failed` with `error.message` populated.

```python theme={null}
dep = client.deployments.get(deployment_id)
if dep.status == "failed":
    print(dep.error)
```

Common fix: bump to a larger GPU (`A100-80GB` → `H100-80GB`) or add a second GPU (`gpu_count=2`).

## Where to go next

<CardGroup cols={2}>
  <Card title="Billing" icon="dollar-sign" href="/deployments/cost">
    Billing states and the idle-stop warning.
  </Card>

  <Card title="OpenAI compatibility" icon="code" href="/deployments/openai-compat">
    What works, what doesn't.
  </Card>

  <Card title="Deployments API" icon="play" href="/api-reference/introduction">
    Endpoint reference for `create` / `chat` / `stop`.
  </Card>
</CardGroup>