AWS - Veri Documentation

AWS is the default serving provider — omit the provider field to use it.

Serving shapes

`gpu_type`	Memory	GPUs per node	Good for
`L4-24GB`	24 GB	1 or 4	Small models (≤4B) on one GPU, larger ones with tensor parallel on 4; cost-sensitive endpoints
`A10G-24GB`	24 GB	1 or 4	Same tier as L4, Ampere
`A100-80GB`	80 GB	8 (whole node)	Large models with tensor parallel
`H100-80GB`	80 GB	8 (whole node)	Fastest per-request latency, TP up to 8

AWS has no single-GPU A100/H100 instance: the smallest SKU is the whole 8-GPU node, and you pay for all of it. Single-GPU A100/H100 serving is not available yet. Until then, serve on L4-24GB or A10G-24GB (x1 or x4) when the model fits, or on the whole 8-GPU node.

Creating a deployment

client.deployments.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    source="huggingface",
    name="my-deploy",
    gpu={"gpu_type": "L4-24GB", "gpu_count": 1},
    # provider defaults to "aws"
)

Deployments run on vLLM. Replicas, autoscaling, and scale-to-zero work as documented in Scaling.

Where to go next

All serving providers

Compare shapes, status, and the live CLI catalog.

GPU sizing

Match GPU memory to model size.