Deployments

​

Model sources

​

Deployment parameters

​

The deployment lifecycle

​

Self-healing

• While a replacement boots, the deployment shows provisioning with failure_code: replica_lost and an error such as Replacing a lost GPU box (attempt 1 of 3). Chat requests get a 503 with Retry-After until it serves again. Each replacement box is billed from its launch, like any replica.
• Replacements follow a fixed ladder: the first goes out at once, the next ones 10, 20, 40 and 80 seconds after the previous launch. A replacement that crashes, that does not become healthy within the boot deadline, or that the cloud rejects counts as one attempt.
• When a replacement reaches serving, failure_code and error clear and the full budget is available again for the next loss. Any PATCH to the deployment also restarts the budget.
• When the attempts are used up, or with self_heal_attempts: 0, a deployment with min_replicas of 1 or more goes failed with failure_code: replica_lost (billing ends; redeploy to bring it back), and a min_replicas: 0 deployment parks (scaled_to_zero, also with replica_lost) and wakes on the next request.

​

Calling a deployment

from veri_sdk import Client

client = Client()
dep = client.deployments.get("<deployment-id>")  # or the object returned by client.deployments.create(...)

# dep carries its own id, so it routes to the right deployment.
response = dep.chat([
    {"role": "user", "content": "Hello"},
])

from openai import OpenAI
oai = OpenAI(
    base_url=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
)
response = oai.chat.completions.create(
    model=dep.name,
    messages=[{"role": "user", "content": "Hello"}],
)

curl https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/chat/completions \
  -H "Authorization: Bearer vk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-deployment-name",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

​

Sizing the GPU

​

Serving newer architectures

veri deployments create \
  --from-hf poolside/Laguna-M.1 \
  --gpu-type A100-80GB --gpu-count 8 \
  --vllm-image vllm/vllm-openai:v0.31.0

[deployment]
vllm_image = "vllm/vllm-openai:v0.31.0"

• Only official upstream images are accepted: vllm/vllm-openai:<tag> (NVIDIA) or vllm/vllm-openai-rocm:<tag> (AMD). Anything else is rejected at create.
• The image runs via Docker on the deployment’s box, on every provider, and is used exactly as written: the platform never rewrites or substitutes it.
• The image is pulled from Docker Hub when the box boots, which adds to the cold start. The boot deadline is the same as for the platform image: 40 minutes plus 5 seconds per GB of the deployment’s total GPU memory (about 42 minutes on L4-24GB x1, about 93 minutes on H100-80GB x8). The pull can still be too slow or rate limited to provision reliably: a first boot that passes the deadline fails with watchdog_timeout, and a wake boot re-parks (a request after the backoff, or POST /wake, wakes it again). Explicit images are supported as literal overrides; the supported route for a newer architecture is the platform’s next validated vLLM version.
• Without vllm_image, the deployment runs the platform-managed vLLM (0.27.1), which never pulls at boot.
• engine: vllm only.
• The image is set at create and immutable, like model and GPU.

​

Observability

dep_metrics = client.deployments.metrics(dep.id)
# request and token totals, latency percentiles, error rate, cost, and uptime

recent = client.deployments.requests(dep.id, limit=50)
# terminal request timing, usage, status, errors, and cancellation state

veri deployments metrics <id>
veri deployments requests <id> --limit 50 --format jsonl

​

Billing

​

Where to go next