Deployments API

Reference for /v1/deployments/*. All endpoints require Bearer token auth (Authorization: Bearer vk_...).

Endpoints

Method	Path	Purpose
`POST`	`/v1/deployments`	Create a deployment
`GET`	`/v1/deployments`	List (paginated, filter by `status`)
`GET`	`/v1/deployments/{id}`	Get one
`PATCH`	`/v1/deployments/{id}`	Update scaling settings (replica bounds, concurrency target, idle windows)
`GET`	`/v1/deployments/{id}/replicas`	List the deployment’s replicas
`POST`	`/v1/deployments/{id}/replicas/{replica_id}/drain`	Retire one replica (scale in, or recycle with `replace=true`)
`POST`	`/v1/deployments/{id}/wake`	Pre-warm a `scaled_to_zero` deployment (202, idempotent)
`POST`	`/v1/deployments/{id}/stop`	Transition to `stopped`
`POST`	`/v1/deployments/{id}/chat/completions`	OpenAI-compatible chat
`GET`	`/v1/deployments/{id}/requests`	Request log
`GET`	`/v1/deployments/{id}/metrics`	Aggregated metrics

POST /v1/deployments

Create a deployment from a base HF model or a completed training job. Request schema:

{
  "model": "Qwen/Qwen2.5-0.5B-Instruct",
  "source": "huggingface",
  "name": "my-deploy",
  "gpu": {"gpu_type": "A100-80GB", "gpu_count": 1},
  "provider": "aws"
}

Field	Required	Notes
`model`	yes	HF model ID (when `source=huggingface`) or training job ID (when `source=training_job`)
`source`	yes	`huggingface` or `training_job`
`name`	yes	Human label; appears in dashboard + as the OpenAI `model` field default
`gpu`	yes	`{"gpu_type": "...", "gpu_count": 1-8}`
`provider`	no	Defaults to `aws` (the only serving provider today)

Validation: When source=training_job, the job must exist, belong to you, and be completed. Otherwise 404 or 400. Response (201):

{
  "object": "deployment",
  "id": "dep_abc123",
  "status": "queued",
  "name": "my-deploy",
  "model": "Qwen/Qwen2.5-0.5B-Instruct",
  "source": "huggingface",
  "endpoint_url": null,
  "gpu": {"type": "A100-80GB", "count": 1},
  "provider": "aws",
  "total_requests": 0,
  "error": null,
  "started_at": null,
  "stopped_at": null,
  "created_at": "2026-05-12T...",
  "updated_at": "2026-05-12T..."
}

endpoint_url populates when status reaches serving.

GET /v1/deployments

List your deployments. Filter by status (queued / provisioning / serving / unhealthy / scaled_to_zero / waking / stopped / failed).

curl "https://api.veri.studio/v1/deployments?status=serving" \
  -H "Authorization: Bearer vk_..."

Returns PaginatedList[DeploymentResponse].

GET /v1/deployments/

Single deployment. Same shape as create response, with timestamps populated as it runs.

PATCH /v1/deployments/

Partial update of the scaling settings on a live deployment. Mutable fields: num_replicas, min_replicas, max_replicas, concurrency_target, scale_to_zero_window_seconds, idle_delete_after_days. Anything else (model, gpu, source, engine, provider) is immutable and fails validation. Returns the updated deployment immediately with the new desired state; the control plane converges the replica fleet in the background. Compare ready_replicas (observed) against num_replicas (desired) to watch convergence.

dep = client.deployments.update(deployment_id, min_replicas=1, max_replicas=4)

GET /v1/deployments//replicas

List the replicas (GPU boxes) behind a deployment: per-replica status, placement, and the latest engine snapshot (in-flight requests, KV-cache usage). Use it to pick a replica_id for the drain endpoint below.

curl "https://api.veri.studio/v1/deployments/dep_abc/replicas" \
  -H "Authorization: Bearer vk_..."

POST /v1/deployments//replicas//drain

Retire one specific replica of a serving deployment. Returns 202 with the replica, now draining (or stopped when it was still booting and was cancelled outright). A replica already taking traffic finishes its in-flight requests before the box is terminated, bounded by the drain timeout. Two modes:

Mode	Effect
Default (scale in)	You pick the victim; the fleet’s desired count shrinks by one and the drained box is not replaced.
`?replace=true` (recycle)	The desired count is unchanged; the control plane launches a replacement box, so capacity is preserved. Use this to rotate out a misbehaving replica.

curl -X POST "https://api.veri.studio/v1/deployments/dep_abc/replicas/rep_123/drain?replace=true" \
  -H "Authorization: Bearer vk_..."

Guards (409): the deployment must be serving; the replica must not already be draining, stopped, or failed; and in default mode the drain may not shrink the fleet below max(min_replicas, 1) or retire the last replica actually serving traffic (use replace=true for those). While autoscaling is armed (min_replicas < max_replicas) the autoscaler may scale back up after a default-mode drain if load warrants it; that is expected. Lower max_replicas to hold a smaller fleet.

POST /v1/deployments//wake

Pre-warm a scaled_to_zero deployment. Returns 202 with the deployment; idempotent (a no-op if it’s already up or booting). A chat request to a parked deployment also wakes it: raw HTTP clients receive 503 with a Retry-After header and error code deployment_waking while the GPU boots, while the Python SDK’s chat() retries through the wake automatically (see the chat endpoint). Fire wake early on a predictive signal to hide the cold start entirely.

client.deployments.wake(deployment_id)

POST /v1/deployments//stop

Transition to stopped. A final billing tick settles the partial-hour spend. Returns the updated deployment.

final = client.deployments.stop(deployment_id)
print(final.status)

POST /v1/deployments//chat/completions

OpenAI-compatible. See OpenAI compatibility for the supported subset of fields. Request:

{
  "model": "my-deploy",
  "messages": [{"role": "user", "content": "Hello"}],
  "temperature": 0.7,
  "max_tokens": 256
}

Response:

{
  "id": "chatcmpl-abc",
  "object": "chat.completion",
  "created": 1715000000,
  "model": "my-deploy",
  "choices": [
    {"index": 0, "message": {"role": "assistant", "content": "..."}, "finish_reason": "stop"}
  ],
  "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}
}

Waking from zero

A chat request to a scaled_to_zero deployment triggers its wake and answers 503 with a Retry-After header while the GPU boots:

{
  "error": {
    "code": "deployment_waking",
    "message": "Deployment was scaled to zero and is waking up; retry shortly"
  }
}

Raw HTTP clients should retry after Retry-After seconds. The Python SDK retries for you: chat() sleeps and re-sends until the deployment serves the request or wake_timeout_s (default 900) elapses, calling on_waking(message) on each 503 so you can surface progress. Set wake_timeout_s=0 to fail fast with the 503 instead.

response = client.deployments.chat(
    deployment_id,
    messages=[{"role": "user", "content": "Hi"}],
    on_waking=lambda msg: print(msg),   # optional progress hook
    wake_timeout_s=900,                 # 0 = fail fast
)

GET /v1/deployments//requests

Per-request log: prompt tokens, completion tokens, latency, status code, error (if any).

curl ".../v1/deployments/dep_abc/requests?limit=50" -H "Authorization: Bearer vk_..."

Response:

{
  "data": [
    {
      "id": "req_xyz",
      "deployment_id": "dep_abc",
      "model_version_id": "mv_1a2b3c4d5e6f",
      "prompt_tokens": 12,
      "completion_tokens": 8,
      "latency_ms": 245.3,
      "status_code": 200,
      "error_message": null,
      "created_at": "2026-05-12T..."
    }
  ],
  "has_more": true
}

Includes failed requests (status_code >= 400) for debugging. model_version_id pins the request to the registered model version that answered it: on a multi-adapter fleet, the version behind the adapter alias the request resolved through (an alias is a moving pointer, so after a production flip the same alias serves different weights); for base-model requests on a fleet built from a registered model, that model’s version. null when nothing served belongs to a model lineage.

GET /v1/deployments//metrics

Aggregated counters across all requests:

{
  "total_requests": 42,
  "total_prompt_tokens": 5230,
  "total_completion_tokens": 1840,
  "avg_latency_ms": 1240.5,
  "error_rate": 0.024,
  "uptime_seconds": 3600.0
}

State machine

queued → provisioning → serving → stopped
                    ↘                    ↘
                     failed             failed
                                ↕
                            unhealthy

unhealthy is a transient state when health checks fail; it returns to serving if the pod recovers. Billing applies during serving and unhealthy only.

Worker-only endpoints

None — deployments don’t have a worker callback API. The serving backend talks to the model server directly via the registered backend interface.

SDK convenience

The SDK wraps the create flow into a Deployment dataclass with bound methods:

dep = client.deployments.create(model="...", source="huggingface", name="...", gpu={...})
dep.wait()                                          # polls until the status settles
response = dep.chat([{"role": "user", "content": "Hi"}])
metrics = dep.metrics()
recent = dep.requests(limit=20)
dep.stop()

These all hit the endpoints above. wait() returns once the status settles: serving, failed, stopped, or scaled_to_zero (a parked deployment won’t move again without traffic, so waiting on it would just burn the timeout). dep.chat() retries through a wake with the default 15-minute budget; call client.deployments.chat() directly to tune wake_timeout_s or on_waking.

OpenAI compatibility

What chat fields are supported.

Billing

Billing states and idle warning.

Hosting & GPU sizing

Pick the right GPU.

Deployment analytics demo

Planned walkthrough for request, token, latency, and replica metrics.

Get started

Training

Deployments

CLI

Volumes

Webhooks

Endpoints

POST /v1/deployments

GET /v1/deployments

GET /v1/deployments/

PATCH /v1/deployments/

GET /v1/deployments//replicas

POST /v1/deployments//replicas//drain

POST /v1/deployments//wake

POST /v1/deployments//stop

POST /v1/deployments//chat/completions

Waking from zero

GET /v1/deployments//requests

GET /v1/deployments//metrics

State machine

Worker-only endpoints

SDK convenience

OpenAI compatibility

Billing

Hosting & GPU sizing

Deployment analytics demo

​Endpoints

​POST /v1/deployments

​GET /v1/deployments

​GET /v1/deployments/

​PATCH /v1/deployments/

​GET /v1/deployments//replicas

​POST /v1/deployments//replicas//drain

​POST /v1/deployments//wake

​POST /v1/deployments//stop

​POST /v1/deployments//chat/completions

​Waking from zero

​GET /v1/deployments//requests

​GET /v1/deployments//metrics

​State machine

​Worker-only endpoints

​SDK convenience

​Related

OpenAI compatibility

Billing

Hosting & GPU sizing

Deployment analytics demo

Endpoints

POST /v1/deployments

GET /v1/deployments

GET /v1/deployments/

PATCH /v1/deployments/

GET /v1/deployments//replicas

POST /v1/deployments//replicas//drain

POST /v1/deployments//wake

POST /v1/deployments//stop

POST /v1/deployments//chat/completions

Waking from zero

GET /v1/deployments//requests

GET /v1/deployments//metrics

State machine

Worker-only endpoints

SDK convenience

Related