Create deployment

What to serve: a completed training job id (source training_job), a Hugging Face repo id (huggingface), or a saved model id (custom_model).

Where model comes from: "training_job" (the default), "huggingface" or "custom_model".

Inference engine. Only "vllm" (the default) is supported; any other value is rejected with a 400. Omitted => vllm.

Number of GPU-box replicas to launch. Omitted => 1 (a single box). Validated 1..=8 at create. Sugar for min_replicas == max_replicas == N (the CLI --replicas flag).

Replica-count bounds. min == max => fixed N (autoscaling off; there is no boolean flag). Omitted => both default to num_replicas (fixed N). min = 0 enables scale to zero: the fleet parks after the idle window and wakes on the next request.

Per-replica concurrency setpoint the autoscaler targets: desired = ceil(total_ongoing / concurrency_target), clamped to [max(min_replicas, 1), max_replicas]. Stored as target_ongoing_requests. Omitted => 8 (throughput-biased; lower it for latency-sensitive chat).

Idle seconds before a min_replicas=0 deployment parks (scaled_to_zero). Omitted => 3600. Minimum 300.

Optional cleanup: a deployment parked at zero with no traffic for this many days goes terminal (stopped). Omitted => kept forever.

How many times Veri replaces the GPU box when the deployment's LAST box dies after it has served (crash or host failure), 0..=5. Replacements go out 0, 10, 20, 40 and 80s after the previous launch. Used up (or 0): a min_replicas >= 1 deployment goes 'failed' with failure_code replica_lost, a min_replicas = 0 one parks. Losing some boxes of a multi-replica fleet is not governed by this. Omitted => 3.

HF cache-through (source=huggingface only): also save the downloaded snapshot to the caller's model library so later deploys of this repo start from S3 instead of the HF hub. Best-effort: a failed upload never affects this deployment. Omitted => no caching.

Explicit vLLM serving image (e.g. "vllm/vllm-openai:v0.25.1") for architectures newer than the platform default vLLM. Official upstream images only (vllm/vllm-openai[-rocm]:). vllm engine only. Forces the docker launcher on every provider, including AWS. Omitted => platform default.

Opt-in vLLM startup diagnostics: the serving agent exports VLLM_LOGGING_LEVEL=DEBUG + VLLM_GC_DEBUG=1 to the engine, records startup phase timings, and reports them on the first healthy heartbeat (or in the failure detail with a deeper stderr tail). Default false: diagnostic logging slows inference, so use for diagnosis launches, not steady customer traffic.

Request traces: capture this deployment's chat request/response bodies and make them queryable via /v1/deployments/{id}/traces. Default false (privacy: enabling stores end-user prompt/completion content, retained per trace_retention). Requires the traces feature; billed per 1k traces beyond the monthly free allowance.

Trace retention class: omitted/null = "base" (14-day retention, base rate), "extended" = 400-day retention at the extended rate. Only meaningful with trace_bodies=true.

"single" (default) or "multi_adapter": a fleet serving the base model plus N hot-loaded LoRA adapters attached via /v1/deployments/{id}/adapters.

vLLM --max-loras (GPU-resident adapter cap) for multi_adapter deployments. Omitted => the default (32).

vLLM --max-lora-rank for multi_adapter deployments. Omitted => the default (64).

vLLM --max-cpu-loras (CPU-offload headroom) for multi_adapter deployments. Omitted => the default (128).

For a custom_model deployment, the saved model this was created from (provenance). Also set on a source=huggingface deployment that took the transparent cache hit (serving from the caller's cached S3 copy). Null otherwise.

HF cache-through: the library model (status importing) this deployment's box is uploading its snapshot into. Null unless the deployment was created with cache=true and the upload is the one this boot owns.

The URL to call once the deployment serves. Managed deployments (aws, gcp, digitalocean) get the platform chat URL {api}/v1/deployments/{id}/chat/completions, authenticated with your Veri API key; BYOC (huggingface) deployments carry the endpoint URL under your own HF account. Null until the deployment first serves, and null while it is parked (scaled_to_zero) until a wake serves again; it stays set after stop or failure.

BYOC (provider='huggingface'): the HF hub repository this endpoint serves under the owner's account. Null for every non-BYOC deployment.

The explicit vLLM serving image this deployment was created with. Null unless supplied at create.

Whether this deployment was created with startup diagnostics on.

Whether request/response bodies are captured as traces.

Trace retention class ("extended" or null = base). Null unless trace_bodies deployments set it.

Startup phase timings reported by a profiled deployment's first healthy heartbeat (seconds: model_sync_s, engine_start_to_healthy_s, total_s). Null unless startup_profiling was set and the engine reached healthy.

Desired replica count (GPU boxes) behind this deployment. The observed count is ready_replicas; the CLI renders "ready/desired" (e.g. 2/3).

Observed: replicas currently serving with a routable endpoint.

Replica-count bounds. min == max => fixed N (autoscaling off).

Per-replica concurrency setpoint (stored as target_ongoing_requests).

Idle window before a min=0 deployment parks (None = 3600s).

Days a parked deployment is kept before it is stopped (None = kept forever).

How many times Veri replaces the GPU box when the deployment's last box dies after it has served, 0..=5 (default 3).

Machine-readable reason for the last failure, e.g. watchdog_timeout, CAPACITY_UNAVAILABLE, replica_lost (the last GPU box was lost after serving: while 'provisioning' Veri is replacing it, with 'failed' or 'scaled_to_zero' self_heal_attempts was used up or 0), or the serving agent's oom / model_source_unsupported. Null when no failure is recorded; cleared with error on an idle park and on wake. A failed wake re-parks with its code (e.g. wake_boot_failed).

When the most recent wake of this scale-to-zero deployment started: a chat request to the parked deployment, POST /wake, or a PATCH that left a parked deployment with min_replicas >= 1. Kept after the wake serves or the deployment parks again. Null if it never woke.

Consecutive failed wake boots since the deployment last reached serving (0 when the last boot served). A failed wake re-parks the deployment (scaled_to_zero, still wakeable) and adds 1; reaching serving resets it to 0. Starting a new wake does not reset it.

After a failed wake: the earliest time a chat request triggers the next wake (backoff of 60s doubling per consecutive failure, up to 1h). POST /wake and a PATCH that raises min_replicas to 1 or more bypass it. Null until a wake fails; cleared when a wake reaches serving. A time in the past means the backoff has elapsed.