Deployment Observability

Veri records terminal telemetry for buffered and streaming inference requests. Use the Analytics dashboard for visual investigation, the SDK or REST API for automation, and MCP when you want an agent to investigate a deployment. This guide is also a smoke test: follow Verify observability end to end after deploying a model to confirm that request telemetry, time-series aggregation, and agent tools are working.

Choose an interface

Interface	Best for
Analytics dashboard	Charts, percentile trends, cost efficiency, engine health, and recent requests
Python SDK	Automated checks, notebooks, alerts, and custom analysis
REST API	Direct integration with any language or monitoring system
Prometheus export	Scraping deployment and replica metrics into your own Prometheus, Grafana, or Datadog for alerting
Veri MCP	Asking an agent to find regressions, errors, or cost changes with bounded context
Official W&B MCP	Deep training-run analysis, artifacts, and W&B reports

MCP is optional. It is useful for agent-driven investigation, but the dashboard, SDK, and REST API expose deployment observability without it.

What you can see, by provider

Request, latency, token, cost, and engine telemetry (queue depth, KV-cache usage) work identically everywhere — they come from the request path and the engine, not the hardware. The GPU hardware snapshot (per-GPU utilization, memory used/total, power draw, temperature, plus aggregates) rides the replica heartbeat and depends on what the serving agent can reach on each provider’s boxes:

Provider	Serving GPUs	Request + engine telemetry	GPU hardware snapshot	Backend
`aws`	A100-80GB, H100-80GB, L4-24GB	Yes	Yes	NVML (vLLM’s vendored binding on the host)
`hotaisle`	MI300X-192GB (ROCm)	Yes	Yes	`amdsmi` bindings, `rocm-smi --json` fallback
`gcp`	A100, H100	Yes	Not yet	Engine runs in docker; the agent cannot reach NVML from outside the container
`digitalocean`	H100-80GB, L40S-48GB	Yes	Not yet	Same docker visibility gap

Two design contracts worth knowing: the snapshot is fail-soft (a box where no GPU library works simply omits the readings — GPU telemetry can never break the heartbeat that liveness monitoring depends on), and it is vendor-neutral in shape — NVIDIA and AMD boxes emit the identical schema, so dashboards and Prometheus queries never branch on vendor.

Verify observability end to end

1. Select a serving deployment

veri deployments list --status serving --format json

Copy the deployment id and export it for the REST examples:

export DEPLOYMENT_ID=dep_...

2. Generate buffered and streaming traffic

Send a regular request with the Veri CLI:

veri deployments chat "$DEPLOYMENT_ID" \
  --message "Reply with exactly: buffered telemetry works"

Then send a streaming request. Replace the model name with the deployment’s configured name:

curl -N "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/chat/completions" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-deployment-name",
    "messages": [{"role": "user", "content": "Reply with exactly: streaming telemetry works"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

include_usage gives Veri authoritative token counts for the stream. Veri requests it automatically when traffic passes through the deployment proxy, but setting it explicitly makes the test self-documenting.

3. Inspect the dashboard

Open Analytics in the Veri dashboard and select the deployment. Choose Past hour, then click refresh. After the two requests finish, expect:

Request and token volume to be non-zero.
The recent requests table to contain both requests.
End-to-end latency samples for successful buffered and streaming requests.
TTFT and TPOT samples for the streaming request.
A zero or near-zero error rate.
Cost-efficiency values once the window contains successful requests with output tokens.

Engine charts populate from replica heartbeats rather than request rows. Queue depth, KV-cache usage, prefix-cache hit rate, generation throughput, GPU utilization, GPU memory, and replica count can therefore appear on a different cadence.

4. Verify with the Python SDK

from datetime import datetime, timedelta, timezone

from veri_sdk import Client

client = Client()
deployment_id = "dep_..."
now = datetime.now(timezone.utc)

lifetime = client.deployments.metrics(deployment_id)
series = client.deployments.metrics_timeseries(
    deployment_id,
    from_time=now - timedelta(hours=1),
    to_time=now,
    interval="auto",
)
errors = client.deployments.requests(
    deployment_id,
    from_time=now - timedelta(hours=1),
    to_time=now,
    errors_only=True,
    limit=50,
)

print(lifetime)
print(series["summary"])
print(f"{len(series['buckets'])} buckets")
print(f"{len(errors)} failed requests")

datetime values are serialized as RFC 3339 timestamps. You can also pass RFC 3339 strings directly. A minimal automated smoke test can assert:

assert lifetime["total_requests"] >= 2
assert series["summary"]["requests"] >= 2
assert series["summary"]["successfulRequests"] >= 2
assert series["summary"]["e2e_ms"]["samples"] >= 2
assert errors == []

5. Verify with REST

Lifetime counters:

curl "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/metrics" \
  -H "Authorization: Bearer $VERI_API_KEY"

Time-series metrics default to the past seven days when from and to are omitted:

curl "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/metrics/timeseries?interval=auto" \
  -H "Authorization: Bearer $VERI_API_KEY"

Only failed requests in a bounded window:

curl --get "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/requests" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  --data-urlencode "errors_only=true" \
  --data-urlencode "limit=50" \
  --data-urlencode "from=2026-07-14T00:00:00Z" \
  --data-urlencode "to=2026-07-14T01:00:00Z"

Scope results to a single replica

The metrics/timeseries, metrics/engine-timeseries, and requests endpoints accept an optional replica_id query parameter. List replica IDs with GET /v1/deployments/$DEPLOYMENT_ID/replicas:

curl --get "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/metrics/timeseries" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  --data-urlencode "interval=auto" \
  --data-urlencode "replica_id=rep_..."

When replica_id is set, buckets, the window summary, engine gauges (queue depth and KV-cache utilization), and recent requests are scoped to that replica. Cost efficiency becomes replica-scoped as well: window cost is the replica’s own active interval multiplied by its per-box GPU rate. Scoped metrics responses echo replica_id at the top level. The ID must be a replica of the deployment; unknown IDs return 404. Omit the parameter to keep deployment-wide aggregates. See the interactive API reference for complete response schemas.

Request traces

Request telemetry (above) records that requests happened; traces additionally capture the full request and response bodies so you can see exactly what was asked and what the model answered. Traces are opt-in per deployment and off by default, because enabling them stores your end users’ prompt and completion content. Opt in at create time:

veri deploy Qwen/Qwen3-4B --trace-bodies                       # 14-day retention
veri deploy Qwen/Qwen3-4B --trace-bodies --trace-retention extended   # 400-day

Or in the SDK: client.deployments.create(..., trace_bodies=True). Browse traces from the deployment page in the dashboard, the CLI, or the API:

veri deployments traces dep_abc --page 1 --limit 20
veri deployments trace dep_abc veri-req000000001

listing = client.deployments.traces(dep.id, page=1, limit=20)
trace = client.deployments.trace(dep.id, listing["data"][0]["id"])
# trace["input"] / trace["output"] are the request and response bodies

Details worth knowing:

Cross-linking. Every trace’s metadata.veri_request_id matches an id from deployments.requests(), so you can join body-level traces with the timing, token, and error telemetry for the same request.
Model version identity. When the request was answered by a registered model version (an adapter alias on a fleet, or a fleet built from a registered model), metadata.model_version_id carries that version’s id — the same value as the request log’s model_version_id — so per-version quality can group traces by identity instead of by a serve-name string that moves on every promote.
Retention. base traces are deleted after 14 days, extended after 400.
Pricing. The first 5,000 base-retention traces per workspace each month are free; beyond that, $2.50 per 1k traces at base retention and$ 5.00 per 1k at extended, metered from credits at month end.
Never in the request path. Capture is asynchronous and drop-tolerant: if the trace store is unavailable, requests are unaffected and the affected traces are dropped.
Streaming. Bodies of streamed (stream: true) responses are not captured yet; those requests still appear in request telemetry.

Export metrics to Prometheus

GET /v1/metrics/export returns your account’s deployment and replica metrics in Prometheus text exposition format (text/plain; version=0.0.4). Authenticate with your API key as a Bearer credential, exactly like every other endpoint. The response covers your deployments only.

curl "https://api.veri.studio/v1/metrics/export" \
  -H "Authorization: Bearer $VERI_API_KEY"

Scrape it on a 30-60 second interval. Values come from replica heartbeat snapshots on roughly that cadence, so scraping faster returns duplicate samples and counts against the API rate limit of 120 requests per minute per key.

Exposed metrics

Per-deployment metrics carry deployment_id and model labels:

Metric	Type	Meaning
`veri_deployment_status_info`	gauge	Always `1`; the `status` label carries the deployment state.
`veri_deployment_replicas_desired`	gauge	Desired replica count.
`veri_deployment_replicas_ready`	gauge	Replicas currently serving traffic.

Per-replica metrics add replica_id, gpu_type, and region labels. Only live replicas (queued, provisioning, serving, unhealthy, or draining) are exported:

Metric	Type	Meaning
`veri_deployment_replica_requests_running`	gauge	Requests actively decoding on the replica.
`veri_deployment_replica_requests_waiting`	gauge	Requests queued on the replica, waiting to be scheduled.
`veri_deployment_replica_kv_cache_usage_ratio`	gauge	GPU KV-cache utilization, 0-1.
`veri_deployment_replica_gpu_utilization_ratio`	gauge	GPU utilization, 0-1, mean across the replica’s GPUs.
`veri_deployment_replica_gpu_memory_used_bytes`	gauge	GPU memory in use, summed across the replica’s GPUs.
`veri_deployment_replica_gpu_memory_total_bytes`	gauge	GPU memory capacity, summed across the replica’s GPUs.
`veri_deployment_replica_metrics_age_seconds`	gauge	Seconds since the replica’s last engine snapshot. Alert on staleness.
`veri_deployment_replica_prompt_tokens_total`	counter	Prompt tokens processed over the replica engine’s lifetime.
`veri_deployment_replica_generation_tokens_total`	counter	Generation tokens produced over the replica engine’s lifetime.

Metrics only appear when the underlying data exists: a replica that has not reported a heartbeat yet exports no series, and GPU series are absent where the agent has no hardware telemetry backend (see What you can see, by provider — NVIDIA boxes use NVML, ROCm boxes use amdsmi/rocm-smi, and docker-isolated providers omit them for now).

Prometheus scrape configuration

prometheus.yml

scrape_configs:
  - job_name: veri
    metrics_path: /v1/metrics/export
    scheme: https
    scrape_interval: 30s
    authorization:
      type: Bearer
      credentials: $VERI_API_KEY
    static_configs:
      - targets: ["api.veri.studio"]

Grafana Cloud

Grafana Alloy forwards the scrape to your Grafana Cloud Prometheus instance:

config.alloy

prometheus.scrape "veri" {
  targets = [{"__address__" = "api.veri.studio", "__scheme__" = "https", "__metrics_path__" = "/v1/metrics/export"}]
  scrape_interval = "30s"
  authorization {
    type        = "Bearer"
    credentials = "$VERI_API_KEY"
  }
  forward_to = [prometheus.remote_write.grafana_cloud.receiver]
}

Datadog agent

Use the OpenMetrics check:

conf.d/openmetrics.d/conf.yaml

instances:
  - openmetrics_endpoint: https://api.veri.studio/v1/metrics/export
    namespace: veri
    metrics: [".*"]
    headers:
      Authorization: Bearer $VERI_API_KEY
    min_collection_interval: 30

Example alert rule

Alert when requests queue up faster than a replica drains them, and when telemetry goes stale:

alerts.yml

groups:
  - name: veri
    rules:
      - alert: VeriReplicaQueueDepthHigh
        expr: veri_deployment_replica_requests_waiting > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Replica {{ $labels.replica_id }} of {{ $labels.deployment_id }} has {{ $value }} queued requests"
      - alert: VeriReplicaMetricsStale
        expr: veri_deployment_replica_metrics_age_seconds > 300
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Replica {{ $labels.replica_id }} has not reported engine metrics for 5+ minutes"

Investigate with MCP

Set up Veri MCP by following Model Context Protocol. In read-only mode, all observability tools remain available; only tools that create, cancel, stop, or rescore resources are removed. Useful deployment tools:

Tool	Use
`veri_get_deployment_metrics`	Compact lifetime counters, percentiles, error rate, uptime, and cost
`veri_query_deployment_metrics`	Bounded time-series buckets with optional metric selection
`veri_list_deployment_requests`	Recent requests, time filters, and error-only investigation
`veri_get_billing_overview`	Balance, current burn, runway, active resources, and recent transactions
`veri_get_cost_summary`	Historical training and deployment costs

Try these prompts:

Inspect deployment dep_... over the past hour. Summarize request volume, p50/p95/p99 TTFT and end-to-end latency, token throughput, error rate, and cost efficiency. Then list failed requests and group them by error category. Do not modify resources.

Compare the past 24 hours with the preceding 24 hours for deployment dep_.... Identify latency, error-rate, throughput, or cost regressions and cite the metric values behind each conclusion.

Review deployment dep_... and my billing overview. Explain whether request volume, output-token efficiency, errors, or the current GPU burn rate is the main cost risk. Do not modify resources.

Agent responses are only as complete as the telemetry window. Include the deployment ID and an explicit period in prompts when you need reproducible results.

Metric semantics

Request timing

Metric	Meaning
TTFB	Request start until upstream response headers arrive
TTFT	Request start until the first non-empty content or reasoning token
E2E latency	Request start until the response body or stream terminates
Generation time	TTFT until stream completion
TPOT	Generation time divided across reported output-token intervals

TTFT and TPOT are most meaningful for streaming responses. TPOT is only calculated when the upstream engine reports usage. Latency percentiles use successful requests so application and transport failures do not distort the serving-latency distribution.

Traffic and errors

Each terminal request record can include:

Prompt, cached prompt, and completion tokens.
Request and response byte counts.
HTTP status and finish reason.
Whether the response streamed and whether usage was reported.
A bounded error category and diagnostic message.
Client cancellation or disconnection state.
Request start and completion timestamps.

Error charts separate client errors from server or interrupted-stream errors. Use request history to inspect the underlying error_category and message.

Cost efficiency

The dashboard and time-series response derive:

Cost per successful request.
Cost per million output tokens.
Tokens per dollar.
Total deployment cost for the selected window.

Cost-per-token values require reported completion tokens. A request can be successful while lacking usage, so compare usage_reported when investigating missing efficiency data.

Engine health

Replica heartbeats provide:

Running and waiting requests.
KV-cache utilization.
Prefix-cache hit rate.
Generation token throughput.
Active replica count.
GPU hardware telemetry: utilization, memory used and total, power draw, and temperature.

Request telemetry and engine telemetry are intentionally separate. Request rows describe completed client operations; engine history describes sampled replica state.

GPU hardware telemetry

Each replica reports NVML readings with its heartbeat. GET /v1/deployments/$DEPLOYMENT_ID/replicas returns the latest snapshot per replica:

Field	Meaning
`gpu_utilization`	Mean utilization percent (0-100) across the replica’s GPUs
`gpu_memory_used_bytes`	GPU memory in use, summed across the replica’s GPUs
`gpu_memory_total_bytes`	GPU memory capacity, summed across the replica’s GPUs
`gpu_power_w`	Power draw in watts, summed across the replica’s GPUs
`gpu_temperature_c`	Hottest GPU on the replica, degrees Celsius
`gpu_devices`	Per-GPU array behind the aggregates (`index`, `utilization`, `memory_used_bytes`, `memory_total_bytes`, `power_w`, `temperature_c`)

The metrics/engine-timeseries endpoint adds gpu_utilization (mean across reporting replicas) and gpu_memory_used_bytes (summed) to each bucket and to each per-replica series. All GPU fields are absent or null when a replica does not report telemetry, for example on hardware without NVML. GPU utilization is a debugging metric, not a scaling signal: it measures the share of time any kernel was resident on the chip, not how much load the engine is under. Scale on queue depth (waiting requests) and KV-cache utilization instead.

Troubleshooting

Symptom	Check
No requests in charts	Confirm the deployment ID, time range, and that requests reached the Veri deployment URL
E2E exists but TTFT/TPOT is empty	Use streaming; confirm the stream emitted content and reported usage
Tokens or cost efficiency are empty	Request streaming usage and inspect `usage_reported`
Errors appear but latency percentiles do not change	Expected: percentiles include successful requests only
Engine charts are empty	Wait for replica heartbeats and confirm the deployment is serving
A stream has `client_cancelled=true`	The downstream client stopped reading before the upstream stream completed
MCP cannot create, stop, or cancel a resource	The server is read-only; observability queries still work

Where to go next

MCP setup

Let an agent query deployment, training, and billing context.

Deployment commands

Generate traffic, benchmark an endpoint, and inspect recent requests.

API reference

Integrate metrics and request history into your own systems.

Deployment billing

Understand hourly GPU billing and scale-to-zero behavior.

Get started

Training

Deployments

CLI

Volumes

Webhooks

Choose an interface

What you can see, by provider

Verify observability end to end

1. Select a serving deployment

2. Generate buffered and streaming traffic

3. Inspect the dashboard

4. Verify with the Python SDK

5. Verify with REST

Scope results to a single replica

Request traces

Export metrics to Prometheus

Exposed metrics

Prometheus scrape configuration

Grafana Cloud

Datadog agent

Example alert rule

Investigate with MCP

Metric semantics

Request timing

Traffic and errors

Cost efficiency

Engine health

GPU hardware telemetry

Troubleshooting

Where to go next

MCP setup

Deployment commands

API reference

Deployment billing

​Choose an interface

​What you can see, by provider

​Verify observability end to end

​1. Select a serving deployment

​2. Generate buffered and streaming traffic

​3. Inspect the dashboard

​4. Verify with the Python SDK

​5. Verify with REST

​Scope results to a single replica

​Request traces

​Export metrics to Prometheus

​Exposed metrics

​Prometheus scrape configuration

​Grafana Cloud

​Datadog agent

​Example alert rule

​Investigate with MCP

​Metric semantics

​Request timing

​Traffic and errors

​Cost efficiency

​Engine health

​GPU hardware telemetry

​Troubleshooting

​Where to go next

MCP setup

Deployment commands

API reference

Deployment billing

Choose an interface

What you can see, by provider

Verify observability end to end

1. Select a serving deployment

2. Generate buffered and streaming traffic

3. Inspect the dashboard

4. Verify with the Python SDK

5. Verify with REST

Scope results to a single replica

Request traces

Export metrics to Prometheus

Exposed metrics

Prometheus scrape configuration

Grafana Cloud

Datadog agent

Example alert rule

Investigate with MCP

Metric semantics

Request timing

Traffic and errors

Cost efficiency

Engine health

GPU hardware telemetry

Troubleshooting

Where to go next