Deployment Observability - Veri Documentation

Veri records terminal telemetry for buffered and streaming inference requests. Use the Analytics dashboard for visual investigation, the SDK or REST API for automation, and MCP when you want an agent to investigate a deployment. This guide is also a smoke test: follow Verify observability end to end after deploying a model to confirm that request telemetry, time-series aggregation, and agent tools are working.

Choose an interface

Interface	Best for
Analytics dashboard	Charts, percentile trends, cost efficiency, engine health, and recent requests
Python SDK	Automated checks, notebooks, alerts, and custom analysis
REST API	Direct integration with any language or monitoring system
Prometheus export	Scraping deployment and replica metrics into your own Prometheus, Grafana, or Datadog for alerting
Veri MCP	Asking an agent to find regressions, errors, or cost changes with bounded context
Official W&B MCP	Deep training-run analysis, artifacts, and W&B reports

MCP is optional. It is useful for agent-driven investigation, but the dashboard, SDK, and REST API expose deployment observability without it.

What you can see, by provider

Request, latency, token, cost, and engine telemetry (queue depth, KV-cache usage) work identically everywhere — they come from the request path and the engine, not the hardware. The GPU hardware snapshot (per-GPU utilization, memory used/total, power draw, temperature, plus aggregates) rides the replica heartbeat and depends on what the serving agent can reach on each provider’s boxes:

Provider	Serving GPUs	Request + engine telemetry	GPU hardware snapshot	Backend
`aws`	A100-80GB, H100-80GB, L4-24GB, A10G-24GB	Yes	Yes	NVML (vLLM’s vendored binding on the host)
`gcp`	A100-80GB	Yes	Not yet	Engine runs in docker; the agent cannot reach NVML from outside the container
`digitalocean`	H100-80GB, L40S-48GB, MI300X-192GB	Yes	Not yet	Same docker visibility gap

GCP and DigitalOcean serving are not enabled yet (see Serving providers); their rows describe what those providers will report. Two design contracts worth knowing: the snapshot is fail-soft (a box where no GPU library works simply omits the readings — GPU telemetry can never break the heartbeat that liveness monitoring depends on), and it is vendor-neutral in shape — NVIDIA and AMD boxes emit the identical schema, so dashboards and Prometheus queries never branch on vendor.

Verify observability end to end

1. Select a serving deployment

veri deployments list --status serving --format json

Copy the deployment id and export it for the REST examples:

export DEPLOYMENT_ID=3f9c2a7d1b04  # the id from the list above

2. Generate buffered and streaming traffic

Send a regular request with the Veri CLI:

veri deployments chat "$DEPLOYMENT_ID" \
  --message "Reply with exactly: buffered telemetry works"

Then send a streaming request. Replace the model name with the deployment’s configured name:

curl -N "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/chat/completions" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-deployment-name",
    "messages": [{"role": "user", "content": "Reply with exactly: streaming telemetry works"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Veri meters the stream’s token counts server-side whether or not you set include_usage. Setting it means your client also receives the final usage-only chunk (empty choices, populated usage), which makes the test self-documenting.

3. Inspect the dashboard

Open Analytics in the Veri dashboard and select the deployment. Choose Past hour, then click refresh. After the two requests finish, expect:

Request and token volume to be non-zero.
The recent requests table to contain both requests.
End-to-end latency samples for successful buffered and streaming requests.
TTFT and TPOT samples for the streaming request.
A zero or near-zero error rate.
Cost-efficiency values once the window contains successful requests with output tokens.

Engine charts populate from replica heartbeats rather than request rows. Queue depth, KV-cache usage, prefix-cache hit rate, generation throughput, GPU utilization, GPU memory, and replica count can therefore appear on a different cadence.

4. Verify with the Python SDK

from datetime import datetime, timedelta, timezone

from veri_sdk import Client

client = Client()
deployment_id = "<deployment-id>"
now = datetime.now(timezone.utc)

lifetime = client.deployments.metrics(deployment_id)
series = client.deployments.metrics_timeseries(
    deployment_id,
    from_time=now - timedelta(hours=1),
    to_time=now,
    interval="auto",
)
errors = client.deployments.requests(
    deployment_id,
    from_time=now - timedelta(hours=1),
    to_time=now,
    errors_only=True,
    limit=50,
)

print(lifetime)
print(series["summary"])
print(f"{len(series['buckets'])} buckets")
print(f"{len(errors)} failed requests")

datetime values are serialized as RFC 3339 timestamps. You can also pass RFC 3339 strings directly. A minimal automated smoke test can assert:

assert lifetime["total_requests"] >= 2
assert series["summary"]["requests"] >= 2
assert series["summary"]["successfulRequests"] >= 2
assert series["summary"]["e2e_ms"]["samples"] >= 2
assert errors == []

5. Verify with REST

Lifetime counters, plus latency and error stats for the past 24 hours:

curl "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/metrics" \
  -H "Authorization: Bearer $VERI_API_KEY"

total_requests, total_prompt_tokens, and total_completion_tokens count every request you sent since the deployment was created (eval calls excluded). avg_latency_ms, p50_latency_ms through p99_latency_ms, and error_rate cover the past 24 hours only (null latency when there were no requests in that window). For any other window, use metrics/timeseries, whose summary carries the same stats for the range you ask for. Time-series metrics default to the past seven days when from and to are omitted:

curl "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/metrics/timeseries?interval=auto" \
  -H "Authorization: Bearer $VERI_API_KEY"

Only failed requests in a bounded window:

curl --get "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/requests" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  --data-urlencode "errors_only=true" \
  --data-urlencode "limit=50" \
  --data-urlencode "from=2026-07-14T00:00:00Z" \
  --data-urlencode "to=2026-07-14T01:00:00Z"

Scope results to a single replica

The metrics/timeseries, metrics/engine-timeseries, and requests endpoints accept an optional replica_id query parameter. List replica IDs with GET /v1/deployments/$DEPLOYMENT_ID/replicas:

curl --get "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/metrics/timeseries" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  --data-urlencode "interval=auto" \
  --data-urlencode "replica_id=$REPLICA_ID"

When replica_id is set, buckets, the window summary, engine gauges (queue depth and KV-cache utilization), and recent requests are scoped to that replica. Cost efficiency becomes replica-scoped as well: window cost is the replica’s own active interval multiplied by its per-box GPU rate. Without replica_id, window cost is the sum of every replica’s billed time in the window. Each bucket also carries cost_usd, the GPU cost billed inside that bucket, so cost lands where the boxes actually ran. Scoped metrics responses echo replica_id at the top level. The ID must be a replica of the deployment; unknown IDs return 404. Omit the parameter to keep deployment-wide aggregates.

Evaluation traffic

Requests that Veri evaluators and experiments send to a deployment are recorded with origin eval and kept out of your production numbers. The lifetime counters, metrics, metrics/timeseries, the dashboard charts, and the default requests listing count only your own API traffic. List the eval calls with origin=eval:

curl --get "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/requests" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  --data-urlencode "origin=eval"

Engine charts (queue depth, KV cache, throughput, GPU readings) come from the GPU box itself, so they include every request it served, eval calls too. Eval calls also count as activity for scale to zero, so a running experiment keeps the deployment awake. Lifetime counters still include eval calls made before 2026-10-04, when this split began.

Retention

Data	Kept for
Per-request telemetry (`requests` rows, which feed `metrics` and `metrics/timeseries`)	90 days; 400 days on deployments with `trace_retention: extended`
Engine samples (`metrics/engine-timeseries`)	30 days, the longest range the charts accept
Lifetime counters (`total_requests` and token totals)	The life of the deployment

Older rows are deleted automatically. Deleting a deployment removes all of it at once. See the interactive API reference for complete response schemas.

Request traces

Request telemetry (above) records that requests happened; traces additionally capture the full request and response bodies so you can see exactly what was asked and what the model answered. Traces are opt-in per deployment and off by default, because enabling them stores your end users’ prompt and completion content. Opt in at create time:

veri deploy Qwen/Qwen3-4B --trace-bodies                       # 14-day retention
veri deploy Qwen/Qwen3-4B --trace-bodies --trace-retention extended   # 400-day

Or in the SDK: client.deployments.create(..., trace_bodies=True). Browse traces from the deployment page in the dashboard, the CLI, or the API:

veri deployments traces $DEPLOYMENT_ID --page 1 --limit 20
veri deployments trace $DEPLOYMENT_ID veri-9b1f2c4a7d01

listing = client.deployments.traces(dep.id, page=1, limit=20)
trace = client.deployments.trace(dep.id, listing["data"][0]["id"])
# trace["input"] / trace["output"] are the request and response bodies

Details worth knowing:

Cross-linking. Every trace’s metadata.veri_request_id matches an id from deployments.requests(), so you can join body-level traces with the timing, token, and error telemetry for the same request.
Model version identity. When the request was answered by a registered model version (an adapter alias on a multi-adapter fleet (beta), or a fleet built from a registered model), metadata.model_version_id carries that version’s id — the same value as the request log’s model_version_id — so per-version quality can group traces by identity instead of by a serve-name string that moves on every promote.
Retention. base traces are deleted after 14 days, extended after 400.
Pricing. The first 5,000 base-retention traces per workspace each month are free; beyond that, $2.50 per 1k traces at base retention and $5.00 per 1k at extended, metered from credits at month end.
Never in the request path. Capture is asynchronous and drop-tolerant: if the trace store is unavailable, requests are unaffected and the affected traces are dropped.
Streaming. Streamed (stream: true) responses are captured once the stream completes: content, reasoning, tool calls, and finish reason are assembled into one response. A stream that is cut off, errors, or exceeds 1 MiB is not captured; the request still appears in request telemetry.
Threads. Send the same X-Veri-Thread-Id header (1-128 characters of A-Z a-z 0-9 . _ : -) on every request of a conversation to group its traces into a thread; the SDK’s session_id sends the equivalent X-Veri-Session-Id. The thread id is stored on the request and in trace metadata, and thread evaluators score the whole conversation. The dashboard and API views of threads are in Agent conversations.
Scores. Each trace carries its evaluator scores (from your own harness, experiments, or human review) under scores.
Traces to training data. A time window of traces can be imported into a dataset stream as chat or prompt rows (veri datasets import-traces), snapshotted, and trained on — production traffic feeding the next fine-tune.

Agent conversations

When an agent calls a deployment, two optional headers let Veri group its requests into conversations:

Header	Value	Effect
`X-Veri-Thread-Id`	1-128 characters of `A-Z a-z 0-9 . _ : -`	Groups the requests of one conversation into a thread. Send the same value on every turn.
`X-Veri-Agent`	1-64 characters of the same set	Names the agent the conversation belongs to. Threads sent without it are grouped under their deployment.

A header value outside these rules is rejected with 400 before the request reaches the model, for example {"error": {"message": "X-Veri-Agent must be 1-64 characters of [A-Za-z0-9._:-]"}}. The SDK’s session_id sends X-Veri-Session-Id, which also counts as the thread id when it follows the same rules.

from openai import OpenAI

oai = OpenAI(
    base_url=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
    default_headers={"X-Veri-Agent": "support-bot"},
)
response = oai.chat.completions.create(
    model=dep.name,
    messages=[{"role": "user", "content": "Where is my order?"}],
    extra_headers={"X-Veri-Thread-Id": "conv-42"},
)

Open Agents in the dashboard (Overview, Conversations, and Agents tabs) to browse them, or read the same data from the API. Each agent has a key: agent:<X-Veri-Agent value>, deployment:<deployment id> for threads sent without an agent name, or env:<environment id> for a managed agent. Eval calls are left out.

Endpoint	Returns
`GET /v1/observability/overview`	Totals, a zero-filled time series (each bucket also carries `tool_errors`, the failed tool calls of managed agent sessions), live counts, and the top tools, agents, and models. Filter with `agent` (a key) or `deployment_id`.
`GET /v1/observability/agents`	One entry per agent: conversations, turns, model and tool calls, tokens, error rate, and a 14-day daily conversation count.
`GET /v1/observability/conversations`	Threads (and managed agent sessions) with activity in the window, newest first. Filters: `agent`, `deployment_id`, `kind` (`thread` or `session`), `active`, `errors_only`, `q` (text search), `tool` and `model` (exact names, as the overview’s top tools and top models list them; at most 200 characters). Page with `limit` (1-100, default 50) and `after` (the previous page’s `next_cursor`).
`GET /v1/deployments/{deployment_id}/threads/{thread_id}`	One thread, turn by turn: every model call and the tool calls between calls. Message text comes from stored traces; `bodies` is `none` when the deployment does not capture them, and `unavailable` when the trace store did not answer (the turns still come from the request records; retry for the text).

The window is from and to (RFC 3339), by default the past 24 hours, at most 30 days, starting no earlier than 2000-01-01. A next_cursor keeps the first page’s window end, so page on without to (or repeat the same to): a different to with a cursor is a 400. To turn one conversation into a dataset row (the dashboard’s Add to dataset), call POST /v1/datasets/{dataset_id}/import-traces with deployment and thread_id. Importing the same conversation into the same dataset again while its newest stored message is unchanged returns 409 export_thread_duplicate, naming the earlier import; send "force": true to append it again.

curl --get "https://api.veri.studio/v1/observability/conversations" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  --data-urlencode "kind=thread" \
  --data-urlencode "limit=20"

curl "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/threads/conv-42" \
  -H "Authorization: Bearer $VERI_API_KEY"

Export metrics to Prometheus

GET /v1/metrics/export returns your account’s deployment and replica metrics in Prometheus text exposition format (text/plain; version=0.0.4). Authenticate with your API key as a Bearer credential, exactly like every other endpoint. The response covers your deployments only.

curl "https://api.veri.studio/v1/metrics/export" \
  -H "Authorization: Bearer $VERI_API_KEY"

Scrape it on a 30-60 second interval. Values come from replica heartbeat snapshots on roughly that cadence, so scraping faster returns duplicate samples and counts against the API rate limit of 120 requests per minute per key.

Exposed metrics

Per-deployment metrics carry deployment_id and model labels:

Metric	Type	Meaning
`veri_deployment_status_info`	gauge	Always `1`; the `status` label carries the deployment state.
`veri_deployment_replicas_desired`	gauge	Desired replica count.
`veri_deployment_replicas_ready`	gauge	Replicas currently serving traffic.

Per-replica metrics add replica_id, gpu_type, and region labels. Only live replicas (queued, provisioning, serving, or draining) are exported:

Metric	Type	Meaning
`veri_deployment_replica_requests_running`	gauge	Requests actively decoding on the replica.
`veri_deployment_replica_requests_waiting`	gauge	Requests queued on the replica, waiting to be scheduled.
`veri_deployment_replica_kv_cache_usage_ratio`	gauge	GPU KV-cache utilization, 0-1.
`veri_deployment_replica_gpu_utilization_ratio`	gauge	GPU utilization, 0-1, mean across the replica’s GPUs.
`veri_deployment_replica_gpu_memory_used_bytes`	gauge	GPU memory in use, summed across the replica’s GPUs.
`veri_deployment_replica_gpu_memory_total_bytes`	gauge	GPU memory capacity, summed across the replica’s GPUs.
`veri_deployment_replica_metrics_age_seconds`	gauge	Seconds since the replica’s last engine snapshot. Alert on staleness.
`veri_deployment_replica_prompt_tokens_total`	counter	Prompt tokens processed over the replica engine’s lifetime.
`veri_deployment_replica_generation_tokens_total`	counter	Generation tokens produced over the replica engine’s lifetime.

Metrics only appear when the underlying data exists: a replica that has not reported a heartbeat yet exports no series, and GPU series are absent where the agent has no hardware telemetry backend (see What you can see, by provider — NVIDIA boxes use NVML, ROCm boxes use amdsmi/rocm-smi, and docker-isolated providers omit them for now).

Prometheus scrape configuration

prometheus.yml

scrape_configs:
  - job_name: veri
    metrics_path: /v1/metrics/export
    scheme: https
    scrape_interval: 30s
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/veri_api_key  # a file holding your API key
    static_configs:
      - targets: ["api.veri.studio"]

Prometheus does not expand environment variables in its config file, so keep the key in a file and point credentials_file at it.

Grafana Cloud

Grafana Alloy forwards the scrape to your Grafana Cloud Prometheus instance:

config.alloy

prometheus.scrape "veri" {
  targets = [{"__address__" = "api.veri.studio", "__scheme__" = "https", "__metrics_path__" = "/v1/metrics/export"}]
  scrape_interval = "30s"
  authorization {
    type        = "Bearer"
    credentials = sys.env("VERI_API_KEY")
  }
  forward_to = [prometheus.remote_write.grafana_cloud.receiver]
}

Datadog agent

Use the OpenMetrics check:

conf.d/openmetrics.d/conf.yaml

instances:
  - openmetrics_endpoint: https://api.veri.studio/v1/metrics/export
    namespace: veri
    metrics: [".*"]
    headers:
      Authorization: Bearer $VERI_API_KEY
    min_collection_interval: 30

Example alert rule

Alert when requests queue up faster than a replica drains them, and when telemetry goes stale:

alerts.yml

groups:
  - name: veri
    rules:
      - alert: VeriReplicaQueueDepthHigh
        expr: veri_deployment_replica_requests_waiting > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Replica {{ $labels.replica_id }} of {{ $labels.deployment_id }} has {{ $value }} queued requests"
      - alert: VeriReplicaMetricsStale
        expr: veri_deployment_replica_metrics_age_seconds > 300
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Replica {{ $labels.replica_id }} has not reported engine metrics for 5+ minutes"

Investigate with MCP

Set up Veri MCP by following Model Context Protocol. In read-only mode, all observability tools remain available; only tools that create, cancel, stop, or rescore resources are removed. Useful deployment tools:

Tool	Use
`veri_get_deployment_metrics`	Compact lifetime counters, percentiles, error rate, uptime, and cost
`veri_query_deployment_metrics`	Bounded time-series buckets with optional metric selection
`veri_list_deployment_requests`	Recent requests, time filters, and error-only investigation
`veri_get_billing_overview`	Balance, current burn, runway, active resources, and recent transactions
`veri_get_cost_summary`	Historical training and deployment costs

Try these prompts:

Inspect deployment <deployment-id> over the past hour. Summarize request volume, p50/p95/p99 TTFT and end-to-end latency, token throughput, error rate, and cost efficiency. Then list failed requests and group them by error category. Do not modify resources.

Compare the past 24 hours with the preceding 24 hours for deployment <deployment-id>. Identify latency, error-rate, throughput, or cost regressions and cite the metric values behind each conclusion.

Review deployment <deployment-id> and my billing overview. Explain whether request volume, output-token efficiency, errors, or the current GPU burn rate is the main cost risk. Do not modify resources.

Agent responses are only as complete as the telemetry window. Include the deployment ID and an explicit period in prompts when you need reproducible results.

Metric semantics

Request timing

Metric	Meaning
TTFB	Request start until upstream response headers arrive
TTFT	Request start until the first non-empty content or reasoning token
E2E latency	Request start until the response body or stream terminates
Generation time	TTFT until stream completion
TPOT	Generation time divided across reported output-token intervals

TTFT and TPOT are most meaningful for streaming responses. TPOT is only calculated when the upstream engine reports usage. Latency percentiles use successful requests so application and transport failures do not distort the serving-latency distribution.

Traffic and errors

Each terminal request record can include:

Prompt, cached prompt, and completion tokens.
Request and response byte counts.
HTTP status and finish reason.
Whether the response streamed and whether usage was reported.
A bounded error category and diagnostic message.
Client cancellation or disconnection state.
Request start and completion timestamps.

Error charts separate client errors from server or interrupted-stream errors. Use request history to inspect the underlying error_category and message.

Cost efficiency

The dashboard and time-series response derive:

Cost per successful request.
Cost per million output tokens.
Tokens per dollar.
Total deployment cost for the selected window.

Cost-per-token values require reported completion tokens. A request can be successful while lacking usage, so compare usage_reported when investigating missing efficiency data.

Engine health

Replica heartbeats provide:

Running and waiting requests.
KV-cache utilization.
Prefix-cache hit rate.
Generation token throughput.
Active replica count.
GPU hardware telemetry: utilization, memory used and total, power draw, and temperature.

Request telemetry and engine telemetry are intentionally separate. Request rows describe completed client operations; engine history describes sampled replica state.

GPU hardware telemetry

Each replica reports NVML readings with its heartbeat. GET /v1/deployments/$DEPLOYMENT_ID/replicas returns the latest snapshot per replica:

Field	Meaning
`gpu_utilization`	Mean utilization percent (0-100) across the replica’s GPUs
`gpu_memory_used_bytes`	GPU memory in use, summed across the replica’s GPUs
`gpu_memory_total_bytes`	GPU memory capacity, summed across the replica’s GPUs
`gpu_power_w`	Power draw in watts, summed across the replica’s GPUs
`gpu_temperature_c`	Hottest GPU on the replica, degrees Celsius
`gpu_devices`	Per-GPU array behind the aggregates (`index`, `utilization`, `memory_used_bytes`, `memory_total_bytes`, `power_w`, `temperature_c`)

The metrics/engine-timeseries endpoint adds gpu_utilization (mean across reporting replicas) and gpu_memory_used_bytes (summed) to each bucket and to each per-replica series. All GPU fields are absent or null when a replica does not report telemetry, for example on hardware without NVML. GPU utilization is a debugging metric, not a scaling signal: it measures the share of time any kernel was resident on the chip, not how much load the engine is under. Scale on queue depth (waiting requests) and KV-cache utilization instead.

Troubleshooting

Symptom	Check
No requests in charts	Confirm the deployment ID, time range, and that requests reached the Veri deployment URL
E2E exists but TTFT/TPOT is empty	Use streaming; confirm the stream emitted content and reported usage
Tokens or cost efficiency are empty	Request streaming usage and inspect `usage_reported`
Errors appear but latency percentiles do not change	Expected: percentiles include successful requests only
Engine charts are empty	Wait for replica heartbeats and confirm the deployment is serving
A stream has `client_cancelled=true`	The downstream client stopped reading before the upstream stream completed
MCP cannot create, stop, or cancel a resource	The server is read-only; observability queries still work

Where to go next

MCP setup

Let an agent query deployment, training, and billing context.

Deployment commands

Generate traffic, benchmark an endpoint, and inspect recent requests.

API reference

Integrate metrics and request history into your own systems.

Deployment billing

Understand hourly GPU billing and scale-to-zero behavior.

​Choose an interface

​What you can see, by provider

​Verify observability end to end

​1. Select a serving deployment

​2. Generate buffered and streaming traffic

​3. Inspect the dashboard

​4. Verify with the Python SDK

​5. Verify with REST

​Scope results to a single replica

​Evaluation traffic

​Retention

​Request traces

​Agent conversations

​Export metrics to Prometheus

​Exposed metrics

​Prometheus scrape configuration

​Grafana Cloud

​Datadog agent

​Example alert rule

​Investigate with MCP

​Metric semantics

​Request timing

​Traffic and errors

​Cost efficiency

​Engine health

​GPU hardware telemetry

​Troubleshooting

​Where to go next