Deployment Observability

Veri records terminal telemetry for buffered and streaming inference requests. Use the Analytics dashboard for visual investigation, the SDK or REST API for automation, and MCP when you want an agent to investigate a deployment. This guide is also a smoke test: follow Verify observability end to end after deploying a model to confirm that request telemetry, time-series aggregation, and agent tools are working.

Choose an interface

Interface	Best for
Analytics dashboard	Charts, percentile trends, cost efficiency, engine health, and recent requests
Python SDK	Automated checks, notebooks, alerts, and custom analysis
REST API	Direct integration with any language or monitoring system
Veri MCP	Asking an agent to find regressions, errors, or cost changes with bounded context
Official W&B MCP	Deep training-run analysis, artifacts, and W&B reports

MCP is optional. It is useful for agent-driven investigation, but the dashboard, SDK, and REST API expose deployment observability without it.

Verify observability end to end

1. Select a serving deployment

veri deployments list --status serving --format json

Copy the deployment id and export it for the REST examples:

export DEPLOYMENT_ID=dep_...

2. Generate buffered and streaming traffic

Send a regular request with the Veri CLI:

veri deployments chat "$DEPLOYMENT_ID" \
  --message "Reply with exactly: buffered telemetry works"

Then send a streaming request. Replace the model name with the deployment’s configured name:

curl -N "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/chat/completions" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-deployment-name",
    "messages": [{"role": "user", "content": "Reply with exactly: streaming telemetry works"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

include_usage gives Veri authoritative token counts for the stream. Veri requests it automatically when traffic passes through the deployment proxy, but setting it explicitly makes the test self-documenting.

3. Inspect the dashboard

Open Analytics in the Veri dashboard and select the deployment. Choose Past hour, then click refresh. After the two requests finish, expect:

Request and token volume to be non-zero.
The recent requests table to contain both requests.
End-to-end latency samples for successful buffered and streaming requests.
TTFT and TPOT samples for the streaming request.
A zero or near-zero error rate.
Cost-efficiency values once the window contains successful requests with output tokens.

Engine charts populate from replica heartbeats rather than request rows. Queue depth, KV-cache usage, prefix-cache hit rate, generation throughput, and replica count can therefore appear on a different cadence.

4. Verify with the Python SDK

from datetime import datetime, timedelta, timezone

from veri_sdk import Client

client = Client()
deployment_id = "dep_..."
now = datetime.now(timezone.utc)

lifetime = client.deployments.metrics(deployment_id)
series = client.deployments.metrics_timeseries(
    deployment_id,
    from_time=now - timedelta(hours=1),
    to_time=now,
    interval="auto",
)
errors = client.deployments.requests(
    deployment_id,
    from_time=now - timedelta(hours=1),
    to_time=now,
    errors_only=True,
    limit=50,
)

print(lifetime)
print(series["summary"])
print(f"{len(series['buckets'])} buckets")
print(f"{len(errors)} failed requests")

datetime values are serialized as RFC 3339 timestamps. You can also pass RFC 3339 strings directly. A minimal automated smoke test can assert:

assert lifetime["total_requests"] >= 2
assert series["summary"]["requests"] >= 2
assert series["summary"]["successfulRequests"] >= 2
assert series["summary"]["e2e_ms"]["samples"] >= 2
assert errors == []

5. Verify with REST

Lifetime counters:

curl "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/metrics" \
  -H "Authorization: Bearer $VERI_API_KEY"

Time-series metrics default to the past seven days when from and to are omitted:

curl "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/metrics/timeseries?interval=auto" \
  -H "Authorization: Bearer $VERI_API_KEY"

Only failed requests in a bounded window:

curl --get "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/requests" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  --data-urlencode "errors_only=true" \
  --data-urlencode "limit=50" \
  --data-urlencode "from=2026-07-14T00:00:00Z" \
  --data-urlencode "to=2026-07-14T01:00:00Z"

Scope results to a single replica

The metrics/timeseries, metrics/engine-timeseries, and requests endpoints accept an optional replica_id query parameter. List replica IDs with GET /v1/deployments/$DEPLOYMENT_ID/replicas:

curl --get "https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/metrics/timeseries" \
  -H "Authorization: Bearer $VERI_API_KEY" \
  --data-urlencode "interval=auto" \
  --data-urlencode "replica_id=rep_..."

When replica_id is set, buckets, the window summary, engine gauges (queue depth and KV-cache utilization), and recent requests are scoped to that replica. Cost efficiency becomes replica-scoped as well: window cost is the replica’s own active interval multiplied by its per-box GPU rate. Scoped metrics responses echo replica_id at the top level. The ID must be a replica of the deployment; unknown IDs return 404. Omit the parameter to keep deployment-wide aggregates. See the interactive API reference for complete response schemas.

Investigate with MCP

Set up Veri MCP by following Model Context Protocol. In read-only mode, all observability tools remain available; only tools that create, cancel, stop, or rescore resources are removed. Useful deployment tools:

Tool	Use
`veri_get_deployment_metrics`	Compact lifetime counters, percentiles, error rate, uptime, and cost
`veri_query_deployment_metrics`	Bounded time-series buckets with optional metric selection
`veri_list_deployment_requests`	Recent requests, time filters, and error-only investigation
`veri_get_billing_overview`	Balance, current burn, runway, active resources, and recent transactions
`veri_get_cost_summary`	Historical training, deployment, and evaluation costs

Try these prompts:

Inspect deployment dep_... over the past hour. Summarize request volume, p50/p95/p99 TTFT and end-to-end latency, token throughput, error rate, and cost efficiency. Then list failed requests and group them by error category. Do not modify resources.

Compare the past 24 hours with the preceding 24 hours for deployment dep_.... Identify latency, error-rate, throughput, or cost regressions and cite the metric values behind each conclusion.

Review deployment dep_... and my billing overview. Explain whether request volume, output-token efficiency, errors, or the current GPU burn rate is the main cost risk. Do not modify resources.

Agent responses are only as complete as the telemetry window. Include the deployment ID and an explicit period in prompts when you need reproducible results.

Metric semantics

Request timing

Metric	Meaning
TTFB	Request start until upstream response headers arrive
TTFT	Request start until the first non-empty content or reasoning token
E2E latency	Request start until the response body or stream terminates
Generation time	TTFT until stream completion
TPOT	Generation time divided across reported output-token intervals

TTFT and TPOT are most meaningful for streaming responses. TPOT is only calculated when the upstream engine reports usage. Latency percentiles use successful requests so application and transport failures do not distort the serving-latency distribution.

Traffic and errors

Each terminal request record can include:

Prompt, cached prompt, and completion tokens.
Request and response byte counts.
HTTP status and finish reason.
Whether the response streamed and whether usage was reported.
A bounded error category and diagnostic message.
Client cancellation or disconnection state.
Request start and completion timestamps.

Error charts separate client errors from server or interrupted-stream errors. Use request history to inspect the underlying error_category and message.

Cost efficiency

The dashboard and time-series response derive:

Cost per successful request.
Cost per million output tokens.
Tokens per dollar.
Total deployment cost for the selected window.

Cost-per-token values require reported completion tokens. A request can be successful while lacking usage, so compare usage_reported when investigating missing efficiency data.

Engine health

Replica heartbeats provide:

Running and waiting requests.
KV-cache utilization.
Prefix-cache hit rate.
Generation token throughput.
Active replica count.

Request telemetry and engine telemetry are intentionally separate. Request rows describe completed client operations; engine history describes sampled replica state.

Troubleshooting

Symptom	Check
No requests in charts	Confirm the deployment ID, time range, and that requests reached the Veri deployment URL
E2E exists but TTFT/TPOT is empty	Use streaming; confirm the stream emitted content and reported usage
Tokens or cost efficiency are empty	Request streaming usage and inspect `usage_reported`
Errors appear but latency percentiles do not change	Expected: percentiles include successful requests only
Engine charts are empty	Wait for replica heartbeats and confirm the deployment is serving
A stream has `client_cancelled=true`	The downstream client stopped reading before the upstream stream completed
MCP cannot create or rescore an eval	The server is read-only; observability queries still work

Where to go next

MCP setup

Let an agent query deployment, training, billing, and evaluation context.

Deployment commands

Generate traffic, benchmark an endpoint, and inspect recent requests.

API reference

Integrate metrics and request history into your own systems.

Deployment billing

Understand hourly GPU billing and scale-to-zero behavior.

Get started

Training

Deployments

Evaluations

CLI

Volumes

Choose an interface

Verify observability end to end

1. Select a serving deployment

2. Generate buffered and streaming traffic

3. Inspect the dashboard

4. Verify with the Python SDK

5. Verify with REST

Scope results to a single replica

Investigate with MCP

Metric semantics

Request timing

Traffic and errors

Cost efficiency

Engine health

Troubleshooting

Where to go next

MCP setup

Deployment commands

API reference

Deployment billing

​Choose an interface

​Verify observability end to end

​1. Select a serving deployment

​2. Generate buffered and streaming traffic

​3. Inspect the dashboard

​4. Verify with the Python SDK

​5. Verify with REST

​Scope results to a single replica

​Investigate with MCP

​Metric semantics

​Request timing

​Traffic and errors

​Cost efficiency

​Engine health

​Troubleshooting

​Where to go next

MCP setup

Deployment commands

API reference

Deployment billing

Choose an interface

Verify observability end to end

1. Select a serving deployment

2. Generate buffered and streaming traffic

3. Inspect the dashboard

4. Verify with the Python SDK

5. Verify with REST

Scope results to a single replica

Investigate with MCP

Metric semantics

Request timing

Traffic and errors

Cost efficiency

Engine health

Troubleshooting

Where to go next