OpenAI compatibility

A Veri deployment exposes the OpenAI Chat Completions API at https://api.veri.studio/v1/deployments/{deployment_id}. Anything that already speaks OpenAI works — point base_url at the deployment and pass your Veri API key.

Use the full URL https://api.veri.studio/v1/deployments/{dep.id} as base_url. The OpenAI client appends /chat/completions for you. The dep.endpoint_url field on the SDK response is a relative path that already includes /chat/completions — it’s for direct curl-style calls, not for the OpenAI SDK’s base_url.

What works

Basic chat with one or more messages, returning a completion. Works in:

OpenAI Python SDK
OpenAI Node SDK
LangChain (ChatOpenAI(base_url=...))
LlamaIndex (OpenAI(api_base=...))
Vercel AI SDK
curl with the standard OpenAI request shape
Anything else that targets /chat/completions

from openai import OpenAI

oai = OpenAI(
    base_url=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
)

response = oai.chat.completions.create(
    model=dep.name,
    messages=[
        {"role": "system", "content": "You are a helpful math tutor."},
        {"role": "user", "content": "What is 12 + 7?"},
    ],
    temperature=0.7,
    max_tokens=256,
)

print(response.choices[0].message.content)
print(response.usage)

Streaming

Streaming works: set stream=true and vLLM’s SSE events are forwarded through unchanged, so for chunk in oai.chat.completions.create(..., stream=True) behaves exactly as it does against OpenAI. Usage accounting is preserved (stream_options.include_usage is injected upstream so the final chunk carries token counts).

Everything else is a passthrough

The proxy keeps a few typed fields (model, messages, temperature, max_tokens, stream) and forwards everything else the client sends — top_p, stop, seed, frequency_penalty, presence_penalty, logit_bias, response_format, tools, tool_choice, … — to vLLM untouched. vLLM is the schema authority: it applies its own defaults for omitted fields and rejects what it doesn’t support. Caveats:

Tool calling (tools, tool_choice) rides through, but the deployment’s vLLM must have tool parsing enabled — pass vllm_extra_args at create time (for example --enable-auto-tool-choice --tool-call-parser hermes). Without those flags vLLM rejects tool requests.
Structured outputs (response_format) are handled by vLLM’s guided decoding; support depends on the engine version and model.
Vision / audio inputs depend on the served model being multimodal; text-only models reject them.

Request shape

{
  "model": "your-deployment-name",
  "messages": [
    {"role": "user", "content": "Hello"}
  ],
  "temperature": 0.7,
  "max_tokens": 256,
  "stream": false
}

The model field in the request is echoed back in the response — it’s not validated against the deployment’s actual model. Pass whatever string you want.

Response shape

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1715000000,
  "model": "your-deployment-name",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "Hello back!"},
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 5,
    "total_tokens": 15
  }
}

finish_reason values: stop (normal end), length (hit max_tokens), tool_calls (when tool calling is enabled). Extra top-level fields vLLM returns (system_fingerprint, …) are preserved in the response.

Scaled-to-zero deployments

If the deployment scaled to zero, the first request triggers a wake and returns 503 with a Retry-After header and error code deployment_waking while the GPU boots (a few minutes). The OpenAI client’s built-in retries give up well before that, so either retry on 503 yourself until the cold start completes, or pre-warm with veri deployments wake <id> before sending traffic. The Veri SDK’s chat() handles this retry loop for you.

LangChain example

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage

llm = ChatOpenAI(
    base_url=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
    model=dep.name,
    temperature=0.7,
)

result = llm.invoke([HumanMessage(content="What is 12 + 7?")])
print(result.content)

LlamaIndex example

from llama_index.llms.openai import OpenAI as LlamaOpenAI

llm = LlamaOpenAI(
    api_base=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
    model=dep.name,
    temperature=0.7,
)

print(llm.complete("What is 12 + 7?"))

curl

curl https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/chat/completions \
  -H "Authorization: Bearer vk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-deployment-name",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

What about embeddings?

Not supported — Veri serves chat completions only, not embeddings. Use a dedicated embeddings provider or run a local model for that surface.

Where to go next

Deployments API

create / chat / stop endpoint reference.

Deployment analytics demo

Planned walkthrough for request, token, latency, and replica metrics.

Hosting & GPU sizing

Pick the right GPU for the model.

Quickstart

First-time deployment in ~10 minutes.

Get started

Training

Deployments

CLI

Volumes

Webhooks

What works

Streaming

Everything else is a passthrough

Request shape

Response shape

Scaled-to-zero deployments

LangChain example

LlamaIndex example

curl

What about embeddings?

Where to go next

Deployments API

Deployment analytics demo

Hosting & GPU sizing

Quickstart

​What works

​Streaming

​Everything else is a passthrough

​Request shape

​Response shape

​Scaled-to-zero deployments

​LangChain example

​LlamaIndex example

​curl

​What about embeddings?

​Where to go next

Deployments API

Deployment analytics demo

Hosting & GPU sizing

Quickstart

What works

Streaming

Everything else is a passthrough

Request shape

Response shape

Scaled-to-zero deployments

LangChain example

LlamaIndex example

curl

What about embeddings?

Where to go next