OpenAI compatibility

​

What works

• OpenAI Python SDK
• OpenAI Node SDK
• LangChain (ChatOpenAI(base_url=...))
• LlamaIndex (OpenAILike(api_base=..., is_chat_model=True))
• Vercel AI SDK
• curl with the standard OpenAI request shape
• Anything else that targets /chat/completions

from openai import OpenAI

oai = OpenAI(
    base_url=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
)

response = oai.chat.completions.create(
    model=dep.name,
    messages=[
        {"role": "system", "content": "You are a helpful math tutor."},
        {"role": "user", "content": "What is 12 + 7?"},
    ],
    temperature=0.7,
    max_tokens=256,
)

print(response.choices[0].message.content)
print(response.usage)

​

Streaming

​

Long generations

• The response starts early. The 200 status and headers are sent at 60 seconds, then a newline every 15 seconds until the answer is ready, and then the completion JSON. This keeps proxies and the API’s network edge (which closes a connection that sends nothing for about 100 seconds) from cutting the request and making your client retry the whole generation. JSON allows leading whitespace: the OpenAI Python and Node SDKs, httpx, requests, and JSON.parse parse the body unchanged.
• A late failure is in the body. The status is already 200, so an error after that point comes back as {"error": {"code": ..., "message": ...}} with no choices, the same error object a non-200 response carries (for example code upstream_unavailable when the model server fails). If you parse responses yourself, check for an error key. With the OpenAI Python SDK, response.choices is None and the error is in response.model_extra["error"]; the Veri SDK raises it as a VeriError.
• Ten-minute limit. A non-streaming request still running after 600 seconds is stopped and answered with error code generation_timeout. Sending the same request again hits the same limit, so switch to stream=true.

​

Everything else is a passthrough

• Tool calling (tools, tool_choice) rides through, but the deployment’s vLLM must have tool parsing enabled — pass vllm_extra_args at create time (for example --enable-auto-tool-choice --tool-call-parser hermes). Without those flags vLLM rejects tool requests.
• Reasoning (Qwen3). Veri starts vLLM with --reasoning-parser qwen3 for the hybrid-thinking Qwen/Qwen3-* checkpoints (0.6B to 235B-A22B, their -FP8, -AWQ and -GPTQ quants, and -Thinking-2507). The model’s thinking comes back in message.reasoning (streaming: delta.reasoning), never inside content; on the Anthropic /v1/messages route it is a thinking block. To pick another parser, pass your own --reasoning-parser in vllm_extra_args. To skip thinking, send chat_template_kwargs: {"enable_thinking": false} per request, or set --default-chat-template-kwargs '{"enable_thinking": false}' at create. A request with reasoning_effort (on /v1/messages: output_config.effort, which Claude Code sends) turns thinking back on for that request.
• Structured outputs (response_format) are handled by vLLM’s guided decoding; support depends on the engine version and model.
• Vision / audio inputs depend on the served model being multimodal; text-only models reject them.

​

Request shape

{
  "model": "your-deployment-name",
  "messages": [
    {"role": "user", "content": "Hello"}
  ],
  "temperature": 0.7,
  "max_tokens": 256,
  "stream": false
}

​

Response shape

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1715000000,
  "model": "your-deployment-name",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "Hello back!"},
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 5,
    "total_tokens": 15
  }
}

​

Scaled-to-zero deployments

​

Upstream failures

​

LangChain example

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage

llm = ChatOpenAI(
    base_url=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
    model=dep.name,
    temperature=0.7,
)

result = llm.invoke([HumanMessage(content="What is 12 + 7?")])
print(result.content)

​

LlamaIndex example

from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    api_base=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
    model=dep.name,
    is_chat_model=True,
    temperature=0.7,
)

print(llm.complete("What is 12 + 7?"))

​

curl

curl https://api.veri.studio/v1/deployments/$DEPLOYMENT_ID/chat/completions \
  -H "Authorization: Bearer vk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-deployment-name",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

​

Responses API

from openai import OpenAI

oai = OpenAI(
    base_url=f"https://api.veri.studio/v1/deployments/{dep.id}",
    api_key="vk_your_api_key",
)
resp = oai.responses.create(model=dep.name, input="Write a haiku about GPUs.")
print(resp.output_text, resp.usage)

model = "your-deployment-name"
model_provider = "veri"

[model_providers.veri]
name = "Veri"
base_url = "https://api.veri.studio/v1/deployments/1a2b3c4d5e6f"
env_key = "VERI_API_KEY"
wire_api = "responses"

• The request goes to the deployment’s own vLLM /v1/responses, the same model server as /chat/completions. Tool calls (tools, function_call items) go through the deployment’s --tool-call-parser, so launch the deployment with --enable-auto-tool-choice --tool-call-parser <parser> for your model (for example hermes for Qwen). A deployment on its own vllm_image needs a vLLM that serves /v1/responses.
• model is rewritten to the deployment’s served name; the rest of the body is passed through. stream: true streams the Responses events (response.created … response.completed) as server-sent events, untouched.
• Clients that append /v1/responses to a base without the version can use https://api.veri.studio/v1/deployments/{deployment_id}/v1/responses, the same route.
• Every call lands in the deployment’s request log with its tokens (prompt_tokens = the response’s input_tokens, cached tokens included; cached_prompt_tokens = input_tokens_details.cached_tokens; completion_tokens = output_tokens).
• The same wake and retry contract as chat: a parked deployment answers 503 deployment_waking while it boots, and 503 upstream_unavailable covers a replica that cannot be reached; both carry Retry-After. Errors from vLLM itself (a prompt longer than the served context, say) come back as vLLM sent them. Request bodies are capped at 8 MB.
• vLLM does not store responses (store is ignored), so previous_response_id is not available: send the conversation in input, as Codex does.

​

What about embeddings?

​

Where to go next