CALL A DEPLOYMENT

A deployment’s endpoint is compatible with the OpenAI and Anthropic APIs. To call it, point the OpenAI or Anthropic SDK at the Oumi endpoint and pass the deployment as the model. The quickest way to start is with your deployment’s own snippets. On the deployment’s page, click View endpoint & code to copy a curl, Python, or TypeScript example that already has your endpoint and deployment path filled in.

ENDPOINT AND AUTHENTICATION

	Value
OpenAI-compatible endpoint	`https://inference.oumi.ai/v1/chat/completions`, for custom, base, and OpenAI deployments
Anthropic-compatible endpoint	`https://inference.oumi.ai/v1/messages`, for Anthropic deployments
`model`	The deployment path, such as `projects/XFZ7K2/deployments/83`, not a model name
API key	A personal Oumi API key, sent as the `X-API-Key` header or as a bearer token

Create a personal API key under User Settings > API Keys, or click Create one in the Endpoint & code dialog.

OPENAI-COMPATIBLE REQUESTS

Use these for custom, base, and OpenAI deployments. Set the base URL to https://inference.oumi.ai/v1.

curl -X POST https://inference.oumi.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $OUMI_API_KEY" \
  -d '{
    "model": "projects/{project_id}/deployments/{deployment_id}",
    "messages": [{"role": "user", "content": "your prompt"}]
  }'

from openai import OpenAI

client = OpenAI(
    api_key="<your-api-key>",
    base_url="https://inference.oumi.ai/v1",
)

response = client.chat.completions.create(
    model="projects/{project_id}/deployments/{deployment_id}",
    messages=[{"role": "user", "content": "your prompt"}],
)
print(response.choices[0].message.content)

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.OUMI_API_KEY,
  baseURL: 'https://inference.oumi.ai/v1',
});

const response = await client.chat.completions.create({
  model: 'projects/{project_id}/deployments/{deployment_id}',
  messages: [{ role: 'user', content: 'your prompt' }],
});

ANTHROPIC-COMPATIBLE REQUESTS

Use these for Anthropic deployments, which /chat/completions rejects. The Anthropic API requires max_tokens.

The Anthropic SDKs add /v1 to the base URL themselves, so set their base URL to https://inference.oumi.ai.

curl -X POST https://inference.oumi.ai/v1/messages \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $OUMI_API_KEY" \
  -d '{
    "model": "projects/{project_id}/deployments/{deployment_id}",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "your prompt"}]
  }'

import anthropic

client = anthropic.Anthropic(
    api_key="<your-api-key>",
    base_url="https://inference.oumi.ai",
)

client.messages.create(
    model="projects/{project_id}/deployments/{deployment_id}",
    max_tokens=1024,
    messages=[{"role": "user", "content": "your prompt"}],
)

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic({
  apiKey: process.env.OUMI_API_KEY,
  baseURL: 'https://inference.oumi.ai',
});

await client.messages.create({
  model: 'projects/{project_id}/deployments/{deployment_id}',
  max_tokens: 1024,
  messages: [{ role: 'user', content: 'your prompt' }],
});

STREAM LONG RESPONSES

Set stream to true to receive the response as it is generated. Stream any request that can produce a long response. A non-streaming request waits for the whole response, and it times out if the model takes too long.

stream = client.chat.completions.create(
    model="projects/{project_id}/deployments/{deployment_id}",
    messages=[{"role": "user", "content": "your prompt"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

SEND THE TRAINING SYSTEM PROMPT

A fine-tuned model performs best with the system prompt it was trained on. If your training data had a system prompt, send the same one with every request. The System prompt section on the deployment’s Info & Configuration tab shows prompts sampled from the training data.

COLD STARTS

A custom or base deployment with min replicas set to 0 has no running replica when it is idle. The first request after an idle period starts a replica and returns 429 with the message “Deployment is warming up” and a Retry-After header. A replica usually takes a few minutes to start.

Retry: Send the request again until it succeeds. The OpenAI and Anthropic SDKs retry a 429 a few times by default, which may not cover the whole warm-up. Raise the SDK’s retry count, or handle the retry yourself.
Wake the deployment first: Before a batch of requests, run oumi-cli deployments wake-up, which waits until the deployment is ready. See the CLI reference.
Avoid cold starts: Set min replicas to 1. See Autoscaling.

External deployments have no cold starts.

ERRORS

Status	Message	What to do
`429`	”Deployment is warming up”	The deployment scaled to zero and is starting. Retry after the `Retry-After` interval.
`429`	”Deployment is scaling to handle load”	All replicas are busy, and a new one is starting. Retry, or raise max replicas.
`429`	A rate limit message	Your plan’s request or token limit was reached. Retry later, or reduce the request rate.
`400`	”The request exceeds this deployment’s context length”	The prompt and the requested output are too long for the model. Shorten the prompt or lower `max_tokens`.

TRY A DEPLOYMENT WITHOUT CODE

On the deployment’s page, click Try in Playground to chat with it in the Model Playground. The button is enabled only when the deployment is Active. You can also ask FAiE (Frontier AI Engineer Agent) to send a prompt to the deployment.

Getting started

Oumi workflow

CALL A DEPLOYMENT

ENDPOINT AND AUTHENTICATION

OPENAI-COMPATIBLE REQUESTS

ANTHROPIC-COMPATIBLE REQUESTS

STREAM LONG RESPONSES

SEND THE TRAINING SYSTEM PROMPT

COLD STARTS

ERRORS

TRY A DEPLOYMENT WITHOUT CODE

​ENDPOINT AND AUTHENTICATION

​OPENAI-COMPATIBLE REQUESTS

​ANTHROPIC-COMPATIBLE REQUESTS

​STREAM LONG RESPONSES

​SEND THE TRAINING SYSTEM PROMPT

​COLD STARTS

​ERRORS

​TRY A DEPLOYMENT WITHOUT CODE

ENDPOINT AND AUTHENTICATION

OPENAI-COMPATIBLE REQUESTS

ANTHROPIC-COMPATIBLE REQUESTS

STREAM LONG RESPONSES

SEND THE TRAINING SYSTEM PROMPT

COLD STARTS

ERRORS

TRY A DEPLOYMENT WITHOUT CODE