> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oumi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# CALL A DEPLOYMENT

> Call your Oumi deployment with the OpenAI or Anthropic SDK, handle cold starts, and fix common errors

A deployment's endpoint is compatible with the OpenAI and Anthropic APIs. To call it, point the OpenAI or Anthropic SDK at the Oumi endpoint and pass the deployment as the model.

The quickest way to start is with your deployment's own snippets. On the deployment's page, click `View endpoint & code` to copy a curl, Python, or TypeScript example that already has your endpoint and deployment path filled in.

## ENDPOINT AND AUTHENTICATION

| | Value |
| - | - |
| **OpenAI-compatible endpoint** | `https://inference.oumi.ai/v1/chat/completions`, for custom, base, and OpenAI deployments |
| **Anthropic-compatible endpoint** | `https://inference.oumi.ai/v1/messages`, for Anthropic deployments |
| **`model`** | The deployment path, such as `projects/XFZ7K2/deployments/83`, not a model name |
| **API key** | A personal Oumi API key, sent as the `X-API-Key` header or as a bearer token |

Create a personal API key under **User Settings** > **API Keys**, or click `Create one` in the `Endpoint & code` dialog.

## OPENAI-COMPATIBLE REQUESTS

Use these for custom, base, and OpenAI deployments. Set the base URL to `https://inference.oumi.ai/v1`.

```bash curl theme={null}
curl -X POST https://inference.oumi.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $OUMI_API_KEY" \
  -d '{
    "model": "projects/{project_id}/deployments/{deployment_id}",
    "messages": [{"role": "user", "content": "your prompt"}]
  }'
```

```python Python theme={null}
from openai import OpenAI

client = OpenAI(
    api_key="<your-api-key>",
    base_url="https://inference.oumi.ai/v1",
)

response = client.chat.completions.create(
    model="projects/{project_id}/deployments/{deployment_id}",
    messages=[{"role": "user", "content": "your prompt"}],
)
print(response.choices[0].message.content)
```

```typescript TypeScript theme={null}
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.OUMI_API_KEY,
  baseURL: 'https://inference.oumi.ai/v1',
});

const response = await client.chat.completions.create({
  model: 'projects/{project_id}/deployments/{deployment_id}',
  messages: [{ role: 'user', content: 'your prompt' }],
});
```

## ANTHROPIC-COMPATIBLE REQUESTS

Use these for Anthropic deployments, which `/chat/completions` rejects. The Anthropic API requires `max_tokens`.

> [!WARNING]
>
> The Anthropic SDKs add `/v1` to the base URL themselves, so set their base URL to `https://inference.oumi.ai`.

```bash curl theme={null}
curl -X POST https://inference.oumi.ai/v1/messages \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $OUMI_API_KEY" \
  -d '{
    "model": "projects/{project_id}/deployments/{deployment_id}",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "your prompt"}]
  }'
```

```python Python theme={null}
import anthropic

client = anthropic.Anthropic(
    api_key="<your-api-key>",
    base_url="https://inference.oumi.ai",
)

client.messages.create(
    model="projects/{project_id}/deployments/{deployment_id}",
    max_tokens=1024,
    messages=[{"role": "user", "content": "your prompt"}],
)
```

```typescript TypeScript theme={null}
import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic({
  apiKey: process.env.OUMI_API_KEY,
  baseURL: 'https://inference.oumi.ai',
});

await client.messages.create({
  model: 'projects/{project_id}/deployments/{deployment_id}',
  max_tokens: 1024,
  messages: [{ role: 'user', content: 'your prompt' }],
});
```

## STREAM LONG RESPONSES

Set `stream` to `true` to receive the response as it is generated. Stream any request that can produce a long response. A non-streaming request waits for the whole response, and it times out if the model takes too long.

```python theme={null}
stream = client.chat.completions.create(
    model="projects/{project_id}/deployments/{deployment_id}",
    messages=[{"role": "user", "content": "your prompt"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")
```

## SEND THE TRAINING SYSTEM PROMPT

A fine-tuned model performs best with the system prompt it was trained on. If your training data had a system prompt, send the same one with every request. The **System prompt** section on the deployment's **Info & Configuration** tab shows prompts sampled from the training data.

## COLD STARTS

A custom or base deployment with min replicas set to 0 has no running replica when it is idle. The first request after an idle period starts a replica and returns `429` with the message "Deployment is warming up" and a `Retry-After` header. A replica usually takes a few minutes to start.

* **Retry:** Send the request again until it succeeds. The OpenAI and Anthropic SDKs retry a `429` a few times by default, which may not cover the whole warm-up. Raise the SDK's retry count, or handle the retry yourself.
* **Wake the deployment first:** Before a batch of requests, run `oumi-cli deployments wake-up`, which waits until the deployment is ready. See the [CLI reference](https://docs.oumi.ai/cli/commands/deployments.md#wake-up).
* **Avoid cold starts:** Set min replicas to 1. See [Autoscaling](https://docs.oumi.ai/guides/deployment/deploy-a-model.md#autoscaling).

External deployments have no cold starts.

## ERRORS

| Status | Message | What to do |
| - | - | - |
| `429` | "Deployment is warming up" | The deployment scaled to zero and is starting. Retry after the `Retry-After` interval. |
| `429` | "Deployment is scaling to handle load" | All replicas are busy, and a new one is starting. Retry, or raise max replicas. |
| `429` | A rate limit message | Your plan's request or token limit was reached. Retry later, or reduce the request rate. |
| `400` | "The request exceeds this deployment's context length" | The prompt and the requested output are too long for the model. Shorten the prompt or lower `max_tokens`. |

## TRY A DEPLOYMENT WITHOUT CODE

On the deployment's page, click `Try in Playground` to chat with it in the [Model Playground](https://docs.oumi.ai/guides/playground.md). The button is enabled only when the deployment is `Active`. You can also ask [FAiE](https://docs.oumi.ai/guides/intro/faie.md) (Frontier AI Engineer Agent) to send a prompt to the deployment.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.