> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oumi.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# DEPLOY A MODEL
> Stand up a managed inference endpoint for your own model or an external one
## OVERVIEW
Once you have [generated training data](/guides/datasets/create), [fine-tuned a model](/guides/training), and [evaluated its performance](/guides/evaluations), the final step is deployment. A deployment serves a model on a managed inference endpoint, ready for real-time use.
The Deployments feature is currently in beta.
This page covers both kinds of deployment: a **custom Oumi model** you trained or imported, and an **external model** hosted by OpenAI or Anthropic. Both are created from the same dialog and, once live, behave identically. See [deployment types](/guides/deployment#deployment-types) for how they compare.
***
## BEFORE YOU START
| To deploy | You need |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| A **custom Oumi model** | A deployable trained or [imported](/guides/deployment/importing-models) model in your project |
| An **external model** | **Optional:** Add your own OpenAI or Anthropic API key. When you use your own key, Oumi does not bill you for that model usage. |
The model picker indicates which of your models can be deployed.
An external model needs no trained model at all, so you can deploy one in a project that has none yet.
***
## CREATE A DEPLOYMENT
From the top of the **Models** page, click the `Deploy Model` button; alternatively, click the `+ Create Deployment` button from the **Deployments** page.
The dialog walks through four stages:
Choose the deployment type and fill in its settings.
Optionally set up [quality monitoring](/guides/deployment/monitoring): add the judge evaluators that will score a sample of live traffic, then adjust the sampling rate. You can also enable monitoring later from the deployment's **Info & Configuration** tab.
Oumi provisions capacity and loads your model. This takes a few minutes, and you can navigate away without interrupting it.
The deployment is serving. The dialog shows a runnable code snippet you can copy straight into your own application.
On the **Deploy Model** dialog, select either `Custom Oumi Model` or `External Model`.
### CUSTOM OUMI MODEL
Serves a model from your project on Oumi-managed GPU capacity.
* Provide a unique `Deployment Name`.
* Select a `Model` from the drop-down.
* Choose whether to enable [inference logging](/guides/deployment/inference-logs). It is on by default.
* Optionally open `Advanced Settings` to change the autoscaling policy.
* Click `Start` to deploy your model.
### EXTERNAL MODEL
Forwards requests to OpenAI or Anthropic.
* Provide a unique `Deployment Name`.
* Select a `Provider` from the drop-down. OpenAI and Anthropic are supported today.
* Select your `External Model` from the drop-down.
* Optionally supply your own provider API key: paste a new one, or select a key already stored in your project.
* Choose whether to enable [inference logging](/guides/deployment/inference-logs). It is on by default.
* Click `Start` to deploy your model.
***
## AUTOSCALING AND COST (CUSTOM MODELS)
External models run on the provider's infrastructure, so they have no replicas to scale and no GPU time to pay for. When you use your own provider API key, Oumi does not bill you for that model usage. The rest of this section applies to custom Oumi models.
Hourly billing starts immediately when a deployment begins and continues while it is active. Leave min replicas at 0 to avoid paying for idle capacity.
Custom deployments default to **scale-to-zero**: min replicas 0, max replicas 1. An idle deployment costs nothing and spins up on its first request, at the cost of some added latency on that call. This is the right default for iterative work, where you deploy a model mainly to try it out.
Set min replicas to 1 when you want a deployment kept warm, with no cold-start latency on the first request. It will be billed hourly around the clock, including while it serves nothing.
| Setting | Default | What it does |
| ------------------------ | ------------------ | -------------------------------------------------------- |
| **Min replicas** | 0 | The warm-replica floor. 0 means scale to zero when idle. |
| **Max replicas** | 1 | The ceiling the deployment can scale up to. |
| **Scale-down idle time** | The server default | How long an idle replica waits before scaling down. |
Scale-down idle time offers presets of 1, 5, 15, 30, and 60 minutes, or the server default. It only applies when max replicas is greater than min replicas, since a fixed replica count never scales down.
***
## AUTO-DEPLOY FROM A TRAINING JOB (CUSTOM MODELS)
You can skip the separate deploy step entirely. The **Run Training Job** dialog has an auto-deploy option; enable it and Oumi deploys the resulting model with scale-to-zero autoscaling as soon as training finishes.
The option is unavailable when the base model and training method you selected aren't deployable, or when your project has reached its deployment limit.
***
## MANAGING DEPLOYMENTS
The **Deployments** page lists every deployment in the project with its `Name`, `Status`, `Model`, `Type`, and `Provider`. Click a deployment to open its detail page, where you can rename it, delete it, view its endpoint and code snippets, [try it in the browser](/guides/deployment/calling-your-endpoint#trying-a-deployment-without-code), and read its [inference logs](/guides/deployment/inference-logs). The detail page also carries the deployment's [health metrics](/guides/deployment/health-metrics), its [quality monitoring](/guides/deployment/monitoring) insights, and an **Info & Configuration** tab where you can adjust monitoring and, for custom Oumi models, autoscaling on an active deployment without redeploying.
A deployment moves through these states:
| Status | Meaning |
| ----------- | ------------------------------------------------------------------------------------ |
| `Deploying` | Capacity is being provisioned and the model loaded. |
| `Active` | Serving requests. This is the only state that accepts inference. |
| `Failed` | Provisioning did not complete. Check the operation on the [Jobs](/guides/jobs) page. |
| `Deleting` | Teardown in progress. |
| `Deleted` | Torn down and no longer billable. |
***
## DEPLOYING WITH THE OUMI AGENT
You can also deploy by asking the Oumi Agent. Tell it which model to deploy and it asks you to approve the deployment before provisioning anything, since GPU capacity costs money. Once the deployment is live it hands back a runnable snippet.
The Agent creates custom Oumi model deployments only. Create external-provider deployments from the **Deployments** page.
***
## WHAT'S NEXT
Call your deployment from Python, TypeScript, or curl.
See what your model is asked in production, and retrain from it.
Score sampled live traffic with LLM judges.