Skip to main content

​
OVERVIEW

Once you have generated training data, fine-tuned a model, and evaluated its performance, the final step is deployment. A deployment serves a model on a managed inference endpoint, ready for real-time use.
The Deployments feature is currently in beta.
This page covers both kinds of deployment: a custom Oumi model you trained or imported, and an external model hosted by OpenAI or Anthropic. Both are created from the same dialog and, once live, behave identically. See deployment types for how they compare.

​
BEFORE YOU START

The model picker indicates which of your models can be deployed.
An external model needs no trained model at all, so you can deploy one in a project that has none yet.

​
CREATE A DEPLOYMENT

From the top of the Models page, click the Deploy Model button; alternatively, click the + Create Deployment button from the Deployments page. The dialog walks through four stages:
1

Configure

Choose the deployment type and fill in its settings.
2

Monitoring

Optionally set up quality monitoring: add the judge evaluators that will score a sample of live traffic, then adjust the sampling rate. You can also enable monitoring later from the deployment’s Info & Configuration tab.
3

Deploy

Oumi provisions capacity and loads your model. This takes a few minutes, and you can navigate away without interrupting it.
4

Live

The deployment is serving. The dialog shows a runnable code snippet you can copy straight into your own application.
On the Deploy Model dialog, select either Custom Oumi Model or External Model.

​
CUSTOM OUMI MODEL

Serves a model from your project on Oumi-managed GPU capacity.
  • Provide a unique Deployment Name.
  • Select a Model from the drop-down.
  • Choose whether to enable inference logging. It is on by default.
  • Optionally open Advanced Settings to change the autoscaling policy.
  • Click Start to deploy your model.

​
EXTERNAL MODEL

Forwards requests to OpenAI or Anthropic.
  • Provide a unique Deployment Name.
  • Select a Provider from the drop-down. OpenAI and Anthropic are supported today.
  • Select your External Model from the drop-down.
  • Optionally supply your own provider API key: paste a new one, or select a key already stored in your project.
  • Choose whether to enable inference logging. It is on by default.
  • Click Start to deploy your model.

​
AUTOSCALING AND COST (CUSTOM MODELS)

External models run on the provider’s infrastructure, so they have no replicas to scale and no GPU time to pay for. When you use your own provider API key, Oumi does not bill you for that model usage. The rest of this section applies to custom Oumi models.
Hourly billing starts immediately when a deployment begins and continues while it is active. Leave min replicas at 0 to avoid paying for idle capacity.
Custom deployments default to scale-to-zero: min replicas 0, max replicas 1. An idle deployment costs nothing and spins up on its first request, at the cost of some added latency on that call. This is the right default for iterative work, where you deploy a model mainly to try it out. Set min replicas to 1 when you want a deployment kept warm, with no cold-start latency on the first request. It will be billed hourly around the clock, including while it serves nothing. Scale-down idle time offers presets of 1, 5, 15, 30, and 60 minutes, or the server default. It only applies when max replicas is greater than min replicas, since a fixed replica count never scales down.

​
AUTO-DEPLOY FROM A TRAINING JOB (CUSTOM MODELS)

You can skip the separate deploy step entirely. The Run Training Job dialog has an auto-deploy option; enable it and Oumi deploys the resulting model with scale-to-zero autoscaling as soon as training finishes. The option is unavailable when the base model and training method you selected aren’t deployable, or when your project has reached its deployment limit.

​
MANAGING DEPLOYMENTS

The Deployments page lists every deployment in the project with its Name, Status, Model, Type, and Provider. Click a deployment to open its detail page, where you can rename it, delete it, view its endpoint and code snippets, try it in the browser, and read its inference logs. The detail page also carries the deployment’s health metrics, its quality monitoring insights, and an Info & Configuration tab where you can adjust monitoring and, for custom Oumi models, autoscaling on an active deployment without redeploying. A deployment moves through these states:

​
DEPLOYING WITH THE OUMI AGENT

You can also deploy by asking the Oumi Agent. Tell it which model to deploy and it asks you to approve the deployment before provisioning anything, since GPU capacity costs money. Once the deployment is live it hands back a runnable snippet. The Agent creates custom Oumi model deployments only. Create external-provider deployments from the Deployments page.

​
WHAT’S NEXT

Calling your endpoint

Call your deployment from Python, TypeScript, or curl.

Inference logs

See what your model is asked in production, and retrain from it.

Quality monitoring

Score sampled live traffic with LLM judges.