SELF-HOSTING

Export a model from Oumi and you can serve it anywhere. Oumi packages trained artifacts in a standard, portable format that works with common inference engines, so you move from the platform to your own serving stack without conversion steps.

WHEN TO SELF-HOST

A managed deployment is the faster path for most teams, and it is the only path that gives you inference logs, health metrics, and quality monitoring inside Oumi. Self-host when you have a requirement a hosted endpoint cannot meet:

Your network is air-gapped, or your data cannot leave your own infrastructure
You have an existing serving stack the model needs to fit into
You want to run the model on hardware you already own
You are testing locally against a model you are still iterating on

You can also export purely for archival, to keep a copy of the weights outside the platform.

STEP 1: EXPORT YOUR MODEL

Go to the Models page and click the model name.
On the model’s detail page, click Export.
Click Continue Export to download the file to your computer.

An export contains everything an inference engine needs:

Model weights: the trained parameters resulting from fine-tuning
Tokenizer and configuration files: required for correct input and output handling
Model metadata: information about the training run and configuration

Export once evaluation results meet your success criteria. If your model does not meet quality expectations yet, keep iterating on training or data synthesis first.

STEP 2: SERVE THE MODEL

vLLM, and its Apple Silicon equivalent vLLM-MLX, run an OpenAI-compatible inference server from your exported directory. The steps below are the same whether you run them on your own machine or on a cloud GPU instance. Follow the installation instructions on those projects’ homepages, for instance:

pip install vllm

Navigate to your exported model’s parent directory and start the server:

vllm serve ./exported_model/ --port 8000

Then call it with OpenAI’s Python library:

pip install openai

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")

response = client.chat.completions.create(
    model="exported_model",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

RUNNING ON A CLOUD GPU INSTANCE

To serve from a cloud instance instead of your own machine, transfer the exported artifacts there first (via scp, S3, or GCS), then run the same two commands above on the instance. Point your client at the instance rather than localhost:

client = OpenAI(base_url="http://<your-instance-ip>:8000/v1", api_key="unused")

Replace <your-instance-ip> with the public IP or hostname of your instance, and open port 8000 in your security group or firewall rules. Do not expose a vLLM server publicly without authentication in front of it.

OTHER SERVING OPTIONS

You can use your exported model directly from Python via the vllm or Hugging Face transformers libraries rather than through a server. Hugging Face transformers also ships a CLI that serves an OpenAI-compatible API; see its serving documentation.

MANAGED PLATFORMS ELSEWHERE

If you want a hosted endpoint outside Oumi, an exported model imports into several third-party platforms.

AMAZON BEDROCK

Bedrock supports custom model import, so you can serve your Oumi-trained model as a managed AWS endpoint. Read the AWS blog post for a full walkthrough. Best for: teams already on AWS who want a managed endpoint with no infrastructure overhead.

LAMBDA

Lambda provides on-demand GPU instances well suited to hosting a vLLM server. Watch the Lambda deployment video for a step-by-step guide to spinning up an instance, loading your exported model, and making requests. Best for: teams who want direct GPU access and control over the serving stack.

CHOOSING AN INSTANCE TYPE

The right instance depends on your model size, latency requirements, and budget. As a general rule, larger models need more GPU memory: a 7B parameter model typically needs at least 16 GB of VRAM, while a 30B+ model needs significantly more. Consult your provider’s documentation for current availability and pricing:

OPERATIONAL CONSIDERATIONS

Cost: cloud GPU instances bill by the hour whether or not they serve traffic. For variable load, consider auto-scaling groups or a serverless inference platform so you are not paying for idle capacity. A managed deployment scales to zero by default and handles this for you. Latency: a network round trip adds latency compared to running locally. Choose a region close to your users and keep request payloads small. Security: restrict access to your endpoint with API keys, VPC networking, or IAM policies. Model versioning: keep exported artifacts versioned in cloud storage so you can roll back to a previous model. Observability: a self-hosted model does not report back into Oumi, so inference logs, health metrics, and quality monitoring are not available for it. Plan for your own logging and evaluation, or keep a managed deployment alongside for comparison.

Getting started

Oumi workflow

SELF-HOSTING

WHEN TO SELF-HOST

STEP 1: EXPORT YOUR MODEL

STEP 2: SERVE THE MODEL

RUNNING ON A CLOUD GPU INSTANCE

OTHER SERVING OPTIONS

MANAGED PLATFORMS ELSEWHERE

AMAZON BEDROCK

LAMBDA

CHOOSING AN INSTANCE TYPE

OPERATIONAL CONSIDERATIONS

WHAT’S NEXT

Deploy a model

Evaluating after deployment

​WHEN TO SELF-HOST

​STEP 1: EXPORT YOUR MODEL

​STEP 2: SERVE THE MODEL

​RUNNING ON A CLOUD GPU INSTANCE

​OTHER SERVING OPTIONS

​MANAGED PLATFORMS ELSEWHERE

​AMAZON BEDROCK

​LAMBDA

​CHOOSING AN INSTANCE TYPE

​OPERATIONAL CONSIDERATIONS

​WHAT’S NEXT

Deploy a model

Evaluating after deployment

WHEN TO SELF-HOST

STEP 1: EXPORT YOUR MODEL

STEP 2: SERVE THE MODEL

RUNNING ON A CLOUD GPU INSTANCE

OTHER SERVING OPTIONS

MANAGED PLATFORMS ELSEWHERE

AMAZON BEDROCK

LAMBDA

CHOOSING AN INSTANCE TYPE

OPERATIONAL CONSIDERATIONS

WHAT’S NEXT