> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oumi.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# SELF-HOSTING
> Export your trained model from Oumi and serve it on your own hardware or cloud GPU instance
Export a model from Oumi and you can serve it anywhere. Oumi packages trained artifacts in a standard, portable format that works with common inference engines, so you move from the platform to your own serving stack without conversion steps.
***
## WHEN TO SELF-HOST
A [managed deployment](/guides/deployment/deploy-a-model) is the faster path for most teams, and it is the only path that gives you inference logs, [health metrics](/guides/deployment/health-metrics), and [quality monitoring](/guides/deployment/monitoring) inside Oumi. Self-host when you have a requirement a hosted endpoint cannot meet:
* Your network is air-gapped, or your data cannot leave your own infrastructure
* You have an existing serving stack the model needs to fit into
* You want to run the model on hardware you already own
* You are testing locally against a model you are still iterating on
You can also export purely for archival, to keep a copy of the weights outside the platform.
***
## STEP 1: EXPORT YOUR MODEL
1. Go to the **Models** page and click the model name.
2. On the model's detail page, click **Export**.
3. Click **Continue Export** to download the file to your computer.
An export contains everything an inference engine needs:
* **Model weights:** the trained parameters resulting from fine-tuning
* **Tokenizer and configuration files:** required for correct input and output handling
* **Model metadata:** information about the training run and configuration
Export once evaluation results meet your success criteria. If your model does not meet quality expectations yet, keep iterating on training or data synthesis first.
***
## STEP 2: SERVE THE MODEL
[vLLM](https://github.com/vllm-project/vllm), and its Apple Silicon equivalent [vLLM-MLX](https://github.com/waybarrios/vllm-mlx), run an OpenAI-compatible inference server from your exported directory. The steps below are the same whether you run them on your own machine or on a cloud GPU instance. Follow the installation instructions on those projects' homepages, for instance:
```bash theme={null}
pip install vllm
```
Navigate to your exported model's parent directory and start the server:
```bash theme={null}
vllm serve ./exported_model/ --port 8000
```
Then call it with [OpenAI's Python library](https://github.com/openai/openai-python):
```bash theme={null}
pip install openai
```
```python theme={null}
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
response = client.chat.completions.create(
model="exported_model",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
```
### RUNNING ON A CLOUD GPU INSTANCE
To serve from a cloud instance instead of your own machine, transfer the exported artifacts there first (via `scp`, S3, or GCS), then run the same two commands above on the instance. Point your client at the instance rather than localhost:
```python theme={null}
client = OpenAI(base_url="http://:8000/v1", api_key="unused")
```
Replace `` with the public IP or hostname of your instance, and open port 8000 in your security group or firewall rules. Do not expose a vLLM server publicly without authentication in front of it.
### OTHER SERVING OPTIONS
You can use your exported model directly from Python via the `vllm` or Hugging Face `transformers` libraries rather than through a server. Hugging Face `transformers` also ships a CLI that serves an OpenAI-compatible API; see [its serving documentation](https://huggingface.co/docs/transformers/v5.2.0/serve-cli/serving).
***
## MANAGED PLATFORMS ELSEWHERE
If you want a hosted endpoint outside Oumi, an exported model imports into several third-party platforms.
### AMAZON BEDROCK
Bedrock supports custom model import, so you can serve your Oumi-trained model as a managed AWS endpoint. Read the [AWS blog post](https://aws.amazon.com/blogs/machine-learning/accelerate-custom-llm-deployment-fine-tune-with-oumi-and-deploy-to-amazon-bedrock/) for a full walkthrough.
**Best for:** teams already on AWS who want a managed endpoint with no infrastructure overhead.
### LAMBDA
Lambda provides on-demand GPU instances well suited to hosting a vLLM server. Watch the [Lambda deployment video](https://www.youtube.com/watch?v=0XpfYRpd_FA) for a step-by-step guide to spinning up an instance, loading your exported model, and making requests.
**Best for:** teams who want direct GPU access and control over the serving stack.
***
## CHOOSING AN INSTANCE TYPE
The right instance depends on your model size, latency requirements, and budget. As a general rule, larger models need more GPU memory: a 7B parameter model typically needs at least 16 GB of VRAM, while a 30B+ model needs significantly more. Consult your provider's documentation for current availability and pricing:
* [AWS EC2 GPU instances](https://aws.amazon.com/ec2/instance-types/#Accelerated_Computing)
* [GCP GPU machine types](https://cloud.google.com/compute/docs/gpus)
* [Lambda GPU cloud](https://lambdalabs.com/service/gpu-cloud)
***
## OPERATIONAL CONSIDERATIONS
**Cost:** cloud GPU instances bill by the hour whether or not they serve traffic. For variable load, consider auto-scaling groups or a serverless inference platform so you are not paying for idle capacity. A [managed deployment](/guides/deployment/deploy-a-model) scales to zero by default and handles this for you.
**Latency:** a network round trip adds latency compared to running locally. Choose a region close to your users and keep request payloads small.
**Security:** restrict access to your endpoint with API keys, VPC networking, or IAM policies.
**Model versioning:** keep exported artifacts versioned in cloud storage so you can roll back to a previous model.
**Observability:** a self-hosted model does not report back into Oumi, so inference logs, [health metrics](/guides/deployment/health-metrics), and [quality monitoring](/guides/deployment/monitoring) are not available for it. Plan for your own logging and evaluation, or keep a managed deployment alongside for comparison.
***
## WHAT'S NEXT
Stand up a managed endpoint instead, with logging and monitoring built in.
Re-evaluate your model as production data evolves.