Skip to main content
Export a model to serve it outside Oumi, on your own hardware or a cloud GPU instance. The export uses standard Hugging Face files, so common inference engines such as vLLM can serve it without conversion.

WHEN TO SELF-HOST

An Oumi deployment is the faster path for most teams, and it is the only path with inference logs, serving metrics, and quality monitoring in Oumi. Self-host when a requirement rules it out:
  • Your data can’t leave your own infrastructure, or your network is air-gapped.
  • The model must fit into a serving stack you already run.
  • You want to run the model on hardware you already own.
You can also export a model to keep a copy of its weights outside the platform.

EXPORT A MODEL

  1. On the Models page, open the model’s actions menu and select Export.
  2. For a model trained with LoRA, choose the Format:
    • Adapter only exports the small LoRA adapter. Serve it together with the base model. Choose it when your serving stack loads adapters, or to serve several adapters on one base model.
    • Merged full weights merges the adapter into the base model and exports a standalone model. Choose it when your serving stack needs a single model. Merging can take a few hours, and you must keep the browser tab open until it finishes.
  3. Click Continue Export, and download the archive.
The archive holds the weights, the tokenizer, and the configuration files that an inference engine needs.

SERVE THE MODEL WITH VLLM

vLLM runs an OpenAI-compatible server from the exported files. Install it, and unpack the archive:
For merged or full weights, serve the directory:
For an adapter, serve the base model and load the adapter on top of it. Replace the base model with the one you trained from:
Then call the server with the OpenAI SDK. For an adapter, use the name you gave it as the model:
To serve from a cloud GPU instance, copy the exported files to it and run the same commands there. Don’t expose a vLLM server to the internet without authentication in front of it.

OTHER PLATFORMS

An exported model also works with other serving options:
  • Hugging Face transformers: Load the model in Python, or serve it with the transformers CLI. See its serving documentation.
  • Amazon Bedrock: Import the model as a Bedrock custom model. See the AWS walkthrough.
A self-hosted model doesn’t report back to Oumi, so plan your own logging and evaluation for it.