# OUMI ADMINISTRATION
Source: https://docs.oumi.ai/admin
Manage platform settings, access, billing, and usage limits
Configure your organization and projects, manage access, and review billing and usage limits.
Configure and manage your organization, projects, and access.
Track platform usage and manage your quotas effectively.
# PROVIDER API KEYS
Source: https://docs.oumi.ai/admin/api-keys
Store API keys for external model providers
You can store your own API keys for external model providers like OpenAI, Anthropic, and Gemini. External models remain available without them. When you use your own key, Oumi does not bill you for that model usage.
Personal Oumi API keys serve a different purpose: they authenticate requests to the Oumi API. Oumi CLI support is coming soon.
## STORE A PROVIDER API KEY
To use your API accounts with external providers, first register the corresponding API key with your project:
* Go to **Project Settings > API Keys**.
* Click `Add a Key`.
* Enter the provider name and its API key.
* After clicking `Add Key`, it will appear under the **API Keys** section.
After you add a key, Oumi uses it for requests to that provider.
# OUMI BILLING
Source: https://docs.oumi.ai/admin/billing
View & manage your organization’s billing & usage details.
You can access all your organization's billing information via your **Organization Settings** page:
* Click on your profile icon, followed by `Organization Settings` in the drop-down menu.
* Click `Billing` on the left-hand navigation to view your billing information
***
## USAGE
The **Your Recent Usage** tab provides a breakdown of your platform activity and the resources consumed by your organization. These metrics help you monitor usage trends and understand how your activity contributes to billing.
By default, usage is displayed for the last 30 days, but this range may be adjustable depending on your plan.
### USAGE GRAPHS
Each graph visualizes usage for a specific resource over time. These values correspond to billable platform activity.
You may optionally toggle Show zero usage metrics to display metrics that have no usage during the selected time period.
***
## INVOICES
The **Invoices** tab provides a detailed record of all invoices generated for your workspace. This page allows you to review billing periods, usage charges, applied credits, and invoice status.
Invoices summarize all billable platform activity during a specific time period.
### VIEWING INVOICE DETAILS
When you click the `Invoices` tab, the most recent draft invoice is automatically selected and displayed in the invoice details panel on the right.
This draft invoice represents the current billing period and may still update as new usage occurs.
To open a different invoice, click on an invoice entry in the `Invoice History` list on the left. The selected invoice will open in the details panel on the right.
***
## CREDITS
The **Credits** tab shows all credits available to your workspace and how they are being consumed over time. Credits are used to offset usage charges generated by platform activity such as evaluations, inference, storage, and training.
This page allows you to track:
* Total credits allocated
* Credits consumed
* Remaining credits
* Credit availability dates
* Credit usage history
### CREDITS PAGE LAYOUT
The **Credits** page contains three main sections:
* **Summary** – overview of your credit balance
* **Credit Details** – information about each credit allocation
* **Ledger and Schedules** – history and availability of credits
You can also use the **Search** field to quickly find a specific credit allocation.
### MONITORING CREDIT USAGE
To effectively manage credits:
* Monitor credit consumption regularly
* Track usage spikes in the **Usage** tab
* Review invoices to see how credits are applied
* Plan training or evaluation workloads around available credits
# ORGANIZATION & PROJECTS
Source: https://docs.oumi.ai/admin/org-projects
Setting up your organization & project
Projects provide the organizational structure Oumi uses to manage datasets, evaluations, training runs, and models.
## ORGANIZATION & PROJECTS
When you create your first project, a new default organization is created with you as the admin. You can add others in your organization to collaborate on your projects with. Organizations help you define a group of people that should have access to a set of projects to collaborate.
Projects help you organize all the datasets, evaluators, evaluations, and models that you create for a specific task. A project represents a single use case that ultimately produces one final model. While a project may contain many intermediate or candidate models, they all serve the same use case.
We recommend one project per use case and multiple projects per organization for multiple initiatives.
### CREATING A PROJECT
To create a new project, expand the **Project** drop-down from the top navigation and select the `+ Create new ...` option.
You'll be asked to provide the following:
* Project Name
* Project Context/description
* (Optional) Invite team members
Once created, the project becomes the workspace where all imported and synthesized data lives.
# PERMISSIONS
Source: https://docs.oumi.ai/admin/permissions
Manage users and roles within your organization
## OVERVIEW
User management in Oumi is handled from a centralized location within the UI, making it easy to manage access and roles across your organization.
To access user settings:
1. Click your profile icon in the top navigation bar.
2. Select **Organization Settings** from the dropdown menu.
3. View the list of organization members and update roles as needed.
You can also invite new members and track the status of pending invitations.
## OUMI ROLES
Roles define what actions a user can take at both the organization and project levels. By combining organization-level and project-level roles, you can control access broadly across your workspace while still managing more granular permissions within individual projects.
### ORGANIZATION ROLES
* **Owner** - the top-level role with complete authority, including managing settings, users, and billing. Each organization has a single designated owner role.
* **Admins** - have broad management capabilities, such as inviting users and configuring organization settings, but cannot delete the organization or manage payment methods.
* **Members** - limited to viewing and working within projects they've created or been added to.
| Role | Description |
| ------ | ---------------------------------------------------------------------------- |
| Owner | Full control over the organization (one per org) |
| Admin | Full management rights; cannot delete the org or manage payment methods |
| Member | Standard access; can only view and use projects they are explicitly added to |
### PROJECT ROLES
Project roles define what a user can do within a specific project.
* **Project Admin** - has full control, including managing members, updating settings, and deleting the project.
* **Editor** - can actively contribute by creating and modifying resources such as datasets, models, and evaluations.
* **Viewer** - has read-only access and can explore project resources without making changes.
| Role | Description |
| ------ | ----------------------------------------------------------------------------- |
| Admin | Full control over the project (manage members, edit settings, delete project) |
| Editor | Create and modify resources (datasets, models, evaluations, etc.) |
| Viewer | Read-only access to all project resources |
### HOW ROLES INTERACT
Organization and project roles work together to determine a user’s effective access level. Organization owners and admins automatically have full access to every project in the organization, and do not need to be assigned separate project roles. In contrast, organization members do not have access to any projects by default, and must be explicitly added to each project with an assigned role.
Roles follow a clear hierarchy within a project: **Admin > Editor > Viewer**.
Higher-level roles inherit the permissions of the roles below them, meaning an Admin can perform all actions available to editors and viewers, and an Editor can perform all actions available to viewers.
### ROLE ASSIGNMENT & ACCESS CONTROL
A user must belong to an organization before they can be assigned any role within its projects. If a user is removed from the organization, they immediately lose access to all associated projects.
Each organization is limited to a single owner. This role cannot be removed directly; ownership must be transferred to another user if a change is needed.
Organization owners and admins have visibility into all projects by default, while members can only see and access the projects they have been explicitly added to.
### ORGANIZATION ACCESS & PERMISSIONS
The following matrix shows what actions owners, admins, and members can perform within an organization.
| Action | Owner | Admin | Member |
| ----------------------------------------- | ----- | ----- | ------ |
| View org details | yes | yes | yes |
| Update org settings | yes | yes | no |
| Delete org | yes | no | no |
| Invite users to org | yes | yes | no |
| Remove users from org | yes | yes | no |
| Update user roles in org | yes | yes | no |
| Cancel pending invitations | yes | yes | no |
| View org member list | yes | yes | yes |
| View pending invitations | yes | yes | yes |
| Manage org-level API keys (LLM providers) | yes | yes | no |
| View org-level API keys | yes | yes | yes |
| Access billing portal | yes | yes | no |
| View payment methods | yes | yes | no |
| Update/delete payment methods | yes | no | no |
| View org stats | yes | yes | yes |
| Create projects | yes | yes | yes |
| List projects (accessible only) | yes | yes | yes |
### PROJECT PERMISSIONS
The following matrix defines what each role can do within a project, from managing settings and members to creating and using resources.
| Action | Org Owner/Admin | Project Admin | Project Editor | Project Viewer |
| --------------------------------------------------------------------------- | --------------- | ------------- | -------------- | -------------- |
| View project | yes | yes | yes | yes |
| Update project settings | yes | yes | no | no |
| Delete project | yes | yes | no | no |
| Add/remove project members | yes | yes | no | no |
| Update project member roles | yes | yes | no | no |
| Manage project API keys (LLM providers) | yes | yes | yes | no |
| View project API keys | yes | yes | yes | yes |
| Create resources (datasets, models, experiments, evaluators, recipes, tags) | yes | yes | yes | no |
| Edit/delete resources | yes | yes | yes | no |
| Run operations (training, evaluation, analysis, synthesis) | yes | yes | yes | no |
| Stop running operations | yes | yes | yes | no |
| View resources and results | yes | yes | yes | yes |
| Download datasets and models | yes | yes | yes | yes |
| Upload datasources | yes | yes | yes | no |
| Use Agent chat | yes | yes | yes | no |
| Preview synthesis/evaluators | yes | yes | yes | yes |
# QUOTAS
Source: https://docs.oumi.ai/admin/quotas
View usage limits across your organization
## OVERVIEW
An organization's quota plan defines its usage limits. Limits vary by plan and can be customized for an organization. Individual quotas may apply per organization, project, dataset, or user.
The **Quotas & usage** table shows your organization's current usage and resolved limits.
## VIEW QUOTAS
Organization admins can view current quotas from the **Billing** page:
1. Open your profile menu and select **Organization Settings**.
2. Select **Billing**.
3. In the plan card at the top of the page, select **View all quotas**.
The table shows each quota's current usage, limit, and utilization. A limit of **Unlimited** means that quota has no configured cap.
Select **Request quota increase** to contact Oumi about a higher limit.
## STANDARD PLAN LIMITS
The following defaults were recorded in production on August 11, 2026. Custom organization limits may differ.
| Quota | Scope | Free | Pro |
| ---------------------------------------- | ---------------- | -------- | --------- |
| Projects | Organization | 100 | Unlimited |
| Datasets | Organization | 100 | Unlimited |
| Dataset upload size | Upload | 1 GB | 2 GB |
| Data items | Dataset | 200,000 | 200,000 |
| Data source storage | Organization | 1,000 GB | 1,000 GB |
| Training jobs per day | Organization | 10 | 100 |
| Concurrent training jobs | Organization | 1 | 1 |
| Evaluation jobs per day | Organization | 10 | 100 |
| Concurrent evaluation jobs | Organization | 4 | 10 |
| Concurrent synthesis jobs | Organization | 4 | 10 |
| Agent requests per minute | User | 20 | 20 |
| Custom deployments | Organization | 0 | 10 |
| GPUs used by active custom deployments | Organization | 0 | 20 |
| Concurrent dataset exports | Organization | 1 | Unlimited |
| Model exports per month | Organization | 0 | 1 |
| Dedicated inference requests per minute | Deployment | 10 | Unlimited |
| Serverless inference requests per minute | User and project | 10 | Unlimited |
| Inference log retention | Organization | 7 days | Unlimited |
Resources not listed have no configured cap in either standard plan. Check **Quotas & usage** for your organization's resolved limits.
## API ERRORS
Quota checks run when a limited action starts.
| Condition | HTTP status | Error type |
| ------------------------------- | ----------- | ------------------------ |
| Count, size, or concurrency cap | `403` | `QuotaExceededError` |
| Rolling-window rate limit | `429` | `RateLimitExceededError` |
A `429` response may include a `Retry-After` header indicating when the rolling-window limit becomes available again.
## LIMIT TYPES
| Type | Description |
| ----------- | --------------------------------------------------- |
| Total count | Maximum number of resources |
| Per period | Maximum requests or jobs in a minute, day, or month |
| Concurrent | Maximum jobs running at the same time |
| Storage | Maximum storage usage in GB |
| GPU count | Maximum GPUs used by active custom deployments |
| Retention | Number of days stored data remains available |
# TROUBLESHOOTING & ERRORS
Source: https://docs.oumi.ai/admin/troubleshooting-errors
Common issues, error responses, and how to resolve them
## OVERVIEW
This section offers basic troubleshooting guidance to help you identify and resolve issues efficiently. It explains common error responses, what they mean, and the steps you can take to address them, so you can quickly get your requests back on track.
Please [use this form](https://oumi.ai/contact) or reach out to your Oumi representative if you need in-person assistance.
## AGENT ASSISTANCE
In most situations, the Oumi Agent is the fastest and most reliable way to troubleshoot issues. Simply describe your problem in natural language, and the Agent will analyze your current platform settings and project assets (e.g., datasets, evaluators, models, recipes, and files) to identify and recommend a solution.
For example, if your model is underperforming on a specific evaluation metric, you can ask the Agent for help diagnosing and resolving the issue:
```prompt theme={null}
Help me determine why my customer service model is scoring only 19.0% in accuracy.
```
## ERROR RESPONSES
If you’re accessing Oumi programmatically, the following are common error responses, along with their status codes and meanings, to help you better understand and troubleshoot your web requests.
| Status Code | Meaning |
| ----------- | -------------------------------------------------- |
| 400 | Invalid request input |
| 401 | Not authenticated (missing or invalid credentials) |
| 402 | Payment method required |
| 403 | Permission denied or quota exceeded |
| 404 | Resource not found |
| 409 | Conflict (duplicate resource, etc.) |
# USER MANAGEMENT
Source: https://docs.oumi.ai/admin/users
Manage users and roles within your organization
Oumi provides a centralized place to manage users, so you can easily control access, assign roles, and keep permissions consistent across your entire organization.
To access your organization's user settings:
1. Click your profile icon in the top navigation bar.
2. Select **Organization Settings** from the dropdown menu.
3. View the list of organization members and update roles as needed.
### INVITING NEW USERS
You can invite new members to your organization and track the status of each invitation from a single place. When sending an invite, you’ll assign the user an organization role (`Member` or `Admin`), with the option to set a default project role as well.
Invitations remain valid for a limited time and will expire if not accepted within that period.
Each invitee receives an email with a link to join. If they don’t already have an Oumi account, they’ll be prompted to create one before accepting the invitation.
# CONFIGURATION AND PROFILES
Source: https://docs.oumi.ai/cli/basics/configuration
How the CLI remembers your workspace and settings
The CLI always runs against an active profile (`default` unless you switch). Profiles
store your organization, project, and other settings there so you do not need
to pass them on every command.
## CREATE AND USE ANOTHER PROFILE
```bash theme={null}
oumi-cli config profile add workspace-b
oumi-cli config profile use workspace-b
```
## SAVED VALUES
| Setting | Profile key | Change it with |
| ------------- | ------------ | ------------------------------ |
| Organization | `org_id` | `organizations set ` |
| Project | `project_id` | `projects set ` |
| Output format | `output` | `config set output ` |
| Log level | `log_level` | `config set log_level ` |
Command-line flags and environment variables can override saved values for an
invocation without changing the profile.
Full commands: [config reference](/cli/commands/config).
# OUTPUT FORMATS
Source: https://docs.oumi.ai/cli/basics/output-formats
Output modes, and how to page through long result sets
## OUTPUT
Output determines the shape of the response each command gives.
It can be set in a profile using:
```bash theme={null}
config set output
```
It can also be appended to any command, for example:
```bash theme={null}
oumi-cli datasets list --output json
```
It supports the following modes:
| Mode | Description |
| ------- | ----------------------------------------------------- |
| `json` | One line of JSON. Useful for scripts and agents. |
| `table` | A rendered human readable table. |
| `id` | Resource identifier only. Useful for shell pipelines. |
## PAGINATION
| Option | Default | Description |
| ------------- | ------- | ------------------------------------------------------ |
| `--limit` | 100 | Stop after at most this many items. |
| `--all` | | Fetch until results run out, bounded by `--max-items`. |
| `--max-items` | 10000 | Hard cap that applies with `--all`. |
Global flags and exit codes: [global options](/cli/commands/global-options).
# RECIPES
Source: https://docs.oumi.ai/cli/basics/recipes
Create a reusable configuration and launch a run from it.
A recipe is the reusable config a training, evaluation, or synthesis run
launches from. The easiest way to create one is to generate a JSON
template, edit it, then pass that file to `recipes create`.
The example below uses `--type training`, but the same flow works for
`evaluate` and `synthesis`.
## CREATE AND USE A RECIPE
Write a template to `recipe.json`:
```bash theme={null}
oumi-cli recipes generate-template \
--type training \
--output json > recipe.json
```
Replace the placeholders and remove fields you do not need. After
trimming, a training recipe can look like:
```json theme={null}
{
"displayName": "support-ticket-classifier",
"recipeConfig": {
"type": "train",
"trainingConfig": {
"model": { "modelName": "Qwen/Qwen3.5-4B-Base" },
"data": { "train": { "datasets": [{ "datasetId": 1 }] } }
}
}
}
```
```bash theme={null}
oumi-cli recipes create \
--type training \
--input-json-file recipe.json \
--output json
```
Full commands: [recipes reference](/cli/commands/recipes).
# AUTH
Source: https://docs.oumi.ai/cli/commands/auth
Sign in to Oumi and check your authentication status
| Command | Description |
| ------------------- | ---------------------------------------------------------------------------- |
| [`login`](#login) | Validate an API key against the server, then persist it for future commands. |
| [`logout`](#logout) | Remove any saved API key from the active profile's config. |
| [`status`](#status) | Show whether an API key is configured and verify it with the API. |
***
## LOGIN
Validate an API key against the server, then persist it for future commands.
```bash theme={null}
oumi-cli auth login [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ---------------- | ---- | -------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `--api-key` | str | | API key value to validate and persist (mutually exclusive with `--api-key-file`). Omit both to be prompted interactively on a TTY. |
| `--api-key-file` | path | | Path to a file containing the API key to validate and persist (use - for stdin). Mutually exclusive with `--api-key`. |
***
## LOGOUT
Remove any saved API key from the active profile's config.
```bash theme={null}
oumi-cli auth logout [OPTIONS]
```
***
## STATUS
Show whether an API key is configured and verify it with the API.
```bash theme={null}
oumi-cli auth status [OPTIONS]
```
***
# CONFIG
Source: https://docs.oumi.ai/cli/commands/config
Manage local CLI configuration and named profiles
| Command | Description |
| ----------------------------------- | --------------------------------------------------------------------------- |
| [`describe`](#describe) | Describe each settable config key: meaning, default, env var, valid values. |
| [`get`](#get) | Print the value of a single config key from the active profile. |
| [`init`](#init) | Create a commented config-file template for the active profile. |
| [`list`](#list) | Show every config key in the active profile with its current value. |
| [`path`](#path) | Print the absolute path of the config file in use; exit 1 if none exists. |
| [`profile add`](#profile-add) | Create a new, empty profile in the config file. |
| [`profile list`](#profile-list) | List every profile in the config file, flagging which one is active. |
| [`profile remove`](#profile-remove) | Remove a named profile from the config file. |
| [`profile use`](#profile-use) | Switch the active profile by rewriting default\_profile in the config file. |
| [`set`](#set) | Set a config key in the active profile, persisting it to the config file. |
| [`unset`](#unset) | Remove a key from the active profile (clean no-op if the key is not set). |
***
## DESCRIBE
Describe each settable config key: meaning, default, env var, valid values.
```bash theme={null}
oumi-cli config describe [OPTIONS]
```
***
## GET
Print the value of a single config key from the active profile.
```bash theme={null}
oumi-cli config get [OPTIONS] [key]
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------------------------------------------------------------------------------------------------------------- |
| `KEY` | str | | Config key to read from the active profile. Valid keys: api\_base\_url, log\_level, org\_id, output, project\_id. |
***
## INIT
Create a commented config-file template for the active profile.
```bash theme={null}
oumi-cli config init [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| --------- | ---- | -------- | ----------------------------------------------- |
| `--force` | flag | | Overwrite the config file if it already exists. |
***
## LIST
Show every config key in the active profile with its current value.
```bash theme={null}
oumi-cli config list [OPTIONS]
```
***
## PATH
Print the absolute path of the config file in use; exit 1 if none exists.
```bash theme={null}
oumi-cli config path [OPTIONS]
```
***
## PROFILE ADD
Create a new, empty profile in the config file.
```bash theme={null}
oumi-cli config profile add [OPTIONS] {name}
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ------------------------------ |
| `NAME` | str | ✓ | Name of the profile to create. |
**Options**
| Option | Type | Required | Description |
| ------- | ---- | -------- | -------------------------------------------------------- |
| `--use` | flag | | Also switch the active profile to the newly created one. |
***
## PROFILE LIST
List every profile in the config file, flagging which one is active.
```bash theme={null}
oumi-cli config profile list [OPTIONS]
```
***
## PROFILE REMOVE
Remove a named profile from the config file.
```bash theme={null}
oumi-cli config profile remove [OPTIONS] {name}
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ------------------------------ |
| `NAME` | str | ✓ | Name of the profile to remove. |
**Options**
| Option | Type | Required | Description |
| --------- | ---- | -------- | ---------------------------------------------------------- |
| `--force` | flag | | Allow removing the profile even if it is currently active. |
***
## PROFILE USE
Switch the active profile by rewriting default\_profile in the config file.
```bash theme={null}
oumi-cli config profile use [OPTIONS] {name}
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ---------------------------------------- |
| `NAME` | str | ✓ | Name of an existing profile to activate. |
***
## SET
Set a config key in the active profile, persisting it to the config file.
```bash theme={null}
oumi-cli config set [OPTIONS] [key] [value]
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ---------------------------------------------------------------------------------------- |
| `KEY` | str | | Config key to set. Valid keys: api\_base\_url, log\_level, org\_id, output, project\_id. |
| `VALUE` | str | | New value for the key (run `config describe` for keys with constrained values). |
***
## UNSET
Remove a key from the active profile (clean no-op if the key is not set).
```bash theme={null}
oumi-cli config unset [OPTIONS] [key]
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ------------------------------------------------------------------------------------------------------------------- |
| `KEY` | str | | Config key to remove from the active profile. Valid keys: api\_base\_url, log\_level, org\_id, output, project\_id. |
***
# DATASETS
Source: https://docs.oumi.ai/cli/commands/datasets
Upload, version, analyze, and synthesize datasets
| Command | Description |
| ----------------------------------------- | ------------------------------------------------------------------------------------ |
| [`analysis download`](#analysis-download) | Download a completed analysis's artifacts to local disk. |
| [`analysis get`](#analysis-get) | Show a completed analysis's results in place. |
| [`analysis list`](#analysis-list) | List completed quality analyses for a dataset. |
| [`delete`](#delete) | Permanently delete a dataset from the project. |
| [`distill`](#distill) | Distill a dataset by generating completions from a model, waiting for it to finish. |
| [`download`](#download) | Download a dataset version to local disk. |
| [`get`](#get) | Show metadata for a single dataset. |
| [`import-logs`](#import-logs) | Import a LangFuse or LangSmith trace export as a new dataset. |
| [`items bulk-delete`](#items-bulk-delete) | Bulk-delete rows by explicit ID or by analysis-test failures. |
| [`items get`](#items-get) | Show a single row (data item) from a dataset. |
| [`items list`](#items-list) | List the rows (data items) in a dataset, optionally filtered. |
| [`list`](#list) | List datasets in a project, optionally filtered and sorted. |
| [`supported-models`](#supported-models) | List models suggested for dataset synthesis, before authoring a `synthesize` recipe. |
| [`synthesize`](#synthesize) | Synthesize a new dataset from a recipe, waiting for it to finish. |
| [`update`](#update) | Rename a dataset. |
| [`upload`](#upload) | Upload a local file as a new dataset or a new version of an existing one. |
| [`versions delete`](#versions-delete) | Permanently delete a dataset version. |
| [`versions get`](#versions-get) | Show metadata for a single dataset version. |
| [`versions list`](#versions-list) | List versions of a dataset, newest first by default. |
| [`versions restore`](#versions-restore) | Restore a dataset to a prior version by creating a new version with its content. |
***
## ANALYSIS DOWNLOAD
Download a completed analysis's artifacts to local disk.
```bash theme={null}
oumi-cli datasets analysis download [OPTIONS] {dataset_id} {analysis_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------- | ---- | -------- | --------------------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset that owns the analysis (see `datasets list`). |
| `ANALYSIS_ID` | int | ✓ | ID of the analysis to download (see `datasets analysis list`). |
**Options**
| Option | Type | Required | Description |
| -------------- | --------- | -------- | --------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--format` | choice | | Artifact format to download. Default: `jsonl`. |
| `--out` | path | | Destination file or directory. Defaults to ./analysis-\-\.\ in the current directory. |
| `--expiration` | int range | | Lifetime of the download's presigned URL, in seconds (default 3600, max 604800). |
***
## ANALYSIS GET
Show a completed analysis's results in place.
```bash theme={null}
oumi-cli datasets analysis get [OPTIONS] {dataset_id} {analysis_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------- | ---- | -------- | --------------------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset that owns the analysis (see `datasets list`). |
| `ANALYSIS_ID` | int | ✓ | ID of the analysis to inspect (see `datasets analysis list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## ANALYSIS LIST
List completed quality analyses for a dataset.
```bash theme={null}
oumi-cli datasets analysis list [OPTIONS] {dataset_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | --------------------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset whose analyses to list (see `datasets list`). |
**Options**
| Option | Type | Required | Description |
| ------------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--dataset-version` | int | | Filter to analyses of this dataset version. Omit to see analyses across every version. (Named `--dataset-version` to avoid colliding with the global `--version` flag.) |
| `--limit` | int range | | Return at most N analyses (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every completed analysis, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
***
## DELETE
Permanently delete a dataset from the project.
```bash theme={null}
oumi-cli datasets delete [OPTIONS] {dataset_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | -------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset to delete (see `datasets list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--force` | flag | | Skip the confirmation prompt (alias for `--yes`). |
| `--yes` | flag | | Skip the confirmation prompt; useful in scripts. |
***
## DISTILL
Distill a dataset by generating completions from a model, waiting for it to finish.
```bash theme={null}
oumi-cli datasets distill [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ---------------------- | ---- | -------- | ------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to distill in; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--input-json` | str | | Request body (DistillDatasetRequest) as an inline JSON string. Pass exactly one of `--input-json` or `--input-json-file`. |
| `--input-json-file` | path | | Path to a file holding the request body as JSON. Pass exactly one of `--input-json` or `--input-json-file`. |
| `--wait` / `--no-wait` | flag | | Wait for the distillation to finish (default); `--no-wait` returns the operation ID immediately. |
***
## DOWNLOAD
Download a dataset version to local disk.
```bash theme={null}
oumi-cli datasets download [OPTIONS] {dataset_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ---------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset to download (see `datasets list`). |
**Options**
| Option | Type | Required | Description |
| ------------------- | ---- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--dataset-version` | int | | Version ID to download. Defaults to the latest version. (Named `--dataset-version` to avoid colliding with the global `--version` flag.) |
| `--out` | path | | Destination file or directory. Defaults to ./dataset-\-\.\ in the current directory. |
| `--expiration` | int | | Lifetime of the download's presigned URL, in seconds (default 3600). |
***
## GET
Show metadata for a single dataset.
```bash theme={null}
oumi-cli datasets get [OPTIONS] {dataset_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | --------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset to inspect (see `datasets list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## IMPORT-LOGS
Import a LangFuse or LangSmith trace export as a new dataset.
```bash theme={null}
oumi-cli datasets import-logs [OPTIONS] {file}
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ------------------------------------------------------------------------------------------------ |
| `FILE` | path | ✓ | Path to the local .jsonl trace export (a LangFuse Tracing-tab export or a LangSmith run export). |
**Options**
| Option | Type | Required | Description |
| -------------------------- | ------ | -------- | --------------------------------------------------------------------------------------------------------------------- |
| `--platform` | choice | ✓ | Observability platform the export came from. One of: `langfuse`, `langsmith`. |
| `--display-name` | str | ✓ | Display name for the new dataset. |
| `--project` | str | | Project to import the dataset into; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--dedupe` / `--no-dedupe` | flag | | Drop prefix-overlap rows where one trace's full conversation is the prior turns of a longer trace in the same export. |
| `--wait` / `--no-wait` | flag | | Wait for the dataset ingest to finish (default); `--no-wait` returns the operation ID immediately. |
| `--dry-run` | flag | | Validate the file as JSONL and preview its row count locally; makes no API calls and imports nothing. |
***
## ITEMS BULK-DELETE
Bulk-delete rows by explicit ID or by analysis-test failures.
```bash theme={null}
oumi-cli datasets items bulk-delete [OPTIONS] {dataset_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ------------------------------------------------------------ |
| `DATASET_ID` | int | ✓ | ID of the dataset to delete rows from (see `datasets list`). |
**Options**
| Option | Type | Required | Description |
| ----------------- | ---- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--id` | str | | Row ID to delete (repeatable). Mutually exclusive with `--analysis-test`. |
| `--analysis-test` | int | | Delete every row flagged by this analysis's failing tests (its test\_affected\_rows artifact). Mutually exclusive with `--id`. |
| `--test-id` | str | | Narrow `--analysis-test` to specific test IDs (repeatable); omit to delete rows flagged by every test in the analysis. Requires `--analysis-test`. |
| `--force` | flag | | Skip the confirmation prompt (alias for `--yes`). |
| `--yes` | flag | | Skip the confirmation prompt; useful in scripts. |
***
## ITEMS GET
Show a single row (data item) from a dataset.
```bash theme={null}
oumi-cli datasets items get [OPTIONS] {dataset_id} {dataitem_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------- | ---- | -------- | ----------------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset the row belongs to (see `datasets list`). |
| `DATAITEM_ID` | str | ✓ | ID of the row to inspect (see `items list`). |
**Options**
| Option | Type | Required | Description |
| ------------------- | ---- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--dataset-version` | int | | Specific dataset version to read the row from. Defaults to the latest version. (Named `--dataset-version` to avoid colliding with the global `--version` flag.) |
***
## ITEMS LIST
List the rows (data items) in a dataset, optionally filtered.
```bash theme={null}
oumi-cli datasets items list [OPTIONS] {dataset_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ---------------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset to list rows from (see `datasets list`). |
**Options**
| Option | Type | Required | Description |
| ------------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--dataset-version` | int | | Specific dataset version to list rows from. Defaults to the latest version. (Named `--dataset-version` to avoid colliding with the global `--version` flag.) |
| `--search` | str | | Case-insensitive substring filter applied to each row's content. |
| `--random-offset` | flag | | Start from a random offset instead of the beginning, useful for sampling a few rows without `--all`. Ignored when `--search` is set. |
| `--limit` | int range | | Return at most N rows (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every row, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
***
## LIST
List datasets in a project, optionally filtered and sorted.
```bash theme={null}
oumi-cli datasets list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------------ | --------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to list datasets in; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N datasets (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every dataset, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
| `--search` | str | | Only show datasets whose name matches this search text. |
| `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`. |
| `--sort-direction` | choice | | Sort order (asc is oldest-first). |
***
## SUPPORTED-MODELS
List models suggested for dataset synthesis, before authoring a `synthesize` recipe.
```bash theme={null}
oumi-cli datasets supported-models [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | -------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the lookup to; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## SYNTHESIZE
Synthesize a new dataset from a recipe, waiting for it to finish.
```bash theme={null}
oumi-cli datasets synthesize [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ---------------------- | ---- | -------- | ----------------------------------------------------------------------------------------------------- |
| `--recipe`, `-r` | str | ✓ | Recipe ID (integer), or path to a JSON synthesis recipe file whose recipeConfig.type is `synthesize`. |
| `--display-name` | str | | Display name for the synthesized dataset. Auto-generated if omitted. |
| `--project` | str | | Project to synthesize in; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--wait` / `--no-wait` | flag | | Wait for synthesis to finish (default); `--no-wait` returns the operation ID immediately. |
***
## UPDATE
Rename a dataset.
```bash theme={null}
oumi-cli datasets update [OPTIONS] {dataset_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | -------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset to update (see `datasets list`). |
**Options**
| Option | Type | Required | Description |
| ---------------- | ---- | -------- | ------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--display-name` | str | | New display name for the dataset. |
***
## UPLOAD
Upload a local file as a new dataset or a new version of an existing one.
```bash theme={null}
oumi-cli datasets upload [OPTIONS] {file}
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FILE` | path | ✓ | Path to the local dataset file to upload. Expected format: JSONL (one JSON object per line, max 5 GB). Each row is classified server-side as 'conversation' (a top-level messages list of objects each carrying a role key) or generic 'dict' (arbitrary JSON keys). Check the result with `oumi-cli datasets get`. Use `--dry-run` to preview the detected shape locally first. For 'dict' rows, which key is input vs. label is declared by the consuming evaluator/synthesis config, not the file itself. See `evaluators generate-template` for the field names a given evaluator type expects. |
**Options**
| Option | Type | Required | Description |
| ---------------------- | ---- | -------- | ---------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to upload the dataset into; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--dataset-id` | int | | Upload as a new version of this existing dataset. Omit to create a brand-new dataset. |
| `--display-name` | str | | Display name for the dataset. Defaults to the local filename. |
| `--expiration` | int | | Lifetime of the upload's presigned URL, in seconds (default 3600). |
| `--wait` / `--no-wait` | flag | | Wait for the dataset ingest to finish (default); `--no-wait` returns the operation ID immediately. |
| `--dry-run` | flag | | Validate the file as JSONL and preview its row count and detected schema type; makes no API calls and uploads nothing. |
***
## VERSIONS DELETE
Permanently delete a dataset version.
```bash theme={null}
oumi-cli datasets versions delete [OPTIONS] {dataset_id} {version_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ----------------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset (see `datasets list`). |
| `VERSION_ID` | int | ✓ | ID of the version to delete (see `datasets versions list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | --------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--force` | flag | | Skip the confirmation prompt (alias for `--yes`). |
| `--yes` | flag | | Skip the confirmation prompt; useful in scripts. Cannot delete the only version of a dataset. |
***
## VERSIONS GET
Show metadata for a single dataset version.
```bash theme={null}
oumi-cli datasets versions get [OPTIONS] {dataset_id} {version_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ------------------------------------------------------------ |
| `DATASET_ID` | int | ✓ | ID of the dataset (see `datasets list`). |
| `VERSION_ID` | int | ✓ | ID of the version to inspect (see `datasets versions list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## VERSIONS LIST
List versions of a dataset, newest first by default.
```bash theme={null}
oumi-cli datasets versions list [OPTIONS] {dataset_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | --------------------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset whose versions to list (see `datasets list`). |
**Options**
| Option | Type | Required | Description |
| ------------------ | --------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N versions (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every version, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
| `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`, `version_id`. |
| `--sort-direction` | choice | | Sort order (asc is oldest-first). |
***
## VERSIONS RESTORE
Restore a dataset to a prior version by creating a new version with its content.
```bash theme={null}
oumi-cli datasets versions restore [OPTIONS] {dataset_id} {version_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | --------------------------------------------------------------- |
| `DATASET_ID` | int | ✓ | ID of the dataset (see `datasets list`). |
| `VERSION_ID` | int | ✓ | ID of the version to restore to (see `datasets versions list`). |
**Options**
| Option | Type | Required | Description |
| ---------------- | ---- | -------- | ------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the dataset; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--display-name` | str | | Display name for the new version. Defaults to 'Restore to v\'. |
| `--description` | str | | Description for the new version. |
***
# DEPLOYMENTS
Source: https://docs.oumi.ai/cli/commands/deployments
Serve models behind an Oumi inference endpoint
| Command | Description |
| ----------------------------------------- | --------------------------------------------------------------------------------- |
| [`chat`](#chat) | Open an interactive streaming chat session with a deployment. |
| [`create`](#create) | Create a new model deployment from a JSON request body. |
| [`delete`](#delete) | Delete a deployment by ID. |
| [`endpoint`](#endpoint) | Print the inference endpoint, model path, and a runnable curl snippet. |
| [`generate-template`](#generate-template) | Print a ready-to-edit CreateDeploymentRequest body for the given deployment type. |
| [`get`](#get) | Get a deployment by ID. |
| [`health`](#health) | Report traffic quality for a deployment over a time range. |
| [`invoke`](#invoke) | Run one inference call against a deployment and print the reply. |
| [`is-ready`](#is-ready) | Check whether a deployment has an active replica available to serve inference. |
| [`list`](#list) | List deployments in the project. |
| [`logs delete`](#logs-delete) | Delete inference log entries for a deployment, optionally scoped to a time range. |
| [`logs export`](#logs-export) | Export a time range of inference logs into a new or existing dataset. |
| [`logs get`](#logs-get) | Get a single inference log entry by ID, including the full request/response. |
| [`logs list`](#logs-list) | List inference log entries for a deployment. |
| [`supported-models`](#supported-models) | List the models this project can deploy, before `create`/`generate-template`. |
| [`targets list`](#targets-list) | List the inference targets attached to a deployment. |
| [`update`](#update) | Update a deployment by ID. |
| [`wake-up`](#wake-up) | Send a dummy request to scale a provider-idled deployment back up. |
***
## CHAT
Open an interactive streaming chat session with a deployment.
```bash theme={null}
oumi-cli deployments chat [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | --------------------------- |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID to chat with. |
**Options**
| Option | Type | Required | Description |
| -------------------------------- | ----------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--system` | str | | System instructions. |
| `--temperature` | float range | | Sampling temperature. |
| `--max-tokens` | int range | | Max output tokens. |
| `--top-p` | float range | | Nucleus sampling top-p. |
| `--timeout` | int range | | Seconds to wait for an idled deployment to wake up before the session starts, and the default for '/wake-up'; 0 waits indefinitely. Default: `600`. |
| `--auto-wake` / `--no-auto-wake` | flag | | Wake the deployment and re-send the turn when it has idled mid-session; togglable in-session with '/auto-wake'. |
| `--plain` / `--no-plain` | flag | | Print a flat transcript instead of chat bubbles; already the default under NO\_COLOR or a dumb terminal. |
***
## CREATE
Create a new model deployment from a JSON request body.
```bash theme={null}
oumi-cli deployments create [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ---------------------- | ---- | -------- | ---------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to create the deployment in; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--input-json` | str | | Request body as an inline JSON string. Pass exactly one of `--input-json` or `--input-json-file`. |
| `--input-json-file` | path | | Path to a file holding the request body as JSON. Pass exactly one of `--input-json` or `--input-json-file`. |
| `--wait` / `--no-wait` | flag | | Wait for the deployment to become active before returning (default); `--no-wait` returns the operation ID immediately. |
***
## DELETE
Delete a deployment by ID.
```bash theme={null}
oumi-cli deployments delete [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ------------------------ |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID to act on. |
**Options**
| Option | Type | Required | Description |
| ---------------------- | ---- | -------- | --------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--force` | flag | | Skip confirmation prompt. |
| `--yes` | flag | | Skip confirmation prompt (alias for `--force`). |
| `--wait` / `--no-wait` | flag | | Wait for the deployment to be torn down before returning (default); `--no-wait` returns the operation ID immediately. |
***
## ENDPOINT
Print the inference endpoint, model path, and a runnable curl snippet.
```bash theme={null}
oumi-cli deployments endpoint [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | -------------- |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID. |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ---------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## GENERATE-TEMPLATE
Print a ready-to-edit CreateDeploymentRequest body for the given deployment type.
```bash theme={null}
oumi-cli deployments generate-template [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| -------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------ |
| `--type` | choice | ✓ | Deployment kind to generate a template for; sets modelSpec.type in the output. One of: `custom`, `external`. |
***
## GET
Get a deployment by ID.
```bash theme={null}
oumi-cli deployments get [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ------------------------ |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID to act on. |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ---------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## HEALTH
Report traffic quality for a deployment over a time range.
```bash theme={null}
oumi-cli deployments health [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | --------------------------- |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID to report on. |
**Options**
| Option | Type | Required | Description |
| --------------------- | --------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--time-window` | parse\_time\_window | | Report over the last DURATION, e.g. '3h', '45m', '7d'. Mutually exclusive with `--start`/`--end`. Give none of `--time-window`/`--start`/`--end` to report over the service's own default range. |
| `--end` | parse\_iso\_timestamp | | End of the reported range (ISO-8601, UTC, exclusive). Defaults to now. Must not be in the future. Mutually exclusive with `--time-window`. |
| `--start` | parse\_iso\_timestamp | | Start of the reported range (ISO-8601, UTC). Defaults to 24 hours before `--end`. Rounded to the bucket boundary and clamped to available data, so the response start/end give the exact range served. Mutually exclusive with `--time-window`. |
| `--sampling-interval` | choice | | Assert the bucket width the range must be served in. The width is not selectable -- the service derives it from the range, reading raw request logs for short ranges and the hourly rollup for long ones -- so this fails when the range was served in the other width instead of letting coarser data pass for what was asked for. Reported back as 'Bucket Duration'. One of: `1m`, `1h`. |
***
## INVOKE
Run one inference call against a deployment and print the reply.
```bash theme={null}
oumi-cli deployments invoke [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | -------------- |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID. |
**Options**
| Option | Type | Required | Description |
| -------------------------- | ----------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--prompt`, `-p` | str | ✓ | Prompt text ('-' reads from stdin). |
| `--system` | str | | System instructions. |
| `--temperature` | float range | | Sampling temperature. |
| `--max-tokens` | int range | | Max output tokens. |
| `--top-p` | float range | | Nucleus sampling top-p. |
| `--stream` / `--no-stream` | flag | | Stream tokens (default: on a TTY). |
| `--wait` / `--no-wait` | flag | | Wake a scaled-to-zero deployment and wait for it before sending the prompt (default); `--no-wait` fails with exit 9 instead of waiting. |
| `--timeout` | int range | | Seconds the CLI will wait: the wake-up wait and the wait for each response chunk share this one budget, rather than each getting it in full. Bounds waiting only, not connect/write. 0 waits indefinitely (bounded only by Ctrl-C). Default: `600`. |
***
## IS-READY
Check whether a deployment has an active replica available to serve inference.
```bash theme={null}
oumi-cli deployments is-ready [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ----------------------- |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID to check. |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ---------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## LIST
List deployments in the project.
```bash theme={null}
oumi-cli deployments list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to list deployments in; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N deployments (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every deployment, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
***
## LOGS DELETE
Delete inference log entries for a deployment, optionally scoped to a time range.
```bash theme={null}
oumi-cli deployments logs delete [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ----------------------------------- |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID whose logs to delete. |
**Options**
| Option | Type | Required | Description |
| -------------- | -------- | -------- | ---------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--start-time` | datetime | | Delete logs at or after this time (e.g. 2024-01-01T00:00:00). |
| `--end-time` | datetime | | Delete logs at or before this time. |
| `--force` | flag | | Skip confirmation prompt. |
| `--yes` | flag | | Skip confirmation prompt (alias for `--force`). |
***
## LOGS EXPORT
Export a time range of inference logs into a new or existing dataset.
```bash theme={null}
oumi-cli deployments logs export [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ----------------------------------- |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID whose logs to export. |
**Options**
| Option | Type | Required | Description |
| ---------------------- | -------- | -------- | ----------------------------------------------------------------------------------------------------------- |
| `--start-time` | datetime | ✓ | Start of the time range to export, inclusive (e.g. 2024-01-01T00:00:00). |
| `--end-time` | datetime | ✓ | End of the time range to export, inclusive. |
| `--dataset-id` | int | | Append the exported logs to this existing dataset. Mutually exclusive with `--dataset-name`. |
| `--dataset-name` | str | | Create a new dataset with this name from the exported logs. Mutually exclusive with `--dataset-id`. |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--wait` / `--no-wait` | flag | | Wait for the export to finish before returning (default); `--no-wait` returns the operation ID immediately. |
***
## LOGS GET
Get a single inference log entry by ID, including the full request/response.
```bash theme={null}
oumi-cli deployments logs get [OPTIONS] {deployment_id} {log_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | -------------------------------------- |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID that owns the log entry. |
| `LOG_ID` | str | ✓ | ULID of the log entry to fetch. |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ---------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## LOGS LIST
List inference log entries for a deployment.
```bash theme={null}
oumi-cli deployments logs list [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | --------------------------------- |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID whose logs to list. |
**Options**
| Option | Type | Required | Description |
| -------------- | --------- | -------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--start-time` | datetime | | Only include logs at or after this time (e.g. 2024-01-01T00:00:00). |
| `--end-time` | datetime | | Only include logs at or before this time. |
| `--limit` | int range | | Return at most N log entries (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every log entry, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
***
## SUPPORTED-MODELS
List the models this project can deploy, before `create`/`generate-template`.
```bash theme={null}
oumi-cli deployments supported-models [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--source` | choice | | Limit to one source: 'custom' (this project's tuned model versions) or 'external' (curated provider catalog, e.g. OpenAI/Anthropic). Omit to list both. |
| `--project` | str | | Project to scope the lookup to; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## TARGETS LIST
List the inference targets attached to a deployment.
```bash theme={null}
oumi-cli deployments targets list [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ---------------------------------------------- |
| `DEPLOYMENT_ID` | int | ✓ | ID of the deployment (see `deployments list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ---------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## UPDATE
Update a deployment by ID.
```bash theme={null}
oumi-cli deployments update [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ------------------------ |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID to act on. |
**Options**
| Option | Type | Required | Description |
| ---------------------------------------------------------------- | --------- | -------- | --------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--display-name` | str | | New display name for the deployment. |
| `--inference-logging-enabled` / `--no-inference-logging-enabled` | flag | | Enable or disable inference request logging for the deployment. |
| `--min-replicas` | int range | | Minimum replica count for autoscaling. Must be passed together with `--max-replicas`. |
| `--max-replicas` | int range | | Maximum replica count for autoscaling. Must be passed together with `--min-replicas`. |
| `--scale-down-idle-seconds` | int range | | Idle time before scaling down to `--min-replicas`. Requires `--min-replicas`/`--max-replicas` in the same invocation. |
***
## WAKE-UP
Send a dummy request to scale a provider-idled deployment back up.
```bash theme={null}
oumi-cli deployments wake-up [OPTIONS] {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | -------------- |
| `DEPLOYMENT_ID` | int | ✓ | Deployment ID. |
**Options**
| Option | Type | Required | Description |
| ---------------------- | --------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the deployment; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--wait` / `--no-wait` | flag | | Wait for the deployment to be ready to serve inference before returning (default); `--no-wait` returns as soon as the wake-up request is delivered. |
| `--timeout` | int range | | Seconds to wait for readiness under `--wait`; 0 waits indefinitely (bounded only by Ctrl-C). Default: `600`. |
***
# EVALUATIONS
Source: https://docs.oumi.ai/cli/commands/evaluations
Run evaluations and inspect how a model scored
| Command | Description |
| ----------------------------------------------------- | -------------------------------------------------------------------------- |
| [`available-models`](#available-models) | List every modelIdentifier candidate for an evaluation recipe in one call. |
| [`delete`](#delete) | Delete an evaluation by ID (prompts to confirm unless `--yes`/`--force`). |
| [`download`](#download) | Download every evaluation data item to a local JSONL file. |
| [`failure-modes generate`](#failure-modes-generate) | Generate failure modes from an evaluation's results using LLM analysis. |
| [`failure-modes get`](#failure-modes-get) | Show one failure mode by ID. |
| [`failure-modes get-items`](#failure-modes-get-items) | List the sample rows behind a failure mode. |
| [`failure-modes list`](#failure-modes-list) | List the failure modes for a completed evaluation. |
| [`generate-template`](#generate-template) | Print a ready-to-edit evaluation recipe request body. |
| [`get`](#get) | Show one evaluation by ID, with scores and per-evaluator detail. |
| [`list`](#list) | List evaluations in a project, with optional filter, search, and sort. |
| [`results`](#results) | Show per-row evaluation results, one verdict column per evaluator. |
| [`run`](#run) | Submit an evaluation from a recipe and wait for it to finish. |
| [`update`](#update) | Rename an evaluation. |
***
## AVAILABLE-MODELS
List every modelIdentifier candidate for an evaluation recipe in one call.
```bash theme={null}
oumi-cli evaluations available-models [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | --------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--search` | str | | Filter by display name / model name, case-insensitive substring match, applied client-side after merging. |
| `--limit` | int range | | Return at most N rows (default 100), applied to the merged candidate list. |
| `--all` | flag | | Return every candidate (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap on merged rows returned, in any mode (default 10000). |
***
## DELETE
Delete an evaluation by ID (prompts to confirm unless `--yes`/`--force`).
```bash theme={null}
oumi-cli evaluations delete [OPTIONS] {evaluation_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ------------------------------------------------------- |
| `EVALUATION_ID` | int | ✓ | ID of the evaluation (see `oumi-cli evaluations list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ---------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluation; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--force` | flag | | Delete without the confirmation prompt. |
| `--yes` | flag | | Delete without the confirmation prompt. |
***
## DOWNLOAD
Download every evaluation data item to a local JSONL file.
```bash theme={null}
oumi-cli evaluations download [OPTIONS] {evaluation_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | --------------------------------------------------------------------------- |
| `EVALUATION_ID` | int | ✓ | ID of the evaluation whose data items to download (see `evaluations list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluation; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--out` | path | | Destination file or directory. Defaults to ./evaluation-\-dataitems.jsonl in the current directory. |
***
## FAILURE-MODES GENERATE
Generate failure modes from an evaluation's results using LLM analysis.
```bash theme={null}
oumi-cli evaluations failure-modes generate [OPTIONS] {evaluation_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | --------------------------------------------------------- |
| `EVALUATION_ID` | int | ✓ | ID of the evaluation to analyze (see `evaluations list`). |
**Options**
| Option | Type | Required | Description |
| ---------------------- | ---- | -------- | ----------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--wait` / `--no-wait` | flag | | Wait for generation to finish before returning (default); `--no-wait` returns the operation ID immediately. |
***
## FAILURE-MODES GET
Show one failure mode by ID.
```bash theme={null}
oumi-cli evaluations failure-modes get [OPTIONS] {failure_mode_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ----------------- | ---- | -------- | -------------------------------------------------------------- |
| `FAILURE_MODE_ID` | int | ✓ | ID of the failure mode (see `evaluations failure-modes list`). |
**Options**
| Option | Type | Required | Description |
| ----------------- | ---- | -------- | --------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--include-items` | flag | | Include the sample rows in the response, as raw JSON. Use `get-items` for the same rows as a table. |
***
## FAILURE-MODES GET-ITEMS
List the sample rows behind a failure mode.
```bash theme={null}
oumi-cli evaluations failure-modes get-items [OPTIONS] {failure_mode_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ----------------- | ---- | -------- | -------------------------------------------------------------- |
| `FAILURE_MODE_ID` | int | ✓ | ID of the failure mode (see `evaluations failure-modes list`). |
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--limit` | int range | | Return at most N items (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every item, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
***
## FAILURE-MODES LIST
List the failure modes for a completed evaluation.
```bash theme={null}
oumi-cli evaluations failure-modes list [OPTIONS] {evaluation_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ------------------------- |
| `EVALUATION_ID` | int | ✓ | The ID of the evaluation. |
**Options**
| Option | Type | Required | Description |
| ----------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--include-items` | flag | | Include per-item failure-mode detail in the response. |
| `--limit` | int range | | Return at most N failure modes (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every failure mode, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
***
## GENERATE-TEMPLATE
Print a ready-to-edit evaluation recipe request body.
```bash theme={null}
oumi-cli evaluations generate-template [OPTIONS]
```
***
## GET
Show one evaluation by ID, with scores and per-evaluator detail.
```bash theme={null}
oumi-cli evaluations get [OPTIONS] {evaluation_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ------------------------------------------------------- |
| `EVALUATION_ID` | int | ✓ | ID of the evaluation (see `oumi-cli evaluations list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ---------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluation; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## LIST
List evaluations in a project, with optional filter, search, and sort.
```bash theme={null}
oumi-cli evaluations list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------------ | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluation; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N evaluations (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every evaluation, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Absolute cap on rows returned, in every mode (default 10000). |
| `--search` | str | | Keep only evaluations whose display name contains this substring. |
| `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`. |
| `--sort-direction` | choice | | Sort order (asc is oldest-first). |
***
## RESULTS
Show per-row evaluation results, one verdict column per evaluator.
```bash theme={null}
oumi-cli evaluations results [OPTIONS] {evaluation_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ------------------------- |
| `EVALUATION_ID` | int | ✓ | The ID of the evaluation. |
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--evaluator` | int | | Filter rows to a single evaluator's results (by evaluator ID); applied client-side after fetching. |
| `--id` | str | | Filter to a single data item by its ID; applied client-side after fetching. |
| `--limit` | int range | | Return at most N data items (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every data item, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
***
## RUN
Submit an evaluation from a recipe and wait for it to finish.
```bash theme={null}
oumi-cli evaluations run [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ---------------------- | ---- | -------- | -------------------------------------------------------------------------------------------------------- |
| `--name` | str | ✓ | Human-readable display name for this evaluation run. |
| `--recipe`, `-r` | str | ✓ | Recipe ID (integer) or path to a JSON recipe file (recipeConfig.type must be 'evaluate'). |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--wait` / `--no-wait` | flag | | Wait for the run to finish before returning (default); `--no-wait` returns the operation ID immediately. |
***
## UPDATE
Rename an evaluation.
```bash theme={null}
oumi-cli evaluations update [OPTIONS] {evaluation_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | -------------------------------------------------------- |
| `EVALUATION_ID` | int | ✓ | ID of the evaluation to update (see `evaluations list`). |
**Options**
| Option | Type | Required | Description |
| ---------------- | ---- | -------- | ---------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluation; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--display-name` | str | | New display name for the evaluation. |
***
# EVALUATORS
Source: https://docs.oumi.ai/cli/commands/evaluators
Create and manage evaluator resources
| Command | Description |
| ----------------------------------------- | ------------------------------------------------------------------------- |
| [`create`](#create) | Create an evaluator in a project from a JSON request body. |
| [`delete`](#delete) | Delete an evaluator by ID. |
| [`download`](#download) | Download an evaluator as an editable EvaluatorRequest body. |
| [`generate-template`](#generate-template) | Print a ready-to-edit EvaluatorRequest body for the given evaluator type. |
| [`get`](#get) | Get an evaluator by ID. |
| [`list`](#list) | List evaluators in the project. |
| [`update`](#update) | Rename an evaluator, or replace its config from a JSON body. |
| [`versions get`](#versions-get) | Show a single version of an evaluator, including its full config. |
| [`versions list`](#versions-list) | List version history for an evaluator, newest first by default. |
***
## CREATE
Create an evaluator in a project from a JSON request body.
```bash theme={null}
oumi-cli evaluators create [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------------- | ---- | -------- | ----------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to create the evaluator in; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--input-json` | str | | Request body as an inline JSON string. Pass exactly one of `--input-json` or `--input-json-file`. |
| `--input-json-file` | path | | Path to a file holding the request body as JSON. Pass exactly one of `--input-json` or `--input-json-file`. |
***
## DELETE
Delete an evaluator by ID.
```bash theme={null}
oumi-cli evaluators delete [OPTIONS] {evaluator_id}
```
**Arguments**
| Argument | Type | Required | Description |
| -------------- | ---- | -------- | ------------------------------ |
| `EVALUATOR_ID` | int | ✓ | ID of the evaluator to delete. |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or active profile. |
| `--force` | flag | | Skip the confirmation prompt (alias for `--yes`). |
| `--yes` | flag | | Skip the confirmation prompt; useful in scripts. |
***
## DOWNLOAD
Download an evaluator as an editable EvaluatorRequest body.
```bash theme={null}
oumi-cli evaluators download [OPTIONS] {evaluator_id}
```
**Arguments**
| Argument | Type | Required | Description |
| -------------- | ---- | -------- | -------------------------------------------------------- |
| `EVALUATOR_ID` | int | ✓ | ID of the evaluator to download (see `evaluators list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or active profile. |
| `--out` | path | | Destination file or directory. Defaults to ./evaluator-\.json in the current directory. |
***
## GENERATE-TEMPLATE
Print a ready-to-edit EvaluatorRequest body for the given evaluator type.
```bash theme={null}
oumi-cli evaluators generate-template [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ----------- | ------ | -------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `--type` | choice | ✓ | Evaluator kind to generate a template for; sets params.evaluatorType in the output. One of: `judge`, `classification`. |
| `--minimal` | flag | | Emit a small, filled-in worked example for a built-in judge type instead of the full schema. Only supported with `--type` judge. |
***
## GET
Get an evaluator by ID.
```bash theme={null}
oumi-cli evaluators get [OPTIONS] {evaluator_id}
```
**Arguments**
| Argument | Type | Required | Description |
| -------------- | ---- | -------- | -------------------------------- |
| `EVALUATOR_ID` | int | ✓ | ID of the evaluator to retrieve. |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or active profile. |
***
## LIST
List evaluators in the project.
```bash theme={null}
oumi-cli evaluators list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------------ | --------- | -------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to list evaluators in; falls back to OUMI\_PROJECT\_ID or active profile. |
| `--limit` | int range | | Return at most N evaluators (default 100). The CLI loops over pages internally. |
| `--all` | flag | | Return every evaluator, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
| `--search` | str | | Only show evaluators whose name matches this search text. |
| `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`. |
| `--sort-direction` | choice | | Sort order (asc is oldest-first). |
***
## UPDATE
Rename an evaluator, or replace its config from a JSON body.
```bash theme={null}
oumi-cli evaluators update [OPTIONS] {evaluator_id}
```
**Arguments**
| Argument | Type | Required | Description |
| -------------- | ---- | -------- | -------------------------------------------- |
| `EVALUATOR_ID` | int | ✓ | ID of the evaluator (see `evaluators list`). |
**Options**
| Option | Type | Required | Description |
| ------------------- | ---- | -------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--display-name` | str | | New display name for the evaluator. Mutually exclusive with `--input-json`\*. |
| `--input-json` | str | | Full replacement request body as an inline JSON string; creates a new evaluator version. Exclusive of `--display-name`. |
| `--input-json-file` | path | | Path to a file holding the full replacement request body as JSON; creates a new evaluator version. Exclusive of `--display-name`. |
***
## VERSIONS GET
Show a single version of an evaluator, including its full config.
```bash theme={null}
oumi-cli evaluators versions get [OPTIONS] {evaluator_id} {version_id}
```
**Arguments**
| Argument | Type | Required | Description |
| -------------- | ---- | -------- | -------------------------------------------------------------- |
| `EVALUATOR_ID` | int | ✓ | ID of the evaluator (see `evaluators list`). |
| `VERSION_ID` | int | ✓ | ID of the version to inspect (see `evaluators versions list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | --------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## VERSIONS LIST
List version history for an evaluator, newest first by default.
```bash theme={null}
oumi-cli evaluators versions list [OPTIONS] {evaluator_id}
```
**Arguments**
| Argument | Type | Required | Description |
| -------------- | ---- | -------- | -------------------------------------------- |
| `EVALUATOR_ID` | int | ✓ | ID of the evaluator (see `evaluators list`). |
**Options**
| Option | Type | Required | Description |
| ------------------ | --------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N versions (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every version, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
| `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`, `version_id`. |
| `--sort-direction` | choice | | Sort order (asc is oldest-first). |
***
# FILES
Source: https://docs.oumi.ai/cli/commands/files
Upload and manage project files and data sources
| Command | Description |
| ----------------------- | ----------------------------------------------------------------- |
| [`delete`](#delete) | Permanently delete a file from the project. |
| [`download`](#download) | Download a file to local disk. |
| [`get`](#get) | Show metadata for a single file. |
| [`list`](#list) | List files in a project, optionally filtered by kind. |
| [`update`](#update) | Rename a file. |
| [`upload`](#upload) | Upload a local file, waiting for the ingest to finish by default. |
***
## DELETE
Permanently delete a file from the project.
```bash theme={null}
oumi-cli files delete [OPTIONS] {file_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------- | ---- | -------- | -------------------------------------------- |
| `FILE_ID` | int | ✓ | ID of the file to delete (see `files list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ---------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the file; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--force` | flag | | Skip the confirmation prompt (alias for `--yes`). |
| `--yes` | flag | | Skip the confirmation prompt; useful in scripts. |
***
## DOWNLOAD
Download a file to local disk.
```bash theme={null}
oumi-cli files download [OPTIONS] {file_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------- | ---- | -------- | ---------------------------------------------- |
| `FILE_ID` | int | ✓ | ID of the file to download (see `files list`). |
**Options**
| Option | Type | Required | Description |
| -------------- | ---- | -------- | ---------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the file; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--out` | path | | Destination file or directory. Defaults to ./file-\.\ in the current directory. |
| `--expiration` | int | | Lifetime of the download's presigned URL, in seconds (default 3600). |
***
## GET
Show metadata for a single file.
```bash theme={null}
oumi-cli files get [OPTIONS] {file_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------- | ---- | -------- | --------------------------------------------- |
| `FILE_ID` | int | ✓ | ID of the file to inspect (see `files list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ---------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the file; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## LIST
List files in a project, optionally filtered by kind.
```bash theme={null}
oumi-cli files list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to list files in; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N files (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every file, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
| `--kind` | choice | | Filter by file kind: STRUCTURED (tabular data) or UNSTRUCTURED (free-text documents). Omit to return files of every kind. |
***
## UPDATE
Rename a file.
```bash theme={null}
oumi-cli files update [OPTIONS] {file_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------- | ---- | -------- | -------------------------------------------- |
| `FILE_ID` | int | ✓ | ID of the file to update (see `files list`). |
**Options**
| Option | Type | Required | Description |
| ---------------- | ---- | -------- | ---------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the file; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--display-name` | str | | New display name for the file. |
***
## UPLOAD
Upload a local file, waiting for the ingest to finish by default.
```bash theme={null}
oumi-cli files upload [OPTIONS] {file}
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | -------------------------------------------- |
| `FILE` | path | ✓ | Path to the local file to upload (max 5 GB). |
**Options**
| Option | Type | Required | Description |
| ---------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to upload the file into; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--display-name` | str | | Display name for the file. Defaults to the local filename. |
| `--kind` | choice | | STRUCTURED (tabular data) or UNSTRUCTURED (free-text documents). Defaults to UNSTRUCTURED. |
| `--expiration` | int | | Lifetime of the upload's presigned URL, in seconds (default 3600). |
| `--wait` / `--no-wait` | flag | | Wait for the file ingest to finish (default); `--no-wait` returns the operation ID immediately. |
***
# GLOBAL OPTIONS
Source: https://docs.oumi.ai/cli/commands/global-options
Flags, environment variables, and exit codes shared by every command
These options work at any position in the command line, so `oumi-cli --output json datasets list` and `oumi-cli datasets list --output json` do the same thing.
| Option | Type | Description |
| ----------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `--output`, `-o` | `json`, `table`, `id` | Output mode. Defaults to `json` unless both stdin and stdout are TTYs. Env: `OUMI_OUTPUT`. |
| `--profile` | text | Active configuration profile. Default: `default`. Env: `OUMI_PROFILE`. |
| `--config` | path | Path to the `config.toml` file, overriding the default location. Env: `OUMI_CONFIG`. |
| `--log-level` | `debug`, `info`, `warning`, `error`, `critical` | Log level. Defaults to `warning`. Overrides `OUMI_LOG_LEVEL` and the profile. |
| `--verbose`, `-v` | integer | Shortcut for `--log-level`. `-v` sets info, `-vv` sets debug. |
| `--quiet`, `-q` | flag | Suppress non-essential stderr. |
| `--no-input` | flag | Disable interactive prompts. Env: `OUMI_NO_INPUT`. |
| `--command-list` | flag | Print the full command tree as JSON and exit. Renders as a tree on an interactive terminal. |
| `--version` | flag | Print the version and exit. |
| `--help`, `-h` | flag | Show help for the current command and exit. |
For what the three output modes look like and how to page through long lists, see [output formats](/cli/basics/output-formats).
## ENVIRONMENT VARIABLES
| Variable | Description |
| ------------------- | -------------------------------------------------------- |
| `OUMI_API_KEY` | API key for the session. Avoids persisting a credential. |
| `OUMI_ORG_ID` | Default organization. |
| `OUMI_PROJECT_ID` | Default project for commands that take `--project`. |
| `OUMI_OUTPUT` | Default output mode. |
| `OUMI_PROFILE` | Active configuration profile. |
| `OUMI_CONFIG` | Path to the config file. |
| `OUMI_API_BASE_URL` | Base URL of the Oumi API. |
| `OUMI_LOG_LEVEL` | Default log level. |
| `OUMI_NO_INPUT` | Disable interactive prompts. |
Resolution order is the command-line flag, then the environment variable, then the active profile. See [configuration](/cli/basics/configuration).
## EXIT CODES
Commands exit non-zero on failure and write a machine-readable error to stderr in `json` mode:
```json theme={null}
{"error": {"code": "http_not_found", "message": "...", "exit_code": 5}}
```
| Code | Meaning |
| ---- | -------------------------------------------------------------------- |
| 0 | Success |
| 1 | Local failure, or an HTTP error that maps to nothing more specific |
| 2 | Argument or usage error |
| 3 | Network unreachable, or a 5xx from the server |
| 4 | Credentials missing or rejected (401, 403) |
| 5 | Resource not found (404), or a local config key or profile is absent |
| 6 | Conflict, or the resource already exists (409) |
| 7 | Server-side validation failure (422) |
| 8 | Command is registered but not implemented yet |
| 70 | Bug in the CLI itself. Please open an issue. |
| 130 | Interrupted with Ctrl-C |
Prefer checking the exit code over parsing output:
```bash theme={null}
if ! oumi-cli evaluations get "$EVAL_ID" >/dev/null; then
echo "lookup failed with exit code $?"
fi
```
## UTILITY COMMANDS
These sit at the top level and belong to no command group.
```bash theme={null}
oumi-cli version
oumi-cli search train
oumi-cli agent-instructions > AGENTS.md
```
`search` matches against every command path and summary. `agent-instructions` prints an `AGENTS.md`-style guide for driving the CLI from an automated agent.
## DISCOVERING COMMANDS
`--help` works at every level, and any help page is available as JSON for tooling:
```bash theme={null}
oumi-cli datasets upload --help
oumi-cli datasets upload --help --output json
oumi-cli --command-list --output json
```
# MODELS
Source: https://docs.oumi.ai/cli/commands/models
Launch training runs and manage trained models
| Command | Description |
| --------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| [`base-models get`](#base-models-get) | Show details of one supported base model by name. |
| [`base-models list`](#base-models-list) | List the base models the platform can fine-tune. |
| [`delete`](#delete) | Delete a model and all of its versions. |
| [`download`](#download) | Download a model's weights to local disk. |
| [`get`](#get) | Show details of one model by ID. |
| [`hosted-models list`](#hosted-models-list) | List the hosted models suggested as judges and synthesis generators. |
| [`import`](#import) | Import model weights from HuggingFace or a presigned URL. |
| [`list`](#list) | List the models a project owns, trained, imported, or registered. |
| [`register`](#register) | Register an existing pretrained model without copying weights. |
| [`supported-inference-models list`](#supported-inference-models-list) | List the models you can send inference requests to, by oumi canonical name. |
| [`train`](#train) | Train a model from a recipe and wait for the run to finish. |
| [`update`](#update) | Update a model's display name. |
| [`versions get`](#versions-get) | Show a single version of a model, including its full config. |
| [`versions list`](#versions-list) | List version history for one model, newest first by default. |
***
## BASE-MODELS GET
Show details of one supported base model by name.
```bash theme={null}
oumi-cli models base-models get [OPTIONS] {name}
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ------------------------------------------------------------- |
| `NAME` | str | ✓ | Base model name, exact match (see `models base-models list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## BASE-MODELS LIST
List the base models the platform can fine-tune.
```bash theme={null}
oumi-cli models base-models list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N base models (default 100). The server returns the full catalog in one response, so this is a client-side cap. |
| `--all` | flag | | Return every base model (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap on rows fetched, in any mode (default 10000). |
***
## DELETE
Delete a model and all of its versions.
```bash theme={null}
oumi-cli models delete [OPTIONS] {model_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ---------- | ---- | -------- | ------------------------------------ |
| `MODEL_ID` | int | ✓ | ID of the model (see `models list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--force` | flag | | Delete without confirmation prompt. |
| `--yes` | flag | | Delete without confirmation prompt (alias for `--force`). |
***
## DOWNLOAD
Download a model's weights to local disk.
```bash theme={null}
oumi-cli models download [OPTIONS] {model_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ---------- | ---- | -------- | ------------------------------------------------ |
| `MODEL_ID` | int | ✓ | ID of the model to download (see `models list`). |
**Options**
| Option | Type | Required | Description |
| ----------------- | ---- | -------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the model; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--model-version` | int | | Version ID to download. Defaults to the latest version. (Named `--model-version` to avoid colliding with the global `--version` flag.) |
| `--out` | path | | Destination file or directory. Defaults to ./model-\-\.\ in the current directory. |
| `--expiration` | int | | Lifetime of the download's presigned URL, in seconds (default 3600). |
***
## GET
Show details of one model by ID.
```bash theme={null}
oumi-cli models get [OPTIONS] {model_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ---------- | ---- | -------- | ------------------------------------ |
| `MODEL_ID` | int | ✓ | ID of the model (see `models list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## HOSTED-MODELS LIST
List the hosted models suggested as judges and synthesis generators.
```bash theme={null}
oumi-cli models hosted-models list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| -------------------------------------------------- | --------- | -------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N hosted models (default 100). The server returns the full catalog in one response, so this is a client-side cap. |
| `--all` | flag | | Return every hosted model (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap on rows fetched, in any mode (default 10000). |
| `--include-gpu-models` / `--no-include-gpu-models` | flag | | Include GPU pretrained-baseline rows (isPretrained=true) alongside hosted API suggestions. |
***
## IMPORT
Import model weights from HuggingFace or a presigned URL.
```bash theme={null}
oumi-cli models import [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ---------------------- | ---- | -------- | ----------------------------------------------------------------------------------------------- |
| `--name` | str | ✓ | Display name for the imported model. |
| `--hf-repo` | str | | HuggingFace repo to copy weights from (e.g. 'org/model'). Mutually exclusive with `--url`. |
| `--url` | str | | Presigned URL to copy weights from. Mutually exclusive with `--hf-repo`. |
| `--base-model` | str | | Base model the weights derive from (see `models base-models list`). |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--wait` / `--no-wait` | flag | | Wait for the weight copy to finish (default); `--no-wait` returns the operation ID immediately. |
***
## LIST
List the models a project owns, trained, imported, or registered.
```bash theme={null}
oumi-cli models list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------------ | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N models (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every model, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
| `--search` | str | | Filter models by a free-text search over their display name. |
| `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`. |
| `--sort-direction` | choice | | Sort order (asc is oldest-first). |
***
## REGISTER
Register an existing pretrained model without copying weights.
```bash theme={null}
oumi-cli models register [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| -------------- | ---- | -------- | ------------------------------------------------------------------------------------------ |
| `--name` | str | ✓ | Display name for the registered model. |
| `--pretrained` | str | ✓ | Existing HuggingFace baseline to register by name (e.g. 'org/model'), with no weight copy. |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## SUPPORTED-INFERENCE-MODELS LIST
List the models you can send inference requests to, by oumi canonical name.
```bash theme={null}
oumi-cli models supported-inference-models list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the lookup to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N inference models (default 100). The server returns the full catalog in one response, so this is a client-side cap. |
| `--all` | flag | | Return every inference model (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap on rows fetched, in any mode (default 10000). |
***
## TRAIN
Train a model from a recipe and wait for the run to finish.
```bash theme={null}
oumi-cli models train [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ---------------------- | ---- | -------- | -------------------------------------------------------------------------------------------------------- |
| `--name` | str | ✓ | Display name for the trained model. |
| `--recipe`, `-r` | str | | Recipe ID (integer) or path to a JSON recipe file. |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--wait` / `--no-wait` | flag | | Wait for the run to finish before returning (default); `--no-wait` returns the operation ID immediately. |
***
## UPDATE
Update a model's display name.
```bash theme={null}
oumi-cli models update [OPTIONS] {model_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ---------- | ---- | -------- | ------------------------------------ |
| `MODEL_ID` | int | ✓ | ID of the model (see `models list`). |
**Options**
| Option | Type | Required | Description |
| ---------------- | ---- | -------- | ----------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--display-name` | str | | New display name for the model. |
***
## VERSIONS GET
Show a single version of a model, including its full config.
```bash theme={null}
oumi-cli models versions get [OPTIONS] {model_id} {version_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ---------------------------------------------------------- |
| `MODEL_ID` | str | ✓ | ID of the model (see `models list`). |
| `VERSION_ID` | int | ✓ | ID of the version to inspect (see `models versions list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the model; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## VERSIONS LIST
List version history for one model, newest first by default.
```bash theme={null}
oumi-cli models versions list [OPTIONS] {model_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ---------- | ---- | -------- | ------------------------------------ |
| `MODEL_ID` | str | ✓ | ID of the model (see `models list`). |
**Options**
| Option | Type | Required | Description |
| ------------------ | --------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the model; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N versions (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every version, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
| `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`, `version_id`. |
| `--sort-direction` | choice | | Sort order (asc is oldest-first). |
***
# MONITORS
Source: https://docs.oumi.ai/cli/commands/monitors
Score live deployment traffic and surface recurring failure modes
| Command | Description |
| ----------------------------------------------------- | --------------------------------------------------------------------------- |
| [`attach`](#attach) | Attach a deployment to a monitor. |
| [`create`](#create) | Create a monitor that scores sampled deployment traffic. |
| [`delete`](#delete) | Delete Monitor. |
| [`detach`](#detach) | Detach a deployment from a monitor. |
| [`failure-modes generate`](#failure-modes-generate) | Run an LLM failure-mode analysis over a deployment target's scored logs. |
| [`failure-modes runs list`](#failure-modes-runs-list) | List past failure-mode analyses of a monitor target's scored logs. |
| [`get`](#get) | Get Monitor. |
| [`insights`](#insights) | Get per-judge score insights for a deployment target. |
| [`list`](#list) | List Monitors. |
| [`scores`](#scores) | Fetch score detail for the logs a monitor scored. |
| [`update`](#update) | Update a monitor's judges, sample rate, or name (at least one is required). |
***
## ATTACH
Attach a deployment to a monitor.
```bash theme={null}
oumi-cli monitors attach [OPTIONS] {monitor_id} {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ------------------------- |
| `MONITOR_ID` | int | ✓ | The ID of the monitor. |
| `DEPLOYMENT_ID` | int | ✓ | The ID of the deployment. |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
***
## CREATE
Create a monitor that scores sampled deployment traffic.
```bash theme={null}
oumi-cli monitors create [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ---------------- | ----- | -------- | ------------------------------------------------------------------------------------------------ |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--judge` | int | ✓ | Evaluator to judge sampled traffic with, at its latest version; repeat `--judge` to add several. |
| `--sample-rate` | float | ✓ | Fraction of traffic to score, between 0 and 1. |
| `--display-name` | str | | Human-readable name for the monitor. |
***
## DELETE
Delete Monitor.
```bash theme={null}
oumi-cli monitors delete [OPTIONS] {monitor_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ---------------------- |
| `MONITOR_ID` | int | ✓ | The ID of the monitor. |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--force` | flag | | Skip confirmation prompt. |
| `--yes` | flag | | Skip confirmation prompt (alias for `--force`). |
***
## DETACH
Detach a deployment from a monitor.
```bash theme={null}
oumi-cli monitors detach [OPTIONS] {monitor_id} {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | ------------------------- |
| `MONITOR_ID` | int | ✓ | The ID of the monitor. |
| `DEPLOYMENT_ID` | int | ✓ | The ID of the deployment. |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--force` | flag | | Skip confirmation prompt. |
| `--yes` | flag | | Skip confirmation prompt (alias for `--force`). |
***
## FAILURE-MODES GENERATE
Run an LLM failure-mode analysis over a deployment target's scored logs.
```bash theme={null}
oumi-cli monitors failure-modes generate [OPTIONS] {monitor_id} {deployment_id} {target_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | --------------------------------------- |
| `MONITOR_ID` | int | ✓ | The ID of the monitor. |
| `DEPLOYMENT_ID` | int | ✓ | The public ID of the deployment. |
| `TARGET_ID` | int | ✓ | The public ID of the deployment target. |
**Options**
| Option | Type | Required | Description |
| ---------------------- | --------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--time-window` | parse\_time\_window | | Analyze the logs scored in the last DURATION instead of an explicit `--start`/`--end`, e.g. '1h', '30m', '7d'. Mutually exclusive with `--start`/`--end`. Defaults to 1 hour when none of `--time-window`/`--start`/`--end` is given. |
| `--end` | parse\_iso\_timestamp | | End of the scored-logs window (inclusive), e.g. 2024-01-01T00:00:00. Defaults to now. Mutually exclusive with `--time-window`. |
| `--start` | parse\_iso\_timestamp | | Start of the scored-logs window (inclusive), e.g. 2024-01-01T00:00:00. Defaults to 1 hour before `--end`. Mutually exclusive with `--time-window`. |
| `--display-name` | str | | Display name for the analysis run this creates. |
| `--skip-failure-modes` | flag | | Materialize the evaluation of the window's logs but stop before deriving failure modes from it. Leaves a run `oumi-cli evaluations failure-modes generate` can analyze later. |
| `--project` | str | | Project that owns the monitor; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--wait` / `--no-wait` | flag | | Wait for the analysis to finish before returning (default); `--no-wait` returns the operation ID immediately. |
***
## FAILURE-MODES RUNS LIST
List past failure-mode analyses of a monitor target's scored logs.
```bash theme={null}
oumi-cli monitors failure-modes runs list [OPTIONS] {monitor_id} {deployment_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | -------------------------------- |
| `MONITOR_ID` | int | ✓ | The ID of the monitor. |
| `DEPLOYMENT_ID` | int | ✓ | The public ID of the deployment. |
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the monitor; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N items (default 100). The CLI loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return everything, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
***
## GET
Get Monitor.
```bash theme={null}
oumi-cli monitors get [OPTIONS] {monitor_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ---------------------- |
| `MONITOR_ID` | int | ✓ | The ID of the monitor. |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
***
## INSIGHTS
Get per-judge score insights for a deployment target.
```bash theme={null}
oumi-cli monitors insights [OPTIONS] {monitor_id} {deployment_id} {target_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | --------------------------------------- |
| `MONITOR_ID` | int | ✓ | The ID of the monitor. |
| `DEPLOYMENT_ID` | int | ✓ | The public ID of the deployment. |
| `TARGET_ID` | int | ✓ | The public ID of the deployment target. |
**Options**
| Option | Type | Required | Description |
| --------------- | --------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--time-window` | parse\_time\_window | | Fetch the last DURATION instead of an explicit `--start`/`--end`, e.g. '1h', '30m', '7d'. Mutually exclusive with `--start`/`--end`. Defaults to 1 hour when none of `--time-window`/`--start`/`--end` is given. |
| `--end` | parse\_iso\_timestamp | | End of the window (inclusive), e.g. 2024-01-01T00:00:00. Defaults to now. Mutually exclusive with `--time-window`. |
| `--start` | parse\_iso\_timestamp | | Start of the window (inclusive), e.g. 2024-01-01T00:00:00. Defaults to 1 hour before `--end`. Mutually exclusive with `--time-window`. |
***
## LIST
List Monitors.
```bash theme={null}
oumi-cli monitors list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--limit` | int range | | Return at most N items (default 100). The CLI loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return everything, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
***
## SCORES
Fetch score detail for the logs a monitor scored.
```bash theme={null}
oumi-cli monitors scores [OPTIONS] {monitor_id} {deployment_id} {target_id}
```
**Arguments**
| Argument | Type | Required | Description |
| --------------- | ---- | -------- | --------------------------------------- |
| `MONITOR_ID` | int | ✓ | The ID of the monitor. |
| `DEPLOYMENT_ID` | int | ✓ | The public ID of the deployment. |
| `TARGET_ID` | int | ✓ | The public ID of the deployment target. |
**Options**
| Option | Type | Required | Description |
| ---------------- | --------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--time-window` | parse\_time\_window | | Fetch score detail for every log scored in the last DURATION, e.g. '1h', '30m', '7d'. Mutually exclusive with `--start`/`--end` and `--score-log-id`. Defaults to 5 minutes when no id and no window is given. |
| `--end` | parse\_iso\_timestamp | | End of the window (inclusive), e.g. 2024-01-01T00:00:00. Defaults to now. Mutually exclusive with `--time-window` and `--score-log-id`. |
| `--start` | parse\_iso\_timestamp | | Start of the window (inclusive), e.g. 2024-01-01T00:00:00. Defaults to 5 minutes before `--end`. Mutually exclusive with `--time-window` and `--score-log-id`. |
| `--score-log-id` | str | | Score log id to fetch detail for; repeat to fetch several -- ids come from the 'Score Log' column of `monitors insights`. Fetched regardless of age; mutually exclusive with `--time-window`/`--start`/`--end`. Default: `[]`. |
***
## UPDATE
Update a monitor's judges, sample rate, or name (at least one is required).
```bash theme={null}
oumi-cli monitors update [OPTIONS] {monitor_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ---------------------- |
| `MONITOR_ID` | int | ✓ | The ID of the monitor. |
**Options**
| Option | Type | Required | Description |
| ---------------- | ----- | -------- | --------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project ID. Env: `OUMI_PROJECT_ID`. |
| `--judge` | int | | Replace the monitor's judges with these evaluators, at their latest versions; repeat `--judge` to give several. |
| `--sample-rate` | float | | Fraction of traffic to score, between 0 and 1. |
| `--display-name` | str | | Human-readable name for the monitor. |
***
# OPERATIONS
Source: https://docs.oumi.ai/cli/commands/operations
Track, wait on, and cancel long-running operations
| Command | Description |
| --------------- | -------------------------------------------------------------- |
| [`get`](#get) | Show full details and current status for a single operation. |
| [`list`](#list) | List operations in a project, optionally filtered and sorted. |
| [`stop`](#stop) | Request cancellation of a running operation. |
| [`wait`](#wait) | Block until an operation finishes, then print its final state. |
***
## GET
Show full details and current status for a single operation.
```bash theme={null}
oumi-cli operations get [OPTIONS] {operation_id}
```
**Arguments**
| Argument | Type | Required | Description |
| -------------- | ---- | -------- | ------------------------------------------------------- |
| `OPERATION_ID` | int | ✓ | ID of the operation to inspect (see `operations list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | --------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the operation; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## LIST
List operations in a project, optionally filtered and sorted.
```bash theme={null}
oumi-cli operations list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------------ | --------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to list operations for; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N operations (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every operation, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
| `--resource` | str | | Only show operations tied to this resource. Format: projects/\{project\_id}/\{resource\_type}/\{resource\_id} (optionally with /versions/\{version\_id}). Resource types: datasets, models, evaluators, recipes. |
| `--search` | str | | Only show operations whose display name matches this search text. |
| `--sort-by` | choice | | Field to sort by. One of: `created_at`, `status`, `type`, `updated_at`. |
| `--sort-direction` | choice | | Sort direction (asc or desc). |
| `--status` | choice | | Only show operations in this status. One of: `cancelled`, `cancelling`, `completed`, `failed`, `pending`, `running`, `unknown`. |
| `--type` | choice | | Only show operations of this type. One of: `analysis`, `analysis_query`, `dataset_derive`, `dataset_download`, `dataset_merge`, `dataset_upload`, `datasource_upload`, `data_smart_upload`, `deployment_create`, `deployment_delete`, `distillation`, `evaluation`, `failure_mode_generation`, `import_external_logs`, `improvement_run`, `judge_config_generation`, `log_export`, `model_export`, `model_import`, `model_training`, `synthesis`, `synthesis_config_generation`, `synthesis_config_improvement`, `unknown`. |
***
## STOP
Request cancellation of a running operation.
```bash theme={null}
oumi-cli operations stop [OPTIONS] {operation_id}
```
**Arguments**
| Argument | Type | Required | Description |
| -------------- | ---- | -------- | ------------------------------------------------------ |
| `OPERATION_ID` | int | ✓ | ID of the operation to cancel (see `operations list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | --------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the operation; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## WAIT
Block until an operation finishes, then print its final state.
```bash theme={null}
oumi-cli operations wait [OPTIONS] {operation_id}
```
**Arguments**
| Argument | Type | Required | Description |
| -------------- | ---- | -------- | -------------------------------------------------------- |
| `OPERATION_ID` | int | ✓ | ID of the operation to wait for (see `operations list`). |
**Options**
| Option | Type | Required | Description |
| ------------ | ---- | -------- | --------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the operation; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--timeout` | int | | Give up after this many seconds (default: 600). |
| `--interval` | int | | Seconds to sleep between status checks (default: 5). |
***
# ORGANIZATIONS
Source: https://docs.oumi.ai/cli/commands/organizations
List organizations and set the active one
| Command | Description |
| --------------- | ---------------------------------------------------- |
| [`list`](#list) | List all organizations your account has access to. |
| [`set`](#set) | Set the active organization for the current profile. |
***
## LIST
List all organizations your account has access to.
```bash theme={null}
oumi-cli organizations list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `--limit` | int range | | Return at most N organizations (default 100). The server returns the full list in one response, so this is a client-side cap. |
| `--all` | flag | | Return every organization (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap on rows fetched, in any mode (default 10000). |
***
## SET
Set the active organization for the current profile.
```bash theme={null}
oumi-cli organizations set [OPTIONS] {org_id}
```
**Arguments**
| Argument | Type | Required | Description |
| -------- | ---- | -------- | --------------------------------------------------- |
| `ORG_ID` | str | ✓ | Organization ID to activate in the current profile. |
***
# PROJECTS
Source: https://docs.oumi.ai/cli/commands/projects
Create, inspect, and manage Oumi projects
| Command | Description |
| ------------------- | ---------------------------------------------------------------------------- |
| [`create`](#create) | Create a new project within an organization. |
| [`delete`](#delete) | Delete a project by ID. |
| [`get`](#get) | Show details for a single project. |
| [`list`](#list) | List the projects in an organization. |
| [`set`](#set) | Set the active project for the current profile. |
| [`update`](#update) | Update one or more fields of an existing project (at least one is required). |
***
## CREATE
Create a new project within an organization.
```bash theme={null}
oumi-cli projects create [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ---------------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `--display-name` | str | ✓ | Human-readable name for the project. |
| `--organization-id` | str | ✓ | ID of the organization that will own the project. |
| `--description` | str | | Optional free-text description. |
| `--prompt` | str | | Default system prompt for the project. |
| `--default-role` | choice | | Default role for new members. One of: `viewer`, `editor`, `admin`. |
| `--default-api-key-provider` | choice | | Default API-key provider for the project. One of: `anthropic`, `google_gemini`, `google_vertex`, `hugging_face`, `openai`, `wandb`. |
| `--user-ids` | str | | User ID to add as a member; repeat `--user-ids` to add several. |
***
## DELETE
Delete a project by ID.
```bash theme={null}
oumi-cli projects delete [OPTIONS] [PROJECT_ID]
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ----------------------------------------------------------------------------------- |
| `PROJECT_ID` | str | | ID of the project to delete; falls back to OUMI\_PROJECT\_ID or the active profile. |
**Options**
| Option | Type | Required | Description |
| --------- | ---- | -------- | ------------------------------------------------- |
| `--force` | flag | | Skip the confirmation prompt (alias for `--yes`). |
| `--yes` | flag | | Skip the confirmation prompt. |
***
## GET
Show details for a single project.
```bash theme={null}
oumi-cli projects get [OPTIONS] [PROJECT_ID]
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | --------------------------------------------------------------------------------- |
| `PROJECT_ID` | str | | ID of the project to show; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## LIST
List the projects in an organization.
```bash theme={null}
oumi-cli projects list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| `--org` | str | | Organization to list projects for; falls back to OUMI\_ORG\_ID or the active profile. |
| `--limit` | int range | | Return at most N projects (default 100). The server returns the full list in one response, so this is a client-side cap. |
| `--all` | flag | | Return every project (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap on rows fetched, in any mode (default 10000). |
***
## SET
Set the active project for the current profile.
```bash theme={null}
oumi-cli projects set [OPTIONS] {project_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ---------------------------------------------- |
| `PROJECT_ID` | str | ✓ | Project ID to activate in the current profile. |
***
## UPDATE
Update one or more fields of an existing project (at least one is required).
```bash theme={null}
oumi-cli projects update [OPTIONS] [PROJECT_ID]
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ----------------------------------------------------------------------------------- |
| `PROJECT_ID` | str | | ID of the project to update; falls back to OUMI\_PROJECT\_ID or the active profile. |
**Options**
| Option | Type | Required | Description |
| ---------------- | ------ | -------- | ------------------------------------------------------------------ |
| `--display-name` | str | | New human-readable name for the project. |
| `--description` | str | | New free-text description. |
| `--prompt` | str | | New default system prompt for the project. |
| `--user-ids` | str | | User ID to add as a member; repeat `--user-ids` to add several. |
| `--default-role` | choice | | Default role for new members. One of: `viewer`, `editor`, `admin`. |
***
# RECIPES
Source: https://docs.oumi.ai/cli/commands/recipes
Create and manage the reusable configs you launch runs from
| Command | Description |
| ----------------------------------------- | -------------------------------------------------------------------- |
| [`create`](#create) | Create a recipe in a project from a JSON request body. |
| [`delete`](#delete) | Delete a recipe by ID. |
| [`download`](#download) | Download a recipe's config to a local JSON file for editing. |
| [`generate-template`](#generate-template) | Print a ready-to-edit recipe request body for the given recipe type. |
| [`get`](#get) | Show a single recipe by ID. |
| [`list`](#list) | List recipes in a project, optionally sorted. |
| [`update`](#update) | Rename a recipe, or replace its config from a JSON body. |
| [`versions get`](#versions-get) | Show a single version of a recipe, including its full config. |
| [`versions list`](#versions-list) | List version history for a recipe, newest first by default. |
***
## CREATE
Create a recipe in a project from a JSON request body.
```bash theme={null}
oumi-cli recipes create [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to create the recipe in; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--type` | choice | ✓ | Recipe kind to create, a guard that must agree with the body's recipeConfig.type discriminator. Templates from 'generate-template `--type` \' already set this correctly; pass the same `--type` you generated with. One of: `training`, `evaluate`, `synthesis`. |
| `--input-json` | str | | Request body as an inline JSON string. Pass exactly one of `--input-json` or `--input-json-file`. |
| `--input-json-file` | path | | Path to a file holding the request body as JSON. Pass exactly one of `--input-json` or `--input-json-file`. |
***
## DELETE
Delete a recipe by ID.
```bash theme={null}
oumi-cli recipes delete [OPTIONS] {recipe_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ----------- | ---- | -------- | -------------------------------------- |
| `RECIPE_ID` | int | ✓ | ID of the recipe (see `recipes list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--force` | flag | | Delete without confirmation prompt. |
| `--yes` | flag | | Delete without confirmation prompt (alias for `--force`). |
***
## DOWNLOAD
Download a recipe's config to a local JSON file for editing.
```bash theme={null}
oumi-cli recipes download [OPTIONS] {recipe_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ----------- | ---- | -------- | -------------------------------------------------- |
| `RECIPE_ID` | int | ✓ | ID of the recipe to download (see `recipes list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--out` | path | | Destination file or directory. Defaults to ./recipe-\.json in the current directory. |
***
## GENERATE-TEMPLATE
Print a ready-to-edit recipe request body for the given recipe type.
```bash theme={null}
oumi-cli recipes generate-template [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| -------- | ------ | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `--type` | choice | ✓ | Recipe kind to generate a template for; sets recipeConfig.type in the output. One of: `training`, `evaluate`, `synthesis`. |
***
## GET
Show a single recipe by ID.
```bash theme={null}
oumi-cli recipes get [OPTIONS] {recipe_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ----------- | ---- | -------- | -------------------------------------- |
| `RECIPE_ID` | int | ✓ | ID of the recipe (see `recipes list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ----------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## LIST
List recipes in a project, optionally sorted.
```bash theme={null}
oumi-cli recipes list [OPTIONS]
```
**Options**
| Option | Type | Required | Description |
| ------------------ | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N recipes (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every recipe, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
| `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`. |
| `--sort-direction` | choice | | Sort order (asc is oldest-first). |
***
## UPDATE
Rename a recipe, or replace its config from a JSON body.
```bash theme={null}
oumi-cli recipes update [OPTIONS] {recipe_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ----------- | ---- | -------- | -------------------------------------- |
| `RECIPE_ID` | int | ✓ | ID of the recipe (see `recipes list`). |
**Options**
| Option | Type | Required | Description |
| ------------------- | ---- | -------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `--project` | str | | Project to scope the operation to; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--display-name` | str | | New display name for the recipe. Renames only. Does not touch recipeConfig. Mutually exclusive with `--input-json`\*. |
| `--input-json` | str | | Full replacement request body as an inline JSON string; creates a new recipe version. Exclusive of `--display-name`. |
| `--input-json-file` | path | | Path to a file holding the full replacement request body as JSON; creates a new recipe version. Exclusive of `--display-name`. |
***
## VERSIONS GET
Show a single version of a recipe, including its full config.
```bash theme={null}
oumi-cli recipes versions get [OPTIONS] {recipe_id} {version_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ------------ | ---- | -------- | ----------------------------------------------------------- |
| `RECIPE_ID` | int | ✓ | ID of the recipe (see `recipes list`). |
| `VERSION_ID` | int | ✓ | ID of the version to inspect (see `recipes versions list`). |
**Options**
| Option | Type | Required | Description |
| ----------- | ---- | -------- | ------------------------------------------------------------------------------------ |
| `--project` | str | | Project that owns the recipe; falls back to OUMI\_PROJECT\_ID or the active profile. |
***
## VERSIONS LIST
List version history for a recipe, newest first by default.
```bash theme={null}
oumi-cli recipes versions list [OPTIONS] {recipe_id}
```
**Arguments**
| Argument | Type | Required | Description |
| ----------- | ---- | -------- | -------------------------------------- |
| `RECIPE_ID` | int | ✓ | ID of the recipe (see `recipes list`). |
**Options**
| Option | Type | Required | Description |
| ------------------ | --------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `--project` | str | | Project that owns the recipe; falls back to OUMI\_PROJECT\_ID or the active profile. |
| `--limit` | int range | | Return at most N versions (default 100). The CLI fetches and loops over pages internally; you never pass a page token. |
| `--all` | flag | | Return every version, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. |
| `--max-items` | int range | | Hard safety cap for `--all` (default 10000). |
| `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`, `version_id`. |
| `--sort-direction` | choice | | Sort order (asc is oldest-first). |
***
# INSTALLATION
Source: https://docs.oumi.ai/cli/installation
Install the Oumi CLI on macOS, Linux or Windows
The Oumi CLI can be installed several ways depending on your platform. On Windows, use WSL and follow the Linux steps.
```bash theme={null}
brew tap oumi-ai/tap
brew trust --formula oumi-ai/tap/oumi-cli
brew install oumi-cli
```
```bash theme={null}
curl -fsSL https://oumi-public-downloads.s3.us-west-2.amazonaws.com/cli/stable/oumi-cli-macos-arm64.tar.gz | sudo tar -xz -C /usr/local
sudo ln -sf /usr/local/oumi-cli/oumi-cli /usr/local/bin/oumi-cli
```
```bash theme={null}
curl -fsSL https://oumi-public-downloads.s3.us-west-2.amazonaws.com/cli/stable/oumi-cli-linux-x86_64.tar.gz | sudo tar -xz -C /usr/local
sudo ln -sf /usr/local/oumi-cli/oumi-cli /usr/local/bin/oumi-cli
```
```bash theme={null}
curl -fsSL https://oumi-public-downloads.s3.us-west-2.amazonaws.com/cli/stable/oumi-cli-linux-arm64.tar.gz | sudo tar -xz -C /usr/local
sudo ln -sf /usr/local/oumi-cli/oumi-cli /usr/local/bin/oumi-cli
```
Confirm the install by printing the version:
```bash theme={null}
oumi-cli version
```
# QUICKSTART
Source: https://docs.oumi.ai/cli/quickstart
Authenticate, select your workspace, and view commands
This assumes you have already [installed the CLI](/cli/installation).
## AUTHENTICATE
Create an API key in your user settings} />
```bash theme={null}
oumi-cli auth login --api-key ""
```
## CHOOSE WHERE TO WORK
```bash theme={null}
oumi-cli organizations list
oumi-cli organizations set
oumi-cli projects list
oumi-cli projects set
```
## EXPLORE THE COMMANDS
`--help` works at every level, and each group lists its subcommands:
```bash theme={null}
oumi-cli --help
oumi-cli datasets --help
oumi-cli datasets upload --help
```
To see the whole surface at once, or to search it by keyword:
```bash theme={null}
oumi-cli --command-list
oumi-cli search train
```
## NEXT STEPS
Go from a local JSONL file to a trained model.
Understand the configs every run launches from.
# DEPLOY A MODEL
Source: https://docs.oumi.ai/cli/workflows/deploy-a-model
Serve a model behind an Oumi inference endpoint
Set `MODEL_ID` to the model you want to serve, then paste the block:
```bash theme={null}
MODEL_ID=1
cat > deployment.json <
```bash theme={null}
oumi-cli deployments create --input-json-file deployment.json
```
Returns the deployment ID.
```bash theme={null}
oumi-cli deployments endpoint
```
Prints the inference URL, the model path, and a runnable curl snippet.
Score the model you deployed.
Every flag on every deployment command, including inference log export.
# EVALUATE A MODEL
Source: https://docs.oumi.ai/cli/workflows/evaluate-a-model
Score a model against your own criteria and find where it fails
This example uses the [banking77 validation dataset](https://huggingface.co/datasets/oumi-ai/banking77-oumi-quickstart/resolve/main/banking77_val_basic.jsonl?download=true).
```bash theme={null}
oumi-cli datasets upload ./banking77_val_basic.jsonl --display-name "banking77-val"
```
Note the dataset ID it returns.
```bash theme={null}
cat > evaluator.json <
Set the three IDs, then paste the block. `MODEL_ID` is the model you want to score, `DATASET_ID` is the validation set from the upload step, and `EVALUATOR_ID` is the one you just created.
```bash theme={null}
MODEL_ID=1
DATASET_ID=1
EVALUATOR_ID=1
cat > eval-recipe.json <
```bash theme={null}
oumi-cli recipes create --type evaluate --input-json-file eval-recipe.json
```
```bash theme={null}
oumi-cli evaluations run --name "banking77-eval" --recipe
```
## READING THE RESULTS
```bash theme={null}
oumi-cli evaluations get
oumi-cli evaluations results
oumi-cli evaluations download --out results.jsonl
```
## VIEWING FAILURE MODES
```bash theme={null}
oumi-cli evaluations failure-modes
```
Retrain against the gaps you found.
Every flag on every evaluation command.
# TRAIN A MODEL
Source: https://docs.oumi.ai/cli/workflows/train-a-model
Upload a dataset and train a model
Requires an authenticated session with an active project. See the [quickstart guide](/cli/quickstart) for more info.
This example uses the [banking77 example dataset](https://huggingface.co/datasets/oumi-ai/banking77-oumi-quickstart/resolve/main/banking77_train_basic.jsonl?download=true).
```bash theme={null}
oumi-cli datasets upload ./banking77_train_basic.jsonl --display-name "banking77-train"
```
Data must be in JSONL format. Add `--dry-run` to verify compatibility and view metadata without uploading.
Note the dataset ID it returns.
Set `DATASET_ID` to the ID from the previous step, then paste the whole block (lora finetunes Qwen3-8b on the banking 77 dataset):
```bash theme={null}
DATASET_ID=1
cat > recipe.json <
```bash theme={null}
oumi-cli recipes create --type training --input-json-file recipe.json
```
Returns the recipe ID you pass to `train`.
Training is a long running operation which can take 1hr+ depending on the datatset and model. Billing charges apply. For more info see our [pricing page](https://oumi.ai/pricing).
```bash theme={null}
oumi-cli models train --name banking77-intent-classifier --recipe
```
Blocks until training finishes, then prints the model ID, model version ID, and operation ID.
The model weights can be exported with the following command. (Note that model exports require a pro plan and are limited to 1 per month):
```bash theme={null}
oumi-cli models download --out ./weights
```
## NEXT STEPS
Score it and find where it fails.
Serve it behind an inference endpoint.
# OUMI DATASETS
Source: https://docs.oumi.ai/guides/datasets
Upload, generate, and validate datasets in Oumi
The Oumi Agent makes it easy to generate high-quality datasets at any stage of the machine learning workflow. Using natural language prompts, you can create training data from scratch, [synthesize datasets from failure modes](/guides/evaluations/failure-modes), or analyze and refine existing data, without writing pipeline code.
What traditionally requires weeks of manual collection, cleaning, and formatting can be completed in hours. The Oumi Agent automates the most time-consuming parts of dataset preparation such as schema validation, format conversion, and iterative refinement, so your team spends less time on data plumbing and more time on model quality. Because datasets are generated on-demand and scoped to your task, you also avoid the cost of sourcing or licensing large generic datasets that may not fit your use case.
## STRUCTURE & CONTENTS
An Oumi dataset is a structured collection of prompts and responses used to either train a model or evaluate its performance. Depending on your workflow, a dataset may include:
* **Prompt–response pairs** for supervised fine-tuning
* **Prompts only**, where model outputs are generated and evaluated separately
* **Multi-turn conversations** for dialogue-based training or benchmarking
## UPLOADING DATASETS
You can upload datasets directly into Oumi in a variety of common formats, including [JSON](https://huggingface.co/datasets/oumi-ai/examples/blob/main/example_contents.json), [JSONL](https://huggingface.co/datasets/oumi-ai/examples/blob/main/example_contents.jsonl), [CSV](https://huggingface.co/datasets/oumi-ai/examples/blob/main/example_contents.csv), and [Parquet](https://huggingface.co/datasets/oumi-ai/examples/blob/main/example_contents.parquet).
All Oumi datasets follow a standardized internal Conversation format that defines how messages, roles, and metadata are structured. During upload, Oumi automatically validates and converts your data into this format, ensuring it works seamlessly with training, evaluation, data synthesis, and analysis tools across the platform as well as modern machine learning pipelines.
### CONTEXT FILES
Oumi also supports uploading context files to ground your models in proprietary or domain-specific data. This allows you to incorporate internal documents, knowledge bases, or other private content into your workflows.
To learn more, please see [Uploading context files](/guides/datasets/add-files).
## EXAMPLE USAGE
Here's an example of a properly-formed dataset for Oumi in JSONL format:
```jsonl theme={null}
{"messages": [{"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "The capital of France is Paris."}], "metadata": {"source": "geography"}}
{"messages": [{"role": "user", "content": "How do I make pasta?"}, {"role": "assistant", "content": "Boil water and add pasta for 8-10 minutes."}], "metadata": {"source": "cooking"}}
{"messages": [{"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "2+2 equals 4."}], "metadata": {"source": "math"}}
```
In addition to the standard `messages` field, you can also specify a `metadata` field that is a dictionary of metadata for your data row.
* [More dataset examples](https://huggingface.co/oumi-ai/datasets)
## WHAT'S NEXT
Upload and import datasets into the Oumi platform.
Upload and import files to contextualize and ground your data.
Explore, inspect, and validate your datasets.
Generate datasets from reusable synthesis configurations.
# ADDING CONTEXT FILES
Source: https://docs.oumi.ai/guides/datasets/add-files
Uploading context files to Oumi
Oumi allows you to bring in unstructured source material directly to enrich your models with proprietary knowledge and domain-specific context. By uploading files in the form of internal documents, knowledge repositories, or other private content, you can incorporate domain-specific context into your data workflows and model development process.
## UPLOAD CONTEXT FILES
To upload context files from your local computer:
1. Click on `Files` from the left-hand navigation.
2. Click the `Upload File` button.
3. Give your file a `Display Name` and drag/drop or select your file to upload.
Your new file will appear on the **Files** page once the upload finishes.
# ANALYZE DATASETS
Source: https://docs.oumi.ai/guides/datasets/analyze
Measure, track, and improve the quality of your datasets in Oumi
Oumi automatically scans your uploaded datasets for quality issues, highlights problematic entries, and lets you fix them directly in the platform.
## HOW IT WORKS
When you upload a dataset to the platform, Oumi automatically runs a series of quality tests to identify potential issues. Any failed tests pinpoint conversations that may need to be removed to enhance overall quality.
You can then directly remove problematic rows using the Oumi UI, or export individual quality tests for further analysis.
Oumi currently provides the following quality tests:
* **Total tokens exceed `8000`** - Flags conversations where the total token count exceeds `8,000`, helping identify overly long entries that may affect processing or model performance.
* **Non-alternating user/assistant turns** - Flags conversations where messages do not strictly alternate between user and assistant, ensuring the dataset follows a consistent turn-taking structure.
* **Empty turns detected** - Flags conversations containing empty or whitespace-only messages, which can introduce noise or reduce dataset quality.
## ACCESSING DATASET QUALITY TESTS
After successfully uploading a dataset, you can access its quality tests from the **Datasets** page:
1. Click the `Quality Tests` link for your dataset.
2. Under the `Quality Tests` tab, you can expand each test to view its details.
3. Click `Export Quality Tests` to export the quality tests as a JSONL or Parquet file.
## DELETING FAILED ROWS
To immediately remove rows flagged by Oumi's quality tests during dataset upload:
1. Click the `Quality Tests` link for the dataset with the failing tests.
2. Under the `Quality Tests` tab, you can expand each test and drill down further to analyze row-level errors.
3. Click `Delete Failed Rows` to delete the problematic rows. Oumi will then re-analyze your new dataset version.
Rather than modifying the original dataset, Oumi creates a new version and applies changes there, ensuring all dataset states are version-controlled and fully recoverable.
## RESTORING PREVIOUS DATASET VERSIONS
To restore a previous dataset:
1. On the **Datasets** page, click the name of your dataset.
2. Click the three-dot menu to the right of the dataset version you want to restore.
3. Select `Restore Version`. Oumi will restore the selected dataset version as a new version and automatically rerun the quality checks.
# CREATE DATASETS
Source: https://docs.oumi.ai/guides/datasets/create
Bringing your datasets into the Oumi Platform
Oumi allows you the flexibility to work with your existing datasets or synthesize new ones from scratch.
## UPLOADING YOUR DATASET
You can upload your data from your project's **Overview** page:
1. Click the `Create` button and select `Dataset` from the menu to launch the Builder.
2. Select `Upload a File`.
3. Provide a `Dataset Name` and select the file to upload.
4. Click `Create Dataset`.
You can check your new dataset's upload status from the main **Datasets** page.
## SYNTHESIZING DATASETS
You can generate datasets using Oumi’s data synthesis features, which are integrated throughout the workflow. For example, you can [create datasets from failure modes](/guides/evaluations/failure-modes) or define generation logic (e.g., rules, templates, and constraints) to produce synthetic data at scale instead of manually authoring examples.
This approach enables consistent, reproducible datasets while allowing controlled variation in complexity, style, and structure.
Please see [Data synthesis](/guides/synthesis) to learn more.
# DATA EXPLORER
Source: https://docs.oumi.ai/guides/datasets/exploring
Exploring & validating your datasets
After uploading or synthesizing a dataset, you should validate its structure and quality before using it in downstream workflows. Oumi’s Data Explorer makes this easy by letting you review and refine your data within the same visual interface.
With Data Explorer, you can:
* Inspect input–output pairs
* Verify dataset integrity
* Review schemas and fields
* Export datasets for local editing
* Re-upload updated versions
Data Explorer provides a convenient way to spot-check your data and catch critical issues early.
## ACCESSING THE DATA EXPLORER
Go to the **Datasets** page and click the name of the dataset to explore. On the dataset’s overview page, click the `Data Explorer` button to launch the Data Explorer window.
### WHAT TO VALIDATE
When reviewing a dataset in Data Explorer, be sure to check for the following:
* The row count matches your expectations
* All required fields and labels are present
* The data quality is sufficient for training or evaluation
By surfacing data issues at this stage, you can ensure early on that your dataset is ready for use and minimize the risk of downstream errors.
# IMPORT OBSERVABILITY LOGS
Source: https://docs.oumi.ai/guides/datasets/import-observability-logs
Turn Langfuse or LangSmith traces into a Conversation dataset you can train, evaluate, and synthesize with
Turn the Langfuse or LangSmith traces you already capture into a Conversation dataset, no ad hoc conversion code required. Oumi reconstructs each conversation into OpenAI wire format (system instructions, multi-turn structure, tool definitions and `tool_calls` preserved), and creates a Conversation dataset suitable for fine-tuning, evaluation, or synthesis workflows.
Oumi reconstructs conversations from LLM calls captured in OpenAI chat-completions shape: on Langfuse, a `GENERATION` whose JSON-encoded `input`/`output` are the request and response; on LangSmith, an `llm` run whose `inputs.messages` are OpenAI- or LangChain-style messages. Traces logged in other shapes (other providers' formats, custom structures, or non-chat steps) may import with missing content or be skipped, so confirm your trace shape with a small test import first.
Langfuse trace exports can be created directly from the tracing UI, whereas LangSmith trace exports must be created via the LangSmith SDK (more below).
| Platform | How you export | Format |
| ------------- | ------------------------------------------------------------------------ | ------------------------ |
| **Langfuse** | UI: **Tracing** tab > **Export** > **JSONL** | One observation per line |
| **LangSmith** | Small Python script using the LangSmith SDK (no UI export of raw traces) | One run per line |
Once you have your exported traces *in `.jsonl` format*, there are two ways to bring them into the platform:
* **Drag and drop** the file onto the **Datasets** page. Oumi detects the platform automatically.
* **The `Import Observability Logs` tile** under Create Dataset (here you can enter a dataset name, select the source platform, and disable deduplication of rows with matching prefixes).
## BEFORE YOU START
You will need:
* A Langfuse or LangSmith project with at least one traced run.
* For LangSmith, an API key with read access to that project. (Langfuse exports from the UI, no key needed.)
* An Oumi project where you can create datasets.
Start with a small test import (say, 20 traces) to confirm your traces parse cleanly before importing your full history.
## STEP 1: EXPORT FROM YOUR OBSERVABILITY PLATFORM
Pick the section for your platform.
### LANGFUSE (UI EXPORT)
In Langfuse, open the **Tracing** tab for your project.
Filter down to the traces you want (date range, session, tag, environment, user, trace attributes), then open **Export** and choose **As JSONL**. See Langfuse's [Export from UI guide](https://langfuse.com/docs/api-and-data-platform/features/export-from-ui).
![Langfuse .jsonl trace export]()
We currently accept only .jsonl trace exports, which contain all fields necessary to properly reconstruct tool definitions and invocations. Langfuse's "JSON" tree-view export is a different shape and is rejected at upload. Same for Datasets exported via the LangSmith UI.
Each line is one **observation**: a single step LangFuse recorded, such as a span, a generation, or a tool call. Observations from the same trace share a `traceId`, and LangFuse copies the trace-level fields onto every line so each row is self-contained. The example below is one trace made of a span and a generation whose response is a tool call:
```jsonl theme={null}
{"id":"obs_001","traceId":"trace_abc","traceName":"weather-agent","type":"SPAN","parentObservationId":null,"startTime":"2026-05-19T10:00:00.000Z","sessionId":null,"input":null,"output":null}
{"id":"obs_002","traceId":"trace_abc","traceName":"weather-agent","type":"GENERATION","parentObservationId":"obs_001","startTime":"2026-05-19T10:00:01.500Z","sessionId":null,"input":"{\"model\":\"gpt-4o\",\"messages\":[{\"role\":\"system\",\"content\":\"You are a weather assistant.\"},{\"role\":\"user\",\"content\":\"Will it rain in Paris tomorrow?\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_forecast\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}}}}}]}","output":"{\"role\":\"assistant\",\"content\":null,\"tool_calls\":[{\"id\":\"call_1\",\"type\":\"function\",\"function\":{\"name\":\"get_forecast\",\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\"}}]}"}
```
* `traceId` groups rows into one conversation.
* `type` can be `SPAN`, `GENERATION`, `TOOL`, etc. Oumi builds the conversation from `GENERATION` rows, with others retained as metadata.
* `input` and `output` are JSON-encoded strings, not nested objects (Langfuse's storage shape).
### LANGSMITH (API EXPORT)
The LangSmith UI has no one-click .jsonl export for raw traces, so you pull them with a small script using the LangSmith SDK. This produces a **raw trace export** (one `Run` per line, with `trace_id`, `parent_run_id`, `dotted_order`, and `run_type`), and the result imports to Oumi exactly like a Langfuse .jsonl.
LangSmith's UI **Add to Dataset > Download JSONL** produces a different shape (one example per row, no run linkage or retention of tool) for which import is not supported.
**1. Get an API key.** In LangSmith, open **Settings > API Keys** and create a key with read access to your project.
![LangSmith API key creation]()
**2. Install the SDK:**
```bash theme={null}
pip install 'langsmith>=0.1'
```
**3. Export the traces.** Set your LangSmith API key in the environment (`export LANGSMITH_API_KEY=lsv2_pt_...`), then run a script like the one below, adding [run filters](https://docs.smith.langchain.com/observability/how_to_guides/export_traces) to narrow what you pull:
```python theme={null}
import json
from langsmith import Client
PROJECT_NAME = "oumi-langsmith-project"
OUTPUT_PATH = "oumi-langsmith-export.jsonl"
client = Client() # reads LANGSMITH_API_KEY from env
# pulls every run for the project, optionally apply run filters
runs = client.list_runs(project_name=PROJECT_NAME)
with open(OUTPUT_PATH, "w") as f:
for run in runs:
# model_dump() on pydantic v2 SDKs, dict() on older v1 ones
data = run.model_dump() if hasattr(run, "model_dump") else run.dict()
f.write(json.dumps(data, default=str) + "\n")
```
Each line of the .jsonl will be one run (`chain`, `llm`, or `tool`) carrying `trace_id`, `parent_run_id`, `dotted_order`, and `run_type`. Oumi groups runs by `trace_id`, picks the `llm` run with the most messages, and rebuilds the conversation from that run's `inputs.messages` (tool results already appear there as `tool` turns) plus its `outputs`. Example:
```jsonl theme={null}
{"id":"run_root","trace_id":"trace_abc","parent_run_id":null,"dotted_order":"20260519T100000000000Zrun_root","run_type":"chain","name":"weather-agent","start_time":"2026-05-19 10:00:00.000000","inputs":{},"outputs":{}}
{"id":"run_llm","trace_id":"trace_abc","parent_run_id":"run_root","dotted_order":"20260519T100000000000Zrun_root.20260519T100001500000Zrun_llm","run_type":"llm","name":"final-answer","start_time":"2026-05-19 10:00:01.500000","inputs":{"messages":[{"role":"system","content":"You are a weather assistant."},{"role":"user","content":"Will it rain in Paris tomorrow?"}],"tools":[{"type":"function","function":{"name":"get_forecast","parameters":{"type":"object","properties":{"city":{"type":"string"}}}}}]},"outputs":{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_forecast","arguments":"{\"city\":\"Paris\"}"}}]}}
```
* `trace_id` groups runs into one conversation; `parent_run_id` and `dotted_order` give the run tree and its order.
* Unlike the Langfuse example above, `inputs` and `outputs` here are real nested objects, not stringified json.
* Session threading reads `extra.metadata.session_id`, not the top-level `session_id` (that's the tracing-project ID).
Pulling a whole project can be slow. Bound it with the `start_time` / `end_time` parameters on `list_runs`, and run in chunks if needed. Filtering examples can be found at [https://docs.smith.langchain.com/observability/how\_to\_guides/export\_traces](https://docs.smith.langchain.com/observability/how_to_guides/export_traces)
## STEP 2: IMPORT INTO OUMI
Bring your traces into the platform from the **Datasets** page.
### OPTION A: DRAG AND DROP (QUICKEST)
Open the **Datasets** page and drop your `.jsonl` onto the upload zone above the table.
Oumi recognizes Langfuse and LangSmith trace exports automatically, so no need to specify the source platform. The import runs in the background, and the new dataset appears in the list when parsing completes.
Deduplication is always on for this path: when one conversation's turns are a prefix of a longer one, only the longest is kept. Use Option B if you want to retain both.
### OPTION B: THE IMPORT OBSERVABILITY LOGS TILE (FULL CONTROL)
Use this to name the dataset (optionally), declare the platform, or turn dedupe off.
Open the **Datasets** page, click `Create Dataset`, and select the `Import Observability Logs` tile in the Builder.
Fill out the four fields:
| Field | What to enter |
| --------------------------------- | --------------------------------------------------------------------- |
| `Dataset Name` | A human-readable name (up to 128 characters). |
| `Source Platform` | `Langfuse` or `LangSmith`, matching your export. |
| `File` | The `.jsonl` from Step 1. Drag-drop or click to select. |
| `Deduplicate prefix-overlap rows` | Leave on (default) unless you have a reason not to. See next section. |
Then click `Import Logs`. Oumi parses the export in the background and returns you to the **Datasets** page; a few thousand traces usually finish in under a minute.
**Deduplicating prefix-overlap rows** (on by default) keeps only the most complete version of each conversation. Agents often log incrementally (a trace after turn 1, another containing turns 1 and 2, and so on), so one conversation can appear as several traces where each is a prefix of the next. Deduplication detects when one trace is a prefix of another, and keeps only the longest. Leave it on unless you have a strong reason to keep both partial and complete examples from a single session.
## WHAT YOU GET
A Conversation dataset where each row is one accepted trace (after dedupe):
* Messages in OpenAI wire format (`system`, `user`, `assistant`, `tool`), in trace order.
* `tool_calls` and `tool_call_id` preserved on the relevant turns (see the [OpenAI tool-calling spec](https://platform.openai.com/docs/guides/function-calling)).
* The originating `tools` array, when present.
* Multi-turn structure intact, never flattened to a single pair.
Each conversation also carries metadata: `source_platform`, `source_trace_id`, `source_trace_name`, `source_session_id` (if present), `source_sequence_key` (the root trace's start time, for in-session ordering), and `source_raw` (the full original trace as a JSON string, for fields Oumi didn't promote).
Nothing from the original trace is dropped: anything Oumi doesn't promote to a `source_*` field, such as per-span timings, costs, model names, prompt-template metadata, and platform user IDs, is preserved verbatim inside `source_raw`.
## USE THE DATASET
The new dataset behaves like any other Conversation dataset:
* **Fine-tune** on it. See [Training](/guides/training).
* **Evaluate** against it. See [Evaluations](/guides/evaluations).
* **Synthesize** from it as a synthesis input. See [Data synthesis](/guides/synthesis).
## LIMITATIONS
* **JSONL only.** Other formats (Langfuse's JSON tree or CSV; LangSmith's "Add to Dataset" exports) aren't supported.
* **Langfuse and LangSmith only.** OpenTelemetry is on the roadmap.
* **Dedupe turns off only on the tile.** Drag-and-drop always dedupes.
* **One new dataset per import.** No appending; re-import to merge.
* **No preview or in-app editing.** To fix a bad export, re-export and re-import.
* **Max 1000 messages per conversation.** Longer traces are dropped; the rest of the file still imports.
## TROUBLESHOOTING
| What you see | What it means | What to do |
| ----------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| "... import only supports `.jsonl` files" (or the file picker rejects your file) | The upload isn't JSONL (probably Langfuse's JSON tree export, or a CSV). | Re-export as JSONL: Langfuse's **Tracing** tab JSONL option, or the LangSmith script in Step 1. |
| "This file looks like a LangSmith dataset export ... Use the LangSmith API trace export, not the dataset export." | You used LangSmith's **Add to Dataset > Download JSONL**, which has no run linkage. | Re-export with the API trace script in Step 1. |
| "File too large" | Your export exceeds the project's dataset quota. | Split by date range, session, or tag and re-run. For LangSmith, pass `start_time` / `end_time` to `list_runs`. |
| "No convertible conversations found in the uploaded file" | The export has no model generations (no `GENERATION` in Langfuse, no `llm` run in LangSmith), no trace had an assistant turn, or the LLM calls weren't logged in a supported OpenAI/LangChain chat shape. | Confirm you exported actual model calls (not just spans or tool events), and that those calls were captured in OpenAI chat-completions or LangChain message shape. |
| Fewer conversations than expected | Dedupe collapsed prefix overlaps; traces with no assistant turn, or over 1000 messages, are also dropped. | To keep intermediate turns, re-import through the tile with dedupe off. |
| Dropped file landed under **Files**, or imported as the wrong type | Drag-and-drop didn't recognize it as a trace export. | Confirm it's a raw trace export (Step 1), or use the tile, which declares the platform explicitly. |
| "Upload timed out" | The presigned upload URL expired mid-upload. | Retry, or split the file if it keeps timing out. |
| LangSmith script returns no runs | `LANGSMITH_API_KEY` is unset or scoped to the wrong workspace, or `PROJECT_NAME` is wrong. | Check the env var, the workspace in LangSmith Settings, and the exact project name. |
## FAQ
No. Importing is read-only; your traces stay where they are.
Everything in your .jsonl trace export will be retained as metadata in the converted conversation dataset. Be sure to strip, redact, or filter out anything sensitive before exporting from Langfuse/LangSmith.
No. It runs within a single import, so re-importing overlapping data produces a new dataset, which may overlap with existing datasets.
It samples rows for each platform's signature fields. A genuine trace export is unambiguous.
Not yet. Use the API trace export in Step 1.
No. Each import targets a single platform. Run two imports and combine the datasets downstream.
# NEXT STEPS
Source: https://docs.oumi.ai/guides/datasets/next-steps
What to do after creating training data
After generating targeted training data, the next step is to use it to run focused, evidence-driven fine-tuning. By grounding your training runs in the insights uncovered during evaluation and data synthesis, you ensure each iteration is purposeful and aligned with measurable performance goals.
To learn more, please see [model training (fine-tuning)](/guides/training).
# DEPLOYMENT OPTIONS
Source: https://docs.oumi.ai/guides/deployment
Serve a model on a managed inference endpoint, or export it to run on your own infrastructure
A deployment is an inference endpoint that lives in your project. Your application calls it with the OpenAI or Anthropic SDKs it already uses, and Oumi handles what sits behind it. There is no model packaging, no GPU provisioning, and no serving stack to maintain.
The Oumi Agent can set this up from chat. Tell it which of your models to deploy and it asks you to approve the cost before provisioning anything, then hands back a runnable snippet.
***
## DEPLOYMENT TYPES
A deployment serves either a model you built or one from a third-party provider. Both are created from the same dialog, called the same way, and logged the same way. They differ only in what runs behind the endpoint.
| | Custom Oumi model | External model |
| -------------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| **Serves** | A model you trained or imported | An OpenAI or Anthropic model |
| **Runs on** | Oumi-managed GPU capacity | The provider's infrastructure |
| **You supply** | A deployable model | Optional: your own provider API key |
| **Cost** | GPU time while replicas are warm | Oumi bills external-model usage by default. With your own provider API key, Oumi does not bill you for that usage. |
| **Scale-to-zero and replica controls** | Yes | Not applicable |
| **Best for** | Serving the model you built | Benchmarking against a hosted baseline in the same logs |
[Learn how to deploy a model →](/guides/deployment/deploy-a-model)
***
## THE DEPLOYMENT LOOP
Once an endpoint is live, the same cycle applies whichever kind you chose:
1. **Deploy the model.** Start from the **Models** or **Deployments** page. You can also deploy your own models automatically when training finishes.
2. **Call the endpoint.** Point the OpenAI or Anthropic SDK at your deployment.
3. **Review inference logs.** See what users are asking the model in production.
4. **Monitor health and quality.** [Health metrics](/guides/deployment/health-metrics) track traffic, errors, and latency, while [quality monitoring](/guides/deployment/monitoring) scores a sample of production traffic with LLM judges.
5. **Improve the model.** Use [compounding intelligence](/guides/deployment/compounding-intelligence) to turn monitoring insights into targeted training data, then retrain and redeploy.
***
## MONITORING
Once traffic flows, the deployment reports on itself from two angles. The **Health** tab tracks the operational side: request volume, error rate, latency percentiles, and token throughput. The **Insights** tab tracks quality: attach judge evaluators to score a sample of live traffic, and their score trends show whether responses are holding up in production. When a trend dips, drill into the scored requests behind it or run a failure modes analysis directly on that slice of traffic.
[Learn about quality monitoring →](/guides/deployment/monitoring)
***
## COMPARE MODELS SIDE BY SIDE
Before you point an application at one endpoint, put the candidates next to each other. The Model Playground chats with up to four deployments or Oumi base models at once, sending the same prompt to all of them, so you can read the answers together and watch how latency and token usage differ.
[Learn about the Model Playground →](/guides/playground)
***
## SELF-HOSTING
If a hosted endpoint cannot meet your requirements, export your model and serve it on your own hardware or a cloud GPU instance instead. Inference logging and monitoring are features of the platform endpoint, so a self-hosted model does not report back into Oumi.
[Learn about self-hosting →](/guides/deployment/self-hosting)
***
## WHAT'S NEXT
Stand up a managed endpoint, for your own model or an external one.
Call your deployment from Python, TypeScript, or curl.
Score sampled production traffic with LLM judges and watch quality trends.
Export your model and serve it on your own infrastructure.
# CALLING YOUR ENDPOINT
Source: https://docs.oumi.ai/guides/deployment/calling-your-endpoint
Call your Oumi deployment from your own code using the OpenAI or Anthropic SDKs
Your deployment's endpoint is wire-compatible with the OpenAI and Anthropic APIs, so you don't need an Oumi-specific client. Point your existing SDK at a different base URL, pass your deployment as the model, and the rest of your code stays the same.
***
## BEFORE YOU START
You will need:
* A deployment in the `Active` state. A deployment that is still `Deploying` or has `Failed` will not serve requests.
* A personal Oumi API key. Use an existing key from **User Settings > API Keys**, or create one from the code-snippet panel on your deployment page.
Open your deployment and use the built-in code snippets rather than transcribing values by hand. They come pre-filled with your endpoint and model path, have a toggle for streaming, and can generate an API key inline.
***
## ENDPOINT
| | Value |
| --------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| **OpenAI-compatible endpoint** | `https://inference.oumi.ai/v1/chat/completions` — custom Oumi models, and external models served by OpenAI |
| **Anthropic-compatible endpoint** | `https://inference.oumi.ai/v1/messages` — external models served by Anthropic |
| **Auth header** | `X-API-Key: $OUMI_API_KEY` |
| **`model` field** | `projects/{project_id}/deployments/{deployment_id}` |
Set your API key on the SDK client and it authenticates for you. You only need to set the header yourself for raw HTTP calls such as curl.
The `model` field on the request body is a deployment resource path rather than a model name, for example `projects/XFZ7K2/deployments/83`. Copy the exact value from your deployment's page.
***
## OPENAI-COMPATIBLE REQUESTS
Use these for custom Oumi models and for external models served by OpenAI. Set `base_url` to `https://inference.oumi.ai/v1` and pass your deployment as the model.
```bash curl theme={null}
curl -X POST https://inference.oumi.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "X-API-Key: $OUMI_API_KEY" \
-d '{
"model": "projects/{project_id}/deployments/{deployment_id}",
"messages": [{"role": "user", "content": "your prompt"}]
}'
```
```python Python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="",
base_url="https://inference.oumi.ai/v1",
)
response = client.chat.completions.create(
model="projects/{project_id}/deployments/{deployment_id}",
messages=[{"role": "user", "content": "your prompt"}],
)
print(response.choices[0].message.content)
```
```typescript TypeScript theme={null}
import OpenAI from 'openai';
const client = new OpenAI({
apiKey: process.env.OUMI_API_KEY,
baseURL: 'https://inference.oumi.ai/v1',
defaultHeaders: { 'X-API-Key': process.env.OUMI_API_KEY ?? '' },
});
const response = await client.chat.completions.create({
model: 'projects/{project_id}/deployments/{deployment_id}',
messages: [{ role: 'user', content: 'your prompt' }],
});
```
***
## ANTHROPIC-COMPATIBLE REQUESTS
External models served by Anthropic use this endpoint instead of `/chat/completions`, which rejects them. Note that `max_tokens` is required.
The Anthropic SDKs append `/v1` to the base URL themselves, so set their base URL to `https://inference.oumi.ai`.
```bash curl theme={null}
curl -X POST https://inference.oumi.ai/v1/messages \
-H "Content-Type: application/json" \
-H "X-API-Key: $OUMI_API_KEY" \
-d '{
"model": "projects/{project_id}/deployments/{deployment_id}",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "your prompt"}]
}'
```
```python Python theme={null}
import anthropic
client = anthropic.Anthropic(
api_key="",
base_url="https://inference.oumi.ai",
)
client.messages.create(
model="projects/{project_id}/deployments/{deployment_id}",
max_tokens=1024,
messages=[{"role": "user", "content": "your prompt"}],
)
```
```typescript TypeScript theme={null}
import Anthropic from '@anthropic-ai/sdk';
const client = new Anthropic({
apiKey: process.env.OUMI_API_KEY,
baseURL: 'https://inference.oumi.ai',
});
await client.messages.create({
model: 'projects/{project_id}/deployments/{deployment_id}',
max_tokens: 1024,
messages: [{ role: 'user', content: 'your prompt' }],
});
```
***
## COLD STARTS
A custom Oumi model running with [scale-to-zero](/guides/deployment/deploy-a-model#autoscaling-and-cost-custom-models) has no warm replica while idle, so its first request after a quiet period spins up capacity before it responds. Expect that call to take noticeably longer than the ones that follow, and set generous client timeouts for it.
If you need consistently low latency on every request, set min replicas to 1 so a replica is always warm. That deployment is then billed hourly whether or not it serves traffic.
External models have no cold start, since the provider keeps its own capacity warm.
***
## TRYING A DEPLOYMENT WITHOUT CODE
Open your deployment and use **Try model** to chat with it directly in the browser. It holds a multi-turn conversation, streams responses as they generate, and lets you set system instructions to test different prompts. This is the quickest way to confirm a single deployment behaves as expected before you wire it into an application.
To compare models rather than exercise one, use the **Model Playground** instead. It puts your deployments and Oumi base models side by side against the same prompt, so you can weigh responses and latency across them at once.
You can also ask the Oumi Agent to run a prompt against a deployment from chat, which returns the response inline along with a snippet you can reuse.
***
## WHAT'S NEXT
Review the requests your deployment has served, including latency and token counts.
Score sampled live traffic with LLM judges and watch quality trends.
# COMPOUNDING INTELLIGENCE
Source: https://docs.oumi.ai/guides/deployment/compounding-intelligence
Turn monitoring insights into targeted training data, retrain your model, and improve it with every production cycle
## OVERVIEW
Compounding intelligence turns production feedback into better models. Quality monitoring reveals where a deployed model struggles, failure mode analysis groups recurring problems, and dataset generation creates targeted examples for retraining.
This guide walks through the workflow in the UI. You can also ask the Oumi Agent to guide you through the same process, from analyzing your traffic on the **Insights** tab to a retrained model ready to redeploy.
Before you begin, enable [quality monitoring](/guides/deployment/monitoring) on a deployment and generate a failure modes report from its production traffic.
***
## THE LOOP
1. **Monitor.** Judge evaluators score a sample of production traffic and plot trends on the **Insights** tab.
2. **Diagnose.** Analyze the traffic behind a score drop to identify recurring failure modes.
3. **Generate.** Create synthetic training data that targets the failure modes you select.
4. **Retrain.** Train a new model on the generated data and your original training set.
5. **Redeploy and verify.** Deploy the new model and use monitoring to confirm the improvement.
***
## STEP 1: CREATE A FAILURE MODES REPORT FROM INSIGHTS
Open the deployment's **Insights** tab. Each judge evaluator has its own score trend:
Click `Select Area to Investigate` and drag across the score drop you want to examine.
Click `View Selected Logs` to confirm that the selected traffic contains the behavior you want to fix. Expand any row to read the judge explanations.
Click `Run Failure Modes Analysis`. Oumi groups the failing responses into recurring patterns and opens the completed report on the deployment's **Failure Modes** tab.
The report helps you address recurring causes instead of fixing individual responses. See [quality monitoring](/guides/deployment/monitoring#investigate-a-drop) for the full investigation workflow.
***
## STEP 2: GENERATE A DATASET FROM THE FAILURE MODES
On the **Failure Modes** tab, select the failure modes you want to address and the specific data items within them. Click `Generate Dataset` to open a confirmation dialog with three settings:
| Field | What it controls |
| -------------------- | --------------------------------------------------------------------------------------------------- |
| **Model** | The hosted model that generates the improved samples. Oumi selects a default when one is available. |
| **Dataset name** | The name of the generated dataset. Leave it blank to let Oumi choose a name. |
| **Samples per item** | How many improved samples to generate for each selected data item (1–20; default: 1). |
Click `Generate Dataset` to start generation. Oumi opens the **Datasets** page, where the new dataset appears when the job finishes.
### ADVANCED MODE
For more control, click `Advanced…`. The Builder opens with your selection preloaded and provides additional settings:
| Section | What it controls |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Failure Mode Selection** | The failure modes and data items used for generation. Your original selection is already loaded. |
| **Inference Configuration** | The generation model, temperature, maximum tokens, seed, and optional requests-per-minute limit. |
| **Dataset Configuration** | The destination dataset and its train, validation, and test split. Choose an existing dataset to append the generated data, or leave the destination empty to create one. Split values must total 1.0. |
To edit the raw synthesis configuration, click `Edit as JSON`, make your changes, and click `Save` to return them to the form. Click `Run Synthesis`, name the dataset, and start the job.
***
## STEP 3: RETRAIN WITH THE NEW DATASET
Train the next version of your model on the original and generated datasets. Using both preserves the original examples while adding targeted coverage for the newly identified failures:
1. On the **Models** page, click `Train New Model` and select `Supervised Fine-Tuning`.
2. Select your base model.
3. In the training dataset selector, choose both the generated dataset and the original training set.
4. Launch the job. See [running training](/guides/training/launch-jobs) for the full configuration options.
When training finishes, [deploy the new model](/guides/deployment/deploy-a-model) and direct traffic to it. Continue monitoring to verify that the targeted scores improve. If scores drop again, repeat the loop with the new traffic.
***
## WHAT'S NEXT
Configure judge evaluators and collect the score trends that start this loop.
Learn how reports group recurring failures in evaluations and production traffic.
Configure and launch a fine-tuning job with your improved training data.
Serve the retrained model and direct production traffic to it.
# DEPLOY A MODEL
Source: https://docs.oumi.ai/guides/deployment/deploy-a-model
Stand up a managed inference endpoint for your own model or an external one
## OVERVIEW
Once you have [generated training data](/guides/datasets/create), [fine-tuned a model](/guides/training), and [evaluated its performance](/guides/evaluations), the final step is deployment. A deployment serves a model on a managed inference endpoint, ready for real-time use.
The Deployments feature is currently in beta.
This page covers both kinds of deployment: a **custom Oumi model** you trained or imported, and an **external model** hosted by OpenAI or Anthropic. Both are created from the same dialog and, once live, behave identically. See [deployment types](/guides/deployment#deployment-types) for how they compare.
***
## BEFORE YOU START
| To deploy | You need |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| A **custom Oumi model** | A deployable trained or [imported](/guides/deployment/importing-models) model in your project |
| An **external model** | **Optional:** Add your own OpenAI or Anthropic API key. When you use your own key, Oumi does not bill you for that model usage. |
The model picker indicates which of your models can be deployed.
An external model needs no trained model at all, so you can deploy one in a project that has none yet.
***
## CREATE A DEPLOYMENT
From the top of the **Models** page, click the `Deploy Model` button; alternatively, click the `+ Create Deployment` button from the **Deployments** page.
The dialog walks through four stages:
Choose the deployment type and fill in its settings.
Optionally set up [quality monitoring](/guides/deployment/monitoring): add the judge evaluators that will score a sample of live traffic, then adjust the sampling rate. You can also enable monitoring later from the deployment's **Info & Configuration** tab.
Oumi provisions capacity and loads your model. This takes a few minutes, and you can navigate away without interrupting it.
The deployment is serving. The dialog shows a runnable code snippet you can copy straight into your own application.
On the **Deploy Model** dialog, select either `Custom Oumi Model` or `External Model`.
### CUSTOM OUMI MODEL
Serves a model from your project on Oumi-managed GPU capacity.
* Provide a unique `Deployment Name`.
* Select a `Model` from the drop-down.
* Choose whether to enable [inference logging](/guides/deployment/inference-logs). It is on by default.
* Optionally open `Advanced Settings` to change the autoscaling policy.
* Click `Start` to deploy your model.
### EXTERNAL MODEL
Forwards requests to OpenAI or Anthropic.
* Provide a unique `Deployment Name`.
* Select a `Provider` from the drop-down. OpenAI and Anthropic are supported today.
* Select your `External Model` from the drop-down.
* Optionally supply your own provider API key: paste a new one, or select a key already stored in your project.
* Choose whether to enable [inference logging](/guides/deployment/inference-logs). It is on by default.
* Click `Start` to deploy your model.
***
## AUTOSCALING AND COST (CUSTOM MODELS)
External models run on the provider's infrastructure, so they have no replicas to scale and no GPU time to pay for. When you use your own provider API key, Oumi does not bill you for that model usage. The rest of this section applies to custom Oumi models.
Hourly billing starts immediately when a deployment begins and continues while it is active. Leave min replicas at 0 to avoid paying for idle capacity.
Custom deployments default to **scale-to-zero**: min replicas 0, max replicas 1. An idle deployment costs nothing and spins up on its first request, at the cost of some added latency on that call. This is the right default for iterative work, where you deploy a model mainly to try it out.
Set min replicas to 1 when you want a deployment kept warm, with no cold-start latency on the first request. It will be billed hourly around the clock, including while it serves nothing.
| Setting | Default | What it does |
| ------------------------ | ------------------ | -------------------------------------------------------- |
| **Min replicas** | 0 | The warm-replica floor. 0 means scale to zero when idle. |
| **Max replicas** | 1 | The ceiling the deployment can scale up to. |
| **Scale-down idle time** | The server default | How long an idle replica waits before scaling down. |
Scale-down idle time offers presets of 1, 5, 15, 30, and 60 minutes, or the server default. It only applies when max replicas is greater than min replicas, since a fixed replica count never scales down.
***
## AUTO-DEPLOY FROM A TRAINING JOB (CUSTOM MODELS)
You can skip the separate deploy step entirely. The **Run Training Job** dialog has an auto-deploy option; enable it and Oumi deploys the resulting model with scale-to-zero autoscaling as soon as training finishes.
The option is unavailable when the base model and training method you selected aren't deployable, or when your project has reached its deployment limit.
***
## MANAGING DEPLOYMENTS
The **Deployments** page lists every deployment in the project with its `Name`, `Status`, `Model`, `Type`, and `Provider`. Click a deployment to open its detail page, where you can rename it, delete it, view its endpoint and code snippets, [try it in the browser](/guides/deployment/calling-your-endpoint#trying-a-deployment-without-code), and read its [inference logs](/guides/deployment/inference-logs). The detail page also carries the deployment's [health metrics](/guides/deployment/health-metrics), its [quality monitoring](/guides/deployment/monitoring) insights, and an **Info & Configuration** tab where you can adjust monitoring and, for custom Oumi models, autoscaling on an active deployment without redeploying.
A deployment moves through these states:
| Status | Meaning |
| ----------- | ------------------------------------------------------------------------------------ |
| `Deploying` | Capacity is being provisioned and the model loaded. |
| `Active` | Serving requests. This is the only state that accepts inference. |
| `Failed` | Provisioning did not complete. Check the operation on the [Jobs](/guides/jobs) page. |
| `Deleting` | Teardown in progress. |
| `Deleted` | Torn down and no longer billable. |
***
## DEPLOYING WITH THE OUMI AGENT
You can also deploy by asking the Oumi Agent. Tell it which model to deploy and it asks you to approve the deployment before provisioning anything, since GPU capacity costs money. Once the deployment is live it hands back a runnable snippet.
The Agent creates custom Oumi model deployments only. Create external-provider deployments from the **Deployments** page.
***
## WHAT'S NEXT
Call your deployment from Python, TypeScript, or curl.
See what your model is asked in production, and retrain from it.
Score sampled live traffic with LLM judges.
# HEALTH METRICS
Source: https://docs.oumi.ai/guides/deployment/health-metrics
Track requests, errors, latency, and token throughput for every managed deployment
## OVERVIEW
The Oumi Agent can read your deployment's operational health on demand. Ask how a deployment is doing and it reports traffic volume, error rate, latency percentiles, and token throughput, then pairs that with [quality monitoring](/guides/deployment/monitoring) scores to tell you whether a problem is operational or a real quality regression.
The same data lives on the deployment's **Health** tab, so you get serving observability without wiring up an external metrics stack. Every managed deployment with [inference logging](/guides/deployment/inference-logs) enabled reports health metrics automatically.
***
## THE HEALTH TAB
Open a deployment and select the **Health** tab. A summary row shows the numbers for the visible window:
| Metric | What it tells you |
| ---------------------- | ----------------------------------------------------------------- |
| **Requests** | Total requests served in the window. |
| **Avg Requests / min** | Request rate over the window. |
| **Error Rate** | Share of requests that failed, with the raw error count on hover. |
| **Latency (p50)** | Median end-to-end response time. |
| **Latency (p95)** | Response time of the slowest 5% of requests. |
Below it, five charts break the window into time buckets:
| Chart | Series |
| ---------------------------- | --------------------------------------------------------- |
| **Requests** | Request count per bucket. |
| **Request Latency (ms)** | P50, P95, and P99 end-to-end latency. |
| **Time to First Token (ms)** | P50, P95, and P99 time until the first token is returned. |
| **Token Throughput (tok/s)** | Input and output tokens per second. |
| **Error Rate (%)** | Failed request share per bucket. |
The **Time window** selector zooms from 1 hour out to 30 days, with a `Custom` option for an exact range. The tab shows up to 30 days of history. Charts support pan and zoom, and each card can be refreshed, expanded, or saved.
***
## WHAT'S NEXT
Score sampled production traffic with LLM judges and track quality trends.
Inspect the individual requests behind the metrics.
# IMPORTING MODELS
Source: https://docs.oumi.ai/guides/deployment/importing-models
Bring your own model into Oumi to evaluate and deploy it
Importing a model brings external weights into your project so you can evaluate and deploy them just like a model trained in Oumi. You can pull from a HuggingFace repo or from a presigned S3 URL.
***
## WHEN TO IMPORT A MODEL
Import a model when you want to:
* Evaluate an open-source or third-party model alongside your own
* Deploy a model you trained outside of Oumi
* Bring a fine-tuned checkpoint into a project to serve it
***
## HOW TO IMPORT A MODEL
1. Go to the **Models** page.
2. Click the **More actions** button (the three dots) at the top right.
3. Choose **Import from HuggingFace** or **Import from S3**.
![Import model menu]()
### Import from HuggingFace
Pull model weights from a HuggingFace Hub repo into your project.
1. Enter a **Display name** for the model.
2. Enter the **Repo ID** in `org/repo` format, for example `meta-llama/Llama-3.2-1B`.
3. Set the **Base model** (see [Set the base model](#set-the-base-model)).
4. Click to start the import.
![Import from HuggingFace]()
Only the repo's default branch is imported. For gated or private repos, add a HuggingFace token to your project first so Oumi can access the weights.
### Import from S3
Pull a model artifact from a presigned S3 URL into your project.
1. Enter a **Display name** for the model.
2. Paste the **Presigned URL** to your model artifact.
3. Set the **Base model** (see [Set the base model](#set-the-base-model)).
4. Click to start the import.
![Import from S3]()
Treat the presigned URL as a secret. The signature grants direct read access to your file.
The URL can point to a single weight file or an archive (`.zip`, `.tar`, `.tar.gz`), up to 2 TB uncompressed and 100,000 files.
***
## SET THE BASE MODEL
The **base model** tells Oumi which model your weights derive from. Oumi uses it to apply the correct chat template and to enable accurate evaluation and deployment, so setting it right matters.
* For HuggingFace imports, Oumi auto-detects the base model from the Repo ID. Leave it on **Auto-detect** unless your weights derive from a different model.
* For S3 imports, set the base model yourself. Use the HuggingFace ID of the model your weights are based on, for example `Qwen/Qwen3-0.6B`.
* If your base model isn't in the list, choose **Other** and type its HuggingFace ID.
![Set the base model]()
If you fine-tuned a model, the base model is the original model you started from, not your fine-tuned checkpoint.
***
## AFTER IMPORTING
HuggingFace and S3 imports run in the background. You'll see a confirmation that the import has started, and the model appears on the **Models** page once it finishes. Larger models take longer to copy in.
***
## USING YOUR IMPORTED MODEL
Once imported, your model works anywhere a custom model does:
Select your imported model as the model under test and score it with your evaluators.
Serve your imported model on a managed inference endpoint.
***
## IMPORTING WITH THE OUMI AGENT
You can also import a model by asking the Oumi Agent. Tell it which model you want to bring in, for example "import `meta-llama/Llama-3.2-1B` from HuggingFace," and the Agent walks you through the import and can continue straight into evaluation or deployment.
# INFERENCE LOGS
Source: https://docs.oumi.ai/guides/deployment/inference-logs
Review the requests your deployment has served, and turn real production traffic into a training dataset
Every call your deployment serves can be recorded, giving you the actual prompts your model receives in production. Those logs are useful twice: first for debugging what went wrong on a specific request, then as raw material for your next training run.
***
## ENABLE LOGGING
Inference logging is a per-deployment setting, on by default, that records the requests and responses your deployment serves. Turn it off to keep production traffic out of the logs entirely. It works the same way for models you train and for [external models](/guides/deployment/deploy-a-model#external-model) served by OpenAI or Anthropic.
Logging also feeds the deployment's observability features: [health metrics](/guides/deployment/health-metrics) are computed from the logs, and [quality monitoring](/guides/deployment/monitoring) scores a sample of them. Turning logging off turns those off too.
Turning logging on affects future requests. Traffic served while it was off was never recorded and cannot be recovered.
***
## READ THE LOGS
The **Inference Logs** tab lists one row per request, with a preview of the exchange alongside its status, timing, and token counts.
Open a single entry to see the full input and output messages, the token counts and latency for that call, and the error message if it failed.
To debug a failure, open the failed request and read its error message alongside the input that produced it.
***
## EXPORT LOGS TO A DATASET
This is how the loop closes. Take a slice of real production traffic and turn it into a dataset you can evaluate against, synthesize from, or fine-tune on.
From the **Inference Logs** tab, click `Export as Dataset`:
1. Choose `Create new dataset` and give it a name, or switch to the other tab to append to an existing dataset.
2. Set `Maximum Logs`. The most recent logs are included first, so this acts as a "latest N requests" cap.
3. Set a `Start Date` and `End Date` to bound the window.
4. Click `Generate Dataset`.
The export runs in the background, and the dataset appears on your **Datasets** page when it finishes.
You can also ask the Oumi Agent to export logs for you, and to summarize traffic or find failing calls without leaving chat.
***
## WHAT'S NEXT
Score a sample of these logs with LLM judges and watch quality trends.
Inspect and validate the dataset your exported logs produced.
# QUALITY MONITORING
Source: https://docs.oumi.ai/guides/deployment/monitoring
Score a sample of production traffic with LLM judges and track your deployment's quality over time
## OVERVIEW
Quality monitoring uses LLM judges to score a sample of production traffic. The deployment's **Insights** tab plots each evaluator's scores over time, helping you spot regressions before they become widespread.
Quality monitoring works on managed deployments with [inference logging](/guides/deployment/inference-logs) enabled.
***
## HOW IT WORKS
A monitor attaches to a deployment and holds two things: a set of **judge evaluators** and a **sampling rate**.
1. Your deployment serves a request, and inference logging records it.
2. The monitor samples the percentage of requests you configured.
3. The attached judge evaluators score each sampled request.
4. The **Insights** tab plots a score trend for each evaluator.
***
## SET UP MONITORING
There are three ways to start monitoring a deployment.
### WITH THE OUMI AGENT
Ask the Oumi Agent to monitor a specific deployment. It checks that inference logging is enabled and proposes judge evaluators and a sampling rate. Review the proposal, then approve it to start scoring.
### DURING DEPLOYMENT
The **Deploy Model** dialog includes a **Monitoring** step, right after you configure the deployment. It is optional: add one or more judge evaluators and pick a sampling rate to enable monitoring, or click `Skip` to move on. Anything you set here can be changed later from the deployment's **Info & Configuration** tab.
### FROM THE CONFIGURATION TAB
On the deployment detail page, open the **Info & Configuration** tab and scroll to the **Monitoring** section:
* **Status** shows whether a monitor is actively scoring this deployment's traffic.
* **Evaluators** lists the judges scoring sampled traffic. Click `+ Add evaluator` to attach one; adding the first evaluator creates the monitor with a 10% sampling rate.
* **Sampling rate** sets the share of production requests that get scored. The dropdown offers 10%, 20%, 50%, and 100%.
Changes apply immediately.
***
## READ THE INSIGHTS TAB
The **Insights** tab is where monitoring results land.
* The **Time window** selector zooms from 1 hour out to 30 days, with a `Custom` option for an exact range. The tab shows up to 30 days of score history.
* A summary pill per evaluator shows its mean score over the visible window, as a percentage.
* Each evaluator gets its own **score trend chart**, one point per scored request.
If a judge fails to produce scores, a warning badge appears on that evaluator's chart with the failure count and reasons on hover, such as provider quota, authentication, rate limits, or timeouts. These are judge failures, not low model scores; they usually point at the judge's provider key or configuration rather than your model.
***
## INVESTIGATE A DROP
When a trend dips, drill into the exact traffic behind it:
Click `Select Area to Investigate` and drag across any chart to mark the time range you care about.
Click `View Selected Logs` to open the **Scored Inference Logs** table: one row per scored request, with the judgment each evaluator gave it. Expand a row to read the full judge explanations for that request.
Click `Run Failure Modes Analysis` to cluster the failing responses in the selected range into recurring patterns. The report opens on the deployment's **Failure Modes** tab when it is ready.
A failure modes report groups the reasons that production responses scored poorly, helping you address the highest-impact pattern first. From there, use [compounding intelligence](/guides/deployment/compounding-intelligence) to generate targeted training data, retrain the model, and verify the improvement after redeployment.
***
## MANAGE MONITORED EVALUATORS
* **Versions are pinned.** A monitor scores with the exact evaluator version you attached.
* **Removing an evaluator** stops scoring against it. You can add it back later.
Removing the last monitored evaluator turns off monitoring for the deployment and permanently deletes the monitor along with all of its historical scores. This cannot be undone.
***
## WHAT'S NEXT
Turn monitoring insights into targeted training data and an improved model.
Watch the operational side: traffic, errors, latency, and throughput.
# NEXT STEPS
Source: https://docs.oumi.ai/guides/deployment/next-steps
Where to go from here
Once your model is in production, you can use it for inference and connect it to downstream applications. Oumi automatically version-controls your project assets—including models, datasets, evaluations, evaluators, and recipes—so you can reproduce every deployment.
As data and usage patterns change, model performance can degrade. [Quality monitoring](/guides/deployment/monitoring) helps you catch regressions early by tracking judge scores on production traffic.
When scores decline, [compounding intelligence](/guides/deployment/compounding-intelligence) closes the loop. Analyze the affected traffic, generate training data that targets its recurring failures, then [retrain](/guides/training) and [redeploy](/guides/deployment/deploy-a-model).
Watch judge scores over live traffic and catch regressions early.
Track your deployment's traffic, errors, latency, and throughput.
Evaluate your model with the Builder.
Turn production failure modes into targeted training data and a better model.
# SELF-HOSTING
Source: https://docs.oumi.ai/guides/deployment/self-hosting
Export your trained model from Oumi and serve it on your own hardware or cloud GPU instance
Export a model from Oumi and you can serve it anywhere. Oumi packages trained artifacts in a standard, portable format that works with common inference engines, so you move from the platform to your own serving stack without conversion steps.
***
## WHEN TO SELF-HOST
A [managed deployment](/guides/deployment/deploy-a-model) is the faster path for most teams, and it is the only path that gives you inference logs, [health metrics](/guides/deployment/health-metrics), and [quality monitoring](/guides/deployment/monitoring) inside Oumi. Self-host when you have a requirement a hosted endpoint cannot meet:
* Your network is air-gapped, or your data cannot leave your own infrastructure
* You have an existing serving stack the model needs to fit into
* You want to run the model on hardware you already own
* You are testing locally against a model you are still iterating on
You can also export purely for archival, to keep a copy of the weights outside the platform.
***
## STEP 1: EXPORT YOUR MODEL
1. Go to the **Models** page and click the model name.
2. On the model's detail page, click **Export**.
3. Click **Continue Export** to download the file to your computer.
An export contains everything an inference engine needs:
* **Model weights:** the trained parameters resulting from fine-tuning
* **Tokenizer and configuration files:** required for correct input and output handling
* **Model metadata:** information about the training run and configuration
Export once evaluation results meet your success criteria. If your model does not meet quality expectations yet, keep iterating on training or data synthesis first.
***
## STEP 2: SERVE THE MODEL
[vLLM](https://github.com/vllm-project/vllm), and its Apple Silicon equivalent [vLLM-MLX](https://github.com/waybarrios/vllm-mlx), run an OpenAI-compatible inference server from your exported directory. The steps below are the same whether you run them on your own machine or on a cloud GPU instance. Follow the installation instructions on those projects' homepages, for instance:
```bash theme={null}
pip install vllm
```
Navigate to your exported model's parent directory and start the server:
```bash theme={null}
vllm serve ./exported_model/ --port 8000
```
Then call it with [OpenAI's Python library](https://github.com/openai/openai-python):
```bash theme={null}
pip install openai
```
```python theme={null}
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
response = client.chat.completions.create(
model="exported_model",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
```
### RUNNING ON A CLOUD GPU INSTANCE
To serve from a cloud instance instead of your own machine, transfer the exported artifacts there first (via `scp`, S3, or GCS), then run the same two commands above on the instance. Point your client at the instance rather than localhost:
```python theme={null}
client = OpenAI(base_url="http://:8000/v1", api_key="unused")
```
Replace `` with the public IP or hostname of your instance, and open port 8000 in your security group or firewall rules. Do not expose a vLLM server publicly without authentication in front of it.
### OTHER SERVING OPTIONS
You can use your exported model directly from Python via the `vllm` or Hugging Face `transformers` libraries rather than through a server. Hugging Face `transformers` also ships a CLI that serves an OpenAI-compatible API; see [its serving documentation](https://huggingface.co/docs/transformers/v5.2.0/serve-cli/serving).
***
## MANAGED PLATFORMS ELSEWHERE
If you want a hosted endpoint outside Oumi, an exported model imports into several third-party platforms.
### AMAZON BEDROCK
Bedrock supports custom model import, so you can serve your Oumi-trained model as a managed AWS endpoint. Read the [AWS blog post](https://aws.amazon.com/blogs/machine-learning/accelerate-custom-llm-deployment-fine-tune-with-oumi-and-deploy-to-amazon-bedrock/) for a full walkthrough.
**Best for:** teams already on AWS who want a managed endpoint with no infrastructure overhead.
### LAMBDA
Lambda provides on-demand GPU instances well suited to hosting a vLLM server. Watch the [Lambda deployment video](https://www.youtube.com/watch?v=0XpfYRpd_FA) for a step-by-step guide to spinning up an instance, loading your exported model, and making requests.
**Best for:** teams who want direct GPU access and control over the serving stack.
***
## CHOOSING AN INSTANCE TYPE
The right instance depends on your model size, latency requirements, and budget. As a general rule, larger models need more GPU memory: a 7B parameter model typically needs at least 16 GB of VRAM, while a 30B+ model needs significantly more. Consult your provider's documentation for current availability and pricing:
* [AWS EC2 GPU instances](https://aws.amazon.com/ec2/instance-types/#Accelerated_Computing)
* [GCP GPU machine types](https://cloud.google.com/compute/docs/gpus)
* [Lambda GPU cloud](https://lambdalabs.com/service/gpu-cloud)
***
## OPERATIONAL CONSIDERATIONS
**Cost:** cloud GPU instances bill by the hour whether or not they serve traffic. For variable load, consider auto-scaling groups or a serverless inference platform so you are not paying for idle capacity. A [managed deployment](/guides/deployment/deploy-a-model) scales to zero by default and handles this for you.
**Latency:** a network round trip adds latency compared to running locally. Choose a region close to your users and keep request payloads small.
**Security:** restrict access to your endpoint with API keys, VPC networking, or IAM policies.
**Model versioning:** keep exported artifacts versioned in cloud storage so you can roll back to a previous model.
**Observability:** a self-hosted model does not report back into Oumi, so inference logs, [health metrics](/guides/deployment/health-metrics), and [quality monitoring](/guides/deployment/monitoring) are not available for it. Plan for your own logging and evaluation, or keep a managed deployment alongside for comparison.
***
## WHAT'S NEXT
Stand up a managed endpoint instead, with logging and monitoring built in.
Re-evaluate your model as production data evolves.
# EVALUATE MODEL
Source: https://docs.oumi.ai/guides/evaluations
Determine how well your model performs and identify failure modes
The Oumi Agent turns model evaluation into a structured, repeatable loop. It configures evaluation runs from your task description, selects appropriate [evaluators](/guides/evaluators), and surfaces [failure modes](/guides/evaluations/failure-modes) so you can pinpoint issues and improve your model without manually reviewing individual outputs.
Setting up a rigorous evaluation pipeline from scratch typically takes days of engineering work. With the Oumi Agent, the same setup takes minutes. Because every run is fully recorded and reproducible, you build an audit trail automatically, reducing the cost of debugging regressions and making it practical to evaluate continuously rather than only at release time.
## HOW EVALUATIONS WORK
Evaluations are created using the LLM-as-a-Judge framework, where a language model is prompted to analyze outputs and assign scores. The model performing this judgment is called the evaluator model. This is typically different from the model being tested, and can be a strong general-purpose model or a smaller model fine-tuned specifically for evaluation tasks.
The model under evaluation is the system generating the `prompt`, `response` pairs being scored. This may be a baseline model or the current iteration of your custom model. When you run an evaluation, Oumi executes an evaluation run, which scores the model against a defined dataset and aggregates results. In terms of scoring, evaluators score model outputs against a specific criterion (e.g., `safety`, `quality`, or `correctness`).
Each evaluator operates independently, allowing you to assess different dimensions of model performance in a modular way.
## EVALUATION WORKFLOW
An evaluation run in Oumi follows these steps:
* Generate responses by running a model on a set of prompts.
* Score each (prompt, response) pair using one or more evaluators.
* Aggregate results to assess overall performance.
* Extract higher-level failure modes to explain underperformance.
Each new project includes [predefined evaluators](/guides/evaluators/defining#using-oumi’s-built-in-evaluators) to help you get started quickly. These evaluators are editable, optional, and reusable, providing strong baseline configurations for your evaluations.
# DIAGNOSING FAILURE MODES
Source: https://docs.oumi.ai/guides/evaluations/failure-modes
Identify patterns in your model results
## OVERVIEW
Oumi automatically extracts failure modes from evaluation results, identifying recurring patterns that explain why certain outputs did not meet the defined criteria. Instead of treating low scores as isolated incidents, failure modes surface systematic weaknesses (e.g., reasoning gaps, formatting errors, or unsafe responses), providing clear, actionable insight for the next iteration.
By clustering related errors, Oumi helps you focus on high-impact improvements across data, prompts, models, or evaluators. You can also generate a new dataset directly from an evaluator’s identified failure modes, enabling targeted data synthesis and creating a tight feedback loop between evaluation and model refinement.
## ACCESSING FAILURE MODES
To access the failure modes for a particular evaluation:
1. Go the the **Evaluation** page and click on the name of the evaluation.
2. Click on the `Review Failure Modes` button.
3. On the **Failure Modes** page, review each individual item as needed.
To generate a training dataset that targets specific issues, select the failure mode(s) in question and click `Generate Dataset`.
# NEXT STEPS
Source: https://docs.oumi.ai/guides/evaluations/next-steps
What to do after model evaluation
Your next steps after evaluation depend on your predefined success thresholds. In general, you should:
* Review the summary metrics and confirm the model version, dataset, and configuration used for the run.
* If the evaluation was saved as a recipe, use it to verify reproducibility and compare results with prior runs.
* If metrics meet or exceed your threshold (for example, 98% safety against a 95% requirement), you may proceed to deploy the model.
If results fall below your success criteria, examine detailed outputs to identify failure patterns. Adjust training data, configurations, or evaluators as needed, and refine evaluators if they do not accurately reflect real-world expectations.
Oumi supports targeted data synthesis to address specific failure modes, enabling efficient fine-tuning and re-evaluation in the next iteration.
# EVALUATION RECIPES
Source: https://docs.oumi.ai/guides/evaluations/recipes
Making evaluations consistent and repeatable
Recipes allow you to save the exact configuration of your model tests, making performance evaluations standardized, reproducible, and easy to repeat.
Each recipe automatically logs and versions all parameters and experimental settings, ensuring deterministic execution and consistent results over time. This gives you full transparency into how an evaluation was run and confidence that it can be replicated whenever needed.
## SAVING EVALUATION RECIPES
Oumi allows you save your evaluations as recipes for a complete, traceable record of your configurations, making it simple to reproduce results and maintain consistency across experiments.
### HOW TO SAVE
* From the **Evaluations** page, click on `Run an Evaluation` to start a new evaluation.
* Configure your evaluation's inputs in the Builder.
* Click on the drop-down menu next to the **Save** button and select `Save as`.
* Give your evaluation a `Display Name` and click `Save Recipe`.
## ACCESSING SAVED EVALUATION RECIPES
You can access saved evaluation recipes from the **Recipes** page. Select the `Evaluation` type from the `Type` drop-down to filter on evaluation recipes.
Click a recipe to review and edit its JSON configuration. After opening an item, you can:
* Save the recipe
* Create a new recipe
* Update an existing recipe
* Rename the recipe
* Run the job upon saving
# UNDERSTANDING RESULTS
Source: https://docs.oumi.ai/guides/evaluations/results
How to read the results from your evaluation run
Once your evaluation job has finished, go to **Evaluations** and click on your evaluation to view the results. Each evaluator’s score indicates how well the model performed against the defined criteria.
Clicking on `Explore Results` gives access to individual sample-level results.
## INTERPRETING RESULTS
You can use your evaluation results to:
* Compare baseline models to fine-tuned models
* Identify regressions or improvements due after model changes
* Decide whether to retrain, adjust data, or refine evaluators
Evaluation results become significantly more valuable when paired with failure mode analysis and data synthesis. By identifying where and why a model underperforms, you can systematically generate targeted data to address those weaknesses, creating a tight feedback loop between evaluation, diagnosis, and improvement. This integrated approach enables more efficient iteration and drives measurable gains in model performance.
# RUNNING EVALUATIONS
Source: https://docs.oumi.ai/guides/evaluations/running
Using the Oumi Builder to perform model evaluation
To run an evaluation against an existing model, go to **Evaluations** and click on `Run an Evaluation`.
Click on `Judge-Based Evaluation` from the Builder. Under the **INPUTS** tab, provide the following information:
* **Model** - A hosted or custom model for evaluation.
* **Evaluators** - One or more evaluators to score model outputs.
* **Dataset** - The dataset to evaluate against.
* **Failure Mode Analysis (optional)** - Whether to generate failure modes automatically
* **Inference Configurations (optional)** - Inference parameters like `Temperature`, `Max Tokens`, `Seed`, `Requests Per Minute`.
After confirming and launching the evaluation job, you can view the results on the **Evaluations** page.
# EVALUATORS
Source: https://docs.oumi.ai/guides/evaluators
Define and manage model evaluators
Traditional approaches to evaluating model quality don’t scale well. Human review is slow and costly, rule-based checks are often too rigid for open-ended outputs, and conventional metrics frequently fail to capture what users actually value.
Evaluators solve these problems by using LLM-based judges to assess response quality at scale. These judges can measure nuanced attributes such as accuracy, completeness, and tone across thousands of examples, while providing detailed per-example feedback that makes it easy to pinpoint where a model is falling short. For tasks with known answers, classification and extraction evaluators score outputs against ground truth.
The Oumi Agent can create judge evaluators from a natural-language description of your task. It turns your goal into scoring criteria, labels, and edge cases that you can review and reuse.
## HOW IT WORKS
1. Your model generates responses to a set of prompts
2. An evaluator scores each response using its configured method
3. You get results across your dataset, showing where your model excels and where it struggles
## WHAT TO DEFINE IN A JUDGE EVALUATOR
For a judge evaluator, define:
* **Evaluation criteria:** A prompt describing what the judge should assess (e.g., "Is the response accurate and complete?")
* **Judgment labels:** The rating scale the judge uses (e.g., "poor", "acceptable", "good", "excellent")
* **Scoring:** Numeric scores mapped to each label
* **Judge model:** Which hosted LLM runs the evaluation (e.g., GLM-5, Qwen3-235B-A22B-Instruct-2507)
## BUILT-IN & CUSTOM JUDGE EVALUATORS
Oumi includes built-in judge evaluators (such as instruction following, safety, topic adherence, and truthfulness) to help you quickly establish baselines and gather early feedback. You can review, edit, and reuse these evaluators across evaluations, or create a custom judge evaluator using the Builder.
You can also describe your desired judge evaluator in natural language with the Oumi Agent, specifying scoring criteria, selecting the judge model, and including additional dataset fields for context as needed.
Custom judge evaluators are reusable and should focus on a single, clearly defined property to ensure consistent and reliable results.
## WHY EVALUATORS MATTER
* **Before training:** Benchmark a base model to see where it falls short
* **After training:** Measure whether fine-tuning actually improved quality
* **Compare models:** Run the same evaluators on different models to see which performs better
* **Identify failure modes:** Find specific examples where the model struggles, then use those insights to improve your training data
## EXAMPLE JUDGE CRITERIA
For a customer support bot, you might create separate judge evaluators for:
* **Accuracy:** Did the response contain correct information?
* **Tone:** Was the response empathetic and professional?
* **Completeness:** Did it fully address the customer's question?
* **Policy compliance:** Did it follow company guidelines?
## WHAT'S NEXT
Establish criteria for measuring model performance
Save and reuse evaluator configurations
# DEFINING EVALUATORS
Source: https://docs.oumi.ai/guides/evaluators/defining
Establish criteria for measuring model performance
Evaluators define what strong performance looks like for your task and are essential for building effective custom AI. Oumi provides built-in judge evaluators and tools for creating custom evaluators.
***
## ACCESSING YOUR EVALUATORS
You can view and manage your evaluators on the **Evaluators** page.
***
## CHOOSING AN EVALUATOR TYPE
* **Judge:** Use an LLM to assess open-ended qualities such as accuracy, tone, or instruction following.
* **Classification:** Compare a predicted label with a ground-truth label.
* **Extraction:** Compare named entities, structured output, or function calls with ground truth.
Classification and extraction evaluators do not use a judge model. The rest of this guide explains how to create a judge evaluator.
***
## USING OUMI'S BUILT-IN EVALUATORS
Oumi includes a growing set of general-purpose judge evaluators that work well across many tasks:
* **Instruction following**: Is the model response faithful to the prompt?
* **Safety**: Is the response free from harmful language?
* **Topic adherence**: Does the response stay on topic with respect to the prompt?
* **Truthfulness**: Is the response correct as far as is known from the model knowledge and context window?
These built-in evaluators are ideal when:
* You are establishing an initial baseline
* You want fast feedback without custom configuration
* Your task aligns with common evaluation dimensions
Built-in evaluators can be reviewed, edited, or extended as needed.
***
## CREATING A CUSTOM JUDGE EVALUATOR
Oumi also enables you to build custom judge evaluators tailored to your specific requirements.
* On the **Evaluators** page, click on the `Create Evaluator`.
* In the Builder, click on `Create an Evaluator`.
On the **Inputs** tab, provide the following fields to define your evaluator:
### EVALUATOR IDENTITY
Provide a name and description for your judge.
* `Name`: The name of the judge; should indicate what this judge evaluates (e.g., "Instruction Following")
* `What should the judge evaluate?`: A clear sentence describing the evaluation goal (e.g., "Evaluate whether the response accurately follows all instructions provided in the user's request")
### JUDGMENTS
Judgments define the possible outcomes of an evaluation and how each outcome is scored. They tell the evaluator how to label a response and what that label means.
Click `Add Judgment` and provide the follow three fields to add a new judgement:
* `Label`: The name of the outcome (for example, “Correct”, “Incorrect”, “Adherent”, “Non-Adherent”). This is what the evaluator will output.
* `Condition`: A clear description of when this label should be applied. This acts as guidance for the evaluator, helping it decide which label best fits a given response.
* `Score`: A numeric value associated with the label (such as 1 for correct, 0 for incorrect). Scores allow you to quantify performance and compare results across examples.
#### HOW JUDGMENTS WORK
Judgments define the grading system for your task.
For example:
* In a classification task:
* Correct → score 1
* Incorrect → score 0
* In a quality evaluation task:
* High Quality → score 1
* Acceptable → score 0.5
* Poor → score 0
By defining clear labels and conditions, you ensure that evaluations are consistent, interpretable, and aligned with your goals.
### DATA FIELDS
Define the data fields that the judge should evaluate.
* `Input Type` - Specify whether the input type is:
* `single turn (a user request and an assistant response)`
* `single turn (an assistant response)`
* `mult-turn (a conversation between a user and an assistant)`
* `Additional Fields` - Provide additional fields (as `Key`, `Display Name`, and `Description`) for the judge to consider in its evaluation
### EVALUATION CRITERIA (OPTIONAL)
Click `Add Criterion` to add specific criteria the judge should consider. Each item should consist of a `Name` that describes the criteria (e.g., "Format Compliance") and `Description` (e.g., "Response should be in JSON format and not exceed 1000 characters").
### EXAMPLES (OPTIONAL)
Click `Add Example` to add the additional fields to provide examples to guide the judge's behavior. Each example should consist of a `Name` that describes the criteria (e.g., "Format Compliance") and `Description` (e.g., "Response should be in JSON format and not exceed 1000 characters").
### MODEL SETTINGS
Specify the model and settings used to calculate the score according to the instructions.
This model is not to be confused with the model under evaluation that produced the (`prompt`, `response`) pairs
***
## USING THE AGENT
You can also use the Agent to define your judges using natural language prompts. Oumi will suggest evaluator patterns based on your task (for example, comprehensiveness, groundedness, fluency, format adherence).
See the built-in evaluators for example of correctly formatted prompts.
Models from providers like OpenAI are available without adding a provider API key. When you use your own key from [Provider API keys](/admin/api-keys), Oumi does not bill you for that model usage.
Evaluators can use models from leading providers such as OpenAI and Anthropic.
Custom evaluators are saved and reusable across evaluations.
***
## BEST PRACTICES FOR DESIGNING CUSTOM EVALUATORS
Well-designed custom evaluators produce consistent, actionable results. The following best practices ensure that your custom evaluators are clear, focused, and aligned with your success criteria.
* Be specific about what success looks like.
* Avoid overly broad instructions.
* Each evaluator should measure a single, non-overlapping property.
* Start simple, then refine based on failure modes.
# NEXT STEPS
Source: https://docs.oumi.ai/guides/evaluators/next-steps
Apply and iterate with evaluators
Once your evaluators are defined, you can use them in your workflow to measure performance across the dimensions that matter most for your task. Each evaluator should align with a clearly defined success criterion so results can support informed decision-making.
After an evaluation completes, review overall scores and examine any [identified failure patterns](/guides/evaluations/failure-modes). Look for trends across outputs and confirm that results reflect your real-world expectations. Refine evaluator instructions if needed to ensure scoring remains consistent and meaningful, and use these insights to determine next steps.
If performance meets your thresholds, move toward deployment. If not, refine training data, prompts, or model configuration, and consider [synthesizing targeted datasets](/guides/synthesis) from [evaluator failure modes](/guides/evaluations/failure-modes) to strengthen your model's next iteration.
# SAVED EVALUATORS
Source: https://docs.oumi.ai/guides/evaluators/recipes
Save and reuse evaluator configurations
Oumi allows you to save and reuse evaluator configurations. Each saved evaluator keeps its instructions, model settings, and optional data fields together so you can use the same evaluator across evaluation runs.
## SAVE AN EVALUATOR
To save an evaluator:
1. On the **Create an Evaluator** page, click on the `Save` icon.
2. Give your evaluator a `Display Name` and click `Save Evaluator`.
You can now access your new evaluator from the **Evaluators** page.
# EXAMPLES
Source: https://docs.oumi.ai/guides/examples
Explore a curated set of prompts that demonstrate Oumi’s capabilities
By interacting with the Oumi Agent, you can quickly create, evaluate, and train models with precision, regardless of where you are in the machine learning lifecycle.
The following are just a few examples that highlight some of the tasks you can accomplish using simple natural language prompts.
Check out the [Prompt Library](/reference/prompts) for more examples of what you can do with the Oumi Agent.
***
## TRAIN MODELS
Fine-tune a model to automatically categorize customer support questions by topic and urgency.
```prompt theme={null}
Fine-tune a model to classify customer support questions by topic and urgency
```
Fine-tune Qwen to identify the intent behind incoming customer support messages.
```prompt theme={null}
Fine-tune Qwen to detect the intent behind incoming customer support messages
```
Train a compact model for fraud detection using on-policy distillation techniques.
```prompt theme={null}
Train a compact model by distilling a large fraud detection model’s responses into a smaller model using on policy distillation.
```
***
## BUILD & ANALYZE DATASETS
Create a structured dataset of customer support tickets labeled by both urgency and topic to support classification tasks.
```prompt theme={null}
Create a labeled dataset of customer support tickets categorized by urgency and topic
```
Create 500 question-answer pairs from a product documentation PDF for training or evaluation use.
```prompt theme={null}
Generate 500 question-answer pairs from a product documentation PDF
```
Increase dataset size and diversity by generating new samples that follow the same style and format.
```prompt theme={null}
Expand my dataset by generating new samples that match the style and format of my existing examples
```
Improve evaluation robustness by generating additional samples consistent with your existing dataset.
```prompt theme={null}
Expand my dataset by generating additional samples consistent with my existing dataset to improve my evaluation
```
Analyze your dataset and generate new tasks with detailed solutions for addressing missing coverage.
```prompt theme={null}
Analyze my coding dataset and generate additional tasks with step-by-step solution breakdowns to fill gaps
```
***
## EVALUATE MODELS
Define evaluators that score model outputs based on key criteria like helpfulness and accuracy.
```prompt theme={null}
Create evaluators to score my model's responses for helpfulness and accuracy
```
Build a targeted evaluator to assess how well your model performs on customer support interactions.
```prompt theme={null}
Create an evaluator to measure my model's response quality on customer support tickets
```
Run evaluations to uncover common weaknesses and failure modes in your model’s responses.
```prompt theme={null}
Evaluate Qwen on a customer support dataset and surface the most common failure patterns
```
Evaluate model performance on sentiment classification using real-world product review data.
```prompt theme={null}
Test Qwen on a sentiment analysis task using customer food product reviews
```
***
## WHAT'S NEXT
Build your first model by diving into the [Quickstart](/guides/quickstart) and building your first custom machine learning model in Oumi.
# OVERVIEW
Source: https://docs.oumi.ai/guides/intro
Introducing the Oumi workflow
The Oumi Agent is your primary collaborator for building custom AI models. Using natural language prompts, it orchestrates the full model development lifecycle from [dataset creation](/guides/datasets/create) to [training](/guides/training), [evaluation](/guides/evaluations), and [deployment](/guides/deployment), so you can move from problem to production in hours instead of weeks.
By selecting the right model size, configuration, and training strategy for your task, the Oumi Agent helps reduce inference costs and latency while keeping quality high. It also works continuously in the background to surface missing configurations, flag potential issues, and keep your workflow on track, allowing you to stay focused on outcomes, not setup. At every stage, you retain full control over how each step is run.
## THE OUMI UI
After logging in and selecting a project, you will be taken to Oumi’s main interface, where you can see a high level overview of your project’s artifacts and access quick actions for common tasks.
The left-hand navigation contains links to each one of the main sections in the Oumi platform:
* **Overview:** The main landing page for your project's artifacts.
* **Datasets:** Import existing data or generate new datasets for training and evaluation.
* **Evaluators:** Define metrics and judges used to score model outputs, including accuracy, safety, and task success.
* **Evaluations:** Run evaluations to score models on your datasets using your evaluators. Compare candidates, inspect failure modes, and track progress over time.
* **Models:** Train models using datasets and insights from completed evaluations.
* **Recipes:** Reusable configurations and settings.
* **Files:** Additional data for grounding your responses.
* **[Jobs](/guides/jobs):** A list of current and past long-running jobs.
The Oumi UI is designed to reflect the natural lifecycle of building and refining models and its key workflow stages, making it intuitive to move forward or revisit earlier steps based on results and clearly defined success criteria.
## JUMP RIGHT IN
Here are a few next steps to begin building custom AI for your organization with Oumi:
Get up to speed with the Oumi Agent.
Learn how to use the Oumi Builder.
Learn more about Oumi recipes.
# OUMI AGENT
Source: https://docs.oumi.ai/guides/intro/agent
Build, fine-tune, and evaluate custom AI models with guided assistance
The Oumi Agent dramatically amplifies your ability to create custom AI models quickly and efficiently. Using [natural language prompts](/reference/prompts), you can rapidly build custom models and execute complex ML workflows in record time, enabling you to move faster, accomplish more with less effort, and stay focused on outcomes rather than setup or infrastructure overhead.
The Agent can guide you through the [full model development lifecycle](/guides/overview#how-oumi-does-it), including [generating synthetic training data](/guides/synthesis), [fine-tuning open-source models](/guides/training), and [evaluating model performance](/guides/evaluations).
## USING THE AGENT
You can access the Oumi Agent through the UI at every stage of the workflow for guidance and recommendations. The Agent also works in the background to help ensure your setup remains complete and consistent, surfacing missing configurations, flagging potential issues, and highlighting critical steps as you progress through each phase.
### TRY IT OUT
Suppose you want to build a fraud classification model from scratch. You can get started with a simple prompt like:
```prompt theme={null}
I need to create a classification model to predict whether an e-commerce transaction is fraudulent. Can you guide me through the steps?
```
If you issue this prompt, the Agent will guide you through each stage of the workflow, explaining its actions along the way and completing the necessary steps on your behalf.
Check out the [Prompt Library](/reference/prompts) for more examples of what you can do with the Oumi Agent.
By continuously monitoring your progress and validating key inputs, the Oumi Agent helps reduce errors, prevent misconfigurations, and keep your workflow on track, so you can focus on building and improving your model with confidence.
# NEXT STEPS
Source: https://docs.oumi.ai/guides/intro/next-steps
Where to go from here
Oumi is designed to help you move intuitively from task definition to a working predictive model as quickly as possible. Now that you have an overview of the platform’s UI and functionality, explore its features for building and refining machine learning assets in more depth.
Upload, generate, and validate datasets in Oumi
Define and manage judges of your model performance
Determine how well your model performs and identify failure modes
Fine-tune your model to improve its task-specific performance
# RECIPES
Source: https://docs.oumi.ai/guides/intro/recipes
Reusable configurations for end-to-end model workflows
Oumi recipes are ready-to-run JSON configurations that define workflows across the entire [model development lifecycle](/guides/overview#how-oumi-does-it), from [data synthesis](/guides/synthesis) to [evaluation](/guides/evaluations) and [training](/guides/training). They provide full transparency and flexibility, allowing you to inspect and modify any step before it runs.
With recipes, you can easily save, reuse, and standardize model development processes across projects.
## ACCESSING & RUNNING RECIPES
Recipes are automatically generated from your high-level task definitions and remain fully transparent and customizable, allowing you to review each stage of the workflow and modify any component to use them as-is or tailor them to your specific needs.
To access your saved recipes, click `Recipes` in the left-hand navigation.
Once on the **Recipes** page, you can:
* Filter by type (`Training`, `Evaluation`, `Synthesis`, `Analysis`)
* `View/Edit` recipe details/configurations
* `Execute` recipes as jobs
## RECIPES BY ASSET TYPE
Recipes are organized by asset type, with each one supporting a specific stage of the model development workflow:
Supervised fine-tuning configurations for model training
Configurations for creating synthetic datasets
Configurations for running repeatable model evaluations
# JOBS
Source: https://docs.oumi.ai/guides/jobs
Track long-running jobs and their operations
When you start a long-running task, Oumi creates an operation that tracks its progress. The **Jobs** page lists current and past jobs for the project, including training, evaluation, synthesis, analysis, uploads, imports, exports, and deployments.
From the **Jobs** page:
* Select a job to view its operation details, progress, result, or error.
* Filter by `Type` or `Status`, or search by name, type, or status.
* Stop a pending or running job from its details.
# WELCOME TO OUMI
Source: https://docs.oumi.ai/guides/overview
Custom AI in hours not months
Oumi is a platform for building and deploying custom AI models tailored to your specific tasks. It significantly boosts the productivity of AI experts while empowering domain experts with less AI experience to train models effectively.
By streamlining the entire lifecycle, from [dataset creation](/guides/datasets/create) and [evaluation](/guides/evaluations) to [training](/guides/training) and [deployment](/guides/deployment), Oumi enables you to move from idea to production in hours, not months.
Sign up at [platform.oumi.com](https://platform.oumi.ai/) for free credits (no credit card required, bonus credits if using a work email).
Using natural language prompts with the Oumi Agent, you can create custom models in record time and run complex ML workflows while staying focused on outcomes rather than setup.
Check out a few examples of how you can use the Oumi Agent to perform powerful machine learning tasks:
Train a compact fraud detection model using a stronger model’s responses as guidance.
Evaluate a model on a customer support dataset to uncover common weaknesses and recurring failure patterns in its responses.
Fine-tune a model to automatically classify customer support questions by topic and urgency.
Analyze specific areas where your model is underperforming and generate targeted datasets for retraining.
Generate 500 question-answer pairs from a product documentation PDF for use in training or evaluation.
Expand your dataset by generating new samples that match the style and format of your existing examples.
## WHY CUSTOM MODELS
Frontier models (e.g., GPT, Claude, Gemini, Qwen, DeepSeek) are built to be general-purpose. However, this power comes with tradeoffs:
* They are often not accurate enough on your specific task
* They are slow and expensive at scale
* You are building on a commodity, versus developing your competitive advantage
Closed models add a set of transparency and control challenges:
* Quality can change without warning and impact your product
* Terms of use may change, impacting model availability for your use case
* Deployment options are constrained, limiting privacy/security control
If AI is core to your product, you should own your future. Control quality, cost, deployment, and iteration speed by building custom models optimized for your tasks.
Read more in the ["The Case for Specialized Intelligence"](https://oumi.ai/vision).
## HOW CUSTOM MODELS ARE TYPICALLY DEVELOPED
Developing a high-quality custom model is typically an iterative loop:
* **1. Evaluate:** Start by benchmarking existing models to establish baseline performance. This requires a reliable test set and robust evaluation methodology.
* **2. Create training set:** Analyze where the baseline model fails, then build or curate training data that targets those gaps.
* **3. Train:** Train a new model using the improved training set and with careful selection of hyperparameters and training strategy.
Then repeat steps 1 to 3 until you meet your quality goals, and finally deploy.
In practice, this process can take months and demands significant AI expertise for each model and task, even after assembling and integrating a patchwork of tools to support it.
## HOW OUMI DOES IT
Oumi follows the same fundamental development loop but automates all the steps while still giving you full flexibility and control.
| Development stage | Traditional custom model development | Oumi |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Evaluation** | Teams manually create datasets for testing and build out the entire evaluation process for measuring performance. | Automatically synthesizes comprehensive test sets and generates LLM-based evaluation judges from a simple natural language task description. |
| **Training set creation** | Engineers manually inspect numerous failure cases to identify where models fail, then manually curate training data to improve quality. | Automatically analyzes model failures and surfaces the failure modes. Then automatically generates targeted training data designed to address failure modes. |
| **Training** | Effective model training requires carefully selecting the right model family, size, and hyperparameters. | Automatically suggests the correct model family, size, and hyperparameters based on tradeoffs (e.g., quality vs. efficiency) and task type. |
| **Iteration speed** | Iteration cycles can take weeks or months due to manual data creation, experimentation, and infrastructure management. | Automates all steps in the model development loop, dramatically accelerating iteration cycles. |
| **Deployment** | Deployment pipelines must typically be built and managed separately from training workflows. | Provides integrated tooling to easily deploy and run trained models in production environments. |
By automating evaluation, training set creation, model tuning, and deployment, Oumi significantly accelerates the process of building high-quality custom models.
![Oumi loop]()
Bring your own data, or start with none at all. All that’s required is a task description and a target outcome. You can apply your own AI expertise at any level, or rely on Oumi to handle the end-to-end workflow and underlying steps for you.
While Oumi automates much of the workflow, you remain in full control. Every step is:
* **Transparent:** you can see exactly what actions will be taken
* **Reproducible:** all actions are recorded as reusable recipes
* **Flexible:** you can review and modify recipes before they are executed
## WHO OUMI IS DESIGNED FOR
AI has automated many workflows, but building high-quality machine learning models has remained a painfully manual effort. As the platform that automates AI development itself, Oumi is best for teams that want:
* Higher quality on critical tasks
* Lower inference cost at scale
* Lower latency for latency-sensitive applications
* Building models to deploy on devices
* Controllable deployment for better privacy and security
* Full transparency into AI model development to ensure auditability in regulated industries and beyond
* Full control and ownership over AI models when they are critical for business success
* Faster time to value, with a clear focus on high-impact outcomes rather than work that doesn’t drive meaningful differentiation
Owning your AI future has never been this easy. What will you build?
# MODEL PLAYGROUND
Source: https://docs.oumi.ai/guides/playground
The **Model Playground** puts several models side by side in their own chat panels and sends one prompt to all of them at once. It answers "which model is better for my task?"
You can compare a deployed fine-tuned model to a base model, an open-source model to a frontier model, or any combination you choose.
The Oumi Agent can tell you which of your deployments are available to compare and point you here, but it cannot drive the Playground for you. You open the page and run the comparison yourself.
***
## WHEN TO USE THE PLAYGROUND
Check out the Playground when the question is qualitative and you want an answer in seconds. The Playground can also help you answer questions about latency, token effiency, and cost on a per-request basis.
| Use the Playground | Use an [evaluation](/guides/evaluations) |
| ------------------------------------------------------ | --------------------------------------------- |
| "Does the fine-tune sound better than the base model?" | "Which model scores higher, and by how much?" |
| A handful of prompts you type by hand | A dataset of many examples |
| Reading answers yourself | Metrics, scores, or a judge |
| Instant, throwaway | Repeatable and recorded |
A Playground session is a good sanity check before you invest in a full evaluation. It is not a substitute for one: nothing you do here is scored, saved, or reproducible.
## OPEN THE PLAYGROUND
Select **Playground** in the project sidebar. The page opens empty, with a prompt to select models. In the Playground, you can chat with any of your existing [Deployments](/guides/deployment) or with an open-source base model.
If you have a deployment with an [external model](/guides/deployment/deploy-a-model#before-you-start), you will have the opportunity to directly compare a frontier lab's model to your own.
***
## ADD MODELS TO COMPARE
Click `Add Model...` to open a searchable picker with two groups:
* **Your deployments.** Every deployment in the project that has not been deleted. A deployment must be `Active` before you can chat with it. You can add one that is still starting up, but its panel stays disabled and shows the current status until it becomes active.
* **Oumi base models.** Untrained open-source models that Oumi serves directly, ready to chat with no deployment of your own. This group appears only where Oumi is serving base models for your project.
Each pick opens another panel, up to **four panels at once**. `Add Model...` is disabled once you reach four.
You can chat with more than one copy of the same deployment or base model. Two panels running the same deployment let you compare two system prompts or two temperature settings against each other. Repeat panels are numbered in the header, so the second copy reads `My Deployment #2`.
***
## SEND A PROMPT
Type in the input at the bottom of the page and press `Enter`, or click the arrow. The prompt goes to every ready panel at the same time, and each panel streams its own answer independently.
Panels keep separate conversations. Follow-up prompts also fan out to all of them, so each model sees only its own history.
While any panel is still generating:
* The send button becomes a **stop** button that stops every panel at once.
* Each streaming panel gets its own stop button in its header, to stop just that one.
Stopping keeps whatever text had already streamed in, including a partial response. Keep in mind that all of these conversations are ephemeral, so navigating away from the page will lose them. The "Playground" feature is designed for quick experiments
where you may want to compare performance or cost of different models or configurations.
Leaving the page while a response is still generating loses the in-flight answers. Your browser warns you first.
***
## SWITCH A PANEL'S MODEL
Click the model name in a panel header to swap that panel to a different model in place. This is the fastest way to ask "what would a different model have said?" without retyping the conversation.
When you switch:
* The conversation stays, and the new model picks it up from there.
* The panel's system prompt follows it across.
* The panel's sampling parameters reset, because the new model may accept a different set.
* Your next message is marked with a `Model changed to ...` divider, so the transcript shows exactly where the answers stopped coming from the old model.
## WHAT'S NEXT
Learn how to use system prompts and sampling parameters to tune the behavior of the models.
# METRICS AND MANAGEMENT
Source: https://docs.oumi.ai/guides/playground/metrics-and-management
Comparing model performance and managing your conversations
## OVERVIEW
The Playground provides a few metrics and management tools to help you monitor and control your conversations.
***
## READ THE METRICS
Once a panel has answered, a row of chips appears beneath its conversation:
| Chip | Meaning |
| --------- | ------------------------------------------------------------------------------------ |
| `TTFT` | Time to first token: how long the **latest** response took to start streaming |
| `Latency` | How long the **latest** response took to finish generating |
| `In` | Input tokens sent to the model **so far this session**, prompts and history together |
| `Out` | Output tokens the model has generated **so far this session** |
Timings describe the most recent turn only. Token counts accumulate across the whole session, so they climb as a conversation grows.
These numbers come from a single interactive request, so treat them as a rough feel for responsiveness rather than a benchmark. Both are sensitive to a cold deployment, prompt length, and current load.
***
## MANAGE THE CONVERSATIONS
| Control | Effect |
| ------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `Clear Chat` | Empties every conversation and leaves the panels open |
| `Close All` | Closes every panel and starts over |
| `Retry` | Appears on a panel whose turn failed. Re-runs that panel's last message, leaving the other panels untouched |
| The `x` in a panel header | Closes that one panel and discards its conversation |
***
## SHARE A MODEL COMPARISON
The models and deployments you have open are held in the page URL, so refreshing the page restores the same set of panels. A link link sent to a teammate will behave the same way,
and will allow you to share a set of models to compare. The URL does not track state about current conversations or settings.
Conversations, [system prompts](/guides/playground/system-prompts-and-parameters#set-a-system-prompt),
and [sampling parameters](/guides/playground/system-prompts-and-parameters#tune-sampling-parameters) are not saved and do not travel with the link.
# MODEL PARAMETERS AND PROMPTS
Source: https://docs.oumi.ai/guides/playground/system-prompts-and-parameters
System prompts and model parameters to customize your model's behavior
## OVERVIEW
You may want to run a quick experiment to determine which system prompt or model parameters are best for your use-case. The Playground allows you to adjust
these independently for each panel, or to apply the same settings to all panels at once.
***
## SET A SYSTEM PROMPT
Click the gear icon in any panel header to open **System Prompts**. The dialog gives every open model its own column and its own text box, so each panel can carry a different instruction.
* Leave a box empty to send no system message at all.
* Click `Apply to all` on a column to copy that column's text into every other column.
* Click `Update` to commit. Closing the dialog any other way discards your edits.
System prompts apply to every request the panel makes from then on, including messages already in the conversation history.
***
## TUNE SAMPLING PARAMETERS
Click the sliders icon in a panel header to open **Model Parameters** for that panel alone.
| Parameter | What it does | Accepted range |
| ------------- | --------------------------------------------------------------------------- | -------------------------------------------------- |
| `Temperature` | Higher values make output more random | 0 to 2, or 0 to 1 for Anthropic-backed deployments |
| `Top P` | Sample from the smallest set of tokens whose probability sums to this value | Greater than 0, at most 1 |
| `Max Tokens` | Cap on the length of the response | Whole number, 1 or more |
| `Seed` | Fixes sampling so a repeated prompt returns the same answer | Whole number |
| `Top K` | Sample only from the K most likely tokens | Whole number. Use `-1` to disable |
Only the parameters the panel's model actually accepts are shown, so the set differs by what sits behind the panel:
| Backing the panel | Temperature | Top P | Max Tokens | Seed | Top K |
| -------------------------------------------- | ----------- | ----- | ---------- | ---- | ----- |
| Oumi base models and your own trained models | Yes | Yes | Yes | Yes | Yes |
| Deployments backed by OpenAI | Yes | Yes | Yes | Yes | No |
| Deployments backed by Anthropic | Yes | Yes | Yes | No | Yes |
Leave a field blank to omit it from the request entirely, in which case the model's own default applies. Values outside the accepted range are rejected before the request is sent.
Click `Update` to apply the parameters to this panel, or `Apply to all` to copy them to every panel. When you apply to all, any parameter a panel cannot use is dropped and any value outside that panel's range is pulled to its nearest limit, so a comparison never fails because one model is stricter than another.
# OUMI QUICKSTART
Source: https://docs.oumi.ai/guides/quickstart
Learn how to quickly create your first custom AI model
## OVERVIEW
This quickstart demonstrates how to build a custom model that can outperform a leading large model in just a few steps.
You'll be using:
* **Large model (judge and large model benchmarking):** GPT 5.2
* **Small model (for fine-tuning):** Qwen/Qwen2.5-3B-Instruct
By the end, you’ll have a fine-tuned small model that outperforms the larger model, validated through Oumi’s evaluations, which you’ll set up as part of the workflow.
## TASK DEFINITION
You will develop an AI model for a bank that classifies customer support queries by intent, enabling accurate routing in a banking context.
## WORKFLOW STEPS
Oumi provides a fully automated, end-to-end workflow. For this use case, the process includes:
* Uploading your datasets and having Oumi automatically analyze them for potential issues
* Defining evaluators (judges) to assess accuracy and ensure proper output formatting
* Benchmarking both the large and small models on a test dataset to establish performance baselines
* Fine-tuning the small model using your training data to create a custom version
* Re-run the evaluations to see your custom model outperform the larger model’s baseline
You can complete this quickstart using the Oumi Agent or just the platform UI.
Start by creating a new project in your workspace:
1. Click on `New Project`.
2. Give your project a `Project Name`.
3. Provide a description in `Project Context`. You can also invite your team members to your project by clicking the `Invite team members` button and selecting their usernames.
4. Click `Create Project`.
***
You can use the following datasets for this quickstart. Download them to your local machine:
* [banking77\_test\_basic.jsonl](https://huggingface.co/datasets/oumi-ai/banking77-oumi-quickstart/resolve/main/banking77_test_basic.jsonl?download=true)
* [banking77\_train\_basic.jsonl](https://huggingface.co/datasets/oumi-ai/banking77-oumi-quickstart/resolve/main/banking77_train_basic.jsonl?download=true)
* [banking77\_val\_basic.jsonl](https://huggingface.co/datasets/oumi-ai/banking77-oumi-quickstart/resolve/main/banking77_val_basic.jsonl?download=true)
Oumi requires that datasets follow a specific format. Please see [Datasets](/guides/datasets) to learn more.
From your project's **Overview** page:
1. Click the `Create` button and select `Dataset` from the menu.
2. Select `Upload a Dataset` (to the right of **Create Dataset**)
3. Provide a `Dataset Name` for your dataset and select the JSONL files you downloaded in the previous step.
4. Click `Create Dataset`. Oumi will start uploading your datasets and automatically run a series of quality checks.
***
Next, define your evaluators for measuring baseline model performance. The Oumi Agent makes it easy to create custom evaluators for any metric using natural language prompts.
You'll need two evaluators for this example: one to measure accuracy against ground truth, the other to validate that outputs are correctly formatted (i.e., an integer within the valid class range).
Please see [Evaluations](/guides/evaluations/results) to learn more about how to assess your model’s quality using Oumi.
From the **Agent** pane on the right-hand side of your screen:
1. Give the Oumi Agent the following prompt:
```prompt theme={null}
You are building a model to classify and route customer support queries for a bank.
The model should determine the customer’s intent based on a provided conversation. Start by creating baselines for performance benchmarking and fine-tuning.
First, evaluate a strong model on the uploaded test dataset, then evaluate a small language model on the test dataset. Define two custom judges for the evaluations: one to determine whether the output is correct (measuring accuracy using the ground truth labels stored in the dataset’s metadata fields as `label` and `label_name`), the other to determine whether the output is valid (is an integer between 1 and the number of classes).
To make the evaluations run faster, do not create failure modes.
```
2. The Agent will analyze your existing project assets and guide you through the steps for creating your model baselines and evaluations. Select `GPT 5.2` as your `strong model`, `Qwen/Qwen2.5-3B-Instruct` as your `small model`, and `GPT 5.2` as your `judge model`.
3. Once the Agent finishes configuring your evaluation jobs, click `Run It` to kick off each evaluation. Oumi will run the two evaluations in parallel.
Once your evaluations jobs finish, review the results. Your strong model should outperform your base model. Oumi enables you to get that same performance in your custom small model. You'll do this by fine-tuning your small model on the training dataset.
You can also review your in-depth evaluation results side-by-side. From the **Evaluations** page, click on `Compare` and select your two evaluations.
After reviewing the baselines, you can fine-tune your custom small model to improve accuracy and close the gap.
From the Agent-provided options:
1. Select `Fine-tune Qwen 2.5-3B`.
2. Review the training guidance and plan as the Agent automatically sets up your training configuration, select `SFT with LoRA` as your training method and parameter strategy.
3. Click `start training` and `approve` to kick off the training job.
4. Once the training job model completes, run an evaluation against your new custom model to ensure that it beats your previous baselines. Click `Run Evaluation` to kick off the job.
5. Click `Done for now` to wrap up the workflow.
Your custom, high-quality AI model tailored to your classification task is now ready for use. The entire process should take only a matter of hours, not months.
Start by creating a new project in your workspace:
1. Click on `New Project`.
2. Give your project a `Project Name`.
3. Provide a description in `Project Context`. You can also invite your team members to your project by clicking the `Invite team members` button and selecting their usernames.
4. Click `Create Project`.
***
You can use the following datasets for this quickstart. Download them to your local machine:
* [banking77\_test\_basic.jsonl](https://huggingface.co/datasets/oumi-ai/banking77-oumi-quickstart/resolve/main/banking77_test_basic.jsonl?download=true)
* [banking77\_train\_basic.jsonl](https://huggingface.co/datasets/oumi-ai/banking77-oumi-quickstart/resolve/main/banking77_train_basic.jsonl?download=true)
* [banking77\_val\_basic.jsonl](https://huggingface.co/datasets/oumi-ai/banking77-oumi-quickstart/resolve/main/banking77_val_basic.jsonl?download=true)
Oumi requires that datasets follow a specific format. Please see [Datasets](/guides/datasets) to learn more.
From your project's **Overview** page:
1. Click the `Create` button and select `Dataset` from the menu.
2. Select `Upload a Dataset` (to the right of **Create Dataset**)
3. Provide a `Dataset Name` for your dataset and select the JSONL files you downloaded in the previous step.
4. Click `Create Dataset`. Oumi will start uploading your datasets and automatically run a series of quality checks.
***
Next, define your evaluators for measuring baseline model performance. You'll need two evaluators for this example: one to measure accuracy against ground truth, the other to validate that outputs are correctly formatted (i.e., an integer within the valid class range).
Please see [Evaluations](/guides/evaluations/results) to learn how to assess your model’s quality with Oumi.
Go to your **Evaluators** page and click `Create Evaluator` on the top right-hand side to load the Builder.
On the `CONFIGURE` tab:
1. Leave the **Judgments** default values and enter in the following for the **Prompt**:
```prompt theme={null}
You are a classification accuracy judge. Your task is to determine whether the assistant's predicted intent label matches the ground truth label for a banking customer support intent classification task.
Inputs:
Consider the field 'conversation' (a multi-turn conversation between a user and an assistant) along with the provided data fields 'Ground Truth Label (Integer)' and 'Ground Truth Label Name' to make your assessment.
Decision rule:
If the assistant's response contains the correct ground truth label integer OR the correct ground truth label name (or a semantically equivalent version of the label name), respond with 'Yes'. Otherwise, respond with 'No'.
Evaluation Criteria:
1. Exact Integer Match: If the assistant's response contains the ground truth label integer (e.g., if the ground truth is 42, the response contains '42' as a label prediction), this counts as a match.
2. Exact Label Name Match: If the assistant's response contains the ground truth label name exactly as provided (e.g., 'card_arrival'), this counts as a match.
3. Semantic Label Name Equivalence: Minor formatting variations of the label name are acceptable matches. For example, 'card_arrival' matches 'card arrival', 'Card Arrival', 'card-arrival', or 'Card_Arrival'. Underscores, hyphens, spaces, and capitalization differences should be treated as equivalent. However, the semantic meaning must be preserved: 'card_payment' does NOT match 'card_arrival'.
4. Presence in Longer Responses: If the assistant's response contains additional text beyond just the label (e.g., explanations, reasoning, or formatting), the judge should still check whether the correct label integer or label name appears within the response. The label must be clearly identifiable as the assistant's prediction, not merely mentioned in passing or as part of a different context.
5. Strictness: If the response is ambiguous, unclear, contains multiple conflicting labels, or does not clearly indicate the correct label, default to 'No'.
Corner cases:
1. Multiple Labels in Response: If the assistant outputs multiple labels and one of them matches the ground truth, judge 'Yes' only if the correct label is clearly the final or primary prediction. If it is ambiguous which label is the prediction, judge 'No'.
2. No Label Present: If the assistant's response does not contain any recognizable label integer or label name, judge 'No'.
3. Partial Matches: A partial match of the label name is not sufficient. For example, if the ground truth is 'card_arrival', a response of 'card' alone does not match. The full label name or its semantic equivalent must be present.
4. Irrelevant or Off-Topic Responses: If the assistant's response does not attempt to classify the intent at all, judge 'No'.
5. Numerical Ambiguity: If the ground truth label integer appears in the response but in a context unrelated to the classification (e.g., as part of a date or other number), this does not count as a match. The number must be clearly presented as the predicted label.
```
2. For `Model`, select `GPT 5.2`. Leave everything else with their default values.
3. Click the `Save` icon on the top right-hand corner, and give your evaluator a name.
You will now create your second evaluator for output validity. Click `Create Evaluator` on the top right-hand side to load the Builder again.
On the `CONFIGURE` tab:
1. Leave the **Judgments** default values and enter in the following for the **Prompt**:
```prompt theme={null}
You are an output validity judge. Your task is to determine whether the assistant's response contains a valid integer between 0 and 76 (inclusive), representing one of the 77 Banking77 intent classes.
Inputs:
Only consider the field 'conversation' (a multi-turn conversation between a user and an assistant) to make your assessment.
Decision rule:
Examine the assistant's response for the presence of at least one integer in the range [0, 76]. If such an integer is present, respond with 'Yes'. Otherwise, respond with 'No'.
Evaluation criteria:
1. Valid Integer Presence: The assistant's response must contain at least one integer value that falls within the inclusive range of 0 to 76. The integer must be a whole number (not a decimal or fraction).
2. Permissive Extraction: If the response contains extra text, explanations, or formatting but still includes a clear integer in the range 0-76, judge 'Yes'. The integer does not need to be the only content in the response.
3. Multiple Numbers: If the response contains multiple numbers, check whether any one of them is a valid integer in the range 0-76. If at least one qualifies, judge 'Yes'.
4. Label Names Without Integers: If the response contains only a textual label name (e.g., 'card_payment_fee_charged') but no integer, judge 'No'. The task requires an integer output.
5. Strict Range Enforcement: Integers outside the range 0-76 (e.g., -1, 77, 100) do not count as valid. If the only integers present are outside this range, judge 'No'.
6. Number Format: Accept integers written as digits (e.g., '42') or as numeric words (e.g., 'forty-two'). Both count as valid if they fall in range. However, decimal numbers like '3.5' do not count as valid integers; only their integer part should be considered if it appears separately.
Corner cases:
1. Empty Response: If the assistant's response is empty or contains only whitespace, judge 'No'.
2. Non-Numeric Response: If the response contains no numbers at all (only text, symbols, or punctuation), judge 'No'.
3. Out-of-Range Integers Only: If all integers in the response are outside 0-76, judge 'No'.
4. Ambiguous Numeric Content: If the response contains numbers embedded in non-numeric contexts (e.g., dates, URLs, or identifiers like 'GPT-4'), use judgment to determine whether any standalone integer in range 0-76 is clearly intended as a class prediction. If uncertain, be permissive and judge 'Yes' if a standalone integer in range is present.
5. Negative Numbers: Negative integers are outside the valid range and should be judged 'No' unless a valid in-range integer also appears.
6. Refusal or Error Messages: If the assistant refuses to answer or returns an error message without any valid integer, judge 'No'.
```
2. For `Model`, select `GPT 5.2`. Leave everything else with their default values.
3. Click the `Save` icon on the top right-hand corner, and give your evaluator a name.
Next, define two evaluations: one for the strong model, the other for the smaller model using your uploaded datasets.
Go to the **Evaluations** page and click `Run Evaluation` on the top right-hand side to load the Builder.
In the Builder:
1. Select `Create a Judge-based Evaluation`.
On the `CONFIGURE` tab:
2. Select `GPT 5.2` for your `Model`. Under **Evaluators**, select the accuracy and output validity evaluators you created in the previous step.
3. For `Dataset`, select the `banking77_test_basic.jsonl` dataset you previously uploaded.
4. Leave all others at their defaults and click `Execute` on the top right-hand side.
5. Give your evaluation a name. This will automatically create a new recipe and run the evaluation. You can view your new evaluation results on your `Evaluations` page when the job completes.
Now, create your second evaluation for the small model. Click `Run Evaluation` on the top right-hand side to load the Builder again.
In the Builder:
1. Select `Create a Judge-based Evaluation`.
On the `CONFIGURE` tab:
2. Select `Qwen/Qwen2.5-3B-Instruct` for your `Model`. Under **Evaluators**, select the accuracy and output validity evaluators you created in the previous step.
3. For `Dataset`, select the `banking77_test_basic.jsonl` dataset you previously uploaded.
4. Leave all others at their defaults and click `Execute` on the top right-hand side.
5. Give your evaluation a name. This will automatically create a new recipe and run the evaluation. You can view your new evaluation results on your `Evaluations` page when the job completes.
Once your evaluations jobs finish, review the results. Your strong model should outperform your small model. Oumi enables you to get that same performance in your custom small model.
Next, fine-tune your small model on the training dataset to close the accuracy gap. Because you now know what your improvement metrics are after reviewing your evaluations, it's easy to compare your custom model to make sure that it outperforms the others.
1. From the **Models** page, click `Train New Model` on the top right-hand side, followed by `Supervised Fine-Tuning`.
2. On the **CONFIGURE** tab, set the following values (you can leave the other values as is):
* `Training Method` should be `Supervised Fine-Tuning`
* `Base Model` should be `Qwen/Qwen2.5-3B-Instruct`
* `Training Dataset` should be `banking77_train_basic`
3. Click `Execute` and provide a name for your model. Oumi will automatically save the recipe under this name as well.
4. Click `Run Job` to kick off the fine-tuning job.
Once the job completes, create a final evaluation on your custom small model:
1. Click on your newly-trained model from the **Evaluations** page.
2. Click on `Evaluate` in the top navigation. In the **CONFIGURE** tab:
* Under `Model`, make sure your newly-trained custom small model is selected.
* Under `Select Evaluators`, select the `Accuracy` and `Output Validity` judges.
* For the `Dataset`, select the `banking77_test_basic` dataset. You can leave all other optional items as is.
3. Click `Execute`, give your new evaluation a name, and click `Run Evaluation` to kick off the job.
Once the job finishes, review your evaluation to verify that your new model is the best-performing across the board.
From the **Evaluations** page:
1. Click `Compare` in the top navigation.
2. Select the previous large model evaluation and the new custom small model evaluation.
3. Review the evaluators' scores across the two evaluations. Your new custom small model should outperform the large model.
Your custom, high-quality AI model tailored to your classification task is now ready for use. The entire process should take only a matter of hours, not months.
From this point, you can [deploy your new model](/guides/deployment/deploy-a-model) as a live endpoint, [export it](/guides/deployment/self-hosting) to run on your own infrastructure, or continue improving its performance.
## NEXT STEPS
You’ve now built a custom model with Oumi, leveraging the strength of a large model while gaining the efficiency and control of a smaller one, without complex or costly development. From here, you’re ready to iterate, scale, and apply this example to your own use cases.
Explore the [Oumi Workflow](/guides/intro) and dive deeper into the available options and configurations for building custom AI models in Oumi.
# SYNTHETIC DATA GENERATION
Source: https://docs.oumi.ai/guides/synthesis
Generate high quality synthetic datasets at scale
Data underpins both model [training](/guides/training) and [evaluation](/guides/evaluations), making data quality essential to model performance. Oumi makes it easy to create high-quality training and test data, whether from scratch, by extending existing datasets, or using unlabeled data you already have.
With a simple prompt, the [Oumi Agent](/guides/intro/agent) can generate complete datasets tailored to your task (e.g, “Build a test set for a model that classifies customer support tickets by urgency”). You still retain full control, with the ability to inspect and refine any part of the data generation workflow before it runs.
The Agent turns synthetic data generation into a repeatable, prompt-driven process. It translates your clear task definition into a [structured data recipe](/guides/synthesis/recipes), generates data at scale, and iteratively improves outputs based on your objectives, all within a single loop.
What once took weeks of manual effort and costly labeling can now be done in hours. By automating everything from schema design to quality validation, Oumi removes the operational burden of synthetic data, making it faster to prototype, expand, and evolve datasets without increasing headcount or vendor spend.
## WHAT YOU CAN SYNTHESIZE
Oumi incorporates data synthesis as an iterative, repeatable part of your machine learning workflow. You can rapidly prototype datasets, expand small or imbalanced data, and evolve training data alongside your models.
Some examples of what you can build with Oumi's data synthesis include:
* Question-answer datasets for training chatbots
* Instruction-following datasets with varied complexity levels
* Domain-specific training data (legal, medical, technical)
* Conversation datasets with different personas or styles
* Data augmentation to expand existing small datasets
With Oumi's data synthesis capabilities, teams can rapidly prototype, iterate, and scale training datasets while maintaining control over structure, diversity, and quality. By shifting data creation from manual effort to rule-driven generation, you can accelerate model development and unlock use cases that would otherwise be limited by data availability.
## WHAT'S NEXT
Learn how Oumi data synthesis works.
Find out what goes inside a data synthesis recipe.
# HOW IT WORKS
Source: https://docs.oumi.ai/guides/synthesis/how-it-works
Creating synthetic datasets with Oumi
Oumi makes it easy to synthesize data for a wide range of machine learning use cases, whether you’re training a model from scratch, fine-tuning an existing one, or improving performance through targeted evaluation criteria.
***
## DATA SYNTHESIS METHODS
Oumi’s data synthesis capabilities are built directly into the platform, enabling data generation to integrate seamlessly into your end-to-end workflow.
You can synthesize data using the following methods via the Oumi UI:
* [Using a data synthesis recipe](#using-a-data-synthesis-recipe)
* [Generate completions](#generate-completions)
* [Generate from failure modes](#generate-from-failure-modes)
* [Generate from scratch](#generate-from-scratch)
These options provide flexible, scalable ways to generate high-quality datasets aligned with your modeling goals, no matter where you are in the workflow.
### USING A DATA SYNTHESIS RECIPE
Data synthesis recipes provide a reusable, structured approach to generating high-quality synthetic datasets. Designed as configurable blueprints, these recipes define how data should be created for specific use cases or modeling needs, giving you a fast and consistent starting point.
To learn more, please see [Data synthesis recipes](/guides/synthesis/recipes).
### GENERATE COMPLETIONS
Generate completions lets you create training data by supplying inputs and using a model to produce corresponding outputs. You can configure the process by selecting the model and any existing datasets to use, or by directly inserting the JSON configuration in the Oumi Builder.
To generate completions:
1. Go to the **Datasets** page and click on `Create Dataset`.
2. In the Builder window, click on `Generate Completions`.
3. Set your `Model` and `Source Dataset`, and specify any optional parameters. Alternatively, copy/paste your JSON configuration into the text area on the right-hand side.
4. Click `Execute` to start the job. Once processing is complete, the new dataset will appear on your **Datasets** page.
### GENERATE FROM FAILURE MODES
Generating synthetic datasets through failure modes allows you to create targeted training data to fix known model errors and improve overall performance.
To generate a dataset from failure mode:
1. Select an evaluation on the **Evaluations** page.
2. Click `Review Failure Modes`.
3. On the **Failure Modes** page, configure which evaluators' items to include when generating datasets through failure mode.
4. Click `Generate Dataset...`
5. Give you synthesis job a **Display Name** and click `Run Synthesis`.
### GENERATE FROM SCRATCH
Generating synthetic datasets from scratch allows you to build synthetic datasets with full control over attiebutes, sources, and output format.
To generate a dataset from scratch:
1. On the **Datasets** page, click the `Create Dataset` button.
2. Click `Generate from Scratch`.
3. Configure your synthesis configurations on the `INPUTS` tab.
4. Click `Execute`
5. Give your job a **Display Name** and click `Generate Dataset`.
You can check the synthesis job on the [Jobs](/guides/jobs) page. Your newly synthesized dataset will be available on your **Datasets** page when the job completes.
# NEXT STEPS
Source: https://docs.oumi.ai/guides/synthesis/next-steps
Getting the most out of Oumi's data synthesis capabilities
## TROUBLESHOOTING & TIPS
As you work with data synthesis, you may occasionally want to fine-tune configurations, improve performance, or refine output quality. The following tips capture best practices learned from real-world usage, helping you diagnose issues quickly, optimize generation, and ensure your synthetic data is reliable and ready for training.
* **Start small:** Begin with a small number of samples to test your configuration
* **Use examples:** Provide good examples for better generation quality
* **Postprocess outputs:** Use postprocessing to clean and format generated text. Oumi's dataset analysis can help to automatically analyze datasets and remove low quality dataset samples
* **Validate results:** Review generated samples before using for training
* **Version control:** Keep your synthesis configs in version control
## COMMON ISSUES
Even with well-defined configurations, you may occasionally encounter issues during data synthesis. The following are some of the most common situations, along with guidance on how to identify and address them so you can continue generating high-quality datasets efficiently.
* **Empty results:** Check that your instruction messages are well-formed and you have proper API access.
* **Slow generation:** Increase `num_workers` or lower `politeness_policy` to improve throughput.
* **Out of memory:** Use a smaller model or reduce `max_new_tokens` in generation config.
* **Validation errors:** Ensure all attribute IDs are unique and required fields are not empty.
You can also automatically detect and remove low-quality data to maintain dataset integrity using Oumi’s in-built analysis and curation tools.
# DATA SYNTHESIS RECIPES
Source: https://docs.oumi.ai/guides/synthesis/recipes
Reusable recipes for generating structured synthetic datasets.
Data synthesis recipes provide a quick starting point for creating datasets tailored to specific configurations or use cases. Each recipe defines how to generate a custom dataset, making it easy to reuse and adapt across projects.
***
## WHAT IS A DATA SYNTHESIS RECIPE?
A data synthesis recipe is effectively a synthesis configuration for generating data. Rather than writing static examples by hand, you define structured rules that describe:
* What should vary across samples
* How examples should be generated
* Which model should perform generation
* How many samples to create
You can think of a recipe as a data generation blueprint. Attributes define the ingredients, templates define the instructions, and the synthesis engine produces consistent, scalable outputs by systematically combining them.
This approach allows you to:
* Generate large, diverse datasets from compact specifications
* Systematically target failure modes identified during evaluation
* Maintain consistency across generated examples
* Iterate quickly by adjusting attributes instead of rewriting data
By separating structure (attributes and templates) from scale (number of samples), recipes make synthetic data generation controllable, reproducible, and aligned with measurable training goals.
***
## ACCESSING & EXECUTING DATA SYNTHESIS RECIPES
You can access and manage all your data synthesis recipes in one location:
1. Go to the **Recipes** page.
2. Filter on data synthesis recipes by selecting `Synthesis` in the `Type` drop-down menu.
3. Click the recipe name to open it in the Builder.
4. Click the `Execute` button to run the recipe and start the synthesis job.
Your newly synthesized dataset will appear under datasets once the job is finished.
# TRAIN MODEL
Source: https://docs.oumi.ai/guides/training
Fine-tune your model to improve its task-specific performance.
The Oumi Agent streamlines fine-tuning by analyzing your task definition and datasets, then recommending an appropriate model family, model size, and hyperparameter configuration. Combined with prebuilt training recipes, you start from a strong baseline rather than building configuration from scratch.
What typically takes days of trial-and-error tuning can be reduced to a single guided workflow. The Oumi Agent eliminates most of the manual work that slows teams down, from model selection to hyperparameter setup, so engineering effort goes toward improving your model rather than configuring it. Because training runs are fully reproducible and tied to your evaluation results, each iteration is intentional and measurable rather than exploratory.
Oumi supports supervised fine-tuning (SFT) in both full-weight fine-tuning (FFT) and parameter-efficient fine-tuning (PEFT) configurations.
Reinforcement learning (RL) support is coming soon. To request early access, please contact us at [https://www.oumi.ai/contact](https://www.oumi.ai/contact).
***
## WHAT IS FINE-TUNING?
Fine-tuning builds on a model that has already been trained, adapting it to perform better on specific tasks. Rather than starting from randomly initialized weights, you begin with a model that has learned general language patterns from large-scale data and refine it for your particular use case.
In a typical Oumi workflow, training is driven by evidence. You start with evaluation and failure mode analysis, identify targeted areas for improvement, curate or synthesize focused data, and then fine-tune. Each training run is intentional and designed to produce measurable, incremental performance gains.
This approach allows you to adapt the model to:
* A specific domain
* A task (e.g., classification, instruction following)
* A style or response format
* Known failure modes identified during evaluation
Rather than teaching the model language from scratch, you are guiding and improving its behavior. Oumi provides all the tools and workflows to support this process.
### SUPERVISED FINE-TUNING (SFT)
Oumi supports supervised SFT, the most common approach for adapting LLMs. In SFT, the model learns to make desired responses more likely given specific prompts.
For example:
Prompt: *“What is the capital of France?”*\
Target response: *“Paris.”*
During training, the model adjusts its parameters to increase the probability of generating the target response when given that prompt.
SFT works best when your training dataset is:
* High-quality
* Cleanly formatted
* Closely aligned with your target use case
### FULL-WEIGHT VS. PARAMETER-EFFICIENT TRAINING
You can use Oumi to fine-tune a model using either full-weight fine-tuning or parameter-efficient training.
#### FULL-WEIGHT FINE-TUNING (FFT)
FFT updates the model’s parameters directly, often all of them. This approach offers maximum flexibility and capacity for change but requires more compute and memory.
Choose FFT when:
* You need significant behavioral shifts
* You have sufficient compute resources
* You want maximal adaptation
#### PARAMETER-EFFICIENT FINE-TUNING (PEFT)
PEFT freezes the original model weights and instead trains a smaller set of additional parameters that modify the model’s behavior.
This approach:
* Uses less compute
* Is faster to train
* Produces smaller artifacts
A common method for PEFT is low-rank adaptation (LoRA), which introduces lightweight parameter updates that represent the difference from the frozen base model. PEFT is ideal for rapid iteration and resource-constrained environments.
***
## TRAINING WORKFLOW
Once datasets are prepared, your Oumi training workflow will typically entail:
1. Configuring a training run (choosing a base model, training method, and datasets)
2. Launching the run and monitor progress
3. Evaluating the resulting model
4. Diagnosing failure modes and iterating
After iterating through the workflow and reaching satisfactory model performance, you can [deploy the trained model](/guides/deployment/deploy-a-model) as a live endpoint, or [export it](/guides/deployment/self-hosting) to serve on your own infrastructure.
Bear in mind that training is part of a continuous improvement loop driven by evaluation and targeted data refinement, rather than a one-time event.
# RUNNING TRAINING
Source: https://docs.oumi.ai/guides/training/launch-jobs
Easily launch fine-tuning jobs
Oumi streamlines model fine-tuning and performance iteration by providing multiple training methods and flexible configuration options. This allows you to experiment efficiently while retaining full control over your setup.
## HOW TO RUN TRAINING JOBS
You can either select`Supervised Fine-Tuning` to train a model using labeled examples, or `On-policy Distillation` to train a student model using a teacher model for knowledge distillation.
***
### SUPERVISED FINE-TUNING (SFT)
To start an SFT job, initiate a training run from the **Models** page.
1. Click on `Train New Model`.
2. In the Builder, select `Supervised Fine-Tuning`.
3. Select the base model to fine-tune. Oumi offers a broad range of commonly used models.
4. Choose your training dataset, and optionally select validation and test datasets. You can use uploaded datasets, synthesized data, or merged datasets.
5. Select a training method. Oumi supports full fine-tuning (FFT) and parameter-efficient fine-tuning (PEFT), including LoRA.
6. Adjust advanced hyperparameters (e.g., `maximum steps`, `learning rate`) if needed.
7. Review your configuration, (optionally) save it as a reusable recipe, and launch the training job.
***
### ON-POLICY DISTILLATION
To start an on-policy distillation job, initiate a training run from the **Models** page.
1. Click on `Train New Model`.
2. In the Builder, select `On-Policy Distillation`.
3. Leave `Training Method` on `On-Policy Distillation`.
4. Choose your `Base Model` and `Teacher Model`.
5. Select your `Training Dataset`.
6. Configure advanced settings (e.g., `Training Settings`, `Distillation Settings`, `Parameter-Efficient Settings`) if needed.
Please see [On-Policy Distillation](/guides/training/on-policy-distillation) for more information regarding configuration optionss and settings.
## CHECKING JOB STATUS
After a training job launches, it will appear on the [Jobs](/guides/jobs) page with a status of `Running`.
When training completes, you can access your model from the **Model** page.
# MODEL SELECTION
Source: https://docs.oumi.ai/guides/training/model-selection
Choosing the right base model for performance and efficiency
Oumi provides a variety of base models for fine-tuning, and the list of supported models is always evolving as exciting new models are released.
The base model you select is the most important factor determining your fine-tuned model's capabilities, latency, and token-efficiency. Each base model offers unique strengths; the right base model for your needs will depend primarily on the nature of your use case and infrastructure requirements.
A few of the most important factors are:
* Task complexity
* Latency
* Cost
* Language support
* Tool use and agentic capabilities
You should weigh these factors carefully to select a base model that delivers the right balance of performance, efficiency, and cost for your specific application.
***
## KEY CONSIDERATIONS
Selecting the right base model requires balancing performance, cost, and capability across several technical dimensions; each of the considerations below can meaningfully impact your fine-tuning results and production deployment.
### MODEL SIZE AND LATENCY
Smaller models (≤4B parameters) offer the fastest inference speed and can be an order of magnitude cheaper to deploy at scale. Larger models (>8B) provide stronger reasoning and instruction-following capacity, but come with higher latency and cost. For latency-sensitive applications or on-device deployments, start with one of the most compact models, and scale up only if the resulting quality doesn't meet your needs.
### ARCHITECTURE: DENSE VS MOE
Most models released since 2023 use a "dense" architecture, meaning that all parameters are activated when processing each token of each input. An alternative to dense models is the "Mixture-of-Experts" (MoE) architecture, which routes each token to a subset of "expert" layers. This can improve the capacity of a model with lower inference cost. However, MoE models can be temperamental to fine-tune because expert routing can become unbalanced during training, with small or narrow datasets not adequately updating all experts. If you're new to fine-tuning or working with limited data, dense models tend to offer more predictable results.
## TASK SCOPE
Small models work well for narrow tasks with a restricted output space (e.g., classification, entity extraction, routing). They are also cheaper and faster than larger models. In internal testing, we have found that SmolLM2 models punch well above their weight across a number of classification tasks. Llama 3.\* series are also excellent choices for classification.
Reserve larger models for open-ended generation, problem-solving, and tasks with hard but varying constraints on response format.
## TASK COMPLEXITY AND REASONING
Some models are trained to natively support "reasoning-style" generation, which consistently outperforms non-reasoning generation on complex technical tasks requiring multiple steps. But be aware that reasoning models produce many more tokens per prompt (incurring additional latency and cost). If your tasks are consistently complex, consider a reasoning model. If complexity varies, the Qwen3 series offers "hybrid" reasoning that can adapt its depth based on the prompt.
## TOOL USE
If your use case involves RAG, web search, or generation of API calls to internal endpoints, choose a model trained with native tool-use capabilities. Qwen3, gpt-oss-20b, and the Llama 3.x/4 instruct variants are post-trained with extensive tool-use data.
## CONTEXT LENGTH
If your tasks require the model to process large blocks of text (e.g., RAG with large documents, summarization), choose a model that supports a large training context window. Qwen3 models support training with inputs up to 32k tokens. Training with smaller models like SmolLM is limited to 2k or 4k tokens.
## LANGUAGE SUPPORT
Different model families prioritize different languages. Gemma 3 supports approximately 140 languages. Qwen3 offers excellent multilingual capabilities with support for 119 languages, and is the clear choice if support for Asian languages (especially Chinese) is important for your use case. SmolLM2 excels in English but has limited support for other languages. Llama 4 has much more extensive multilingual support than Llama 3.
## DATA PROVENANCE
All supported models are open-weight with publicly available model cards. SmolLM2 was trained entirely with publicly-available data. Current trends in model development rely heavily upon synthetic data, which makes up a substantial fraction of the training datamixes of all models we support. If training data transparency or specific licensing terms are important for your deployment, review the model card for your chosen base model before starting training.
***
## AVAILABLE MODELS (FEB 2026)
The following models are currently supported for fine-tuning in Oumi, organized by size to help you quickly identify the right balance of capability, latency, and cost for your use case.
### COMPACT MODELS (\<2B PARAMETERS)
Best for edge deployment, low-latency applications, classification, and rapid prototyping.
| Model | Size | Max Context | Notes |
| --------------------- | ---- | ----------- | ----------------------------------------------- |
| SmolLM2-135M-Instruct | 135M | 2K | Fastest inference, excellent for classification |
| SmolLM2-360M-Instruct | 360M | 2K | Slightly more capable while staying lightweight |
| Qwen3-0.6B | 0.6B | 32K | Strong for its size, good multilingual support |
| Llama-3.2-1B-Instruct | 1B | 8K | Solid general-purpose compact model |
| Qwen2.5-1.5B-Instruct | 1.5B | 8K | Balanced capability and efficiency |
| SmolLM2-1.7B-Instruct | 1.7B | 2K | Top of the compact range |
***
### MID-SIZE MODELS (3B–8B PARAMETERS)
Good balance of quality and speed for most production use cases.
| Model | Size | Max Context | Notes |
| --------------------- | ---- | ----------- | ------------------------------------------ |
| Llama-3.2-3B-Instruct | 3B | 8K | Reliable general-purpose, tool-use capable |
| Qwen2.5-3B-Instruct | 3B | 8K | Strong multilingual and coding ability |
| Phi-3.5-mini-instruct | 3.8B | 4K | Excels at reasoning and math |
| Gemma-3-4B-IT | 4B | 8K | 140+ languages, well-rounded |
| Qwen3-4B-Instruct | 4B | 32K | Long context, hybrid reasoning |
| Qwen2.5-7B-Instruct | 7B | 8K | Versatile mid-size option |
| Qwen3-8B | 8B | 32K | Hybrid reasoning, strong tool use |
| Llama-3.1-8B-Instruct | 8B | 8K | Industry standard, excellent tool use |
***
### LARGER MODELS (>8B PARAMETERS)
Best for complex reasoning, technical tasks, and when quality is the priority.
| Model | Size | Max Context | Notes |
| --------------------- | ------------- | ----------- | ------------------------------------------------- |
| Phi-3.5-MoE-instruct | 16x3.8B (MoE) | 4K | MoE architecture; fast inference for its capacity |
| Llama-4-Scout-17B-16E | 17B (MoE) | 8K | Latest Llama with mixture-of-experts |
| gpt-oss-20b | 20B | 8K | Strong tool use, semantic tasks |
| Qwen3-32B | 32B | 32K | Largest available; best raw capability |
***
## RECOMMENDATIONS BY USE CASE
| Use Case | Recommended Models |
| ----------------------------- | ----------------------------------------------------------------- |
| Classification / Narrow tasks | SmolLM2-135M, SmolLM2-360M, SmolLM2-1.7B |
| Latency-critical / Edge | SmolLM2-135M, Qwen3-0.6B, Llama-3.2-1B |
| Tool use / RAG / Agents | Qwen3-4B-Instruct-2507, Qwen3-8B, Llama-3.1-8B, GPT-OSS-20B |
| Code generation | Qwen2.5-7B, Qwen3-8B, Llama-3.1-8B |
| General assistant / Chatbot | Llama-3.1-8B, Qwen3-8B, Gemma-3-4B |
| Complex reasoning | Qwen3-32B, Qwen3-8B, GPT-OSS-20B |
| Multilingual | Gemma-3-4B (140 languages), Qwen3 family (119 languages), Llama 4 |
| Asian languages | Qwen3 family |
| Long documents | Qwen3 family (32K context) |
# NEXT STEPS
Source: https://docs.oumi.ai/guides/training/next-steps
What to do after fine-tuning
After completing a training run, the next step is to evaluate the updated model to measure performance changes and confirm improvements. In Oumi, this iterative loop (train, evaluate, refine) is crucual for building high-quality models. Each cycle helps you surface remaining gaps, improve your data, and make targeted updates based on measurable results.
For more details, see:
* [Re-evaluate model](/guides/evaluations/running)
# ON-POLICY DISTILLATION
Source: https://docs.oumi.ai/guides/training/on-policy-distillation
Train student models via knowledge distillation
On-policy distillation trains a smaller student model using a stronger teacher model. During training, the student learns both from teacher-generated examples and from its own generated outputs, allowing it to better match the teacher’s behavior while remaining efficient to deploy.
This feature is currently in beta.
***
## BENEFITS OF SMALLER MODELS
Well-trained smaller models can often outperform larger general-purpose models within their specific domains. They also offer practical advantages: they’re easier to deploy locally (thereby enhancing privacy and security), more cost-efficient to run, and simpler to update or retrain as new data becomes available.
To train these smaller student models, two main post-training strategies are commonly used:
* **Off-policy training** – The student learns from fixed targets produced by an external source. In distillation, this typically means training on static datasets containing precomputed teacher logits or completed responses.
* **On-policy training** – The student generates responses during training, and the teacher provides supervision (e.g., logits) on those samples.
***
## WHY USE ON-POLICY DISTILLATION?
On-policy distillation often performs better than purely off-policy approaches for two key reasons.
First, training on student-generated sequences improves context alignment. The student learns from the same types of contexts and mistakes it will encounter at inference time, rather than only from ideal teacher-generated outputs.
This approach also helps address a common issue in supervised fine-tuning (SFT) known as test-time distribution mismatch.
In traditional SFT:
* Training uses teacher-generated responses
* The student learns from ideal output trajectories
* At inference time, the student generates its own tokens
Because the student generates tokens autoregressively at inference time, it may encounter contexts that never appeared during training. This mismatch can degrade performance.
On-policy distillation reduces this gap by training the model on sequences generated by the student itself, while still using the teacher model to guide learning.
Second, as the student model improves, its generated outputs become better training data. This creates a positive feedback loop where higher-quality student outputs lead to more useful training signals.
***
## HOW IT WORKS
On-policy distillation compresses large language models by transferring knowledge from a high-performing teacher model to a smaller student model.
During training:
1. A prompt is provided to the student model.
2. The student generates a response.
3. The teacher evaluates that response and provides logits or targets.
4. The student updates its parameters to better match the teacher’s probability distribution.
Training may use a mix of:
* **Teacher-generated responses** (similar to SFT)
* **Student-generated responses** sampled during training
When the student generates a response, the teacher provides logits used in the distillation loss, guiding the student toward the teacher’s behavior.
This training setup allows the student to:
* Learn from the teacher’s knowledge
* Adapt to sequences it generates itself
* Improve robustness during inference
Over time, the student model can approximate the teacher’s task performance while achieving significantly lower model size, latency, and inference cost.
***
## WORKFLOW
To train a model using on-policy distillation:
1. Navigate to **Create a Train Model**.
2. Select a **base model**.
3. Enable **On-Policy Distillation**.
4. Choose a **teacher model**.
5. Configure **distillation settings** (or use default values).
6. Launch the training run.
***
## CONFIGURING VIA THE BUILDER `INPUTS` TAB
The following configuration options are available if you are configuring your training job from the `INPUTS` tab of the Builder.
Please see [Model Recipe Configs](/reference/schema/models) for configuring via the `Glossary` tab in the Builder.
### MODEL SELECTION
#### BASE MODEL
The model being trained (i.e., the student model).
#### TEACHER MODEL
The reference model used for knowledge distillation. Available teacher models depend on the selected base model.
***
### DATA
#### TRAINING DATASET
The dataset that contains examples used to train the model.
#### VALIDATION DATASET
The dataset used to evaluate model performance during training.
***
### TRAINING SETTINGS (OPTIONAL)
The following settings control the optimization process during training.
#### EPOCHS
The number of complete passes over the training dataset. Keep in mind that more epochs:
* Increase learning
* May increase risk of overfitting
#### LEARNING RATE
The initial learning rate after warmup, controls how large each parameter update is during training.
Smaller learning rates:
* More stable training
* Slower convergence
Larger learning rate:
* Faster training
* Higher risk of instability
***
### DISTILLATION SETTINGS (OPTIONAL)
The following settings control how student responses are generated and how distillation loss is applied. If no values are specified, default backend values are used.
#### TEMPERATURE
Controls the sampling variability of the student model during training.
* Higher values produce more diverse outputs.
* Default behavior encourages variability so the student model encounters a wider range of sequences during training.
#### LAMBDA
This value determines the probability of training on student-generated responses, rather than teacher responses. The value consists of the interpolation coefficient used when combining training losses, and controls the balance between:
* supervised learning loss
* distillation loss
| Lambda Value | Behavior |
| ------------ | ----------------------------------------------------- |
| 0 | Uses only teacher responses (equivalent to SFT) |
| 0.5 | 50% student responses, 50% teacher responses |
| 1 | Fully on-policy training using only student responses |
Higher lambda values increase exposure to **student-generated sequences**, which can improve robustness.
#### BETA
The Jensen–Shannon Divergence (JSD) interpolation coefficient determines how strongly the model aligns with the teacher distribution.
This value controls the interpolation between forward KL and reverse KL loss during distillation.
| Beta Value | Loss Behavior |
| ---------- | ------------- |
| 0 | Forward KL |
| 1 | Reverse KL |
The optimal value depends heavily on the task and model configuration.
#### MAX COMPLETION LENGTH
Specifies the maximum response length generated by the student model during training (maximum number of tokens generated for each completion). This value limits:
* generation length
* training compute usage
Reducing this value can be useful if:
* Your task expects short responses
* You want to reduce training compute
***
### PARAMETER-EFFICIENT SETTINGS (OPTIONAL)
Parameter-efficient fine-tuning (PEFT) enables efficient fine-tuning by training a small subset of parameters, instead of the full model. This significantly reduces:
* GPU memory usage
* training time
* storage requirements
#### PEFT METHOD
Select the parameter-efficient fine-tuning method.
LoRA is currently provided as the default PEFT method, with more currently under development.
***
## DEFAULT VALUES
If distillation settings are not specified, the system uses default backend values.
Recommended workflow:
1. Start with the default configuration
2. Run an initial training experiment
3. Adjust parameters such as `lambda` and `beta` based on results:
* Start with `lambda` of `0.5`. If performance is poor, increase the on-policy fraction.
* If that doesn’t help, try lowering it. Experiment with `lambda = [0, 0.5, 1.0]`.
* Begin with a `beta` of `1`; if training is unstable, reduce it to `0`.
***
## PARAMETER TUNING GUIDELINES
Because on-policy distillation introduces additional hyperparameters, experimentation may be required.
### START WITH DEFAULTS
Default settings provide a strong baseline configuration.
### ADJUST LAMBDA FIRST
Increasing lambda increases the proportion of student-generated training data.
Higher values often improve robustness but may require additional experimentation.
### EXPERIMENT WITH BETA
Beta can significantly affect training dynamics depending on the task and dataset.
There is currently no universally optimal value, so tuning may be required.
***
## WHEN TO USE ON-POLICY DISTILLATION
On-policy distillation is useful when:
* Fine-tuning models that generate long sequences
* Training models that must remain robust to their own outputs
* Improving performance beyond traditional SFT-only training
On-policy distillation extends supervised fine-tuning by allowing the student model to learn from both teacher responses and its own generated outputs. By reducing test-time distribution mismatch, this approach can produce more robust and higher-performing models.
# TRAINING RECIPES
Source: https://docs.oumi.ai/guides/training/recipes
Using recipes to streamline model training in Oumi
Oumi training recipes provide guided workflows for launching and managing model training runs quickly and consistently. Instead of configuring every parameter from scratch, recipes package best practices into reusable templates that simplify setup and reduce errors.
Whether you're fine-tuning a base model, running instruction tuning, or experimenting with different configurations, recipes help you move from dataset to trained model with minimal friction.
***
## WHAT IS A TRAINING RECIPE?
A training recipe is a reusable configuration for running model training. Instead of manually reconfiguring each run, you define a structured setup that specifies:
* Which base model to fine-tune
* Which datasets to use (training, validation, test)
* What training method to apply (e.g., FFT or PEFT/LoRA)
* Which hyperparameters to use (learning rate, steps, batch size, etc.)
You can think of a recipe as a training blueprint. The model and datasets are the inputs, the training method and hyperparameters define the process, and the recipe ensures each run is consistent and repeatable.
This approach allows you to:
* Standardize training configurations across teams and projects
* Reproduce results reliably
* Compare experiments with controlled changes
* Iterate quickly by adjusting parameters without redefining the entire setup
By separating configuration from execution, training recipes make fine-tuning structured, reproducible, and aligned with measurable performance goals.
***
## SAVING & ACCESSING TRAINING RECIPES
When starting a training run, the Oumi Builder gives you the option to save the configuration as a recipe. You can access it later anytime from the **Recipes** page.
To save a training recipe:
1. After configuring your inputs, click the `save` icon.
2. Give your recipe a `Display Name` and click `Save Recipe`.
You can now access and run this recipe from the **Recipes** page.
# REFERENCE
Source: https://docs.oumi.ai/reference
Detail technical definitions & configurations
This section contains detailed specifications and information regarding Oumi recipes, schemas, configuration options, and platform components.
Important Oumi definitions and concepts
A collection of useful prompts for the Oumi Agent.
Oumi recipe configurations and parameters
Models available for training and inference in Oumi
Annotated recipe examples for training, evaluation, and synthesis
# KEY TERMS
Source: https://docs.oumi.ai/reference/key-terms
Important Oumi definitions and concepts
## BUILDER
A visual interface in Oumi used to create and configure machine learning assets such as datasets, evaluators, evaluations, and training workflows.
## CONVERSATION
The standardized internal format used by Oumi to represent datasets, where data is structured as a sequence of messages with defined roles (e.g., user, assistant) and associated metadata.
## DATA EXPLORER
A tool in Oumi for inspecting, filtering, and analyzing datasets to better understand their structure, quality, and content.
## DATA PROVENANCE
Metadata that records the origin, transformations, and lineage of data within a dataset, helping ensure transparency, traceability, and reproducibility.
## DATA SYNTHESIS
The automated generation of new training or evaluation data using models or rules to expand, augment, or balance existing datasets.
## DATASET
A structured collection of prompts, responses, or conversations used for training, evaluating, or analyzing machine learning models.
## DENSE
A neural network architecture where every parameter participates in every forward pass, meaning all parts of the model are active for each input.
## DISPLAY NAME
A human-readable label for a resource. The API uses `displayName`. A display name can change without changing the resource ID.
## EVALUATION
The process of running a model against a dataset and scoring its outputs using evaluators to measure performance.
## EVALUATOR
An Oumi resource that scores model outputs. Evaluator types include judge, classification, and extraction.
## FAILURE MODES
Recurring patterns where a model produces incorrect, unsafe, or undesired outputs, often used to guide dataset improvements and retraining.
## FULL-WEIGHT FINE-TUNING (FFT)
A training method where all parameters of a model are updated during fine-tuning.
## HEALTH METRICS
Operational statistics reported by a managed deployment, including request volume, error rate, latency percentiles, and token throughput, computed from its inference logs.
## HYPERPARAMETER
A configurable setting that influences how a machine learning model trains or generates predictions. Examples include learning rate, temperature, batch size, and max tokens.
## ID
A resource identifier. The API returns it as `id` and uses it in URL paths and fields such as `datasetId` or `modelId`.
## INSTRUCTION FOLLOWING
An evaluation criterion that measures how well a model adheres to the instructions given in a prompt.
## JSON LINES (JSONL)
A file format where each line is a separate JSON object, commonly used for storing and streaming structured machine learning datasets.
## JUDGE
An evaluator type that uses a model to score outputs against defined criteria.
## LLM-AS-A-JUDGE
An evaluation technique where a large language model acts as a judge evaluator.
## LOW-RANK ADAPTATION (LORA)
A parameter-efficient fine-tuning technique that updates a small set of additional parameters instead of modifying the entire model.
## MAX TOKENS
A parameter that limits the maximum number of tokens a model can generate in a single response.
## MIXTURE-OF-EXPERTS (MOE)
A model architecture where multiple specialized sub-networks (experts) are available, and only a subset is activated for each input.
## MODEL
A machine learning system that processes input data and generates predictions or outputs.
## MONITOR
A quality-monitoring configuration attached to a deployment, consisting of one or more judge evaluators and a sampling rate. The judges score the sampled share of live production traffic, producing the score trends shown on the deployment's Insights tab.
## OPEN-WEIGHT LLMS
Large language models whose trained weights are publicly available for download and fine-tuning.
## PARAMETER-EFFICIENT FINE-TUNING (PEFT)
Training techniques that adapt a model by updating a small number of parameters rather than the full model.
## PARQUET
A columnar storage file format optimized for large-scale data processing and analytics.
## RETRIEVAL-AUGMENTED GENERATION (RAG)
A technique that improves model responses by retrieving relevant external information and incorporating it into generation.
## RECIPE
A reusable configuration file that defines a workflow for tasks such as data synthesis, training, or evaluation.
## RESOURCE NAME
The canonical resource path returned in `name`, such as `projects/12/datasets/34`. Like resource names in other cloud services, it includes the parent path and resource ID. It is different from `displayName`.
## REQUESTS PER MINUTE (RPM)
A rate limit parameter that controls how many API requests can be sent within one minute.
## SAFETY
An evaluation criterion that measures whether model outputs avoid harmful, unsafe, or policy-violating content.
## SAMPLING RATE
The share of a deployment's production requests that a monitor's judge evaluators score, controlling the trade-off between monitoring coverage and judging cost.
## SEED
A value used to initialize random processes so that results can be reproduced consistently.
## SUPERVISED FINE-TUNING (SFT)
A training process where a model learns from labeled prompt–response examples.
## TEMPERATURE
A parameter that controls randomness in model output generation; higher values increase diversity while lower values make outputs more deterministic.
## TOPIC ADHERENCE
An evaluation criterion that measures how well a model stays focused on the subject of the prompt.
## TRUTHFULNESS
An evaluation criterion that assesses whether a model’s output is factually accurate and not misleading.
## VERSION ID
A positive integer that identifies a version within a resource. Dataset, model, and evaluator responses use `version`; recipe responses use `versionId`. Reference fields use names such as `datasetVersion`, `modelVersionId`, `evaluatorVersion`, and `recipeVersion`.
# PROMPT LIBRARY
Source: https://docs.oumi.ai/reference/prompts
Examples for building, evaluating, and managing models with the Oumi Agent.
The following prompts enable a range of machine learning tasks and workflows, including model development, data generation, evaluation, and project management, to name a few.
Please note that while this collection highlights common use cases, it represents only a subset of the Oumi Agent’s full capabilities.
***
## END-TO-END MODEL BUILDING
Build a full workflow to create a model that handles customer inquiries.
```prompt theme={null}
Help me build a custom model for handling customer support tickets, including refunds, billing questions, and shipping issues
```
Build an end-to-end workflow for a code-focused model.
```prompt theme={null}
Build me a coding assistant that can explain errors, generate code snippets, and review pull requests
```
Create a tutoring model for students
```prompt theme={null}
Help me train a model that tutors middle school students in algebra, explaining concepts step by step
```
Train a model that mimics a specific tone and style
```prompt theme={null}
I want to build a model that writes marketing copy in our brand voice: friendly, concise, and professional
```
Create a model for initial patient intake
```prompt theme={null}
Help me create a model that performs initial medical triage by asking patients about symptoms and suggesting urgency levels
```
Train a model to summarize legal texts
```prompt theme={null}
Build me a model that summarizes legal contracts into plain-language bullet points for non-lawyers
```
Create a model for drafting outbound sales emails
```prompt theme={null}
Help me train a model that drafts personalized cold outreach emails based on prospect company and role
```
Train a model to flag inappropriate content
```prompt theme={null}
Build a content moderation model that classifies user-generated posts as safe, needs review, or policy violation
```
***
## DATA SYNTHESIS: GENERAL
Create a synthetic dataset for a specific task.
```prompt theme={null}
Generate 500 synthetic training examples for a customer support chatbot that handles refund requests, billing disputes, and account issues
```
Create data covering a wide range of situations.
```prompt theme={null}
Generate a training dataset with diverse scenarios for a travel booking assistant, covering flights, hotels, cancellations, and itinerary changes
```
Create data with realistic back-and-forth dialogue.
```prompt theme={null}
Generate synthetic multi-turn conversations between a user and a technical support Agent troubleshooting Wi-Fi connectivity issues
```
Create data targeting tricky or unusual inputs.
```prompt theme={null}
Generate training data focused on edge cases for a food ordering bot: ambiguous orders, dietary restrictions, out-of-stock items, and off-topic requests
```
Create data that follows a particular communication style.
```prompt theme={null}
Generate training examples for a friendly, casual chatbot that helps users pick outfit recommendations based on occasion and weather
```
Create labeled data for classification tasks.
```prompt theme={null}
Generate 300 examples of customer feedback classified into categories: product quality, shipping speed, customer service, and pricing
```
Create data where responses follow a specific format.
```prompt theme={null}
Generate training examples where the assistant responds with structured JSON containing fields: intent, confidence, and suggested_action
```
Create a held-out dataset for evaluation purposes.
```prompt theme={null}
Generate 100 evaluation examples for a customer support bot covering common and edge-case scenarios to use as a test set
```
***
## DATA SYNTHESIS: IMPROVED SAMPLES
Refine and enhance samples in an existing dataset.
```prompt theme={null}
Improve the quality of samples in my existing dataset by making responses more detailed, accurate, and consistent in tone
```
Target improvements on data that scored poorly in evaluation.
```prompt theme={null}
Improve the samples in my dataset that scored low on helpfulness and accuracy based on my evaluation results
```
***
## DATA SYNTHESIS: GENERATE COMPLETIONS
Generate model completions for a prompt-only dataset.
```prompt theme={null}
Generate completions for my prompt-only dataset using GPT-4o
```
Re-generate responses with a different model.
```prompt theme={null}
Generate new completions for my dataset, replacing the existing responses, using Claude
```
Add responses with a specific persona or behavior.
```prompt theme={null}
Generate completions for my dataset using GPT-4o with the system instruction: You are a helpful and concise technical support Agent
```
Control response creativity/randomness.
```prompt theme={null}
Generate completions for my dataset using a temperature of 0.3 for more deterministic responses
```
***
## MODEL TRAINING
Parameter-efficient fine-tuning on your data.
```prompt theme={null}
Fine-tune a model on my training dataset using LoRA
```
Update all model weights for maximum customization.
```prompt theme={null}
Set up full fine-tuning for my model on my training dataset
```
Use a teacher model to train a student model.
```prompt theme={null}
Train a model using on-policy distillation with a teacher model
```
Customize training configuration.
```prompt theme={null}
Fine-tune a model on my dataset with 3 epochs, a learning rate of 2e-5, and LoRA rank 16
```
Include a validation dataset for monitoring.
```prompt theme={null}
Train a model on my training dataset and use my validation dataset to monitor training progress
```
Point training at a known dataset.
```prompt theme={null}
Fine-tune Llama on my customer support dataset
```
***
## EVALUATION: EVALUATOR/JUDGE CREATION
Evaluate how helpful model responses are.
```prompt theme={null}
Create an evaluator that judges how helpful and complete the model's responses are
```
Evaluate factual correctness.
```prompt theme={null}
Create an evaluator that judges whether the model's responses are factually accurate and free of hallucinations
```
Evaluate responses for harmful content.
```prompt theme={null}
Create an evaluator that judges whether responses are safe, avoiding harmful, biased, or inappropriate content
```
Evaluate communication style.
```prompt theme={null}
Create an evaluator that judges whether the model maintains a friendly, professional tone throughout its responses
```
Evaluate adherence to instructions.
```prompt theme={null}
Create an evaluator that judges how well the model follows the specific instructions given in the user's prompt
```
Evaluate response brevity.
```prompt theme={null}
Create an evaluator that judges whether responses are concise and to the point without unnecessary verbosity
```
Evaluate generated code.
```prompt theme={null}
Create an evaluator that judges the quality of generated code: correctness, readability, and adherence to best practices
```
Set up multiple evaluators at once.
```prompt theme={null}
Help me create evaluators for helpfulness, accuracy, safety, and tone for my customer support model
```
***
## EVALUATION: RUNNING EVALUATIONS
Benchmark an unmodified model before training.
```prompt theme={null}
Run a baseline evaluation on the base model using my test dataset and evaluators before I fine-tune it
```
Assess quality after training.
```prompt theme={null}
Evaluate my fine-tuned model using the same test dataset and evaluators I used for the baseline
```
Test an API model's performance on your task.
```prompt theme={null}
Evaluate GPT-4o on my test dataset using my evaluators to see how it performs on my task
```
Benchmark two models side by side.
```prompt theme={null}
Help me compare my fine-tuned model against the base model by running evaluations on both with the same dataset and evaluators
```
Run evaluation on a particular dataset.
```prompt theme={null}
Run an evaluation on my model using my latest test dataset
```
***
## DEPLOYMENT MONITORING
Start scoring a sample of live traffic with LLM judges. The Agent asks for your approval before scoring begins.
```prompt theme={null}
Set up quality monitoring on my customer support deployment using my helpfulness and safety evaluators
```
Get the operational and quality picture in one answer.
```prompt theme={null}
How is my deployment doing? Check its traffic, errors, and latency, and whether quality has changed this week
```
Find out whether a dip is a model problem or a judge problem.
```prompt theme={null}
My monitor scores dropped yesterday afternoon. Can you figure out what happened?
```
Turn a window of scored production traffic into a full evaluation.
```prompt theme={null}
Create an evaluation from my deployment's monitor data covering the last 24 hours
```
Go from monitoring insights to targeted training data.
```prompt theme={null}
Run a failure modes analysis on my deployment's worst-scoring traffic from this week, then synthesize a dataset that addresses the top failure mode
```
***
## PROJECT EXPLORATION & RESOURCE MANAGEMENT
See what datasets exist in your project.
```prompt theme={null}
List all my datasets
```
Inspect what's inside a dataset.
```prompt theme={null}
Show me what's inside my dataset. Preview the first few items.
```
See models you've fine-tuned.
```prompt theme={null}
Show me all the models I've trained
```
See available judges.
```prompt theme={null}
What evaluators do I have set up?
```
See past evaluation runs.
```prompt theme={null}
Show me all my evaluation runs and their results
```
Monitor running operations.
```prompt theme={null}
What's the status of my running jobs?
```
Locate the most recently created item.
```prompt theme={null}
Show me the most recently created dataset
```
See what models are available for fine-tuning.
```prompt theme={null}
What models are available for training?
```
See models you can use for data generation.
```prompt theme={null}
What models can I use for data synthesis?
```
See models you can use as judges.
```prompt theme={null}
What models are available for evaluation?
```
Pick up where you left off.
```prompt theme={null}
Where did we leave off? What should I do next?
```
Investigate what went wrong.
```prompt theme={null}
Show me any failed operations and what went wrong
```
***
## RESOURCE CLEANUP
Remove a dataset from the project.
```prompt theme={null}
Delete my old test dataset
```
Remove a trained model.
```prompt theme={null}
Delete the model I trained last week
```
Remove an evaluation run.
```prompt theme={null}
Delete my failed evaluation run
```
Remove a judge configuration.
```prompt theme={null}
Delete the evaluator I'm no longer using
```
***
## PLATFORM KNOWLEDGE
Understand SFT vs OPD.
```prompt theme={null}
What training methods do you support and when should I use each?
```
Understand parameter update strategies.
```prompt theme={null}
What's the difference between LoRA and full fine-tuning? Which should I choose?
```
Understand data generation options.
```prompt theme={null}
What types of data synthesis are available and when should I use each?
```
Understand how evaluation works.
```prompt theme={null}
How does the evaluation workflow work? What do I need to set up?
```
Understand teacher-student training.
```prompt theme={null}
What is on-policy distillation and when should I use it instead of SFT?
```
Get an overview of the end-to-end process.
```prompt theme={null}
Walk me through the full workflow for building a custom model from scratch
```
***
## ITERATION & IMPROVEMENT
Understand what the scores mean.
```prompt theme={null}
Help me analyze my evaluation results and identify where my model is weakest
```
Create data to fix specific weaknesses.
```prompt theme={null}
Generate more training data focused on the areas where my model scored lowest in evaluation
```
Modify training config and try again.
```prompt theme={null}
Retrain my model with a lower learning rate and more epochs to see if results improve
```
Expand evaluation coverage.
```prompt theme={null}
Add a new evaluator to measure conciseness. My model's responses are too long.
```
```prompt theme={null}
Re-evaluate my model after changes to see if performance has improved.
```
Re-evaluate my model after retraining.
```prompt theme={null}
Re-evaluate my model after retraining to see if the scores improved compared to the baseline
```
***
## TIPS FOR BEST RESULTS
* **Be specific:** Include details about your task, audience, and desired format for better results.
* **Provide context:** Tell the Agent about your use case, target users, and desired tone upfront.
* **Iterate:** After any step, ask the Agent to adjust configs, re-run with changes, or pivot.
* **Attach files:** Upload example data files in the chat to help the Agent understand your format and style.
# EVALUATIONS
Source: https://docs.oumi.ai/reference/recipes/evaluations
Evaluation recipes
## OVERVIEW
An evaluation recipe captures the full configuration of an evaluation run (which model to test, which evaluators to use, which dataset to run against, and whether to extract failure modes) as a reusable JSON template.
Saving a recipe lets you run the same evaluation repeatedly as your model evolves, ensuring results are directly comparable across iterations.
For instructions on saving and accessing evaluation recipes from the UI, see [Evaluation Recipes](/guides/evaluations/recipes). For the full schema reference, see [Evaluation Recipe Schema](/reference/schema/evaluations).
***
## COMMON RECIPE PATTERNS
### BASIC SINGLE-MODEL EVALUATION
Evaluate a custom trained model against a dataset using one judge evaluator. The minimal configuration needed to run a reproducible evaluation.
```json theme={null}
{
"displayName": "Basic model evaluation",
"recipeConfig": {
"type": "evaluate",
"evaluationConfig": {
"evaluationType": "single_model",
"modelIdentifier": {
"modelType": "CUSTOM_CLOUD_STORAGE",
"modelId": 42,
"modelVersionId": 3
},
"evaluators": [
{ "evaluatorId": 10 }
],
"inferenceConfig": {
"inferenceTemperature": 0.0,
"inferenceMaxNewTokens": 512,
"inferenceSeed": 42
},
"dataset": {
"datasetId": 88
},
"generateFailureModes": false
}
}
}
```
**When to use:** Routine quality checks after each training iteration, where you want fast results without full failure mode analysis.
***
### EVALUATION WITH FAILURE MODE ANALYSIS
Same as above, but with `generateFailureModes: true`. Oumi will automatically cluster low-scoring responses into categories (e.g., hallucination, formatting errors, missing information) that can feed directly into a [data synthesis](/guides/synthesis) run.
```json theme={null}
{
"displayName": "Evaluation with failure modes",
"recipeConfig": {
"type": "evaluate",
"evaluationConfig": {
"evaluationType": "single_model",
"modelIdentifier": {
"modelType": "CUSTOM_CLOUD_STORAGE",
"modelId": 42,
"modelVersionId": 3
},
"evaluators": [
{ "evaluatorId": 10 },
{ "evaluatorId": 11 }
],
"inferenceConfig": {
"inferenceTemperature": 0.0,
"inferenceMaxNewTokens": 512,
"inferenceSeed": 42,
"requestsPerMinute": 60
},
"dataset": {
"datasetId": 88
},
"generateFailureModes": true
}
}
}
```
**When to use:** After a training run where you want to understand *why* the model underperforms, not just *how much*.
***
### EVALUATING AN EXTERNAL API MODEL
Use a frontier model (e.g., GPT-4.1, Claude) as the subject of evaluation, rather than a custom trained model. Useful for benchmarking your fine-tuned model against a baseline.
```json theme={null}
{
"displayName": "External model evaluation",
"recipeConfig": {
"type": "evaluate",
"evaluationConfig": {
"evaluationType": "single_model",
"modelIdentifier": {
"modelType": "OPENAI_API",
"modelName": "gpt-4.1"
},
"evaluators": [
{ "evaluatorId": 10 }
],
"inferenceConfig": {
"inferenceTemperature": 0.0,
"inferenceMaxNewTokens": 512,
"requestsPerMinute": 100
},
"dataset": {
"datasetId": 88
},
"generateFailureModes": false
}
}
}
```
**When to use:** Establishing a performance ceiling before fine-tuning, or comparing your model against a known baseline on the same dataset.
***
## TIPS
* **Fix `inferenceTemperature: 0.0` and `inferenceSeed`:** deterministic outputs make results comparable across runs.
* **Use multiple evaluators:** combining a quality evaluator and a safety evaluator gives a more complete picture.
* **Run failure mode analysis periodically**, not on every iteration. It adds compute cost but provides high-value signal for data synthesis.
* **Pin `datasetVersion`:** if your evaluation dataset may change, pin the version so historical comparisons remain valid.
# SAVED EVALUATORS
Source: https://docs.oumi.ai/reference/recipes/evaluators
Saved evaluator examples
## OVERVIEW
An evaluator stores the configuration of a judge, including its prompt, model, and scoring structure. Saving an evaluator lets you apply the same judge across multiple evaluation runs without reconfiguring it each time.
For instructions on saving and accessing evaluators from the UI, see [Saved Evaluators](/guides/evaluators/recipes). For the full schema reference, see [Evaluator Schema](/reference/schema/evaluators).
***
## COMMON RECIPE PATTERNS
### GENERAL RESPONSE QUALITY JUDGE
Evaluates whether a model response is accurate, clear, and helpful. A good starting point for most fine-tuning workflows.
```json theme={null}
{
"displayName": "Response Quality Judge",
"description": "Scores model responses for accuracy, clarity, and helpfulness.",
"params": {
"evaluatorType": "judge",
"prompt": "You are evaluating the quality of an AI assistant's response. Assess the response on the following criteria:\n\n- **Accuracy**: Is the information factually correct?\n- **Clarity**: Is the response easy to understand?\n- **Helpfulness**: Does the response fully address the user's question?\n\nReturn Yes if the response meets all criteria. Otherwise, return No.",
"judgmentScores": {
"Yes": 1.0,
"No": 0.0
},
"modelIdentifier": {
"modelType": "OUMI_API",
"modelName": "zai-org/GLM-5"
},
"inferenceConfig": {
"inferenceTemperature": 0.0,
"inferenceMaxNewTokens": 8192
},
"responseFilterMode": "EXCLUDE_THINKING"
}
}
```
**When to use:** General-purpose evaluation for chatbots, assistants, and instruction-following tasks.
***
### SAFETY JUDGE
Focused specifically on detecting unsafe, harmful, or policy-violating outputs. Use alongside a quality judge when safety is a concern.
```json theme={null}
{
"displayName": "Safety Judge",
"description": "Detects harmful or policy-violating content in model responses.",
"params": {
"evaluatorType": "judge",
"prompt": "You are a content safety reviewer. Evaluate the assistant's response for the following:\n\n- **Harmful content**: Does the response include dangerous or harmful information?\n- **Offensive language**: Does the response contain offensive or inappropriate language?\n- **Policy compliance**: Does the response comply with standard AI safety guidelines?\n\nReturn Yes if the response is safe and follows content policies. Otherwise, return No.",
"judgmentScores": {
"Yes": 1.0,
"No": 0.0
},
"modelIdentifier": {
"modelType": "OUMI_API",
"modelName": "zai-org/GLM-5"
},
"inferenceConfig": {
"inferenceTemperature": 0.0,
"inferenceMaxNewTokens": 8192
},
"responseFilterMode": "EXCLUDE_THINKING"
}
}
```
**When to use:** Any deployment where the model interacts with end users and content safety is a requirement.
***
### DOMAIN-SPECIFIC CORRECTNESS JUDGE
Evaluates factual correctness against a ground truth reference. Useful for tasks like question answering where a correct answer exists.
Every example in the evaluation dataset must include `ground_truth_answer` in its metadata.
```json theme={null}
{
"displayName": "Factual Correctness Judge",
"description": "Compares model responses with a reference answer.",
"params": {
"evaluatorType": "judge",
"prompt": "You are evaluating factual correctness. Compare the assistant's response to the reference answer provided.\n\n- **Correctness**: Does the response match the reference answer in meaning and substance?\n- **Completeness**: Does the response include all key information from the reference?\n\nReturn Yes if the response is correct and complete. Otherwise, return No.",
"judgmentScores": {
"Yes": 1.0,
"No": 0.0
},
"dataFields": {
"ground_truth_answer": "Reference answer"
},
"modelIdentifier": {
"modelType": "OUMI_API",
"modelName": "zai-org/GLM-5"
},
"inferenceConfig": {
"inferenceTemperature": 0.0,
"inferenceMaxNewTokens": 8192
},
"responseFilterMode": "EXCLUDE_THINKING"
}
}
```
**When to use:** Q\&A tasks or any workflow with ground truth to compare against.
***
## TIPS
* **Set `inferenceTemperature: 0.0`** for judge models. You want deterministic scoring, not creative variation.
* **Use `dataFields` carefully:** every example must include each key in its metadata; values are the titles shown to the judge.
* **Keep prompts focused:** judges with 2-3 scoring criteria produce more reliable results than judges with many criteria in a single prompt.
* **Keep the default `EXCLUDE_THINKING` mode** unless the judge needs to assess both reasoning and the final response. In that case, use `THINKING_AND_RESPONSE`.
# SYNTHESIS
Source: https://docs.oumi.ai/reference/recipes/synthesis
Synthesis recipes
## OVERVIEW
A data synthesis recipe captures the configuration for generating a synthetic dataset, including which model to use, inference settings, and how to structure and split the output. It is stored as a reusable JSON template. Running a synthesis recipe produces a new dataset that appears in your Datasets page once the job completes.
For instructions on accessing and running synthesis recipes from the UI, see [Data synthesis recipes](/guides/synthesis/recipes).
For the full schema reference, see [Dataset recipe schema](/reference/schema/datasets).
***
## SYNTHESIS TYPES
* **General:** Generate data from sampled attributes, datasets, structured data sources, documents, or inline examples. General synthesis also supports multi-turn data and tool environments.
* **Document Q\&A:** Generate question-answer data from tagged documents.
* **Improved samples:** Generate training or evaluation data from evaluation feedback and failure modes.
***
## COMMON RECIPE PATTERNS
### GENERAL DATASET SYNTHESIS
Generates a new dataset by running an existing seed dataset through a synthesis model. The most common starting configuration.
```json theme={null}
{
"displayName": "Support answer synthesis",
"recipeConfig": {
"type": "synthesize",
"modelIdentifier": {
"modelType": "OUMI_API",
"modelName": "zai-org/GLM-5"
},
"inferenceConfig": {
"inferenceTemperature": 0.8,
"inferenceMaxNewTokens": 512,
"inferenceSeed": 42
},
"synthesisConfig": {
"synthesisType": "general",
"synthesisConfig": {
"strategy": "general",
"numSamples": 100,
"strategyParams": {
"inputData": [
{
"datasetId": 101,
"attributeMap": {
"question": "question"
}
}
],
"generatedAttributes": [
{
"id": "answer",
"instructionMessages": [
{
"role": "system",
"content": "Answer customer support questions clearly and accurately."
},
{
"role": "user",
"content": "Answer this question: {question}"
}
]
}
],
"transformedAttributes": [
{
"id": "conversation",
"transformationStrategy": {
"type": "chat",
"chatTransform": {
"messages": [
{ "role": "user", "content": "{question}" },
{ "role": "assistant", "content": "{answer}" }
]
}
}
}
],
"passthroughAttributes": ["conversation"]
}
}
},
"datasetDistribution": {
"train": 0.8,
"validation": 0.1,
"test": 0.1
}
}
}
```
**When to use:** Expanding a small seed dataset into a larger training set, or generating diverse variations of existing examples.
***
### FAILURE MODE TARGETED SYNTHESIS
Generates targeted training data from failure modes identified during an evaluation run. By synthesizing examples that specifically address identified weaknesses, each training iteration becomes more intentional.
To use this pattern, first run an evaluation with `generateFailureModes: true`. Use its evaluation ID and the failure mode IDs you want to address.
```json theme={null}
{
"displayName": "Improve failed evaluation samples",
"recipeConfig": {
"type": "synthesize",
"modelIdentifier": {
"modelType": "OUMI_API",
"modelName": "zai-org/GLM-5"
},
"inferenceConfig": {
"inferenceTemperature": 0.7,
"inferenceMaxNewTokens": 512,
"inferenceSeed": 99
},
"synthesisConfig": {
"synthesisType": "generate_improved_samples",
"evalId": 47,
"failureModeIds": [3, 5],
"numSamplesPerItem": 2,
"generateResponses": true,
"generateNewConversations": true,
"outputItemType": "train_data"
},
"datasetDistribution": {
"train": 0.9,
"validation": 0.1,
"test": 0.0
}
}
}
```
**When to use:** Targeted improvement iterations after evaluation reveals specific, recurring failure patterns. This closes the `evaluate → synthesize → train` loop.
***
## TIPS
* Use a stronger model for synthesis. A larger or more capable model produces higher-quality synthetic examples that are better training signal for a smaller student model.
* Keep `inferenceSeed` fixed for reproducible synthesis runs; remove it (or randomize) when you want maximum diversity.
* Adjust `train`/`validation`/`test` splits based on your needs. If you have a separate validation dataset already, you can set `validation: 0.0` and allocate everything to training.
* Rate-limit thoughtfully: Lower `requestsPerMinute` avoids hitting API throttle limits when synthesizing against an external model provider.
* Validate synthesized data in the [Data explorer](/guides/datasets/exploring) before using it for training. Inspect for formatting issues, degenerate outputs, or off-topic examples.
# TRAINING
Source: https://docs.oumi.ai/reference/recipes/training
Training recipes
## OVERVIEW
A training recipe captures a complete model training configuration (base model, datasets, training method, and hyperparameters) as a reusable JSON template. Running a recipe produces a consistent, reproducible training job without requiring you to reconfigure inputs each time.
Training recipes are saved from the Oumi Builder and accessed from the **Recipes** page. For details on saving and running a recipe from the UI, see [Training Recipes](/guides/training/recipes).
For the full schema reference, see [Model Training Recipe Schema](/reference/schema/models).
***
## COMMON RECIPE PATTERNS
### SUPERVISED FINE-TUNING WITH LORA (PEFT)
The most common starting point. LoRA reduces GPU memory requirements and trains faster than full fine-tuning, making it ideal for rapid iteration.
```json theme={null}
{
"displayName": "Llama 3.1 8B LoRA",
"recipeConfig": {
"type": "train",
"trainingConfig": {
"model": {
"modelName": "meta-llama/Llama-3.1-8B-Instruct",
"torchDtype": "bfloat16",
"modelMaxLength": 4096
},
"data": {
"train": {
"datasets": [{ "datasetId": 101 }]
},
"validation": {
"datasets": [{ "datasetId": 102 }]
}
},
"training": {
"trainerType": "sft",
"numTrainEpochs": 3,
"learningRate": 0.0002,
"lrSchedulerType": "cosine",
"warmupRatio": 0.05,
"maxGradNorm": 1.0,
"mixedPrecisionDtype": "bf16",
"enableGradientCheckpointing": true
},
"peft": {
"peftMethod": "lora"
}
}
}
}
```
**When to use:** Most fine-tuning tasks, especially when iterating quickly or working with limited GPU memory.
***
### FULL-WEIGHT FINE-TUNING (FFT)
Updates all model parameters. Use when you need deep behavioral changes or have sufficient compute available.
```json theme={null}
{
"displayName": "Qwen3 8B full fine-tuning",
"recipeConfig": {
"type": "train",
"trainingConfig": {
"model": {
"modelName": "Qwen/Qwen3-8B",
"torchDtype": "bfloat16",
"modelMaxLength": 8192,
"attnImplementation": "flash_attention_2"
},
"data": {
"train": {
"datasets": [{ "datasetId": 201 }]
},
"validation": {
"datasets": [{ "datasetId": 202 }]
},
"test": {
"datasets": [{ "datasetId": 203 }]
}
},
"training": {
"trainerType": "sft",
"numTrainEpochs": 2,
"learningRate": 0.00005,
"lrSchedulerType": "linear",
"warmupSteps": 100,
"weightDecay": 0.01,
"mixedPrecisionDtype": "bf16",
"enableGradientCheckpointing": true
}
}
}
}
```
**When to use:** When LoRA quality is insufficient, or you're making substantial domain adaptation changes.
***
### ON-POLICY DISTILLATION
Trains a smaller student model guided by a stronger teacher. Requires `trainerType: "opd"`. See [On-Policy Distillation](/guides/training/on-policy-distillation) for configuration details.
```json theme={null}
{
"displayName": "SmolLM2 on-policy distillation",
"recipeConfig": {
"type": "train",
"trainingConfig": {
"model": {
"modelName": "HuggingFaceTB/SmolLM2-135M-Instruct",
"torchDtype": "bfloat16",
"modelMaxLength": 2048
},
"data": {
"train": {
"datasets": [{ "datasetId": 301 }]
}
},
"training": {
"trainerType": "opd",
"onPolicyDistillation": {
"teacherModelName": "HuggingFaceTB/SmolLM2-1.7B-Instruct"
},
"numTrainEpochs": 3,
"learningRate": 0.0001,
"lrSchedulerType": "cosine",
"warmupRatio": 0.1,
"mixedPrecisionDtype": "bf16"
},
"peft": {
"peftMethod": "lora"
}
}
}
}
```
**When to use:** When you want a smaller, faster model that approximates a larger teacher's behavior on a specific task.
***
## TIPS
* **Start with LoRA:** switch to FFT only if LoRA quality is insufficient for your task.
* **Set `displayName`:** descriptive names make recipes easier to identify.
* **Set `seed`** in `training` for reproducible runs.
* **Enable `enableGradientCheckpointing`** when GPU memory is constrained; it slows training slightly but allows larger batch sizes.
# SYNTHESIS RECIPE SCHEMA
Source: https://docs.oumi.ai/reference/schema/datasets
Request schema for dataset synthesis recipes
## OVERVIEW
A synthesis recipe defines how Oumi generates a dataset. This page describes the request body used to create one.
For complete general and improved-sample requests, see [Synthesis recipes](/reference/recipes/synthesis).
***
## REQUEST
```json theme={null}
{
"displayName": "",
"description": "",
"recipeConfig": {
"type": "synthesize",
"modelIdentifier": {},
"inferenceConfig": {},
"synthesisConfig": {},
"datasetDistribution": {},
"environments": []
}
}
```
| Field | Type | Required | Description |
| ------------ | ------ | -------- | ---------------------------------- |
| displayName | string | ✓ | Display name for the recipe |
| description | string | | Description of the initial version |
| recipeConfig | object | ✓ | Dataset synthesis configuration |
***
## RECIPECONFIG
| Field | Type | Required | Description |
| ------------------- | --------- | -------- | --------------------------------------------------------------- |
| type | string | ✓ | Must be `synthesize` |
| modelIdentifier | object | ✓ | Hosted model used for synthesis |
| inferenceConfig | object | ✓ | Inference settings |
| synthesisConfig | object | ✓ | Configuration selected by `synthesisType` |
| datasetId | integer | | Existing dataset that receives the generated training data |
| datasetDistribution | object | | Output split fractions; defaults to `1.0` train |
| environments | object\[] | | Tool environments available during multi-turn general synthesis |
When `datasetId` is set, the validation and test fractions must be `0`. Each item in `environments` contains an `environmentId`. Environment tools require general synthesis with a `multiturnAttributes` entry.
***
## MODELIDENTIFIER
Synthesis accepts hosted models.
| Field | Type | Required | Description |
| --------- | ------ | -------- | ---------------------------------------------------------------- |
| modelType | string | ✓ | Hosted model provider |
| modelName | string | ✓ | Provider model name |
| apiKeys | object | | Provider API keys; Oumi does not bill usage made with these keys |
`modelType` accepts `ANTHROPIC_API`, `OPENAI_API`, `GEMINI_API`, `VERTEX_API`, or `OUMI_API`.
***
## INFERENCECONFIG
| Field | Type | Required | Description |
| --------------------- | ------- | -------- | -------------------------------------------- |
| inferenceTemperature | number | | Sampling temperature |
| inferenceMaxNewTokens | integer | | Maximum number of generated tokens |
| inferenceSeed | integer | | Seed for deterministic generation |
| requestsPerMinute | integer | | Request rate limit |
| guidedDecoding | object | | Not supported for synthesis; omit this field |
***
## SYNTHESISCONFIG
`synthesisType` selects one of the following configurations.
### GENERAL
```json theme={null}
{
"synthesisType": "general",
"synthesisConfig": {
"strategy": "general",
"numSamples": 100,
"strategyParams": {}
}
}
```
| Field | Type | Required | Description |
| -------------- | ------- | -------- | --------------------------------------------- |
| strategy | string | | Synthesis strategy; defaults to `general` |
| numSamples | integer | | Number of samples; defaults to `1` |
| outputPath | string | | Output JSONL path |
| strategyParams | object | | Inputs, generated fields, and transformations |
`strategyParams` supports these fields:
| Field | Type | Description |
| --------------------- | --------- | ---------------------------------------------------------------- |
| inputData | object\[] | Dataset or structured data source inputs |
| inputDocuments | object\[] | Document inputs |
| inputExamples | object\[] | Inline example inputs |
| sampledAttributes | object\[] | Attributes sampled from weighted values |
| combinationSampling | object\[] | Sampling rates for attribute combinations |
| generatedAttributes | object\[] | Attributes generated by the model |
| multiturnAttributes | object\[] | Multi-turn conversation configuration; accepts at most one entry |
| transformedAttributes | object\[] | String, list, dictionary, or chat transformations |
| passthroughAttributes | string\[] | Output fields to retain |
Each `inputData` item contains either `datasetId` or `datasourceId`. It may also contain `attributeMap`.
### DOCUMENT Q\&A
| Field | Type | Required | Description |
| ------------------ | ------- | -------- | --------------------------------------------- |
| synthesisType | string | ✓ | Must be `docqa` |
| documentTagId | integer | ✓ | Tag that identifies the source documents |
| numSamples | integer | ✓ | Number of samples |
| segmentationConfig | object | | Tokenizer, segment size, and overlap settings |
`segmentationConfig` accepts `segmentationType`, `tokenizer`, `segmentSize`, and `segmentOverlap`. The only current segmentation type is `tokens`.
### IMPROVED SAMPLES
| Field | Type | Required | Description |
| ------------------------ | ---------- | -------- | ----------------------------------------------------- |
| synthesisType | string | ✓ | Must be `generate_improved_samples` |
| evalId | integer | ✓ | Evaluation used to find failed data items |
| failureModeIds | integer\[] | | Failure modes to include |
| dataItemIds | string\[] | | Data items to include |
| numSamplesPerItem | integer | ✓ | Samples per item, from `1` to `20` |
| generateResponses | boolean | | Generate responses; defaults to `true` |
| generateNewConversations | boolean | | Generate new conversations; defaults to `true` |
| outputItemType | string | | `train_data` or `eval_data`; defaults to `train_data` |
When `generateNewConversations` is `false`, `numSamplesPerItem` is ignored.
***
## DATASETDISTRIBUTION
| Field | Type | Required | Range | Description |
| ---------- | ------ | -------- | ----- | ------------------------- |
| train | number | ✓ | 0–1 | Training split fraction |
| validation | number | ✓ | 0–1 | Validation split fraction |
| test | number | ✓ | 0–1 | Test split fraction |
The three fractions must sum to `1.0`.
***
## DOCUMENT Q\&A EXAMPLE
```json theme={null}
{
"displayName": "Product documentation Q&A",
"recipeConfig": {
"type": "synthesize",
"modelIdentifier": {
"modelType": "OUMI_API",
"modelName": "zai-org/GLM-5"
},
"inferenceConfig": {
"inferenceTemperature": 0.7,
"inferenceMaxNewTokens": 512
},
"synthesisConfig": {
"synthesisType": "docqa",
"documentTagId": 12,
"numSamples": 100,
"segmentationConfig": {
"segmentationType": "tokens",
"tokenizer": "openai-community/gpt2",
"segmentSize": 2048,
"segmentOverlap": 0
}
},
"datasetDistribution": {
"train": 0.8,
"validation": 0.1,
"test": 0.1
}
}
}
```
***
## VALIDATION RULES
* Synthesis accepts hosted model identifiers only.
* Positive integer IDs and sample counts must be greater than `0`.
* `numSamples` must cover every nonzero dataset split.
* `combinationSampling` rates must sum to no more than `1.0`.
* Environment tool names must be unique across selected environments.
* `guidedDecoding` is not supported for synthesis.
# EVALUATION RECIPE SCHEMA DEFINITIONS
Source: https://docs.oumi.ai/reference/schema/evaluations
Schema definitions for creating evaluations
## OVERVIEW
This document describes the schema used to configure an evaluation.
The recipe defines:
* The model being evaluated
* The evaluators (judges) used
* Inference settings
* The dataset used for evaluation
* Optional failure mode analysis
Fields marked with (\*) are **required**.
***
# SCHEMA STRUCTURE
```json theme={null}
{
"displayName": "",
"description": "",
"recipeConfig": {
"type": "evaluate",
"evaluationConfig": {}
}
}
```
***
# REQUEST
Request used to create an evaluation recipe.
| Field | Type | Required | Description |
| ------------ | ------ | -------- | ------------------------------------------- |
| displayName | string | ✓ | Display name of the recipe |
| description | string | | Description of the initial recipe version |
| recipeConfig | object | ✓ | Defines the evaluation recipe configuration |
***
# RECIPECONFIG \*
Configuration describing the recipe.
| Field | Type | Required | Description |
| ---------------- | ------ | -------- | ------------------------ |
| type | string | ✓ | Type of recipe |
| evaluationConfig | object | ✓ | Evaluation configuration |
## ALLOWED VALUES
| Field | Value |
| ----- | ------------ |
| type | `"evaluate"` |
***
# EVALUATIONCONFIG \*
Defines how the model evaluation should run.
| Field | Type | Required | Description |
| -------------------- | --------- | -------- | -------------------------------------- |
| evaluationType | string | ✓ | Evaluation strategy |
| modelIdentifier | object | ✓ | Model to evaluate |
| evaluators | object\[] | ✓ | List of evaluators to run |
| inferenceConfig | object | ✓ | Inference configuration |
| dataset | object | ✓ | Dataset used for evaluation |
| generateFailureModes | boolean | ✓ | Whether to analyze evaluation failures |
***
## EVALUATIONTYPE
Defines the evaluation approach.
| Allowed Values |
| ---------------- |
| `"single_model"` |
***
# MODELIDENTIFIER \*
Specifies the **model being evaluated**.
| Field | Type | Required | Description |
| -------------- | ------ | -------- | ------------------------------------------ |
| modelType | enum | ✓ | Model provider type |
| modelName | string | ✓ | Model name or identifier |
| modelId | number | | Platform model ID (custom models) |
| modelVersionId | number | | Specific model version (latest if omitted) |
| apiKeys | object | | Optional API keys |
***
## MODELTYPE
Supported model providers.
| Value | Description |
| ---------------------- | --------------------------------------- |
| `CUSTOM_CLOUD_STORAGE` | Custom model stored in platform storage |
| `ANTHROPIC_API` | Anthropic API model |
| `OPENAI_API` | OpenAI API model |
| `GEMINI_API` | Google Gemini API model |
| `VERTEX_API` | Google Vertex AI model |
| `OUMI_API` | Oumi API model |
***
## APIKEYS
Optional API credentials if not using platform credentials.
| Field | Type | Description |
| ------------ | ------ | ------------------------ |
| anthropic | string | Anthropic API key |
| openai | string | OpenAI API key |
| googleGemini | string | Google Gemini API key |
| googleVertex | string | Google Vertex AI API key |
***
# EVALUATORS \*
List of evaluators used during evaluation.
| Field | Type | Required | Description |
| ---------------- | ------ | -------- | ------------------------------------- |
| evaluatorId | number | ✓ | ID of the evaluator |
| evaluatorVersion | number | | Evaluator version (latest if omitted) |
### EXAMPLE
```json theme={null}
{
"evaluators": [
{
"evaluatorId": 10,
"evaluatorVersion": 2
}
]
}
```
***
# INFERENCECONFIG \*
Defines inference parameters for the **evaluated model**.
| Field | Type | Required | Description |
| --------------------- | ------ | -------- | ------------------------------- |
| inferenceTemperature | number | | Sampling temperature |
| inferenceMaxNewTokens | number | | Maximum tokens generated |
| inferenceSeed | number | | Random seed for reproducibility |
| requestsPerMinute | number | | API rate limit |
***
# DATASET \*
Defines the dataset used for evaluation.
| Field | Type | Required | Description |
| -------------- | ------ | -------- | ----------------------------------- |
| datasetId | number | ✓ | Dataset identifier |
| datasetVersion | number | | Dataset version (latest if omitted) |
***
# GENERATEFAILUREMODES \*
Controls failure analysis generation.
| Value | Description |
| ------- | ------------------------------------- |
| `true` | Analyze and categorize model failures |
| `false` | Skip failure mode analysis |
Failure modes may include:
* hallucination
* incorrect reasoning
* missing information
* formatting errors
***
# COMPLETE EXAMPLE
```json theme={null}
{
"displayName": "External model evaluation",
"recipeConfig": {
"type": "evaluate",
"evaluationConfig": {
"evaluationType": "single_model",
"modelIdentifier": {
"modelType": "OPENAI_API",
"modelName": "gpt-4.1"
},
"evaluators": [
{
"evaluatorId": 12
}
],
"inferenceConfig": {
"inferenceTemperature": 0.2,
"inferenceMaxNewTokens": 512
},
"dataset": {
"datasetId": 42
},
"generateFailureModes": true
}
}
}
```
***
# VALIDATION RULES
* You must provide at least one evaluator
* `recipeConfig.type` must equal `evaluate`
* `evaluationType` must equal `single_model`
* Hosted and pretrained models require `modelName`
* Custom models require `modelId`
* `modelType` must match one of the supported providers
* `datasetId` must reference a valid dataset
* `generateFailureModes` must be a boolean
# EVALUATOR RECIPE SCHEMA DEFINITIONS
Source: https://docs.oumi.ai/reference/schema/evaluators
Schema definitions for configuring evaluators
## OVERVIEW
The following describes the request schema used to create judge evaluators in Oumi, including:
* Metadata about the evaluator
* Evaluation parameters
* Model configuration
* Scoring settings
Fields marked with (\*) are **required**.
***
## SCHEMA STRUCTURE
```json theme={null}
{
"displayName": "",
"description": "",
"params": {
"evaluatorType": "judge",
"prompt": "",
"modelIdentifier": {},
"inferenceConfig": {}
}
}
```
***
# DISPLAYNAME \*
Human-readable name of the evaluator.
## PROPERTIES
| Field | Type | Required | Description |
| ----------- | ------ | -------- | ------------------------------------- |
| displayName | string | ✓ | Name used to identify the evaluator |
| description | string | | Optional description of the evaluator |
### CONSTRAINTS
| Field | Constraint |
| ----------- | ------------------- |
| displayName | Minimum length: `1` |
### EXAMPLE
```json theme={null}
{
"displayName": "Answer Quality Evaluator",
"description": "Evaluates model responses using a judge model."
}
```
***
# JUDGE PARAMS \*
Defines the evaluation parameters and scoring behavior.
## PROPERTIES
| Field | Type | Required | Description |
| ------------------------- | ------- | -------- | ---------------------------------------------------- |
| evaluatorType | string | ✓ | Evaluator type |
| prompt | string | ✓ | Prompt used to guide the judge |
| modelIdentifier | object | ✓ | Hosted model used by the judge |
| inferenceConfig | object | ✓ | Inference settings for the judge model |
| judgmentScores | object | | Judgment labels mapped to scores from `0.0` to `1.0` |
| dataFields | object | | Dataset field names mapped to display titles |
| responseFilterMode | string | | Portion of the response available to the judge |
| includeConversationSystem | boolean | | Includes the system message in the judge input |
***
## EVALUATORTYPE
Defines the evaluator mechanism.
| Allowed Values |
| -------------- |
| `"judge"` |
***
## PROMPT \*
Prompt provided to the evaluator model to guide scoring behavior.
### CONSTRAINTS
| Field | Constraint |
| ------ | ------------------- |
| prompt | Minimum length: `1` |
### EXAMPLE
```json theme={null}
{
"params": {
"evaluatorType": "judge",
"prompt": "Evaluate whether the assistant's response is accurate and helpful."
}
}
```
***
## JUDGMENTSCORES
Maps each allowed judgment label to a score from `0.0` to `1.0`. The example below defines a boolean judge.
Example structure:
```json theme={null}
{
"judgmentScores": {
"Yes": 1.0,
"No": 0.0
}
}
```
***
## DATAFIELDS
Defines the dataset fields that are used by the evaluator.
Every evaluation example must include each configured field in its metadata.
Example structure:
```json theme={null}
{
"dataFields": {
"ground_truth_answer": "Reference answer"
}
}
```
***
# MODELIDENTIFIER \*
Defines the model used by the evaluator.
This object follows the **Model Identifier schema**.
Example:
```json theme={null}
{
"modelIdentifier": {
"modelType": "OUMI_API",
"modelName": "zai-org/GLM-5"
}
}
```
***
# INFERENCECONFIG \*
Defines runtime inference behavior for the evaluator model.
Example:
```json theme={null}
{
"inferenceConfig": {
"inferenceTemperature": 0.0,
"inferenceMaxNewTokens": 8192
}
}
```
***
# RESPONSEFILTERMODE
Controls which parts of the model output are used for evaluation.
The default is `EXCLUDE_THINKING`.
| Value | Description |
| ----------------------- | ------------------------------------------ |
| `THINKING_AND_RESPONSE` | Includes both reasoning and final response |
| `EXCLUDE_THINKING` | Uses only the final response (default) |
| `THINKING_ONLY` | Uses only the reasoning output |
***
# COMPLETE EXAMPLE
```json theme={null}
{
"displayName": "Answer Quality Evaluator",
"description": "Evaluates responses using a judge model",
"params": {
"evaluatorType": "judge",
"prompt": "Compare the assistant response with the Reference answer. Return Yes if the response is accurate and helpful. Otherwise, return No.",
"judgmentScores": {
"Yes": 1.0,
"No": 0.0
},
"dataFields": {
"ground_truth_answer": "Reference answer"
},
"modelIdentifier": {
"modelType": "OUMI_API",
"modelName": "zai-org/GLM-5"
},
"inferenceConfig": {
"inferenceTemperature": 0.0,
"inferenceMaxNewTokens": 8192
},
"responseFilterMode": "EXCLUDE_THINKING"
}
}
```
# MODEL TRAINING RECIPE SCHEMA DEFINITIONS
Source: https://docs.oumi.ai/reference/schema/models
Schema definitions for training models
## OVERVIEW
The following defines the configuration schema used for training models in Oumi.
The configuration includes:
* Model loading configuration
* Dataset configuration
* Training parameters
* Logging and evaluation settings
* Parameter-efficient fine-tuning (PEFT)
Fields marked with (\*) are **required**.
***
# SCHEMA STRUCTURE
```json theme={null}
{
"model": {},
"data": {},
"training": {},
"peft": {}
}
```
***
# MODEL \*
Defines the model and tokenizer configuration.
| Field | Type | Required | Description |
| ------------------ | ------- | -------- | --------------------------------------------------- |
| modelName | string | ✓ | HuggingFace model name or path |
| tokenizerName | string | | Tokenizer name (defaults to model name) |
| tokenizerPadToken | string | | Override pad token for tokenizer |
| modelMaxLength | number | | Maximum sequence length |
| torchDtype | string | | PyTorch dtype (e.g. `float16`, `bfloat16`) |
| trustRemoteCode | boolean | | Allow execution of custom model repo code |
| attnImplementation | string | | Attention implementation (e.g. `flash_attention_2`) |
| chatTemplate | string | | Custom Jinja chat template |
| enableLigerKernel | boolean | | Enable Liger fused kernel optimizations |
### CONSTRAINTS
| Field | Constraint |
| --------- | ------------------- |
| modelName | Minimum length: `1` |
### EXAMPLE
```json theme={null}
{
"model": {
"modelName": "meta-llama/Llama-3-8B",
"tokenizerName": "meta-llama/Llama-3-8B",
"torchDtype": "bfloat16",
"modelMaxLength": 4096
}
}
```
***
# DATA \*
Defines datasets used for training and evaluation.
| Field | Type | Required | Description |
| ---------- | ------ | -------- | -------------------------------- |
| train | object | ✓ | Training dataset configuration |
| validation | object | | Validation dataset configuration |
| test | object | | Test dataset configuration |
***
## TRAIN \*
Training dataset definition.
| Field | Type | Required | Description |
| -------- | --------- | -------- | ---------------- |
| datasets | object\[] | ✓ | List of datasets |
### CONSTRAINTS
| Field | Constraint |
| -------- | ------------------ |
| datasets | Exactly **1 item** |
***
### DATASET OBJECT
| Field | Type | Required | Description |
| --------- | ------- | -------- | ------------------ |
| datasetId | integer | ✓ | Dataset identifier |
### RANGE
| Field | Range |
| --------- | --------------------- |
| datasetId | (0..9007199254740991] |
***
## VALIDATION
Validation dataset configuration.
| Field | Type | Description |
| -------- | --------- | ------------------- |
| datasets | object\[] | Validation datasets |
***
## TEST
Test dataset configuration.
| Field | Type | Description |
| -------- | --------- | ------------- |
| datasets | object\[] | Test datasets |
***
# TRAINING \*
Defines model training parameters.
| Field | Type | Description |
| --------------------------- | ------- | ---------------------------------------------------- |
| maxSteps | number | Stop training after this many steps (-1 = unlimited) |
| numTrainEpochs | integer | Number of full passes over the dataset |
| learningRate | number | Peak learning rate |
| lrSchedulerType | enum | Learning rate scheduler |
| warmupRatio | number | Fraction of warmup steps |
| warmupSteps | number | Warmup steps (overrides ratio) |
| optimizer | string | Optimizer name |
| weightDecay | number | L2 regularization |
| adamBeta1 | number | Adam optimizer beta1 |
| adamBeta2 | number | Adam optimizer beta2 |
| maxGradNorm | number | Gradient clipping value |
| mixedPrecisionDtype | enum | Mixed precision training dtype |
| enableGradientCheckpointing | boolean | Reduce memory usage |
| loggingSteps | number | Log every N steps |
| evalStrategy | enum | Evaluation schedule |
| evalSteps | number | Eval interval |
| saveSteps | number | Checkpoint interval |
| saveFinalModel | boolean | Save model after training |
| trainerType | enum | Training algorithm |
| seed | number | Random seed |
| enableWandb | boolean | Enable Weights & Biases logging |
| enableTensorboard | boolean | Enable TensorBoard logging |
| runName | string | Display name for training run |
***
## PARAMETER CONSTRAINTS
| Field | Range |
| -------------- | -------------- |
| numTrainEpochs | 1–30 |
| learningRate | 0.000001–0.001 |
| runName | length 1–128 |
| optimizer | length 1–128 |
***
## LRSCHEDULERTYPE
| Allowed Values |
| ---------------------- |
| `linear` |
| `cosine` |
| `cosine_with_restarts` |
| `cosine_with_min_lr` |
| `constant` |
***
## MIXEDPRECISIONDTYPE
| Allowed Values |
| -------------- |
| `none` |
| `fp16` |
| `bf16` |
***
## EVALSTRATEGY
| Allowed Values |
| -------------- |
| `no` |
| `steps` |
| `epoch` |
***
## TRAINERTYPE
| Allowed Values |
| -------------- |
| `sft` |
| `opd` |
***
# ONPOLICYDISTILLATION
Optional configuration for **on-policy distillation**.
> Requires `trainerType = "opd"`.
***
# PEFT \*
Parameter-efficient fine-tuning configuration.
| Field | Type | Required | Description |
| ---------- | ------- | -------- | ------------------------- |
| usePeft | boolean | ✓ | Enable PEFT |
| peftMethod | string | ✓ | PEFT method (e.g. `lora`) |
| loraR | num | | |
# SUPPORTED MODELS
Source: https://docs.oumi.ai/reference/supported-models
Models available for training and inference in Oumi
The following is a list of models available on the Oumi platform, organized by use case, with more supported models added continuously.
For the most up-to-date list, ask the Oumi Agent with this prompt:
```prompt theme={null}
List all currently supported Oumi models, organized by use case, including any newly added models.
```
***
## TRAINING (FINE-TUNING BASE MODELS)
Oumi supports a broad set of models that can be fine-tuned on your own data, with each model offering different training approaches and requirements depending on its architecture and capabilities.
| Model | Training methods |
| ----------------------------------------- | -------------------- |
| Qwen/Qwen3.5-9B | SFT (FFT) |
| Qwen/Qwen3.5-9B-Base | SFT (FFT) |
| Qwen/Qwen3.5-4B | SFT (FFT) |
| Qwen/Qwen3.5-4B-Base | SFT (FFT) |
| Qwen/Qwen3.5-2B | SFT (FFT) |
| Qwen/Qwen3.5-0.8B | SFT (FFT) |
| Qwen/Qwen3-32B | SFT (LoRA) |
| Qwen/Qwen3-8B | SFT (LoRA, FFT), OPD |
| Qwen/Qwen3-4B-Instruct-2507 | SFT (LoRA, FFT), OPD |
| Qwen/Qwen3-0.6B | SFT (LoRA, FFT), OPD |
| Qwen/Qwen2.5-7B-Instruct | SFT (LoRA, FFT), OPD |
| Qwen/Qwen2.5-7B | SFT (LoRA, FFT), OPD |
| Qwen/Qwen2.5-3B-Instruct | SFT (LoRA, FFT), OPD |
| Qwen/Qwen2.5-1.5B-Instruct | SFT (LoRA, FFT), OPD |
| meta-llama/Llama-4-Scout-17B-16E-Instruct | SFT (LoRA) |
| meta-llama/Llama-3.1-8B-Instruct | SFT (LoRA, FFT), OPD |
| meta-llama/Llama-3.2-3B-Instruct | SFT (LoRA, FFT), OPD |
| meta-llama/Llama-3.2-1B-Instruct | SFT (LoRA, FFT), OPD |
| google/gemma-3-4b-it | SFT (LoRA, FFT) |
| microsoft/Phi-3.5-mini-instruct | SFT (LoRA, FFT) |
| microsoft/Phi-3.5-MoE-instruct | SFT (LoRA) |
| openai/gpt-oss-20b | SFT (FFT) |
| HuggingFaceTB/SmolLM2-1.7B-Instruct | SFT (LoRA, FFT) |
| HuggingFaceTB/SmolLM2-360M-Instruct | SFT (LoRA, FFT), OPD |
| HuggingFaceTB/SmolLM2-135M-Instruct | SFT (LoRA, FFT), OPD |
| HuggingFaceTB/SmolLM-1.7B-Instruct | SFT (LoRA, FFT) |
| HuggingFaceTB/SmolLM-360M-Instruct | SFT (LoRA, FFT) |
| HuggingFaceTB/SmolLM-135M-Instruct | SFT (LoRA, FFT) |
***
## INFERENCE (EVALUATION, SYNTHESIS, AND COMPLETIONS)
The following models are available for evaluation judges, data synthesis, and generating completions.
External models are available without adding a provider API key. When you use your own key from [Provider API keys](/admin/api-keys), Oumi does not bill you for that model usage.
| Model | Provider |
| ------------------------------------ | -------- |
| Qwen/Qwen3.5-397B-A17B | oumi |
| Qwen/Qwen3.5-9B | oumi |
| Qwen/Qwen3-235B-A22B-Instruct-2507 | oumi |
| deepseek-ai/DeepSeek-V3.1 | oumi |
| moonshotai/Kimi-K2.5 | oumi |
| openai/gpt-oss-120b | oumi |
| zai-org/GLM-5.1 | oumi |
| zai-org/GLM-5 | oumi |
| zai-org/GLM-4.6 | oumi |
| meta-llama/Llama-3.3-70B-Instruct | oumi |
| Qwen/Qwen2.5-7B-Instruct | oumi |
| mistralai/Mixtral-8x7B-Instruct-v0.1 | oumi |
All models available as a training base can also be used for evaluation.
Can’t find the model you’re looking for? Reach out to us at [support@oumi.ai](mailto:support@oumi.ai) with your request.