| Command | Description |
|---|---|
available-models | List the models an evaluation recipe can score: your project’s models plus hosted models. |
delete | Permanently delete an evaluation. |
download | Download every evaluation data item to a local JSONL file. |
failure-modes generate | Generate failure modes from an evaluation’s results using LLM analysis. |
failure-modes get | Show one failure mode by ID. |
failure-modes get-items | List the sample rows behind a failure mode. |
failure-modes list | List the failure modes for a completed evaluation. |
generate-template | Print a ready-to-edit evaluation recipe request body. |
get | Show an evaluation with its scores and per-evaluator detail. |
list | List the evaluations in a project, optionally searched by name and sorted. |
results | Show per-row evaluation results, one verdict column per evaluator. |
run | Submit an evaluation from a recipe and wait for it to finish. |
update | Rename an evaluation. |
AVAILABLE-MODELS
List the models an evaluation recipe can score: your project’s models plus hosted models.oumi-cli evaluations available-models [OPTIONS]
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project whose models to list; falls back to OUMI_PROJECT_ID or the active profile. | |
--search | text | Filter by display name / model name, case-insensitive substring match, applied client-side after merging. | |
--limit | integer range | Maximum number of models to return (default 100). | |
--all | flag | Return all models (capped by --max-items). Cannot be combined with --limit. | |
--max-items | integer range | Cap on how many models --all returns (default 10000). |
DELETE
Permanently delete an evaluation.oumi-cli evaluations delete [OPTIONS] EVALUATION_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATION_ID | integer | ✓ | ID of the evaluation (see oumi-cli evaluations list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile. | |
--force | flag | Skip the confirmation prompt. Without a TTY or under --no-input, this (or --yes) is required; otherwise the command exits 2 without changing anything. | |
--yes | flag | Same as --force. |
DOWNLOAD
Download every evaluation data item to a local JSONL file.oumi-cli evaluations download [OPTIONS] EVALUATION_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATION_ID | integer | ✓ | ID of the evaluation (see oumi-cli evaluations list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile. | |
--out | path | Destination file or directory; defaults to ./evaluation-<id>-dataitems.jsonl. Refuses to overwrite an existing file (exit 6). |
FAILURE-MODES GENERATE
Generate failure modes from an evaluation’s results using LLM analysis.oumi-cli evaluations failure-modes generate [OPTIONS] EVALUATION_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATION_ID | integer | ✓ | ID of the evaluation (see oumi-cli evaluations list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile. | |
--wait / --no-wait | flag | Wait for generation to finish before returning (default); --no-wait returns the operation ID immediately. |
FAILURE-MODES GET
Show one failure mode by ID.oumi-cli evaluations failure-modes get [OPTIONS] FAILURE_MODE_ID
| Argument | Type | Required | Description |
|---|---|---|---|
FAILURE_MODE_ID | integer | ✓ | ID of the failure mode (see evaluations failure-modes list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile. | |
--include-items | flag | Include the sample rows in the response, as raw JSON. Use get-items for the same rows as a table. |
FAILURE-MODES GET-ITEMS
List the sample rows behind a failure mode.oumi-cli evaluations failure-modes get-items [OPTIONS] FAILURE_MODE_ID
| Argument | Type | Required | Description |
|---|---|---|---|
FAILURE_MODE_ID | integer | ✓ | ID of the failure mode (see evaluations failure-modes list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile. | |
--limit | integer range | Maximum number of items to return (default 100). | |
--all | flag | Return all items, following pages until none are left (capped by --max-items). Cannot be combined with --limit. | |
--max-items | integer range | Cap on how many items --all returns (default 10000). |
FAILURE-MODES LIST
List the failure modes for a completed evaluation.oumi-cli evaluations failure-modes list [OPTIONS] EVALUATION_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATION_ID | integer | ✓ | ID of the evaluation (see oumi-cli evaluations list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile. | |
--include-items | flag | Include per-item failure-mode detail in the response. | |
--limit | integer range | Maximum number of failure modes to return (default 100). | |
--all | flag | Return all failure modes, following pages until none are left (capped by --max-items). Cannot be combined with --limit. | |
--max-items | integer range | Cap on how many failure modes --all returns (default 10000). |
GENERATE-TEMPLATE
Print a ready-to-edit evaluation recipe request body.oumi-cli evaluations generate-template [OPTIONS]
GET
Show an evaluation with its scores and per-evaluator detail.oumi-cli evaluations get [OPTIONS] EVALUATION_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATION_ID | integer | ✓ | ID of the evaluation (see oumi-cli evaluations list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile. |
LIST
List the evaluations in a project, optionally searched by name and sorted.oumi-cli evaluations list [OPTIONS]
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project to list evaluations from; falls back to OUMI_PROJECT_ID or the active profile. | |
--limit | integer range | Maximum number of evaluations to return (default 100). | |
--all | flag | Return all evaluations, following pages until none are left (capped by --max-items). Cannot be combined with --limit. | |
--max-items | integer range | Cap on how many evaluations --all returns (default 10000). | |
--search | text | Only show evaluations whose display name contains this text. | |
--sort-by | choice | Field to sort by. One of: created_at, updated_at. | |
--sort-direction | choice | Sort order (asc is oldest-first). |
RESULTS
Show per-row evaluation results, one verdict column per evaluator.oumi-cli evaluations results [OPTIONS] EVALUATION_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATION_ID | integer | ✓ | ID of the evaluation (see oumi-cli evaluations list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile. | |
--evaluator | integer | Only show this evaluator’s verdicts (evaluator ID). Applied after --limit, so add --all to search every row. | |
--id | text | Only show the row with this data item ID. Cannot be combined with --limit, --all or --max-items. | |
--limit | integer range | Maximum number of data items to return (default 100). | |
--all | flag | Return all data items, following pages until none are left (capped by --max-items). Cannot be combined with --limit. | |
--max-items | integer range | Cap on how many data items --all returns (default 10000). |
RUN
Submit an evaluation from a recipe and wait for it to finish.oumi-cli evaluations run [OPTIONS]
| Option | Type | Required | Description |
|---|---|---|---|
--name | text | ✓ | Human-readable display name for this evaluation run. |
--recipe, -r | text | ✓ | Recipe ID (integer) or path to a JSON recipe file (type ‘evaluate’); start from oumi-cli evaluations generate-template. |
--project | text | Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile. | |
--wait / --no-wait | flag | Wait for the run to finish and print the evaluation (default); --no-wait prints the operation ID immediately. |
UPDATE
Rename an evaluation.oumi-cli evaluations update [OPTIONS] EVALUATION_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATION_ID | integer | ✓ | ID of the evaluation (see oumi-cli evaluations list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile. | |
--display-name | text | New display name for the evaluation. |