EVALUATIONS

Command	Description
`available-models`	List every modelIdentifier candidate for an evaluation recipe in one call.
`delete`	Delete an evaluation by ID (prompts to confirm unless `--yes`/`--force`).
`download`	Download every evaluation data item to a local JSONL file.
`failure-modes generate`	Generate failure modes from an evaluation’s results using LLM analysis.
`failure-modes get`	Show one failure mode by ID.
`failure-modes get-items`	List the sample rows behind a failure mode.
`failure-modes list`	List the failure modes for a completed evaluation.
`generate-template`	Print a ready-to-edit evaluation recipe request body.
`get`	Show one evaluation by ID, with scores and per-evaluator detail.
`list`	List evaluations in a project, with optional filter, search, and sort.
`results`	Show per-row evaluation results, one verdict column per evaluator.
`run`	Submit an evaluation from a recipe and wait for it to finish.
`update`	Rename an evaluation.

AVAILABLE-MODELS

List every modelIdentifier candidate for an evaluation recipe in one call.

oumi-cli evaluations available-models [OPTIONS]

Options

Option	Type	Description
`--project`	str	Project to scope the operation to; falls back to OUMI_PROJECT_ID or the active profile.
`--search`	str	Filter by display name / model name, case-insensitive substring match, applied client-side after merging.
`--limit`	int range	Return at most N rows (default 100), applied to the merged candidate list.
`--all`	flag	Return every candidate (bounded by `--max-items`). Mutually exclusive with `--limit`.
`--max-items`	int range	Hard safety cap on merged rows returned, in any mode (default 10000).

DELETE

Delete an evaluation by ID (prompts to confirm unless --yes/--force).

oumi-cli evaluations delete [OPTIONS] {evaluation_id}

Arguments

Argument	Type	Required	Description
`EVALUATION_ID`	int	✓	ID of the evaluation (see `oumi-cli evaluations list`).

Options

Option	Type	Description
`--project`	str	Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile.
`--force`	flag	Delete without the confirmation prompt.
`--yes`	flag	Delete without the confirmation prompt.

DOWNLOAD

Download every evaluation data item to a local JSONL file.

oumi-cli evaluations download [OPTIONS] {evaluation_id}

Arguments

Argument	Type	Required	Description
`EVALUATION_ID`	int	✓	ID of the evaluation whose data items to download (see `evaluations list`).

Options

Option	Type	Required	Description
`--project`	str		Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile.
`--out`	path		Destination file or directory. Defaults to ./evaluation-<id>-dataitems.jsonl in the current directory.

FAILURE-MODES GENERATE

Generate failure modes from an evaluation’s results using LLM analysis.

oumi-cli evaluations failure-modes generate [OPTIONS] {evaluation_id}

Arguments

Argument	Type	Required	Description
`EVALUATION_ID`	int	✓	ID of the evaluation to analyze (see `evaluations list`).

Options

Option	Type	Required	Description
`--project`	str		Project ID. Env: `OUMI_PROJECT_ID`.
`--wait` / `--no-wait`	flag		Wait for generation to finish before returning (default); `--no-wait` returns the operation ID immediately.

FAILURE-MODES GET

Show one failure mode by ID.

oumi-cli evaluations failure-modes get [OPTIONS] {failure_mode_id}

Arguments

Argument	Type	Required	Description
`FAILURE_MODE_ID`	int	✓	ID of the failure mode (see `evaluations failure-modes list`).

Options

Option	Type	Required	Description
`--project`	str		Project ID. Env: `OUMI_PROJECT_ID`.
`--include-items`	flag		Include the sample rows in the response, as raw JSON. Use `get-items` for the same rows as a table.

FAILURE-MODES GET-ITEMS

List the sample rows behind a failure mode.

oumi-cli evaluations failure-modes get-items [OPTIONS] {failure_mode_id}

Arguments

Argument	Type	Required	Description
`FAILURE_MODE_ID`	int	✓	ID of the failure mode (see `evaluations failure-modes list`).

Options

Option	Type	Description
`--project`	str	Project ID. Env: `OUMI_PROJECT_ID`.
`--limit`	int range	Return at most N items (default 100). The CLI fetches and loops over pages internally; you never pass a page token.
`--all`	flag	Return every item, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`.
`--max-items`	int range	Hard safety cap for `--all` (default 10000).

FAILURE-MODES LIST

List the failure modes for a completed evaluation.

oumi-cli evaluations failure-modes list [OPTIONS] {evaluation_id}

Arguments

Argument	Type	Required	Description
`EVALUATION_ID`	int	✓	The ID of the evaluation.

Options

Option	Type	Description
`--project`	str	Project ID. Env: `OUMI_PROJECT_ID`.
`--include-items`	flag	Include per-item failure-mode detail in the response.
`--limit`	int range	Return at most N failure modes (default 100). The CLI fetches and loops over pages internally; you never pass a page token.
`--all`	flag	Return every failure mode, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`.
`--max-items`	int range	Hard safety cap for `--all` (default 10000).

GENERATE-TEMPLATE

Print a ready-to-edit evaluation recipe request body.

oumi-cli evaluations generate-template [OPTIONS]

GET

Show one evaluation by ID, with scores and per-evaluator detail.

oumi-cli evaluations get [OPTIONS] {evaluation_id}

Arguments

Argument	Type	Required	Description
`EVALUATION_ID`	int	✓	ID of the evaluation (see `oumi-cli evaluations list`).

Options

Option	Type	Required	Description
`--project`	str		Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile.

LIST

List evaluations in a project, with optional filter, search, and sort.

oumi-cli evaluations list [OPTIONS]

Options

Option	Type	Description
`--project`	str	Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile.
`--limit`	int range	Return at most N evaluations (default 100). The CLI fetches and loops over pages internally; you never pass a page token.
`--all`	flag	Return every evaluation, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`.
`--max-items`	int range	Absolute cap on rows returned, in every mode (default 10000).
`--search`	str	Keep only evaluations whose display name contains this substring.
`--sort-by`	choice	Field to sort by. One of: `created_at`, `updated_at`.
`--sort-direction`	choice	Sort order (asc is oldest-first).

RESULTS

Show per-row evaluation results, one verdict column per evaluator.

oumi-cli evaluations results [OPTIONS] {evaluation_id}

Arguments

Argument	Type	Required	Description
`EVALUATION_ID`	int	✓	The ID of the evaluation.

Options

Option	Type	Description
`--project`	str	Project ID. Env: `OUMI_PROJECT_ID`.
`--evaluator`	int	Filter rows to a single evaluator’s results (by evaluator ID); applied client-side after fetching.
`--id`	str	Filter to a single data item by its ID; applied client-side after fetching.
`--limit`	int range	Return at most N data items (default 100). The CLI fetches and loops over pages internally; you never pass a page token.
`--all`	flag	Return every data item, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`.
`--max-items`	int range	Hard safety cap for `--all` (default 10000).

RUN

Submit an evaluation from a recipe and wait for it to finish.

oumi-cli evaluations run [OPTIONS]

Options

Option	Type	Required	Description
`--name`	str	✓	Human-readable display name for this evaluation run.
`--recipe`, `-r`	str	✓	Recipe ID (integer) or path to a JSON recipe file (recipeConfig.type must be ‘evaluate’).
`--project`	str		Project to scope the operation to; falls back to OUMI_PROJECT_ID or the active profile.
`--wait` / `--no-wait`	flag		Wait for the run to finish before returning (default); `--no-wait` returns the operation ID immediately.

UPDATE

Rename an evaluation.

oumi-cli evaluations update [OPTIONS] {evaluation_id}

Arguments

Argument	Type	Required	Description
`EVALUATION_ID`	int	✓	ID of the evaluation to update (see `evaluations list`).

Options

Option	Type	Required	Description
`--project`	str		Project that owns the evaluation; falls back to OUMI_PROJECT_ID or the active profile.
`--display-name`	str		New display name for the evaluation.

Getting started

Basics

Workflows

Command reference

EVALUATIONS

AVAILABLE-MODELS

DELETE

DOWNLOAD

FAILURE-MODES GENERATE

FAILURE-MODES GET

FAILURE-MODES GET-ITEMS

FAILURE-MODES LIST

GENERATE-TEMPLATE

GET

LIST

RESULTS

RUN

UPDATE

​AVAILABLE-MODELS

​DELETE

​DOWNLOAD

​FAILURE-MODES GENERATE

​FAILURE-MODES GET

​FAILURE-MODES GET-ITEMS

​FAILURE-MODES LIST

​GENERATE-TEMPLATE

​GET

​LIST

​RESULTS

​RUN

​UPDATE

AVAILABLE-MODELS

DELETE

DOWNLOAD

FAILURE-MODES GENERATE

FAILURE-MODES GET

FAILURE-MODES GET-ITEMS

FAILURE-MODES LIST

GENERATE-TEMPLATE

GET

LIST

RESULTS

RUN

UPDATE