| Command | Description |
|---|---|
create | Create an evaluator in a project from a JSON request body. |
delete | Permanently delete an evaluator. |
download | Download an evaluator’s config as an editable JSON file. |
generate-template | Print a ready-to-edit evaluator request body for the given evaluator type. |
get | Show an evaluator’s type, config and current version. |
list | List evaluators in the project. |
preview | Score an evaluator config against a handful of rows without running an evaluation. |
update | Rename an evaluator, or replace its config from a JSON body. |
versions get | Show a single version of an evaluator, including its full config. |
versions list | List version history for an evaluator, newest first by default. |
CREATE
Create an evaluator in a project from a JSON request body.oumi-cli evaluators create [OPTIONS]
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or the active profile. | |
--input-json | text | Request body for the new evaluator, as inline JSON; see oumi-cli evaluators generate-template --type <type> for its shape. Pass exactly one of --input-json or --input-json-file. | |
--input-json-file | path | Path to a JSON file with the request body for the new evaluator. Pass exactly one of --input-json or --input-json-file. |
DELETE
Permanently delete an evaluator.oumi-cli evaluators delete [OPTIONS] EVALUATOR_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | integer | ✓ | ID of the evaluator (see oumi-cli evaluators list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or the active profile. | |
--force | flag | Skip the confirmation prompt. Without a TTY or under --no-input, this (or --yes) is required; otherwise the command exits 2 without changing anything. | |
--yes | flag | Same as --force. |
DOWNLOAD
Download an evaluator’s config as an editable JSON file.oumi-cli evaluators download [OPTIONS] EVALUATOR_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | integer | ✓ | ID of the evaluator (see oumi-cli evaluators list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or the active profile. | |
--out | path | Destination file or directory; defaults to ./evaluator-<id>.json. Refuses to overwrite an existing file (exit 6). |
GENERATE-TEMPLATE
Print a ready-to-edit evaluator request body for the given evaluator type.oumi-cli evaluators generate-template [OPTIONS]
| Option | Type | Required | Description |
|---|---|---|---|
--type | choice | ✓ | Evaluator kind to generate a template for (judge, rubric_judge, classification or extraction); sets params.evaluatorType in the output. |
--minimal | flag | Emit a small, filled-in worked example for a built-in judge type instead of the full schema. Only supported with --type judge. |
GET
Show an evaluator’s type, config and current version.oumi-cli evaluators get [OPTIONS] EVALUATOR_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | integer | ✓ | ID of the evaluator (see oumi-cli evaluators list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or the active profile. |
LIST
List evaluators in the project.oumi-cli evaluators list [OPTIONS]
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project to list evaluators from; falls back to OUMI_PROJECT_ID or the active profile. | |
--limit | integer range | Maximum number of evaluators to return (default 100). | |
--all | flag | Return all evaluators, following pages until none are left (capped by --max-items). Cannot be combined with --limit. | |
--max-items | integer range | Cap on how many evaluators --all returns (default 10000). | |
--search | text | Only show evaluators whose name matches this search text. | |
--sort-by | choice | Field to sort by. One of: created_at, updated_at. | |
--sort-direction | choice | Sort order (asc is oldest-first). |
PREVIEW
Score an evaluator config against a handful of rows without running an evaluation.oumi-cli evaluators preview [OPTIONS]
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project to run the preview in; falls back to OUMI_PROJECT_ID or the active profile. | |
--input-json | text | Evaluator request body to score, as inline JSON (same shape evaluators create takes; see oumi-cli evaluators generate-template --type <type>). Pass exactly one of --input-json or --input-json-file. | |
--input-json-file | path | Path to a JSON file with the evaluator request body to score. Pass exactly one of --input-json or --input-json-file. | |
--sample | path | Local JSONL file of 1-20 conversations to score, one JSON object per line with a “messages” list and optional “metadata”. Cannot be combined with --dataset/--data-item-id. | |
--dataset | integer | Dataset the --data-item-id values belong to. Judge and rubric_judge only; requires --data-item-id; mutually exclusive with --sample. | |
--data-item-id | text | Data item to score (repeatable, 1-20), from oumi-cli datasets items list. Judge and rubric_judge only; requires --dataset; mutually exclusive with --sample. |
UPDATE
Rename an evaluator, or replace its config from a JSON body.oumi-cli evaluators update [OPTIONS] EVALUATOR_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | integer | ✓ | ID of the evaluator (see oumi-cli evaluators list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or the active profile. | |
--display-name | text | New display name for the evaluator. Mutually exclusive with --input-json*. | |
--input-json | text | Request body with the full new config, as inline JSON; saved as a new version (see oumi-cli evaluators generate-template --type <type> for its shape). Cannot be combined with --display-name. | |
--input-json-file | path | Path to a JSON file with the full new config, saved as a new version (evaluators download writes one). Cannot be combined with --display-name. |
VERSIONS GET
Show a single version of an evaluator, including its full config.oumi-cli evaluators versions get [OPTIONS] EVALUATOR_ID VERSION_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | integer | ✓ | ID of the evaluator (see oumi-cli evaluators list). |
VERSION_ID | integer | ✓ | ID of the version to inspect (see evaluators versions list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or the active profile. |
VERSIONS LIST
List version history for an evaluator, newest first by default.oumi-cli evaluators versions list [OPTIONS] EVALUATOR_ID
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | integer | ✓ | ID of the evaluator (see oumi-cli evaluators list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | text | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or the active profile. | |
--limit | integer range | Maximum number of versions to return (default 100). | |
--all | flag | Return all versions, following pages until none are left (capped by --max-items). Cannot be combined with --limit. | |
--max-items | integer range | Cap on how many versions --all returns (default 10000). | |
--sort-by | choice | Field to sort by. One of: created_at, updated_at, version_id. | |
--sort-direction | choice | Sort order (asc is oldest-first). |