| Command | Description |
|---|---|
create | Create an evaluator in a project from a JSON request body. |
delete | Delete an evaluator by ID. |
download | Download an evaluator as an editable EvaluatorRequest body. |
generate-template | Print a ready-to-edit EvaluatorRequest body for the given evaluator type. |
get | Get an evaluator by ID. |
list | List evaluators in the project. |
update | Rename an evaluator, or replace its config from a JSON body. |
versions list | List version history for an evaluator, newest first by default. |
CREATE
Create an evaluator in a project from a JSON request body.oumi-cli evaluators create [OPTIONS]
| Option | Type | Required | Description |
|---|---|---|---|
--project | str | Project to create the evaluator in; falls back to OUMI_PROJECT_ID or the active profile. | |
--input-json | str | Request body as an inline JSON string. Pass exactly one of --input-json or --input-json-file. | |
--input-json-file | path | Path to a file holding the request body as JSON. Pass exactly one of --input-json or --input-json-file. |
DELETE
Delete an evaluator by ID.oumi-cli evaluators delete [OPTIONS] {evaluator_id}
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | int | ✓ | ID of the evaluator to delete. |
| Option | Type | Required | Description |
|---|---|---|---|
--project | str | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or active profile. | |
--force | flag | Skip the confirmation prompt (alias for --yes). | |
--yes | flag | Skip the confirmation prompt; useful in scripts. |
DOWNLOAD
Download an evaluator as an editable EvaluatorRequest body.oumi-cli evaluators download [OPTIONS] {evaluator_id}
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | int | ✓ | ID of the evaluator to download (see evaluators list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | str | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or active profile. | |
--out | path | Destination file or directory. Defaults to ./evaluator-<id>.json in the current directory. |
GENERATE-TEMPLATE
Print a ready-to-edit EvaluatorRequest body for the given evaluator type.oumi-cli evaluators generate-template [OPTIONS]
| Option | Type | Required | Description |
|---|---|---|---|
--type | choice | ✓ | Evaluator kind to generate a template for; sets params.evaluatorType in the output. One of: judge, classification. |
--minimal | flag | Emit a small, filled-in worked example for a built-in judge type instead of the full schema. Only supported with --type judge. |
GET
Get an evaluator by ID.oumi-cli evaluators get [OPTIONS] {evaluator_id}
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | int | ✓ | ID of the evaluator to retrieve. |
| Option | Type | Required | Description |
|---|---|---|---|
--project | str | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or active profile. |
LIST
List evaluators in the project.oumi-cli evaluators list [OPTIONS]
| Option | Type | Required | Description |
|---|---|---|---|
--project | str | Project to list evaluators in; falls back to OUMI_PROJECT_ID or active profile. | |
--limit | int range | Return at most N evaluators (default 100). The CLI loops over pages internally. | |
--all | flag | Return every evaluator, looping until the cursor is exhausted (bounded by --max-items). Mutually exclusive with --limit. | |
--max-items | int range | Hard safety cap for --all (default 10000). | |
--search | str | Only show evaluators whose name matches this search text. | |
--sort-by | choice | Field to sort by. One of: created_at, updated_at. | |
--sort-direction | choice | Sort order (asc is oldest-first). |
UPDATE
Rename an evaluator, or replace its config from a JSON body.oumi-cli evaluators update [OPTIONS] {evaluator_id}
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | int | ✓ | ID of the evaluator (see evaluators list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | str | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or the active profile. | |
--display-name | str | New display name for the evaluator. Mutually exclusive with --input-json*. | |
--input-json | str | Full replacement request body as an inline JSON string; creates a new evaluator version. Exclusive of --display-name. | |
--input-json-file | path | Path to a file holding the full replacement request body as JSON; creates a new evaluator version. Exclusive of --display-name. |
VERSIONS LIST
List version history for an evaluator, newest first by default.oumi-cli evaluators versions list [OPTIONS] {evaluator_id}
| Argument | Type | Required | Description |
|---|---|---|---|
EVALUATOR_ID | int | ✓ | ID of the evaluator (see evaluators list). |
| Option | Type | Required | Description |
|---|---|---|---|
--project | str | Project that owns the evaluator; falls back to OUMI_PROJECT_ID or the active profile. | |
--limit | int range | Return at most N versions (default 100). The CLI loops over pages internally. | |
--all | flag | Return every version, looping until the cursor is exhausted (bounded by --max-items). Mutually exclusive with --limit. | |
--max-items | int range | Hard safety cap for --all (default 10000). | |
--sort-by | choice | Field to sort by. One of: created_at, updated_at, version_id. | |
--sort-direction | choice | Sort order (asc is oldest-first). |