> ## Documentation Index > Fetch the complete documentation index at: https://docs.oumi.ai/llms.txt > Use this file to discover all available pages before exploring further. # EVALUATORS > Create and manage evaluator resources | Command | Description | | ----------------------------------------- | ------------------------------------------------------------------------- | | [`create`](#create) | Create an evaluator in a project from a JSON request body. | | [`delete`](#delete) | Delete an evaluator by ID. | | [`download`](#download) | Download an evaluator as an editable EvaluatorRequest body. | | [`generate-template`](#generate-template) | Print a ready-to-edit EvaluatorRequest body for the given evaluator type. | | [`get`](#get) | Get an evaluator by ID. | | [`list`](#list) | List evaluators in the project. | | [`update`](#update) | Rename an evaluator, or replace its config from a JSON body. | | [`versions list`](#versions-list) | List version history for an evaluator, newest first by default. | *** ## CREATE Create an evaluator in a project from a JSON request body. ```bash theme={null} oumi-cli evaluators create [OPTIONS] ``` **Options** | Option | Type | Required | Description | | ------------------- | ---- | -------- | ----------------------------------------------------------------------------------------------------------- | | `--project` | str | | Project to create the evaluator in; falls back to OUMI\_PROJECT\_ID or the active profile. | | `--input-json` | str | | Request body as an inline JSON string. Pass exactly one of `--input-json` or `--input-json-file`. | | `--input-json-file` | path | | Path to a file holding the request body as JSON. Pass exactly one of `--input-json` or `--input-json-file`. | *** ## DELETE Delete an evaluator by ID. ```bash theme={null} oumi-cli evaluators delete [OPTIONS] {evaluator_id} ``` **Arguments** | Argument | Type | Required | Description | | -------------- | ---- | -------- | ------------------------------ | | `EVALUATOR_ID` | int | ✓ | ID of the evaluator to delete. | **Options** | Option | Type | Required | Description | | ----------- | ---- | -------- | ----------------------------------------------------------------------------------- | | `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or active profile. | | `--force` | flag | | Skip the confirmation prompt (alias for `--yes`). | | `--yes` | flag | | Skip the confirmation prompt; useful in scripts. | *** ## DOWNLOAD Download an evaluator as an editable EvaluatorRequest body. ```bash theme={null} oumi-cli evaluators download [OPTIONS] {evaluator_id} ``` **Arguments** | Argument | Type | Required | Description | | -------------- | ---- | -------- | -------------------------------------------------------- | | `EVALUATOR_ID` | int | ✓ | ID of the evaluator to download (see `evaluators list`). | **Options** | Option | Type | Required | Description | | ----------- | ---- | -------- | ------------------------------------------------------------------------------------------- | | `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or active profile. | | `--out` | path | | Destination file or directory. Defaults to ./evaluator-\.json in the current directory. | *** ## GENERATE-TEMPLATE Print a ready-to-edit EvaluatorRequest body for the given evaluator type. ```bash theme={null} oumi-cli evaluators generate-template [OPTIONS] ``` **Options** | Option | Type | Required | Description | | ----------- | ------ | -------- | -------------------------------------------------------------------------------------------------------------------------------- | | `--type` | choice | ✓ | Evaluator kind to generate a template for; sets params.evaluatorType in the output. One of: `judge`, `classification`. | | `--minimal` | flag | | Emit a small, filled-in worked example for a built-in judge type instead of the full schema. Only supported with `--type` judge. | *** ## GET Get an evaluator by ID. ```bash theme={null} oumi-cli evaluators get [OPTIONS] {evaluator_id} ``` **Arguments** | Argument | Type | Required | Description | | -------------- | ---- | -------- | -------------------------------- | | `EVALUATOR_ID` | int | ✓ | ID of the evaluator to retrieve. | **Options** | Option | Type | Required | Description | | ----------- | ---- | -------- | ----------------------------------------------------------------------------------- | | `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or active profile. | *** ## LIST List evaluators in the project. ```bash theme={null} oumi-cli evaluators list [OPTIONS] ``` **Options** | Option | Type | Required | Description | | ------------------ | --------- | -------- | ---------------------------------------------------------------------------------------------------------------------------- | | `--project` | str | | Project to list evaluators in; falls back to OUMI\_PROJECT\_ID or active profile. | | `--limit` | int range | | Return at most N evaluators (default 100). The CLI loops over pages internally. | | `--all` | flag | | Return every evaluator, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. | | `--max-items` | int range | | Hard safety cap for `--all` (default 10000). | | `--search` | str | | Only show evaluators whose name matches this search text. | | `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`. | | `--sort-direction` | choice | | Sort order (asc is oldest-first). | *** ## UPDATE Rename an evaluator, or replace its config from a JSON body. ```bash theme={null} oumi-cli evaluators update [OPTIONS] {evaluator_id} ``` **Arguments** | Argument | Type | Required | Description | | -------------- | ---- | -------- | -------------------------------------------- | | `EVALUATOR_ID` | int | ✓ | ID of the evaluator (see `evaluators list`). | **Options** | Option | Type | Required | Description | | ------------------- | ---- | -------- | --------------------------------------------------------------------------------------------------------------------------------- | | `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or the active profile. | | `--display-name` | str | | New display name for the evaluator. Mutually exclusive with `--input-json`\*. | | `--input-json` | str | | Full replacement request body as an inline JSON string; creates a new evaluator version. Exclusive of `--display-name`. | | `--input-json-file` | path | | Path to a file holding the full replacement request body as JSON; creates a new evaluator version. Exclusive of `--display-name`. | *** ## VERSIONS LIST List version history for an evaluator, newest first by default. ```bash theme={null} oumi-cli evaluators versions list [OPTIONS] {evaluator_id} ``` **Arguments** | Argument | Type | Required | Description | | -------------- | ---- | -------- | -------------------------------------------- | | `EVALUATOR_ID` | int | ✓ | ID of the evaluator (see `evaluators list`). | **Options** | Option | Type | Required | Description | | ------------------ | --------- | -------- | -------------------------------------------------------------------------------------------------------------------------- | | `--project` | str | | Project that owns the evaluator; falls back to OUMI\_PROJECT\_ID or the active profile. | | `--limit` | int range | | Return at most N versions (default 100). The CLI loops over pages internally. | | `--all` | flag | | Return every version, looping until the cursor is exhausted (bounded by `--max-items`). Mutually exclusive with `--limit`. | | `--max-items` | int range | | Hard safety cap for `--all` (default 10000). | | `--sort-by` | choice | | Field to sort by. One of: `created_at`, `updated_at`, `version_id`. | | `--sort-direction` | choice | | Sort order (asc is oldest-first). | ***