EVALUATE A MODEL

This example uses the banking77 validation dataset.

Upload the validation data

oumi-cli datasets upload ./banking77_val_basic.jsonl --display-name "banking77-val"

Note the dataset ID it returns.

Create a classification evaluator

cat > evaluator.json <<EOF
{
  "displayName": "banking77-label-match",
  "description": "Checks the predicted intent label against the ground-truth label.",
  "params": {
    "evaluatorType": "classification",
    "labelField": "label",
    "labelExtractionMode": "auto"
  }
}
EOF

oumi-cli evaluators create --input-json-file evaluator.json

Note the evaluator ID it returns. Valid labels are auto-discovered from the dataset; run oumi-cli evaluators generate-template --type classification to see every option.

Write an evaluation recipe

Set the three IDs, then paste the block. MODEL_ID is the model you want to score, DATASET_ID is the validation set from the upload step, and EVALUATOR_ID is the one you just created.

MODEL_ID=1
DATASET_ID=1
EVALUATOR_ID=1

cat > eval-recipe.json <<EOF
{
  "displayName": "banking77-eval",
  "recipeConfig": {
    "type": "evaluate",
    "evaluationConfig": {
      "evaluationType": "single_model",
      "modelIdentifier": { "modelType": "CUSTOM_CLOUD_STORAGE", "modelId": $MODEL_ID },
      "evaluators": [{ "evaluatorId": $EVALUATOR_ID }],
      "inferenceConfig": {},
      "dataset": { "datasetId": $DATASET_ID },
      "generateFailureModes": true
    }
  }
}
EOF

To score a base model instead of one you trained, use { "modelType": "PRETRAINED", "modelName": "..." }, or list what is available with oumi-cli evaluations available-models.

Create the recipe and run it

oumi-cli recipes create --type evaluate --input-json-file eval-recipe.json

oumi-cli evaluations run --name "banking77-eval" --recipe <RECIPE_ID>

READING THE RESULTS

oumi-cli evaluations get <EVALUATION_ID>
oumi-cli evaluations results <EVALUATION_ID>
oumi-cli evaluations download <EVALUATION_ID> --out results.jsonl

VIEWING FAILURE MODES

oumi-cli evaluations failure-modes <EVALUATION_ID>

Train a model

Retrain against the gaps you found.

Evaluations reference

Every flag on every evaluation command.

Getting started

Basics

Workflows

Command reference

EVALUATE A MODEL

READING THE RESULTS

VIEWING FAILURE MODES

Train a model

Evaluations reference

​READING THE RESULTS

​VIEWING FAILURE MODES

Train a model

Evaluations reference

READING THE RESULTS

VIEWING FAILURE MODES