RUNNING EVALUATIONS

An evaluation runs a model on each row of a dataset and scores the results with your evaluators. You can also attach tools and a simulated user, so that the model works through each row as an agent, or score conversations that already happened without running a model.

RUN AN EVALUATION IN THE BUILDER

Go to the Evaluations page and click Run Evaluation.
In the Builder, click Run an Evaluation.
Under Inference Source, select Use model to generate new responses, or Use trace to score existing conversations.
In the Assistant Model section, choose the model under test.
Optional: in the Tools and User Model sections, add tools and a simulated user.
In the Evaluators section, click Select Evaluators and choose your evaluators.
In the Dataset section, choose the dataset.
Optional: in the Failure Mode Analysis section, check Generate failure modes after evaluation.
Optional: in the Inference Configuration section, set inference parameters for the model under test, such as Temperature, Max Tokens, and Seed.
Click Run Evaluation, enter a name for the evaluation, and confirm.

When the job finishes, open the evaluation from the Evaluations page to see the results.

ADD TOOLS AND A SIMULATED USER

To evaluate a model as an agent, attach the tools it can call and, optionally, a simulated user that continues the conversation. Each evaluator then scores the full conversation, including every tool call and tool result. Use this when the quality you care about depends on how the model works through a task, not only on its final answer.

Expand the Tools section and open the Select tools and implementations list. Under each tool, check the tool implementation that answers its calls. After you check one implementation for a tool, the other implementations for that tool name are disabled. If a tool has no implementation yet, click Create an implementation.
Optional: expand the User Model section and choose a model to play the user. Set Max user turns to limit the number of user messages in each conversation, including the first.

The Tools section appears only when your project has at least one tool. To remove the simulated user, click Remove user model. For each row, the evaluation keeps the messages up to and including the first user message and discards every later message. The model under test answers and calls tools until it gives a final answer. With a user model, the simulated user replies and the model answers again, until the simulated user ends the conversation or it reaches the maximum number of user turns. To give the simulated user a goal for each row, add a user persona.

The simulated user takes turns only when the run also attaches at least one tool implementation. A run with a user model and no tool implementations generates one assistant response to the first user message.

These requirements apply:

Tool calling: The model under test must support tool calling. Oumi checks Oumi-served pretrained models and custom models against the tool-calling support of their base model, and rejects the run if the base model cannot call tools. Hosted API models are accepted, because the provider handles tool calls.
User model: The user model must be a hosted API model. The User Model section lists only models that can play the user.
Interactive tool implementations: A run that attaches an interactive tool implementation must also set a user model, because the simulated user answers those calls.

To decide what your evaluators should check in a tool-using conversation, see Judging tool use and conversations. To run your own agent loop instead of the platform’s, see Harnesses.

SCORE EXISTING CONVERSATIONS

When your dataset holds conversations that already happened, such as traces from Import observability logs, you can score them without running a model. The evaluators score each conversation as it is, including its tool calls. In the Builder, select Use trace under Inference Source. The model, tool, user model, and inference sections are hidden, because no model runs. Choose your evaluators and dataset, and run the evaluation.

COMPARE TRACES WITH MODEL RUNS

When you plan to replace or fine-tune the model behind your traces, evaluate four sets of conversations with the same evaluators:

The traces as they are, with Use trace.
The original model (the teacher), run again through your tool implementations.
The untuned candidate model, as a baseline.
The fine-tuned candidate model.

The teacher run is the reference point. Compare 1 with 2 to measure how closely your tool implementations and simulated user reproduce the original traces. Compare 2 with 3 and 4 to compare models under the same tool implementations and inference settings. Do not compare 1 directly with 3, because that comparison mixes the model difference with the difference between production and the platform.

RUN AN EVALUATION WITH THE API

Send a POST request to /evaluations:run. Authenticate with your Oumi API key in the X-API-Key header. This example attaches tools and a simulated user. Omit agenticInferenceConfig for a plain evaluation.

curl -X POST "https://api.oumi.ai/v1/projects/{project_id}/evaluations:run" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $OUMI_API_KEY" \
  -d '{
    "displayName": "support-agent-eval",
    "recipe": {
      "recipeConfig": {
        "type": "evaluate",
        "evaluationConfig": {
          "evaluationType": "single_model",
          "modelIdentifier": {"modelType": "OPENAI_API", "modelName": "gpt-5-mini"},
          "evaluators": [{"evaluatorId": 7}, {"evaluatorId": 9}],
          "inferenceConfig": {},
          "dataset": {"datasetId": 4},
          "generateFailureModes": true,
          "agenticInferenceConfig": {
            "toolImplementations": [
              {"toolImplementationId": 22},
              {"toolImplementationId": 23}
            ],
            "userModelSpec": {
              "modelIdentifier": {"modelType": "OPENAI_API", "modelName": "gpt-5-mini"},
              "inferenceConfig": {}
            },
            "maxUserTurns": 5
          }
        }
      }
    }
  }'

import os
import requests

response = requests.post(
    "https://api.oumi.ai/v1/projects/{project_id}/evaluations:run",
    headers={"X-API-Key": os.environ["OUMI_API_KEY"]},
    json={
        "displayName": "support-agent-eval",
        "recipe": {
            "recipeConfig": {
                "type": "evaluate",
                "evaluationConfig": {
                    "evaluationType": "single_model",
                    "modelIdentifier": {"modelType": "OPENAI_API", "modelName": "gpt-5-mini"},
                    "evaluators": [{"evaluatorId": 7}, {"evaluatorId": 9}],
                    "inferenceConfig": {},
                    "dataset": {"datasetId": 4},
                    "generateFailureModes": True,
                    "agenticInferenceConfig": {
                        "toolImplementations": [
                            {"toolImplementationId": 22},
                            {"toolImplementationId": 23},
                        ],
                        "userModelSpec": {
                            "modelIdentifier": {"modelType": "OPENAI_API", "modelName": "gpt-5-mini"},
                            "inferenceConfig": {},
                        },
                        "maxUserTurns": 5,
                    },
                },
            }
        },
    },
)
response.raise_for_status()
print(response.json()["id"])

The response is the evaluation operation. You can also pass {"recipeId": ..., "recipeVersion": ...} as recipe to run a saved evaluation recipe. For agenticInferenceConfig, see its field reference. The same object configures synthesis jobs. For the other fields, see the evaluation schema. To score existing conversations with the API, set skipInference to true and forceInference to false. forceInference defaults to true, so you must set it explicitly. modelIdentifier and inferenceConfig are still required by the schema, but the run does not use them. Omit agenticInferenceConfig.

VALIDATION RULES

At least one evaluator is required.
forceInference and skipInference cannot both be true.
maxUserTurns requires userModelSpec, and must be between 1 and 20.
The tool implementations in one run must serve distinct tool names.
When the run attaches tool implementations, the model under test must support tool calling.
A run that attaches an interactive tool implementation must set userModelSpec.
All simulated tool implementations in one run must use the same model.
The user model must be a hosted API model.

Getting started

Oumi workflow

RUNNING EVALUATIONS

RUN AN EVALUATION IN THE BUILDER

ADD TOOLS AND A SIMULATED USER

SCORE EXISTING CONVERSATIONS

COMPARE TRACES WITH MODEL RUNS

RUN AN EVALUATION WITH THE API

VALIDATION RULES

​RUN AN EVALUATION IN THE BUILDER

​ADD TOOLS AND A SIMULATED USER

​SCORE EXISTING CONVERSATIONS

​COMPARE TRACES WITH MODEL RUNS

​RUN AN EVALUATION WITH THE API

​VALIDATION RULES

RUN AN EVALUATION IN THE BUILDER

ADD TOOLS AND A SIMULATED USER

SCORE EXISTING CONVERSATIONS

COMPARE TRACES WITH MODEL RUNS

RUN AN EVALUATION WITH THE API

VALIDATION RULES