> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oumi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# EVALUATE AGENTS

> Evaluate a model that calls tools and talks with a simulated user over several turns

An agent evaluation scores how a model works through a task, not only its final answer. The model calls your [tools](https://docs.oumi.ai/guides/agents/tools.md), reads their results, and keeps talking with a [simulated user](https://docs.oumi.ai/guides/agents/simulated-users.md) over several turns. Each evaluator then scores the full conversation, including every tool call and tool result.

Use it when you build a model that calls tools or holds a conversation, such as a support agent that looks up an order before it answers. To run your own agent loop instead of the platform's, see [Evaluate with a harness](https://docs.oumi.ai/guides/harnesses/evaluations.md).

## BEFORE YOU START

You need:

* At least one tool with a [tool implementation](https://docs.oumi.ai/guides/agents/tool-implementations.md) that answers its calls. The **Tools** section appears only when your project has at least one tool.
* A model under evaluation that supports tool calling. Hosted API models are accepted. For an Oumi-served or custom model, Oumi checks whether its base model can call tools, and rejects the run if it can't.
* For a simulated user, a hosted API model to play the user. The **User Model** section lists only models that can play the user.

## RUN AN AGENT EVALUATION

1. Set up the run as in [Evaluate a model](https://docs.oumi.ai/guides/evaluations/running.md#evaluate-a-model): keep `Use model`, and select the **Assistant Model**, **Evaluators**, and **Dataset**.
2. Expand **Tools** and open `Select tools and implementations`. For each tool the model may call, check the [tool implementation](https://docs.oumi.ai/guides/agents/tool-implementations.md) that answers its calls. You can check one implementation for each tool. If a tool has no implementation yet, click `Create an implementation`.
3. Optionally, expand **User Model**, choose a model to play the user, and set `Max user turns`. The count includes the first user message. Fewer turns keep conversations short and the run cheaper.
4. Click `Execute`, enter a name, and click `Run Evaluation`.

![Tools section with the Look Up Card Status tool and its simulated implementation, and a User Model section with claude-sonnet-5.5 and Max user turns set to 5](https://mintcdn.com/oumi/5eBJSZOshiPI5WMs/images/evaluations-agent-sections.jpg?fit=max\&auto=format\&n=5eBJSZOshiPI5WMs\&q=85\&s=89a65de16ba8698298a9aae4ae392af4)

To remove the simulated user, click `Remove user model`.

> [!NOTE]
>
> A run that attaches an interactive tool implementation must also set a user model, because the simulated user answers those calls.

## HOW EACH CONVERSATION RUNS

For each row, the evaluation keeps the messages up to and including the first user message and discards the rest. The model answers, calling tools as needed, until it gives a final answer. With a user model, the simulated user replies and the model answers again, until the simulated user ends the conversation or the conversation reaches `Max user turns`. To give the simulated user a goal for each row, add a [user persona](https://docs.oumi.ai/guides/agents/simulated-users.md#steer-the-user-with-a-persona).

> [!WARNING]
>
> Without a harness, the simulated user takes turns only when the run also attaches at least one tool implementation. A run with a user model and no tool implementations generates one assistant response to the first user message. With a harness, the simulated user answers when your harness pauses for input. See [Evaluate with a harness](https://docs.oumi.ai/guides/harnesses/evaluations.md#add-a-simulated-user).

## WHAT TO CHECK

A correct final answer can hide a poor process, such as a wrong tool call that happened to return useful data. To decide what your evaluators should check in a tool-using conversation, see [Judging tool use and conversations](https://docs.oumi.ai/guides/evaluators/defining.md#judging-tool-use-and-conversations). The [results](https://docs.oumi.ai/guides/evaluations/results.md#results-for-tool-using-conversations) show the turns and tool calls of each conversation.

To run an agent evaluation with the API, add `agenticInferenceConfig` to the request. See [Run an evaluation with the API](https://docs.oumi.ai/guides/evaluations/running.md#run-an-evaluation-with-the-api).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.