Skip to main content
An agent evaluation scores how a model works through a task, not only its final answer. The model calls your tools, reads their results, and keeps talking with a simulated user over several turns. Each evaluator then scores the full conversation, including every tool call and tool result. Use it when you build a model that calls tools or holds a conversation, such as a support agent that looks up an order before it answers. To run your own agent loop instead of the platform’s, see Evaluate with a harness.

​
BEFORE YOU START

You need:
  • At least one tool with a tool implementation that answers its calls. The Tools section appears only when your project has at least one tool.
  • A model under evaluation that supports tool calling. Hosted API models are accepted. For an Oumi-served or custom model, Oumi checks whether its base model can call tools, and rejects the run if it can’t.
  • For a simulated user, a hosted API model to play the user. The User Model section lists only models that can play the user.

​
RUN AN AGENT EVALUATION

  1. Set up the run as in Evaluate a model: keep Use model, and select the Assistant Model, Evaluators, and Dataset.
  2. Expand Tools and open Select tools and implementations. For each tool the model may call, check the tool implementation that answers its calls. You can check one implementation for each tool. If a tool has no implementation yet, click Create an implementation.
  3. Optionally, expand User Model, choose a model to play the user, and set Max user turns. The count includes the first user message. Fewer turns keep conversations short and the run cheaper.
  4. Click Execute, enter a name, and click Run Evaluation.
Tools section with the Look Up Card Status tool and its simulated implementation, and a User Model section with claude-sonnet-5.5 and Max user turns set to 5 To remove the simulated user, click Remove user model.
A run that attaches an interactive tool implementation must also set a user model, because the simulated user answers those calls.

​
HOW EACH CONVERSATION RUNS

For each row, the evaluation keeps the messages up to and including the first user message and discards the rest. The model answers, calling tools as needed, until it gives a final answer. With a user model, the simulated user replies and the model answers again, until the simulated user ends the conversation or the conversation reaches Max user turns. To give the simulated user a goal for each row, add a user persona.
Without a harness, the simulated user takes turns only when the run also attaches at least one tool implementation. A run with a user model and no tool implementations generates one assistant response to the first user message. With a harness, the simulated user answers when your harness pauses for input. See Evaluate with a harness.

​
WHAT TO CHECK

A correct final answer can hide a poor process, such as a wrong tool call that happened to return useful data. To decide what your evaluators should check in a tool-using conversation, see Judging tool use and conversations. The results show the turns and tool calls of each conversation. To run an agent evaluation with the API, add agenticInferenceConfig to the request. See Run an evaluation with the API.