> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oumi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# EVALUATE WITH A HARNESS

> Run and review an evaluation of your agent from the Builder

"Bring Your Own Harness" lets you evaluate models in Oumi using your existing agent harness and application logic.

Instead of recreating your agent workflow in Oumi, you can connect your existing harness and run Oumi evaluations against your complete system. This lets you evaluate performance while keeping your existing orchestration, prompts, tools, context management, memory, and other application logic in place.

Your harness continues to run in your environment. Choose **Your harness** in the evaluation Builder to run your agent against a dataset. Oumi sends evaluation inputs to your harness, receives the resulting responses, and evaluates them using your Oumi datasets and evaluators.

***

## BEFORE YOU START

You need:

* A [harness registered in your project](/guides/harnesses/register).
* A custom Oumi model or an API-hosted model to evaluate.
* A dataset of conversations, each ending with a user message.
* At least one evaluator.

***

## RUN THE EVALUATION

1. Go to **Evaluations** and choose **Run an Evaluation** to open the Builder.
2. Under **Inference Source**, select **Use model**. A harness needs to generate new responses, so **Use trace** is not supported.
3. Under **Assistant Model**, select the model you want your agent to use.
4. Under **Agent Runtime**, select **Your harness**. The **Harness** selector appears; choose the endpoint registered in this project.

![Evaluation Builder showing the Inference Source, Assistant Model, Agent Runtime, and Harness sections](https://mintcdn.com/oumi/1xlA72C9TkG2ilgC/images/harnesses-evaluation-builder.png?fit=max\&auto=format\&n=1xlA72C9TkG2ilgC\&q=85\&s=671b2c3bedaa882bc0db69f6d6781d35)

Next, select the **Evaluators** and **Dataset** for the run. Configure any other settings you need:

| Input | What to select |
| - | - |
| **User Model** | A model to simulate user replies if your agent asks follow-up questions. See below. |
| **Inference Configuration** | Optional generation settings, such as temperature and maximum tokens. |
| **Failure Mode Analysis** | Optionally analyze the completed results for recurring issues. |

Review the configuration and click **Execute**. In the confirmation dialog, name the run and click **Run Evaluation**. For custom models, Oumi uses the same model version throughout the run.

***

## ADD A SIMULATED USER

If your agent asks questions during a conversation, expand **User Model** in the Builder and select a model to answer them. This is separate from the **Assistant Model** being evaluated.

When your agent pauses for input, Oumi generates the user's reply and sends it back so the conversation can continue. Dataset metadata, such as a user persona, provides context for these replies and is not shared with your agent.

You can leave **User Model** empty if your agent does not need follow-up input. If it requests input without a user model configured, that row fails.

***

## BILLING

A harness evaluation is billed like a regular evaluation of the same model:

* **Custom model:** Serving capacity for the duration of the evaluation, plus token usage through the model connection.
* **API-hosted model:** Token usage through the model connection.
* **Simulated user and evaluators:** Token usage, as in other evaluations.

Oumi does not bill for your agent's infrastructure or for model calls made outside the provided connection.

***

## TROUBLESHOOTING

| Issue | What to check |
| - | - |
| Your registered endpoint is missing from the Builder. | Confirm that you registered it in the same project as the evaluation. |
| Oumi cannot connect to the endpoint. | Check the URL on the **Harnesses** page and confirm with your team that the service is running and publicly reachable. |
| The endpoint rejects the evaluation. | Check its authentication settings and update the credential if needed. |
| Rows fail when the agent asks a question. | Add a **User Model** in the Builder so Oumi can generate replies. |
| Some conversations fail to complete. | Confirm that each dataset conversation ends with a user message. Ask your team to check the endpoint logs for errors or timeouts. |
| No rows complete. | Check the dataset, connection settings, and endpoint logs before running the evaluation again. |

For endpoint implementation details, see the [harness protocol](/guides/harnesses/protocol).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.