> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oumi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# BUILD AN AGENT

> Build a tool-using support agent, evaluate its actions, and decide what to improve next

Build an agent that can check the status of a customer's order. You'll define the tool it can call, give the tool a simulated implementation, and evaluate whether the agent calls it correctly and answers using its result.

You'll finish with an evaluation of the agent's behavior, including its tool calls and final responses. You don't need a live order system or a trained model to get started.

## BUILD WITH FAiE

[FAiE](https://docs.oumi.ai/guides/intro/faie.md) (Frontier AI Engineer Agent) can help create tools, test implementations, and configure evaluations. Open **Ask FAiE** in your project and describe the workflow you want:

```text theme={null}
Help me set up a customer support agent that uses get_order_status to
answer order-status questions. Define the tool with a required order_id,
give it a simulated implementation for a fictional store, and test it.
Then help me evaluate whether the agent calls the tool with the right
arguments and gives an answer grounded in the tool result.
```

Review the tools, settings, and evaluators FAiE proposes before approving jobs. You can also complete the same workflow manually below.

## BEFORE YOU START

Create a project in Oumi, or open an existing one. You'll need access to a model that supports tool calling and a hosted model for the simulated tool and judge evaluator. These model calls are billed as inference.

This tutorial uses one function, `get_order_status`, and a small test dataset. A model evaluation in Oumi runs the tool-calling conversation; it does not deploy a complete production agent.

## 1. DEFINE THE ORDER-STATUS TOOL

1. Open your project's **Tools** page and click `Create Tool`.
2. Enter **Get Order Status** as the display name and `get_order_status` as the function name.
3. Describe when the model should call it: "Return the current status and expected delivery date of one order. Call when the customer asks where an order is or when it will arrive."
4. Add one required string parameter named `order_id`. In its description, tell the model to use the order ID the customer provided.
5. Optionally, specify the output schema below so the tool returns predictable fields. Click `Create Tool`.

```json theme={null}
{
  "type": "object",
  "properties": {
    "status": {
      "type": "string",
      "enum": ["processing", "shipped", "delivered", "cancelled"]
    },
    "expected_delivery": {
      "type": "string",
      "description": "Expected delivery date in YYYY-MM-DD format"
    }
  },
  "required": ["status"]
}
```

A tool defines what the model can request, not how that request is answered. For details on parameter schemas and later changes, see [Tools](https://docs.oumi.ai/guides/agents/tools.md).

## 2. SIMULATE THE TOOL'S RESPONSE

Oumi opens the implementation form after you create the tool.

1. Choose **Simulated** as the implementation type.
2. Select a hosted model to generate tool responses.
3. In the **Tool persona**, enter:

```text theme={null}
You represent a fictional retailer's order system.
Return plausible order statuses and delivery dates.
Be consistent about an order within the same conversation.
Do not include real customer information.
```

4. Create the implementation. On its page, use the **Test** card to call it with the arguments below, then click `Run test`.

```json theme={null}
{"order_id": "A-48213"}
```

Confirm that the output has a `status` and, if present, an `expected_delivery` date. Because a simulated implementation uses a model, its results can vary. For deterministic tests, replace it later with a **Lookup** implementation; for real order records, use an **Endpoint** implementation connected to your HTTPS or MCP service. See [Tool implementations](https://docs.oumi.ai/guides/agents/tool-implementations.md).

## 3. UPLOAD A FEW TEST REQUESTS

Create a local file named `order-status-test.jsonl` and paste in the following three lines. Each line is one conversation that ends with a user request.

```json theme={null}
{"messages":[{"role":"user","content":"Where is my order A-48213?"}]}
{"messages":[{"role":"user","content":"Has order B-10872 shipped yet?"}]}
{"messages":[{"role":"user","content":"When should I expect order C-73910?"}]}
```

Open **Datasets** and upload the file. Oumi creates a Conversation dataset, which you can select when you run the evaluation. See [Upload datasets](https://docs.oumi.ai/guides/datasets/upload.md) if you need help with the format or quality checks.

## 4. CREATE A TOOL-USE EVALUATOR

Go to **Evaluators**, click `Create Evaluator`, and choose an **LLM** judge evaluator. Select a judge model and create two judgments: `Correct` with score `1` and `Incorrect` with score `0`.

Use these instructions for the judge:

```text theme={null}
Judge the entire conversation, including tool calls and tool results.
Return Correct only when the assistant calls get_order_status
with the order_id from the user's request, then answers the question
accurately using the returned status and delivery date (if present).
Return Incorrect if it skips the required lookup, uses the wrong
order_id, invents a status, or contradicts the tool result.
```

Save the evaluator as **Order status tool use**. The evaluation should measure the steps taken, not just whether the final sentence sounds helpful. See [Create judge evaluators](https://docs.oumi.ai/guides/evaluators/judge.md) for evaluator configuration.

## 5. RUN YOUR AGENT EVALUATION

1. Open **Evaluations**, click `Run Evaluation`, and choose `Run an Evaluation`.
2. Under **Inference Source**, keep `Use model`. Choose a tool-capable model as the **Assistant Model**.
3. Select **Order status tool use** as the evaluator and `order-status-test` as the dataset.
4. Expand **Tools** and click `Select tools and implementations`. Select the simulated implementation of `get_order_status`.
5. Click `Execute`, name the evaluation, and click `Run Evaluation`.

For this first run, leave **User Model** empty. Each example asks one question, so you can focus on whether the model chooses the right tool. To test follow-up conversations later, add a simulated user and set **Max user turns**. See [Evaluate agents](https://docs.oumi.ai/guides/evaluations/agents.md).

## 6. INSPECT THE RESULTS

Open the completed evaluation. Review the scores and inspect individual conversations, including the assistant's `get_order_status` call, the arguments, the simulated tool output, and the final response.

If a row fails, decide whether the problem is the tool description, the implementation output, or the model's decision. Fix the appropriate part, then run the same dataset and evaluator again to compare results. See [Evaluation results](https://docs.oumi.ai/guides/evaluations/results.md#results-for-tool-using-conversations).

You've now built and evaluated a tool-using agent workflow in Oumi.

## 7. SYNTHESIZE AGENT DATA (OPTIONAL)

Use [Synthesize agent data](https://docs.oumi.ai/guides/synthesis/agent-data.md) to generate more tool-using conversations for training or evaluation. With **Generate Completions**, you can reuse your test prompts and tool implementations to produce full conversations, optionally adding a simulated user for multiple turns. Review the generated examples before using them in a training or evaluation dataset.

## NEXT STEPS

> [card] [Train a specialized model](https://docs.oumi.ai/guides/training.md)
>
> Fine-tune a model on high-quality conversations, then compare it with your baseline.

> [card] [Run your agent](https://docs.oumi.ai/guides/agents/run-your-agent.md)
>
> Deploy the underlying model for inference and connect it to an agent harness such as pi or Hermes.

> [card] [Bring your own harness](https://docs.oumi.ai/guides/harnesses.md)
>
> Evaluate an existing agent while keeping its orchestration, tools, and state in your runtime.

> [card] [Continuous improvement](https://docs.oumi.ai/guides/deployment/compounding-intelligence.md)
>
> Use evaluated results and production evidence to guide further model improvements.

> [!NOTE]
>
> Deploying a model endpoint does not by itself host the full agent runtime. The tool loop, orchestration, and state must run in an agent harness or application. See [Run your agent](https://docs.oumi.ai/guides/agents/run-your-agent.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.