BUILD AN AGENT

Build an agent that can check the status of a customer’s order. You’ll define the tool it can call, give the tool a simulated implementation, and evaluate whether the agent calls it correctly and answers using its result. You’ll finish with an evaluation of the agent’s behavior, including its tool calls and final responses. You don’t need a live order system or a trained model to get started.

BUILD WITH FAiE

FAiE (Frontier AI Engineer Agent) can help create tools, test implementations, and configure evaluations. Open Ask FAiE in your project and describe the workflow you want:

Help me set up a customer support agent that uses get_order_status to
answer order-status questions. Define the tool with a required order_id,
give it a simulated implementation for a fictional store, and test it.
Then help me evaluate whether the agent calls the tool with the right
arguments and gives an answer grounded in the tool result.

Review the tools, settings, and evaluators FAiE proposes before approving jobs. You can also complete the same workflow manually below.

BEFORE YOU START

Create a project in Oumi, or open an existing one. You’ll need access to a model that supports tool calling and a hosted model for the simulated tool and judge evaluator. These model calls are billed as inference. This tutorial uses one function, get_order_status, and a small test dataset. A model evaluation in Oumi runs the tool-calling conversation; it does not deploy a complete production agent.

1. DEFINE THE ORDER-STATUS TOOL

Open your project’s Tools page and click Create Tool.
Enter Get Order Status as the display name and get_order_status as the function name.
Describe when the model should call it: “Return the current status and expected delivery date of one order. Call when the customer asks where an order is or when it will arrive.”
Add one required string parameter named order_id. In its description, tell the model to use the order ID the customer provided.
Optionally, specify the output schema below so the tool returns predictable fields. Click Create Tool.

{
  "type": "object",
  "properties": {
    "status": {
      "type": "string",
      "enum": ["processing", "shipped", "delivered", "cancelled"]
    },
    "expected_delivery": {
      "type": "string",
      "description": "Expected delivery date in YYYY-MM-DD format"
    }
  },
  "required": ["status"]
}

A tool defines what the model can request, not how that request is answered. For details on parameter schemas and later changes, see Tools.

2. SIMULATE THE TOOL’S RESPONSE

Oumi opens the implementation form after you create the tool.

Choose Simulated as the implementation type.
Select a hosted model to generate tool responses.
In the Tool persona, enter:

You represent a fictional retailer's order system.
Return plausible order statuses and delivery dates.
Be consistent about an order within the same conversation.
Do not include real customer information.

Create the implementation. On its page, use the Test card to call it with the arguments below, then click Run test.

{"order_id": "A-48213"}

Confirm that the output has a status and, if present, an expected_delivery date. Because a simulated implementation uses a model, its results can vary. For deterministic tests, replace it later with a Lookup implementation; for real order records, use an Endpoint implementation connected to your HTTPS or MCP service. See Tool implementations.

3. UPLOAD A FEW TEST REQUESTS

Create a local file named order-status-test.jsonl and paste in the following three lines. Each line is one conversation that ends with a user request.

{"messages":[{"role":"user","content":"Where is my order A-48213?"}]}
{"messages":[{"role":"user","content":"Has order B-10872 shipped yet?"}]}
{"messages":[{"role":"user","content":"When should I expect order C-73910?"}]}

Open Datasets and upload the file. Oumi creates a Conversation dataset, which you can select when you run the evaluation. See Upload datasets if you need help with the format or quality checks.

4. CREATE A TOOL-USE EVALUATOR

Go to Evaluators, click Create Evaluator, and choose an LLM judge evaluator. Select a judge model and create two judgments: Correct with score 1 and Incorrect with score 0. Use these instructions for the judge:

Judge the entire conversation, including tool calls and tool results.
Return Correct only when the assistant calls get_order_status
with the order_id from the user's request, then answers the question
accurately using the returned status and delivery date (if present).
Return Incorrect if it skips the required lookup, uses the wrong
order_id, invents a status, or contradicts the tool result.

Save the evaluator as Order status tool use. The evaluation should measure the steps taken, not just whether the final sentence sounds helpful. See Create judge evaluators for evaluator configuration.

5. RUN YOUR AGENT EVALUATION

Open Evaluations, click Run Evaluation, and choose Run an Evaluation.
Under Inference Source, keep Use model. Choose a tool-capable model as the Assistant Model.
Select Order status tool use as the evaluator and order-status-test as the dataset.
Expand Tools and click Select tools and implementations. Select the simulated implementation of get_order_status.
Click Execute, name the evaluation, and click Run Evaluation.

For this first run, leave User Model empty. Each example asks one question, so you can focus on whether the model chooses the right tool. To test follow-up conversations later, add a simulated user and set Max user turns. See Evaluate agents.

6. INSPECT THE RESULTS

Open the completed evaluation. Review the scores and inspect individual conversations, including the assistant’s get_order_status call, the arguments, the simulated tool output, and the final response. If a row fails, decide whether the problem is the tool description, the implementation output, or the model’s decision. Fix the appropriate part, then run the same dataset and evaluator again to compare results. See Evaluation results. You’ve now built and evaluated a tool-using agent workflow in Oumi.

7. SYNTHESIZE AGENT DATA (OPTIONAL)

Use Synthesize agent data to generate more tool-using conversations for training or evaluation. With Generate Completions, you can reuse your test prompts and tool implementations to produce full conversations, optionally adding a simulated user for multiple turns. Review the generated examples before using them in a training or evaluation dataset.

NEXT STEPS

Train a specialized model

Fine-tune a model on high-quality conversations, then compare it with your baseline.

Run your agent

Deploy the underlying model for inference and connect it to an agent harness such as pi or Hermes.

Bring your own harness

Evaluate an existing agent while keeping its orchestration, tools, and state in your runtime.

Continuous improvement

Use evaluated results and production evidence to guide further model improvements.

Deploying a model endpoint does not by itself host the full agent runtime. The tool loop, orchestration, and state must run in an agent harness or application. See Run your agent.

Getting started

Oumi workflow

BUILD AN AGENT

BUILD WITH FAiE

BEFORE YOU START

1. DEFINE THE ORDER-STATUS TOOL

2. SIMULATE THE TOOL’S RESPONSE

3. UPLOAD A FEW TEST REQUESTS

4. CREATE A TOOL-USE EVALUATOR

5. RUN YOUR AGENT EVALUATION

6. INSPECT THE RESULTS

7. SYNTHESIZE AGENT DATA (OPTIONAL)

NEXT STEPS

Train a specialized model

Run your agent

Bring your own harness

Continuous improvement

​BUILD WITH FAiE

​BEFORE YOU START

​1. DEFINE THE ORDER-STATUS TOOL

​2. SIMULATE THE TOOL’S RESPONSE

​3. UPLOAD A FEW TEST REQUESTS

​4. CREATE A TOOL-USE EVALUATOR

​5. RUN YOUR AGENT EVALUATION

​6. INSPECT THE RESULTS

​7. SYNTHESIZE AGENT DATA (OPTIONAL)

​NEXT STEPS

Train a specialized model

Run your agent

Bring your own harness

Continuous improvement

BUILD WITH FAiE

BEFORE YOU START

1. DEFINE THE ORDER-STATUS TOOL

2. SIMULATE THE TOOL’S RESPONSE

3. UPLOAD A FEW TEST REQUESTS

4. CREATE A TOOL-USE EVALUATOR

5. RUN YOUR AGENT EVALUATION

6. INSPECT THE RESULTS

7. SYNTHESIZE AGENT DATA (OPTIONAL)

NEXT STEPS