Skip to main content
An agent is a model that does more than answer a single prompt. It calls tools to look up or change information, and it holds a conversation with a user across several turns. Oumi lets you define the tools your agent can call, decide how each tool call is answered, and then generate training data and run evaluations in which your model acts as that agent. Testing an agent against production systems is slow, costly, and risky. Oumi lets you simulate the tools and the user, so you can generate and judge thousands of realistic agent conversations in hours. You can then fine-tune a smaller model on the best conversations and serve it at a fraction of the inference cost of a frontier model.

MULTI-STEP AND MULTI-TURN

Agent behavior varies along two independent axes. Your agent can use either one or both:
  • Multi-step: The model calls tools. It may make several tool calls, reading each result, before it answers.
  • Multi-turn: The model holds a conversation across several user turns. A separate model simulates the user, so you do not need a person in the loop.

HOW THE PIECES FIT TOGETHER


BUILD AN AGENT WITH FAiE

FAiE (Frontier AI Engineer Agent) can set up each part of an agent for you. Ask for the part you need:

START WITH AN AGENT EVALUATION

Build an Agent walks through a complete order-status support example. You’ll define a tool, add a simulated implementation, upload test prompts, create an evaluator, and inspect the agent’s tool calls and final answers. Once the agent behaves reliably, generate training conversations and fine-tune a model if you need to improve quality or economics. To use that model outside evaluation, deploy it for inference and run it in a harness.

TIPS

These practices help you get reliable results from agentic synthesis and evaluation without spending inference budget on data you later discard.
  • Get the tools right first: Your tool definitions shape every conversation that follows. Test each tool implementation with sample calls before you attach it to a job.
  • Check prompts before generating responses: Generating conversations with tool calls costs far more than generating prompts. See Split the work into two jobs.
  • Write varied personas and cap max conversation length: See Write good personas and Turn limits.
  • Benchmark the teacher first: A model trained on teacher conversations rarely beats the teacher. Evaluate the teacher before you generate training data with it.
  • Choose a long-context model for training: Agent conversations run long, and training truncates anything past the maximum sequence length.
  • Sweep temperature: Start low for reliable tool-call formatting, but also try higher values. Some models repeat tool calls more at low temperature.

WHAT’S NEXT

Build an Agent

Follow the end-to-end agent tutorial.

Tools

Define the functions your agent can call.

Tool implementations

Decide how each tool call is answered.

Simulated users

Let a model play the user across several turns.

Harnesses

Evaluate your existing agent loop with Oumi.