Skip to main content
An agent is a model that does more than answer a single prompt. It calls tools to look up or change information, and it holds a conversation with a user across several turns. Oumi lets you define the tools your agent can call, decide how each tool call is answered, and then generate training data and run evaluations in which your model acts as that agent. Testing an agent against production systems is slow, costly, and risky. With Oumi, you simulate the tools and the user, so you can generate and judge thousands of realistic agent conversations in hours. You can then fine-tune a smaller model on the best conversations and serve it at a fraction of the inference cost of a frontier model.

​
MULTI-STEP AND MULTI-TURN

Agent behavior varies along two independent axes. Your agent can use either one or both:
  • Multi-step: The model calls tools. It may make several tool calls, reading each result, before it answers.
  • Multi-turn: The model holds a conversation across several user turns. A model plays the user, so you do not need a person in the loop.

​
HOW THE PIECES FIT TOGETHER

​
A TYPICAL WORKFLOW

1

Define your tools

Create a tool for each function your agent calls. Then give each tool a tool implementation and test it with a few sample calls.
2

Assemble your prompts

Collect the prompts your agent must handle, from your own data or from data synthesis. For multi-turn agents, add a varied user persona to each row.
3

Benchmark a teacher model

Run an evaluation with your tools on a strong model and on your untuned candidate model. Judge tool selection and grounding, not only the final answer. If you imported production traces, score those traces as well.
4

Generate training conversations

Generate completions with your tools to have the teacher model run your prompts with your tools and simulated user. Review the results and keep only the strongest conversations.
5

Fine-tune and evaluate

Train a model that supports tool calling on those conversations. Evaluate it against the teacher and the untuned baseline, then iterate.

​
WHAT’S NEXT

Tools

Define the functions your agent can call.

Tool implementations

Decide how each tool call is answered.

Simulated users

Let a model play the user across several turns.

Harnesses

Evaluate your own agent loop. In beta.