Skip to main content
Use Bring Your Own Harness (BYOH) to generate conversation datasets with your existing agent. Your harness runs in your environment and keeps its prompts, tools, context management, memory, and application logic. When you generate a dataset from scratch, choose Your harness in the Builder. Oumi generates the opening messages for each conversation, your harness produces the assistant responses, and Oumi saves the full conversations to a new dataset.

​
BEFORE YOU START

You need:
Harness synthesis is available only through Generate from Scratch. Generate Completions and Generate from Failure Modes do not support harnesses.

​
HOW A HARNESS RUN WORKS

A synthesis run with a harness has two stages:
  1. Oumi generates the conversation openings. The model you select under Model & Inference generates the recipe’s attributes, such as a question, a persona, or a scenario. A chat transform assembles them into the opening messages of each conversation.
  2. Your harness completes the conversations. Oumi sends each opening to your harness with a connection to the same model. Your harness runs the agent and its tools, then returns the resulting messages. Oumi saves the full conversation to the output dataset.
The messages produced by the chat transform must end with a user message so your harness can generate the next assistant response. Include a system prompt if your recipe uses one. If an opening ends with an assistant message, that row fails.

​
GENERATE A DATASET

  1. Go to Datasets and click Create Dataset, then choose Generate from Scratch to open the Builder.
  2. Under Model & Inference, select the API-hosted model your agent should call.
  3. Under Agent Runtime, select Your harness. In the Harness section that appears, choose a registered endpoint.
Your harness runs its own tools. Selecting Your harness hides the Builder’s Tools section, and Oumi’s tool implementations are not used during the run. Configure the rest of the recipe as you normally would: Review the configuration and click Execute. Give the job a Display Name and click Generate Dataset. You can follow the run on the Jobs page. When the job completes, the new dataset appears on your Datasets page.

​
ADD A SIMULATED USER

If your agent asks questions during a conversation, expand User Model in the Builder and select a model to answer them. This setting is separate from the model selected under Model & Inference. When your agent pauses for input, Oumi generates the user’s reply and sends it back so the conversation can continue. If your recipe synthesizes a user persona, Oumi uses it to guide those replies. The persona is not shared with your agent. You can leave User Model empty if your agent does not need follow-up input. If it requests input without a user model configured, that row fails.

​
BILLING

Synthesis with a harness follows the same billing model as other synthesis runs:
  • Attribute generation: Token usage to generate the conversation openings.
  • Harness turns: Token usage for the calls your harness makes through the model connection.
  • Simulated user: Token usage to generate user replies, if a user model is configured.
Oumi does not bill for your agent’s infrastructure or for model calls made outside the provided connection.

​
TROUBLESHOOTING

For endpoint implementation details, see the harness protocol.