HOW IT WORKS

Oumi makes it easy to synthesize data for a wide range of machine learning use cases, whether you’re training a model from scratch, fine-tuning an existing one, or improving performance through targeted evaluation criteria.

DATA SYNTHESIS METHODS

Oumi’s data synthesis capabilities are built directly into the platform, enabling data generation to integrate seamlessly into your end-to-end workflow. You can synthesize data using the following methods via the Oumi UI:

Using a data synthesis recipe
Generate completions
Generate from failure modes
Generate synthetic data from scratch

These options provide flexible, scalable ways to generate high-quality datasets aligned with your modeling goals, no matter where you are in the workflow.

USING A DATA SYNTHESIS RECIPE

Data synthesis recipes provide a reusable, structured approach to generating high-quality synthetic datasets. Designed as configurable blueprints, these recipes define how data should be created for specific use cases or modeling needs, giving you a fast and consistent starting point. To learn more, please see Data synthesis recipes.

GENERATE COMPLETIONS

Generate completions lets you create training data by supplying inputs and using a model to produce corresponding outputs. You can configure the process by selecting the model and any existing datasets to use, or by directly inserting the JSON configuration in the Oumi Builder. To generate completions:

Go to the Datasets page and click on Create Dataset.
In the Builder window, click on Generate Completions.
Set your Model and Source Dataset, and specify any optional parameters. Alternatively, copy/paste your JSON configuration into the text area on the right-hand side.
Click Generate Completions, enter a name for the new dataset, and click Generate Completions to start the job. Once processing is complete, the new dataset will appear on your Datasets page.

To have the model call tools or continue the conversation with a simulated user, see Add tools and a simulated user.

GENERATE FROM FAILURE MODES

Generating synthetic datasets through failure modes allows you to create targeted training data to fix known model errors and improve overall performance. To generate a dataset from failure mode:

Select an evaluation on the Evaluations page.
Click Review Failure Modes.
On the Failure Modes page, configure which evaluators’ items to include when generating datasets through failure mode.
Click Generate Dataset...
Give you synthesis job a Display Name and click Run Synthesis.

WHAT EACH FAILING ROW PRODUCES

For each failing row, Oumi generates Samples per item new conversations. Each one keeps the row’s subject but makes a different request, in a situation where the same failure could happen again. The synthesis model writes the assistant’s replies and is guided to avoid that failure. Because the new conversations don’t repeat the original requests, you can still use the original evaluation to test a model trained on them. To correct the original conversations instead, set generateNewConversations to false in the API. Each failing row then produces one corrected version of itself. These rows contain the original requests, so don’t test on the original evaluation after training on them.

CONVERSATIONS THAT CALL TOOLS

The job needs tool implementations. Generate Dataset reuses the ones from the evaluation. Under Advanced…, set them in the Tools section. Without them, rows that called tools are skipped.
With lookup or endpoint tools, new conversations reuse the original’s IDs. These tools answer only for records that exist, so new conversations keep the names, numbers, and IDs from the original. With simulated tools, new conversations can use new ones.
The synthesis model plays the user unless you choose a model in the User Model section.

RUBRIC JUDGE FAILURE MODES

For a rubric evaluator, generation targets only the failed criteria behind the failure modes you selected. To fix one behavior, select only its failure mode.

FEWER ROWS THAN EXPECTED

The dataset can be smaller than the number of failing rows × Samples per item. Oumi drops new conversations that nearly duplicate each other. A failing row can also produce nothing, for example when it called tools and the job has none. The job lists the rows that produced nothing.

TRACE A ROW TO ITS FAILURE

Each generated row stores the ID of the evaluation row it came from in its metadata, under source_data_item_id. Use it to compare a generated conversation with the failure it targets.

GENERATE SYNTHETIC DATA FROM SCRATCH

Generating synthetic datasets from scratch allows you to build synthetic datasets with full control over attiebutes, sources, and output format. To generate a dataset from scratch:

On the Datasets page, click Create Dataset.
Click Generate Synthetic Data from Scratch.
Configure your synthesis settings.
Click Generate Dataset, enter a name for the new dataset, and click Generate Dataset to start the job.

You can check the synthesis job on the Jobs page. Your newly synthesized dataset will be available on your Datasets page when the job completes.

ADD TOOLS AND A SIMULATED USER

The Generate Completions and Generate Synthetic Data from Scratch forms can generate conversations in which the model calls your tools and a simulated user continues the conversation for several turns. Use the resulting datasets to train a tool-calling model or to evaluate one. You need:

At least one tool with a tool implementation that answers its calls. The Tools section appears only when your project has at least one tool.
A model that supports tool calling. The GET /v1/projects/{project_id}/synthesis:supported_models response reports supportsToolCalling for each model, and oumi-cli datasets supported-models lists the same catalog.
For a simulated user, a hosted API model to play the user.

To add tools and a simulated user:

On the Datasets page, click Create Dataset.
Click Generate Completions to add responses to an existing dataset, or Generate Synthetic Data from Scratch to synthesize new conversations.
Configure the model and inputs as usual. The model you select writes the assistant turns.
In the Tools section, open Select tools and implementations and check one implementation for each tool the model may call. You can pick only one implementation per tool name.
Optional: in the User Model section, select a model to play the user, and set Max user turns. The count includes the first user message.
Click Generate Completions or Generate Dataset, enter a name for the new dataset, and confirm.

The simulated user takes turns only when the job also attaches at least one tool implementation. A user model without tool implementations produces single-turn output.

For each row, Oumi keeps the conversation up to and including the first user message and regenerates the rest. Existing assistant responses, tool results, and later user messages are not reused. For how the conversation proceeds and how to steer the simulated user with a persona, see Simulated users.

REVIEW THE OUTPUT

Each output row contains the full conversation: the tools the model could call, assistant messages with structured tool_calls, tool results, and any simulated user messages. A tool-calling assistant message often has empty text content because the call is in tool_calls. The dataset’s Overview tab lists, under Tools, the tools that the generating job made available. When a conversation reaches the assistant turn limit, Oumi replaces the last assistant message with a fixed note and handles the row as follows:

Generate Synthetic Data from Scratch drops the row. The job completes with a partial failure that lists the row with the reason “The assistant exceeded the maximum tool-call turns.”
Generate Completions keeps the row and sets tool_call_limit_exceeded to "true" in its metadata. Filter these rows out before you train on the dataset.

SPLIT THE WORK INTO TWO JOBS

Generating conversations with tools and a simulated user costs far more than generating prompts, so split the work:

Run a synthesis job without tools that produces the opening user messages and a user_persona for each row.
Review a sample of rows. Check for duplicates, confirm that personas vary, and confirm that the prompts exercise each tool.
Run Generate Completions on that dataset with tools and a user model, using a strong model as the assistant.
Evaluate the output and keep only rows that pass your evaluators. Also remove rows with repeated tool calls or the tool_call_limit_exceeded flag.

WITH THE API

Set agenticInferenceConfig on a POST /synthesis:generate-responses request, or in the recipeConfig of a POST /synthesis:run request. For its fields, see agenticInferenceConfig. Calls use your API key in the X-API-Key header, and both endpoints require a valid payment card on the account.

curl -X POST "https://api.oumi.ai/v1/projects/$OUMI_PROJECT_ID/synthesis:generate-responses" \
  -H "X-API-Key: $OUMI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "datasetId": 42,
    "displayName": "Support agent trajectories",
    "modelIdentifier": {"modelType": "ANTHROPIC_API", "modelName": "claude-sonnet-5"},
    "inferenceConfig": {"inferenceTemperature": 0.7},
    "systemInstruction": "You are a customer support agent for an online store.",
    "agenticInferenceConfig": {
      "toolImplementations": [{"toolImplementationId": 22}],
      "userModelSpec": {
        "modelIdentifier": {"modelType": "ANTHROPIC_API", "modelName": "claude-sonnet-5"},
        "inferenceConfig": {}
      },
      "maxUserTurns": 5
    }
  }'

import os
import requests

project_id = os.environ["OUMI_PROJECT_ID"]
response = requests.post(
    f"https://api.oumi.ai/v1/projects/{project_id}/synthesis:generate-responses",
    headers={"X-API-Key": os.environ["OUMI_API_KEY"]},
    json={
        "datasetId": 42,
        "displayName": "Support agent trajectories",
        "modelIdentifier": {"modelType": "ANTHROPIC_API", "modelName": "claude-sonnet-5"},
        "inferenceConfig": {"inferenceTemperature": 0.7},
        "systemInstruction": "You are a customer support agent for an online store.",
        "agenticInferenceConfig": {
            "toolImplementations": [{"toolImplementationId": 22}],
            "userModelSpec": {
                "modelIdentifier": {"modelType": "ANTHROPIC_API", "modelName": "claude-sonnet-5"},
                "inferenceConfig": {},
            },
            "maxUserTurns": 5,
        },
    },
)
response.raise_for_status()
print(response.json()["id"])

synthesis:run requires a hosted model, and Document Q&A synthesis does not accept tool implementations. synthesis:generate-responses also accepts custom and pretrained models that support tool calling.

Getting started

Oumi workflow

HOW IT WORKS

DATA SYNTHESIS METHODS

USING A DATA SYNTHESIS RECIPE

GENERATE COMPLETIONS

GENERATE FROM FAILURE MODES

WHAT EACH FAILING ROW PRODUCES

CONVERSATIONS THAT CALL TOOLS

RUBRIC JUDGE FAILURE MODES

FEWER ROWS THAN EXPECTED

TRACE A ROW TO ITS FAILURE

GENERATE SYNTHETIC DATA FROM SCRATCH

ADD TOOLS AND A SIMULATED USER

REVIEW THE OUTPUT

SPLIT THE WORK INTO TWO JOBS

WITH THE API

​DATA SYNTHESIS METHODS

​USING A DATA SYNTHESIS RECIPE

​GENERATE COMPLETIONS

​GENERATE FROM FAILURE MODES

​WHAT EACH FAILING ROW PRODUCES

​CONVERSATIONS THAT CALL TOOLS

​RUBRIC JUDGE FAILURE MODES

​FEWER ROWS THAN EXPECTED

​TRACE A ROW TO ITS FAILURE

​GENERATE SYNTHETIC DATA FROM SCRATCH

​ADD TOOLS AND A SIMULATED USER

​REVIEW THE OUTPUT

​SPLIT THE WORK INTO TWO JOBS

​WITH THE API

DATA SYNTHESIS METHODS

USING A DATA SYNTHESIS RECIPE

GENERATE COMPLETIONS

GENERATE FROM FAILURE MODES

WHAT EACH FAILING ROW PRODUCES

CONVERSATIONS THAT CALL TOOLS

RUBRIC JUDGE FAILURE MODES

FEWER ROWS THAN EXPECTED

TRACE A ROW TO ITS FAILURE

GENERATE SYNTHETIC DATA FROM SCRATCH

ADD TOOLS AND A SIMULATED USER

REVIEW THE OUTPUT

SPLIT THE WORK INTO TWO JOBS

WITH THE API