Skip to main content

​
OVERVIEW

Compounding intelligence turns production feedback into better models. Quality monitoring reveals where a deployed model struggles, failure mode analysis groups recurring problems, and dataset generation creates targeted examples for retraining. This guide walks through the workflow in the UI. You can also ask the Oumi Agent to guide you through the same process, from analyzing your traffic on the Insights tab to a retrained model ready to redeploy.
Before you begin, enable quality monitoring on a deployment and generate a failure modes report from its production traffic.

​
THE LOOP

  1. Monitor. Judge evaluators score a sample of production traffic and plot trends on the Insights tab.
  2. Diagnose. Analyze the traffic behind a score drop to identify recurring failure modes.
  3. Generate. Create synthetic training data that targets the failure modes you select.
  4. Retrain. Train a new model on the generated data and your original training set.
  5. Redeploy and verify. Deploy the new model and use monitoring to confirm the improvement.

​
STEP 1: CREATE A FAILURE MODES REPORT FROM INSIGHTS

Open the deployment’s Insights tab. Each judge evaluator has its own score trend:
1

Select a time range

Click Select Area to Investigate and drag across the score drop you want to examine.
2

Inspect the scored logs

Click View Selected Logs to confirm that the selected traffic contains the behavior you want to fix. Expand any row to read the judge explanations.
3

Analyze the failures

Click Run Failure Modes Analysis. Oumi groups the failing responses into recurring patterns and opens the completed report on the deployment’s Failure Modes tab.
The report helps you address recurring causes instead of fixing individual responses. See quality monitoring for the full investigation workflow.

​
STEP 2: GENERATE A DATASET FROM THE FAILURE MODES

On the Failure Modes tab, select the failure modes you want to address and the specific data items within them. Click Generate Dataset to open a confirmation dialog with three settings: Click Generate Dataset to start generation. Oumi opens the Datasets page, where the new dataset appears when the job finishes.

​
ADVANCED MODE

For more control, click Advanced…. The Builder opens with your selection preloaded and provides additional settings: To edit the raw synthesis configuration, click Edit as JSON, make your changes, and click Save to return them to the form. Click Run Synthesis, name the dataset, and start the job.

​
STEP 3: RETRAIN WITH THE NEW DATASET

Train the next version of your model on the original and generated datasets. Using both preserves the original examples while adding targeted coverage for the newly identified failures:
  1. On the Models page, click Train New Model and select Supervised Fine-Tuning.
  2. Select your base model.
  3. In the training dataset selector, choose both the generated dataset and the original training set.
  4. Launch the job. See running training for the full configuration options.
When training finishes, deploy the new model and direct traffic to it. Continue monitoring to verify that the targeted scores improve. If scores drop again, repeat the loop with the new traffic.

​
WHAT’S NEXT

Quality monitoring

Configure judge evaluators and collect the score trends that start this loop.

Diagnosing failure modes

Learn how reports group recurring failures in evaluations and production traffic.

Running training

Configure and launch a fine-tuning job with your improved training data.

Deploy a model

Serve the retrained model and direct production traffic to it.