Skip to main content
Continuous improvement uses evidence from production traffic to guide the next version of a specialized model. Monitoring reveals where quality drops, failure mode analysis groups the likely causes, and synthesis creates targeted examples. You can retrain, evaluate the new model against your baseline, and deploy it when the results justify the change. You can run the whole loop with FAiE (Frontier AI Engineer Agent), from reading the Insights tab to a retrained model ready to deploy.

WHY GENERATE DATA INSTEAD OF TRAINING ON LOGS

Raw production logs record what the model answered, including its mistakes. Training on them teaches the model its own failures. Failure-mode synthesis writes corrected examples for the requests that failed, so the next model can learn a correction. Combining these examples with your original training data can help retain useful behavior, but you should check for regressions with held-out evaluations.

THE LOOP

  1. Monitor: Quality monitoring scores a sample of the deployment’s traffic.
  2. Find failure modes: Group the failing requests in a time range by cause.
  3. Generate data: Write training examples that target the failure modes you choose.
  4. Retrain: Train a new version of the model on the generated data and the original training set.
  5. Evaluate, deploy, and verify: Evaluate the new model against your held-out baseline before deploying it, then confirm its quality on live traffic.
Before you start, set up quality monitoring on the deployment. Failure mode analysis and data generation are billed model work, and their cost grows with the number of failures.

FIND FAILURE MODES

  1. On the deployment’s Insights tab, click Select area to investigate, and drag across the time range you want to fix.
  2. Click View selected logs, and read some low-scoring requests to confirm that the model, not the judge, is wrong.
  3. Click Run failure-mode analysis. Oumi groups the failures in the range by cause, and the report opens on the Failure Modes tab.
To learn how to read a failure mode and its rows, see Diagnosing failure modes.

GENERATE DATA

  1. On the Failure Modes tab, select the failure modes to fix, or click Select All.
  2. Click Generate Dataset.
  3. Choose the Model that writes the examples, and optionally a Dataset name and the Samples per item. Samples per item sets how many new examples Oumi writes for each failing request.
  4. Click Generate Dataset. The new dataset appears on the Datasets page when the job finishes.
To split the data into training, validation, and test sets, or to add it to an existing dataset, click Advanced… instead. See Generate from failure modes.

RETRAIN

Open the deployed model’s page, and click Retrain. The Builder opens a training job for a new version of the model. Select both the generated dataset and your original training set, then launch the job. See Launch training jobs.

DEPLOY AND VERIFY

A deployment serves one model version, so the retrained model needs a new deployment. A monitor belongs to one deployment, so add the same evaluators to the new deployment’s monitor.
  1. Deploy the new version, with the same evaluators in its Monitoring step.
  2. Point your application at the new deployment path.
  3. Compare the new deployment’s Insights trends with the old deployment’s for the same evaluators.
  4. When the new deployment meets your quality requirements, retire the old one.
If new failures appear, run the loop again on the new deployment’s traffic.