WHY GENERATE DATA INSTEAD OF TRAINING ON LOGS
Raw production logs record what the model answered, including its mistakes. Training on them teaches the model its own failures. Failure-mode synthesis writes corrected examples for the requests that failed, so the next model can learn a correction. Combining these examples with your original training data can help retain useful behavior, but you should check for regressions with held-out evaluations.THE LOOP
- Monitor: Quality monitoring scores a sample of the deployment’s traffic.
- Find failure modes: Group the failing requests in a time range by cause.
- Generate data: Write training examples that target the failure modes you choose.
- Retrain: Train a new version of the model on the generated data and the original training set.
- Evaluate, deploy, and verify: Evaluate the new model against your held-out baseline before deploying it, then confirm its quality on live traffic.
FIND FAILURE MODES
- On the deployment’s Insights tab, click
Select area to investigate, and drag across the time range you want to fix. - Click
View selected logs, and read some low-scoring requests to confirm that the model, not the judge, is wrong. - Click
Run failure-mode analysis. Oumi groups the failures in the range by cause, and the report opens on the Failure Modes tab.
GENERATE DATA
- On the Failure Modes tab, select the failure modes to fix, or click
Select All. - Click
Generate Dataset. - Choose the Model that writes the examples, and optionally a Dataset name and the Samples per item. Samples per item sets how many new examples Oumi writes for each failing request.
- Click
Generate Dataset. The new dataset appears on the Datasets page when the job finishes.
Advanced… instead. See Generate from failure modes.
RETRAIN
Open the deployed model’s page, and clickRetrain. The Builder opens a training job for a new version of the model. Select both the generated dataset and your original training set, then launch the job. See Launch training jobs.
DEPLOY AND VERIFY
A deployment serves one model version, so the retrained model needs a new deployment. A monitor belongs to one deployment, so add the same evaluators to the new deployment’s monitor.- Deploy the new version, with the same evaluators in its Monitoring step.
- Point your application at the new deployment path.
- Compare the new deployment’s Insights trends with the old deployment’s for the same evaluators.
- When the new deployment meets your quality requirements, retire the old one.