Skip to main content
Once your model is in production, you can begin using it for inference and applying it to downstream applications. Oumi keeps all your project assets (models, datasets, evaluations, evaluators, and recipes) automatically version controlled, so every deployment is reproducible. As data and usage patterns change, performance will invariably degrade, making periodic retraining necessary. Quality monitoring tells you when that moment arrives: judge scores over live traffic trend downward before users complain. Your deployment’s inference logs are the best source of material for the fix: export a slice of real production traffic to a dataset, then evaluate, retrain, and redeploy within the same workflow.

Quality monitoring

Watch judge scores over live traffic and catch regressions early.

Health metrics

Track your deployment’s traffic, errors, latency, and throughput.

Running evaluations

Using the Builder to evaluate your model.

Diagnosing failure modes

Generate a new dataset directly from an evaluator’s identified failure modes.