Skip to main content

​
OVERVIEW

Quality monitoring uses LLM judges to score a sample of production traffic. The deployment’s Insights tab plots each evaluator’s scores over time, helping you spot regressions before they become widespread.
Quality monitoring works on managed deployments with inference logging enabled.

​
HOW IT WORKS

A monitor attaches to a deployment and holds two things: a set of judge evaluators and a sampling rate.
  1. Your deployment serves a request, and inference logging records it.
  2. The monitor samples the percentage of requests you configured.
  3. The attached judge evaluators score each sampled request.
  4. The Insights tab plots a score trend for each evaluator.

​
SET UP MONITORING

There are three ways to start monitoring a deployment.

​
WITH THE OUMI AGENT

Ask the Oumi Agent to monitor a specific deployment. It checks that inference logging is enabled and proposes judge evaluators and a sampling rate. Review the proposal, then approve it to start scoring.

​
DURING DEPLOYMENT

The Deploy Model dialog includes a Monitoring step, right after you configure the deployment. It is optional: add one or more judge evaluators and pick a sampling rate to enable monitoring, or click Skip to move on. Anything you set here can be changed later from the deployment’s Info & Configuration tab.

​
FROM THE CONFIGURATION TAB

On the deployment detail page, open the Info & Configuration tab and scroll to the Monitoring section:
  • Status shows whether a monitor is actively scoring this deployment’s traffic.
  • Evaluators lists the judges scoring sampled traffic. Click + Add evaluator to attach one; adding the first evaluator creates the monitor with a 10% sampling rate.
  • Sampling rate sets the share of production requests that get scored. The dropdown offers 10%, 20%, 50%, and 100%.
Changes apply immediately.

​
READ THE INSIGHTS TAB

The Insights tab is where monitoring results land.
  • The Time window selector zooms from 1 hour out to 30 days, with a Custom option for an exact range. The tab shows up to 30 days of score history.
  • A summary pill per evaluator shows its mean score over the visible window, as a percentage.
  • Each evaluator gets its own score trend chart, one point per scored request.
If a judge fails to produce scores, a warning badge appears on that evaluator’s chart with the failure count and reasons on hover, such as provider quota, authentication, rate limits, or timeouts. These are judge failures, not low model scores; they usually point at the judge’s provider key or configuration rather than your model.

​
INVESTIGATE A DROP

When a trend dips, drill into the exact traffic behind it:
1

Select the range

Click Select Area to Investigate and drag across any chart to mark the time range you care about.
2

View the scored logs

Click View Selected Logs to open the Scored Inference Logs table: one row per scored request, with the judgment each evaluator gave it. Expand a row to read the full judge explanations for that request.
3

Run a failure modes analysis

Click Run Failure Modes Analysis to cluster the failing responses in the selected range into recurring patterns. The report opens on the deployment’s Failure Modes tab when it is ready.
A failure modes report groups the reasons that production responses scored poorly, helping you address the highest-impact pattern first. From there, use compounding intelligence to generate targeted training data, retrain the model, and verify the improvement after redeployment.

​
MANAGE MONITORED EVALUATORS

  • Versions are pinned. A monitor scores with the exact evaluator version you attached.
  • Removing an evaluator stops scoring against it. You can add it back later.
Removing the last monitored evaluator turns off monitoring for the deployment and permanently deletes the monitor along with all of its historical scores. This cannot be undone.

​
WHAT’S NEXT

Compounding intelligence

Turn monitoring insights into targeted training data and an improved model.

Health metrics

Watch the operational side: traffic, errors, latency, and throughput.