SERVING METRICS

Serving metrics show whether a deployment serves requests reliably and quickly. They come from the inference logs, so every deployment with logging on reports them, with no metrics stack to set up. Serving metrics tell you whether the deployment works. To learn whether its answers are good, use quality monitoring. You can also ask FAiE (Frontier AI Engineer Agent) how a deployment is doing. It reads the same metrics, and with monitoring on, it can tell you whether a problem is operational or a drop in quality.

WHAT EACH METRIC MEASURES

The deployment’s Health tab shows these metrics for the Time window you choose.

Health tab with the summary metrics and the Requests, Request Latency, Time to First Token, Token Throughput, and Error Rate charts

Metric	What it measures
Requests and Avg Requests / min	How much traffic the deployment served
Error Rate	The share of requests that failed. Hover to see the count.
Request Latency	The time from request to complete response, at the 50th, 95th, and 99th percentiles. It includes failed requests.
Time to First Token	The time until the first token of the response. For a non-streaming request, it equals the full latency.
Token Throughput	Input and output tokens per second

READ THE CHARTS

What you see	Likely cause	What to do
Errors rise	Cold starts, rate limits, or prompts that are too long	Open the failed requests on the Inference Logs tab and read their messages. See Errors.
Time to first token is high on some requests	Cold starts, or requests that wait for a busy replica	Set min replicas to 1 to avoid cold starts, or raise max replicas for concurrent traffic. See Autoscaling.
Latency is high, but time to first token is normal	Long responses	Lower `max_tokens`, or stream the response so users see it sooner.
Requests drop to zero	No traffic, or an application error before the request reaches Oumi	Check your application. With no traffic, quality monitoring has nothing to score.

Getting started

Oumi workflow

SERVING METRICS

WHAT EACH METRIC MEASURES

READ THE CHARTS

​WHAT EACH METRIC MEASURES

​READ THE CHARTS

WHAT EACH METRIC MEASURES

READ THE CHARTS