Skip to main content
Traditional approaches to evaluating model quality don’t scale well. Human review is slow and costly, rule-based checks are often too rigid for open-ended outputs, and conventional metrics frequently fail to capture what users actually value. Evaluators solve these problems by using LLM-based judges to assess response quality at scale. These judges can measure nuanced attributes such as accuracy, completeness, and tone across thousands of examples, while providing detailed per-example feedback that makes it easy to pinpoint where a model is falling short. For tasks with known answers, classification and extraction evaluators score outputs against ground truth. The Oumi Agent can create judge evaluators from a natural-language description of your task. It turns your goal into scoring criteria, labels, and edge cases that you can review and reuse.

​
HOW IT WORKS

  1. Your model generates responses to a set of prompts
  2. An evaluator scores each response using its configured method
  3. You get results across your dataset, showing where your model excels and where it struggles

​
WHAT TO DEFINE IN A JUDGE EVALUATOR

For a judge evaluator, define:
  • Evaluation criteria: A prompt describing what the judge should assess (e.g., “Is the response accurate and complete?”)
  • Judgment labels: The rating scale the judge uses (e.g., “poor”, “acceptable”, “good”, “excellent”)
  • Scoring: Numeric scores mapped to each label
  • Judge model: Which hosted LLM runs the evaluation (e.g., GLM-5, Qwen3-235B-A22B-Instruct-2507)

​
BUILT-IN & CUSTOM JUDGE EVALUATORS

Oumi includes built-in judge evaluators (such as instruction following, safety, topic adherence, and truthfulness) to help you quickly establish baselines and gather early feedback. You can review, edit, and reuse these evaluators across evaluations, or create a custom judge evaluator using the Builder. You can also describe your desired judge evaluator in natural language with the Oumi Agent, specifying scoring criteria, selecting the judge model, and including additional dataset fields for context as needed.
Custom judge evaluators are reusable and should focus on a single, clearly defined property to ensure consistent and reliable results.

​
WHY EVALUATORS MATTER

  • Before training: Benchmark a base model to see where it falls short
  • After training: Measure whether fine-tuning actually improved quality
  • Compare models: Run the same evaluators on different models to see which performs better
  • Identify failure modes: Find specific examples where the model struggles, then use those insights to improve your training data

​
EXAMPLE JUDGE CRITERIA

For a customer support bot, you might create separate judge evaluators for:
  • Accuracy: Did the response contain correct information?
  • Tone: Was the response empathetic and professional?
  • Completeness: Did it fully address the customer’s question?
  • Policy compliance: Did it follow company guidelines?

​
WHAT’S NEXT

Defining evaluators

Establish criteria for measuring model performance

Saved evaluators

Save and reuse evaluator configurations