Skip to main content
Traditional approaches to evaluating model quality don’t scale well. Human review is slow and costly, rule-based checks are often too rigid for open-ended outputs, and conventional metrics frequently fail to capture what users actually value. Evaluators solve these problems by using LLM-based judges to assess response quality at scale. These judges can measure nuanced attributes such as accuracy, completeness, and tone across thousands of examples, while providing detailed per-example feedback that makes it easy to pinpoint where a model is falling short. Rubric evaluators check several criteria in one pass and score each one separately. For tasks with known answers, classification and extraction evaluators score outputs against ground truth. FAiE can create judge evaluators from a natural-language description of your task. It turns your goal into scoring criteria, labels, and edge cases that you can review and reuse.

​
HOW IT WORKS

  1. Your model generates responses to a set of prompts
  2. An evaluator scores each response using its configured method
  3. You get results across your dataset, showing where your model excels and where it struggles

​
WHAT TO DEFINE IN A JUDGE EVALUATOR

For a judge evaluator, define:
  • Evaluation criteria: A prompt describing what the judge should assess (e.g., “Is the response accurate and complete?”)
  • Judgment labels: The rating scale the judge uses (e.g., “poor”, “acceptable”, “good”, “excellent”)
  • Scoring: Numeric scores mapped to each label
  • Judge model: Which hosted LLM runs the evaluation (e.g., GLM-5, Qwen3-235B-A22B-Instruct-2507)

​
CREATING JUDGE EVALUATORS

Create a judge evaluator in the Builder, then reuse it across evaluations. You can also describe your desired judge evaluator in natural language with FAiE, specifying scoring criteria, selecting the judge model, and including additional dataset fields for context as needed.
Custom judge evaluators are reusable and should focus on a single, clearly defined property to ensure consistent and reliable results.

​
WHY EVALUATORS MATTER

  • Before training: Benchmark a base model to see where it falls short
  • After training: Measure whether fine-tuning actually improved quality
  • Compare models: Run the same evaluators on different models to see which performs better
  • Identify failure modes: Find specific examples where the model struggles, then use those insights to improve your training data

​
EXAMPLE JUDGE CRITERIA

For a customer support bot, you might create separate judge evaluators for:
  • Accuracy: Did the response contain correct information?
  • Tone: Was the response empathetic and professional?
  • Completeness: Did it fully address the customer’s question?
  • Policy compliance: Did it follow company guidelines?
To see all four for each conversation in one place, create a single rubric evaluator with one criterion for each.

​
WHAT’S NEXT

Defining evaluators

Establish criteria for measuring model performance

Saved evaluators

Save and reuse evaluator configurations