Skip to main content
Evaluators define what strong performance looks like for your task and are essential for building effective custom AI.

​
ACCESSING YOUR EVALUATORS

You can view and manage your evaluators on the Evaluators page.

​
CHOOSING AN EVALUATOR TYPE

  • Judge: Use an LLM to assess open-ended qualities such as accuracy, tone, or instruction following.
  • Rubric: Use an LLM to check several criteria in one pass, with a separate verdict and score for each.
  • Classification: Compare a predicted label with a ground-truth label.
  • Extraction: Compare named entities, structured output, or function calls with ground truth.
Classification and extraction evaluators do not use a judge model. The rest of this guide explains how to create judge and rubric evaluators.

​
CREATING A CUSTOM JUDGE EVALUATOR

Build a judge evaluator tailored to your specific requirements.
  • On the Evaluators page, click on the Create Evaluator.
  • In the Builder, click on Create an Evaluator.
On the Inputs tab, provide the following fields to define your evaluator:

​
EVALUATOR IDENTITY

Provide a name and description for your judge.
  • Name: The name of the judge; should indicate what this judge evaluates (e.g., “Instruction Following”)
  • What should the judge evaluate?: A clear sentence describing the evaluation goal (e.g., “Evaluate whether the response accurately follows all instructions provided in the user’s request”)

​
JUDGMENTS

Judgments define the possible outcomes of an evaluation and how each outcome is scored. They tell the evaluator how to label a response and what that label means. Click Add Judgment and provide the follow three fields to add a new judgement:
  • Label: The name of the outcome (for example, “Correct”, “Incorrect”, “Adherent”, “Non-Adherent”). This is what the evaluator will output.
  • Condition: A clear description of when this label should be applied. This acts as guidance for the evaluator, helping it decide which label best fits a given response.
  • Score: A numeric value associated with the label (such as 1 for correct, 0 for incorrect). Scores allow you to quantify performance and compare results across examples.

​
HOW JUDGMENTS WORK

Judgments define the grading system for your task. For example:
  • In a classification task:
    • Correct → score 1
    • Incorrect → score 0
  • In a quality evaluation task:
    • High Quality → score 1
    • Acceptable → score 0.5
    • Poor → score 0
By defining clear labels and conditions, you ensure that evaluations are consistent, interpretable, and aligned with your goals.

​
DATA FIELDS

The judge always sees the full conversation: each user message, each assistant message, and each tool call and tool result. Use this section to show the judge more context.
  • Conversation system message - Check this box to show the judge the system message of each conversation. Use it when a criterion depends on the assistant’s instructions, such as a policy it must follow. In the API, this is includeConversationSystem (false by default).
  • Metadata fields - Click Add Field to show the judge a value from each row’s metadata. Give each field a Key in the dataset’s metadata, such as ground_truth, and a Display Title for the judge.

​
EVALUATION CRITERIA (OPTIONAL)

Click Add Criterion to add specific criteria the judge should consider. Each item should consist of a Name that describes the criteria (e.g., “Format Compliance”) and Description (e.g., “Response should be in JSON format and not exceed 1000 characters”).

​
EXAMPLES (OPTIONAL)

Click Add Example to add the additional fields to provide examples to guide the judge’s behavior. Each example should consist of a Name that describes the criteria (e.g., “Format Compliance”) and Description (e.g., “Response should be in JSON format and not exceed 1000 characters”).

​
MODEL SETTINGS

Specify the model and settings used to calculate the score according to the instructions. By default, the judge does not see content inside <think> tags. Check Include thinking tokens in response? to show it. In the API, this is responseFilterMode (EXCLUDE_THINKING by default, or THINKING_AND_RESPONSE).
This model is not to be confused with the model under evaluation that produced the (prompt, response) pairs

​
CREATING A RUBRIC EVALUATOR

A rubric evaluator checks several criteria in a single judge call. Each criterion gets its own label, score, and explanation, and the criterion scores combine into one overall score for the row. Use a rubric when you want to check several behaviors of the same conversation, such as the ones in Judging tool use and conversations. The results show exactly which criterion failed, instead of one score for everything. In the Builder, click Create an Evaluator, then choose Rubric Evaluator.

​
SYSTEM INSTRUCTION

Tell the judge what it is assessing and how to approach it. For example: “You review conversations between a support assistant and a customer. Judge each criterion on its own.”

​
CRITERIA

Click Add Criterion for each behavior you want to check, and fill in:
  • ID: A short name, such as grounded. It starts with a letter or underscore and contains only letters, digits, and underscores.
  • Description: The question the judge answers for this criterion. Write it so that the conversation alone can decide it.
  • Value and Score: The labels the judge can return. Click Add Label to add one. Each criterion needs at least two.
  • Weight: How much the criterion counts in the overall score. The default is 1.0.
  • Ask for an explanation: The judge explains its reasoning before it picks a label. This is on by default.

​
LABELS AND SCORES

The judge picks exactly one label for each criterion. Give each label a score from 0 to 1:
  • Pass or fail: yes = 1, no = 0
  • Graded: correct = 1, minor_error = 0.5, wrong = 0
  • No score: Leave the score empty for a label that should not count, such as not_applicable when the situation the criterion checks never came up. An unscored label is left out of the overall score instead of counting as 0.
The judge sees the labels in the order you list them. If more than one label could apply, say in the description which one to pick.

​
AGGREGATION

Choose how the criterion scores combine into the row’s overall score: Only criteria with a scored label count. With Weighted mean, at least one criterion needs a weight above 0 and a label with a score.

​
DATA FIELDS AND MODEL SETTINGS

These work the same way as for a judge evaluator. See Data fields and Model settings. The judge answers every criterion in one response, so allow enough Max Tokens for all of them.

​
USING FAiE

You can also use FAiE to define your judges using natural language prompts. Oumi will suggest evaluator patterns based on your task (for example, comprehensiveness, groundedness, fluency, format adherence).
Models from providers like OpenAI are available without adding a provider API key. When you use your own key from Provider API keys, Oumi does not bill you for that model usage.
Evaluators can use models from leading providers such as OpenAI and Anthropic. Custom evaluators are saved and reusable across evaluations.

​
BEST PRACTICES FOR DESIGNING CUSTOM EVALUATORS

Well-designed custom evaluators produce consistent, actionable results. The following best practices ensure that your custom evaluators are clear, focused, and aligned with your success criteria.
  • Be specific about what success looks like.
  • Avoid overly broad instructions.
  • Each evaluator should measure a single, non-overlapping property.
  • Start simple, then refine based on failure modes.

​
JUDGING TOOL USE AND CONVERSATIONS

When the model under test calls tools, a correct final answer is not enough. Judge the process that the conversation shows. Consider the following behaviors, in addition to ordinary correctness and tone:
  • Tool selection: The model used the right tool, and looked information up before it stated it instead of answering from memory.
  • Argument grounding: Every ID and parameter that the model sent to a tool comes from the user or from an earlier tool result. The model did not invent values.
  • Answer grounding: Every fact in the model’s answer has a tool result behind it, unless the model states its uncertainty. This is usually the largest class of failures.
  • Efficiency: The model did not repeat identical calls. It called a tool again only when a constraint changed or the user asked something new.
  • When not to call: The model declined out-of-scope or unsafe requests directly, and did not refuse a harmless request that looks similar.
  • Error handling: After an empty or failed result, the model retried with a changed query before it told the user that something does not exist. It did not loop.
  • Faithful self-report: The model did not claim an action that it never took, or describe an outcome before the tool call returned.
  • Conversation management: The model asked a question only when the request was truly underspecified. Otherwise it proceeded and stated its assumption. It covered every part of a multi-part request and recognized when the task was done.
A rubric evaluator fits this well: make each behavior one criterion. Write each criterion so that the transcript alone can decide it, and keep tool names and numbers in the wording. Give each behavior its own criterion or evaluator, so a strong score on style cannot hide a fabricated fact. Decide in advance how a criterion scores when its trigger never occurs. For example, “did not call a tool on an out-of-scope request” says nothing about a conversation with no out-of-scope request. In a rubric, give such a criterion a not_applicable label with no score. LLM judges are least reliable on answer grounding when the user pushes back, so write those criteria most precisely. Also check tool activation. A conversation can pass every criterion and still never call the tool that the row was built to exercise.