Skip to main content
Evaluators define what strong performance looks like for your task and are essential for building effective custom AI. Oumi provides built-in judge evaluators and tools for creating custom evaluators.

​
ACCESSING YOUR EVALUATORS

You can view and manage your evaluators on the Evaluators page.

​
CHOOSING AN EVALUATOR TYPE

  • Judge: Use an LLM to assess open-ended qualities such as accuracy, tone, or instruction following.
  • Classification: Compare a predicted label with a ground-truth label.
  • Extraction: Compare named entities, structured output, or function calls with ground truth.
Classification and extraction evaluators do not use a judge model. The rest of this guide explains how to create a judge evaluator.

​
USING OUMI’S BUILT-IN EVALUATORS

Oumi includes a growing set of general-purpose judge evaluators that work well across many tasks:
  • Instruction following: Is the model response faithful to the prompt?
  • Safety: Is the response free from harmful language?
  • Topic adherence: Does the response stay on topic with respect to the prompt?
  • Truthfulness: Is the response correct as far as is known from the model knowledge and context window?
These built-in evaluators are ideal when:
  • You are establishing an initial baseline
  • You want fast feedback without custom configuration
  • Your task aligns with common evaluation dimensions
Built-in evaluators can be reviewed, edited, or extended as needed.

​
CREATING A CUSTOM JUDGE EVALUATOR

Oumi also enables you to build custom judge evaluators tailored to your specific requirements.
  • On the Evaluators page, click on the Create Evaluator.
  • In the Builder, click on Create an Evaluator.
On the Inputs tab, provide the following fields to define your evaluator:

​
EVALUATOR IDENTITY

Provide a name and description for your judge.
  • Name: The name of the judge; should indicate what this judge evaluates (e.g., “Instruction Following”)
  • What should the judge evaluate?: A clear sentence describing the evaluation goal (e.g., “Evaluate whether the response accurately follows all instructions provided in the user’s request”)

​
JUDGMENTS

Judgments define the possible outcomes of an evaluation and how each outcome is scored. They tell the evaluator how to label a response and what that label means. Click Add Judgment and provide the follow three fields to add a new judgement:
  • Label: The name of the outcome (for example, “Correct”, “Incorrect”, “Adherent”, “Non-Adherent”). This is what the evaluator will output.
  • Condition: A clear description of when this label should be applied. This acts as guidance for the evaluator, helping it decide which label best fits a given response.
  • Score: A numeric value associated with the label (such as 1 for correct, 0 for incorrect). Scores allow you to quantify performance and compare results across examples.

​
HOW JUDGMENTS WORK

Judgments define the grading system for your task. For example:
  • In a classification task:
    • Correct → score 1
    • Incorrect → score 0
  • In a quality evaluation task:
    • High Quality → score 1
    • Acceptable → score 0.5
    • Poor → score 0
By defining clear labels and conditions, you ensure that evaluations are consistent, interpretable, and aligned with your goals.

​
DATA FIELDS

Define the data fields that the judge should evaluate.
  • Input Type - Specify whether the input type is:
    • single turn (a user request and an assistant response)
    • single turn (an assistant response)
    • mult-turn (a conversation between a user and an assistant)
  • Additional Fields - Provide additional fields (as Key, Display Name, and Description) for the judge to consider in its evaluation

​
EVALUATION CRITERIA (OPTIONAL)

Click Add Criterion to add specific criteria the judge should consider. Each item should consist of a Name that describes the criteria (e.g., “Format Compliance”) and Description (e.g., “Response should be in JSON format and not exceed 1000 characters”).

​
EXAMPLES (OPTIONAL)

Click Add Example to add the additional fields to provide examples to guide the judge’s behavior. Each example should consist of a Name that describes the criteria (e.g., “Format Compliance”) and Description (e.g., “Response should be in JSON format and not exceed 1000 characters”).

​
MODEL SETTINGS

Specify the model and settings used to calculate the score according to the instructions.
This model is not to be confused with the model under evaluation that produced the (prompt, response) pairs

​
USING THE AGENT

You can also use the Agent to define your judges using natural language prompts. Oumi will suggest evaluator patterns based on your task (for example, comprehensiveness, groundedness, fluency, format adherence).
See the built-in evaluators for example of correctly formatted prompts.
Models from providers like OpenAI are available without adding a provider API key. When you use your own key from Provider API keys, Oumi does not bill you for that model usage.
Evaluators can use models from leading providers such as OpenAI and Anthropic. Custom evaluators are saved and reusable across evaluations.

​
BEST PRACTICES FOR DESIGNING CUSTOM EVALUATORS

Well-designed custom evaluators produce consistent, actionable results. The following best practices ensure that your custom evaluators are clear, focused, and aligned with your success criteria.
  • Be specific about what success looks like.
  • Avoid overly broad instructions.
  • Each evaluator should measure a single, non-overlapping property.
  • Start simple, then refine based on failure modes.