Skip to main content
An evaluator scores each row of an evaluation. Evaluators are a separate resource from recipes. An evaluation recipe references evaluators by ID, so you can apply the same evaluator across many evaluation runs. To create and manage evaluators in the app, see Manage evaluators. Request and response fields use camelCase. The API rejects unknown fields. params.evaluatorType selects one of four evaluator types. rubric_judge, classification, and extraction are behind feature flags. The API rejects these types in environments where their flags are off.

REQUEST


JUDGE PARAMS

A judge evaluator sends each row to a judge model with your prompt. The judge returns one of the labels in judgmentScores, and the label’s value becomes the row’s score. The API also accepts isMultiturn for compatibility. Oumi always stores true, so every judge reads the conversation in multi-turn format.

JUDGMENT SCORES

judgmentScores maps each label that the judge can return to a score from 0.0 to 1.0. The following example defines a yes or no judge.
When you set judgmentScores, it must have at least two keys. A key cannot be an empty string, and every value must be from 0 to 1.

DATA FIELDS

dataFields maps a field name in each row’s metadata to the title that the judge prompt shows for it. Every row in the evaluation dataset must include each configured field in its metadata.

RESPONSE FILTER MODE

responseFilterMode controls which part of the model output the judge sees. The default is EXCLUDE_THINKING.

RUBRIC JUDGE PARAMS

A rubric evaluator asks the judge several questions, called criteria, about each row. Each criterion has its own labels and scores, and aggregation combines the criterion scores into the row’s score. See Create a rubric evaluator. A rubric evaluator takes the same fields as a judge evaluator, except judgmentScores. It adds criteria and aggregation.

CRITERIA

Each criterion is one question that the judge answers. Each label is an object with two fields. A criterion id starts with a letter or underscore, followed by letters, digits, and underscores, up to 64 characters. An id cannot end in _explanation. An id also cannot reuse an evaluation result field name, such as score, explanation, judgment, request, response, conversation, or metadata. Criterion ids must be unique, and the labels within a criterion must be unique.

AGGREGATION

Only criteria with a scored label count toward the row’s score.

CLASSIFICATION PARAMS

A classification evaluator reads a predicted label from the model response and compares it with a ground-truth label in the row metadata. It does not use a judge model.

EXTRACTION PARAMS

An extraction evaluator scores what the model extracts against ground-truth labels in the row metadata. It does not use a judge model. taskConfig.taskMode selects one of three modes.

NER

STRUCTURED OUTPUT

FUNCTION CALLING


MODEL IDENTIFIER

modelIdentifier names the judge model for judge and rubric_judge evaluators. Evaluators accept only hosted models, and the model must be one that Oumi supports for evaluation.

INFERENCE CONFIG

inferenceConfig sets inference parameters for the judge model. It takes the fields described in Inference config, except guidedDecoding, which judges do not support.

VALIDATION RULES

  • displayName must be 1 to 128 characters without <, >, or &.
  • judge and rubric_judge evaluators require prompt, modelIdentifier, and inferenceConfig.
  • Judge models must be hosted models that Oumi supports for evaluation.
  • Judge inferenceConfig cannot include guidedDecoding.
  • judgmentScores, when set, needs at least two keys, no empty keys, and values from 0 to 1.
  • Rubric criterion ids must be unique, and labels within a criterion must be unique.
  • weighted_mean aggregation requires at least one criterion with a weight above 0 and at least one scored label.
  • validLabels, when set, needs at least two unique, nonempty labels.
  • groundTruthField must differ from sourceTextField. The ner and structured_output modes require sourceTextField.
  • rubric_judge, classification, and extraction require their feature flags.

EXAMPLES

RESPONSE QUALITY JUDGE

This judge checks whether a response is accurate, clear, and helpful. Use it for general evaluation of assistants and instruction-following tasks.

SAFETY JUDGE

This judge detects unsafe or policy-violating responses. Use it together with a quality judge when the model talks to end users.

CORRECTNESS JUDGE WITH A REFERENCE ANSWER

This judge compares each response with a reference answer. Every row in the evaluation dataset must include ground_truth_answer in its metadata.

TOOL-USE RUBRIC

This rubric checks how an agent handles a conversation with tools. The results show a separate score for each behavior. The not_applicable label has no score, so a conversation where a situation never came up is not penalized. With min aggregation, any failing criterion fails the conversation.

WEIGHTED RUBRIC

This rubric gives one criterion twice the weight of the other in the weighted_mean aggregation.
Set inferenceTemperature to 0.0 for judge models to get consistent scores across runs. Keep the default EXCLUDE_THINKING mode unless the judge must assess the reasoning as well as the final response.