Request and response fields use camelCase. The API rejects unknown fields.
params.evaluatorType selects one of four evaluator types.
rubric_judge, classification, and extraction are behind feature flags. The API rejects these types in environments where their flags are off.
REQUEST
JUDGE PARAMS
A judge evaluator sends each row to a judge model with your prompt. The judge returns one of the labels injudgmentScores, and the label’s value becomes the row’s score.
The API also accepts
isMultiturn for compatibility. Oumi always stores true, so every judge reads the conversation in multi-turn format.
JUDGMENT SCORES
judgmentScores maps each label that the judge can return to a score from 0.0 to 1.0. The following example defines a yes or no judge.
judgmentScores, it must have at least two keys. A key cannot be an empty string, and every value must be from 0 to 1.
DATA FIELDS
dataFields maps a field name in each row’s metadata to the title that the judge prompt shows for it. Every row in the evaluation dataset must include each configured field in its metadata.
RESPONSE FILTER MODE
responseFilterMode controls which part of the model output the judge sees. The default is EXCLUDE_THINKING.
RUBRIC JUDGE PARAMS
A rubric evaluator asks the judge several questions, called criteria, about each row. Each criterion has its own labels and scores, andaggregation combines the criterion scores into the row’s score. See Create a rubric evaluator.
A rubric evaluator takes the same fields as a judge evaluator, except judgmentScores. It adds criteria and aggregation.
CRITERIA
Each criterion is one question that the judge answers.
Each label is an object with two fields.
A criterion
id starts with a letter or underscore, followed by letters, digits, and underscores, up to 64 characters. An id cannot end in _explanation. An id also cannot reuse an evaluation result field name, such as score, explanation, judgment, request, response, conversation, or metadata. Criterion ids must be unique, and the labels within a criterion must be unique.
AGGREGATION
Only criteria with a scored label count toward the row’s score.CLASSIFICATION PARAMS
A classification evaluator reads a predicted label from the model response and compares it with a ground-truth label in the row metadata. It does not use a judge model.EXTRACTION PARAMS
An extraction evaluator scores what the model extracts against ground-truth labels in the row metadata. It does not use a judge model.taskConfig.taskMode selects one of three modes.
NER
STRUCTURED OUTPUT
FUNCTION CALLING
MODEL IDENTIFIER
modelIdentifier names the judge model for judge and rubric_judge evaluators. Evaluators accept only hosted models, and the model must be one that Oumi supports for evaluation.
INFERENCE CONFIG
inferenceConfig sets inference parameters for the judge model. It takes the fields described in Inference config, except guidedDecoding, which judges do not support.
VALIDATION RULES
displayNamemust be 1 to 128 characters without<,>, or&.judgeandrubric_judgeevaluators requireprompt,modelIdentifier, andinferenceConfig.- Judge models must be hosted models that Oumi supports for evaluation.
- Judge
inferenceConfigcannot includeguidedDecoding. judgmentScores, when set, needs at least two keys, no empty keys, and values from0to1.- Rubric criterion ids must be unique, and labels within a criterion must be unique.
weighted_meanaggregation requires at least one criterion with a weight above0and at least one scored label.validLabels, when set, needs at least two unique, nonempty labels.groundTruthFieldmust differ fromsourceTextField. Thenerandstructured_outputmodes requiresourceTextField.rubric_judge,classification, andextractionrequire their feature flags.
EXAMPLES
RESPONSE QUALITY JUDGE
This judge checks whether a response is accurate, clear, and helpful. Use it for general evaluation of assistants and instruction-following tasks.SAFETY JUDGE
This judge detects unsafe or policy-violating responses. Use it together with a quality judge when the model talks to end users.CORRECTNESS JUDGE WITH A REFERENCE ANSWER
This judge compares each response with a reference answer. Every row in the evaluation dataset must includeground_truth_answer in its metadata.
TOOL-USE RUBRIC
This rubric checks how an agent handles a conversation with tools. The results show a separate score for each behavior. Thenot_applicable label has no score, so a conversation where a situation never came up is not penalized. With min aggregation, any failing criterion fails the conversation.
WEIGHTED RUBRIC
This rubric gives one criterion twice the weight of the other in theweighted_mean aggregation.
inferenceTemperature to 0.0 for judge models to get consistent scores across runs. Keep the default EXCLUDE_THINKING mode unless the judge must assess the reasoning as well as the final response.