OVERVIEW

The following describes the request schema used to create judge and rubric evaluators in Oumi, including:

Metadata about the evaluator
Evaluation parameters
Model configuration
Scoring settings

Fields marked with (*) are required.

SCHEMA STRUCTURE

{
  "displayName": "",
  "description": "",
  "params": {
    "evaluatorType": "judge",
    "prompt": "",
    "modelIdentifier": {},
    "inferenceConfig": {}
  }
}

DISPLAYNAME *

Human-readable name of the evaluator.

PROPERTIES

Field	Type	Required	Description
displayName	string	✓	Name used to identify the evaluator
description	string		Optional description of the evaluator

CONSTRAINTS

Field	Constraint
displayName	Minimum length: `1`

EXAMPLE

{
  "displayName": "Answer Quality Evaluator",
  "description": "Evaluates model responses using a judge model."
}

JUDGE PARAMS *

Defines the evaluation parameters and scoring behavior.

PROPERTIES

Field	Type	Required	Description
evaluatorType	string	✓	Evaluator type
prompt	string	✓	Prompt used to guide the judge
modelIdentifier	object	✓	Hosted model used by the judge
inferenceConfig	object	✓	Inference settings for the judge model
judgmentScores	object		Judgment labels mapped to scores from `0.0` to `1.0`
dataFields	object		Dataset field names mapped to display titles
responseFilterMode	string		Portion of the response available to the judge
includeConversationSystem	boolean		Includes the system message in the judge input

EVALUATORTYPE

Defines the evaluator mechanism.

Allowed Values	Evaluator
`"judge"`	Judge evaluator
`"rubric_judge"`	Rubric evaluator. See Rubric judge params

PROMPT *

Prompt provided to the evaluator model to guide scoring behavior.

CONSTRAINTS

Field	Constraint
prompt	Minimum length: `1`

EXAMPLE

{
  "params": {
    "evaluatorType": "judge",
    "prompt": "Evaluate whether the assistant's response is accurate and helpful."
  }
}

JUDGMENTSCORES

Maps each allowed judgment label to a score from 0.0 to 1.0. The example below defines a boolean judge. Example structure:

{
  "judgmentScores": {
    "Yes": 1.0,
    "No": 0.0
  }
}

DATAFIELDS

Defines the dataset fields that are used by the evaluator. Every evaluation example must include each configured field in its metadata. Example structure:

{
  "dataFields": {
    "ground_truth_answer": "Reference answer"
  }
}

RUBRIC JUDGE PARAMS

A rubric evaluator uses the same parameters as a judge evaluator, except judgmentScores. Instead, it defines criteria, each with its own labels and scores, and an aggregation that combines them. See Creating a rubric evaluator.

PROPERTIES

Field	Type	Required	Description
evaluatorType	string	✓	`"rubric_judge"`
prompt	string	✓	What the judge is assessing and how to approach it
criteria	array	✓	The criteria to judge, in the order the judge answers them
aggregation	string		How criterion scores combine into the row’s score
modelIdentifier	object	✓	Hosted model used by the judge
inferenceConfig	object	✓	Inference settings for the judge model
dataFields	object		Dataset field names mapped to display titles
responseFilterMode	string		Portion of the response available to the judge
includeConversationSystem	boolean		Includes the system message in the judge input

CRITERIA *

Each criterion is one question the judge answers.

Field	Type	Required	Description
id	string	✓	Short name of the criterion, such as `grounded`
description	string	✓	The question the judge answers for this criterion
labels	array	✓	The values the judge can return, each with an optional score
weight	number		Weight in the `weighted_mean` aggregation. Default `1.0`
includeExplanation	boolean		Whether the judge explains before it picks a label. Default `true`

Each label is an object with a label and an optional score from 0.0 to 1.0. A label without a score is allowed but does not count toward the overall score, which suits values such as not_applicable.

CONSTRAINTS

Field	Constraint
criteria	`1` to `200` criteria, each with a unique `id`
id	Starts with a letter or underscore, then letters, digits, and underscores. Up to `64` characters. Cannot end in `_explanation` or reuse a result field name such as `score` or `explanation`
labels	`2` to `50` labels per criterion, each unique
weight	`0` or greater

AGGREGATION

The default is weighted_mean. Only criteria with a scored label count.

Value	Overall score
`weighted_mean`	The average of the criterion scores, using each criterion’s weight. Needs at least one criterion with a weight above `0` and a scored label
`min`	The lowest criterion score
`all`	`1.0` if every scored criterion scores `1.0`, otherwise `0.0`
`none`	No overall score. Each criterion is reported on its own

EXAMPLE

{
  "displayName": "Support Agent Rubric",
  "params": {
    "evaluatorType": "rubric_judge",
    "prompt": "You review conversations between a support assistant and a customer. Judge each criterion on its own.",
    "criteria": [
      {
        "id": "grounded",
        "description": "Does every fact in the assistant's replies come from a tool result?",
        "labels": [
          { "label": "yes", "score": 1.0 },
          { "label": "no", "score": 0.0 }
        ]
      },
      {
        "id": "honest_about_actions",
        "description": "Did the assistant avoid claiming an action that no tool performed? Answer not_applicable if the customer asked for no action.",
        "weight": 2.0,
        "labels": [
          { "label": "honest", "score": 1.0 },
          { "label": "claims_action", "score": 0.0 },
          { "label": "not_applicable" }
        ]
      }
    ],
    "aggregation": "weighted_mean",
    "modelIdentifier": {
      "modelType": "OUMI_API",
      "modelName": "zai-org/GLM-5"
    },
    "inferenceConfig": {
      "inferenceTemperature": 0.0
    }
  }
}

MODELIDENTIFIER *

Defines the model used by the evaluator. This object follows the Model Identifier schema. Example:

{
  "modelIdentifier": {
    "modelType": "OUMI_API",
    "modelName": "zai-org/GLM-5"
  }
}

INFERENCECONFIG *

Defines runtime inference behavior for the evaluator model. Example:

{
  "inferenceConfig": {
    "inferenceTemperature": 0.0,
    "inferenceMaxNewTokens": 8192
  }
}

RESPONSEFILTERMODE

Controls which parts of the model output are used for evaluation. The default is EXCLUDE_THINKING.

Value	Description
`THINKING_AND_RESPONSE`	Includes both reasoning and final response
`EXCLUDE_THINKING`	Uses only the final response (default)
`THINKING_ONLY`	Uses only the reasoning output

COMPLETE EXAMPLE

{
  "displayName": "Answer Quality Evaluator",
  "description": "Evaluates responses using a judge model",
  "params": {
    "evaluatorType": "judge",
    "prompt": "Compare the assistant response with the Reference answer. Return Yes if the response is accurate and helpful. Otherwise, return No.",
    "judgmentScores": {
      "Yes": 1.0,
      "No": 0.0
    },
    "dataFields": {
      "ground_truth_answer": "Reference answer"
    },
    "modelIdentifier": {
      "modelType": "OUMI_API",
      "modelName": "zai-org/GLM-5"
    },
    "inferenceConfig": {
      "inferenceTemperature": 0.0,
      "inferenceMaxNewTokens": 8192
    },
    "responseFilterMode": "EXCLUDE_THINKING"
  }
}

Resources

Recipe configs

Agent schemas

Recipe examples

EVALUATOR RECIPE SCHEMA DEFINITIONS

OVERVIEW

SCHEMA STRUCTURE

DISPLAYNAME *

PROPERTIES

CONSTRAINTS

EXAMPLE

JUDGE PARAMS *

PROPERTIES

EVALUATORTYPE

PROMPT *

CONSTRAINTS

EXAMPLE

JUDGMENTSCORES

DATAFIELDS

RUBRIC JUDGE PARAMS

PROPERTIES

CRITERIA *

CONSTRAINTS

AGGREGATION

EXAMPLE

MODELIDENTIFIER *

INFERENCECONFIG *

RESPONSEFILTERMODE

COMPLETE EXAMPLE

​OVERVIEW

​SCHEMA STRUCTURE

​DISPLAYNAME *

​PROPERTIES

​CONSTRAINTS

​EXAMPLE

​JUDGE PARAMS *

​PROPERTIES

​EVALUATORTYPE

​PROMPT *

​CONSTRAINTS

​EXAMPLE

​JUDGMENTSCORES

​DATAFIELDS

​RUBRIC JUDGE PARAMS

​PROPERTIES

​CRITERIA *

​CONSTRAINTS

​AGGREGATION

​EXAMPLE

​MODELIDENTIFIER *

​INFERENCECONFIG *

​RESPONSEFILTERMODE

​COMPLETE EXAMPLE

OVERVIEW

SCHEMA STRUCTURE

DISPLAYNAME *

PROPERTIES

CONSTRAINTS

EXAMPLE

JUDGE PARAMS *

PROPERTIES

EVALUATORTYPE

PROMPT *

CONSTRAINTS

EXAMPLE

JUDGMENTSCORES

DATAFIELDS

RUBRIC JUDGE PARAMS

PROPERTIES

CRITERIA *

CONSTRAINTS

AGGREGATION

EXAMPLE

MODELIDENTIFIER *

INFERENCECONFIG *

RESPONSEFILTERMODE

COMPLETE EXAMPLE