OVERVIEW

The following describes the request schema used to create judge evaluators in Oumi, including:

Metadata about the evaluator
Evaluation parameters
Model configuration
Scoring settings

Fields marked with (*) are required.

SCHEMA STRUCTURE

{
  "displayName": "",
  "description": "",
  "params": {
    "evaluatorType": "judge",
    "prompt": "",
    "modelIdentifier": {},
    "inferenceConfig": {}
  }
}

DISPLAYNAME *

Human-readable name of the evaluator.

PROPERTIES

Field	Type	Required	Description
displayName	string	✓	Name used to identify the evaluator
description	string		Optional description of the evaluator

CONSTRAINTS

Field	Constraint
displayName	Minimum length: `1`

EXAMPLE

{
  "displayName": "Answer Quality Evaluator",
  "description": "Evaluates model responses using a judge model."
}

JUDGE PARAMS *

Defines the evaluation parameters and scoring behavior.

PROPERTIES

Field	Type	Required	Description
evaluatorType	string	✓	Evaluator type
prompt	string	✓	Prompt used to guide the judge
modelIdentifier	object	✓	Hosted model used by the judge
inferenceConfig	object	✓	Inference settings for the judge model
judgmentScores	object		Judgment labels mapped to scores from `0.0` to `1.0`
dataFields	object		Dataset field names mapped to display titles
responseFilterMode	string		Portion of the response available to the judge
includeConversationSystem	boolean		Includes the system message in the judge input

EVALUATORTYPE

Defines the evaluator mechanism.

Allowed Values
`"judge"`

PROMPT *

Prompt provided to the evaluator model to guide scoring behavior.

CONSTRAINTS

Field	Constraint
prompt	Minimum length: `1`

EXAMPLE

{
  "params": {
    "evaluatorType": "judge",
    "prompt": "Evaluate whether the assistant's response is accurate and helpful."
  }
}

JUDGMENTSCORES

Maps each allowed judgment label to a score from 0.0 to 1.0. The example below defines a boolean judge. Example structure:

{
  "judgmentScores": {
    "Yes": 1.0,
    "No": 0.0
  }
}

DATAFIELDS

Defines the dataset fields that are used by the evaluator. Every evaluation example must include each configured field in its metadata. Example structure:

{
  "dataFields": {
    "ground_truth_answer": "Reference answer"
  }
}

MODELIDENTIFIER *

Defines the model used by the evaluator. This object follows the Model Identifier schema. Example:

{
  "modelIdentifier": {
    "modelType": "OUMI_API",
    "modelName": "zai-org/GLM-5"
  }
}

INFERENCECONFIG *

Defines runtime inference behavior for the evaluator model. Example:

{
  "inferenceConfig": {
    "inferenceTemperature": 0.0,
    "inferenceMaxNewTokens": 8192
  }
}

RESPONSEFILTERMODE

Controls which parts of the model output are used for evaluation. The default is EXCLUDE_THINKING.

Value	Description
`THINKING_AND_RESPONSE`	Includes both reasoning and final response
`EXCLUDE_THINKING`	Uses only the final response (default)
`THINKING_ONLY`	Uses only the reasoning output

COMPLETE EXAMPLE

{
  "displayName": "Answer Quality Evaluator",
  "description": "Evaluates responses using a judge model",
  "params": {
    "evaluatorType": "judge",
    "prompt": "Compare the assistant response with the Reference answer. Return Yes if the response is accurate and helpful. Otherwise, return No.",
    "judgmentScores": {
      "Yes": 1.0,
      "No": 0.0
    },
    "dataFields": {
      "ground_truth_answer": "Reference answer"
    },
    "modelIdentifier": {
      "modelType": "OUMI_API",
      "modelName": "zai-org/GLM-5"
    },
    "inferenceConfig": {
      "inferenceTemperature": 0.0,
      "inferenceMaxNewTokens": 8192
    },
    "responseFilterMode": "EXCLUDE_THINKING"
  }
}

Resources

Recipe configs

Recipe examples

EVALUATOR RECIPE SCHEMA DEFINITIONS

OVERVIEW

SCHEMA STRUCTURE

DISPLAYNAME *

PROPERTIES

CONSTRAINTS

EXAMPLE

JUDGE PARAMS *

PROPERTIES

EVALUATORTYPE

PROMPT *

CONSTRAINTS

EXAMPLE

JUDGMENTSCORES

DATAFIELDS

MODELIDENTIFIER *

INFERENCECONFIG *

RESPONSEFILTERMODE

COMPLETE EXAMPLE

​OVERVIEW

​SCHEMA STRUCTURE

​DISPLAYNAME *

​PROPERTIES

​CONSTRAINTS

​EXAMPLE

​JUDGE PARAMS *

​PROPERTIES

​EVALUATORTYPE

​PROMPT *

​CONSTRAINTS

​EXAMPLE

​JUDGMENTSCORES

​DATAFIELDS

​MODELIDENTIFIER *

​INFERENCECONFIG *

​RESPONSEFILTERMODE

​COMPLETE EXAMPLE

OVERVIEW

SCHEMA STRUCTURE

DISPLAYNAME *

PROPERTIES

CONSTRAINTS

EXAMPLE

JUDGE PARAMS *

PROPERTIES

EVALUATORTYPE

PROMPT *

CONSTRAINTS

EXAMPLE

JUDGMENTSCORES

DATAFIELDS

MODELIDENTIFIER *

INFERENCECONFIG *

RESPONSEFILTERMODE

COMPLETE EXAMPLE