> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oumi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# CREATE GROUND-TRUTH EVALUATORS

> Create classification and extraction evaluators that compare responses with the correct answers in your dataset

A ground-truth evaluator compares each response with the correct answer stored in the row's metadata. It uses no judge model, so it is fast, cheap, and gives the same result every time. Use one when your task has one right answer and your dataset holds it. For help choosing, see [Types of evaluators](https://docs.oumi.ai/guides/evaluators.md#types-of-evaluators).

To create one, go to the **Evaluators** page, click `Create Evaluator`, and choose `Classification Evaluator` or `Extraction Evaluator`. To save it, click the `Save` icon, enter a name, and click `Save Evaluator`.

***

## CREATE A CLASSIFICATION EVALUATOR

Use a classification evaluator when the model picks one label from a fixed set, such as an intent, a sentiment, or a category. Each row's metadata must hold the correct label.

1. Under **Label field**, choose the metadata field that holds the correct label. With `Discover from dataset`, pick a dataset and click one of the fields Oumi finds. With `Enter manually`, type the field's key.
2. Under **Valid labels**, keep `Auto-discover from dataset` to use every label that appears as a correct answer. Choose `Enter manually` to restrict scoring to a fixed set, for example when the dataset doesn't contain every label.
3. Optionally, under **Label Extraction Mode**, choose how Oumi finds the predicted label in the response:
   * `Auto (recommended)`: Tries an exact match first, then looks for a label as a whole word in the response. It suits most tasks, including short and numeric labels.
   * `Exact match only`: The whole response must equal a valid label. Use it for a fine-tuned model that returns only the label.
   * `Substring match`: Matches a label anywhere in the response. Use it when labels are phrases of several words.
4. Click the `Save` icon, enter a name, and click `Save Evaluator`.

![Classification evaluator form with the label field discovered from banking77\_test\_basic.jsonl and valid labels set to auto-discover](https://mintcdn.com/oumi/5eryyJDD07V8QVa7/images/evaluators-classification-form.jpg?fit=max\&auto=format\&n=5eryyJDD07V8QVa7\&q=85\&s=63f98d08c4a41e356597fffb274e3763)

### HOW ROWS ARE SCORED

Oumi finds the predicted label in the model's last response, using the extraction mode, and compares it with the correct label. The comparison ignores case. A row passes only on an exact match. A row whose response contains no valid label, or whose correct label is missing, can't be scored, and it is left out of the metrics.

### CLASSIFICATION RESULTS

A classification evaluator reports more than one score, so you can see which labels the model gets wrong:

* **Macro F1:** The headline score. It averages the F1 score of every label, so a rare label counts as much as a common one.
* **Macro precision, macro recall, and accuracy.**
* **Per-label metrics:** Precision, recall, F1, and the number of rows for each label.
* **Top confusions:** The pairs of labels the model mixes up most often.
* **Confusion matrix:** Each row is a correct label and each column a predicted label, so every number off the diagonal is a mistake. It appears when the evaluation has a small number of labels.

![Classification results with macro F1, macro precision, macro recall, and accuracy, and the start of the per-label metrics table](https://mintcdn.com/oumi/5eryyJDD07V8QVa7/images/evaluators-classification-results.jpg?fit=max\&auto=format\&n=5eryyJDD07V8QVa7\&q=85\&s=baf0bfe27727c778d39410b02dc430e9)

![Confusion matrix in which label 11 is predicted as 12 four times and label 12 as 11 five times](https://mintcdn.com/oumi/5eryyJDD07V8QVa7/images/evaluators-confusion-matrix.jpg?fit=max\&auto=format\&n=5eryyJDD07V8QVa7\&q=85\&s=1b75c9143ccd301a7d0d238a5d7ab8ef)

In this example, the model confuses intent `11`, `card_arrival`, with intent `12`, `card_delivery_estimate`, in both directions. That points to a specific fix: training data that separates the two intents.

***

## CREATE AN EXTRACTION EVALUATOR

Use an extraction evaluator when the model extracts information instead of picking a label. Choose the mode that matches your task:

| Mode | The model returns | The correct answer in the metadata |
| - | - | - |
| `Named-entity recognition` | A list of labeled spans, such as names of people or organizations | A list of entities, each with a label and a value |
| `Structured output` | A JSON document in a fixed shape | The correct JSON document |
| `Function calling` | One or more tool calls | The correct tool calls |

![Extraction evaluator form with the three modes and the source, ground-truth, and entity type fields for named-entity recognition](https://mintcdn.com/oumi/5eryyJDD07V8QVa7/images/evaluators-extraction-form.jpg?fit=max\&auto=format\&n=5eryyJDD07V8QVa7\&q=85\&s=17f826287ede1520e1409de84cf99366)

1. Choose the mode.
2. Point the evaluator at your metadata fields, with `Choose from dataset` or `Manual entry`:
   * **Ground-truth field:** The field that holds the correct answer. Every mode needs it.
   * **Source text field:** The field that holds the text the model extracted from. Named-entity recognition and structured output use it to check that each extracted value appears in the source.
3. Fill in the mode's settings:
   * **Named-entity recognition:** Add the `Entity types` the model may return, such as `PERSON` or `ORG`. Oumi can suggest them from the ground-truth field.
   * **Structured output:** Paste the `JSON Schema` the output must follow. Under **Advanced settings**, choose whether `1` and `"1"` count as the same value, and whether a `null` value differs from a missing key. Under **Check for Groundedness**, turn off fields whose values aren't expected to appear in the source text, such as IDs or booleans.
   * **Function calling:** Add each function the model may call, with its `Function name` and the JSON Schema of its parameters.
4. Click the `Save` icon, enter a name, and click `Save Evaluator`.

### HOW ROWS ARE SCORED

* **Named-entity recognition and structured output:** A row passes only when the extraction matches the correct answer exactly, with nothing missing and nothing extra.
* **Function calling:** Each predicted call is matched with the correct call of the same function name, and it is correct when every argument matches. A row passes when every call is correct and the model made the right number of calls. Only the tool calls in the final assistant turn are scored.

The results report a headline score for each mode, along with detail tables:

* **Named-entity recognition:** Strict F1, with type F1, partial F1, and results for each entity type.
* **Structured output:** Field accuracy, with sample accuracy and results for each field.
* **Function calling:** Sample accuracy, with function and argument accuracy.

Named-entity recognition and structured output also report a **grounding rate**: the share of extracted values found in the source text. A value that isn't in the source is likely made up.

***

## RUN A GROUND-TRUTH EVALUATOR

Select the evaluator in [Evaluate models](https://docs.oumi.ai/guides/evaluations/running.md), like any other evaluator. To score responses you already have, such as an earlier model's outputs, use [Score existing conversations](https://docs.oumi.ai/guides/evaluations/running.md#score-existing-conversations). That run calls no model and no judge, so it finishes in seconds.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.