CREATE GROUND-TRUTH EVALUATORS

A ground-truth evaluator compares each response with the correct answer stored in the row’s metadata. It uses no judge model, so it is fast, cheap, and gives the same result every time. Use one when your task has one right answer and your dataset holds it. For help choosing, see Types of evaluators. To create one, go to the Evaluators page, click Create Evaluator, and choose Classification Evaluator or Extraction Evaluator. To save it, click the Save icon, enter a name, and click Save Evaluator.

---

​

CREATE A CLASSIFICATION EVALUATOR

Use a classification evaluator when the model picks one label from a fixed set, such as an intent, a sentiment, or a category. Each row’s metadata must hold the correct label.

1. Under Label field, choose the metadata field that holds the correct label. With Discover from dataset, pick a dataset and click one of the fields Oumi finds. With Enter manually, type the field’s key.
2. Under Valid labels, keep Auto-discover from dataset to use every label that appears as a correct answer. Choose Enter manually to restrict scoring to a fixed set, for example when the dataset doesn’t contain every label.
3. Optionally, under Label Extraction Mode, choose how Oumi finds the predicted label in the response:
   • Auto (recommended): Tries an exact match first, then looks for a label as a whole word in the response. It suits most tasks, including short and numeric labels.
   • Exact match only: The whole response must equal a valid label. Use it for a fine-tuned model that returns only the label.
   • Substring match: Matches a label anywhere in the response. Use it when labels are phrases of several words.
4. Click the Save icon, enter a name, and click Save Evaluator.

​

HOW ROWS ARE SCORED

Oumi finds the predicted label in the model’s last response, using the extraction mode, and compares it with the correct label. The comparison ignores case. A row passes only on an exact match. A row whose response contains no valid label, or whose correct label is missing, can’t be scored, and it is left out of the metrics.

​

CLASSIFICATION RESULTS

A classification evaluator reports more than one score, so you can see which labels the model gets wrong:

• Macro F1: The headline score. It averages the F1 score of every label, so a rare label counts as much as a common one.
• Macro precision, macro recall, and accuracy.
• Per-label metrics: Precision, recall, F1, and the number of rows for each label.
• Top confusions: The pairs of labels the model mixes up most often.
• Confusion matrix: Each row is a correct label and each column a predicted label, so every number off the diagonal is a mistake. It appears when the evaluation has a small number of labels.

In this example, the model confuses intent 11, card_arrival, with intent 12, card_delivery_estimate, in both directions. That points to a specific fix: training data that separates the two intents.

---

​

CREATE AN EXTRACTION EVALUATOR

Use an extraction evaluator when the model extracts information instead of picking a label. Choose the mode that matches your task:

1. Choose the mode.
2. Point the evaluator at your metadata fields, with Choose from dataset or Manual entry:
   • Ground-truth field: The field that holds the correct answer. Every mode needs it.
   • Source text field: The field that holds the text the model extracted from. Named-entity recognition and structured output use it to check that each extracted value appears in the source.
3. Fill in the mode’s settings:
   • Named-entity recognition: Add the Entity types the model may return, such as PERSON or ORG. Oumi can suggest them from the ground-truth field.
   • Structured output: Paste the JSON Schema the output must follow. Under Advanced settings, choose whether 1 and "1" count as the same value, and whether a null value differs from a missing key. Under Check for Groundedness, turn off fields whose values aren’t expected to appear in the source text, such as IDs or booleans.
   • Function calling: Add each function the model may call, with its Function name and the JSON Schema of its parameters.
4. Click the Save icon, enter a name, and click Save Evaluator.

​

HOW ROWS ARE SCORED

• Named-entity recognition and structured output: A row passes only when the extraction matches the correct answer exactly, with nothing missing and nothing extra.
• Function calling: Each predicted call is matched with the correct call of the same function name, and it is correct when every argument matches. A row passes when every call is correct and the model made the right number of calls. Only the tool calls in the final assistant turn are scored.

The results report a headline score for each mode, along with detail tables:

• Named-entity recognition: Strict F1, with type F1, partial F1, and results for each entity type.
• Structured output: Field accuracy, with sample accuracy and results for each field.
• Function calling: Sample accuracy, with function and argument accuracy.

Named-entity recognition and structured output also report a grounding rate: the share of extracted values found in the source text. A value that isn’t in the source is likely made up.

---

​

RUN A GROUND-TRUTH EVALUATOR

Select the evaluator in Evaluate models, like any other evaluator. To score responses you already have, such as an earlier model’s outputs, use Score existing conversations. That run calls no model and no judge, so it finishes in seconds.