REINFORCEMENT FINE-TUNING

Reinforcement fine-tuning (RFT) trains a model from scores instead of target responses. For each prompt, the model generates a group of responses, an LLM judge scores each one, and Group Relative Policy Optimization (GRPO) increases the likelihood of responses that score better than others in the group.

Only some base models support Reinforcement Fine-Tuning.

WHEN TO USE RFT

Use RFT when all of the following are true:

The task is objectively gradable. Given the same prompt, response, criteria, and optional reference answer, careful reviewers should reach the same verdict.
The base model sometimes succeeds. GRPO needs score differences among responses to the same prompt. Aim for prompts the base model passes roughly 20% to 80% of the time.
The base model already produces the required output type. For example, it can already emit well-formed tool calls, even if it often chooses the wrong tool or arguments.
Several responses can be correct, or you do not have good responses to imitate. Examples include equivalent SQL queries, valid tool calls, math solutions, and compliance with explicit format or policy rules.

Use another training method when:

You have good target responses or need the model to match a particular response style. Use supervised fine-tuning (SFT).
You have preferred and rejected response pairs. Use direct preference optimization.
You want a smaller model to learn a hard reasoning task from a stronger model. Use on-policy distillation.
The base model never succeeds at the task. Use SFT or on-policy distillation to teach the missing behavior instead (multiphase training is coming soon!).

PREREQUISITES

Before you configure a run, you need:

A paid plan.
A base model that supports RFT.
Separate training and validation conversation datasets (assistant response is not required).
A standard LLM judge evaluator that meets the reward judge requirements.
(Optional) Held-out test dataset for evaluation of tuned model and/or for calibrating the reward judge.

PREPARE THE DATA

Each row contains the conversation before the response you want the model to generate. An assistant response is not required.

{
  "messages": [
    {"role": "system", "content": "Answer with the final number only."},
    {"role": "user", "content": "A box holds 12 eggs. How many eggs are in 17 boxes?"}
  ]
}

If the judge needs a reference answer, provide it as metadata.ground_truth or as the final assistant message. metadata.ground_truth takes precedence when both are present.

{
  "messages": [
    {"role": "system", "content": "Answer with the final number only."},
    {"role": "user", "content": "A box holds 12 eggs. How many eggs are in 17 boxes?"}
  ],
  "metadata": {"ground_truth": "204"}
}

During RFT, the reference answer (metadata.ground_truth or the final assistant message) is used only by the judge. The model never trains on it directly, so unlike SFT, it picks up the reference’s wording, length, or style only to the extent the judge rewards them. The reference can be anything the judge needs to decide whether a response is correct, not only a finished answer. For example, it can be an expected value, a required JSON schema, or the tool call the response must make. Write the judge prompt to check whether the response satisfies the reference, not whether it matches it exactly. For example, “two thousand nineteen” satisfies a reference of “2019”, and a response containing every required field satisfies a schema in any field order.

DESIGN THE REWARD JUDGE

The judge defines the training objective. The reward evaluator must:

Be a standard LLM judge. Rubric judges, classification evaluators, and extraction evaluators cannot be used as rewards.
Include scoring instructions and valid labels with scores.
Use a hosted judge model with a configured provider API key.

CONTROL WHAT THE JUDGE SEES

The judge always receives the prompt turns and one sampled response. Configure the following inputs only when the task needs them:

If correctness depends on a reference answer, add ground_truth to the judge’s data fields and refer to it in the judge prompt.
Enable Conversation system message when instructions or source material are in the system message. It is excluded by default.
Add other metadata fields only after confirming that every relevant dataset row provides them.

CHOOSE USEFUL SCORES

Use labels that distinguish typical responses to the same prompt:

Use two labels, such as Correct and Incorrect, when the base model produces a mix of clear passes and failures.
Use three to five graded labels when responses are close in quality. For example: Excellent 1.0, Good 0.66, Partial 0.33, and Poor 0.0.
Write a clear rule for each label so the judge knows when to use it. This is especially important if semi-subjective scales like Good and Partial are used as they are hard to apply consistently without one. Describe what the response does, for example “uses the correct method but makes an arithmetic error,” instead of “partially correct.”
Give a response the lowest score when it breaks a hard requirement. Empty responses already receive the lowest score.

CLOSE REWARD LOOPHOLES

The model learns whatever behavior the judge rewards, including unintended shortcuts. Check for:

Unnecessary length or answers that are too short to support their conclusion.
Hedging or refusing instead of completing the task.
Repeating the prompt or source material.
Correct formatting around incorrect content.
Claims that the response is correct or complete.
Incomplete responses truncated at the maximum completion length.

TEST BEFORE TRAINING

Test and refine the judge before starting an RFT run.

Run the base model on a small, representative set of prompts, and label a sample of its responses yourself.
Score those responses with each candidate judge model, using the same judge prompt. Pick the model that agrees most with your labels, and weigh cost and latency too, since the judge scores every sampled response during training.
Review the results, revise the judge prompt, and re-run until these checks pass:

Score spread: Scores do not cluster at the minimum or maximum. Otherwise, adjust the prompts or scoring levels.
Human agreement: The judge’s verdicts match your labels. Review examples from each label to see where they differ.
Robustness: Edge cases and deliberately flawed responses score low. Include answers that are wrong but long, confident, or well formatted, along with the loopholes listed above. If they score well, the model will learn to exploit that weakness.
Parse rate: The judge always returns one of its configured labels. Repeated parse failures can stop an RFT job.

Keep a separate held-out test dataset for measuring the trained model.

CALIBRATE THE REWARD JUDGE

When practical, calibrate the reward judge against a stronger judge from another provider. Before RFT, have both judges score the same base-model responses using the same criteria. The difference between their mean scores establishes a baseline for how much stricter one judge is than the other. Inspect individual disagreements and revise the reward criteria when the external judge exposes a blind spot. After RFT, repeat the comparison on the same held-out prompts. If the gap between the judges grows because the reward judge’s score improves while the external judge’s score stays flat or falls, the model may be exploiting the reward judge rather than improving at the task. Inspect the disagreements, revise the reward criteria, retest the reward judge, and retrain. Without a pre-RFT baseline, a post-RFT score gap is harder to distinguish from a normal difference in judge strictness.

CONFIGURE TRAINING

On the Models page, click Train New Model.
Set Training Method to Reinforcement Fine-Tuning.
Select a base model, training dataset, and validation dataset.
Select the tested LLM judge under Reward evaluator.
Choose FFT or LoRA, then review the optional settings.
Launch the training job.

The selected evaluator version is pinned for the run. The Builder shows its judge model, instructions, scores, and required dataset fields.

TRAINING SETTINGS

Epochs: Number of passes over the training dataset. The presets use one epoch.
Learning rate: Initial learning rate after warmup. Use the preset unless you have evidence from a previous run that it needs adjustment.

GRPO SETTINGS

Setting	Default	Description
`Generations per prompt`	`8`	Responses sampled for each prompt. Larger groups can improve the reward estimate but increase generation and judge cost.
`Maximum prompt length`	`512` tokens	Prompts above this limit are dropped from training. The prompt includes the system message and every turn before a final assistant reference answer.
`Maximum completion length`	`1024` tokens	Responses above this limit are truncated before judging.
`Sampling temperature`	`1.0`	Controls response diversity. Valid values range from `0` to `2`.
`KL coefficient`	`0.001`	Penalizes movement away from the base model. `0` disables the penalty.

MONITOR TRAINING

The model resource stores RFT curves in metrics.training_metrics and metrics.validation_metrics.

Metric	Meaning
`reward`	Mean judge score for sampled responses.
`reward_group_std`	Mean within-group score spread. Near zero means little training signal.
`frac_reward_zero_std`	Fraction of prompt groups whose responses all received the same score. These groups provide no learning signal.
`response_length`	Mean response length in tokens.
`response_length_clip_ratio`	Fraction of responses truncated at the maximum completion length.
`kl_loss`	Divergence from the base model.
`entropy`	Variation in sampled responses.
`grad_norm`	Update magnitude. Repeated spikes can indicate instability.

Use the training curves to decide what to inspect next:

Pattern	Check
Training and validation reward rise	Review samples to confirm that task quality, not only judge score, improved.
Training reward rises while validation stays flat or falls	Check for overfitting and reward gaming.
Reward stays flat and `frac_reward_zero_std` is high	Use prompts closer to the model’s ability or a judge with finer-grained scores. If the model rarely succeeds, a supervised warmup phase can establish the behavior before RFT. Multiphase training support is coming soon.
Response length moves steadily or clipping is high	Inspect whether length affects the score. Tighten the judge or raise the completion limit when correct answers are genuinely longer.
Entropy and reward-group spread fall quickly	Responses are converging and training may be stalling. Consider finer-grained scores or a higher sampling temperature.
KL loss rises while reward stays flat	The model is drifting without improving. Consider a higher KL coefficient or LoRA for the next run.

Getting started

Oumi workflow

REINFORCEMENT FINE-TUNING

WHEN TO USE RFT

PREREQUISITES

PREPARE THE DATA

DESIGN THE REWARD JUDGE

CONTROL WHAT THE JUDGE SEES

CHOOSE USEFUL SCORES

CLOSE REWARD LOOPHOLES

TEST BEFORE TRAINING

CALIBRATE THE REWARD JUDGE

CONFIGURE TRAINING

TRAINING SETTINGS

GRPO SETTINGS

MONITOR TRAINING

​WHEN TO USE RFT

​PREREQUISITES

​PREPARE THE DATA

​DESIGN THE REWARD JUDGE

​CONTROL WHAT THE JUDGE SEES

​CHOOSE USEFUL SCORES

​CLOSE REWARD LOOPHOLES

​TEST BEFORE TRAINING

​CALIBRATE THE REWARD JUDGE

​CONFIGURE TRAINING

​TRAINING SETTINGS

​GRPO SETTINGS

​MONITOR TRAINING

WHEN TO USE RFT

PREREQUISITES

PREPARE THE DATA

DESIGN THE REWARD JUDGE

CONTROL WHAT THE JUDGE SEES

CHOOSE USEFUL SCORES

CLOSE REWARD LOOPHOLES

TEST BEFORE TRAINING

CALIBRATE THE REWARD JUDGE

CONFIGURE TRAINING

TRAINING SETTINGS

GRPO SETTINGS

MONITOR TRAINING