Reflect, Retry, Reward
Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
- Identifier
- arXiv:2505.24726
- Published
- May 30, 2025
- Authors
- 8
- Type
- Paper
- Area
- Practical evaluations
In one paragraph
When a model fails, it writes a reflection on what went wrong and tries again. If the retry succeeds, only the reflection tokens get rewarded. With nothing more than a binary pass/fail signal, small models learn to self-correct well enough to beat models ten times their size.
- math equation writing (Countdown)
- +34.7%
- math equation writing (Countdown)
- function calling (APIGen)
- +18.1%
- function calling (APIGen)
- fine-tuned 1.5B–7B models outperform same-family models 10× larger
- 10×
- fine-tuned 1.5B–7B models outperform same-family models 10× larger
- only a pass/fail signal is needed, with no labeled data and no teacher
- binary
- only a pass/fail signal is needed, with no labeled data and no teacher
FIG. 02: Mechanism
How it works
Attempt
The model tries the task, for example writing a function call or building an equation that hits a target number. A simple verifier checks the answer.
Reflect
If the attempt fails, the model is prompted to write a short self-reflection on what went wrong.
Retry
The model tries again with its own reflection in context.
Reward only the reflection
If the retry succeeds, a GRPO reward is applied to the reflection tokens alone and every other token is masked. The model is learning to reflect, not memorizing answers.
Reflect better, fail less
After training, the model self-corrects better, and its first-attempt accuracy improves too.
FIG. 03: Interactive figure
Pick the reflection. See which tokens learn.
Step through a failed attempt, choose how the model reflects on it, and turn on the gradient view to see exactly which tokens the reward trains.
Loading interactive demo…
FIG. 04: Notes
The problem
Improving a model on a specific task usually takes labeled data or a larger teacher model. For many enterprise tasks, neither is available. What you can usually tell is whether an answer is right: whether the API call executed, or whether the equation equals the target.
The approach
The method uses that binary signal to teach a skill that carries over to other tasks: self-reflection. After a failed attempt, the model writes a reflection and retries. When the retry succeeds, we use Group Relative Policy Optimization (GRPO) to reward only the tokens in the reflection. The model never gets rewarded for the answer itself. It gets rewarded for the kind of reflection that leads to a correct answer.
We trained on two verifiable tasks: function calling (APIGen) and math equation writing (Countdown).
What we found
- Up to 18.1% better on function calling and up to 34.7% better on math equation writing.
- Fine-tuned models from 1.5B to 7B parameters outperform models in the same family that are 10× larger.
- Catastrophic forgetting was minimal on GSM8K, MMLU-Pro, HellaSwag and MATH.
Reference
@misc{bensal2025reflect,
title={Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning},
author={Shelly Bensal and Umar Jamil and Christopher Bryant and Melisa Russak and Kiran Kamble and Dmytro Mozolevskyi and Muayad Ali and Waseem AlShikh},
year={2025},
eprint={2505.24726},
archivePrefix={arXiv},
primaryClass={cs.CL}
}