Reflect, Retry, Reward

Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning

Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Kiran Kamble, Dmytro Mozolevskyi, Muayad Ali, Waseem AlShikh

FIG. 01 Attempt → Reflect → Retry → Reward only the reflection → Reflect better, fail less
Identifier
arXiv:2505.24726
Published
May 30, 2025
Authors
8
Type
Paper
Area
Practical evaluations

In one paragraph

When a model fails, it writes a reflection on what went wrong and tries again. If the retry succeeds, only the reflection tokens get rewarded. With nothing more than a binary pass/fail signal, small models learn to self-correct well enough to beat models ten times their size.

math equation writing (Countdown)
+34.7%
math equation writing (Countdown)
function calling (APIGen)
+18.1%
function calling (APIGen)
fine-tuned 1.5B–7B models outperform same-family models 10× larger
10×
fine-tuned 1.5B–7B models outperform same-family models 10× larger
only a pass/fail signal is needed, with no labeled data and no teacher
binary
only a pass/fail signal is needed, with no labeled data and no teacher

FIG. 02: Mechanism

How it works

  1. Attempt

    The model tries the task, for example writing a function call or building an equation that hits a target number. A simple verifier checks the answer.

  2. Reflect

    If the attempt fails, the model is prompted to write a short self-reflection on what went wrong.

  3. Retry

    The model tries again with its own reflection in context.

  4. Reward only the reflection

    If the retry succeeds, a GRPO reward is applied to the reflection tokens alone and every other token is masked. The model is learning to reflect, not memorizing answers.

  5. Reflect better, fail less

    After training, the model self-corrects better, and its first-attempt accuracy improves too.

FIG. 03: Interactive figure

Pick the reflection. See which tokens learn.

Step through a failed attempt, choose how the model reflects on it, and turn on the gradient view to see exactly which tokens the reward trains.

Loading interactive demo…

FIG. 04: Notes

The problem

Improving a model on a specific task usually takes labeled data or a larger teacher model. For many enterprise tasks, neither is available. What you can usually tell is whether an answer is right: whether the API call executed, or whether the equation equals the target.

The approach

The method uses that binary signal to teach a skill that carries over to other tasks: self-reflection. After a failed attempt, the model writes a reflection and retries. When the retry succeeds, we use Group Relative Policy Optimization (GRPO) to reward only the tokens in the reflection. The model never gets rewarded for the answer itself. It gets rewarded for the kind of reflection that leads to a correct answer.

We trained on two verifiable tasks: function calling (APIGen) and math equation writing (Countdown).

What we found

  • Up to 18.1% better on function calling and up to 34.7% better on math equation writing.
  • Fine-tuned models from 1.5B to 7B parameters outperform models in the same family that are 10× larger.
  • Catastrophic forgetting was minimal on GSM8K, MMLU-Pro, HellaSwag and MATH.

Reference

Cite this work
@misc{bensal2025reflect,
  title={Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning},
  author={Shelly Bensal and Umar Jamil and Christopher Bryant and Melisa Russak and Kiran Kamble and Dmytro Mozolevskyi and Muayad Ali and Waseem AlShikh},
  year={2025},
  eprint={2505.24726},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}