Shorthand for Thought

Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens

Zhenyu Zhao, Sander Land, Daniel M. Bikel, Waseem AlShikh

FIG. 01 Collect the model's own traces → Mine cross-word merges → Check that they are predictable → Teach the shorthand
Identifier
arXiv:2604.26355
Published
Apr 29, 2026
Authors
4
Type
Paper
Area
Enterprise-optimized models

In one paragraph

Much of a reasoning trace is predictable scaffolding, such as "wait, let me check". The model learns supertokens for these low-entropy patterns, mined from its own traces. Its reasoning gets 8.1% shorter on average, and accuracy holds on 13 of 15 model–benchmark pairs.

average reasoning-trace length
−8.1%
average reasoning-trace length
model–benchmark cells equivalent or inconclusive (TOST, ±2 pp)
13 / 15
model–benchmark cells equivalent or inconclusive (TOST, ±2 pp)
model families × math benchmarks evaluated
3 × 5
model families × math benchmarks evaluated

FIG. 02: Mechanism

How it works

  1. Collect the model's own traces

    Each model answers OpenThoughts3 prompts, and its reasoning traces become the training corpus. Every model family gets its own supertokens.

  2. Mine cross-word merges

    BPE merges that may cross word boundaries run over the traces, with each trace's pair counts capped at 10. A filter keeps structural shapes such as capitalized phrase openers and comma-led continuations, and the top 250 become supertokens.

  3. Check that they are predictable

    Tokens inside a merge are much easier to predict than the rest. Continuation-token entropy drops 33% for two-token merges and 79% for merges of 16 or more.

  4. Teach the shorthand

    The supertokens join the vocabulary, each embedding starts as the mean of its parts, and supervised fine-tuning on retokenized traces teaches the model to use them.

FIG. 03: Interactive figure

Scaffolding compresses. The math does not.

Pick a trace, toggle supertokens and drag the merge count to watch predictable phrases collapse while problem-specific steps stay as they are.

Loading interactive demo…

FIG. 04: Notes

Summary

Reasoning traces mix two kinds of tokens:

  • Low-entropy structural tokens: the predictable connective moves of reasoning.
  • Higher-entropy organic tokens: the problem-specific content.

The method compresses the structural part:

  1. Mine the traces. Cross-word BPE merges run over the model’s own reasoning traces and produce supertokens for frequent structural patterns.
  2. Teach the shorthand. Supervised fine-tuning teaches the model to use them.
  3. Read the structure. Learned supertokens often line up with backtracking, verification and strategy shifts, so they also make traces easier to analyze. Correct traces use them more.

Results

Across three model families and five math benchmarks, traces are 8.1% shorter on average. Under a TOST analysis with a ±2 pp margin, accuracy is equivalent or inconclusive on 13 of 15 cells. Most inconclusive cells are AIME, which has only 30 problems. Two cells show non-equivalent degradation: DeepSeek-R1-Distill-Llama-70B on MATH-500 and on OlympiadBench. The full pipeline is open source.

Reference

Cite this work
@misc{zhao2026shorthand,
  title={Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens},
  author={Zhenyu Zhao and Sander Land and Daniel M. Bikel and Waseem AlShikh},
  year={2026},
  eprint={2604.26355},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}