Shorthand for Thought
Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens
- Identifier
- arXiv:2604.26355
- Published
- Apr 29, 2026
- Authors
- 4
- Type
- Paper
- Area
- Enterprise-optimized models
In one paragraph
Much of a reasoning trace is predictable scaffolding, such as "wait, let me check". The model learns supertokens for these low-entropy patterns, mined from its own traces. Its reasoning gets 8.1% shorter on average, and accuracy holds on 13 of 15 model–benchmark pairs.
- average reasoning-trace length
- −8.1%
- average reasoning-trace length
- model–benchmark cells equivalent or inconclusive (TOST, ±2 pp)
- 13 / 15
- model–benchmark cells equivalent or inconclusive (TOST, ±2 pp)
- model families × math benchmarks evaluated
- 3 × 5
- model families × math benchmarks evaluated
FIG. 02: Mechanism
How it works
Collect the model's own traces
Each model answers OpenThoughts3 prompts, and its reasoning traces become the training corpus. Every model family gets its own supertokens.
Mine cross-word merges
BPE merges that may cross word boundaries run over the traces, with each trace's pair counts capped at 10. A filter keeps structural shapes such as capitalized phrase openers and comma-led continuations, and the top 250 become supertokens.
Check that they are predictable
Tokens inside a merge are much easier to predict than the rest. Continuation-token entropy drops 33% for two-token merges and 79% for merges of 16 or more.
Teach the shorthand
The supertokens join the vocabulary, each embedding starts as the mean of its parts, and supervised fine-tuning on retokenized traces teaches the model to use them.
FIG. 03: Interactive figure
Scaffolding compresses. The math does not.
Pick a trace, toggle supertokens and drag the merge count to watch predictable phrases collapse while problem-specific steps stay as they are.
Loading interactive demo…
FIG. 04: Notes
Summary
Reasoning traces mix two kinds of tokens:
- Low-entropy structural tokens: the predictable connective moves of reasoning.
- Higher-entropy organic tokens: the problem-specific content.
The method compresses the structural part:
- Mine the traces. Cross-word BPE merges run over the model’s own reasoning traces and produce supertokens for frequent structural patterns.
- Teach the shorthand. Supervised fine-tuning teaches the model to use them.
- Read the structure. Learned supertokens often line up with backtracking, verification and strategy shifts, so they also make traces easier to analyze. Correct traces use them more.
Results
Across three model families and five math benchmarks, traces are 8.1% shorter on average. Under a TOST analysis with a ±2 pp margin, accuracy is equivalent or inconclusive on 13 of 15 cells. Most inconclusive cells are AIME, which has only 30 problems. Two cells show non-equivalent degradation: DeepSeek-R1-Distill-Llama-70B on MATH-500 and on OlympiadBench. The full pipeline is open source.
Reference
@misc{zhao2026shorthand,
title={Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens},
author={Zhenyu Zhao and Sander Land and Daniel M. Bikel and Waseem AlShikh},
year={2026},
eprint={2604.26355},
archivePrefix={arXiv},
primaryClass={cs.CL}
}