Multi-layer Indexer
Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention
- Identifier
- arXiv:2611.03768
- Published
- Nov 2, 2026
- Authors
- 4
- Type
- Paper
- Area
- Enterprise-optimized models
In one paragraph
In sparse attention, each layer's indexer scans the whole KV cache to choose which tokens to attend to, and at long context that scan dominates decoding. Adjacent layers' selections overlap enough that each layer can reuse the previous selection and run its indexer alongside attention. Decoding gets faster with no retraining.
- decode speed over IndexCache on DeepSeek-V3.2 (retention 1/2)
- +14.5%
- decode speed over IndexCache on DeepSeek-V3.2 (retention 1/2)
- decode speed over IndexCache on GLM-5 (retention 1/2)
- +9.8%
- decode speed over IndexCache on GLM-5 (retention 1/2)
- unmodified model's decode throughput at 128K (retention 1/4)
- 1.32×
- unmodified model's decode throughput at 128K (retention 1/4)
- retraining or new weights required
- 0
- retraining or new weights required
FIG. 02: Mechanism
How it works
Each layer picks its tokens
In DeepSeek Sparse Attention, every layer's lightning indexer scores the whole KV cache and keeps the top 2,048 tokens for attention. The scan grows with context. Attention over 2,048 tokens does not.
Attention waits for the scan
In the standard schedule a layer cannot attend until its own indexer finishes, so the scan sits on the critical path of every layer.
Borrow the previous selection
Adjacent layers pick strongly overlapping sets (Jaccard near 0.42 on DeepSeek-V3.2, 0.26 on GLM-5). So each layer attends the selection of the previous indexer layer, and its own indexer's output goes to the next one.
Run the indexer alongside attention
With that dependency gone, each indexer runs on a second CUDA stream while its layer computes attention and the feed-forward block. No retraining or new weights are needed.
Stack it on IndexCache
IndexCache keeps the indexer on only 1 in r layers and copies selections to the rest. MLI hides the scan on the layers that keep one, so the two speedups add up.
FIG. 03: Interactive figure
Take the indexer off the critical path.
Switch the schedule from unmodified to IndexCache to IndexCache + MLI, change the retention, and watch the indexer scan move onto a second stream.
Loading interactive demo…
FIG. 04: Notes
Summary
DeepSeek Sparse Attention gives every layer a lightweight “lightning indexer”. The indexer scores the whole KV cache and keeps the top 2,048 tokens for attention. At long context, that scan dominates decode time.
Adjacent layers’ selections overlap substantially (Jaccard similarity of about 0.42 on DeepSeek-V3.2 and 0.26 on GLM-5 in the paper’s measurements), enough that each layer can attend to the previous indexer layer’s selection instead of waiting for its own while staying within a few points of accuracy, which lets every indexer run concurrently with its layer’s attention. The method needs no retraining or new weights, and it composes with IndexCache, which removes indexers from some layers entirely, across retention rates from 1/2 to 1/32.
Results
- Up to 14.5% faster decoding than IndexCache on DeepSeek-V3.2 and 9.8% on GLM-5 at retention 1/2. At 1/4 the gains are 7.9% and 5.1%.
- At 128K context the method reaches 1.32× the unmodified model’s decode throughput at retention 1/4, above IndexCache’s 1.28× at 1/8.
- Accuracy on LongBench, RULER and LongBench v2 stays within a few points of IndexCache’s.
Reference
@misc{dilbaz2026multilayer,
title={Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention},
author={Serdarcan Dilbaz and Arjun Krishna and Kiran Kamble and Daniel M. Bikel},
year={2026},
eprint={2611.03768},
archivePrefix={arXiv},
primaryClass={cs.LG}
}