Multi-layer Indexer

Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention

Serdarcan Dilbaz, Arjun Krishna, Kiran Kamble, Daniel M. Bikel

FIG. 01 Each layer picks its tokens → Attention waits for the scan → Borrow the previous selection → Run the indexer alongside attention → Stack it on IndexCache
Identifier
arXiv:2611.03768
Published
Nov 2, 2026
Authors
4
Type
Paper
Area
Enterprise-optimized models

In one paragraph

In sparse attention, each layer's indexer scans the whole KV cache to choose which tokens to attend to, and at long context that scan dominates decoding. Adjacent layers' selections overlap enough that each layer can reuse the previous selection and run its indexer alongside attention. Decoding gets faster with no retraining.

decode speed over IndexCache on DeepSeek-V3.2 (retention 1/2)
+14.5%
decode speed over IndexCache on DeepSeek-V3.2 (retention 1/2)
decode speed over IndexCache on GLM-5 (retention 1/2)
+9.8%
decode speed over IndexCache on GLM-5 (retention 1/2)
unmodified model's decode throughput at 128K (retention 1/4)
1.32×
unmodified model's decode throughput at 128K (retention 1/4)
retraining or new weights required
0
retraining or new weights required

FIG. 02: Mechanism

How it works

  1. Each layer picks its tokens

    In DeepSeek Sparse Attention, every layer's lightning indexer scores the whole KV cache and keeps the top 2,048 tokens for attention. The scan grows with context. Attention over 2,048 tokens does not.

  2. Attention waits for the scan

    In the standard schedule a layer cannot attend until its own indexer finishes, so the scan sits on the critical path of every layer.

  3. Borrow the previous selection

    Adjacent layers pick strongly overlapping sets (Jaccard near 0.42 on DeepSeek-V3.2, 0.26 on GLM-5). So each layer attends the selection of the previous indexer layer, and its own indexer's output goes to the next one.

  4. Run the indexer alongside attention

    With that dependency gone, each indexer runs on a second CUDA stream while its layer computes attention and the feed-forward block. No retraining or new weights are needed.

  5. Stack it on IndexCache

    IndexCache keeps the indexer on only 1 in r layers and copies selections to the rest. MLI hides the scan on the layers that keep one, so the two speedups add up.

FIG. 03: Interactive figure

Take the indexer off the critical path.

Switch the schedule from unmodified to IndexCache to IndexCache + MLI, change the retention, and watch the indexer scan move onto a second stream.

Loading interactive demo…

FIG. 04: Notes

Summary

DeepSeek Sparse Attention gives every layer a lightweight “lightning indexer”. The indexer scores the whole KV cache and keeps the top 2,048 tokens for attention. At long context, that scan dominates decode time.

Adjacent layers’ selections overlap substantially (Jaccard similarity of about 0.42 on DeepSeek-V3.2 and 0.26 on GLM-5 in the paper’s measurements), enough that each layer can attend to the previous indexer layer’s selection instead of waiting for its own while staying within a few points of accuracy, which lets every indexer run concurrently with its layer’s attention. The method needs no retraining or new weights, and it composes with IndexCache, which removes indexers from some layers entirely, across retention rates from 1/2 to 1/32.

Results

  • Up to 14.5% faster decoding than IndexCache on DeepSeek-V3.2 and 9.8% on GLM-5 at retention 1/2. At 1/4 the gains are 7.9% and 5.1%.
  • At 128K context the method reaches 1.32× the unmodified model’s decode throughput at retention 1/4, above IndexCache’s 1.28× at 1/8.
  • Accuracy on LongBench, RULER and LongBench v2 stays within a few points of IndexCache’s.

Reference

Cite this work
@misc{dilbaz2026multilayer,
  title={Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention},
  author={Serdarcan Dilbaz and Arjun Krishna and Kiran Kamble and Daniel M. Bikel},
  year={2026},
  eprint={2611.03768},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}