Self-Indexing Attention Unifies Token Retrieval Across Prefill and Decode

A new arXiv paper proposes a self-indexing attention mechanism for sparse long-context LLM inference. The authors note that current sparse methods typically rely on separate retrieval strategies for the prefill and decode stages, which prevents a single retrieval representation from being shared across the whole inference run. Their approach aims to make one retrieval representation reusable throughout, while remaining compatible with compression.