1. Bibliographic Information

1.1. Title

The central topic of the paper is "Efficient Streaming Language Models with Attention Sinks." It focuses on enhancing the capability of Large Language Models (LLMs) to handle long, continuous streams of text efficiently and stably, particularly in applications like multi-round dialogue.

1.2. Authors

The authors are:

• Guangxuan Xiao (Massachusetts Institute of Technology)

• Yuandong Tian (Meta AI)

• Beidi Chen (Carnegie Mellon University)

• Song Han (Massachusetts Institute of Technology, NVIDIA)

• Mike Lewis (Meta AI)

  Their affiliations indicate a collaboration between top academic institutions (MIT, CMU) and leading industry research labs (Meta AI, NVIDIA), suggesting a strong foundation in both theoretical research and practical application of LLMs.

1.3. Journal/Conference

This paper was published on arXiv, a preprint server for scientific papers. While arXiv is not a peer-reviewed journal or conference in itself, it is a widely recognized and influential platform where researchers disseminate their work rapidly before, or concurrently with, formal peer review processes for conferences or journals. Its presence on arXiv indicates its immediate accessibility to the research community.

1.4. Publication Year

The paper was published on 2023-09-29.

1.5. Abstract

Deploying Large Language Models (LLMs) in streaming applications, such as multi-round dialogues, faces two main challenges: extensive memory consumption due to caching Key and Value states (KV) during decoding, and the inability of LLMs to generalize to texts longer than their training sequence length. While window attention (caching only recent KVs) is a natural approach, it fails when text length exceeds cache size. The authors observe an "attention sink" phenomenon, where keeping the KV states of initial tokens significantly recovers the performance of window attention. This phenomenon is attributed to strong attention scores towards initial tokens acting as a "sink," even if semantically unimportant. Based on this, they introduce StreamingLLM, an efficient framework enabling pre-trained LLMs to generalize to infinite sequence lengths without fine-tuning, by preserving initial attention sink tokens alongside a sliding window of recent KVs. StreamingLLM allows models like Llama-2, MPT, Falcon, and Pythia to stably and efficiently process up to 4 million tokens or more. Furthermore, adding a placeholder token as a dedicated attention sink during pre-training can enhance streaming deployment. In streaming settings, StreamingLLM achieves up to a 22.2x speedup over the sliding window recomputation baseline.

1.6. Original Source Link

The original source link is: https://arxiv.org/abs/2309.17453. The PDF link is: https://arxiv.org/pdf/2309.17453v4.pdf. This paper is a preprint available on arXiv.

2. Executive Summary

2.1. Background & Motivation

The widespread adoption of Large Language Models (LLMs) in applications like dialogue systems, document summarization, and code completion necessitates their efficient and accurate performance on long sequence generation. However, deploying LLMs in streaming applications (where interactions are continuous and potentially infinite, like a day-long chatbot conversation) presents two major challenges:

1. Memory Consumption: During the decoding stage, Transformer-based LLMs typically cache the Key (K) and Value (V) states (KV cache) of all previously processed tokens. This KV cache grows linearly with the sequence length, leading to excessive memory usage and increasing decoding latency, making continuous operation impractical.

2. Limited Length Extrapolation: Popular LLMs are trained on a finite sequence length (e.g., 4K tokens for Llama-2). Their performance degrades sharply when the input text length exceeds the sequence length they were trained on, failing to generalize to longer contexts.

   An intuitive solution, window attention, involves caching only the KV states of the most recent tokens within a fixed-size sliding window. While this approach effectively manages memory and decoding speed, the paper demonstrates that it fails catastrophically when the text length surpasses the cache size. Specifically, performance collapses once the initial tokens (the very first tokens of the input stream) are evicted from the cache. Another existing approach, sliding window with re-computation, maintains performance but is prohibitively slow due to quadratic attention computations within its window.

The paper's innovative idea stems from an observation: LLMs exhibit an "attention sink" phenomenon. Even if the initial tokens of a sequence are not semantically important, a surprisingly large amount of attention score is consistently allocated to them. This suggests that these initial tokens play a crucial, albeit unexpected, role in maintaining the model's stability. The core motivation is to understand why these attention sinks emerge and how to leverage this phenomenon to build an efficient and stable streaming LLM solution without fine-tuning.

2.2. Main Contributions / Findings

The paper makes several significant contributions to enabling efficient and stable LLM deployment in streaming applications:

1. Identification and Explanation of "Attention Sinks": The authors discover and empirically demonstrate that autoregressive LLMs disproportionately allocate attention scores to initial tokens (referred to as "attention sinks"), regardless of their semantic importance. This phenomenon is attributed to the properties of the SoftMax function in attention computation, which necessitates distributing attention values even when no strong semantic match exists. Initial tokens are preferred as sinks because they are visible to all subsequent tokens during autoregressive training.
2. Introduction of StreamingLLM Framework: Based on the "attention sink" insight, the paper proposes StreamingLLM, a simple and efficient framework. It enables LLMs trained with a finite attention window to generalize to infinite sequence lengths without any fine-tuning. The core idea is to preserve the KV states of a small number of initial attention sink tokens (typically 4) alongside a rolling KV cache of recent tokens. This effectively "anchors" the attention computation and stabilizes model performance.
3. Robust Generalization to Long Sequences: StreamingLLM is shown to enable various LLM families (Llama-2, MPT, Falcon, Pythia) and scales to perform stable and efficient language modeling with millions of tokens (up to 4 million and more). This significantly extends the practical applicability of these models in long-running streaming scenarios.
4. Pre-training with Dedicated Sink Tokens: The research further discovers that adding a dedicated placeholder token as a learnable attention sink during pre-training can further improve streaming deployment. Models pre-trained with such a sink token can achieve stable streaming performance by retaining only this single sink token, demonstrating a more explicit and efficient mechanism for managing attention sinks.
5. Significant Efficiency Gains: In streaming settings, StreamingLLM achieves up to a 22.2x speedup in per-token decoding latency compared to the sliding window recomputation baseline, while maintaining a similar memory footprint. This makes StreamingLLM a practical and highly efficient solution for real-world streaming applications.
6. Decoupling Pre-training Window from Generation Length: StreamingLLM fundamentally decouples the LLM's pre-training window size from its actual text generation length, addressing a long-standing limitation in LLM deployment.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully grasp the innovations presented in this paper, a reader should be familiar with the following foundational concepts:

• Large Language Models (LLMs): These are advanced artificial intelligence models, typically based on the Transformer architecture, trained on vast amounts of text data to understand, generate, and process human language. They excel at tasks like text generation, translation, summarization, and question answering. Their "largeness" refers to the number of parameters they contain (billions or even trillions) and the scale of their training data.

• Transformer Architecture: The Transformer is a neural network architecture introduced by Vaswani et al. (2017) that revolutionized sequence modeling. It primarily relies on self-attention mechanisms to weigh the importance of different parts of the input sequence when processing each element. It eschews recurrent or convolutional layers, making it highly parallelizable and efficient for long sequences. LLMs are overwhelmingly built upon this architecture.

• Attention Mechanism: At the heart of the Transformer is the attention mechanism. For each token in a sequence, it computes a weighted sum of all other tokens (or itself) in the sequence to determine its new representation. The weights (attention scores) are dynamically calculated based on the relevance between tokens. This allows the model to "attend" to relevant parts of the input, regardless of their position. The attention calculation involves three learned matrices:

  • Query (Q): Represents what a token is looking for.
  • Key (K): Represents what a token offers.
  • Value (V): The actual information a token carries. The attention scores are typically computed by taking the dot product of Query and Key vectors, followed by a SoftMax function, and then multiplying by the Value vectors.

• KV Cache: In autoregressive language models (models that predict tokens one by one based on previous predictions), when generating a sequence, the Key (K) and Value (V) states for previously computed tokens are often cached (stored). This KV cache prevents redundant re-computation of $K$ and $V$ for already processed tokens during subsequent token generation steps. While it speeds up inference, the size of this cache grows linearly with the sequence length, leading to significant memory consumption for long sequences.

• Perplexity (PPL): Perplexity is a common intrinsic evaluation metric for language models. It quantifies how well a probability distribution (the language model) predicts a sample. Conceptually, it's the exponentiated average negative log-likelihood of a sequence, normalized by the number of tokens. A lower perplexity indicates that the model is better at predicting the next word in a sequence and, therefore, has a better understanding of the language.

• SoftMax Function: The SoftMax function is a mathematical function that converts a vector of arbitrary real numbers into a probability distribution. In the attention mechanism, it's applied to the raw attention scores (logits) to ensure that they sum up to 1 and can be interpreted as probabilities. The formula for SoftMax is: $ \mathrm{SoftMax}(x)i = \frac{e^{x_i}}{\sum{j=1}^{N} e^{x_j}} $ Where:

  • $x$ is the input vector of raw attention scores (logits).
  • $x_i$ is the $i$-th element of the input vector.
  • $N$ is the total number of elements in the vector.
  • $e^{x_i}$ is the exponential of $x_i$. The paper highlights that this property (attention scores summing to one) is crucial to the emergence of "attention sinks."

• Positional Encoding (RoPE, ALiBi): Since Transformers do not inherently process sequential information (they process all tokens in parallel), positional encodings are added to the token embeddings to provide information about the relative or absolute position of tokens in a sequence.

  • Rotary Position Embeddings (RoPE): A technique that applies a rotation matrix to Query and Key vectors based on their absolute positions. This implicitly encodes relative positional information, making it suitable for length extrapolation.
  • Attention with Linear Biases (ALiBi): This method directly biases the query-key attention scores based on their distance. Instead of adding positional embeddings to the tokens, ALiBi adds a penalty to attention scores for tokens that are farther apart, encouraging attention to more recent tokens.

3.2. Previous Works

The paper contextualizes its contribution by discussing prior research in three main areas concerning LLMs and long texts:

1. Length Extrapolation: This area aims to enable LLMs trained on shorter texts to handle much longer ones during inference.

   • Rotary Position Embeddings (RoPE) (Su et al., 2021): A popular method that transforms Queries and Keys in each attention layer to integrate relative positional information. While promising, subsequent research (Press et al., 2022; Chen et al., 2023) showed its performance degrades when text vastly exceeds the training window.
   • ALiBi (Press et al., 2022): Biases query-key attention scores based on distance. While offering improved extrapolation over RoPE, the paper notes it still breaks down with extremely long texts (much greater than training length).
   • Overall Limitation: Current methods have not achieved "infinite length extrapolation," making them unsuitable for true streaming applications.

2. Context Window Extension: This focuses on expanding the LLM's effective context window to process more tokens in a single forward pass.

   • Efficiency Optimizations: Due to the quadratic complexity of attention computation, efforts have focused on making long context training feasible.
     • FlashAttention (Dao et al., 2022; Dao, 2023): System-focused optimization that accelerates attention computation and reduces memory footprint.
     • Approximate Attention Methods (Zaheer et al., 2020b; Beltagy et al., 2020; Wang et al., 2020; Kitaev et al., 2020): Trade model quality for efficiency by approximating the full attention mechanism (e.g., sparse attention, windowed attention).
   • Extending Pre-trained LLMs: Recent work involves extending pre-trained LLMs (often with RoPE) through position interpolation and fine-tuning (Chen et al., 2023; kaiokendev, 2023; bloc97, 2023; Peng et al., 2023).
   • Overall Limitation: These techniques extend context windows only to a limited extent, still falling short of handling limitless inputs required by streaming applications.

3. Improving LLMs' Utilization of Long Text: This research line focuses on whether LLMs actually use the information in long contexts effectively, rather than just taking them as input.

   • Findings by Liu et al. and Li et al. suggest that simply extending context size or extrapolation capabilities does not guarantee effective utilization of the long context.
   • Overall Limitation: This is an ongoing challenge. The paper notes its work concentrates on stably harnessing the most recent tokens, enabling streaming, but does not directly aim to enhance LLMs' ability to utilize all information across extremely long contexts.

3.3. Technological Evolution

The field of LLMs has rapidly evolved from early recurrent neural networks (RNNs) and LSTMs to the dominant Transformer architecture. Initial Transformers focused on fixed-length contexts, leading to quadratic memory and computational scaling with sequence length. This spurred innovations in:

1. Efficiency: Techniques like sparse attention, windowed attention, and system optimizations (FlashAttention) to reduce the computational burden.

2. Length Generalization: Methods like RoPE and ALiBi to allow models to handle sequences longer than seen during training, often combined with position interpolation and fine-tuning.

   This paper's work (StreamingLLM) fits into this evolution by addressing the practical deployment challenge of infinite sequence lengths in streaming applications. While previous works tried to expand the finite context window or generalize to longer finite sequences, StreamingLLM introduces a mechanism to operate continuously without a fixed hard limit, effectively moving towards "infinite" streaming. It leverages existing pre-trained LLMs without fine-tuning, offering a practical solution to a persistent deployment problem. It identifies a fundamental aspect of Transformer attention (attention sinks) that was previously overlooked in the context of streaming and uses this understanding to devise a novel, efficient strategy.

3.4. Differentiation Analysis

The StreamingLLM approach differentiates itself from previous and concurrent works primarily by:

• Focus on True Streaming vs. Finite Context Extension:

  • Existing Length Extrapolation and Context Window Extension methods: These aim to increase the finite maximum sequence length an LLM can handle in a single forward pass (e.g., from 4K to 32K or 128K tokens). They don't typically allow for truly infinite or continuous streaming where old tokens are constantly evicted.
  • StreamingLLM: Explicitly targets streaming applications that require continuous operation on infinite sequence lengths without resetting or losing coherence. It decouples the pre-training window from the inference generation length.

• Leveraging "Attention Sinks" for Stability:

  • Existing Window Attention: Naively discards older KV states once the window is full, leading to catastrophic performance collapse.
  • StreamingLLM: Recognizes and exploits the "attention sink" phenomenon. By intelligently preserving a small number of initial tokens (the attention sinks) alongside the sliding window of recent tokens, it stabilizes the attention distribution and recovers performance, turning a perceived problem into a solution.

• No Fine-tuning Requirement for Existing LLMs:

  • Many Context Window Extension methods: Often require fine-tuning (e.g., position interpolation) of pre-trained models to adapt them to longer contexts.
  • StreamingLLM (for already trained models): Operates as a simple, efficient framework that works with pre-trained LLMs off-the-shelf without any fine-tuning, making it immediately applicable and cost-effective.

• Efficiency Compared to Sliding Window with Re-computation:

  • Sliding Window with Re-computation: Achieves good performance by re-computing KV states for the entire window at each step, but this is quadratically slow and impractical for real-time streaming.
  • StreamingLLM: Offers comparable performance while achieving a 22.2x speedup by avoiding expensive re-computation, thanks to its fixed-size rolling KV cache plus attention sinks.

• Pre-training for Optimal Streaming (Dedicated Sink Token):

  • The paper's proposal to add a learnable placeholder token during pre-training is a proactive design choice for future LLMs. This allows a single, dedicated token to serve as an attention sink, further streamlining the streaming deployment compared to implicitly relying on multiple initial content tokens. This is a novel architectural modification specifically for streaming.

    In essence, StreamingLLM provides a practical and principled solution for a previously unmet need: enabling existing LLMs to operate robustly and efficiently in genuinely infinite streaming environments, bridging the gap between finite pre-training and continuous deployment.

4. Methodology

4.1. Principles

The core idea behind StreamingLLM is to address the catastrophic failure of window attention in Large Language Models (LLMs) when dealing with infinite input streams. This failure occurs because LLMs lose stability once the initial tokens, which act as "attention sinks," are evicted from the KV cache. The principle is that these initial tokens, despite often lacking semantic importance, are critical for anchoring the SoftMax distribution in the attention mechanism. By explicitly preserving these attention sink tokens alongside a rolling window of recent, semantically relevant tokens, StreamingLLM can maintain stable language modeling perplexity and efficiently generalize to arbitrarily long sequences.

The theoretical basis for attention sinks is rooted in the properties of the SoftMax function and the autoregressive training nature of LLMs:

1. SoftMax Property: The SoftMax function requires attention scores to sum to one. Even if a query token doesn't have a strong semantic match with any specific key token, the attention mechanism must still distribute attention values. The model learns to "dump" these "unneeded" attention values onto specific tokens.

2. Autoregressive Training: Initial tokens are visible to all subsequent tokens during autoregressive training. This constant visibility makes them ideal candidates for the model to learn to use as consistent attention sinks, as they are always present and can reliably collect attention scores.

   The StreamingLLM framework leverages this observation by ensuring that these crucial attention sinks are never evicted from the KV cache, thus preventing the attention distribution from destabilizing.

4.2. Core Methodology In-depth (Layer by Layer)

The StreamingLLM framework modifies the KV cache management strategy during the decoding phase of autoregressive LLMs. It comprises two main components: Attention Sinks and a Rolling KV Cache.

4.2.1. The Failure of Window Attention and the Role of Attention Sinks

The paper first empirically demonstrates the failure of window attention. While window attention (which only keeps the KV states of the most recent $L$ tokens) offers efficiency by maintaining constant memory usage, its performance (perplexity) drastically increases once the text length exceeds the cache size and the initial tokens are evicted. This suggests that initial tokens, regardless of their semantic content, are crucial for the stability of LLMs.

To illustrate this, the paper shows that even replacing the initial four tokens with semantically irrelevant linebreak tokens ($"\n"$) still restores the model's perplexity to acceptable levels, comparable to keeping the original initial tokens. This indicates that it's the absolute position of these starting tokens, rather than their specific semantic value, that is significant.

The paper explains this phenomenon using the SoftMax function. The attention score for a query $Q_i$ with respect to key $K_j$ is given by $QK^T/\sqrt{d_k}$, and then normalized by SoftMax. Specifically, for the $i$-th element of the SoftMax output: $ \mathrm { S o f t M a x } ( x ) _ { i } = \frac { e ^ { x _ { i } } } { e ^ { x _ { 1 } } + \sum _ { j = 2 } ^ { N } e ^ { x _ { j } } } $ where $x_j$ are the raw attention logits. If the model learns to assign a very high logit $x_1$ to the first token (i.e., $x_1 \gg x_j$ for $j \in \{2, \dots, N\}$), this first token acts as an "attention sink" because its exponential term dominates the denominator. Removing the KV state of this initial token (or other strong sink tokens) would significantly alter the denominator, leading to a drastic shift in the distribution of attention scores across the remaining tokens, causing instability and perplexity surge.

The reason initial tokens become sinks is their global visibility in autoregressive training. Since they are always present for all subsequent tokens, the model can reliably train them to absorb "unneeded" attention, ensuring the SoftMax normalization property is met without forcing attention onto semantically irrelevant tokens elsewhere. The paper finds that typically multiple initial tokens (e.g., four) are used as sinks because LLMs are not trained with a consistent single starting token (e.g., Llama-2's $<s>$ token might not consistently be at the very first position after text chunking).

4.2.2. Rolling KV Cache with Attention Sinks

To implement StreamingLLM with already trained LLMs, the method is straightforward:

1. Conceptual Division of KV Cache: The KV cache is logically divided into two parts:

   • Attention Sinks: A fixed, small number of initial tokens (typically 4, as determined by ablation studies) whose KV states are permanently preserved. These tokens stabilize the attention computation.

   • Rolling KV Cache: A fixed-size sliding window that stores the KV states of the most recent tokens. These tokens are crucial for language modeling as they provide the most relevant recent context. As new tokens are generated, the oldest tokens in this rolling cache are evicted to maintain constant memory.

     This strategy ensures that the essential attention sinks are never removed, preventing the SoftMax distribution collapse, while the rolling cache provides recent contextual information.

The KV cache of StreamingLLM is conceptually divided into Attention sinks (four initial tokens) and Rolling KV Cache (retains the most recent tokens).

该图像是一个示意图，展示了StreamingLLM中的KV缓存机制。图中显示了在生成第7、8和9个token时的不同状态，包括注意力汇聚（Attention Sinks）、被驱逐的token和滚动KV缓存。该图旨在说明如何利用初始token来改进窗口注意力的性能。

Figure 4: The KV cache of StreamingLLM shows the process of generating tokens 7, 8, and 9. The "Attention Sinks" (tokens 0, 1, 2, 3) are always kept. The "Rolling KV Cache" (tokens 4, 5, 6 for generating token 7; tokens 5, 6, 7 for generating token 8; and tokens 6, 7, 8 for generating token 9) slides, with older tokens being evicted.

4.2.3. Positional Encoding Handling

StreamingLLM is designed to be compatible with LLMs that use relative positional encoding techniques like RoPE (Rotary Position Embeddings) and ALiBi (Attention with Linear Biases). A crucial aspect is how positional information is assigned:

• Cache-based Positioning: Instead of using positions from the original full text, StreamingLLM assigns positions based on the tokens' positions within the current KV cache. For example, if the cache contains tokens [0, 1, 2, 3] (sinks) and [6, 7, 8] (recent), and the model is decoding the 9th token, the positions assigned internally for attention computation would be [0, 1, 2, 3, 4, 5, 6, 7] (where 4, 5, 6, 7 correspond to 6, 7, 8, 9 in the original text, effectively treating the cache as a contiguous block).

• RoPE Integration: For RoPE-based models, the Keys of tokens are cached before the rotary transformation is applied. Then, during each decoding phase, the rotary transformation is applied to the keys within the rolling cache using their cache-relative positions.

• ALiBi Integration: ALiBi is more direct, as it applies a contiguous linear bias to attention scores based on distance. In StreamingLLM, this bias is applied using the cache-relative distances rather than original text distances.

  This cache-centric positional embedding strategy is fundamental to StreamingLLM's ability to operate beyond the pre-training window.

4.2.4. Pre-Training LLMs with Attention Sinks

To further optimize streaming deployment, the paper proposes a modification during the pre-training phase of LLMs. The goal is to consolidate the attention sink role into a single, explicit token, rather than relying on multiple implicit initial tokens.

Two alternative approaches are explored:

1. SoftMax-Off-by-One (Zero Sink): This variant of the SoftMax function (Miller, 2023) does not enforce attention scores to sum to one across all contextual tokens. The formula is: $ \mathrm { S o f t M a x } _ { 1 } ( x ) _ { i } = \frac { e ^ { x _ { i } } } { 1 + \sum _ { j = 1 } ^ { N } e ^ { x _ { j } } } $ This is equivalent to prepending a token with all-zero Key and Value features, effectively creating an implicit sink that can absorb attention without contributing semantic information. The paper denotes this as "Zero Sink."

2. Learnable Placeholder Token (Sink Token): The recommended approach is to explicitly add an extra learnable token at the beginning of all training samples. This Sink Token serves as a designated, trainable repository for unnecessary attention scores.

   The experiments demonstrate that while Zero Sink helps alleviate the attention sink problem, the model still requires other initial tokens to fully stabilize. In contrast, training with a learnable Sink Token is highly effective. By simply pairing this single Sink Token with recent tokens, the model's perplexity is stabilized, and even marginally improved. This suggests that future LLMs could be pre-trained with such a dedicated Sink Token to optimize for streaming deployment.

5. Experimental Setup

5.1. Datasets

The experiments primarily use the following datasets:

• PG19 Test Set: This dataset (Rae et al., 2020) consists of 100 long books. It is used for evaluating language modeling perplexity on super long texts (up to 4 million tokens). The books are concatenated to create continuous long streams.

  • Source: A collection of public domain books.
  • Scale: 100 books, providing very long sequences.
  • Characteristics: Diverse narrative text, suitable for evaluating language model performance on extensive, coherent texts.
  • Domain: General literature.
  • Purpose: Ideal for testing LLMs' ability to maintain stable performance over millions of tokens, far exceeding their typical training context length.

• The Pile (Gao et al., 2020): A large, diverse, open-source dataset used for pre-training the 160-million parameter language models from scratch to validate the sink token hypothesis.

  • Source: A vast collection of text from 22 diverse high-quality subsets, including books, scientific papers, web pages, code, etc.
  • Scale: 800 GB of text.
  • Characteristics: Highly diverse, covering many domains and styles. The deduplicated version was used.
  • Domain: General-purpose text, scientific, code, etc.
  • Purpose: To train new LLMs from scratch under controlled conditions to observe the impact of adding a sink token during pre-training.

• ARC-[Challenge, Easy] Datasets (Clark et al., 2018): Used for evaluating multi-round question-answering with instruction-tuned LLMs.

  • Source: A dataset of science questions designed to be challenging for AI requiring multi-hop reasoning.
  • Characteristics: Focuses on natural language understanding and reasoning.
  • Purpose: To assess StreamingLLM's applicability in real-world interactive scenarios like dialogue systems by concatenating QA pairs into a stream.

• StreamEval: A custom dataset inspired by LongEval (Li et al., 2023), designed to evaluate streaming question-answering.

  • Characteristics: Differs from LongEval by querying the model every 10 lines of new information, with answers consistently 20 lines prior. This reflects real-world scenarios where questions pertain to recent information.
  • Example Data Sample (Conceptual):

    Line 1: ... (some text)
    Line 2: ... (some text)
    ...
    Line 19: ... (some text)
    Line 20: This is the answer to Query A.
    Line 21: ... (some text)
    ...
    Query A: What is the answer to Query A? (query about Line 20)
    Line 22: ... (some text)
    ...
    Line 40: This is the answer to Query B.
    Line 41: ... (some text)
    ...
    Query B: What is the answer to Query B? (query about Line 40)
    

  • Purpose: To specifically test StreamingLLM's ability to maintain accuracy on instruction-tuned models as input length grows, focusing on recent context retrieval.

• LongBench (Bai et al., 2023): A comprehensive benchmark for long context understanding, covering single-document QA, multi-document QA, and summarization.

  • Source: Includes various datasets like NarrativeQA, Qasper, HotpotQA, 2WikiMQA, GovReport, MultiNews.
  • Purpose: To evaluate StreamingLLM's performance on standard long-range NLP tasks and compare it against a default truncation baseline.

5.2. Evaluation Metrics

The paper uses several standard metrics to evaluate StreamingLLM's performance:

• Language Modeling Perplexity (PPL):

  1. Conceptual Definition: Perplexity is a measure of how well a probability model predicts a sample. In language modeling, it quantifies how well the model predicts a sequence of words. A lower perplexity score indicates a better language model, as it means the model is more confident and accurate in its predictions. It can be interpreted as the inverse probability of the test set, normalized by the number of words.
  2. Mathematical Formula: For a sequence of tokens $W = (w_1, w_2, \ldots, w_N)$, the perplexity is defined as: $ \mathrm{PPL}(W) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i | w_1, \ldots, w_{i-1})\right) $
  3. Symbol Explanation:
     • $W$: The sequence of tokens being evaluated.
     • $N$: The total number of tokens in the sequence $W$.
     • $\exp(\cdot)$: The exponential function (base $e$).
     • $\log(\cdot)$: The natural logarithm.
     • $P(w_i | w_1, \ldots, w_{i-1})$: The probability assigned by the language model to the $i$-th token $w_i$, given all the preceding tokens $w_1, \ldots, w_{i-1}$.

• Exact Match (EM) Accuracy:

  1. Conceptual Definition: Exact Match accuracy is a stringent metric often used in question-answering tasks. A prediction is considered correct only if it is an exact, character-for-character match with the ground truth answer. This metric measures the model's ability to produce precise and identical answers.
  2. Mathematical Formula: $ \mathrm{EM} = \frac{\text{Number of exact matches}}{\text{Total number of questions}} \times 100% $
  3. Symbol Explanation:
     • Number of exact matches: The count of predictions that are identical to their respective ground truth answers.
     • Total number of questions: The total number of questions in the evaluation set.

• Speedup (Efficiency):

  1. Conceptual Definition: Speedup measures the improvement in computation time (or reduction in latency) of a new method compared to a baseline. A speedup of $X$ means the new method is $X$ times faster. In this paper, it's particularly used for per-token decoding latency.
  2. Mathematical Formula: $ \mathrm{Speedup} = \frac{\text{Time taken by Baseline}}{\text{Time taken by StreamingLLM}} $
  3. Symbol Explanation:
     • Time taken by Baseline: The time (e.g., decoding latency per token) required by the comparison method (e.g., sliding window with re-computation).
     • Time taken by StreamingLLM: The time required by the proposed StreamingLLM method for the same task.

• Memory Usage (Efficiency):

  1. Conceptual Definition: Memory usage quantifies the amount of computer memory (e.g., GPU memory) consumed by a model during its operation. Lower memory usage indicates higher efficiency, especially critical for deploying large models.
  2. Mathematical Formula: Not a single formula, but typically measured in Gigabytes (GB) or Megabytes (MB).
  3. Symbol Explanation: N/A (direct measurement).

5.3. Baselines

The paper compares StreamingLLM against several established methods:

• Dense Attention:

  • Description: The standard Transformer attention where KV states for all previous tokens are maintained in the KV cache.
  • Representativeness: This is the default, most common attention mechanism in LLMs.
  • Limitations: Suffers from quadratic memory and computational complexity with increasing sequence length, leading to Out-of-Memory (OOM) errors and performance degradation when exceeding the pre-training window.

• Window Attention (Sliding Window Attention):

  • Description: Maintains a fixed-size sliding window of KV states for only the most recent tokens. Older KV states outside this window are discarded.
  • Representativeness: A common approach for managing memory in long sequence processing.
  • Limitations: While efficient in terms of memory and speed, the paper shows it collapses in performance (high perplexity) once initial tokens are evicted from the cache.

• Sliding Window with Re-computation:

  • Description: For each new token generation, the KV states for the entire sliding window of recent tokens are recomputed from scratch.
  • Representativeness: Represents an "oracle" baseline for performance, as it avoids the collapse of simple window attention by always having the full recent context, even if inefficiently.
  • Limitations: Offers strong performance but is significantly slower due to the quadratic attention computation within its window at every step, making it impractical for real-world streaming.

• Context-extended models (e.g., LongChat-7b-v1.5-32k, Llama-2-7B-32KInstruct):

  • Description: These are LLMs that have been specifically fine-tuned or modified (e.g., using position interpolation) to extend their context window to a larger, but still finite, size (e.g., 32K tokens).
  • Representativeness: Showcases the state-of-the-art in finite context window extension.
  • Purpose: Used to demonstrate that StreamingLLM can complement these methods (by broadening the maximum cache size of streaming LLMs, enabling broader local information capture) rather than replace them.

Models Evaluated with StreamingLLM: The paper evaluates StreamingLLM across a diverse range of prominent LLM families and scales to ensure the robustness and generalizability of its findings:

• Llama-2: Llama-2-[7, 13, 70]B models (Touvron et al., 2023b). These use RoPE.

• MPT: MPT-[7, 30]B models (Team, 2023). These employ ALiBi.

• Pythia: Pythia-[2.9, 6.9, 12]B models (Biderman et al., 2023). These use RoPE.

• Falcon: Falcon-[7, 40]B models (Almazrouei et al., 2023). These use RoPE.

  This diverse selection, encompassing different model sizes and leading positional encoding techniques, strengthens the validity of StreamingLLM's applicability.

6. Results & Analysis

6.1. Core Results Analysis

The experimental results consistently demonstrate the effectiveness and efficiency of StreamingLLM across various LLM families, scales, and tasks.

1. Language Modeling on Long Texts: The paper first evaluates StreamingLLM's language modeling perplexity on the concatenated PG19 test set (100 long books).

• Failure of Baselines: Dense attention fails (OOM or high PPL) when input length surpasses its pre-training window. Window attention collapses dramatically once initial tokens are evicted (i.e., input length exceeds cache size).

• StreamingLLM's Stability: StreamingLLM consistently matches the oracle baseline (sliding window with re-computation) in terms of perplexity, demonstrating stable performance. More importantly, it maintains this stability over exceptionally extended texts, reaching up to 4 million tokens and more across all tested LLM families (Llama-2, MPT, Falcon, Pythia) and scales. This is a critical validation of its ability to generalize to effectively infinite sequence lengths.

  该图像是一个图表，展示了不同语言模型在处理20K tokens文本时的语言建模困惑度(log perplexity)。观察到，当输入长度超过预训练的注意力窗口时，密集注意力表现出明显的崩溃，而StreamingLLM则保持稳定，几乎与滑动窗口的重新计算基线相当。

Figure 3: Language modeling perplexity on texts with 20K tokens across various LLM. Observations reveal consistent trends:(1) Dense attention fails once the input length surpasses the pre-training attention window s Windowattenti colapses nce the iput lnth exceds the cache ize, ie the iiial tokens e evicted. (3) StreamingLLM demonstrates stable performance, with its perplexity nearly matching that of the sliding window with re-computation baseline.

该图像是一个图表，展示了StreamingLLM在超长文本上进行语言建模时的困惑度（perplexity）变化。图中分别列出了Llama-2、Pythia、Falcon和MPT这几种LLM的表现，困惑度在不同输入长度下保持稳定。

Figure 5: Language modeling perplexity of StreamingLLM on super long texts with 4 million tokens across varius LLM families and scales. The perplexity remains stable throughout. We use the concatenated test set of PG19 (100 books) to perform language modeling, with perplexity fluctuations due to book transitions.

2. Results of Pre-Training with a Sink Token: The paper validates the hypothesis that pre-training with a dedicated sink token can improve streaming LLMs.

• Convergence and Normal Performance: Pre-training a 160M-parameter model with a sink token shows similar convergence dynamics to a vanilla model (Figure 6) and no negative impact on performance across 7 NLP benchmarks (ARC-[Challenge, Easy], HellaSwag, LAMBADA, OpenbookQA, PIQA, and Winogrande) (Table 4). This indicates that the architectural change doesn't harm the model's general capabilities.

  该图像是训练损失曲线图，展示了模型在使用和不使用 sink token 时的收敛趋势。蓝色线条表示未使用 sink token 的模型，橙色线条表示添加了 sink token 的模型，二者在 k 步骤的训练损失趋于相似。

Figure 6: Pre-training loss curves of models w/ and w/o sink tokens. Two models have a similar convergence trend.

The following are the results from Table 4 of the original paper:

• Streaming Performance: Critically, the vanilla model requires multiple initial tokens as attention sinks to maintain stable streaming perplexity. In contrast, the model trained with a sink token achieves satisfactory streaming performance using only the single designated sink token (Table 3), simplifying the cache management and making the attention sink mechanism explicit.

  The following are the results from Table 3 of the original paper:

  Cache Config	0+1024	1+1023	2+1022	4+1020
  Vanilla	27.87	18.49	18.05	18.05
  Zero Sink	29214	19.90	18.27	18.01
  Learnable Sink	1235	18.01	18.01	18.02

• Attention Visualization: Visualization (Figure 7) confirms that models without a sink token distribute attention locally in lower layers and then heavily on initial tokens in deeper layers. Models with a sink token consistently concentrate attention on this sink token across all layers and heads, effectively offloading attention and reducing focus on other initial content tokens.

  该图像是一个比较图，显示了在使用和不使用池化标记（sink token）预训练的模型在不同层和头上的平均注意力日志的可视化。左侧为未使用池化标记的模型，其低层显示局部注意力，而右侧则是使用池化标记的模型，初始标记的注意力增强了流式性能。

Figure 7: Visualization of average attention logits over 256 sentences, each 16 tokens long, comparing models pre-trained without (left) and with (right) a sink token. Both maps show the same layers and heads. Key observations: (1) Without a sink token, models show local attention in lower layers and increased attention tkes deyersWi tke the ee e , eivel ceanatenion.Wit he ec the tokn attenion ive initial tokens, supporting the benefit of designating the sink token to enhance the streaming performance.

3. Streaming Question Answering with Instruction-Tuned Models:

• Real-world Applicability: On a simulated multi-round QA task using concatenated ARC datasets and instruction-tuned Llama-2-Chat models: Dense attention resulted in OOM errors. Window attention was efficient but had low accuracy due to random outputs. StreamingLLM efficiently handled the streaming format, maintaining accuracy aligned with a one-shot, sample-by-sample baseline.

• StreamEval Benchmark: On StreamEval, LLMs employing StreamingLLM maintain reasonable accuracy even with inputs approaching 120K tokens. In contrast, dense and window attention fail at much shorter lengths.

• Complementary to Context Extension: StreamingLLM can effectively complement context extension methods (e.g., LongChat-7b-v1.5-32k). StreamingLLM's cache size can be "extended" by these methods, allowing it to capture broader local information, thus demonstrating its versatility.

  该图像是一个性能比较图，展示了不同模型（如Llama-2系列）在StreamEval基准测试上的精度表现。通过密集注意力、窗口注意力和StreamingLLM三种方法，随着输入长度增加，精度变化趋势有所不同，StreamingLLM在长输入下显示出更稳健的性能。

Figure 9: Performance on the StreamEval benchmark. Accuracies are averaged over 100 samples.

6.2. Data Presentation (Tables)

The following are the results from Table 1 of the original paper:

The following are the results from Table 2 of the original paper:

The following are the results from Table 3 of the original paper:

The following are the results from Table 4 of the original paper:

The following are the results from Table 6 of the original paper:

The following are the results from Table 7 of the original paper:

The following are the results from Table 8 of the original paper:

The following are the results from Table 9 of the original paper:

The following are the results from Table 10 of the original paper:

6.3. Ablation Studies / Parameter Analysis

The paper conducts several ablation studies to understand the impact of different parameters and design choices within StreamingLLM.

1. Numbers of Initial Tokens:

• Purpose: To determine the optimal number of initial tokens required as attention sinks to stabilize performance.
• Experiment: Streaming perplexity is evaluated by varying the number of initial tokens (0, 1, 2, 4, 8) kept with a rolling window of recent tokens (e.g., $x + y$ denotes $x$ initial tokens and $y$ recent tokens).
• Results (Table 2):
  • Window attention ($0 + y$) leads to a drastic increase in perplexity, confirming its failure without initial tokens.
  • Introducing only one or two initial tokens does not fully restore model perplexity, indicating that LLMs don't solely rely on the very first token as an attention sink.
  • Introducing four initial tokens generally suffices to achieve full recovery, with further additions (e.g., eight tokens) yielding only diminishing returns.
• Conclusion: This justifies the choice of using four initial tokens as attention sinks in the standard StreamingLLM configuration, balancing performance restoration with minimal KV cache overhead.

2. Cache Sizes:

• Purpose: To investigate the effect of the rolling KV cache size on StreamingLLM's perplexity.
• Experiment: Streaming perplexity is evaluated with a fixed number of initial attention sinks (4) but varying the size of the rolling window (e.g., $4 + 252$, $4 + 508$, $4 + 1020$, $4 + 2044$).
• Results (Table 6): Contrary to intuition, increasing the rolling cache size does not consistently lower the language modeling perplexity. In some cases (e.g., MPT-7B, Falcon-7B), perplexity can even slightly increase with a larger cache.
• Conclusion: This finding suggests a potential limitation: current LLMs might not be able to maximize the utility of the entire context they receive within a very large rolling window. It highlights a broader challenge in LLM research regarding effective long-context utilization, aligning with observations from other studies (Liu et al.).

3. Using More Sink Tokens in the Pre-Training Stage:

• Purpose: To explore if pre-training with more than one dedicated sink token could further optimize performance.
• Experiment: Models are pre-trained with 0, 1, or 2 learnable sink tokens. Their pre-training loss curves, zero-shot accuracy on NLP benchmarks, and streaming perplexity are compared.
• Results (Figure 15, Table 9, Table 10):
  • Adding either one or two sink tokens results in similar pre-training loss curves to the baseline, indicating no convergence issues.
  • The addition of a second sink token does not yield substantial improvements in zero-shot accuracy across most benchmark tasks (Table 9).
  • For streaming perplexity, the model trained with 2 sink tokens appears to rely on both to maintain stable performance, and the results are not better than using just one optimally.
• Conclusion: A single dedicated sink token is adequate and optimal for improving streaming performance. Adding more sink tokens during pre-training does not lead to further enhancements in overall language model performance or streaming stability, contrasting with observations in Vision Transformers (ViTs) where multiple "registers" have been found beneficial.

Efficiency Results:

• Purpose: To quantify the speedup and memory footprint of StreamingLLM compared to the sliding window with re-computation baseline.

• Experiment: Benchmarking decoding latency and memory usage on Llama-2-[7, 13]B models, varying cache size.

• Results (Figure 10):

  • Decoding Latency: StreamingLLM shows linear growth in decoding speed as cache size increases, whereas sliding window with re-computation exhibits a quadratic rise. This results in an impressive speedup of up to 22.2x per token for StreamingLLM.
  • Memory Usage: StreamingLLM maintains a memory footprint similar to the re-computation baseline.

• Conclusion: StreamingLLM offers a significant practical advantage by dramatically improving decoding speed while keeping memory usage in check, making it highly efficient for streaming deployments.

  该图像是图表，比较了 Llama-2-7B 和 Llama-2-13B 在不同缓存大小下，滑动窗口重新计算方法与 StreamingLLM 的每个 token 解码延迟及内存使用。结果显示，StreamingLLM 在延迟和内存使用方面均优于滑动窗口方案。

Figure 10: Comparison of per-token decoding latency and memory usage between the sliding window approach with re-computation baseline and StreamingLLM, plotted against the cache size (attention window size) on the $\\mathrm { X }$ -axis. StreamingLLM delivers a remarkable speedup of up to $2 2 . 2 \\times$ per token and retains a memory footprint similar to the re-computation baseline.

6.4. Long-Range Benchmark Evaluation

• Purpose: To evaluate StreamingLLM's performance on standard long-range NLP tasks using LongBench and compare it to a default truncation baseline.
• Experiment: Llama-2-7B-chat (max context 4k) is evaluated on LongBench tasks (single-document QA, multi-document QA, summarization). The baseline truncates inputs to 1750 initial and 1750 final tokens ($1750+1750$). StreamingLLM is tested with $4+3496$ (4 sinks + 3496 recent) and $1750+1750$ (1750 sinks + 1750 recent) cache configurations.
• Results (Table 8):
  • StreamingLLM 4+3496 (using only 4 initial tokens as sinks) underperforms the truncation baseline. This is because LongBench tasks often require information from the very beginning of the long document, and simply having 4 general sink tokens is insufficient to preserve this task-specific critical initial context.
  • However, when StreamingLLM is configured with $1750+1750$ (i.e., preserving the first 1750 tokens as "sinks" and a recent window of 1750 tokens), its performance is restored to be comparable to the truncation baseline.
• Conclusion: This indicates that StreamingLLM's effectiveness is contingent on the information within its cache. If crucial initial prompt information is needed for a specific task (like in LongBench), the "sink" part of the cache might need to be expanded to retain that information. It corroborates the finding that StreamingLLM cannot extend the context length itself but rather makes the utilization of the available cache stable and efficient.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces StreamingLLM, a novel and highly efficient framework for deploying Large Language Models (LLMs) in streaming applications. The core innovation stems from the observation of "attention sinks"—initial tokens that disproportionately attract attention scores, stabilizing the attention mechanism even when semantically unimportant. StreamingLLM leverages this by preserving a small, fixed number of initial attention sink tokens alongside a sliding window of recent tokens in the KV cache. This strategy enables LLMs (like Llama-2, MPT, Falcon, and Pythia) to generalize to effectively infinite sequence lengths (up to 4 million tokens and more) without any fine-tuning, maintaining stable language modeling perplexity. Furthermore, the paper demonstrates that pre-training LLMs with a dedicated learnable placeholder token can serve as a more explicit and efficient attention sink, further improving streaming deployment. StreamingLLM achieves a remarkable 22.2x speedup in decoding latency compared to sliding window recomputation, while maintaining similar memory footprint. Ultimately, StreamingLLM successfully decouples an LLM's pre-training window size from its operational text generation length, paving the way for practical and efficient streaming LLM deployment.

7.2. Limitations & Future Work

The authors explicitly acknowledge several limitations and suggest future research directions:

• No Context Window Extension or Long-Term Memory Enhancement: StreamingLLM does not inherently extend the LLMs' context window or improve their long-term memory capabilities. It efficiently utilizes the information within its current cache. This means it's still bounded by the cache size for tasks requiring information from very distant past tokens.
• Unsuitability for Long-Term Data Dependency: Consequently, StreamingLLM is not suitable for tasks that strictly demand very long-term memory and extensive data dependency, such such as long document question-answering (QA) or summarization that needs to recall facts from thousands of tokens ago. This is demonstrated in the LongBench evaluation, where StreamingLLM needs a large "sink" portion to match truncation baselines on tasks requiring initial document context.
• Suboptimal Context Utilization: The ablation study on cache sizes revealed that increasing the rolling cache size doesn't consistently decrease perplexity, suggesting that current LLMs might not fully maximize the utility of the entire context provided within a large cache.
• Future Research on Context Utilization: The authors suggest that future research should focus on enhancing LLMs' capabilities to better utilize extensive contexts, even those available within their cache.

7.3. Personal Insights & Critique

This paper offers a highly practical and insightful solution to a critical deployment challenge for LLMs. The discovery of "attention sinks" is a significant finding, revealing a fundamental, often overlooked, aspect of Transformer behavior. The elegance of StreamingLLM lies in its simplicity – it's a minimal intervention (just preserving a few initial tokens) that yields massive benefits in stability and efficiency without requiring complex fine-tuning for existing models.

Innovations and Transferability:

• The "Attention Sink" Phenomenon: Identifying and characterizing "attention sinks" is the paper's most profound contribution. It provides a deeper understanding of how Transformers manage their attention distribution, particularly under the constraint of the SoftMax normalization. This phenomenon, observed across decoder-only, encoder-only (BERT), and Vision Transformers (ViTs), suggests a universal property of Transformer architectures.
• Simple yet Effective Solution: The StreamingLLM framework is remarkably simple to implement yet incredibly effective. This low-cost, high-impact approach is highly valuable in a field often characterized by computationally expensive solutions.
• Pre-training for Streaming: The idea of explicitly designing LLMs for streaming by adding a learnable sink token during pre-training is a forward-thinking approach. It moves beyond post-hoc fixes to integrate streaming considerations into the foundational model design. This could inspire similar design considerations for other LLM deployment aspects.
• Broad Applicability: The method's success across diverse LLM families (Llama-2, MPT, Falcon, Pythia) and scales demonstrates its generalizability, making it immediately useful for a wide range of current LLMs.

Potential Issues and Areas for Improvement:

• Understanding Attention Sink Semantics: While the paper argues attention sinks are semantically unimportant, a deeper investigation into what information, if any, these tokens implicitly encode or aggregate over long sequences could be insightful. Could they be learning a compressed representation of initial context or acting as a "reset" mechanism?

• Hard Limit on Context: StreamingLLM solves the "infinite stream" problem, but it doesn't solve the "infinite context" problem. The fixed-size rolling cache still means a hard limit on how far back the model can semantically attend. Future work could explore hybrid approaches, perhaps combining StreamingLLM with sparse retrieval mechanisms to access truly old, relevant information beyond the rolling cache.

• "Effective" Cache Size vs. Actual Use: The observation that increasing cache size doesn't always improve perplexity is a critical insight. This points to a fundamental limitation in current LLMs regarding their ability to effectively utilize long contexts. This could be due to attention dilution, where too much context makes it harder to focus, or positional encoding limitations even with relative schemes. Research into making LLMs "smarter" at long-context understanding is crucial.

• Generalization of Attention Sinks across Architectures: The paper briefly mentions attention sinks in BERT and ViTs. Further dedicated research could explore the nuances of attention sink behavior in these different Transformer variants and whether similar (or adapted) sink token strategies could benefit them. For instance, in ViTs, the "registers" (Darcet et al., 2023) are found to be beneficial in multiples, which contrasts with the single sink token finding for LLMs. Understanding this divergence could yield new architectural insights.

• Dynamic Sink Management: The current StreamingLLM uses a fixed number of initial tokens as sinks. Could a dynamic approach, where attention sink tokens are identified or created on-the-fly based on attention patterns or task requirements, yield even better results or adaptability?

  In conclusion, StreamingLLM is an excellent example of how deep empirical observation of model behavior can lead to simple, yet highly impactful, practical solutions. It's a significant step towards making LLMs truly robust for continuous, real-world streaming applications.