1. Bibliographic Information

1.1. Title

OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting

1.2. Authors

Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, Sifan Zhou. Affiliations include Houmo AI, Nanjing University, and Southeast University.

1.3. Journal/Conference

This paper is a preprint published on arXiv. It is currently under review or has not yet been officially published in a peer-reviewed journal or conference proceeding. arXiv is a reputable open-access archive for preprints of scientific papers, particularly in physics, mathematics, computer science, and related disciplines, widely used by the academic community to disseminate research findings prior to formal publication.

1.4. Publication Year

2025

1.5. Abstract

Post-training quantization (PTQ) is a critical technique for compressing and accelerating Large Language Models (LLMs). A significant challenge in LLM quantization arises from uneven and heavy-tailed data distributions, which expand the quantization range and consequently reduce bit precision for most values. Existing methods often employ heuristic linear transformations to address outliers and inter-channel differences but frequently overlook optimizing the data distribution across the entire quantization space.

This paper introduces Quantization Space Utilization Rate (QSUR), a novel metric designed to assess the quantizability of transformed data by measuring its space utilization within the quantization space. Complementing QSUR with mathematical derivations, the authors examine the effects and limitations of various transformations. This theoretical foundation guides the development of Orthogonal and Scaling Transformation-based Quantization (OSTQuant). OSTQuant employs learnable equivalent transformations, comprising both orthogonal and scaling components, to optimize weight and activation distributions across the entire quantization space. Additionally, the paper proposes KL-Top loss function, which mitigates noise during optimization while preserving richer semantic information, particularly valuable given the limited calibration data typical in PTQ.

OSTQuant demonstrates superior performance over existing methods across various LLMs and benchmarks. In a W4-only setting, it retains 99.5% of the floating-point accuracy. For the more challenging W4A4KV4 configuration, OSTQuant reduces the performance gap by 32% on the LLaMA-3-8B model compared to state-of-the-art methods.

1.6. Original Source Link

https://arxiv.org/abs/2501.13987v1 (This is a preprint on arXiv.) PDF Link: https://arxiv.org/pdf/2501.13987v1.pdf

2. Executive Summary

2.1. Background & Motivation

The rapid advancement of Large Language Models (LLMs) has led to their widespread adoption, but their substantial memory and computational requirements pose significant deployment challenges. These challenges exist not only for resource-constrained edge devices but also for powerful cloud servers, limiting their practical applicability.

Post-training quantization (PTQ) is a widely adopted technique to compress and accelerate LLMs by reducing the bit-width of weights and activations after the model has been trained. However, a major hurdle in LLM quantization is the presence of uneven and heavy-tailed data distributions in weights and activations. These distributions mean that a few extreme values (outliers) dictate a large quantization range. Consequently, the limited number of available quantization levels must be stretched over this wide range, leading to reduced precision for the majority of values that fall within the narrower, central part of the distribution. This ultimately results in significant quantization errors and a drop in model performance.

Prior research has attempted to address these issues using linear transformations, such as smooth-based methods (e.g., SmoothQuant) that redistribute quantization difficulty between weights and activations, and rotation-based methods (e.g., QuaRot, SpinQuant) that aim to suppress outliers. While these methods have shown some effectiveness in specific regions of the quantization space, they are often heuristic and fail to holistically optimize the data distribution across the entire quantization space. This leads to sub-optimal utilization of the available quantization levels and leaves a notable gap in ensuring robust performance at very low bit-widths. The paper identifies this gap as the lack of a quantitative metric to assess quantizability and the absence of a method that globally optimizes data distribution for better fit within the quantization space.

2.2. Main Contributions / Findings

The paper introduces several primary contributions to address the challenges in LLM quantization:

• Quantization Space Utilization Rate (QSUR): The authors propose QSUR, a novel and effective metric to quantitatively assess the quantizability of transformed data. QSUR measures how effectively data utilizes the available quantization space. Mathematical derivations are provided to analyze the effects and limitations of various linear transformations on QSUR, establishing a theoretical foundation for designing more effective transformations. Experiments demonstrate a positive correlation between higher QSUR values and improved quantization accuracy.
• Orthogonal and Scaling Transformation-based Quantization (OSTQuant): Based on the insights from QSUR, OSTQuant is proposed. It employs learnable equivalent transformation pairs, each consisting of an orthogonal transformation and a scaling transformation, assigned to each fully connected (FC) layer in LLMs. These transformations are optimized to globally reshape the distributions of weights and activations across the entire quantization space, making them more quantization-friendly. Critically, these transformation pairs and their inverses are fused into their respective FC layers after optimization, preserving the original computational graph and incurring no additional computational overhead during inference. The method also introduces Weight Outlier Minimization Initialization (WOMI) for effective initialization of orthogonal matrices.
• KL-Top Loss Function: Recognizing that PTQ typically uses small calibration datasets (e.g., 1000 samples), which can lead to overfitting with standard cross-entropy loss or noise with full Kullback-Leibler divergence on large vocabularies, the KL-Top loss function is introduced. This loss function computes KL divergence only over the top $k$ highest-probability logits from the full-precision model. This approach aims to capture more nuanced semantic information and provide clearer, more informative gradients while mitigating noise from long-tail distributions and reducing computational costs.
• Robust Performance and Efficiency: OSTQuant consistently outperforms existing state-of-the-art PTQ methods across various LLMs (LLaMA-1, LLaMA-2, LLaMA-3) and benchmarks.

  • In the W4-only (4-bit weights, 16-bit activations/KV cache) setting, it retains over 99.5% of the floating-point accuracy.

  • In the more aggressive W4A4KV4 (4-bit weights, 4-bit activations, 4-bit KV cache) configuration, it reduces the performance gap by 32% on the LLaMA-3-8B model compared to state-of-the-art methods, maintaining at least 96% of the original model's performance.

  • It achieves significant inference speedups (over $2 \times$ average, up to $3 \times$ for LLaMA-30B) and memory savings (exceeding $3.5 \times$).

  • The optimization process is highly efficient, quantizing LLaMA-7B in about 20 minutes, up to $5.3 \times$ faster than block reconstruction methods like OmniQuant.

    These contributions collectively provide a more theoretically grounded and empirically effective approach to LLM quantization, making these powerful models more accessible and efficient for diverse deployment scenarios.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand OSTQuant, a foundational grasp of Large Language Models (LLMs), quantization principles, and linear algebra concepts is essential.

• Large Language Models (LLMs): These are advanced artificial intelligence models, typically based on the transformer architecture, that are trained on vast amounts of text data to understand, generate, and process human language. They excel at tasks like translation, summarization, question-answering, and creative writing. Examples include OpenAI's GPT series, Google's Bard/Gemini, and Meta's LLaMA series. Their immense size (billions of parameters) makes them computationally intensive and memory-hungry, necessitating compression techniques for practical deployment.

• Post-Training Quantization (PTQ): This is a model compression technique applied after a neural network has been fully trained (post-training). The goal is to reduce the numerical precision (bit-width) of the model's parameters (weights) and intermediate computations (activations) from high-precision floating-point numbers (e.g., FP32 or FP16) to lower-precision integers (e.g., INT8 or INT4).

  • Benefits: PTQ significantly reduces memory footprint (allowing larger models on smaller devices or more models on a single device) and accelerates inference speed (as low-bit integer operations are faster and more energy-efficient than floating-point operations).
  • Challenge: The main challenge is to perform this reduction without significant loss in model accuracy, which often happens due to quantization errors.

• Quantization Range & Outliers: In quantization, a continuous range of floating-point values is mapped to a discrete set of integer values. This mapping typically involves defining a quantization range (e.g., min to max value in a tensor) and then dividing this range into $2^N$ discrete steps, where $N$ is the bit-width.

  • Outliers are a few data points that are significantly different from the majority of the data. In LLMs, weights and activations often exhibit heavy-tailed distributions, meaning there are a few extreme outlier values far from the mean.
  • If these outliers are included in the quantization range, they force the min and max values to be very far apart. This widens the overall quantization range, making each quantization step larger. A larger step size means less precision for the vast majority of values clustered near the center of the distribution, leading to substantial quantization error and accuracy degradation.

• Data Distributions (Uneven, Heavy-tailed):

  • Uneven Distribution: Refers to data where values are not uniformly spread or symmetrically centered. In multidimensional data (like activation tensors, where each channel might have its own distribution), unevenness can also refer to significant disparities in variance or mean across different channels.
  • Heavy-tailed Distribution: A probability distribution where the tails (the extreme ends) are "heavier" than those of a normal (Gaussian) distribution. This means outliers occur more frequently or with larger magnitudes than expected in a Gaussian distribution. This characteristic is common in LLM activations, making quantization difficult.

• Linear Transformations: In linear algebra, a linear transformation is a function that maps vectors from one vector space to another, preserving vector addition and scalar multiplication. In neural networks, this typically involves multiplying a data vector or matrix by another matrix.

  • Linear transformations can rotate, scale, shear, or project data. The paper focuses on orthogonal transformations (rotations/reflections) and scaling transformations.

• Orthogonal Matrices & Stiefel Manifold:

  • An orthogonal matrix $\pmb{O}$ is a square matrix whose columns and rows are orthonormal vectors. This means $\pmb{O}^\top \pmb{O} = \pmb{O} \pmb{O}^\top = \pmb{I}$, where $\pmb{I}$ is the identity matrix and $\pmb{O}^\top$ is the transpose of $\pmb{O}$.
  • Orthogonal transformations preserve vector lengths and angles, meaning they are norm-preserving. They primarily rotate or reflect data without distorting its shape or relative distances. This property is crucial for quantization as it allows reshaping distributions without introducing unwanted scaling biases.
  • The Stiefel manifold $V_k(\mathbb{R}^n)$ is the set of all orthonormal $k$-frames in $\mathbb{R}^n$. In the context of orthogonal matrices, the Stiefel manifold represents the space of all possible orthogonal matrices. Optimizing parameters that must remain orthogonal requires specialized Riemannian optimization techniques that respect the geometric constraints of this manifold.

• Diagonal Matrices & Scaling Transformations:

  • A diagonal matrix $\pmb{\Lambda}$ is a square matrix where all entries outside the main diagonal are zero.
  • A scaling transformation applied by a diagonal matrix scales each dimension of the input data independently. If the diagonal entries are reciprocals, it acts as an inverse scaling. This is used to balance variances across different channels, making distributions more uniform.

• Kullback-Leibler (KL) Divergence: Also known as relative entropy, KL divergence is a non-symmetric measure of the difference between two probability distributions, $P$ and $Q$. It quantifies how much information is lost when $Q$ is used to approximate $P$.

  • Formula: $D_{KL}(P || Q) = \sum_i P(i) \log \left(\frac{P(i)}{Q(i)}\right)$
  • In quantization, KL divergence is often used as a loss function to ensure that the output distribution of the quantized model ($\hat{y}$) remains as close as possible to that of the full-precision model ($y$).

• Cross-Entropy Loss (CE Loss): A common loss function in classification tasks that measures the performance of a classification model whose output is a probability value between 0 and 1. It increases as the predicted probability diverges from the actual label.

  • Formula for single-class classification: $L_{CE} = -\sum_c y_c \log(\hat{y}_c)$, where $y_c$ is 1 if class $c$ is the true class and 0 otherwise, and $\hat{y}_c$ is the predicted probability for class $c$.
  • In LLMs, CE loss is typically used for next-token prediction. However, when training on small calibration datasets during PTQ, focusing solely on the single correct label might lead to overfitting and a loss of the model's broader semantic capabilities.

3.2. Previous Works

The paper contextualizes OSTQuant within the landscape of Post-Training Quantization (PTQ) for LLMs, broadly categorizing existing methods into weight-only and weight-activation quantization.

• Weight-Only Quantization: Aims to reduce memory by quantizing only the model weights, while activations remain in higher precision (e.g., FP16). This provides memory savings but less inference speedup compared to quantizing activations.

  • GPTQ (Frantar et al., 2022): Utilizes Hessian-based error compensation to minimize quantization errors layer-by-layer, achieving high compression rates. It specifically quantizes weights greedily by selecting optimal quantization values for each weight given the errors in previously quantized weights.
  • AWQ (Lin et al., 2023) and OWQ (Lee et al., 2023): These methods focus on the observation that activation outliers significantly impact the performance of weight quantization. They aim to protect salient weights or channels from aggressive quantization error by scaling weights or identifying specific channels that are robust to quantization.
  • QuIP (Chee et al., 2023) and QuIP# (Tseng et al., 2024): These approaches use random Hadamard matrices for incoherent processing of weights (making them more uniformly distributed) and apply vector quantization to weights, leading to better performance at very low precisions. Hadamard transformations are a type of orthogonal transformation.

• Weight-Activation Quantization: Aims to quantize both weights and activations (including the Key-Value (KV) cache in transformers) to achieve maximum memory savings and inference speedup. This is more challenging due to the dynamic nature and wide distributions of activations.

  • ZeroQuant (Yao et al., 2022): Proposes a fine-grained, hardware-friendly quantization scheme for both weights and activations, often utilizing per-channel and per-token quantization strategies.
  • SmoothQuant (Xiao et al., 2022): A pioneering smooth-based transformation method. It addresses activation quantization difficulty by mathematically shifting some of the quantization challenge from activations to weights, which are more static and easier to quantize. This is achieved by introducing a diagonal scaling matrix that smooths out large inter-channel variances in activations.
  • OmniQuant (Shao et al., 2023): Further enhances performance by actively training both quantization parameters (scales, zero-points) and transformation coefficients through an optimization process, often using block-wise reconstruction to minimize quantization error.
  • I-LLM (Hu et al., 2024): Focuses on achieving integer-only quantization and inference, which is highly efficient for specialized hardware. It uses fully-smooth block reconstruction and fully integer operators.
  • QuaRot (Ashkboos et al., 2024): Utilizes random rotation matrices (a type of orthogonal transformation) to suppress outliers in both weights and activations, enabling 4-bit quantization. The rotations are fixed (randomly initialized Hadamard-like matrices).
  • SpinQuant (Liu et al., 2024): Extends QuaRot by learning these rotation matrices to further refine 4-bit weight-activation quantization. This moves from fixed random rotations to optimized ones.

• Riemannian Optimization: Optimizing orthogonal matrices (which lie on a Stiefel manifold) requires specialized optimization techniques that respect their orthonormality constraints.

  • Cayley SGD (Li et al., 2020): Relies on iterative approximations of the Cayley Transform to optimize rotation matrices for arbitrary loss functions efficiently, primarily using matrix multiplication.
  • RAOM (Bécigneul & Ganea, 2018): Extends popular adaptive optimizers like ADAM (Kingma, 2014) and ADAGRAD to the realm of Riemannian optimization, making them suitable for Stiefel manifolds.
  • Geoopt (Kochurov et al., 2020): A Python library providing Riemannian optimization algorithms for various manifolds, facilitating their integration into deep learning models.

3.3. Technological Evolution

The field of LLM quantization has evolved from basic uniform quantization to sophisticated methods that adaptively handle the complex distributions of LLMs. Initially, quantization focused on simple per-tensor or per-channel scaling (RTN - Round-to-Nearest). The discovery of activation outliers in LLMs highlighted the need for more advanced techniques. Early methods like GPTQ and AWQ addressed weight quantization by mitigating outlier impact. Later, SmoothQuant introduced the idea of shifting quantization difficulty by scaling activations and inverse-scaling weights. More recent work, including QuaRot and SpinQuant, introduced orthogonal transformations (rotations) to directly reduce outliers and balance distributions in both weights and activations. OmniQuant pushed towards learnable quantization parameters and transformations.

OSTQuant builds upon this evolution by:

1. Introducing a principled metric (QSUR) to quantitatively guide the optimization of data distributions.
2. Combining the strengths of both orthogonal and scaling transformations in a learnable, equivalent transformation pair for global optimization.
3. Addressing the limited calibration data challenge with KL-Top loss.

3.4. Differentiation Analysis

OSTQuant differentiates itself from previous methods primarily through its holistic approach to optimizing data distribution across the entire quantization space, guided by a novel quantitative metric, and its specific combination of learnable transformations.

• From Heuristic to Principled:

  • Prior Methods (e.g., SmoothQuant, QuaRot, SpinQuant): These methods often operate based on empirical observations and heuristics (e.g., "smooth outliers," "rotate outliers"). While effective, they lack a unified metric to quantify "quantizability" or to rigorously guide the optimization process.
  • OSTQuant: Introduces QSUR as a theoretically grounded metric to directly measure how effectively data utilizes the quantization space. This metric provides a clear optimization target and allows for a more principled design of transformations, moving beyond pure heuristics.

• Combined Orthogonal and Scaling Transformations for Global Optimization:

  • Smooth-based Methods (e.g., SmoothQuant): Primarily use diagonal scaling matrices to reduce inter-channel variance in activations by shifting it to weights. They are effective at balancing scale but are sensitive to outliers and uneven mean values. Their optimization is typically localized (per-layer or per-channel) and doesn't explicitly optimize the overall shape of the distribution in the quantization space.
  • Rotation-based Methods (e.g., QuaRot, SpinQuant): Focus on orthogonal transformations (rotations) to suppress outliers in both weights and activations by rotating the data. QuaRot uses fixed random rotations, while SpinQuant learns them. These are good at making distributions more ball-shaped but might not optimally balance inter-channel variances or ensure maximum utilization of the quantization range. Their scope is often limited to reducing outliers rather than globally fitting the entire distribution.
  • OSTQuant: Combines the strengths of both. It employs learnable equivalent transformation pairs consisting of both an orthogonal transformation and a scaling transformation. This allows it to:
    1. Rotate the data to align its principal components with the quantization axes and reduce outlier impact (like rotation-based methods).
    2. Scale dimensions to balance variances and mean values across channels (like smooth-based methods), ensuring a flatter, more uniform distribution. This combined approach allows for a more comprehensive optimization of the data distribution across the entire quantization space, leading to higher QSUR and better precision.

• Equivalence Preservation and Fusion: OSTQuant explicitly designs equivalent transformation pairs that, in the absence of quantization, preserve the network's original output. This prevents overfitting to the calibration data and ensures better generalization. These transformations are then fused into the model weights during deployment, incurring no runtime overhead. While SmoothQuant also fuses scaling factors, OSTQuant extends this to orthogonal matrices as well.

• Addressing Limited Calibration Data:

  • Prior Methods: Often rely on standard cross-entropy loss or full KL divergence for optimization. Cross-entropy can lead to overfitting on small calibration sets, while full KL divergence can be noisy due to long-tail distributions in LLM vocabularies.

  • OSTQuant: Proposes KL-Top loss, which focuses KL divergence computation on only the top $k$ highest-probability logits. This provides more stable and semantically rich gradients for optimization with limited data.

    In essence, OSTQuant provides a more theoretically grounded, comprehensive, and adaptively optimized solution by integrating a novel metric, a powerful combination of learnable transformations, and a robust loss function specifically designed for PTQ challenges.

4. Methodology

4.1. Principles

The core principle behind OSTQuant is to enhance the quantizability of Large Language Models (LLMs) by actively reshaping the data distributions of weights and activations to better utilize the available quantization space. This is achieved through a novel metric, Quantization Space Utilization Rate (QSUR), which quantifies the effectiveness of this utilization, and a sophisticated learnable equivalent transformation mechanism.

The theoretical intuition is that quantization error is minimized when the data distribution is as compact and uniform as possible within the fixed quantization range. Uneven and heavy-tailed distributions lead to large quantization ranges and sparse quantization levels for most values. OSTQuant hypothesizes that a combination of orthogonal transformations (rotations) and scaling transformations can effectively transform these sub-optimal distributions into more quantization-friendly ones, characterized by higher QSUR.

Orthogonal transformations (rotations) can reduce the maximum extent of outliers by distributing their magnitude across multiple dimensions, making the data more "ball-shaped." Scaling transformations (diagonal matrices) can then balance variances across different channels, making the data distribution flatter and more uniform. By learning these transformations, OSTQuant aims to dynamically adapt to the specific distributions within each layer of an LLM, thereby minimizing quantization loss across the entire network.

The process is guided by minimizing the KL-Top loss, which ensures that the quantized model's output distribution remains semantically close to the full-precision model's output, even with limited calibration data, preventing overfitting and preserving performance.

4.2. Core Methodology In-depth (Layer by Layer)

4.2.1. Quantization Notations

The paper first defines the standard uniform quantization and dequantization process. This process maps a floating-point number to a discrete integer representation and then approximates the original floating-point number from that integer.

Given a floating-point tensor $\boldsymbol{X}$, the quantization function $\mathcal{Q}$ converts it into its quantized integer counterpart $X^I$. The dequantization process then approximates the original value as $X'$. The quantization process is defined by: $ X^I = \mathrm{clamp}\left(\left\lfloor \frac{X}{s} \right\rceil + zp^I, 0, 2^{n^I}-1\right) $ where:

• $X$: The original floating-point tensor.

• $X^I$: The quantized integer tensor, with values clamped between 0 and $2^{n^I}-1$.

• $n^I$: The number of bits for quantization (e.g., 4 bits for INT4).

• $s$: The quantization step size (also known as scale), which determines the width of each discrete quantization level.

• $zp^I$: The zero-point, an integer value that represents the floating-point value 0 in the quantized integer range.

• $\lfloor \cdot \rceil$: The rounding function (typically round-to-nearest).

• $\mathrm{clamp}(\cdot, \mathrm{min}, \mathrm{max})$: A function that clips values to be within the specified min and max range.

  The quantization step size $s$ and zero-point $zp^I$ are calculated from the minimum ($x_{\mathrm{min}}$) and maximum ($x_{\mathrm{max}}$) values observed in the floating-point tensor $\boldsymbol{X}$: $ s = \frac{x_{\mathrm{max}} - x_{\mathrm{min}}}{2^{n^I}-1} $ $ zp^I = \left\lfloor \frac{-x_{\mathrm{min}}}{s} \right\rceil $ After quantization, the dequantized floating-point approximation $\boldsymbol{X}'$ is computed as: $ \boldsymbol{X}' = (\boldsymbol{X}^I - zp^I) \cdot s $ This dequantization formula maps the integer back to a floating-point value within the original approximate range. The choice of $s$ (scale) and $zp^I$ (zero-point) is crucial for quantization accuracy. These can be determined statically from a few calibration samples (static quantization) or dynamically from runtime statistics (dynamic quantization). Quantization can also be applied per-channel or per-token, referring to the granularity at which $s$ and $zp^I$ are computed.

4.2.2. Quantization Space Utilization Rate (QSUR)

To quantitatively evaluate the quantizability of data and the effectiveness of transformations, the paper introduces QSUR.

Definition 1: Given a set of $d$-dimensional data $\boldsymbol{X} \in \mathbb{R}^{n \times d}$, let $V_X$ denote the hypervolume occupied by $\boldsymbol{X}$, and $V_{S_X}$ denote the hypervolume of the quantization space $S$ corresponding to $\boldsymbol{X}$. The quantization space $S_X$ is modeled as a hypercube whose edge lengths are defined by the maximum quantization range across all dimensions of $\boldsymbol{X}$. The Quantization Space Utilization Rate of $\boldsymbol{X}$ is then defined as: $ \mathrm{QSUR}X = \frac{V_X}{V{S_X}} \quad (1) $ A higher QSUR indicates that the data effectively fills the allocated quantization space, suggesting better quantizability.

For data typically following a Gaussian distribution ($X \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$), $V_X$ is calculated based on the ellipsoid formed by the covariance matrix $\boldsymbol{\Sigma}$ and mean vector $\boldsymbol{\mu}$. The covariance matrix can be diagonalized via eigenvalue decomposition: $\boldsymbol{\Sigma} = \boldsymbol{Q} \boldsymbol{\Lambda} \boldsymbol{Q}^\top$, where $\boldsymbol{Q}$ is a unit orthogonal matrix of eigenvectors, and $\boldsymbol{\Lambda} = \mathrm{diag}(\lambda_1, \lambda_2, ..., \lambda_d)$ contains the eigenvalues (representing variance along principal axes) in descending order.

The hypervolume of this ellipsoid at a given confidence level $\alpha$ (e.g., $\alpha = 0.99$) is given by: $ V_X = \frac{\pi^{d/2}}{\Gamma(d/2+1)} \times \left(\chi_d^2(\alpha)\right)^{d/2} \times \sqrt{\operatorname*{det}(\boldsymbol{\Sigma})} \quad (2) $ where:

• $\Gamma$: The Gamma function.

• $\chi_d^2(\alpha)$: The chi-squared quantile for $d$ degrees of freedom at confidence level $\alpha$. This factor scales the ellipsoid to encompass a certain percentage of the data.

• $\operatorname*{det}(\boldsymbol{\Sigma})$: The determinant of the covariance matrix. Since $\boldsymbol{Q}$ is orthogonal, $\operatorname*{det}(\boldsymbol{\Sigma}) = \operatorname*{det}(\boldsymbol{Q} \boldsymbol{\Lambda} \boldsymbol{Q}^\top) = \operatorname*{det}(\boldsymbol{Q}) \operatorname*{det}(\boldsymbol{\Lambda}) \operatorname*{det}(\boldsymbol{Q}^\top) = \operatorname*{det}(\boldsymbol{\Lambda}) = \prod_{i=1}^{d} \lambda_i$. Thus, $\sqrt{\operatorname*{det}(\boldsymbol{\Sigma})} = \sqrt{\prod_{i=1}^{d} \lambda_i}$.

  The volume of the quantization hypercube ($V_{S_X}$) is determined by the range of the distribution along each principal axis, considering the maximum and minimum coordinate values. The extremal points of the ellipsoid correspond to these maximum and minimum values. Let $\lambda_{\mathrm{max}}$ and $\lambda_{\mathrm{min}}$ be eigenvalues, and $\pmb{q}_{\mathrm{max}}$ and $\pmb{q}_{\mathrm{min}}$ be corresponding eigenvectors (though the paper simplifies this by later considering only the largest eigenvalue). The maximum and minimum coordinate values $v_{\mathrm{max}}^{\mathrm{org}}$ and $v_{\mathrm{min}}^{\mathrm{org}}$ are given as: $ v_{\mathrm{max}}^{\mathrm{org}} = \sqrt{\chi_d^2(\alpha) \cdot \lambda_{\mathrm{max}}} \cdot \pmb{q}{\mathrm{max}} + \boldsymbol{\mu} $ $ v{\mathrm{min}}^{\mathrm{org}} = -\sqrt{\chi_d^2(\alpha) \cdot \lambda_{\mathrm{min}}} \cdot \pmb{q}{\mathrm{min}} + \boldsymbol{\mu} $ The volume of the quantization hypercube is then: $ V{S_X} = (\operatorname*{max}(v_{\mathrm{max}}^{\mathrm{org}}) - \operatorname*{min}(v_{\mathrm{min}}^{\mathrm{org}}))^d \quad (5) $ Combining these, the QSUR expression is: $ \mathrm{QSUR}X = \frac{ \frac{\pi^{d/2}}{\Gamma(d/2+1)} \cdot \left(\chi_d^2(\alpha)\right)^{d/2} \cdot \sqrt{\operatorname*{det}(\boldsymbol{\Lambda})} }{ \left( \operatorname*{max}(\sqrt{\chi_d^2(\alpha) \cdot \lambda{\mathrm{max}}} \cdot |\pmb{q}{\mathrm{max}}| + \boldsymbol{\mu}) - \operatorname*{min}(\sqrt{\chi_d^2(\alpha) \cdot \lambda{\mathrm{min}}} \cdot |\pmb{q}_{\mathrm{min}}| + \boldsymbol{\mu}) \right)^d } \quad (6) $ Simplified QSUR: The paper further simplifies this by neglecting the mean vector $\boldsymbol{\mu}$ (assuming its magnitude is small relative to the largest eigenvalue) and assuming $\lambda_{\mathrm{max}} = \lambda_{\mathrm{min}} = \lambda_1$ (the largest eigenvalue) for extremal points. This simplification leads to: $ \mathrm{QSUR}X = \frac{ \frac{\pi^{d/2}}{\Gamma(d/2+1)} \cdot \sqrt{\prod{i=1}^{d} \lambda_i} }{ 2^d \left( \operatorname*{max}(\sqrt{\lambda_1} \cdot \pmb{q}_1) \right)^d } \quad (7) $ From this simplified QSUR formula (Equation 7), two key observations are made:

1. QSUR is proportional to the product of the ratios of each eigenvalue $\lambda_i$ to the largest eigenvalue $\lambda_1$. This implies that QSUR is maximized when all eigenvalues are similar (i.e., the distribution is more ball-shaped).

2. The maximum component of the eigenvector $\pmb{q}_1$ corresponding to $\lambda_1$ is inversely proportional to QSUR. QSUR is minimized when this component is large, and maximized when it is small. The denominator in Equation 7 is minimized when the components of $\pmb{q}_1$ are $\pm d^{-1/2}$ (Appendix A.2.1), which corresponds to the most uniform spread of eigenvector components.

   Influence of linear transformation on QSUR: Applying a linear transformation $\pmb{T}$ to $X \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ results in a transformed distribution $\hat{X} \sim \mathcal{N}(\hat{\boldsymbol{\mu}}, \hat{\boldsymbol{\Sigma}})$, where $\hat{\boldsymbol{\mu}} = \pmb{T}\boldsymbol{\mu}$ and $\hat{\boldsymbol{\Sigma}} = \pmb{T} \boldsymbol{Q} \boldsymbol{\Lambda} \boldsymbol{Q}^\top \pmb{T}^\top$.

• Smoothing-based approaches (e.g., SmoothQuant) treat $\pmb{T}$ as a diagonal matrix that scales variances, reducing disparities among eigenvalues $\lambda_i$. However, they are sensitive to outliers and uneven mean values.

• Rotation-based methods (e.g., QuaRot, SpinQuant) use orthogonal matrices to modify $\boldsymbol{Q}$, reducing outliers and the hypercube volume, thereby increasing QSUR. This ability improves with dimensionality (Appendix A.2.2). The best outlier reduction is achieved when the orthogonal matrix is $\pmb{T} = d^{-\frac{1}{2}} \pmb{H} \pmb{Q}^\top$, where $\pmb{H}$ is a Hadamard matrix.

  The Best Transformation Matrix: Combining the insights from Equation 7 and the influence of transformations, the paper deduces that the maximum QSUR is achieved when the transformation matrix is: $ \pmb{T} = c \cdot \pmb{\Lambda}^{-\frac{1}{2}} \pmb{Q}^\top \quad (8) $ where $c$ is an arbitrary scalar. This transformation effectively spheres the data, making its covariance matrix an identity matrix ($\hat{\boldsymbol{\Sigma}} = \boldsymbol{I}$, see Appendix A.2.3). At this point, the maximum QSUR is achieved: $ \mathrm{QSUR}'' = \frac{ \frac{\pi^{d/2}}{\Gamma(d/2+1)} }{ 2^d } \quad (28) $ This theoretical derivation highlights that an ideal transformation involves both scaling (by $\pmb{\Lambda}^{-\frac{1}{2}}$) to normalize variances and rotation (by $\pmb{Q}^\top$) to align with principal axes, effectively making the data distribution isotropic (uniform in all directions).

4.2.3. Orthogonal and Scaling Transformation-based Quantization (OSTQuant)

OSTQuant leverages these theoretical insights to design a practical PTQ framework. It uses learnable equivalent transformation pairs within LLM layers to optimize weight and activation distributions for improved quantization performance.

Overall Flow (Figure 5): OSTQuant applies multiple transformation pairs globally across and within LLM blocks. Four equivalent transformation pairs are learned per block, each comprising a learnable diagonal scaling matrix $\pmb{S}$ (denoted as $\pmb{\Lambda}$ in formulas) and a learnable orthogonal matrix $\pmb{O}$ (denoted as $\pmb{R}$ in figures/text for rotation).

Equivalent Transformation Pair: A transformation pair is defined as $\pmb{T} = \pmb{\Lambda} \pmb{O}$, where $\pmb{\Lambda}$ is a diagonal scaling matrix and $\pmb{O}$ is a unit orthogonal matrix. The forward inference process for a pair of matrix multiplications (e.g., $W_1 W_2$) is reformulated as: $ \pmb{y} = \mathcal{Q}(\pmb{x} W_1 O \pmb{\Lambda}) \mathcal{Q}(\pmb{\Lambda}^{-1} O^T W_2) \quad (9) $ where:

• $\pmb{x}$: Input to the first linear layer.
• $W_1, W_2$: Weights of two consecutive linear layers.
• $\pmb{O}$: Learnable orthogonal matrix.
• $\pmb{\Lambda}$: Learnable diagonal scaling matrix.
• $\mathcal{Q}(\cdot)$: The quantization operation. The intermediate product $\pmb{O} \pmb{\Lambda}$ transforms the output of $W_1$, and its inverse $\pmb{\Lambda}^{-1} \pmb{O}^\top$ transforms the input of $W_2$. This maintains mathematical equivalence in full-precision, but allows for optimization of intermediate distributions during quantization.

Advantages of Equivalent Transformation Pair:

1. Learnability and Computational Efficiency: Both $\pmb{O}$ and $\pmb{\Lambda}$ are learnable parameters. The inverse of a diagonal matrix $\pmb{\Lambda}$ is trivial (reciprocal of diagonal elements). The orthogonal matrix $\pmb{O}$ is optimized using Riemannian optimizers like RiemannAdam, leveraging first-order gradient information.

2. Equivalence Preservation: Without quantization, the transformations cancel out ($\pmb{O} \pmb{\Lambda} \pmb{\Lambda}^{-1} \pmb{O}^\top = \pmb{I}$), ensuring the network's output is identical to the original model. This reduces the risk of overfitting to the small calibration set.

3. No Runtime Overhead: After optimization, $\pmb{O}$ and $\pmb{\Lambda}$ are fused directly into the existing weight matrices. For example, $W_1 \leftarrow W_1 \pmb{O} \pmb{\Lambda}$ and $W_2 \leftarrow \pmb{\Lambda}^{-1} \pmb{O}^\top W_2$. This means no additional computational cost or parameters are introduced during inference.

   The overall optimization objective for the network is to minimize the loss between the quantized network output $\hat{y}$ and the full-precision output $y$, by learning the optimal scaling ($\pmb{A}_i$) and orthogonal ($\pmb{O}_i$) matrices: $ \arg \operatorname*{min}_{A_i, O_i} \mathcal{L}(\hat{y}, y; A_i, O_i, \theta) \quad (10) $ where $\theta$ represents the frozen full-precision network parameters.

Weight Outlier Minimization Initialization (WOMI): Since weights often follow a Gaussian distribution with zero mean, the orthogonal matrix $\pmb{O}$ is initialized to optimally reduce weight outliers. For a global orthogonal matrix $R_{res}$, weights of all linear layers receiving residual inputs are concatenated. Eigenvalue decomposition is performed on their covariance matrix to obtain the eigenmatrix $\boldsymbol{Q}_W$. Then, $\pmb{O}$ is initialized as $\pmb{O} = \pmb{H} \boldsymbol{Q}_W^\top$, where $\pmb{H}$ is a normalized Hadamard matrix (Equation 27 from Appendix A.2.2 is $\pmb{T} = d^{-\frac{1}{2}} \pmb{H} \pmb{Q}^\top$, suggesting a similar structure for initialization). This initialization leverages Hadamard matrices' ability to spread out values and covariance matrices to align with principal components, minimizing outliers and inter-channel disparities. For scaling matrices, they are initialized as identity matrices.

Inter-Block Learning (Global Transformations): As shown in the upper part of Figure 5, a global orthogonal transformation $R_{res}$ is applied at the embedding layer and propagates through each residual path in the LLM. This transformation rotates activations across the entire residual path. Due to the norm-preserving property of unitary orthogonal matrices, this allows bypassing RMSNorm layers (which normalize based on norm) and applying an inverse transformation to inputs of projection layers and the final output head to maintain equivalence. Additionally, for the two normalization layers in each block, and their respective projection layers ($S_{attn}^i$ and $S_{ffn}^i$), scaling matrices are applied. The orthogonal matrix $R_{res}$ rotates activations along residual paths and adjusts weights of multiple projection layers. The scaling matrices $S_{attn}^i$ and $S_{ffn}^i$ scale the outputs of RMSNorm layers and corresponding projection layers. These transformations are absorbed into the corresponding weight matrices, optimizing for distribution shifts and mitigating quantization errors.

Intra-Block Learning (Local Transformations): The lower part of Figure 5 illustrates local equivalent transformation pairs within each transformer block:

1. Multi-Head Self-Attention (MHSA) Layer: Two pairs are introduced.

   • Value (V) and Output (O) Projection Layers: For each attention head $h$, the data flow is: $ \pmb{Y} = \pmb{P}^h \cdot \pmb{X}^h \cdot (\pmb{W}v^h \pmb{R}{ov}^h \pmb{S}{ov}^h) \cdot (\pmb{S}{ov}^{h^{-1}} \pmb{R}_{ov}^{h^T} \pmb{W}_o^h) \quad (11) $ Here:
     • $h$: Head index.
     • $\pmb{P}^h$: Attention matrix for head $h$.
     • $\pmb{X}^h$: Input to head $h$.
     • $\pmb{W}_v^h$: Weight for value projection.
     • $\pmb{W}_o^h$: Weight for output projection.
     • $\pmb{R}_{ov}^h$: Learnable orthogonal matrix for V/O layers.
     • $\pmb{S}_{ov}^h$: Learnable scaling matrix for V/O layers. These transformations improve QSUR for both the value cache and output projection layer.
   • Query (Q) and Key (K) Projection Layers: After Rotary Positional Encoding (ROPE), the query and key outputs undergo an equivalent scaling transformation $S_{qk}$, which can be incorporated into weights $W_q$ and $W_k$. An additional Hadamard transformation is applied to $Q$ and $K$ outputs to enhance key cache quantization.

2. Feed-Forward Network (FFN) Layer: The diagonal matrices in the up-projection and down-projection layers of the FFN are optimized. The inverse of the Hadamard transformation (if applied) is fused into $W_{down}$ from the start.

4.2.4. KL-Top Loss

While LLMs are trained on massive datasets, PTQ optimization is typically performed on small calibration sets (e.g., 1000 samples). In this limited data regime:

• Standard cross-entropy (CE) loss can lead to overfitting, where the model performs well on the calibration set but degrades on broader tasks (as shown in Table 1). This is because small datasets may not fully utilize LLM capacity, causing CE loss (focusing on a single correct label) to overfit to narrow features.

• Full Kullback-Leibler (KL) divergence (comparing the full prediction distributions) is also problematic. LLMs have very large vocabularies (e.g., LLaMA-3-8B has over 100,000 tokens), and their prediction results often follow a severe long-tail distribution (Figure 6), meaning only a few tokens have significant probabilities. Directly applying KL divergence over all classes means the loss can be dominated by uninformative low-probability classes, adding noise to training and incurring high computational/memory costs.

  To mitigate these issues, OSTQuant proposes the KL-Top loss function. This function computes KL divergence over only the top $k$ classes with the highest probabilities. This approach focuses optimization on the model's primary predictions, enhancing gradient quality by avoiding noise from low-probability values. It also reduces computational and memory costs for large vocabularies.

The KL-Top loss is calculated as follows: $ idxs = \operatorname*{top}k(z) \quad (12) $ $ \mathcal{L} = \sum_{i \in idxs} z[i] \log \left( \frac{z[i]}{\hat{z}[i]} \right) $ where:

• $z$: The prediction distribution (logits, usually after softmax) from the full-precision model.

• $\hat{z}$: The prediction distribution from the quantized model.

• $\operatorname*{top}k(z)$: A function that returns the indices of the top $k$ highest-probability logits from $z$.

• $\mathcal{L}$: The KL-Top loss. The sum is taken only over the selected top $k$ indices.

  Experiments (Table 6) show that a $k$ value of 1,000 typically yields the best results, balancing semantic richness with optimization stability.

5. Experimental Setup

5.1. Datasets

The experimental validation of OSTQuant is conducted across a range of Large Language Models (LLMs) from the LLaMA family and evaluated on standard benchmarks.

• Models:

  • LLaMA-1 (7B, 13B, 30B) (Touvron et al., 2023a)
  • LLaMA-2 (7B, 13B, 70B) (Touvron et al., 2023b)
  • LLaMA-3-8B, LLaMA-3-70B

• Calibration Data: For the PTQ optimization phase, a small calibration set is used.

  • Source: 1,000 samples from WikiText2 (Merity et al., 2016)
  • Characteristics: Each sample has a token length of 2,048. WikiText2 is a common dataset for language model evaluation, particularly for perplexity measurements, as it is derived from high-quality Wikipedia articles.
  • Why chosen: It's a standard and relatively clean dataset suitable for calibrating quantization parameters due to its representative linguistic patterns, while being small enough to fit the PTQ paradigm of limited data.

• Evaluation Datasets (Zero-Shot Tasks): To assess the generalized performance and emergent capabilities of the quantized LLMs, the models are evaluated on a suite of nine zero-shot tasks using the lm-evaluation-harness (version 0.4.4) (Gao et al., 2024). These tasks are designed to test various aspects of language understanding and reasoning.

  • BoolQ (Clark et al., 2019): Boolean Question Answering.
  • HellaSwag (Zellers et al., 2019): Commonsense reasoning for sentence completion.
  • LAMBADA (Radford et al., 2019): Language Modeling Broadened Across Diverse Activities, testing language understanding by predicting the last word of a sentence.
  • OpenBookQA (OBQA) (Mihaylov et al., 2018): Open-book question answering requiring commonsense and factual knowledge.
  • PIQA (Bisk et al., 2020): Physical Interaction Question Answering, focusing on physical commonsense.
  • SIQA (Sap et al., 2019): Social IQa, evaluating commonsense reasoning about social interactions.
  • WinoGrande (Sakaguchi et al., 2021): An adversarial Winograd Schema Challenge to test commonsense reasoning.
  • ARC-Easy and ARC-Challenge (Boratko et al., 2018): AI2 Reasoning Challenge, testing scientific reasoning.
  • Why chosen: These datasets represent a diverse set of zero-shot NLP tasks and are standard benchmarks for evaluating the capabilities and robustness of LLMs after quantization. They are crucial because perplexity alone (from WikiText2) might not fully reflect the true functional performance of the model, especially after quantization.

5.2. Evaluation Metrics

For comprehensive evaluation, OSTQuant uses both perplexity and zero-shot accuracy (averaged across multiple tasks), along with inference speedup and memory savings.

• Perplexity (PPL)

  • Conceptual Definition: Perplexity is a common metric for evaluating language models. It measures how well a probability distribution or language model predicts a sample. A lower perplexity score indicates that the model predicts the test set more accurately and assigns higher probabilities to the observed sequences, meaning it is a better language model. In simple terms, it's a measure of how "surprised" the model is by new data; less surprise means better performance.
  • Mathematical Formula: For a sequence of words $W = (w_1, w_2, \ldots, w_N)$, the perplexity is defined as: $ PPL(W) = \exp\left(-\frac{1}{N} \sum_{i=1}^N \log P(w_i | w_1, \dots, w_{i-1})\right) $
  • Symbol Explanation:
    • $W$: The test corpus or sequence of words.
    • $N$: The total number of words in the test corpus.
    • $P(w_i | w_1, \dots, w_{i-1})$: The probability assigned by the language model to the $i$-th word $w_i$, given the preceding words $w_1, \dots, w_{i-1}$. This probability is typically computed by the LLM during its next-token prediction task.

• Zero-Shot Accuracy (Avg. Zero-Shot Score)

  • Conceptual Definition: Zero-shot accuracy measures a model's ability to perform tasks it has not been explicitly trained on, demonstrating its generalization capabilities and emergent properties. The average zero-shot score refers to the mean accuracy across a suite of predefined zero-shot tasks. For each task, accuracy is simply the percentage of correct predictions.
  • Mathematical Formula: Since this metric is an average across multiple tasks, there isn't a single universal formula beyond the definition of accuracy for each task. For an individual task, Accuracy is: $ \text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}} \times 100% $ The Average Zero-Shot Score is then: $ \text{Avg. Zero-Shot Score} = \frac{1}{M} \sum_{j=1}^M \text{Accuracy}_j $
  • Symbol Explanation:
    • $\text{Accuracy}_j$: The accuracy achieved on the $j$-th zero-shot task.
    • $M$: The total number of zero-shot tasks evaluated (in this paper, $M=9$).

• Speedup

  • Conceptual Definition: Speedup quantifies how much faster the quantized model performs inference compared to its full-precision (e.g., FP16) counterpart. A speedup of $2 \times$ means the quantized model is twice as fast.
  • Mathematical Formula: $ \text{Speedup} = \frac{\text{Inference Time (Full Precision)}}{\text{Inference Time (Quantized)}} $
  • Symbol Explanation:
    • $\text{Inference Time (Full Precision)}$: The time taken for a specific inference operation (e.g., prefill, decoding a token) using the full-precision model.
    • $\text{Inference Time (Quantized)}$: The time taken for the same inference operation using the quantized model.

• Memory Saving Factor

  • Conceptual Definition: Memory saving factor indicates how much less memory the quantized model consumes compared to its full-precision version. A factor of $3 \times$ means the quantized model uses one-third of the memory.
  • Mathematical Formula: $ \text{Memory Saving Factor} = \frac{\text{Memory Usage (Full Precision)}}{\text{Memory Usage (Quantized)}} $
  • Symbol Explanation:
    • $\text{Memory Usage (Full Precision)}$: The memory consumed by the full-precision model (e.g., for storing weights or activations).
    • $\text{Memory Usage (Quantized)}$: The memory consumed by the quantized model.

5.3. Baselines

OSTQuant is benchmarked against several representative Post-Training Quantization (PTQ) methods, ranging from basic approaches to state-of-the-art techniques.

• RTN (Round-to-Nearest): A basic quantization method where floating-point values are simply rounded to the nearest available integer quantization level. It serves as a strong baseline to show the benefits of more sophisticated methods over naive quantization.
• SmoothQuant (Xiao et al., 2022): A smooth-based transformation method that shifts quantization difficulty from activations to weights by scaling activations and inversely scaling weights. This aims to reduce inter-channel variance in activations.
• GPTQ (Frantar et al., 2022): An accurate weight-only PTQ method that uses Hessian-based error compensation to minimize quantization errors in generative pre-trained transformers.
• OmniQuant (Shao et al., 2023): A method that further enhances quantization performance by training quantization parameters (scales and zero-points) and transformation coefficients through an optimization process, often involving block-wise reconstruction.
• AWQ (Lin et al., 2023): Activation-aware weight quantization, which focuses on protecting salient weights (those critical for model performance) from quantization errors by identifying activation outliers and applying appropriate scaling.
• QuaRot (Ashkboos et al., 2024): A rotation-based method that uses random rotation matrices to make activations and weights more uniform and outlier-free, enabling 4-bit quantization for both.
• SpinQuant (Liu et al., 2024): An advancement over QuaRot that learns the rotation matrices for 4-bit weight-activation quantization rather than relying on random ones, aiming for better optimization.

5.4. Implementation Details

The paper specifies key aspects of OSTQuant's implementation and optimization:

• Quantization Scheme:
  • Activations: Per-token asymmetric quantization without any pruning operations. Per-token means scales and zero-points are calculated for each token in a sequence, allowing adaptation to dynamic ranges. Asymmetric means the quantization range does not necessarily center around zero.
  • Weights: Per-channel symmetric quantization. Per-channel means scales and zero-points are calculated independently for each output channel of a weight tensor. Symmetric means the quantization range is centered around zero (e.g., from -Max to +Max).
• Optimizer: RiemannAdam (Bécigneul & Ganea, 2018) is used to optimize all unit orthogonal matrices and scaling matrices. RiemannAdam is suitable for Riemannian optimization on Stiefel Manifolds, where orthogonal matrices reside.
• Calibration Data and Iterations:
  • Calibration Samples: 1,000 samples from WikiText2.
  • Token Length: Each sample has a token length of 2,048.
  • Iterations: 150 iterations.
  • Batch Size: 8.
• Learning Rate Schedule: Cosine learning rate decay is applied.
  • Initial Learning Rate for orthogonal matrix parameters: $2 \times 10^{-2}$.
  • Initial Learning Rate for scaling parameters: $3 \times 10^{-2}$.
  • The paper notes that using a learning rate for the Stiefel manifold (orthogonal matrices) 10 times larger than that for scaling parameters typically leads to better results.
• Hardware: Quantization and inference speed tests were conducted on an A800 GPU for optimization timing, and on a 3090 GPU for detailed prefill/memory speedup tests (Table 9), and an A6000 GPU for decoding speed (Table 10).
• Kernel Implementation: The 4-bit matrix multiplication kernel was implemented using CUTLASS of Nvidia, a high-performance library for CUDA. The self-attention mechanism used PyTorch's native SDPA (scaled dot product attention) function.

6. Results & Analysis

6.1. Core Results Analysis

OSTQuant consistently outperforms existing state-of-the-art methods across various LLMs and quantization configurations, demonstrating its effectiveness in preserving accuracy at low bit-widths and achieving significant efficiency gains.

Quantization Performance (Table 2):

• W4-only (4-16-16) Setting: OSTQuant achieves remarkable accuracy retention, maintaining at least 99.5% of the floating-point (FP) accuracy in zero-shot tasks. For the challenging LLaMA-3-8B model, OSTQuant has only a 0.29-point performance drop (67.80 vs FP 68.09), significantly outperforming other methods that incur losses exceeding 1.55 points. This demonstrates its superior ability to quantize weights with minimal impact.

• W4A4KV16 Setting (4-4-16): When activations are also quantized but the KV cache remains at FP16, OSTQuant shows substantial performance boosts. For instance, on LLaMA-2-7B, OSTQuant (63.90) outperforms SpinQuant (57.37) by 6.53 points in zero-shot accuracy. This highlights its effectiveness in handling activation distributions.

• W4A4KV4 Setting (4-4-4): In this most aggressive 4-bit quantization configuration for weights, activations, and KV cache, OSTQuant still maintains significant performance gains. It outperforms the previous SOTA method, SpinQuant, by approximately 1 point across multiple models (e.g., LLaMA-3-8B: 65.37 vs 64.10, LLaMA-2-7B: 63.18 vs 62.01). This demonstrates its robustness in extreme compression scenarios, making 4-bit inference truly feasible.

• Comparison to Smooth-based vs. Rotation-based: The results clearly show that rotation-based methods (like QuaRot, SpinQuant, OSTQuant) significantly outperform smooth-based methods (like SmoothQuant) once activations are quantized. This confirms the paper's argument that smooth-based methods struggle more with the prominent outliers and uneven distributions in activations.

  QSUR and Accuracy Correlation (Figure 3): The evaluation of QSUR across different quantization methods for LLaMA variants reveals a clear positive correlation with model performance (Zero-Shot accuracy). OSTQuant achieves the highest QSUR, which directly translates to improved model accuracy. This validates the QSUR metric as an effective indicator of quantizability and the success of OSTQuant in optimizing data distributions.

Activation Distribution Uniformity (Figure 2, 11, 12): Visualizations (Figure 2) show that LLaMA-3-8B's activation distributions prior to OSTQuant exhibit substantial variation across channels and numerous outliers. After OSTQuant's transformations, the distributions become significantly more uniform across channels and outliers are mitigated. This qualitative observation supports the quantitative QSUR improvements and performance gains.

Speedup and Memory Savings (Table 3, 9, 10):

• Inference Speedup: OSTQuant enables 4-bit inference that delivers an average inference speedup of over $2 \times$ compared to FP16. For larger models like LLaMA-30B, it achieves approximately $3 \times$ acceleration. This is attributed to reduced memory access overhead and efficient low-precision computation units.
• Memory Savings: OSTQuant provides memory savings exceeding $3.5 \times$. This is crucial for deploying LLMs on resource-constrained devices or running larger models on existing hardware.
• Decoding Stage Efficiency (Table 10): In the decoding stage (which is often memory-bound), quantizing to 4-bit significantly reduces memory access overhead. This allows running very large models like LLaMA-3-70B on a single A6000 GPU at a respectable speed of 15 tokens per second, which would be Out of Memory (OOM) for FP16.
• Training Speed (Table 11): OSTQuant also offers substantial advantages in optimization speed compared to block reconstruction-based methods (like OmniQuant). With only 150 iterations and minimal learnable parameters, it can quantize smaller models (7B) in around 20 minutes (a $5.3 \times$ speedup over OmniQuant) and larger models (30B) in 120 minutes (a $3.3 \times$ speedup).

6.2. Data Presentation (Tables)

The following are the results from Table 1 of the original paper:

The following are the results from Table 2 of the original paper:

The following are the results from Table 3 of the original paper:

The following are the results from Table 4 of the original paper:

The following are the results from Table 5 of the original paper:

The following are the results from Table 6 of the original paper:

The following are the results from Table 7 of the original paper:

The following are the results from Table 8 of the original paper:

The following are the results from Table 9 of the original paper:

The following are the results from Table 10 of the original paper:

The following are the results from Table 11 of the original paper:

The following are the results from Table 12 of the original paper:

The following are the results from Table 13 of the original paper:

The following are the results from Table 14 of the original paper:

6.3. Ablation Studies / Parameter Analysis

6.3.1. Effect of Different Transformation Matrices (Table 4)

The ablation study investigates the contribution of different learnable transformation matrices on LLaMA-2 7B under W4A4KV4 quantization.

• Baseline (Zero-Shot9 33.51): This represents the performance without any of the proposed transformations, likely using a basic RTN or SmoothQuant-like approach, resulting in poor accuracy.

• $+Rres$ (Zero-Shot9 54.33, Wiki PPL 9.70): Adding the global orthogonal transformation $R_{res}$ brings the most significant improvement (over 20 points in Zero-Shot9 accuracy). This highlights the critical role of rotating activations globally across the residual paths to manage distributions.

• $+Sres$ (Zero-Shot9 53.74, Wiki PPL 9.46): This refers to the global scaling transformation $S_{res}$. While the paper indicates $R_{res}$ provides the largest single improvement, $S_{res}$ also shows a positive effect, suggesting the importance of global scaling, though less impactful than global rotation. The specific combination is crucial.

• $+Rdn$ (Zero-Shot9 61.75, Wiki PPL 6.16): Adding the orthogonal transformation $R_{down}$ (likely related to FFN down-projection or similar intra-block rotation) provides the next largest jump in performance. This indicates that local rotations within critical layers are also highly effective.

• $+Su|d$ (Zero-Shot9 61.79, Wiki PPL 6.00): This represents scaling transformations for FFN up/down-projections. It offers a slight improvement over $+Rdn$, confirming that scaling helps balance variances across channels after rotation.

• $+Rqk$ (Zero-Shot9 62.35, Wiki PPL 5.92): The Hadamard transformation (referred to as Rqk for Query/Key rotation) applied after ROPE further improves accuracy.

• $+Sqk$ (Zero-Shot9 62.56, Wiki PPL 5.92): The scaling transformation $S_{qk}$ for Query/Key also contributes.

• $+Rov$ (Zero-Shot9 63.11, Wiki PPL 5.94): Orthogonal transformation for Value/Output projections in MHSA significantly boosts performance.

• $+Sov$ (Zero-Shot9 63.18, Wiki PPL 5.91): The scaling transformation for Value/Output projections provides the final marginal improvement.

  Conclusion: The results demonstrate that orthogonal transformations (especially global ones like $R_{res}$ and local ones like $R_{dn}$, $R_{ov}$) provide the most substantial gains by reshaping the distributions. Scaling transformations ($S_{res}$, $S_{u|d}$, $S_{qk}$, $S_{ov}$) then build upon these by finely balancing variance across channels, minimizing residual quantization losses and maximizing QSUR. The cumulative effect of all these learnable transformations is critical for OSTQuant's strong performance.

6.3.2. Different Manifold Optimizers (Table 5)

The paper compares different Riemannian optimizers for the unit orthogonal matrices (which reside on a Stiefel manifold) on LLaMA-2-7B and LLaMA-2-13B under W4A4KV4 configuration.

• Cayley SGD (Stochastic Gradient Descent): Requires a relatively high learning rate (LR1 1.50, LR2 0.20) and achieves good results (63.11 for 7B, 64.77 for 13B) within 150-200 steps.

• Riemann SGD: Needs more iterations (500 steps) and lower learning rates (LR1 0.10, LR2 0.02) to reach similar or slightly worse performance (63.09 for 7B, 65.19 for 13B).

• Riemann Adam: Delivers the best results (63.18 for 7B, 65.41 for 13B) with the fewest iterations (150 steps) and lower learning rates (LR1 0.02, LR2 1e-3 or 0.002). This highlights RiemannAdam's efficiency and effectiveness in optimizing on Stiefel manifolds.

  Learning Rate Ratio: A key finding is that setting the learning rate for the Stiefel manifold (orthogonal matrices, LR2) approximately 10 times larger than that for scaling transformation parameters (LR1) leads to better results. This suggests that orthogonal transformations might require larger steps or adapt faster during optimization.

6.3.3. Influence of $k$ in KL-Top Loss (Table 6)

The parameter $k$ in KL-Top loss (Equation 12) determines the number of top-probability classes considered for KL divergence calculation. The ablation study explores the impact of different $k$ values on LLaMA-2 7B's Zero-Shot9 score and Wiki PPL under W3-only and W4A4KV4 settings.

• Balanced $k$ is Key: Both excessively small ($k=5$, $k=50$) or excessively large ($k=5000$, $k=10000$) values for $k$ negatively impact optimization.
  • Small $k$: May discard too much semantic information, leading to sub-optimal optimization.
  • Large $k$: Reintroduces noise from long-tail distributions and increases computational cost, similar to full KL divergence.
• Optimal $k$ Value: For both W3-only and W4A4KV4 settings, $k=1000$ consistently yields the best Zero-Shot9 scores (62.30 and 63.18 respectively). This $k$ value appears to strike the right balance between retaining semantic information and mitigating noise.
• PPL Stability: Wiki PPL shows less sensitivity to $k$ compared to Zero-Shot scores, indicating that Zero-Shot tasks are a more robust indicator of model performance after quantization.

6.3.4. Effect of Weight Outlier Minimization Initialization (WOMI) (Table 7, Figure 7, 13)

WOMI is designed to initialize trainable orthogonal matrices by leveraging eigenvalue decomposition of weights and Hadamard matrices.

• Performance Improvement (Table 7): WOMI consistently achieves lower perplexity and higher few-shot accuracy compared to random Hadamard initialization for LLaMA-2-7B and LLaMA-3-8B under both W4A16KV16 and W4A4KV4 configurations. This confirms its effectiveness in providing a better starting point for optimization.
• Enhanced W4-only Performance: Notably, WOMI shows greater performance improvements in W4-only quantization settings compared to W4A4KV4. This suggests that WOMI is particularly effective at minimizing weight quantization errors, which is crucial when weights are the primary quantized component.
• Visualization (Figure 7, 13): Visualizations illustrate that original weight distributions often have significant variations across input/output channels. While Hadamard transformations reduce some of these differences, WOMI (by incorporating the covariance matrix) further smooths inter-channel disparities and minimizes outliers. This leads to a more compact distribution within the quantization space and reduced relative quantization error.

6.3.5. Effect of KL-Top Loss with SpinQuant (Table 8)

This ablation investigates the impact of KL-Top loss when applied to SpinQuant versus OSTQuant.

• SpinQuant with KL-Top: When KL-Top loss is introduced to SpinQuant alone, it does not lead to significant performance improvements and may even cause slight degradation (Avg. Zero-Shot for LLaMA3-8B 64.10 -> 64.07, LLaMA2-7B 62.01 -> 62.09). This suggests that KL-Top loss is most effective when coupled with powerful distribution-reshaping transformations.
• OSTQuant with KL-Top: For OSTQuant, using Cross-Entropy (CE) loss (referred to as Origin in Table 1) results in overfitting on the calibration set, leading to higher Wiki PPL and lower Zero-Shot scores. The introduction of KL-Top loss (OSTQuant + KL-Top) alleviates this overfitting problem, leading to better generalization and improved Zero-Shot scores (e.g., LLaMA3-8B 65.13 -> 65.37, LLaMA2-7B 62.45 -> 63.11). This highlights that KL-Top loss is crucial for OSTQuant to leverage its transformation capabilities effectively on limited PTQ data.

6.4. QSUR and Evaluation Loss Trends (Figure 10)

Figure 10 illustrates the dynamic behavior of QSUR and evaluation loss during the training process for the LLaMA3-8B model.

• As the number of training iterations increases, the local and average QSURs (Quantization Space Utilization Rates) consistently increase and stabilize at a high level. This directly shows that OSTQuant's learnable transformations are effectively optimizing the data distributions to better utilize the quantization space.
• Concurrently, the evaluation loss gradually decreases, reflecting the model's improved performance during the quantization-aware optimization.
• The positive correlation between increasing QSUR and decreasing evaluation loss (or improving accuracy, as seen in Figure 3) empirically validates QSUR as an effective metric for guiding and assessing quantization quality.

6.5. Activation and Weight Distribution Visualizations (Figure 11, 12, 14)

These visualizations provide qualitative evidence of OSTQuant's impact on data distributions and quantization error.

• Activation Distributions (Figure 11, 12): These figures display the activation distributions of different layers in LLaMA-2-7B and LLaMA-3-8B before and after OSTQuant.
  • Before OSTQuant: The distributions show significant variations across different channels and often exhibit prominent outliers, leading to wide ranges and low QSUR.
  • After OSTQuant: The distributions become noticeably more uniform across channels, and outliers are substantially mitigated. This "flattening" and "centering" of distributions allows for a more efficient mapping to discrete quantization levels, reducing the relative quantization error.
• Weight Distributions and WOMI (Figure 13): This figure compares the weight distribution and relative L1 error for LLaMA-2-7B when quantized with original, Hadamard transformed, and WOMI transformed weights.
  • Original: Shows large variations and outliers.
  • Hadamard Transformed: Reduces some inter-channel differences, but some spikes or extreme values might remain.
  • WOMI Transformed: Further smooths these inter-channel differences and reduces outliers more effectively, leading to a smaller relative L1 error. This demonstrates WOMI's role in providing a better initial distribution for weight quantization.
• Comparison of Activation Quantization Errors (Figure 14): This figure compares the activation distribution and relative L1 error for QuaRot, SpinQuant, and OSTQuant on LLaMA-2-7B.
  • QuaRot and SpinQuant show improvements over baselines, but OSTQuant achieves the most uniform activation distributions with the lowest relative L1 errors. This visually confirms that OSTQuant's combined orthogonal and scaling transformations are superior in making activations quantization-friendly by effectively managing outliers and variances, resulting in less information loss during per-token quantization.

7. Conclusion & Reflections

7.1. Conclusion Summary

In this paper, the authors introduce OSTQuant, a novel post-training quantization (PTQ) method aimed at significantly improving the efficiency of Large Language Models (LLMs). The central innovation is the Quantization Space Utilization Rate (QSUR), a new metric that quantitatively assesses the quantizability of transformed data by measuring how effectively it occupies and utilizes the available quantization space. This metric, supported by rigorous mathematical derivations on the effects and limitations of various linear transformations, provides a principled guide for optimizing data distributions.

Leveraging these insights, OSTQuant employs learnable equivalent transformation pairs, consisting of both orthogonal and scaling transformations, to optimize the distributions of weights and activations across the entire quantization space. This combined approach effectively addresses outliers and inter-channel disparities, leading to more quantization-friendly distributions. Furthermore, to overcome the challenges of limited calibration data inherent in PTQ, the KL-Top loss function is proposed. This loss function mitigates optimization noise while retaining richer semantic information by focusing KL divergence on only the top $k$ highest-probability logits.

Extensive experiments across various LLMs (LLaMA-1, -2, -3) and benchmarks demonstrate OSTQuant's superior performance. It retains over 99.5% of the floating-point accuracy in W4-only settings and reduces the performance gap by 32% on LLaMA-3-8B in the more challenging W4A4KV4 configuration compared to state-of-the-art methods. The method also delivers significant inference speedups (over $2 \times$) and memory savings (over $3.5 \times$), while accelerating the quantization process itself by up to $5.3 \times$. These results underscore the effectiveness of OSTQuant's principled approach to optimizing data distributions for practical and efficient LLM deployment.

7.2. Limitations & Future Work

The authors identify clear directions for future research, primarily focusing on extending OSTQuant to fully-quantized LLMs.

• Extension to Full Quantization: The current OSTQuant framework, while effective, is primarily presented for traditional PTQ where some components (like KV cache or certain activations) might still be in higher precision. The future work section (Appendix A.5.1) outlines an extension for full quantization, where all activations within each Transformer Block are quantized to low bits. This would further reduce memory transfer overhead and fully leverage low-precision computational units.
  • New Transformation Pairs: This extension proposes introducing more equivalence transformations around ROPE (Rotary Positional Encoding) and SiLU (Sigmoid Linear Unit) operations, which are critical components in modern LLMs.
    • ROPE Handling: Treating ROPE as a lightweight GEMM layer and constructing a weight matrix based on its principle allows for pre-ROPE and post-ROPE transformation pairs. This involves scaling transformations ($S_q^i, S_k^i, S_{post}^i$) and orthogonal transformations ($R_{pre}^i, R_{post}^i$) to optimize query and key distributions before and after ROPE and attention computation.
    • SiLU Smoothing: Decomposing SiLU(X) =X \cdot \sigma(X)$$ allows for equivalence transformations (e.g., $X_1 \cdot X_2 \cdot \sigma(X_1) = (X_1 \cdot S) \cdot (S_2 \cdot \frac{1}{S}) \cdot \sigma((X_1 \cdot S) \cdot \frac{1}{S})$) to alleviate inter-channel discrepancies of activations before and after SiLU using scaling matrices like $S_{u|g}$. The authors plan to conduct experiments in the full-quantization domain to explore the full potential of OSTQuant.

7.3. Personal Insights & Critique

This paper presents a rigorous and well-motivated approach to LLM quantization. Here are some personal insights and critiques:

• Strengths and Innovations:

  • Principled Metric (QSUR): The introduction of QSUR is a significant contribution. It moves beyond heuristic observation to provide a quantitative, theoretically-backed metric to guide quantization optimization. This allows for a more direct and transparent evaluation of transformation effectiveness. The correlation between QSUR and accuracy (Figure 3) strongly supports its utility.
  • Holistic Transformation Strategy: The combination of learnable orthogonal and scaling transformations is powerful. Orthogonal transformations excel at reshaping distributions to be more ball-shaped and outlier-resistant, while scaling transformations fine-tune inter-channel variances. This dual approach provides a more comprehensive solution than methods relying solely on one type of linear transformation.
  • Addressing PTQ Constraints: The KL-Top loss function is a clever solution to the problem of limited calibration data and large vocabulary sizes in LLMs. It effectively balances the need for semantic preservation with the practical challenges of PTQ optimization, preventing overfitting and noise.
  • Efficiency: The method not only achieves high accuracy but also demonstrates impressive speedups in inference and memory savings, making LLMs more deployable. The fast training time for quantization is also a major practical advantage.
  • Equivalence Preservation: The design of equivalent transformation pairs that are fused into weights ensures no additional runtime overhead, which is crucial for real-world deployment.

• Potential Issues, Unverified Assumptions, or Areas for Improvement:

  • QSUR Formula Simplifications: While the QSUR derivation provides a solid theoretical basis, some simplifications (e.g., neglecting the mean vector $\boldsymbol{\mu}$, assuming $\lambda_{\mathrm{max}} = \lambda_{\mathrm{min}} = \lambda_1$) are made to arrive at a tractable form (Equation 7). The practical impact of these simplifications on the optimality of the derived transformation (Equation 8) could be further explored. Do real LLM distributions always conform well enough to these assumptions for the derived optimum to hold globally?
  • Complexity of Riemannian Optimization: While RiemannAdam is effective, optimizing on Stiefel manifolds is inherently more complex than standard Euclidean optimization. This might pose a barrier for practitioners less familiar with Riemannian geometry or require specialized libraries. The ease of integration into common ML frameworks (like PyTorch) is good, but understanding the underlying mechanics remains a learning curve.
  • Sensitivity to $k$ in KL-Top: While an optimal $k=1000$ is identified for the tested models, this value might be sensitive to specific LLM architectures, vocabulary sizes, or downstream tasks. A more adaptive or data-driven way to select $k$ could enhance robustness.
  • Generalizability of WOMI: WOMI relies on the assumption that weights follow a Gaussian distribution with zero mean. While often true for typical weight initializations, deviations could affect its optimality.
  • Full Quantization Details: The future work section on full quantization is promising, but the mathematical details of how the ROPE and SiLU transformations are maintained as equivalent transformation pairs (similar to Equation 9) could be further elaborated in a subsequent paper. The decomposition of SiLU into a product and its transformation for inter-channel discrepancies (Equation 31) is an intriguing idea, and its empirical validation would be valuable.

• Transferability and Future Value:

  • The QSUR metric is highly transferable and could be adopted by other quantization methods as a standard evaluation and optimization target, fostering more principled research.

  • The combined orthogonal and scaling transformation strategy is broadly applicable to any neural network layer that performs matrix multiplication, not just LLMs.

  • The insights from KL-Top loss are relevant for any PTQ scenario involving large output spaces (e.g., in sparse expert models) and limited calibration data.

  • The paper's direction towards full quantization is critical, as this is the ultimate goal for maximizing hardware efficiency. OSTQuant seems well-positioned to contribute significantly to this challenging area.

    Overall, OSTQuant represents a significant step forward in LLM quantization, offering a robust, theoretically grounded, and highly effective solution that addresses key challenges in LLM compression and acceleration.