1. Bibliographic Information

1.1. Title

VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction

1.2. Authors

• Sinan Du

• Jiahao Guo

• Bo Li

• Shuhao Cui

• Zhengzhuo Xu

• Yifu Luo

• Yongxian Wei

• Kun Gai

• Xinggang Wang

• Kai Wu

• Chun Yuan

  Affiliations: iua University, Huazhong University of Science and Technology, Kolors Team Kuaishou Technology.

1.3. Journal/Conference

This paper is a preprint published on arXiv. The official publication venue is not specified, but its Published at (UTC): 2025-11-28T17:26:34.000Z timestamp suggests it is a recent or forthcoming submission.

1.4. Publication Year

2025

1.5. Abstract

The paper addresses the challenge of unifying multimodal understanding, generation, and reconstruction representations within a single tokenizer for unified models. Existing research often uses a dual-encoder paradigm or balances features with contrastive loss. This work introduces VQRAE (Vector Quantization version of Representation AutoEncoders), which explores a unified representation approach. VQRAE is designed to produce continuous semantic features for image understanding and discrete tokens for visual generation using a single tokenizer. It builds upon pretrained vision foundation models with a symmetric ViT decoder and employs a two-stage training strategy: first, freezing the encoder to learn a high-dimensional semantic VQ codebook with pixel reconstruction; then, jointly optimizing the encoder with self-distillation constraints. This design allows VQRAE to maintain multimodal understanding capabilities with negligible semantic information loss, generate discrete tokens compatible with generation, and achieve fine-grained reconstruction. A notable finding is the efficacy of a high-dimensional codebook (e.g., 1536 dimensions) for quantizing semantic encoders, achieving a 100% utilization ratio, contrasting with prior common practice of low-dimensional codebooks for image reconstruction. VQRAE demonstrates competitive performance across various benchmarks for visual understanding, generation, and reconstruction, showing promising scaling properties for autoregressive models due to its discrete nature.

1.6. Original Source Link

https://arxiv.org/abs/2511.23386 Publication Status: Preprint.

1.7. PDF Link

https://arxiv.org/pdf/2511.23386v1.pdf

2. Executive Summary

2.1. Background & Motivation

The advancement of Multimodal Large Language Models (MLLMs), such as GPT-4o, has highlighted the immense potential for unifying visual understanding and generation within a single autoregressive architecture. However, a fundamental challenge persists: how to design a visual tokenizer that can produce appropriate representations to achieve an optimal trade-off across three critical tasks: visual understanding, generation, and reconstruction.

Core Problems & Challenges:

• Representation Dilemma: Traditional discrete tokenizers (e.g., VQGAN) are excellent for next token prediction (NTP)-based generation (due to compatibility and efficiency) but often produce pixel-level features for fine-grained details. These features can conflict with the semantic-level representations needed for visual understanding tasks (like CLIP-based recognition), leading to performance degradation.

• Dual Encoder Complexity: To mitigate this, previous research often adopted a dual encoder paradigm (e.g., Janus series, TokenFlow), employing separate encoders for understanding and generation. This increases model complexity, hinders deeper interaction between representations, and often requires immense batch sizes to balance conflicting losses.

• Loss Balance & Quantization Errors: Approaches using contrastive loss (QLIP, UniTok) to balance semantic and low-level features still face challenges with large batch sizes and potential loss conflicts. Discrete methods also inherently suffer from quantization errors which can impact semantic understanding.

• Lack of Unified Tokenizer: The field lacked a truly unified tokenizer capable of simultaneously producing continuous semantic features for understanding and discrete fine-grained tokens for generation and reconstruction from a single, cohesive architecture without relying on complex dual-encoder designs or convolutional pixel encoders.

  The paper aims to address this dilemma by proposing a novel unified tokenizer that can achieve this trade-off effectively.

2.2. Main Contributions / Findings

The paper introduces VQRAE, a Vector Quantization version of Representation AutoEncoders, making the following primary contributions:

• Pioneering Unified Tokenizer: VQRAE is the first work to explore a unified tokenizer that produces both continuous semantic features for visual understanding and discrete tokens for visual generation and reconstruction. This eliminates the need for complex dual-encoder paradigms and convolutional blocks, using a pure ViT-based model.

• Novel High-Dimensional VQ Codebook: VQRAE successfully trains a high-dimensional VQ codebook (e.g., 1536 dimensions, comparable to CLIP encoders) with a remarkable 100% utilization ratio. This finding contradicts previous common practices in CNN-based VQ codebook training, which suggested low-dimensional codebooks for reconstruction. The semantic nature of the VFMs allows for this novel high-dimensional approach.

• Effective Two-Stage Training Strategy: The proposed two-stage training strategy (freezing encoder for codebook/decoder optimization, then unfreezing encoder with self-distillation) effectively preserves semantic understanding while enhancing fine-grained reconstruction, achieving a superior trade-off.

• Competitive Performance Across Tasks: VQRAE demonstrates competitive performance on various benchmarks spanning visual understanding, generation, and reconstruction tasks, highlighting its efficacy as a unified tokenizer.

• Promising Scaling Properties: By leveraging its discrete nature and semantic high-dimensional latent space built on VFMs, VQRAE shows promising scaling properties in the autoregressive paradigm, benefiting training dynamics.

  These findings address the fundamental dilemma of visual tokenizers by providing a single, efficient, and robust solution for multimodal representation learning, paving the way for more integrated and powerful MLLMs.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully understand the VQRAE paper, a beginner should be familiar with several core concepts in deep learning and computer vision:

• Autoencoders (AEs): An autoencoder is a type of artificial neural network used to learn efficient data codings (representations) in an unsupervised manner. It consists of two main parts: an encoder that compresses the input into a latent-space representation (bottleneck) and a decoder that reconstructs the input from this latent space. The goal is for the reconstructed output to be as close as possible to the original input.

  • Variational Autoencoders (VAEs): VAEs are generative models that learn a latent space with a probabilistic interpretation. Instead of encoding the input into a fixed latent vector, VAEs encode it into a distribution (mean and variance) over the latent space. This allows for sampling from the latent space to generate new, similar data.
  • Representation Autoencoders (RAEs): As mentioned in the paper, RAEs [96] are a recent variant that replaces the VAE encoder with pre-trained vision encoders (like ViT or CLIP) and pairs them with trained decoders. RAEs demonstrated that the structured semantic space learned by these powerful encoders could benefit the convergence of diffusion models. VQRAE builds directly on this idea, but adds Vector Quantization.

• Vector Quantization (VQ): Vector Quantization is a technique that maps input vectors from a continuous space to a finite set of discrete codebook vectors. It's often used in autoencoders to introduce discreteness into the latent space.

  • VQ-VAE (Vector Quantized Variational Autoencoder): VQ-VAE [66] combines VQ with VAEs (though it simplifies some aspects of the VAE objective). The key idea is that the encoder outputs a continuous latent representation, which is then "quantized" by finding the closest vector in a learned codebook. This discrete latent representation is then passed to the decoder. This makes the latent space discrete, which is beneficial for autoregressive models.
  • VQ-GAN (Vector Quantized Generative Adversarial Network): VQ-GAN [15] replaces the VAE's reconstruction loss with a combination of perceptual loss and adversarial loss from a Generative Adversarial Network (GAN) to produce higher-fidelity image reconstructions. It also uses a VQ codebook for discrete latent representations, making it suitable for generating images with discrete tokens.

• Vision Foundation Models (VFMs): These are large, pre-trained neural networks trained on vast amounts of image and sometimes text data, capable of extracting rich, versatile visual features.

  • Vision Transformer (ViT): ViT [2] adapts the Transformer architecture (originally for natural language processing) to computer vision tasks. It treats image patches as sequences of tokens, allowing Transformers to process them. ViT-based encoders are known for learning powerful semantic representations.
  • CLIP (Contrastive Language-Image Pre-training): CLIP [50] is a VFM trained on a massive dataset of image-text pairs. It learns to associate images with their textual descriptions by bringing their representations closer in a shared latent space. This makes CLIP encoders excellent at extracting semantic features relevant to natural language. SigLIP [93] and InternViT [98] are similar powerful vision-language models.

• Multimodal Large Language Models (MLLMs): These are Large Language Models (LLMs) extended to process and understand multiple modalities, typically text and images. They combine the reasoning and generation capabilities of LLMs with visual perception. Unified MLLMs aim to use a single underlying architecture for various multimodal tasks.

• Autoregressive (AR) Models / Next Token Prediction (NTP): Autoregressive models predict the next item in a sequence based on previous items. In LLMs and sequence generation (like image generation from discrete tokens), this often involves Next Token Prediction (NTP), where the model predicts the most probable next token given the context of already generated tokens. Discrete tokens are highly compatible with this paradigm.

• Self-Distillation: A training technique where a model (the "student") is trained to mimic the outputs of another model (the "teacher"), often a larger or more robust version of itself, or even the same model at an earlier, frozen state. This helps the student model learn better representations or maintain performance.

• Perceptual Loss (LPIPS): Instead of comparing pixels directly (like L2 loss), perceptual loss [15] compares high-level features extracted by a pre-trained deep neural network (e.g., VGG or ConvNeXt). This makes the reconstructed images visually more realistic and less blurry, even if pixel-wise differences are larger. LPIPS (Learned Perceptual Image Patch Similarity) is a common way to calculate this.

• Adversarial Loss (GANs): Adversarial loss is a component of Generative Adversarial Networks (GANs). A GAN consists of a generator (which creates fake data) and a discriminator (which tries to distinguish real data from fake data). The generator is trained to fool the discriminator, and the discriminator is trained to be accurate. This adversarial process helps the generator produce highly realistic outputs.

3.2. Previous Works

The paper contextualizes VQRAE by discussing prior attempts at multimodal representation, primarily focusing on visual tokenizers for generation and unified tokenizers.

• Visual Tokenizers for Generation:

  • VQGAN [15], VQ-VAE [66], LlamaGen [57], Open-MAGVIT2 [40]: These methods use Vector Quantization to encode raw pixels into compact discrete latent representations. These discrete tokens are highly compatible with autoregressive (AR) and masked generative models [3, 51, 57, 64, 88].
  • Limitation: While effective for generation, these discrete tokenizers (especially those trained with pixel-level reconstruction objectives) often lead to performance degradation in visual understanding tasks (e.g., CLIP-based tasks) due to quantization errors [36, 67, 72]. The fine-grained, pixel-level tokens produced also incur significant alignment costs when integrated with LLMs for semantic understanding.

• Unified Tokenizers (Addressing the Representation Dilemma):

  • Dual-Encoder Paradigm (Janus series [7, 43, 76], TokenFlow [49], MUSE-VL [82]): These approaches attempt to disentangle representations by employing separate visual encoders: one semantic encoder (often ViT-based) for understanding and another pixel encoder (often CNN-based) for generation.
    • Limitation: This leads to increased model complexity and training overhead. More importantly, it can hinder deeper interaction and alignment between different representations, which is crucial for truly unified MLLMs [11, 12, 24, 33, 45, 59, 61, 68, 73, 77, 81].
  • Contrastive Learning Supervision (QLIP [95], VILA-U [78], UniTok [41]): These methods supervise latent features from Vision Foundation Models (VFMs) (like CLIP [50, 93]) with contrastive learning loss. The goal is to enforce semantic alignment.
    • Limitation: Contrastive learning often demands immense batch sizes for effective training and struggles to balance conflicting losses between semantic and low-level features.
  • Diffusion-based Tokenizers [6, 58, 92]: These typically employ continuous representations for reconstruction.
    • Limitation: They are difficult to converge in autoregressive paradigms due to the high dimensionality of CLIP features [31, 50].
  • Semantic Supervision without Reconstruction (Tar [21], X-Omni [19]): These approaches propose VQ tokenizers trained with semantic supervision (e.g., self-distillation from VFMs) but discarded reconstruction capabilities.
    • Limitation: By discarding reconstruction, they lose their nature as autoencoders and cannot serve as a full solution for multimodal tasks requiring both generation/reconstruction and understanding. They also suffer from quantization errors impacting understanding.
  • RAE [96] (Representation Autoencoders): A recent work that utilizes pre-trained VFMs as encoders paired with trained decoders for image reconstruction. It demonstrated that the structured semantic space of VFMs benefits the convergence of diffusion transformers. VQRAE builds upon RAE by introducing Vector Quantization to achieve discrete tokens while maintaining semantic properties.

3.3. Technological Evolution

The field of multimodal models has evolved from separate systems for understanding and generation to increasingly unified architectures:

1. Separate Systems: Early approaches often used distinct models for visual understanding (e.g., CNN classifiers, CLIP for embedding) and visual generation (e.g., GANs, VAEs, then VQGAN for discrete tokens).

2. Discrete Tokens for AR Generation: The success of Transformer-based LLMs led to attempts to apply autoregressive generation to images by tokenizing them into discrete units using VQ-VAE or VQGAN. This allowed MLLMs to generate images token by token, compatible with NTP.

3. Semantic vs. Pixel-level Features Conflict: The core issue emerged when these pixel-level discrete tokens from generation models proved suboptimal for semantic understanding tasks.

4. Dual-Encoder Attempts: The dual-encoder paradigm arose as a compromise, with one encoder specialized for semantic understanding and another for pixel-level generation. This, however, introduced complexity and limited deeper integration.

5. Weak Semantic Supervision: Efforts to train a single tokenizer with contrastive loss or self-distillation from VFMs aimed to bridge the gap, but often struggled with training stability, resource demands, or quantization errors.

6. RAE's Insight: The realization that powerful pre-trained VFMs (like ViT encoders) already learn structured semantic spaces suitable for reconstruction (as shown by RAE) opened new avenues.

   This paper's work, VQRAE, fits into this timeline by taking the RAE insight a step further. It aims to unify the continuous semantic features from VFMs for understanding with discrete tokens for generation/reconstruction within a single tokenizer, avoiding the pitfalls of dual-encoders and semantic information loss from quantization errors in understanding tasks.

3.4. Differentiation Analysis

Compared to the main methods in related work, VQRAE presents several core differences and innovations:

• Unified Continuous and Discrete Output from a Single Encoder:
  • Differentiation: Unlike dual-encoder methods (e.g., Janus [76], TokenFlow [49], MUSE-VL [82]) which use separate encoders for continuous understanding features and discrete generation tokens, VQRAE uses a single, unified VFM (ViT-based) encoder. This encoder directly produces continuous semantic features for understanding (without quantization) and, via a VQ codebook, also generates discrete tokens for generation and reconstruction.
  • Innovation: This design reduces model complexity, improves training efficiency, and fosters deeper interaction between different representations, which is crucial for truly unified MLLMs.
• High-Dimensional, 100% Utilization Semantic VQ Codebook:
  • Differentiation: Previous VQ methods (e.g., VQGAN [15], VQVAE [66]) typically train low-dimensional codebooks (e.g., 8-256) primarily with CNN-extracted features for pixel reconstruction. Scaling to higher dimensions often led to codebook collapse or low utilization. VQRAE is the first to successfully train a high-dimensional codebook (e.g., 1536 dimensions, matching VFM embedding dimensions) with a nearly 100% utilization ratio for semantic features.
  • Innovation: This finding is a novel empirical property for VFM-based quantization, indicating that semantic encoders benefit from larger codebook dimensions, contrary to prior beliefs for pixel-level features. This allows for rich, semantic discrete tokens.
• Pure ViT-based Architecture without Convolutional Blocks:
  • Differentiation: Many prior unified tokenizers or pixel-level VQGANs rely on CNN-based architectures or hybrid designs. VQRAE utilizes pre-trained VFMs (which are ViT-based) as the encoder and a symmetric ViT-based decoder.
  • Innovation: This streamlines the architecture, leverages the powerful semantic capabilities of ViTs, and avoids the need for specialized pixel encoders, simplifying the overall design.
• Robust Two-Stage Training Strategy:
  • Differentiation: While some methods use self-distillation [21, 47], VQRAE employs a specific two-stage strategy: first, freezing the VFM encoder to optimize the VQ codebook and ViT decoder for pixel reconstruction; then, unfreezing the encoder and jointly optimizing all components with self-distillation loss (from the frozen encoder itself) to maintain semantic features and enhance reconstruction.
  • Innovation: This strategy effectively balances the trade-off, enabling fine-grained reconstruction without degrading the VFM's strong semantic understanding capabilities, and even improving them in some cases. It ensures that the continuous features directly from the VFM encoder (without quantization errors) are always available for understanding.
• Superior Trade-off for Understanding and Reconstruction:
  • Differentiation: Unlike Tar [21] or X-Omni [19] which perform semantic distillation directly on discrete tokens (potentially leading to quantization errors for understanding) and discard reconstruction, VQRAE offers a superior trade-off. It provides continuous features for understanding (avoiding quantization errors) while also maintaining strong reconstruction capabilities through its autoencoder nature.
  • Innovation: This comprehensive approach allows VQRAE to be a truly versatile unified tokenizer for AR-only paradigms, suitable for all three tasks.

4. Methodology

4.1. Principles

The core idea behind VQRAE is to design a single, unified visual tokenizer that can effectively serve three distinct multimodal tasks: visual understanding, visual generation, and image reconstruction. This is achieved by simultaneously producing two types of representations:

1. Continuous semantic features: Directly from a powerful pre-trained Vision Foundation Model (VFM) encoder, these features are ideal for high-level visual understanding tasks, maintaining rich semantic information without quantization errors.

2. Discrete tokens: Derived from the same semantic features through Vector Quantization (VQ), these tokens are compatible with autoregressive (AR) models for efficient image generation and can be used for fine-grained image reconstruction.

   The theoretical basis and intuition are rooted in the observation that VFMs (like ViT-based models) are excellent at learning structured semantic spaces. RAE [96] showed that these continuous semantic features could be directly used for reconstruction. VQRAE extends this by introducing Vector Quantization to make these semantic features discrete, thus compatible with AR generation, while carefully preserving the original continuous semantics for understanding. The two-stage training strategy is crucial for balancing the conflicting objectives of maintaining semantic integrity (for understanding) and achieving high-fidelity pixel-level reconstruction.

4.2. Core Methodology In-depth (Layer by Layer)

As illustrated in Figure 3a (not shown, but conceptually described), VQRAE consists of three main components: a unified tokenizer built upon pre-trained Vision Foundation Models (VFMs) for encoding, a high-dimensional semantic VQ codebook, and a symmetric ViT decoder for pixel reconstruction.

4.2.1. VFMs as Unified Encoder

Unlike prior dual-encoder approaches that use separate encoders for semantic understanding (e.g., ViT-based) and pixel-level generation (e.g., CNN-based), VQRAE employs a single, pre-trained VFM (such as CLIP, SigLIP, or InternViT) as its unified encoder, denoted as $E$. This design choice aims to reduce model complexity, training overhead, and improve representation interaction.

Given an input image $\boldsymbol{X} \in \mathbb{R}^{h \times w \times 3}$, where $h$ is height, $w$ is width, and 3 represents the RGB channels, and an encoder $E$ with a patch size $p$ and hidden size $d$, the encoder produces latent features $Z_I \in \mathbb{R}^{\frac{hw}{p^2} \times d}$.

• $Z_I$ represents the continuous semantic features.

• $\frac{hw}{p^2}$ is the number of patches (tokens) extracted from the image.

• $d$ is the dimensionality of each patch embedding.

  These intermediate continuous features $Z_I$ serve a dual purpose:

1. They are directly utilized for multimodal understanding tasks without any quantization errors.

2. They are fed into the semantic VQ codebook for quantization to produce discrete tokens.

   The paper notes that while frozen VFMs can reconstruct images, they might lose fine details like color and texture. However, slight fine-tuning of the encoder can refine these representations to recover missing details, often without degrading semantic understanding, and sometimes even improving it.

4.2.2. High Dimensional VQ Vector Quantization

Vector Quantization (VQ) is a technique used to transform continuous representations into a set of discrete tokens. VQRAE applies this to the semantic features extracted from VFMs, rather than pixel features. The method utilized is SimVQ [100].

The key components for VQ are:

• A VQ codebook $C \in \mathbb{R}^{k \times e} = \{ c^i \}_{i=1}^k$, where $k$ is the codebook size (number of entries) and $e$ is the codebook dimension. Each $c^i$ is a codebook vector or prototype.

• A learnable projection matrix $W \in \mathbb{R}^{e \times e} = \{ w^i \}_{i=1}^e$, where $w^i$ are the basis vectors for projection.

  The process involves:

1. Projection: The continuous semantic features $Z_I$ from the VFM encoder are first projected to $\hat{Z}_c \in \mathbb{R}^{\frac{hw}{p^2} \times e}$.

2. Quantization: For each vector in $\hat{Z}_c$, the closest vector from the projected codebook entries $\{ c^i w^i \}$ is selected based on l2-norm distances. This selection yields the quantized vectors $Z_q \in \mathbb{R}^{\frac{hw}{p^2} \times e}$.

   The mathematical formulation for the quantization step is: $$Z_q = lookup \bigl( argmin || \hat{Z}_c - c^i w^i || \bigr), \mathrm{for} i = 1, \dots, k$$ Where:

• $Z_q$: The quantized vectors (discrete representation) that are selected from the codebook $C$.

• $lookup(\cdot)$: An operation that retrieves the selected codebook vector (or its index).

• argmin: The argument that minimizes the expression. In this case, it finds the index $i$ of the codebook vector that is closest to $\hat{Z}_c$.

• $|| \cdot ||$: Denotes the l2-norm (Euclidean distance), measuring the distance between vectors.

• $\hat{Z}_c$: The projected semantic features from the VFM encoder.

• $c^i$: The $i$-th codebook vector from the VQ codebook $C$.

• $w^i$: The $i$-th basis vector from the learnable projection matrix $W$.

  A crucial observation highlighted by the paper is that VQRAE's codebook performs effectively in a high-dimensional formulation, where its dimensionality $e$ must be at least that of the VFMs encoder's hidden size $d$. This contrasts with previous studies (e.g., VQGAN [15]), which typically suggested low-dimensional codebooks (e.g., 8-256) for pixel reconstruction objectives using CNN-extracted features.

4.2.3. Symmetric Decoder

VQRAE replaces traditional CNN-like pixel decoders [15, 28, 59] with a ViT-based decoder that mirrors the encoder's structure, similar to RAE [96]. This symmetric decoder, denoted as $D$, maps the latent features back to the pixel space to reconstruct the image.

The process is:

1. Projection to Bottleneck: The quantized vectors $Z_q$ are projected to bottleneck features $Z_{bot} \in \mathbb{R}^{\frac{hw}{p^2} \times d}$. This step aligns the dimensionality of the quantized features $Z_q$ (which have dimension $e$) with the input dimension required by the symmetric decoder $D$ (which expects dimension $d$).
2. Decoding: The decoder $D(Z_{bot})$ processes these bottleneck features.
3. Pixel Reconstruction: The decoded features are then projected to the reconstructed image $X' \in \mathbb{R}^{\frac{h q}{p} \times \frac{w q'}{p} \times 3}$.
   • $q$ and $q'$ are hyperparameters used to adjust the resolution of the reconstructed images.
   • For VQRAE, $q = q' = p$ is set to maintain a constant resolution equal to the input image's patch size.

4.2.4. Two-Stage Training

The training of VQRAE employs a two-stage strategy, designed to balance the objectives of semantic feature preservation and fine-grained pixel reconstruction.

Stage 1: Codebook and Decoder Optimization with Frozen Encoder In this initial stage, the VFMs encoder $E$ is kept frozen (sg(E) operation is applied implicitly by freezing parameters) to preserve the integrity of its learned semantic features. The VQ codebook $C$ and the symmetric ViT decoder $D$ are jointly optimized. The primary objective is to achieve high-fidelity pixel reconstruction and effective vector quantization.

The loss function for Stage 1, $\mathcal{L}_{\mathrm{stage1}}$, is a combination of reconstruction loss and quantization loss: $$\mathcal{L}_{\mathrm{stage1}} = \mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{quant}}$$

Where:

• $\mathcal{L}_{\mathrm{rec}}$ is the reconstruction loss, comprising three components: $$\mathcal{L}_{\mathrm{rec}} = \ell_2 (X, X') + \mathcal{L}_{\mathrm{P}} (X, X') + \lambda_{\mathrm{G}} \mathcal{L}_{\mathrm{G}} (X')$$

  • $\ell_2 (X, X')$: Represents the pixel-wise reconstruction loss (Mean Squared Error or MSE) between the original input image $X$ and the reconstructed image $X'$. This measures direct pixel differences.
  • $\mathcal{L}_{\mathrm{P}} (X, X')$: Denotes the perceptual loss (e.g., using LPIPS). This loss compares high-level feature representations of $X$ and $X'$ extracted by a pre-trained network, aiming to make reconstructed images visually more similar and less blurry.
  • $\mathcal{L}_{\mathrm{G}} (X')$: Represents the adversarial loss from a discriminator (not explicitly shown in the formula but implied by $\mathcal{L}_{\mathrm{G}}$ and its coefficient $\lambda_{\mathrm{G}}$). The discriminator is trained to distinguish real images from reconstructed images, pushing the decoder to generate more realistic outputs.
  • $\lambda_{\mathrm{G}}$: Is the weight coefficient for the adversarial loss.

• $\mathcal{L}_{\mathrm{quant}}$ is the vector quantization loss, which ensures the codebook is effectively learned and utilized: $$\mathcal{L}_{\mathrm{quant}} = || sg(C) - Z_q ||_2^2 + \beta \cdot || Z_q - sg(C) ||_2^2$$

  • $|| sg(C) - Z_q ||_2^2$: This term encourages the encoder's output $\hat{Z}_c$ (which is implicitly $Z_q$ here as $Z_q$ is directly selected from $C$) to be close to the codebook entries. The stop-gradient operation sg(C) means that gradients from this term do not flow back into the codebook $C$ itself, but rather encourage $\hat{Z}_c$ to align with the chosen codebook vector.
  • $\beta \cdot || Z_q - sg(C) ||_2^2$: This is the codebook commitment loss. It encourages the codebook vectors to move towards the encoder's output for the selected entries. Here, $sg(Z_q)$ means gradients from this term do not flow back to $Z_q$, ensuring that the codebook $C$ itself is updated.
  • sg[·]: Denotes the stop-gradient operation, which blocks the gradient flow through its argument.
  • $\beta$: Is a hyperparameter, typically set to 0.25 by default, balancing the commitment loss.

Stage 2: Joint Optimization with Self-Distillation In the second stage, the VFMs encoder $E$ is unfrozen. The goal is to further augment reconstruction quality while crucially maintaining (or even strengthening) the performance of multimodal understanding. This is achieved by introducing a self-distillation loss constraint. All components (encoder, VQ codebook, decoder) are jointly optimized.

The loss function for Stage 2, $\mathcal{L}_{\mathrm{stage2}}$, includes the Stage 1 losses plus the new self-distillation loss: $$\mathcal{L}_{\mathrm{stage2}} = \mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{quant}} + \lambda_d \mathcal{L}_{\mathrm{distill}}$$

Where:

• $\mathcal{L}_{\mathrm{rec}}$ and $\mathcal{L}_{\mathrm{quant}}$ are the same as in Stage 1.
• $\mathcal{L}_{\mathrm{distill}}$ is the self-distillation loss: $$\mathcal{L}_{\mathrm{distill}} = || Z_I - T(X) ||_2^2$$
  • $|| \cdot ||_2^2$: Denotes the l2-norm (squared Euclidean distance) for the distillation objective.
  • $Z_I$: The continuous features produced by the current, unfrozen encoder $E$ for the input image $X$. It's important to note that this uses the continuous features $Z_I$ before quantization, thereby avoiding quantization errors in the distillation process.
  • T(X): The features produced by a teacher model $T$ for the input image $X$. This teacher model $T$ is typically initialized from an earlier, frozen version of the encoder $E$ (often the encoder from the end of Stage 1 or the original pre-trained VFM) and remains frozen throughout Stage 2. This self-distillation encourages the unfrozen encoder to maintain its semantic properties while it's also learning to improve reconstruction.
  • $\lambda_d$: Is the coefficient for the distillation loss, balancing its impact.

4.2.5. Semantic VQ with 100% Utilization Ratio

A significant finding of VQRAE is the successful training of a high-dimensional codebook (e.g., 1536 dimensions) with a 100% utilization ratio (meaning all codebook entries are actively used during training). This contrasts sharply with previous VQ methods (like VQVAE [66] and VQGAN [15]) that typically operated with low-dimensional codebook entries (e.g., 8-256) when quantizing features extracted from CNN-based encoders for pixel reconstruction. Previous studies often encountered codebook collapse or sharp declines in utilization when attempting to scale codebook dimensions to match CLIP-based encoders (e.g., 1152).

VQRAE empirically demonstrates that when quantizing features extracted from VFMs (which are ViT-based and inherently semantic), a larger codebook dimension is necessary. Using lower dimensions can lead to non-convergence in reconstruction training and codebook collapse. The high utilization ratio achieved in VQRAE's high-dimensional semantic codebook is a key factor enabling its performance.

4.2.6. Multimodal Understanding with VQRAE

A core advantage of VQRAE is its ability to directly provide intermediate continuous features ($Z_I$) from the VFM encoder for image understanding tasks. Since these features are used without quantization, they are free from quantization errors that often degrade the understanding performance of discrete tokenizers [21, 47].

Furthermore, because VQRAE is built upon pre-trained VFMs, it can be seamlessly integrated into existing MLLMs. This significantly reduces training overhead, as the tokenizer can be directly incorporated without extensive additional pre-training or supervised fine-tuning for understanding tasks. For example, by replacing the ViT encoder in an MLLM with VQRAE's encoder, the MLLM can immediately leverage VQRAE's strong understanding capabilities. The paper explicitly states that the MLLMs used for evaluation were not specifically trained for the proposed unified tokenizer, demonstrating its plug-and-play nature for understanding.

4.2.7. Visual Generation with VQRAE

For visual generation, VQRAE leverages its discrete VQ codebook. While continuous autoregressive methods like MAR [31, 61] exist, the discrete nature of VQRAE's tokens offers better compatibility with highly optimized AI infrastructure for training acceleration in autoregressive models.

The generation pipeline involves:

1. Text Encoding: A text tokenizer encodes the input text prompt.

2. Image Encoding: VQRAE encodes the image (if image-to-image generation is involved, or generates tokens from scratch for text-to-image).

3. LLM Backbone: A Large Language Model (LLM) backbone (e.g., Qwen3 [87]) is used.

4. Vocabulary Expansion: The vocabulary size of the LLM is expanded to include the visual tokens from VQRAE.

5. Autoregressive Training: The LLM is trained with Next Token Prediction (NTP) loss exclusively on these visual tokens, learning to generate image sequences.

   The paper highlights VQRAE's disentangled representations. As visualized in Figure 4 (not shown, but described), the continuous semantic features tend to cluster similar objects and animals, while the discrete tokens (used for reconstruction and generation) cluster images based on similar textures. This indicates that a single unified tokenizer can provide distinct yet complementary representations, suggesting redundancy in dual-encoder paradigms. The VQRAE-InternViT version with a codebook size of 16k and dimension of 1536 is used for generation experiments.

5. Experimental Setup

5.1. Datasets

VQRAE's training and evaluation span multiple datasets, tailored to different tasks:

• VQRAE Pretraining:

  • BLIP3-o [6] open-sourced data: This forms the primary pretraining dataset for VQRAE. It comprises:
    • 27 million (M) samples recaptioned by Qwen2.5-VL-7B [1].
    • 5M samples from CC12M [4] (Conceptual Captions 12M, a large dataset of image-text pairs).
    • 4M synthesized images from JourneyDB [56] (a benchmark for generative image understanding).
  • Characteristics: These datasets provide a diverse collection of image-text pairs, suitable for training a multimodal tokenizer that can learn rich visual representations.

• Image Understanding Task (LLaVA-1.5 Setting):

  • LLaVA-Pretrain-595K [37]: A dataset used for pretraining LLaVA models, focusing on image-text alignment.
  • LLaVA-v1.5-mix665K [37]: A dataset used for supervised fine-tuning (SFT) of LLaVA models, containing mixed instruction-following data.
  • Characteristics: These datasets are standard for training Multimodal Large Language Models (MLLMs) for visual instruction tuning and general multimodal understanding.

• Visual Generation Task:

  • BLIP3-o [6] data: The same base data used for VQRAE pretraining.
  • Additional 80M high-quality images: Supplements the BLIP3-o data to enhance the quality and diversity of generated images.
  • Characteristics: Large-scale, high-quality image-text data is essential for training robust generative models.

• Ablation Studies (VQ Codebook):

  • ImageNet-1K [50k validation set]: A subset of the popular ImageNet dataset, typically used for image classification, but here used for efficient ablation studies on VQ codebook hyperparameters.

  • Characteristics: Contains a diverse set of natural images, suitable for evaluating reconstruction quality.

    The paper does not provide concrete examples of data samples (e.g., specific image-text pairs or images) from these datasets within the main text.

5.2. Evaluation Metrics

The paper evaluates VQRAE using a comprehensive suite of metrics across reconstruction, multimodal understanding, and visual generation tasks.

5.2.1. Reconstruction Metrics

Reconstruction quality is assessed using the following metrics, evaluated on the $256 \times 256$ ImageNet $50 \mathrm{k}$ validation set.

• rFID (reconstruction Frechet Inception Distance) $\downarrow$:

  • Conceptual Definition: Frechet Inception Distance (FID) is a metric used to assess the quality of images generated by generative models. It quantifies the similarity between the distribution of generated images and the distribution of real images. A lower FID score indicates better image quality and greater similarity between the generated and real image distributions. rFID specifically refers to FID calculated for image reconstruction tasks, comparing reconstructed images to their original counterparts.
  • Mathematical Formula: $ \mathrm{FID} = ||\mu_1 - \mu_2||^2 + \mathrm{Tr}(\Sigma_1 + \Sigma_2 - 2(\Sigma_1\Sigma_2)^{1/2}) $
  • Symbol Explanation:
    • $\mu_1$: The mean of the feature vectors for the real images.
    • $\mu_2$: The mean of the feature vectors for the generated/reconstructed images.
    • $\Sigma_1$: The covariance matrix of the feature vectors for the real images.
    • $\Sigma_2$: The covariance matrix of the feature vectors for the generated/reconstructed images.
    • $||\cdot||^2$: Squared Euclidean distance (L2 norm).
    • $\mathrm{Tr}(\cdot)$: Trace of a matrix.
    • Feature vectors are typically extracted from an intermediate layer of a pre-trained Inception-v3 network.

• PSNR (Peak Signal-to-Noise Ratio) $\uparrow$:

  • Conceptual Definition: PSNR is a common metric for measuring the quality of reconstruction of lossy compression codecs (or in this case, image reconstruction). It is defined as the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. Higher PSNR values generally indicate better quality reconstruction.
  • Mathematical Formula: $ \mathrm{PSNR} = 10 \cdot \log_{10} \left( \frac{\mathrm{MAX}I^2}{\mathrm{MSE}} \right) $ Where: $ \mathrm{MSE} = \frac{1}{mn} \sum{i=0}^{m-1} \sum_{j=0}^{n-1} [I(i,j) - K(i,j)]^2 $
  • Symbol Explanation:
    • $\mathrm{MAX}_I$: The maximum possible pixel value of the image (e.g., 255 for 8-bit images).
    • $\mathrm{MSE}$: Mean Squared Error between the original image $I$ and the reconstructed image $K$.
    • m, n: Dimensions (height and width) of the image.
    • I(i,j): Pixel value at position (i,j) in the original image.
    • K(i,j): Pixel value at position (i,j) in the reconstructed image.

• SSIM (Structural Similarity Index Measure) $\uparrow$:

  • Conceptual Definition: SSIM is a perceptual metric that quantifies the similarity between two images. It aims to more closely align with human perception of quality compared to traditional metrics like PSNR or MSE. SSIM considers luminance, contrast, and structure differences between images. Higher SSIM values indicate greater similarity and better quality reconstruction.
  • Mathematical Formula: $ \mathrm{SSIM}(x, y) = [l(x,y)]^{\alpha} \cdot [c(x,y)]^{\beta} \cdot [s(x,y)]^{\gamma} $ where, for common usage, $\alpha = \beta = \gamma = 1$. $ l(x,y) = \frac{2\mu_x\mu_y + C_1}{\mu_x^2 + \mu_y^2 + C_1} $ $ c(x,y) = \frac{2\sigma_x\sigma_y + C_2}{\sigma_x^2 + \sigma_y^2 + C_2} $ $ s(x,y) = \frac{\sigma_{xy} + C_3}{\sigma_x\sigma_y + C_3} $
  • Symbol Explanation:
    • x, y: Two image patches being compared (from the original and reconstructed images).
    • $\mu_x, \mu_y$: The average (mean) pixel intensity of $x$ and $y$.
    • $\sigma_x, \sigma_y$: The standard deviation of pixel intensities of $x$ and $y$.
    • $\sigma_{xy}$: The covariance of $x$ and $y$.
    • $C_1, C_2, C_3$: Small constants to prevent division by zero and stabilize the metrics (e.g., $C_1 = (K_1 L)^2$, $C_2 = (K_2 L)^2$, $C_3 = C_2/2$, where $L$ is dynamic range like 255, and $K_1, K_2$ are small constants).
    • l(x,y): Luminance comparison function.
    • c(x,y): Contrast comparison function.
    • s(x,y): Structure comparison function.

5.2.2. Multimodal Understanding Metrics

Multimodal understanding performance is evaluated on a range of benchmarks, following the setups in LLaVA-1.5 [37].

• MME-Perception (MME-P) [18] $\uparrow$:
  • Conceptual Definition: A comprehensive benchmark for multimodal large language models focusing on various perception abilities, including object recognition, counting, position, and attributes. Higher scores indicate better perceptual understanding.
• GQA [25] $\uparrow$:
  • Conceptual Definition: A dataset for real-world visual reasoning and compositional question answering. It requires models to perform multi-step reasoning over images to answer questions. Higher scores indicate better visual reasoning capabilities.
• POPE [32] $\uparrow$:
  • Conceptual Definition: A benchmark specifically designed to evaluate object hallucination in large vision-language models. It measures how often models "hallucinate" objects that are not present in an image. Higher scores (closer to 100%) indicate less hallucination (better accuracy in identifying absence/presence).
• MMBench-en (MMB) [38] $\uparrow$:
  • Conceptual Definition: A benchmark that evaluates multimodal models as all-around players across various domains and tasks, including fine-grained perception, spatial reasoning, and common sense. Higher scores indicate stronger general multimodal capabilities.
• SEEDBench-Img (SEED) [29] $\uparrow$:
  • Conceptual Definition: A benchmark for multimodal LLMs focusing on generative comprehension, requiring models to provide detailed and coherent textual descriptions or answers based on image content. Higher scores indicate better comprehension and generation of image-related text.
• MMMU [91] $\uparrow$:
  • Conceptual Definition: MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning benchmark) is designed to evaluate expert AGI (Artificial General Intelligence) through diverse, multi-discipline multimodal understanding and reasoning tasks, often requiring advanced knowledge. Higher scores suggest superior performance in expert-level multimodal reasoning.
• TextVQA (TQA) [54] $\uparrow$:
  • Conceptual Definition: A dataset for Visual Question Answering (VQA) where questions often require reading text present in the image (e.g., signs, labels, documents). Higher scores indicate better ability to read and understand text within images and answer questions based on it.
• AI2D [27] $\uparrow$:
  • Conceptual Definition: A dataset for diagram understanding and question answering. It tests a model's ability to interpret and reason about scientific diagrams and their components. Higher scores indicate better understanding of diagrammatic information.

5.2.3. Image Generation Metrics

Image generation performance is evaluated on two benchmarks.

• GenEval [20] $\uparrow$:
  • Conceptual Definition: An object-focused framework for evaluating text-to-image alignment. It assesses how well generated images adhere to specific object properties described in the text prompt, including single object, two objects, counting, position, color, and attributes. Higher scores indicate better alignment between text prompts and generated visual content.
• DPG-Bench [22] $\uparrow$:
  • Conceptual Definition: A benchmark designed to evaluate diffusion models with LLM for enhanced semantic alignment. It assesses the model's ability to generate images that accurately reflect global scene properties, entities, attributes, and relations specified in the text prompt. Higher scores indicate better semantic control over generation.

5.3. Baselines

VQRAE is compared against various baseline models, categorized by their primary function or architecture:

• Generative Only Tokenizers (for Reconstruction):

  • VQGAN [15]: A foundational model for discrete image tokenization.
  • LlamaGen [57]: An autoregressive image generation model.
  • VAR [64]: Visual Autoregressive modeling.
  • Open-MAGVIT2 [40]: An open-source autoregressive visual generation project.
  • RAE [96]: Representation Autoencoders, a direct inspiration for VQRAE.

• Unified Tokenizers (for Reconstruction):

  • Show-o [80]: One single transformer to unify multimodal understanding and generation.
  • TokenFlow [49]: Unified image tokenizer for multimodal understanding and generation.
  • DualViTok [23]: Dual visual tokenization for unified MLLM.
  • MUSE-VL [82]: Modeling unified VLM through semantic discrete encoding.

• Understanding Only MLLMs (for Multimodal Understanding): These are SOTA MLLMs without specific unified tokenizer design.

  • Emu3-Chat [70]: A native multimodal model.
  • LLaVA-1.5 [37]: Improved baselines with visual instruction tuning (used as a strong baseline, specifically with CLIP-L vision encoder and Vicuna-7B/13B LLM).
  • InternVL2.5 [8]: A powerful open-source multimodal model.
  • InternVL3 [98]: Advanced training and test-time recipes for open-source multimodal models.
  • Qwen2.5-VL [1]: A Qwen-series VLM.

• MLLMs with Unified Tokenizer (for Multimodal Understanding):

  • VILA-U [78]: A unified foundation model integrating visual understanding and generation.
  • UniTok [41]: A unified tokenizer for visual generation and understanding.
  • SemHiTok [9]: A unified image tokenizer via semantic-guided hierarchical codebook.
  • QLIP [95]: Text-aligned visual tokenization unifying AR multimodal understanding and generation.
  • TokenFlow-L/XL [49]: TokenFlow variants with different scales.
  • TokLIP [34]: Marrying visual tokens to CLIP for multimodal comprehension and generation.
  • Tar [21]: Unifying visual understanding and generation via text-aligned representations.

• Diffusion-based Models (for Visual Generation):

  • SDv1.5 [52]: Stable Diffusion v1.5.
  • PixArt-α [5]: Diffusion transformer for 4k text-to-image generation.
  • SDv2.1 [52]: Stable Diffusion v2.1.
  • SDXL [48]: Improving latent diffusion models for high-resolution image synthesis.
  • DALLE3 [35]: Text-to-image generation model.
  • SD3-Medium [16]: Scaling rectified flow transformers.
  • SANA-1.5 [79]: Efficient scaling of training-time and inference-time compute in linear diffusion transformer.

• Autoregressive-based Models (for Visual Generation):

  • Chameleon [60]: Mixed-modal early-fusion foundation models.

  • LlamaGen [57]: Autoregressive model that beats diffusion.

  • EMU3-Gen [70]: Next-token prediction is all you need.

  • TokenFlow [49]: Unified image tokenizer.

  • Janus [76]: Decoupling visual encoding for unified multimodal understanding and generation.

  • SimpleAR [67]: Pushing the frontier of autoregressive visual generation.

  • Janus-Pro [7]: Unified multimodal understanding and generation with data and model scaling.

    These baselines represent the state-of-the-art in their respective categories, allowing for a thorough comparison of VQRAE's performance and its unique advantages in unification.

6. Results & Analysis

6.1. Core Results Analysis

6.1.1. Pilot Experiments (Table 1)

The pilot experiments in Table 1 serve to elucidate VQRAE's design rationale by comparing different approaches to unified tokenization under LLaVA-1.5 [37] settings for understanding and ImageNet for reconstruction.

The following are the results from Table 1 of the original paper:

Analysis:

• Conflict between Reconstruction and Understanding (Exp. 1-2): VQGAN (Exp. 1), a discrete tokenizer trained primarily for pixel reconstruction, shows low MME-P (756.1) and SEED (38.2) scores, indicating poor multimodal understanding. VQD (Exp. 2), which distills VFMs for discrete tokens, improves understanding (MME-P: 1252.4, SEED: 57.8) but still lags behind continuous tokenizers due to quantization errors. This confirms the inherent trade-off.
• Continuous vs. Discrete (Exp. 3 vs. 1, 2): RRA (Exp. 3), a continuous representation method, achieves the best understanding scores (MME-P: 1544.3, SEED: 70.0, TQA: 61.7) among the single-encoder methods. However, its reconstruction (PSNR: 19.23, SSIM: 0.620) is not outstanding, suggesting that continuous representations, while good for semantics, might struggle with fine-grained pixel details without specific optimization.
• Dual Encoder Complexity (Exp. 4-5): TokenFlow (Exp. 4) and Janus (Exp. 5) adopt dual-encoder paradigms. TokenFlow uses discrete representations for both, while Janus uses continuous for understanding and discrete for generation. They show decent reconstruction and understanding, but VQRAE outperforms them.
• VQRAE's Superior Trade-off (Exp. 6): VQRAE (Exp. 6) demonstrates a more favorable trade-off. It achieves competitive reconstruction quality (rFID: 1.31, PSNR: 22.23, SSIM: 0.762) and strong understanding performance (MME-P: 1543.3, SEED: 70.0, TQA: 61.7). Notably, it achieves this with a single tokenizer ($C$ for understanding, $D$ for generation), unlike the dual-encoder TokenFlow and Janus. Its PSNR and SSIM are the highest among all methods, indicating excellent reconstruction. Its understanding scores are on par with RRA (the best for understanding) while offering discrete tokens for generation.

6.1.2. Unified Visual Tokenizers (Table 2)

Table 2 compares VQRAE's reconstruction quality against both generative-only and other unified tokenizers.

The following are the results from Table 2 of the original paper:

Analysis:

• Generative Only Tokenizers: RAE [96], which uses continuous features, achieves the best rFID (0.49), indicating high semantic fidelity, but its PSNR and SSIM are not the highest. VAR and Open-MAGVIT2 show very strong PSNR values.

• Unified Tokenizers: VQRAE (SigLIP2) achieves an rFID of 1.31, PSNR of 22.23, and SSIM of 0.762.

  • It surpasses dual-encoder methods like TokenFlow (rFID: 1.37, PSNR: 21.41, SSIM: 0.690) and MUSE-VL (rFID: 2.26, PSNR: 20.14, SSIM: 0.646) in reconstruction quality, especially SSIM.
  • DualViTok has a slightly higher PSNR (22.53) but a slightly lower SSIM (0.740) and similar rFID (1.37).

• Key Finding: The results validate that VQRAE, despite using pre-trained VFMs (which are ViT-based) as unified encoders and a ViT-based decoder without any convolutional blocks, can achieve competitive reconstruction quality. This supports the observation that VFM continuous features are usable for reconstruction and that discretizing these features (as VQRAE does) can still maintain high fidelity. The visualization results in Figure 5 further illustrate this fine-grained reconstruction ability.

  As can be seen from the results in Figure 5, VQRAE-InternViT version effectively reconstructs diverse images, preserving details in human faces, natural scenes, and objects. The clarity and fidelity of the reconstructed outputs highlight the model's capability in accurately mapping latent features back to pixel space.

  该图像是复原结果的可视化，来源于VQRAEInternViT版本。左侧为输入图像，右侧为输出图像，展示了模型在图像重建方面的表现。

6.1.3. Multimodal Understanding (Table 3)

Table 3 presents a comprehensive comparison of VQRAE against various MLLMs (understanding-only and unified tokenizers) on downstream multimodal understanding benchmarks.

The following are the results from Table 3 of the original paper:

Analysis:

• Performance vs. LLaVA-1.5 Baseline: VQRAE (SigLIP2-so400m, Vicuna-13B, 512px) achieves MME-P of 1543.3 and SEED of 69.9, which is comparable to or slightly better than the LLaVA-1.5 baseline (CLIP-L, Vicuna-13B, 336px) with MME-P of 1531.3 and SEED of 68.1. This is a significant finding, as it demonstrates that VQRAE can maintain or even improve understanding performance even after being optimized for reconstruction and generation.
• Outperforming Other Unified Tokenizers: VQRAE consistently outperforms most other unified tokenizers under similar settings. For instance, comparing 13B models, VQRAE (SigLIP2, Vicuna-13B, 512px) yields MME-P of 1543.3, significantly higher than TokenFlow-L (1365.4) and competitive with TokenFlow-XL (1545.9). VQRAE also surpasses Tar [21] (MME-P: 1571.0 vs VQRAE's 1746.8 with InternViT) which performs semantic distillation directly on discrete tokens, highlighting the benefit of VQRAE's continuous features for understanding and avoiding quantization errors.
• Efficiency and Seamless Integration: The paper notes that VQRAE is more efficient because it doesn't require multimodal alignment or instruction tuning for its pre-trained tokenizers. By simply replacing the ViT encoder in a base model (e.g., InternVL3), VQRAE can be directly applied to downstream tasks without degradation, often leading to improvements. This supports the effectiveness of the two-stage training strategy in preserving understanding.
• High-Resolution Benefit: Comparing VQRAE (SigLIP2, Vicuna-13B) at 256px resolution (MME-P: 1491.1) to 512px resolution (MME-P: 1543.3), there is a clear performance gain with higher resolution, as expected.
• Strongest Variant: The VQRAE with InternViT-300M vision encoder and Qwen2.5-7B LLM backbone performs exceptionally well, achieving MME-P of 1746.8, TQA of 80.6, and MMMU of 61.6. This is competitive with top understanding-only MLLMs like InternVL3 (MME-P: 1748.4) and Qwen2.5-VL (MME-P: 1698.1), demonstrating that VQRAE can serve as a powerful vision backbone for MLLMs.

6.1.4. Visual Generation (Table 4)

Table 4 evaluates VQRAE's visual generation performance on GenEval [20] and DPG-Bench [22] benchmarks, comparing it with both diffusion-based and autoregressive models.

The following are the results from Table 4 of the original paper:

Analysis:

• Competitive Performance for Lightweight Models: VQRAE, with only 0.6B parameters (using Qwen3-0.6B as LLM backbone), shows highly competitive generation capabilities, particularly when compared to other autoregressive models of similar or even larger parameter sizes.

  • On GenEval (Overall $\uparrow$), VQRAE scores 0.76, which is higher than Chameleon (0.54, 7B params), LlamaGen (0.54, 0.8B params), EMU3-Gen (0.54, 8B params), TokenFlow (0.55, 13B params), Janus (0.61, 1.3B params), SimpleAR (0.63, 1.5B params), and Janus-Pro (0.73, 1B params). This indicates strong text-to-image alignment.
  • On DPG-Bench (Overall $\uparrow$), VQRAE scores 86.67, surpassing all listed autoregressive models, including Janus-Pro (82.63), SimpleAR (81.97), and others. This highlights its superior ability in generating images that align semantically with global, entity, attribute, and relation prompts.

• Benefits of Semantic High-Dimensional Latent Space: The strong performance of VQRAE in generation, especially its lightweight variant, suggests that the semantic high-dimensional latent space built on VFMs is beneficial not only for the convergence of diffusion-based models (as shown by RAE [96], SANA-1.5 [79], SD3-Medium [16], DALLE3 [35]) but also significantly improves the training dynamics of autoregressive models in the discrete scaling paradigm.

• Comparison to Diffusion Models: While VQRAE excels among autoregressive models, top-tier diffusion-based models like DALLE3 (0.67 GenEval Overall, 83.50 DPG-Bench Overall) and SD3-Medium (0.74 GenEval Overall, 84.08 DPG-Bench Overall), especially SANA-1.5 (0.81 GenEval Overall, 82.63 DPG-Bench Overall), generally achieve higher GenEval scores. However, VQRAE (0.76 GenEval Overall, 86.67 DPG-Bench Overall) is still highly competitive, especially with DPG-Bench, and stands out for its autoregressive nature and unified approach.

  As can be seen from the results in Figure 8, VQRAE can generate a wide array of images spanning various styles, subjects, and scenarios, demonstrating its versatility in visual generation.

  该图像是一个展示不同生成结果的视觉集，共包含多个主题，如自然、人物与动物等。通过统一的表示方法，图中展示的结果表现出多样化的图像风格和结构，突出VQRAE方法在视觉生成和理解上的应用潜力。

6.2. Ablation Studies / Parameter Analysis

6.2.1. Codebook Dimension and Size (Table 5)

This ablation study investigates the impact of codebook dimension and codebook size on VQRAE's reconstruction quality and utilization ratio.

The following are the results from Table 5 of the original paper:

Analysis of Codebook Dimension:

• The results present a contrary conclusion to previous CNN-based VQ codebook practices, which favored low dimensions.
• For VQRAE (quantizing features from ViT-based VFMs), $low dimensions (≤ 256)$ lead to training non-convergence (NA values).
• As the codebook dimension increases from 384 to 1536 (with fixed size 16384):
  • rFID decreases significantly (from 7.69 to 2.65).
  • PSNR and SSIM increase substantially (e.g., PSNR from 8.24 to 20.14, SSIM from 0.261 to 0.668).
  • The utilization ratio improves dramatically (from 64% to 100%).
• This indicates that a semantic codebook generally requires a larger dimension to effectively capture the richness of VFM features, preventing codebook collapse and ensuring better reconstruction and higher utilization. At 1536 dimensions, VQRAE achieves a 100% utilization ratio.

Analysis of Codebook Size:

• With a fixed codebook dimension of 1920, increasing the codebook size from 4096 to 16384 generally improves reconstruction quality:
  • rFID decreases (from 7.07 to 2.65).
  • PSNR and SSIM increase (e.g., PSNR from 8.02 to 20.14, SSIM from 0.253 to 0.668).
  • The utilization ratio remains high (98% to 100%).
• However, when the codebook size exceeds 16K (e.g., 32768), there's a slight degradation in rFID, PSNR, and SSIM, and the utilization ratio drops slightly (to 96%). This is attributed to the slow convergence of the training process with excessively large codebooks.

6.2.2. Training Strategies (Table 6)

This ablation study investigates the effect of the two-stage training strategy and self-distillation on VQRAE's ability to balance image understanding and reconstruction.

The following are the results from Table 6 of the original paper:

Analysis:

• End-to-End Training without Self-Distillation (Row 1: Two Stage ×, Self-Distillation ×):

  • Achieves the best PSNR (21.35) and SSIM (0.704) for reconstruction among the ablations. However, its performance on understanding tasks is catastrophically low (MME-P: 608.9, MMB: 22.3, TQA: 7.0). This indicates that jointly training all components without explicit constraints to preserve VFM semantics leads to severe degradation in understanding as the encoder shifts its focus heavily towards pixel reconstruction, losing its semantic properties.

• End-to-End Training with Self-Distillation (Row 2: Two Stage ×, Self-Distillation ✓):

  • Introducing self-distillation significantly alleviates the degradation of visual understanding (MME-P: 1435.2, MMB: 64.9, TQA: 42.6), bringing it closer to competitive levels. However, it leads to a slight decrease in reconstruction quality (PSNR: 19.68, SSIM: 0.644) compared to the no-distillation case. This suggests a trade-off: preserving semantics helps understanding but might slightly compromise direct reconstruction when optimized end-to-end.

• Two-Stage Training with Self-Distillation (Row 3: Two Stage ✓, Self-Distillation ✓):

  • This is the proposed VQRAE strategy. It achieves a better balance. While its raw PSNR (20.52) and SSIM (0.680) are slightly lower than the best reconstruction-only case (Row 1), its understanding performance (MME-P: 1439.1, MMB: 65.8, TQA: 44.0) is comparable to, and slightly better than, the end-to-end distillation case (Row 2). The rFID (2.71) is also good.

• Conclusion: The two-stage training strategy, coupled with self-distillation, effectively allows VQRAE to achieve a strong trade-off. It minimizes the update of the pre-trained encoder in the initial phase (Stage 1), allowing the codebook and decoder to focus on reconstruction. Then, in Stage 2, self-distillation guides the encoder's fine-tuning to retain its semantic understanding while further improving reconstruction details. This confirms the importance of this strategic training approach.

  As shown in Figure 6, the two-stage training strategy with self-distillation loss (Stage 2) yields reconstruction results that are both fine-grained and semantically consistent, representing an optimal balance. In contrast, end-to-end training without distillation (E2E) fails to preserve semantic meaning, highlighting the necessity of the proposed training approach.

  该图像是一个示意图，展示了不同训练阶段下的重建效果。图中的四组内容分别为原始图像、阶段1（Stage 1）、阶段2（Stage 2）和端到端训练（E2E），可以看出阶段2的重建在细节和语义保持上更为完善。

6.3. Visualizations

• Figure 1: Comparisons of different unified tokenizers. This figure (conceptual diagram) visually illustrates the architectural differences between dual-encoder paradigms (a), contrastive loss supervision (b), and VQRAE's single-tokenizer approach capable of producing both continuous and discrete tokens (c). It highlights the VQRAE's innovation in unification.

• Figure 2: Showcase of the visual understanding, generation and reconstruction ability of our VQRAE model. This figure (collage of examples) provides qualitative evidence of VQRAE's versatility across the three key tasks, showcasing diverse images for understanding, generated content, and reconstructed outputs.

• Figure 3: VQRAE achieves a superior trade-off with the unified encoder in the autoregressive style. This figure (architectural diagram) is a critical visual aid for understanding the VQRAE model structure (a) and its two-stage training strategy (b). It details the VFM encoder, VQ codebook, symmetric ViT decoder, and the specific loss functions and gradient flows in each stage.

• Figure 4: K-means clustering on the ImageNet-1K validation set. This figure (clustering visualization) provides insight into VQRAE's disentangled representations. It shows that continuous features from the VFM encoder cluster semantically similar objects, while discrete tokens tend to cluster images with similar textures. This supports the claim that the unified tokenizer can provide distinct yet complementary representations, indicating potential redundancy in dual-encoder designs.

• Figure 7: Additional reconstruction results. This figure (collage of reconstructed images) further demonstrates VQRAE's fine-grained reconstruction capabilities across various subjects like human faces, complex scenes, and detailed objects, reinforcing the quantitative metrics.

  该图像是包含多种场景的插图，包括儿童、太空主题、自然景观和美食。每个场景都有独特的元素，展现了创意和多样化的视觉表现。

• Figure 9: Failure cases in reconstruction. This figure (collage of reconstructed images with issues) highlights limitations in VQRAE's reconstruction, particularly regarding text legibility and high-density scenarios. This indicates areas for future improvement.

  该图像是多张示意图组合，展现了不同场景下的多模态理解与生成。每个部分分别呈现了街道标牌、网络安全中心页面、住宅区规划图、家庭游戏场景和药品包装，这些内容强调了多模态表现对图像理解和生成的重要性。

• Figure 10: Failure cases in generation. This figure (collage of generated images with artifacts) shows common failure modes in image generation, such as artifacts in human fingers and faces. This points to challenges that often require additional training or reinforcement learning to mitigate.

  该图像是一个包含多种场景的拼贴画，展现了不同主题的主体，如节日庆祝、艺术创作和烹饪等，体现了多模态表现的特点。

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper successfully introduces VQRAE, a novel Vector Quantization version of Representation AutoEncoders, which pioneers a truly unified tokenizer for multimodal understanding, generation, and reconstruction. VQRAE addresses the long-standing challenge of balancing semantic understanding with fine-grained reconstruction and discrete token generation within a single architecture.

Key contributions include:

• The first exploration of a unified tokenizer that simultaneously produces continuous semantic representations for understanding and fine-grained discrete tokens for generation/reconstruction, eliminating the need for complex dual-encoder paradigms.

• A pure ViT-based model, leveraging pre-trained Vision Foundation Models (VFMs) as the unified encoder and a symmetric ViT decoder, thereby removing dependency on convolutional pixel encoders.

• The discovery and successful training of a high-dimensional VQ codebook (e.g., 1536 dimensions) with a 100% utilization ratio, which is a significant empirical finding contrary to previous CNN-based VQ practices.

• An effective two-stage training strategy, employing self-distillation constraints, to achieve a competitive trade-off between preserving semantic understanding and enhancing reconstruction quality.

  Extensive experiments across multimodal understanding, generation, and reconstruction benchmarks demonstrate VQRAE's competitive performance, showcasing its efficiency, robustness, and promising scaling properties within the autoregressive paradigm due to its discrete merits.

7.2. Limitations & Future Work

The authors acknowledge several limitations of VQRAE and propose directions for future research:

• Trade-off in Understanding and Reconstruction: The primary limitation is the ongoing challenge of finding more effective methods to perfectly balance understanding and reconstruction performance, aiming to minimize any compromise on understanding capabilities.
• Underexplored Synergy: The potential for reconstruction and generation capabilities to actually enhance understanding abilities remains largely underexplored. This could represent a significant area for future work.
• Quantization Loss vs. Continuous VAEs: Due to the inherent quantization loss in discrete tokenizers, VQRAE may find it challenging to fully compete with state-of-the-art continuous VAEs in terms of pure reconstruction quality.
• Generation Quality: There is still room for improvement in generation quality, particularly in handling spatial relationships, texture rendering, and addressing specific artifacts in human faces and fingers (as highlighted in failure cases). These issues often require post-training refinement or advanced techniques.

Future Research Directions: The authors suggest several promising avenues for future exploration:

• Developing methods to integrate various multimodal tasks into a single, cohesive model using VQRAE's representations.
• Investigating the complex interplay of conflicts and synergies among different multimodal tasks.
• Exploring efficient model scaling strategies for unified models.
• Research into advanced post-training techniques, reinforcement learning, and leveraging extensive training data to address generation artifacts and improve output quality.

7.3. Personal Insights & Critique

VQRAE represents a significant step towards truly unified Multimodal Large Language Models (MLLMs). The paper's core innovation lies in successfully bridging the gap between continuous semantic understanding and discrete token-based generation/reconstruction within a single, elegant ViT-based tokenizer.

Personal Insights:

• Elegant Unification: The idea of a single tokenizer providing both continuous and discrete representations is intuitively appealing. It avoids the architectural overhead and conceptual complexity of dual-encoder systems, paving the way for more streamlined MLLMs. The autoregressive paradigm benefits greatly from the discrete nature of the tokens, offering efficiency and scalability advantages.
• Empirical Breakthrough: The finding that semantic encoders (like VFMs) require high-dimensional codebooks for effective VQ and can achieve 100% utilization is a crucial empirical contribution. It challenges a long-held assumption from CNN-based VQ and provides valuable guidance for future VQ tokenizer designs in the era of ViT and Transformers. This property alone makes VQRAE stand out.
• Practical Applicability: The claim that VQRAE can be seamlessly integrated into existing MLLMs by simply replacing the ViT encoder is a powerful demonstration of its practicality. This "plug-and-play" capability, combined with maintained or improved understanding performance, significantly lowers the barrier to adoption for MLLM developers.
• Disentangled Representations: The visualization of K-means clustering (Figure 4) showing distinct semantic and texture-based clusters from continuous features and discrete tokens, respectively, is quite insightful. It provides strong evidence that the unified tokenizer is learning a rich, multi-faceted representation space, further justifying the redundancy of dual-encoder architectures.

Critique & Areas for Improvement:

• "Negligible Semantic Information" Clarification: The abstract states negligible semantic information for maintaining the ability of multimodal understanding, which could be interpreted as the model having little semantic information. The body text clarifies this as negligible semantic information loss. This distinction is important for clarity, and a slight rephrasing in the abstract could avoid potential misinterpretation.

• Quantization Loss Impact: While the paper successfully mitigates quantization errors for understanding by using continuous features, the inherent quantization loss will always be a factor for the discrete tokens. A deeper theoretical analysis of the bounds or characteristics of this specific semantic VQ loss in high-dimensional spaces could be valuable.

• Text Reconstruction and High-Density Scenarios: The failure cases (Figure 9) in text reconstruction and high-density images suggest areas where VQRAE might still struggle with extremely fine-grained details or complex compositional structures. This could be due to the patch-based nature of ViTs or limitations in the codebook's capacity for very specific visual patterns. Specialized techniques for text rendering or enhanced local attention mechanisms could be explored.

• Generative Artifacts: The presence of artifacts in human faces and fingers (Figure 10) is a common challenge in image generation. While the authors suggest post-training or reinforcement learning, future work could explore how the VQRAE's specific architecture might be inherently improved to reduce these issues (e.g., incorporating more localized detail refinement in the decoder).

  Overall, VQRAE provides a compelling and rigorously evaluated solution to a central problem in multimodal AI. Its elegant design and empirical successes make it a valuable contribution, offering a promising foundation for the next generation of unified MLLMs.