1. Bibliographic Information

1.1. Title

End-to-End Speech Recognition Contextualization with Large Language Models

1.2. Authors

The paper is authored by Egor Lakomkin, Chunyang Wu, Yassir Fathullah, Ozlem Kalinli, Michael L. Seltzer, and Christian Fuegen. All authors are affiliated with Meta AI, the artificial intelligence research lab of Meta Platforms, Inc. This affiliation suggests that the research has access to large-scale computational resources and datasets, which is typical for state-of-the-art work in fields like speech recognition and large language models.

1.3. Journal/Conference

The paper was published on arXiv, a popular open-access repository for preprints of scientific papers. As a preprint, it has not yet undergone a formal peer-review process for publication in a conference or journal. However, arXiv is a standard platform in the machine learning community for rapidly disseminating cutting-edge research.

1.4. Publication Year

2023. The paper was submitted to arXiv on September 19, 2023.

1.5. Abstract

The authors propose a new method for incorporating contextual information into speech recognition systems by leveraging Large Language Models (LLMs). Their approach reframes Automatic Speech Recognition (ASR) as a mixed-modal language modeling task. A pretrained LLM is given audio features and optional textual context (like a video title) as input and is trained to generate the corresponding transcription. This end-to-end training implicitly teaches the model to use the unstructured context. The experiments show a significant 6% Word Error Rate (WER) reduction when textual context is provided. Furthermore, their model, named "Speech LLaMA," outperforms a strong baseline contextualized system (an RNN-T model) by 7.5% in overall WER and 17% in rare word WER, despite the baseline being trained on over 25 times more speech data. The authors highlight that this contextual ASR capability is unlocked by adding a very small number of trainable parameters via adapters, while preserving the LLM's original text-only functionality.

1.6. Original Source Link

• Original Source Link: https://arxiv.org/abs/2309.10917
• PDF Link: https://arxiv.org/pdf/2309.10917v1
• Publication Status: This is a preprint available on arXiv and has not been formally peer-reviewed or published in a conference or journal at the time of this analysis.

2. Executive Summary

2.1. Background & Motivation

The core problem this paper addresses is the challenge of contextualization in Automatic Speech Recognition (ASR). ASR systems often struggle to correctly transcribe words that are rare, ambiguous, or domain-specific, such as named entities (e.g., "Zendaya"), technical jargon (e.g., "LoRA adapters"), or proper nouns. Providing the ASR system with external textual context—like the title of a video, a meeting agenda, or a user's contact list—can help it disambiguate these terms and improve accuracy.

However, traditional methods for contextualization have several limitations:

1. Limited Scope: They typically operate at a word or phrase level, "biasing" the model towards a predefined list of terms rather than understanding the broader topic from unstructured text.

2. Complex Tuning: The strength of this biasing often needs to be carefully controlled with hyperparameters to prevent the model from becoming "over-biased" and hallucinating context words that weren't actually spoken.

3. Architectural Constraints: Many methods involve adding specialized modules or only influencing the decoder part of the ASR model, without allowing for a deep, end-to-end interaction between the context, audio, and the full model.

   This paper's innovative entry point is to leverage the powerful contextual understanding and world knowledge inherent in Large Language Models (LLMs). Instead of treating contextualization as a separate add-on, the authors propose a fundamental shift in perspective: they cast ASR itself as a mixed-modal language modeling task. The idea is to feed both audio information and unstructured text context directly to a pretrained LLM and train it to "complete" the sequence by generating the spoken transcript.

2.2. Main Contributions / Findings

The paper's primary contributions and findings are:

1. A Novel Architecture ("Speech LLaMA"): The paper introduces a decoder-only ASR system built upon a pretrained LLM (LLaMA 7B). It integrates audio by processing it through an audio encoder and treating the resulting features as a sequence of "audio tokens" that are concatenated with standard text tokens.

2. End-to-End Contextualization via Prompting: The method allows for contextualization by simply prepending unstructured text (e.g., video title and description) to the audio tokens. The LLM then processes this entire mixed-modal sequence, giving it the flexibility to cross-reference the text context while transcribing the audio, all within a unified next-token prediction framework.

3. High Efficiency and Performance: By freezing the vast majority of the LLM's parameters and only fine-tuning an audio encoder and a small number of LoRA adapters (adding only 30 million trainable parameters to a 6.7 billion parameter model), the method is highly parameter-efficient. Despite this efficiency and being trained on 25 times less speech data, the proposed Speech LLaMA significantly outperforms a massive, 1-billion-parameter industrial baseline system on both overall and rare-word recognition accuracy.

4. Effective Context Utilization: Ablation studies confirm that the model learns to effectively leverage relevant context, ignore noisy or irrelevant context, and even perform phonetic disambiguation when presented with similar-sounding words in the context, demonstrating a sophisticated use of the provided information.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

• Automatic Speech Recognition (ASR): The technology that converts human speech into a sequence of text. Modern ASR systems are based on deep learning and are often "end-to-end," meaning a single neural network learns to map raw audio features directly to text.

• Large Language Models (LLMs): These are massive neural networks (often with billions of parameters) trained on vast quantities of text data from the internet. Their primary training objective is next-token prediction: given a sequence of text, predict the most likely next word or sub-word token. This simple objective enables them to acquire a deep understanding of language, grammar, facts, and reasoning. Examples include OpenAI's GPT series and Meta's LLaMA.

• Decoder-Only Architecture: A type of Transformer architecture used by many modern LLMs like GPT. It consists of a stack of "decoder" blocks. Each block uses self-attention to look at all previous tokens in the sequence to predict the next token. This is "causal" or "auto-regressive" because the model can only attend to past information, not future information, which is ideal for generating text one token at a time.

• RNN-T (Recurrent Neural Network Transducer): A popular architecture for streaming ASR. It consists of three main parts:

  1. Audio Encoder: Processes audio frames and produces a sequence of acoustic representations.
  2. Prediction Network: A recurrent neural network (like an LSTM) that processes the previously predicted text tokens and produces a linguistic representation.
  3. Joint Network: Combines the outputs of the audio encoder and prediction network to produce a probability distribution over the next possible token (including a special "blank" token, which allows the model to advance in time without emitting a character).

• LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning (PEFT) technique. When adapting a huge pretrained model (like an LLM) to a new task, fine-tuning all of its billions of parameters is computationally expensive. LoRA freezes the original model weights ($W$) and injects small, trainable "rank decomposition" matrices ($A$ and $B$) into specific layers (e.g., the attention mechanism). During fine-tuning, only $A$ and $B$ are updated, where the number of parameters in $A$ and $B$ is far less than in $W$. The update is represented as $\Delta W = BA$. This drastically reduces the number of trainable parameters and memory usage.

• Contextual Biasing: The process of providing an ASR model with additional information (context) to improve its accuracy on specific, often difficult, words or phrases. For example, if the context is "medical conference," the ASR system can be biased to favor words like "pharmacology" over the phonetically similar "farm-ecology."

3.2. Previous Works

The paper positions its work in contrast to two main lines of research: traditional ASR contextualization and recent efforts to integrate LLMs with speech.

• Traditional Contextualization:

  • Shallow Fusion: This approach works at the decoding stage. The final score for a word is an interpolation of the score from the main ASR model and a score from an external language model or a specialized structure like a Weighted Finite-State Transducer (WFST). Le et al. [4] describe a method where a WFST is built from biasing strings (e.g., named entities) and combined with an RNN-T's scores. This is flexible as it can be added to any trained ASR model, but the integration is not "deep."
  • Deep Biasing / Deep Context: These methods integrate contextual information directly into the ASR model's architecture and training process. For example, Pundak et al. [6] introduced methods for end-to-end contextual speech recognition. These approaches allow for a tighter coupling between context and acoustics but often require specialized modules and training procedures, and are typically limited to biasing on specific phrases rather than unstructured text.

• LLMs for Speech Tasks:

  • Researchers have recently started using LLMs for speech-related tasks. Wu et al. [12] used LLaMA for speech translation by concatenating an audio representation with a text prompt like "Translate audio to language X."
  • Rubenstein et al. [13] proposed AudioPaLM, a model that can process and generate both text and audio by representing them as a sequence of discrete tokens.
  • Fathullah et al. [14] explored prompting LLMs to unlock ASR capabilities.
  • Radford et al. [15] developed Whisper, which uses the transcription of a previous audio segment as a prompt to improve the transcription of the current segment in long-form speech recognition.

3.3. Technological Evolution

The field of contextual ASR has evolved from post-processing and shallow fusion methods to more integrated deep biasing techniques.

1. Early Stages: Simple find-and-replace or boosting scores of context words during decoding.
2. WFST-based Biasing: A more systematic approach using WFSTs to represent context phrases, combined with ASR model scores during decoding (Shallow Fusion).
3. Deep Biasing: Integrating context directly into the neural network architecture, often using attention mechanisms to attend to a list of biasing phrases.
4. LLM-based Paradigm (This Paper): A fundamental shift away from biasing on lists of phrases to contextualizing on unstructured, free-form text. This leverages the general knowledge and reasoning abilities of LLMs, treating ASR as a language modeling task over mixed (audio + text) modalities.

3.4. Differentiation Analysis

The core innovations of this paper compared to previous work are:

• Unstructured vs. Structured Context: Traditional methods require a structured list of biasing phrases. This paper's method can take raw, unstructured text (e.g., "A documentary about the marine biologist Sylvia Earle") and let the LLM figure out which parts are relevant (e.g., "Sylvia Earle," "marine biologist").
• End-to-End Learning vs. Heuristic Fusion: Instead of heuristically combining scores (like in shallow fusion) or adding specialized biasing modules, this method learns to use context end-to-end through a standard language modeling objective. This eliminates the need for extra hyperparameters to control biasing strength.
• Unified Architecture vs. Separate Components: The Speech LLaMA architecture is simple and unified. Both audio and text are treated as parts of a single input sequence for a decoder-only LLM. This contrasts with more complex architectures that have separate encoders and decoders for different modalities with specialized cross-attention mechanisms.
• Parameter Efficiency: Compared to training a large ASR model from scratch or fine-tuning an entire LLM, the use of LoRA makes this approach highly efficient, enabling powerful capabilities with minimal additional training cost.

4. Methodology

4.1. Principles

The core principle of this work is to re-imagine Automatic Speech Recognition not as a separate audio-to-text conversion task, but as a mixed-modal, conditional language modeling task. The intuition is that a powerful, pretrained Large Language Model (LLM) already possesses immense knowledge about language, topics, and entities. By providing it with both the audio signal and relevant textual context, the LLM can be trained to generate the correct transcription by drawing upon both its internal knowledge and the provided cues.

The system is designed to process a single, continuous sequence of tokens. This sequence starts with optional text tokens (the context), followed by a sequence of "audio tokens" representing the speech. The LLM's task is then simply to predict the subsequent tokens, which correspond to the words spoken in the audio. This decoder-only, auto-regressive setup implicitly incentivizes the model to learn the complex relationships between the textual context, the acoustic information, and the final transcript.

4.2. Core Methodology In-depth (Layer by Layer)

The proposed model, named Speech LLaMA, is depicted in Figure 1 of the paper. It consists of two main parts: an Audio Encoder and a Text Decoder based on a pretrained LLM.

The overall data flow is as follows:

1. Raw audio is converted into log Mel spectrograms.

2. The Audio Encoder processes these spectrograms to produce a sequence of high-level audio representations (audio tokens).

3. External textual context (e.g., video title) is tokenized using the LLM's tokenizer.

4. The text tokens and audio tokens are concatenated to form a single input sequence.

5. This mixed-modal sequence is fed into the LLM decoder, which then auto-regressively generates the transcript tokens.

   The architecture and training process are detailed below.

   该图像是关于如何使用大语言模型（LLM）进行语音识别的示意图。图中展示了音频编码器输出的音频标记、上下文文本标记（如视频标题和描述）以及通过文本-文本注意力机制（Text-text attention）处理后得到的识别文本。

4.2.1. Audio Encoder

The purpose of the audio encoder is to transform the low-level audio features into a sequence of higher-level representations that are compatible with the LLM's embedding space.

• Input: The process starts with 80-dimensional log Mel features, calculated from the audio with a 25ms window and a 10ms step.
• Downsampling and Encoding:
  1. The features first pass through four downsampling blocks, which collectively reduce the temporal resolution by a factor of 16. This makes the sequence much shorter and computationally manageable.
  2. The downsampled features are then fed into a stack of Conformer blocks. The Conformer architecture is a state-of-the-art model for ASR that effectively combines convolutions and self-attention. The paper uses a hidden dimensionality of 512 and a kernel size of 9. Rotary positional embeddings (RoPE) are used to encode positional information.
  3. An additional downsampling block is applied at the end.
• Output: The final output of the audio encoder is a sequence of "audio tokens." Each token has a dimensionality of 4,096, which matches the embedding dimension of the LLaMA 7B model. Due to the total downsampling, each audio token represents a 320ms segment of the original audio.
• Pretraining: To ensure the audio encoder produces meaningful representations before being connected to the LLM, it is pretrained for 300,000 steps on the training data using the Connectionist Temporal Classification (CTC) loss. CTC is a common loss function for ASR that handles the alignment between variable-length audio and text sequences.

4.2.2. Text Decoder (LLM Backbone)

The core of the system is the text decoder, which is responsible for both understanding the mixed-modal prompt and generating the final transcription.

• Base Model: The authors use a pretrained 7B LLaMA (v1) model as the decoder. Crucially, the vast majority of its 6.7 billion parameters are frozen during the ASR fine-tuning stage. This preserves the powerful linguistic and world knowledge learned during its original pretraining.
• Adaptation with LoRA: To adapt the text-only LLM for the speech recognition task, Low-Rank Adapters (LoRA) are inserted into every decoder layer. Specifically, LoRA matrices are added to the query, key, value, and output projection matrices within each self-attention block.
  • Trainable Parameters: Only the audio encoder and these LoRA adapters are trained. The LoRA configuration (rank=32, dropout=5%, scaling=0.05) adds only about 30 million trainable parameters to the decoder. This makes fine-tuning vastly more efficient in terms of memory and computation compared to training the entire 7B model.

4.2.3. Input Formulation and Training

The key to the method is how the inputs are combined and how the model is trained.

• Input Concatenation: For a given audio sample, the input sequence for the LLM is constructed as follows:
  1. The textual context (concatenated video title and description) is tokenized. The paper limits this to a maximum of 50 tokens.
  2. A special beginning-of-sequence token ($<bus>$) is prepended.
  3. The sequence of audio tokens from the audio encoder is appended. The final input looks like: $[<bus>, text_token_1, ..., text_token_N, audio_token_1, ..., audio_token_M]$
• Training Objective: The model is trained using a standard cross-entropy loss for next-token prediction, the same objective used to train LLMs. However, the loss calculation is masked:
  • Loss Masking: The loss is not computed for the predictions made at the positions of the input context tokens and audio tokens. The loss is only calculated for the tokens corresponding to the ground-truth transcription, which the model is expected to generate after processing the entire input prompt.
  • This forces the model to learn to generate the transcript conditioned on both the text context and the audio content. When no textual context is available, the input simply consists of the audio tokens, and the model functions as a standard non-contextual ASR system.

5. Experimental Setup

5.1. Datasets

• Training Data: The models are trained on a large, in-house dataset of 150,000 hours of English speech. This data is derived from public Facebook and Instagram videos and has been de-identified to remove Personally Identifiable Information (PII).
  • Data Augmentation: To improve robustness, the training data is augmented using two common techniques:
    1. Speed Perturbation: The speed of the audio is slightly altered (e.g., played at 0.9x or 1.1x speed).
    2. Additive Noise: Randomly sampled background noise is added to the audio clips.
• Contextual Information: For contextualization, the video title and video description associated with the videos are used. Approximately 25% of the videos in the training set have this non-empty text context. If the context is longer than 50 tokens, it is randomly cropped during training and truncated from the beginning during inference.
• Evaluation Data: The evaluation set consists of 3,200 videos, totaling about 34 hours of speech. This set was specifically curated to test the model's contextualization abilities: each video has a context of at least 100 characters, and the corresponding transcript contains at least one "rare word" that also appears in the context.

5.2. Evaluation Metrics

The paper uses two variants of Word Error Rate (WER) to measure performance.

5.2.1. Word Error Rate (WER)

• Conceptual Definition: WER is the standard metric for evaluating ASR systems. It measures the number of errors a model makes in transcribing speech, calculated by comparing the model's output (hypothesis) to a ground-truth reference transcript. The errors are categorized into three types: substitutions (wrong word), deletions (missed word), and insertions (extra word). A lower WER indicates higher accuracy.
• Mathematical Formula: The WER is calculated using the Levenshtein distance at the word level. $ WER = \frac{S + D + I}{N} $
• Symbol Explanation:
  • $S$: The number of substitutions, where a word in the reference is replaced by a different word in the hypothesis.
  • $D$: The number of deletions, where a word in the reference is missing from the hypothesis.
  • $I$: The number of insertions, where a word appears in the hypothesis but not in the reference.
  • $N$: The total number of words in the reference transcript.

5.2.2. Rare WER

• Conceptual Definition: This is the same as WER, but it is calculated only on a specific subset of words defined as "rare." This metric is crucial for evaluating contextualization because the primary benefit of context is often to help the model recognize rare words (like names or jargon) that it might otherwise miss.
• Definition of Rare Word: In this paper, a word is considered rare if it does not belong to the 90th percentile of the most frequent words in the training dataset's vocabulary.

5.3. Baselines

The paper compares its Speech LLaMA model against a very strong industrial-grade baseline system.

• Model: A 1-billion-parameter Transformer-based RNN-T system. This is a large and powerful ASR architecture. The encoder has 60 Transformer layers, and the decoder has 3 LSTM layers.
• Training Data: This baseline was trained on 4 million hours of supervised and semi-supervised speech data. This is more than 25 times larger than the 150k hours used to train Speech LLaMA.
• Contextualization Method: The baseline uses a state-of-the-art contextualization technique combining WFST-based biasing with neural language modeling shallow fusion [4]. The biasing WFST is constructed from the same video title and description information that Speech LLaMA uses, ensuring a fair comparison of how effectively the context is utilized.

6. Results & Analysis

6.1. Core Results Analysis

The main results, presented in Table 1, compare the proposed Speech LLaMA with the powerful RNN-T baseline under different conditions. The analysis reveals several key insights about the effectiveness of the LLM-based approach.

The following are the results from Table 1 of the original paper:

• Superiority of the LLM Backbone: Even without any contextual information, the Speech LLaMA model (trained on just 150k hours) achieves a WER of 11.70%. This is a 5.2% relative improvement over the non-contextual 1B RNN-T baseline (12.34% WER), which was trained on 4 million hours of data. This demonstrates the immense power of leveraging a pretrained LLM as a decoder, as its rich linguistic knowledge provides a significant advantage.

• Effective Contextualization: When context is provided during both training and evaluation, Speech LLaMA's WER drops to 11.22%. This represents a 6.3% relative reduction compared to the same model evaluated without context (11.98% WER), confirming that the model successfully learns to use the provided information.

• Dominance in Rare Word Recognition: The most impressive result is the improvement on Rare WER. The contextual Speech LLaMA achieves a Rare WER of 23.88%, which is a 17.5% relative improvement over the contextualized RNN-T baseline (28.96%). This highlights that the LLM-based approach is substantially better at leveraging context to recognize difficult, out-of-vocabulary, or domain-specific terms.

• Efficiency: The Speech LLaMA model achieves these superior results with only 130M trainable parameters (Audio Encoder + LoRA) and 25x less speech training data, making it a far more efficient solution than training a massive RNN-T model from scratch.

6.2. Ablation Studies / Parameter Analysis

The authors conduct several ablation studies to better understand the model's behavior and architectural choices.

6.2.1. Context Sensitivity

This study, summarized in Table 2, investigates how the model reacts to different types of context during inference to probe its understanding.

The following are the results from Table 2 of the original paper:

• Robustness to Irrelevant Context: When the original context is replaced with random words, the WER (12.07%) is nearly identical to when context is removed entirely (11.98%). This indicates the model is robust and can effectively ignore irrelevant or noisy context.
• "Copying" Capability: When the context is seeded with ground truth rare words from the transcript, the WER drops significantly to 10.50% (a 6.4% relative improvement) and Rare WER plummets to 19.54% (an 18.2% relative improvement). This confirms the model is highly effective at identifying and "copying" relevant entities from the context when they are present.
• Phonetic Disambiguation: The experiments with phonetic respellings are particularly insightful.
  • When a context word (e.g., "ball") is replaced with a phonetic competitor (e.g., "bawl"), the performance degrades significantly (WER 11.89%), almost as if there were no context. This suggests the model relies heavily on exact string matching.
  • However, when the competitor is appended to the context (i.e., both "ball" and "bawl" are present), the performance drop is much smaller (WER 11.46%). This indicates that when faced with ambiguity, the model can use the acoustic signal to disambiguate between the competing context words and choose the correct one.

6.2.2. Architectural Choices

The authors briefly explore two alternative architectural designs, with results presented in Table 3.

The following are the results from Table 3 of the original paper:

• Causal vs. Full Masking: The standard decoder-only model uses causal masking for the entire input sequence. The authors test an alternative where the context and audio tokens can attend to each other bidirectionally (full mask), while the generated transcript remains causal. This yields a marginal improvement (11.22% → 11.15% WER) but comes at a 10% training slowdown due to less optimized implementations.
• Decoder-only vs. Encoder-decoder: They also compare their concatenated decoder-only approach to a more traditional encoder-decoder setup where the LLM decoder attends to the audio encoder's output via cross-attention layers. The encoder-decoder model performs slightly better (11.22% → 11.18% WER), but the difference is minimal. This finding validates that the simpler and more elegant decoder-only concatenation approach is a highly effective and viable method for mixed-modal ASR.

7. Conclusion & Reflections

7.1. Conclusion Summary

The paper successfully presents Speech LLaMA, a novel and effective method for end-to-end contextual speech recognition using a pretrained Large Language Model. By framing ASR as a mixed-modal language modeling task—where audio representations and unstructured text context are fed as a prompt to an LLM—the authors demonstrate a system that is both powerful and efficient. The key findings are:

1. The proposed method significantly outperforms a strong, industrial-scale RNN-T baseline that was trained on over 25 times more data.
2. The performance gains are particularly large for recognizing rare words, with a 17% relative Rare WER reduction over the baseline, proving the model's superior ability to utilize context.
3. The system is highly parameter-efficient, achieving these results by freezing the pretrained LLM and only training an audio encoder and a small set of LoRA adapters.
4. Ablation studies confirm the model is robust to noisy context and capable of phonetic disambiguation, indicating a sophisticated level of learned behavior.

7.2. Limitations & Future Work

The authors acknowledge some limitations and propose directions for future research.

• Stated Limitations:

  • There is a minor performance degradation when the model trained with context is evaluated without it (11.98% WER), compared to a model trained from the start without context (11.70% WER). The authors suggest that adding "context jitter" during training might improve generalization.
  • The authors note that the quadratic complexity of self-attention in the decoder-only approach can be a bottleneck for very long sequences (long audio + long context).

• Future Work:

  • The primary direction for future work is to extend the method to handle long contexts and long-form audio, potentially using linear attention approximations or other techniques to manage the computational cost.
  • The authors also plan to explore extending the approach to other modalities beyond audio and text.

7.3. Personal Insights & Critique

This paper is a compelling example of the "foundation model" paradigm, showcasing how a general-purpose model like LLaMA can be effectively adapted to a specialized, multi-modal task with remarkable success and efficiency.

• Inspirations and Strengths:

  • Elegance and Simplicity: The core idea of concatenating audio and text tokens into a single sequence for a standard LLM is elegant. It unifies ASR and contextualization under the single, well-understood principle of next-token prediction, removing the need for complex, task-specific architectural components.
  • Data and Compute Efficiency: The most striking aspect is achieving superior performance with drastically less task-specific training data (150k vs. 4M hours). This highlights the power of transfer learning from massive text corpora and suggests a more sustainable path for developing high-performance models.
  • Strong Empirical Validation: The comparison against a powerful industrial baseline and the detailed ablation studies provide strong evidence for the method's effectiveness and robustness.

• Potential Issues and Areas for Improvement:

  • Nature of "Understanding": The ablation studies suggest that a significant portion of the gain comes from the model's ability to "copy" exact strings from the context. While the phonetic disambiguation experiment shows it's more than just a simple string search, the depth of the model's semantic understanding of the context remains an open question. Is it truly understanding the topic, or is it just getting very good at finding keywords?
  • Short Context Limitation: The experiments are limited to a context of 50 tokens. Real-world applications might involve much longer documents (e.g., meeting transcripts, scientific papers). The model's performance and the viability of the simple concatenation approach with extremely long contexts are unverified.
  • Reproducibility: The use of a large, in-house dataset makes the results difficult to reproduce externally. While standard practice in industrial research, it limits the ability of the broader academic community to build directly upon this work.
  • Modality Gap: The paper treats audio features as just another type of token. While effective, this may not be the optimal way to fuse information from two very different modalities. Further research into more sophisticated fusion techniques within the LLM framework could yield even better results.