1. Bibliographic Information

• Title: MUSE: Mamba Is Efficient Multi-scale Learner for Text-video Retrieval
• Authors: Haoran Tang, Meng Cao, Jinfa Huang, Ruyang Liu, Peng Jin, Ge Li, Xiaodan Liang.
• Affiliations: The authors are affiliated with Peking University, Peng Cheng Laboratory, and Sun Yat-sen University. These are prominent research institutions, particularly in the field of computer vision and AI in China.
• Journal/Conference: The paper provides an arXiv link, indicating it is a preprint. The specific conference or journal where it was submitted or published is not mentioned in the provided text, but the format is typical of top-tier AI conferences like CVPR, ICCV, or NeurIPS.
• Publication Year: The arXiv preprint link indicates 2024.
• Abstract: The paper addresses the task of Text-Video Retrieval (TVR), which involves finding relevant videos for a given text query. The authors observe that most existing methods, built on the CLIP model, fail to leverage multi-scale visual representations, which can provide richer contextual information. To solve this, they propose MUSE (Mamba is efficient mUlti-ScalE learner). MUSE first generates multi-scale features from a single-scale feature map using a feature pyramid. Then, it uses the Mamba architecture, known for its linear computational complexity, to efficiently model the relationships across these different scales. The paper includes extensive experiments on three benchmarks (MSR-VTT, DiDeMo, ActivityNet) to validate MUSE's superiority.
• Original Source Link: The paper provides a link to an extended version on arXiv: https://arxiv.org/abs/2408.10575. It also gives a code repository link: https://github.com/hrtang22/MUSE. The provided PDF is /files/papers/68f0e43e1e3a059405607731/paper.pdf.

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: The primary task is Text-Video Retrieval (TVR), which is fundamental to multi-modal understanding. State-of-the-art TVR models heavily rely on pre-trained vision-language models like CLIP. However, CLIP has a "plain" or non-hierarchical structure, meaning it processes visual information at a single, fixed resolution throughout its layers.
  • Gap in Prior Work: This single-scale nature prevents models from capturing multi-scale visual details. For instance, small but crucial objects (like a "torch" in a video) might be missed at a coarse resolution, while the overall scene context might be lost at a very fine resolution. Existing methods focus on fine-grained alignment (e.g., frame-word) but not on cross-resolution alignment.
  • Innovation: The paper introduces two key ideas:
    1. Generating Multi-scale Features: Instead of redesigning the entire backbone, it proposes a simple and efficient way to create a feature pyramid from the final layer of a standard CLIP model.
    2. Efficiently Modeling These Features: Modeling interactions across multiple scales can be computationally prohibitive, especially with standard Transformer attention which has quadratic complexity. The paper proposes using Mamba, a State Space Model (SSM) with linear complexity, as an efficient and effective learner for these multi-scale features.

• Main Contributions / Findings (What):

  • Novel Model (MUSE): The paper proposes MUSE, a framework that integrates multi-scale feature generation with a Mamba-based learner for TVR. MUSE is designed as a plug-and-play module that can enhance existing CLIP-based TVR models.
  • Efficiency and Effectiveness of Mamba: The authors demonstrate that Mamba is not only significantly more memory-efficient than the Transformer for modeling long sequences of multi-scale features but also achieves superior performance. This provides strong empirical evidence for Mamba's suitability as a cross-resolution context learner.
  • State-of-the-Art Performance: By integrating MUSE into existing SOTA models, the paper achieves new state-of-the-art results on three widely-used TVR benchmarks: MSR-VTT, DiDeMo, and ActivityNet.
  • Comprehensive Exploratory Studies: The paper provides a thorough investigation into various design choices, including different feature aggregation strategies (scale-wise, frame-wise, spatial-wise), Mamba scan strategies, scale combinations, and model architectures, offering valuable insights for future research in this area.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • Text-Video Retrieval (TVR): A cross-modal task where the goal is to retrieve the most relevant video from a large collection given a natural language query, and vice-versa. This is typically framed as a similarity matching problem in a shared embedding space.
  • CLIP (Contrastive Language-Image Pre-training): A powerful model from OpenAI trained on a massive dataset of image-text pairs from the internet. It learns to map images and text into a joint embedding space where corresponding pairs are close to each other. Its strong generalization ability has made it a popular backbone for many vision-language tasks, including TVR.
  • Vision Transformer (ViT): A model that applies the Transformer architecture, originally designed for text, to image data. It does this by splitting an image into a sequence of fixed-size patches, embedding them, and feeding them to a Transformer encoder. CLIP uses a ViT as its visual encoder.
  • Multi-scale Representations: Representing an image or video at multiple resolutions or scales. Low-resolution features capture global context and coarse shapes, while high-resolution features capture fine-grained details and textures. This is a standard technique in computer vision, especially for tasks like object detection and segmentation.
  • Feature Pyramid: A common method to generate multi-scale features. It typically starts with a high-resolution feature map and progressively downsamples it to create a "pyramid" of feature maps at different scales.
  • State Space Models (SSM): A class of models from classical control theory used to model sequences. They maintain a hidden state that evolves over time based on the current input. Recent work has adapted them for deep learning, showing promise in modeling long-range dependencies.
  • Mamba: A recent deep learning architecture based on SSMs. Its key innovation is a selective scan mechanism that allows the model to selectively focus on or ignore parts of the input sequence. This makes it data-dependent and powerful for modeling complex sequences. Crucially, it has linear time complexity with respect to sequence length, unlike the quadratic complexity of standard Transformers.

• Previous Works:

  • Text-Video Retrieval: Early works focused on designing custom architectures. More recent works, such as CLIP4clip, X-CLIP, and DiCoSA, have shifted to fine-tuning CLIP for the video domain. They primarily improve performance by exploring fine-grained alignments (e.g., matching frames to words or video clips to phrases) but overlook the potential of multi-scale visual features due to CLIP's plain structure.
  • Multi-scale Video Modeling: Methods like SlowFast explore multi-scale in the temporal domain (using different frame rates). Others, like ViTDet and ViT-Adapter in the image domain, show how to build feature pyramids from plain ViT backbones. However, these methods often don't jointly model the correlations across the different scales in a holistic and efficient manner.
  • Mamba for Vision/Video: Following its success in language, Mamba has been adapted for vision (Vim, VMamba). VideoMamba and Video mamba suite have shown its efficiency and effectiveness for video understanding tasks. This paper builds on this trend, specifically applying Mamba to the new problem of learning from multi-scale representations in TVR.

• Differentiation: MUSE differentiates itself by being the first to systematically address the lack of multi-scale modeling in CLIP-based TVR. While others focused on temporal or semantic granularity, MUSE introduces spatial resolution granularity. Its key innovation is coupling a simple feature pyramid generator with the highly efficient Mamba architecture to model cross-resolution dependencies, a task for which the standard Transformer is computationally too expensive.

4. Methodology (Core Technology & Implementation)

The MUSE pipeline consists of four main stages: feature extraction, multi-scale feature generation, aggregation, and learning with ResMamba.

该图像是MUSE模型的整体架构示意图。它展示了视频输入经 Backbone 和 Multi-Scale Generator 产生多尺度特征。这些特征再由 Multi-Scale Aggregator 聚合，并送入堆叠 L 次的 ResMamba 模块处理输出。下方详细描绘了尺度内、帧间和空间维度的三种特征聚合策略。

• Principles: The core idea is that different semantic concepts in a text query correspond to visual elements of varying scales in a video. By explicitly generating and modeling features from multiple resolutions, the model can create a more comprehensive representation that aligns better with the text. Mamba is used to make this process computationally feasible.

• Steps & Procedures:

  1. Feature Extraction:

     • Given a video $v$ and a text query $t$, the model uses a pre-trained CLIP model with a ViT backbone to extract initial features.
     • For the text, the representation of the [EOT] (End of Text) token is used.
     • For the video, instead of just using the frame-wise [CLS] token (which summarizes the whole frame), MUSE utilizes all visual patch tokens. This provides a rich, spatially-aware feature map. The initial video features are denoted as $\pmb{f} \in \mathbb{R}^{T \times N \times C}$, where $T$ is the number of frames, $N$ is the number of patches (visual tokens), and $C$ is the feature dimension.

  2. Multi-scale Feature Generation:

     • Starting from the base feature map $\pmb{f}$, a feature pyramid is constructed to generate multi-scale features $\pmb{f}_t^s$ for each frame $t$ at each scale $s$.
     • This is achieved using simple convolution and pooling operations, following the spirit of ViTDet. The formula is: $$\pmb { f } _ { t } ^ { s } = \operatorname { P o o l } \big ( \operatorname { C o n v } ( \pmb { f } ) \big ) .$$
     • Symbol Explanation:
       • $\pmb{f}$: The single-scale feature map from the ViT backbone.
       • $\operatorname{Conv}(\cdot)$: A convolution operation to process the feature map.
       • $\operatorname{Pool}(\cdot)$: A pooling operation (e.g., max or average pooling) to downsample the feature map and create different scales.
       • $\pmb{f}_t^s$: The resulting feature representation for the $t$-th frame at scale $s$.

  3. Multi-scale Feature Aggregation:

     • The multi-scale features from all frames need to be arranged into a single 1D sequence to be fed into the Mamba learner. The paper explores three strategies, illustrated in Image 2:
     • (a) Scale-wise (Best performing): Tokens are grouped by scale first, and then by time. All tokens from the lowest resolution scale are concatenated, followed by all tokens from the next resolution, and so on. $$\begin{array} { l } { { { \pmb f } ^ { s } = \{ { \pmb f } _ { 1 } ^ { s } , { \pmb f } _ { 2 } ^ { s } , \cdot \cdot \cdot , { \pmb f } _ { T } ^ { s } \} } } \\ { { { \pmb v } ^ { a } = \{ { \pmb f } ^ { s } \} _ { s = 1 } ^ { S } . } } \end{array}$$
     • (b) Frame-wise: Tokens are grouped by frame first, and then by scale. For each frame, tokens from all scales are concatenated, and then these frame-level blocks are concatenated temporally. $$\begin{array} { l } { { \pmb f } _ { t } = \{ { \pmb f } _ { t } ^ { 1 } , { \pmb f } _ { t } ^ { 2 } , \cdot \cdot \cdot , { \pmb f } _ { t } ^ { S } \} } \\ { { \pmb v } ^ { a } = \{ { \pmb f } _ { t } \} _ { t = 1 } ^ { T } . } \end{array}$$
     • (c) Spatial-wise: Temporal information is first collapsed by mean pooling across the time dimension for each scale. Then, the resulting spatial-only features are concatenated by scale. $$\begin{array} { l } { { { \pmb f } ^ { s } = \mathrm { meanpool } ( { \pmb f } _ { 1 } ^ { s } , { \pmb f } _ { 2 } ^ { s } , \cdots , { \pmb f } _ { T } ^ { s } ) } } \\ { { { \pmb v } ^ { a } = \{ { \pmb f } ^ { s } \} _ { s = 1 } ^ { S } . } } \end{array}$$
     • The final aggregated sequence is denoted by $\pmb{v}^a$.

  4. Mamba As Video Learner (ResMamba):

     • The aggregated sequence $\pmb{v}^a$ is processed by a stack of $L$ ResMamba blocks.
     • Each block consists of a bidirectional Mamba layer followed by a gated residual connection. The formulation is: $$v ^ { o } = \operatorname { ResMamba } ( v ^ { a } ) .$$
     • The core Mamba SSM operates via a recurrence: $$\pmb { h } _ { l } = \mathbf { A } \pmb { h } _ { l - 1 } + \mathbf { B } \pmb { v } _ { l } ^ { a }$$ $$\pmb { y } _ { l } = \mathbf { C } \pmb { h } _ { l }$$
     • Symbol Explanation:
       • $\pmb{v}_l^a$: The input token at step $l$.
       • $\pmb{h}_l$: The hidden state at step $l$.
       • $\mathbf{A}, \mathbf{B}, \mathbf{C}$: State matrix, input projection, and output projection, respectively. These are learned parameters that are functions of the input data in Mamba's selective scan mechanism.
       • $\pmb{y}_l$: The output at step $l$.
     • The output $\pmb{y}_l$ is then passed through a gated residual connection: $$\pmb { v } _ { l + 1 } ^ { a } = \mathcal { G } ( \mathrm { Norm } ( \pmb { y } _ { l } ) ) + \pmb { v } _ { l } ^ { a }$$
     • Symbol Explanation:
       • $\operatorname{Norm}(\cdot)$: A normalization layer (e.g., LayerNorm).
       • $\mathcal{G}(\cdot)$: A gating function, implemented as a single linear layer with zero initialization. This helps stabilize training.
       • The addition provides the residual connection, which is crucial for training deep networks.

  5. Optimization:

     • The final video representation $v^o$ is aggregated (e.g., by mean pooling) to get a single vector $\boldsymbol{v}$.
     • The model is trained using a standard contrastive, symmetric cross-entropy loss to align the video vector $\boldsymbol{v}$ with its corresponding text vector $\boldsymbol{t}$ and push it away from unmatched text vectors $t^-$. $$\mathcal { L } = - \log \frac { \exp ( { \boldsymbol { v } } \cdot { \boldsymbol { t } } / { \tau } ) } { \sum _ { t ^ { - } } \exp \left( { \boldsymbol { v } } \cdot { \boldsymbol { t } } ^ { - } / { \tau } \right) }$$
     • Symbol Explanation:
       • $\boldsymbol{v} \cdot \boldsymbol{t}$: The cosine similarity between the final video and text embeddings.
       • $\tau$: A learned temperature parameter that scales the logits.
       • $t^-$: A negative (unmatched) text sample from the same batch.

5. Experimental Setup

• Datasets:

  • MSR-VTT: Contains 10,000 YouTube videos, each with 20 text descriptions. It's a standard benchmark for TVR. The 1k-A split is used (9k train, 1k test).
  • ActivityNet: A large-scale dataset with 20,000 untrimmed videos of human activities, averaging two minutes long. It's more challenging due to the longer video durations. The val1 split is used.
  • DiDeMo: Consists of over 10,000 unedited personal videos with diverse content.

• Evaluation Metrics:

  • Recall at K (R@K):
    1. Conceptual Definition: This metric measures the percentage of queries for which the correct (ground-truth) video is found within the top K retrieved results. A higher R@K is better. It is the most common metric for retrieval tasks.
    2. Mathematical Formula: $$\mathrm{R@K} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}(\text{rank}_i \le K)$$
    3. Symbol Explanation:
       • $N$: The total number of queries.
       • $\text{rank}_i$: The rank of the correct item for the $i$-th query.
       • $\mathbb{I}(\cdot)$: The indicator function, which is 1 if the condition inside is true, and 0 otherwise.
  • Median Rank (MdR):
    1. Conceptual Definition: This is the median of the ranks of all correct items across all queries. A lower MdR is better, indicating that the correct item typically appears higher in the ranking list. It is less sensitive to outliers than the mean rank.
    2. Mathematical Formula: $$\mathrm{MdR} = \text{median}(\{\text{rank}_1, \text{rank}_2, \dots, \text{rank}_N\})$$
    3. Symbol Explanation:
       • $\{\text{rank}_1, \dots, \text{rank}_N\}$: The set of ranks for all ground-truth items.
  • Mean Rank (MnR):
    1. Conceptual Definition: This is the average of the ranks of all correct items. Like MdR, a lower MnR is better.
    2. Mathematical Formula: $$\mathrm{MnR} = \frac{1}{N} \sum_{i=1}^{N} \text{rank}_i$$
    3. Symbol Explanation: Same as above.

• Baselines:

  • For the plug-and-play experiments, MUSE is added to CLIP4clip, EMCL-Net, STAN, and T-MASS.
  • For state-of-the-art comparisons, MUSE is compared against a comprehensive list of strong methods including X-Pool, HBI, DiffusionRet, and T-MASS.

6. Results & Analysis

• Core Results:

  • Plug-and-play Capability: Table 1 shows that MUSE consistently improves the performance of four different baseline models on MSR-VTT. For example, it boosts the R@1 score of CLIP4clip by +2.2 and T-MASS by +0.9 in text-to-video retrieval. The improvements are particularly large in video-to-text retrieval for EMCL-Net (+3.0) and T-MASS (+3.7), suggesting MUSE effectively enhances the video representation.

    (Manual transcription of Table 1)

    Methods	Text->Video			Video->Text
    	R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑
    CLIP4Clip† (Luo et al. 2022)	42.6	70.8	79.9	43.9	70.0	81.4
    + MUSE (Ours)	44.8 (+2.2)	71.6 (+0.8)	82.1 (+2.2)	44.9 (+1.0)	70.8 (+0.8)	82.2 (+0.8)
    EMCL-Net† (Jin et al. 2022)	47.1	72.7	82.3	44.4	72.6	82.6
    + MUSE (Ours)	48.8 (+1.7)	74.1 (+1.4)	83.4 (+1.1)	47.4 (+3.0)	75.8 (+3.2)	82.9 (+0.3)
    STAN† (Liu et al. 2023)	46.2	72.6	81.1	44.5	71.9	81.7
    + MUSE (Ours)	47.3 (+1.1)	73.1 (+0.5)	82.2 (+1.1)	45.5 (+1.0)	73.1 (+1.2)	81.8 (+0.1)
    T-MASS† (Wang et al. 2024)	50.0	75.3	84.2	46.0	77.1	86.2
    + MUSE (Ours)	50.9 (+0.9)	76.7 (+1.5)	85.6 (+1.4)	49.7 (+3.7)	77.8 (+0.7)	86.5 (+0.3)

  • State-of-the-Art Comparison: Tables 2 and 3 show that by adding MUSE to a strong baseline (T-MASS or DiffusionRet), the resulting model achieves new SOTA performance on all three datasets. On MSR-VTT (Table 2), MUSE achieves an R@1 of 50.9 (text-to-video) and 49.7 (video-to-text). On DiDeMo and ActivityNet (Table 3), it also surpasses all previous methods.

    (Manual transcription of Table 2: MSR-VTT)

    Methods	Text → Video					Video → Text
    	R@1↑	R@5↑	R@10↑	MdR↓	MnR↓	R@1↑	R@5↑	R@10↑	MdR↓	MnR↓
    CLIP4Clip(Luo et al. 2022)	44.5	71.4	81.6	2.0	15.3	42.7	70.9	80.6	2.0	11.6
    X-Pool(Gorti et al. 2022)	46.9	72.8	82.2	2.0	14.3	44.4	73.3	84.0	2.0	9.0
    ... (other methods omitted for brevity) ...
    T-MASS(Wang et al. 2024)	50.2	75.3	85.1	1.0	11.9	47.7	78.0	86.3	2.0	8.0
    MUSE (Ours)	50.9	76.7	85.6	1.0	10.9	49.7	77.8	86.5	2.0	7.4

    (Manual transcription of Table 3: DiDeMo and ActivityNet)

    	DiDeMo					ActivityNet
    Methods	R@1↑	R@5↑	R@10↑	MdR↓	MnR↓	R@1↑	R@5↑	R@10↑	MdR↓	MnR↓
    ... (other methods omitted for brevity) ...
    T-MASS(Wang et al. 2024)	50.9	77.2	85.3	1.0	12.1	-	-	-	-	-
    MUSE(Ours)	51.5	77.7	86.0	1.0	11.3	46.2	76.9	86.8	2.0	5.8

• Ablations / Parameter Sensitivity:

  • Why use Mamba? This is a central question answered by Figure 3 and Table 4.

    • Memory Efficiency (Figure 3): The plot starkly illustrates the quadratic vs. linear complexity. As the number of input frames increases, the Transformer's memory usage explodes, becoming infeasible (OOM) around 16 frames. In contrast, Mamba's memory usage grows linearly and remains manageable, saving 79.9% of memory at 16 frames.

      该图像是 Figure 3，对比了 Transformer、Mamba 和 Baseline 的内存使用量。图中显示，随着帧数的增加，Transformer 的内存使用量呈指数级增长，在约15.5帧时达到75GB并超出内存限制（OOM!）。相比之下，Mamba 和 Baseline（标记为None）模型的内存使用量增长缓慢，Mamba 尤其在内存效率上表现出显著优势。在特定点，Mamba 相较于 Transformer 节省了约79.9%的内存。

    • Performance (Table 4): Mamba not only is efficient but also outperforms other architectures. It achieves the highest R@1 (44.8) compared to Transformer (43.0), FlashAttention (42.6), and MambaOut (42.4). The poor performance of MambaOut (which removes the core SSM module from Mamba) confirms that the selective scan mechanism is crucial for its effectiveness.

      (Manual transcription of Table 4)

      Module	R@1↑	R@5↑	R@10↑	MnR↓	Memory(GB)↓
      Transformer	43.0	71.1	80.0	16.3	36.80
      FlashAttention	42.6	69.3	79.7	16.3	2.38
      MambaOut	42.4	70.2	80.7	15.4	3.28
      Mamba	44.8	71.6	82.1	15.6	3.40

  • Scan Strategies (Table 5): The ablation on scan strategies shows that a bidirectional scan with separate weights ($v2$) performs best. This suggests that processing the multi-scale sequence from both low-to-high resolution and high-to-low resolution is beneficial, and using separate projection weights for each direction allows the model to learn distinct patterns.

    (Manual transcription of Table 5)

    Scan Type	R@1↑	R@5↑	R@10↑	MnR↓
    "none"	44.1	71.7	80.5	14.4
    "v1"	44.0	71.0	80.7	14.9
    "v2"	44.8	71.6	82.1	15.6

  • Scale Combination:

    • Aggregation Manners (Table 6): The Scale-wise aggregation manner significantly outperforms Frame-wise and Spatial-wise. This is a key finding: it's more effective to first process all temporal information within a single scale before moving to the next scale. This allows the Mamba model to learn scale-specific temporal dynamics first. (Manual transcription of Table 6)

      Agg. Mode	R@1↑	R@5↑	R@10↑	MnR↓
      Frame	43.5	70.6	80.0	15.2
      Spatial	44.4	70.3	80.9	15.4
      Scale	44.8	71.6	82.1	15.6

    • Scale Selection (Table 7): Performance generally improves as more scales are added, from {1} up to {1, 3, 7, 14}. However, adding an even larger scale (28) hurts performance and dramatically increases memory usage. This indicates a trade-off: more scales add richer information but can also introduce noise and redundancy, alongside a heavy computational cost. The choice {1, 3, 7, 14} provides the best balance. (Manual transcription of Table 7)

      Scale	Memory(GB)↓	R@1↑	R@5↑	R@10↑	MnR↓
      {1}	7.62	43.6	71.2	81.8	15.2
      {1, 3}	7.82	44.0	70.8	81.2	15.8
      {1, 3, 7}	8.76	44.3	71.8	81.7	15.9
      {1, 3, 7, 14}	12.60	44.8	71.6	82.1	15.6
      {1, 3, 7, 14, 28}	29.36	42.5	71.4	81.6	15.1

  • Layer Numbers (Table 8): Performance steadily increases with more ResMamba layers, from $L=0$ (no MUSE) to $L=16$. This demonstrates that the ResMamba architecture is effective and well-designed for this task. The authors choose $L=4$ as a good trade-off between performance and computational cost. (Manual transcription of Table 8)

    Layers	Memory(GB)↓	R@1↑	R@5↑	R@10↑	MnR↓
    L=0	9.20	42.6	70.8	79.9	16.1
    L=2	10.46	44.0	70.9	80.6	15.0
    L=4	12.60	44.8	71.6	82.1	15.6
    L=8	16.69	45.0	72.1	81.4	14.6
    L=16	25.05	45.6	72.4	81.5	14.7

• Visualization Results: Figure 4 provides qualitative evidence for MUSE's effectiveness. In the examples, the base model fails by matching general themes but missing specific details. MUSE succeeds by correctly identifying these details, which are often multi-scale in nature:

  • Small objects: It correctly identifies "a little brush".

  • Multi-granularity objects: It correctly identifies "A man" (large scale) in one scene.

  • Fine-grained recognition: It distinguishes "a cat" from other animals.

  • Relational understanding: It understands the action "into the crowd". These examples visually confirm the paper's central hypothesis that modeling multi-scale features is crucial for nuanced text-video understanding.

    该图像是图4，展示了文本-视频检索的示例。它对比了使用MUSE模型与不使用MUSE模型时视频检索的排名第一结果。绿色边框表示MUSE的正确检索结果，而红色边框表示未采用MUSE时的错误检索。橙色方框标注了关键视觉提示。图像通过四个不同的场景，直观地证明了MUSE模型在准确理解和关联文本与视频内容方面的优越性，尤其是在处理多尺度信息时。

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully introduces MUSE, a novel and efficient method for text-video retrieval that leverages multi-scale representations. By generating features at multiple resolutions and using the Mamba architecture to model their inter-dependencies, MUSE provides richer context for video understanding. The method is computationally efficient, highly effective (achieving state-of-the-art results), and versatile (acting as a plug-and-play module for existing models). The extensive experiments and ablations provide strong validation and offer valuable insights into designing models for multi-scale learning.

• Limitations & Future Work:

  • Reliance on CLIP: The model's performance is still fundamentally tied to the quality of the initial features from the CLIP backbone. Biases or weaknesses in CLIP will be inherited.
  • Simple Multi-scale Generation: The method for generating the feature pyramid (convolution and pooling) is relatively simple. More sophisticated or learnable feature pyramid networks could potentially yield better results.
  • Focus on Spatial Scales: The paper focuses on spatial resolutions. A joint exploration of spatial and temporal multi-scale modeling could be a promising future direction.
  • Generalization: While tested on three popular benchmarks, its performance on more niche or specialized video domains remains to be seen.

• Personal Insights & Critique:

  • Novelty and Impact: The paper's primary strength is its timely and effective application of Mamba to a well-defined problem in vision-language research. It smartly sidesteps the computational bottleneck of Transformers for multi-scale modeling, offering a practical and powerful solution. The "plug-and-play" design makes it highly impactful, as it can be easily adopted by the research community to boost a wide range of existing models.
  • Rigor: The experimental section is exceptionally thorough. The authors not only demonstrate SOTA results but also conduct a comprehensive set of ablation studies that dissect their design choices and provide clear justifications. This level of rigor strengthens the paper's conclusions significantly.
  • Future Directions: This work opens up several interesting avenues. The success of Mamba as a "multi-scale learner" suggests it could be valuable for other dense prediction tasks in video, such as video object segmentation or action detection, where integrating context from different resolutions is key. Furthermore, the concept could be extended beyond vision to other modalities that have inherent multi-scale structures, such as audio or sensor data. The paper is a strong piece of engineering and empirical research that effectively marries a new architectural advance (Mamba) with a classic computer vision concept (multi-scale pyramids) to solve a modern problem.