1. Bibliographic Information

1.1. Title

Characteristics Matching Based Hash Codes Generation for Efficient Fine-grained Image Retrieval

1.2. Authors

The paper is authored by Zhen-Duo Chen, Li-Jun Zhao, Zi-Chao Zhang, Xin Luo, and Xin-Shun Xu. They are affiliated with the School of Software, Shandong University, China.

1.3. Journal/Conference

This paper was published at CVPR 2024. The Conference on Computer Vision and Pattern Recognition (CVPR) is one of the premier annual computer vision conferences, renowned for publishing high-impact research in the field. Its proceedings are highly influential in computer vision, machine learning, and artificial intelligence.

1.4. Publication Year

2024

1.5. Abstract

The abstract introduces the challenge of efficient fine-grained image retrieval (FGIR) amidst rapidly growing data. It highlights two inherent contradictions in designing hashing-based models for FGIR:

1. Feature Learning Contradiction: Low-dimensional binary hash codes have limited information embedding capacity, which conflicts with the detailed information required to describe fine-grained categories.

2. Efficiency Contradiction: Complex fine-grained feature extraction models, while effective, lead to higher computational costs, undermining the primary goal of efficient retrieval.

   To address these issues, the paper proposes a characteristics matching based hash codes generation method (CMBH). This method is enhanced by two auxiliary modules: a cross-layer semantic information transfer module and a multi-region feature embedding module. These modules enable the generation of hash codes that effectively capture subtle fine-grained differences between samples while ensuring efficient inference. Extensive experiments on widely used datasets demonstrate that CMBH significantly outperforms state-of-the-art methods in both effectiveness and efficiency.

1.6. Original Source Link

https://openaccess.thecvf.com/content/CVPR2024/papers/Chen_Characteristics_Matching_Based_Hash_Codes_Generation_for_Efficient_Fine-grained_Image_CVPR_2024_paper.pdf This is the official PDF link to the paper published at CVPR 2024.

2. Executive Summary

2.1. Background & Motivation

The explosive growth of digital image data necessitates efficient retrieval systems. Fine-grained image retrieval (FGIR), which involves distinguishing between highly similar sub-categories (e.g., different bird species or car models), presents a particularly challenging scenario. Hashing-based models have emerged as a promising solution for efficient retrieval by mapping high-dimensional image features into compact binary codes, enabling faster similarity searches and reduced storage.

However, the authors identify two fundamental contradictions that hinder the effective design of hashing-based models for FGIR:

1. Limited Information Embedding Capacity vs. Detailed Fine-Grained Features: Fine-grained categories are characterized by subtle, often minute differences that require comprehensive and high-dimensional feature representations. Conversely, hashing relies on generating low-dimensional binary hash codes. There's an inherent conflict: how can compact binary codes embed enough detailed information to distinguish fine-grained variations without significant information loss? Simply using more complex feature extractors or higher-dimensional real-valued features doesn't guarantee better hash codes because of this information bottleneck.

2. Complexity of Feature Extraction vs. Retrieval Efficiency: Achieving high accuracy in FGIR often demands sophisticated deep learning models with intricate architectures (e.g., multi-region analysis, multi-layer feature fusion) to extract discriminative fine-grained features. However, such complex models introduce more parameters and computational costs during inference, directly counteracting the primary goal of hashing-based retrieval, which is efficiency (fast query speed and low storage).

   The paper's entry point is to challenge the traditional approach of simply mapping complex fine-grained feature vectors directly to hash codes. Instead, it proposes that the ultimate goal is to represent fine-grained differences, not to fully describe input samples. This leads to the novel idea of a characteristics matching based hash codes generation strategy, which aims to abstract critical characteristics and match samples against them, simplifying the information that needs to be encoded in hash codes.

2.2. Main Contributions / Findings

The paper makes several significant contributions to the field of efficient fine-grained image retrieval:

• Identification and Analysis of Core Contradictions: The authors are the first to explicitly analyze the inherent feature learning contradiction and efficiency contradiction specific to hashing-based FGIR. This clear problem definition provides a new perspective for designing fine-grained hashing methods.
• Proposal of Characteristics Matching Based Hashing (CMBH): The paper introduces a novel CMBH method that addresses these contradictions. Instead of directly encoding comprehensive image features, CMBH generates hash codes based on how well an image's features match a set of pre-defined characteristic vectors (C-vectors) representing abstract fine-grained categories. This shifts the focus from detailed description to abstract matching, simplifying information embedding in hash codes and enhancing efficiency during inference.
• Design of Auxiliary Training Modules: To ensure the C-vectors are discriminative and representative, two auxiliary modules are proposed, which operate only during the training phase without adding inference overhead:
  • Cross-layer Semantic Information Transfer Module: This module integrates fine-grained information from multiple network layers into the C-vectors, ensuring they capture richer semantic details. It uses both early and later fusion strategies combined with classification and knowledge distillation-inspired losses.
  • Multi-region Feature Embedding Module: This module associates C-vectors with specific local details by extracting features from multiple informative image regions. It employs a parameter-free algorithm for localizing distinct regions, promoting distinctiveness among C-vectors for the same category.
• Demonstrated Superiority in Effectiveness and Efficiency: Extensive experiments on five widely-used fine-grained datasets (CUB-200-2011, FGVC-Aircrafts, Food101, NABirds, VegFru) show that the proposed CMBH method significantly outperforms state-of-the-art (SOTA) methods. Importantly, it achieves this with fewer parameters and comparable or reduced computational costs during inference, particularly for very short hash code lengths. This directly validates its success in resolving both the feature learning and efficiency contradictions.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully understand the CMBH paper, a foundational understanding of several key concepts in computer vision and machine learning is necessary.

• Fine-Grained Image Retrieval (FGIR):

  • Concept: FGIR is a specialized task in image retrieval where the goal is to distinguish between subordinate categories that share very similar overall appearances but differ in subtle details. Examples include differentiating between species of birds, types of aircraft, or models of cars.
  • Challenge: The primary challenge in FGIR is the small inter-category difference (e.g., two bird species might look very similar) and large intra-category variance (e.g., the same bird species can appear very different due to pose, lighting, or background). This requires models to learn highly discriminative, localized features.

• Hashing-Based Retrieval:

  • Concept: Hashing-based retrieval is a technique used for Approximate Nearest Neighbor (ANN) search in large-scale databases. It involves mapping high-dimensional data points (like image feature vectors) into compact binary codes (or hash codes). The core idea is that similar data points should map to similar hash codes (or codes with small Hamming distance).
  • Advantages:
    • Efficiency: Comparing binary codes (e.g., using XOR operations for Hamming distance) is significantly faster than comparing high-dimensional floating-point vectors (e.g., using Euclidean distance).
    • Storage: Binary codes require much less storage space than floating-point vectors.
  • Challenge: The main challenge is to design hashing functions that preserve the semantic similarity of original data in the Hamming space of binary codes, especially given the limited information capacity of short binary codes.

• Convolutional Neural Networks (CNNs):

  • Concept: CNNs are a class of deep learning models specifically designed for processing grid-like data, such as images. They consist of multiple layers, including convolutional layers (which apply learnable filters to extract local features), pooling layers (which reduce spatial dimensions and create translation invariance), and fully connected layers (which perform high-level reasoning).
  • Backbone Networks: In computer vision, pre-trained CNNs like ResNet (Residual Network) are often used as backbone networks to extract feature representations from images. ResNet introduces residual connections (skip connections) to allow deeper networks to be trained more effectively, addressing the vanishing gradient problem.

• L2 Normalization:

  • Concept: L2 normalization (also known as Euclidean normalization or vector normalization) rescales a vector so that its L2 norm (Euclidean length) is 1. This means all normalized vectors lie on the surface of a unit hypersphere.
  • Purpose: In machine learning, it's often used to make feature vectors comparable regardless of their magnitude, focusing solely on their direction. This can be important for similarity measures like cosine similarity, which is equivalent to the dot product of L2-normalized vectors.

• Softmax Function:

  • Concept: The softmax function (or normalized exponential function) is often used in multi-class classification to convert a vector of arbitrary real numbers into a probability distribution. It takes a vector of scores and squashes them into a vector of probabilities that sum to 1.
  • Formula: For a vector $\mathbf{z} = [z_1, z_2, \ldots, z_K]$, the softmax function calculates probabilities $\hat{p}_i$ as: $ \hat{p}i = \frac{e^{z_i}}{\sum{j=1}^{K} e^{z_j}} $
  • Purpose: It's typically applied to the output of the final layer of a neural network to produce class probabilities.

• Cross-Entropy Loss (CE):

  • Concept: Cross-entropy loss is a common loss function used in classification tasks to measure the difference between the predicted probability distribution and the true probability distribution. It penalizes predictions that are confident and wrong more heavily than predictions that are less confident and wrong.
  • Formula: For a multi-class classification problem with $C$ classes, given true one-hot label $\mathbf{y}$ and predicted probabilities $\hat{\mathbf{y}}$, the cross-entropy loss for a single sample is: $ \mathrm{CE}(\mathbf{y}, \hat{\mathbf{y}}) = - \sum_{c=1}^{C} y_c \log(\hat{y}_c) $ where $y_c$ is 1 if class $c$ is the true class, and 0 otherwise.
  • Purpose: It guides the model to learn parameters that produce probability distributions closer to the true labels.

• Sign Function (sign(·)):

  • Concept: The sign function is a mathematical function that returns the sign of a real number. It returns 1 for positive numbers, -1 for negative numbers, and 0 for zero.
  • Purpose in Hashing: In hashing, it's often used to binarize real-valued outputs into binary codes. For example, values greater than 0 become 1, and values less than or equal to 0 become -1 (or 0, depending on the desired binary representation).

• Tanh Function (tanh(·)):

  • Concept: The hyperbolic tangent function (tanh) is an activation function similar to the sigmoid function, but its output range is from -1 to 1.
  • Purpose in Hashing: In some hashing methods, tanh is used as a "relaxed" version of the sign function during training. Its continuous and differentiable nature allows for gradient-based optimization, while still pushing values towards -1 or 1, which can then be hard-thresholded by sign to produce binary codes during inference.

3.2. Previous Works

The paper discusses a progression of fine-grained image retrieval (FGIR) methods, particularly focusing on hashing-based approaches.

• Early FGIR Methods (Real-valued vectors):

  • SCDA [20], CRL [29], DGCRL [30], MPFE [2], KAE-Net [16]: These earlier works primarily focused on maximizing retrieval accuracy using long real-valued feature vectors. They often employed techniques like local feature aggregation, contextual reasoning, or attribute learning to capture fine-grained details. Their limitation, in the context of large-scale data, was the high computational cost and storage requirements associated with real-valued vector comparisons.

• Hashing-Based FGIR Methods (Balancing Accuracy and Efficiency): The main trend shifted towards hashing to balance accuracy with efficiency. The common underlying principle of these methods, prior to CMBH, is to extract more comprehensive fine-grained feature vectors and then map them into binary hash codes.

  • FPH [26] (Two-pyramid hashing architecture): One of the earliest hashing methods for FGIR. It aimed to capture subtle differences by integrating features from multiple network layers, often resembling a feature pyramid network structure to handle different scales of features.
  • DSaH [11] (Attention network): Also an early method. It introduced an attention mechanism to identify and focus on salient regions within images, assuming these regions contain the most discriminative fine-grained information.
  • ExchNet [4] (Localized regions with attention and alignment): This method localized multiple regions using an attention mechanism and then aligned the extracted part feature vectors across different images using an exchanging operation. This helped to ensure that corresponding parts were compared.
  • A$^2$-Net [21] (Localization and attribute vectors): Adopted a localization module (similar to ExchNet) and aimed to learn high-level attribute vectors for hash code generation. These attributes could represent specific fine-grained characteristics.
  • A$^2$-Net$^{++}$ [23] (Enhanced self-consistency): An improved version of $A$^2-Net that further enhanced the model's self-consistency, likely through better regularization or training strategies, to produce more robust attribute vectors and hash codes.
  • DLTH [12] (Novel list-wise triplet loss): This method focused on the loss function aspect, designing a list-wise triplet loss to capture relative similarity among samples more effectively in the hashing space. A triplet loss typically aims to ensure that an anchor sample is closer to a positive sample than to a negative sample in the embedding space.
  • sRLH [24] (Sub-region localization): Proposed a sub-region localization module to identify peaks of non-overlap sub-regions, aiming to capture diverse local information without redundancy.
  • FISH [3] and FCAENet [28] (Attention-based erasing strategy): These methods employed an attention-based erasing strategy. This involves identifying salient regions, then erasing them or making them less prominent, forcing the model to learn from other, potentially less obvious, discriminative regions. FISH also included a feature refinement module, while FCAENet added an enhancing space relation loss.
  • SEMICON [18] (Suppression-enhancing mask): This method used a suppression-enhancing mask to explore the relationships between different discovered regions, indicating a focus on contextual understanding of local features.
  • CNET [27] (Cascaded network and attention-guided data augmentation): Designed a cascaded network and an attention-guided data augmentation strategy to improve fine-grained feature learning. It also introduced a novel approach to balance multi-task losses.
  • AGMH [13] (Attention dispersion loss and step-wise interactive external attention): One of the most recent SOTA methods, AGMH proposed an attention dispersion loss and a step-wise interactive external attention module. Its goal was to group and embed category-specific visual attributes into multiple descriptors for a comprehensive feature representation.

3.3. Technological Evolution

The evolution of FGIR methods, particularly hashing-based ones, can be traced through several stages:

1. Early Focus on Accuracy with Real-valued Features: Initially, the primary goal was high accuracy, often achieved through complex CNN architectures and real-valued feature vectors. Methods like SCDA, CRL, MPFE extracted rich, high-dimensional features. While effective for accuracy, these were computationally intensive and memory-demanding for large datasets.
2. Introduction of Hashing for Efficiency: The increasing scale of data necessitated more efficient retrieval. FPH and DSaH marked the transition by introducing hashing to FGIR, aiming to convert real-valued features into binary codes for faster search and reduced storage.
3. Enhancing Feature Extraction for Hashing: Subsequent hashing-based FGIR methods (ExchNet, $A$^2-Net, sRLH, FISH, FCAENet, SEMICON, CNET, AGMH) largely retained the philosophy of their real-valued predecessors: the core challenge was seen as extracting ever more comprehensive and discriminative fine-grained features. They integrated various techniques like attention mechanisms, multi-region analysis, multi-layer feature fusion, and advanced loss functions to improve the quality of feature vectors before binarization. The assumption was that better real-valued features would directly lead to better hash codes.
4. CMBH: A Paradigm Shift – Characteristics Matching: This paper (CMBH) represents a shift in thinking. Instead of focusing solely on generating more comprehensive fine-grained features to be directly embedded into hash codes, CMBH argues that this approach leads to inherent contradictions. CMBH proposes that the goal is not a detailed description of the input sample in the hash code, but rather to capture fine-grained differences through matching against abstract characteristics. It seeks to embed the degree of matching into the hash codes, rather than the raw fine-grained features themselves. This new paradigm aims to simplify the information embedded in hash codes while preserving discriminative power, and to decouple the complexity of feature learning during training from the efficiency of inference.

3.4. Differentiation Analysis

Compared to the main methods in related work, CMBH introduces a fundamental conceptual shift in how hash codes are generated for fine-grained image retrieval.

• Core Conceptual Difference:

  • Previous Methods: The prevailing approach in hashing-based FGIR (e.g., ExchNet, $A$^2-Net, FISH, AGMH) has been to develop increasingly sophisticated feature extraction modules (employing attention, multi-region analysis, multi-layer fusion) to generate highly comprehensive real-valued fine-grained feature vectors. These feature vectors are then directly mapped or compressed into binary hash codes. The implicit assumption is that richer feature vectors will yield better hash codes.
  • CMBH's Approach: CMBH challenges this assumption. It posits that directly embedding all fine-grained features into low-dimensional binary codes is inherently contradictory due to the limited information capacity of hash codes. Instead, CMBH proposes that hash codes should represent the degree of matching between an input sample's abstract features and a set of predefined characteristic vectors (C-vectors). These C-vectors act as abstract descriptors of fine-grained categories. The hash code then reflects how well a sample exhibits these characteristics, rather than being a direct compressed representation of its full feature vector.

• Addressing the Contradictions:

  • Feature Learning Contradiction: By shifting from direct feature embedding to characteristics matching, CMBH simplifies the information that needs to be encoded in hash codes. It focuses on capturing significant differences among samples via matching scores, effectively alleviating the information bottleneck of low-dimensional binary codes. Previous methods often struggled with this, as adding more fine-grained features didn't always translate to corresponding performance gains in hash code quality.
  • Efficiency Contradiction: CMBH explicitly designs its cross-layer semantic information transfer and multi-region feature embedding modules to operate only during the training phase. During inference, only the simple characteristics matching component is used to generate hash codes. This modular design allows for complex, fine-grained feature learning to optimize C-vectors during training without incurring additional computational costs or parameters during the crucial retrieval inference stage. In contrast, many SOTA methods (e.g., SEMICON, CNET mentioned in Table 5) incorporate their multi-region or multi-layer components into the inference path, leading to higher parameters and FLOPs.

• Analogy to Human Cognition: CMBH draws an analogy to how humans swiftly assess similarity—by matching against key characteristics from memory, not by meticulously checking every detail. This intuitive alignment suggests a more natural and efficient way to handle fine-grained distinctions for hashing.

  In essence, CMBH distinguishes itself by fundamentally rethinking the role of hash codes in FGIR: from a compressed feature representation to an abstract matching score representation, thereby offering a more principled solution to the effectiveness-efficiency dilemma.

4. Methodology

The proposed method, Characteristics Matching Based Hashing (CMBH), aims to address the inherent contradictions in fine-grained image retrieval (FGIR) by generating hash codes based on characteristics matching rather than direct feature embedding. The methodology is structured around a core hash code generation component and two auxiliary modules that operate during training to optimize the characteristic vectors (C-vectors).

4.1. Principles

The core idea of CMBH is to move away from the traditional paradigm of directly compressing comprehensive fine-grained feature vectors into binary hash codes. Instead, it proposes that hash codes should represent the degree of matching between an input sample's feature vector and a set of predefined characteristic vectors (C-vectors). These C-vectors are learned to abstractly represent the critical fine-grained characteristics of each category.

This approach offers two key advantages:

1. Information Simplification: By focusing on matching scores against abstract characteristics, the information embedded in the hash codes is significantly simplified, alleviating the information bottleneck of low-dimensional binary codes when dealing with detailed fine-grained differences.
2. Decoupling Training Complexity from Inference Efficiency: Complex fine-grained feature learning (e.g., multi-layer and multi-region analysis) is used only during training to optimize the C-vectors. During inference, only a single feature vector is extracted, and its matching scores against the learned C-vectors are used to generate the hash code. This ensures high retrieval efficiency by minimizing computational cost and parameters at test time.

4.2. Core Methodology In-depth

4.2.1. Framework and Notations

The overall framework of CMBH involves a main component for hash code generation and training, which includes a backbone network for feature extraction. Two auxiliary modules, the cross-layer semantic information transfer module and the multi-region feature learning module, are used exclusively during training to optimize the C-vectors. During inference, only the main hash code generation component is retained.

Let's define the notations used:

• $\mathbf{I}_i$: Represents the $i$-th raw pixel input image from a total of $n$ training instances. So, $\mathbf{I}_i \in \{ \mathbf{I}_1, \ldots, \mathbf{I}_n \}$.

• $N(\cdot)$: Denotes the backbone network (e.g., ResNet).

• $\mathbf{F}_i^j$: Represents the feature map output from the $j$-th stage (from top to bottom) of the backbone ResNet for image $\mathbf{I}_i$. Its dimension is $\mathbb{R}^{c_j \times w_j \times h_j}$, where $c_j$ is the number of channels, and $w_j, h_j$ are the spatial dimensions.

• $L$: The total number of stages from which feature maps are extracted (e.g., for the last three stages of ResNet-50, $L=3$).

  The feature extraction procedure is defined as: $$\{ \mathbf{F}_i^j | j \in \{ 1, \dots, L \} \} = N(\mathbf{I}_i)$$ This means the backbone network $N$ processes an input image $\mathbf{I}_i$ to produce a set of feature maps from different stages.

Each feature map $\mathbf{F}_i^j$ is then transformed into a feature vector $\mathbf{f}_i^j$: $$\mathbf{f}_i^j = N_{sub-j}(\mathbf{F}_i^j) \in \mathbb{R}^d$$ Here, $N_{sub-j}(\cdot)$ is a sub-network specific to stage $j$. It consists of two convolutional layers, a global max pooling layer, and a fully connected layer. This sub-network processes the feature map $\mathbf{F}_i^j$ to produce a fixed-dimensional feature vector $\mathbf{f}_i^j$ of dimension $d$.

• $\mathbf{b}_i \in \{-1, 1\}^b$: The $b$-bit binary hash code for instance $\mathbf{I}_i$.
• $\mathbf{y}_i \in \{0, 1\}^y$: The one-hot label vector for instance $\mathbf{I}_i$, where $y$ is the total number of categories.
• $\mathbf{C} \in \mathbb{R}^{y \times k \times d}$: The Characteristic vectors (C-vectors). This is a tensor of predefined C-vectors. $y$ is the number of categories, $k$ is the number of critical characteristics preset for each category, and $d$ is the dimension of each C-vector (matching the dimension of $\mathbf{f}_i^j$). These C-vectors are randomly initialized and optimized during training.

4.2.2. C-vectors Matching and Hash Codes Learning

The core mechanism is to calculate a matching score between an image's feature vector and the C-vectors. The paper specifies that for hash code generation during inference, only the feature vector from a single network layer, $\mathbf{f}_i^l$, is used (where $l$ is a specific chosen layer index, e.g., $l=2$ or the penultimate layer).

The matching score for input instance $\mathbf{I}_i$ is defined as a matrix $\mathbf{M}_i \in \mathbb{R}^{y \times k}$. Each element $\mathbf{M}_{i, (a,b)}$ represents the matching score between the $a$-th category's $b$-th C-vector and the image's feature vector $\mathbf{f}_i^l$. This score is calculated using L2-normalized dot product (cosine similarity): $$\mathbf{M}_{i, (a,b)} = \left( \left( \frac{\mathbf{C}_{a,b,*}}{||\mathbf{C}_{a,b,*}||_2} \right)^\top \frac{\mathbf{f}_i^l}{||\mathbf{f}_i^l||_2} \right)$$ Here:

• $\mathbf{C}_{a,b,*}$: Represents the $b$-th C-vector for the $a$-th category. The * denotes selection across the $d$ dimension.
• $||\cdot||_2$: Denotes L2 normalization, which scales the vector to have a Euclidean length of 1.
• $\top$: Denotes the transpose operation. The formula computes the cosine similarity between the L2-normalized C-vector $\mathbf{C}_{a,b,*}$ and the L2-normalized feature vector $\mathbf{f}_i^l$.

After computing the matching score matrix $\mathbf{M}_i$, the to-be-learned hash code $\mathbf{b}_i$ is obtained by a mapping layer $N_{hash}(\cdot)$. This involves reshaping $\mathbf{M}_i$ into a single vector, applying a linear transformation, and then binarizing it. $$\mathbf{b}_i = N_{hash}(\mathbf{M}_i) = \mathrm{sign} (\mathbf{W}_h (\mathrm{vec}(\mathbf{M}_i)))$$ Here:

• $\mathrm{vec}(\cdot)$: Is a simple function to reshape the matching score matrix $\mathbf{M}_i$ (of size $y \times k$) into a single vector of size $y \times k$.

• $\mathbf{W}_h \in \mathbb{R}^{b \times (yk)}$: Is the hash codes mapping matrix, which transforms the concatenated matching scores into a $b$-dimensional vector, where $b$ is the desired hash code length.

• $\mathrm{sign}(\cdot)$: Is an element-wise sign function that converts real-valued outputs into binary codes (e.g., positive values become 1, non-positive values become -1).

  The hash codes learning uses a simple loss function that combines pair-wise and proxy-based similarity preservation training: $$\mathcal{L}_{hash} = \mathcal{L}_{hash-pair} + \mathcal{L}_{hash-proxy}$$ This total hash loss consists of two components:

1. Pair-wise Loss ($\mathcal{L}_{hash-pair}$): This term encourages the hash codes of similar images to be close in Hamming space and dissimilar images to be far apart. $$\mathcal{L}_{hash-pair} = \frac{1}{nn} \sum_{i=1}^n \sum_{j=1}^n (\mathbf{u}_i^{(t)\top} \mathbf{b}_j^{(t-1)} - b \mathbf{S}_{i,j}^p)^2$$

   • $n$: Total number of training instances.
   • $\mathbf{u}_i^{(t)}$: The relaxed version of $\mathbf{b}_i^{(t)}$. It's obtained by replacing the sign operation with tanh in Equation (4). This makes the binarization differentiable for gradient-based optimization.
   • $\mathbf{b}_j^{(t-1)}$: The binary code of the $j$-th instance recorded from the previous training epoch (t-1).
   • $b$: The hash code length.
   • $\mathbf{S}^p$: A pair-wise similarity matrix of size $n \times n$.
     • Initially, $\mathbf{S}_{i,j}^p = 1$ if images $i$ and $j$ belong to the same category, and 0 otherwise.
     • Then, every 0 in $\mathbf{S}^p$ is replaced with $- (\sum \mathbf{S}^p / \sum (1 - \mathbf{S}^p))$. This scaling balances the contributions of positive (similar) and negative (dissimilar) pairs, making the loss function more robust. The term $\mathbf{u}_i^{(t)\top} \mathbf{b}_j^{(t-1)}$ approximates the inner product (or Hamming similarity) in the binary space.

2. Proxy-based Loss ($\mathcal{L}_{hash-proxy}$): This term ensures that the hash code of an image is similar to the hash code proxy of its true category and dissimilar to proxies of other categories. $$\mathcal{L}_{hash-proxy} = \frac{1}{ny} \sum_{i=1}^n \sum_{j=1}^y (\mathbf{u}_i^{(t)\top} \mathbf{c}_j^{(t-1)} - b \mathbf{S}_{i,j}^c)^2$$

   • $y$: Total number of categories.
   • $\mathbf{c}_j^{(t-1)}$: The hash code proxy for the $j$-th category, which is the average of the relaxed hash codes $\mathbf{u}^{(t-1)}$ belonging to the $j$-th category, recorded from the previous epoch.
   • $\mathbf{S}^c$: A proxy similarity matrix of size $n \times y$.

     • Initially, $\mathbf{S}_{i,j}^c = 1$ if image $i$ belongs to the $j$-th category, and 0 otherwise.

     • Then, every 0 in $\mathbf{S}^c$ is replaced with $- (\sum \mathbf{S}^c / \sum (1 - \mathbf{S}^c))$ for balancing.

       During the first training epoch, both $\mathbf{c}_j^{(t-1)}$ and $\mathbf{b}_j^{(t-1)}$ are randomly initialized.

4.2.3. C-vectors Training and Optimization

For the characteristics matching based hash codes generation to be effective, the C-vectors must accurately describe fine-grained categories. This section details how C-vectors are trained and optimized through cross-layer and multi-region learning, with these modules being active only during training.

4.2.3.1. Cross-layer semantic information transfer

Integrating information from multiple network layers is crucial for capturing different granularities of features, which is particularly beneficial for fine-grained tasks. However, traditional cascading methods lead to higher-dimensional vectors and increased computational costs, which is undesirable for hashing. CMBH addresses this by embedding multi-layer information into the C-vectors, allowing them to act as an intermediary for transferring this rich information into the hash code generation process, without impacting inference efficiency.

Given the feature vectors $\{ \mathbf{f}_i^j | j \in \{1, \dots, L\} \}$ generated as per Equation (1), the information transfer has two components:

1. Early Fusion Strategy: A unified cross-layer feature representation $\mathbf{f}_i^0$ is obtained by concatenating feature vectors from different layers and mapping them to a common dimension: $$\mathbf{f}_i^0 = N_{con} (\mathrm{concat}(\mathbf{f}_i^1; \cdot \cdot \cdot; \mathbf{f}_i^L)) \in \mathbb{R}^d$$

   • $\mathrm{concat}(\cdot)$: Concatenates the feature vectors from all $L$ stages.
   • $N_{con}$: A mapping layer, including a fully-connected layer, to unify the concatenated feature dimension to $d$. This $\mathbf{f}_i^0$ is then used to calculate a matching score matrix $\mathbf{M}_i^{con}$ in a similar fashion to Equation (3): $$\mathbf{M}_{i, (a,b)}^{con} = \left( \left( \frac{\mathbf{C}_{a,b,*}}{||\mathbf{C}_{a,b,*}||_2} \right)^\top \frac{\mathbf{f}_i^0}{||\mathbf{f}_i^0||_2} \right)$$ By jointly optimizing this matching score matrix with the main hash code generation (Equation (3)), the integrated multi-layer information is embedded into the C-vectors. To ensure semantic consistency, classification losses are applied to these matching score matrices and individual layer features: $$L_{cls} = \sum_{i=1}^n \sum_{j=0}^L \mathrm{CE} (\hat{\mathbf{y}}_i^j, \mathbf{y}_i)$$
   • $\mathrm{CE}(\cdot, \cdot)$: Standard softmax-cross-entropy loss.
   • $\hat{\mathbf{y}}_i^0 = \alpha \sum_{j=1}^k \mathbf{M}_{i, (*,j)}^{con}$: The class vector generated from the concatenated feature matching. The sum is across the $k$ C-vectors for each category.
   • $\hat{\mathbf{y}}_i^l = \alpha \sum_{j=1}^k \mathbf{M}_{i, (*,j)}$: The class vector generated from the matching of the specific layer $\mathbf{f}_i^l$ (used for hash code generation).
   • $\hat{\mathbf{y}}_i^j = N_{cls-j}(\mathbf{f}_i^j)$ for $j \in \{1, \dots, L\}$ and $j \neq l$: Class vectors generated from individual feature vectors $\mathbf{f}_i^j$ using a simple one-layer classification layer $N_{cls-j}(\cdot)$.
   • $\alpha$: A hyper-parameter greater than 1, used to magnify the gradient and accelerate training.

2. Later Fusion Strategy (Knowledge Distillation inspired): This component uses a knowledge distillation approach to supervise the training of the output layer $l$ (which is used for hash code generation) with the combined knowledge from all layers. $$L_{trans} = - \sum_{i=1}^n \left( \mathrm{softmax} \left( \sum_{j=0}^L \hat{\mathbf{y}}_i^j \right)^\top \log \left( \mathrm{softmax} (\hat{\mathbf{y}}_i^l) \right) \right)$$

   • This is a form of Kullback-Leibler (KL) divergence or cross-entropy loss where the target distribution is the ensemble softmax output from all class vectors $(\sum_{j=0}^L \hat{\mathbf{y}}_i^j)$, and the student distribution is the softmax output of the main layer $l$ $(\hat{\mathbf{y}}_i^l)$. This forces the C-vectors associated with layer $l$ to learn the integrated semantic information from all layers.

     Finally, to further reduce information loss during the mapping from matching scores to hash codes, especially for shorter hash codes, the semantic-preserving information obtained from the cross-layer transfer is used to reinforce the weights of C-vectors in hash code mapping. Equation (4) is modified as: $$\mathbf{b}_i = \mathrm{sign} \left( \mathbf{W}_h \left( \mathrm{vec} \left( \frac{\exp(\hat{\mathbf{y}}_i^l)}{\sum_{j=1}^y \exp(\hat{\mathbf{y}}_{i,j}^l)} \odot \mathbf{M}_i \right) \right) \right)$$

• $\frac{\exp(\hat{\mathbf{y}}_i^l)}{\sum_{j=1}^y \exp(\hat{\mathbf{y}}_{i,j}^l)}$: This is the softmax of the class vector $\hat{\mathbf{y}}_i^l$ derived from the main layer $l$'s matching scores. This softmax output essentially provides a probability distribution over categories based on the matching scores of layer $l$.
• $\odot$: Represents element-wise multiplication with broadcasting operation. The softmax probabilities are used to weight the matching score matrix $\mathbf{M}_i$. This means that matching scores for categories that the model is confident about for layer $l$ will be emphasized more, effectively guiding the hash code generation towards the most salient semantic information learned from layer $l$.

4.2.3.2. Multi-region feature embedding

To ensure the C-vectors are distinct and capable of describing specific characteristics of a fine-grained category, multi-region feature embedding is employed. This associates C-vectors with specific local details from different regions of the input images. A parameter-free algorithm is designed for this purpose to avoid adding trainable parameters and computational costs.

First, all feature maps generated in Equation (1) are resized to the same spatial size as the last stage feature map ($\overline{\mathbf{F}}_i^j \in \mathbb{R}^{c_j \times w_L \times h_L}$). Then, a channel-wise mean feature map $\mathbf{A}_i \in \mathbb{R}^{w_L \times h_L}$ is computed: $$\mathbf{A}_i = \frac{1}{\sum_{j=1}^L c_j} \sum_{a=1}^{\sum_{j=1}^L c_j} \mathrm{stack}(\overline{\mathbf{F}}_i^1; \ldots; \overline{\mathbf{F}}_i^L)_{(a,*,*)}$$

• $\mathrm{stack}(\cdot)$: A channel-wise stack operation that concatenates all resized feature maps along the channel dimension.

• The formula then computes the average across all channels at each spatial location $(*,*)$ to create a 2D attention map $\mathbf{A}_i$.

  This attention map $\mathbf{A}_i$ serves as input to Algorithm 1 to localize M distinct informative regions:

Algorithm 1: Distinct critical image regions localization Input: Feature map matrix $\mathbf{A}_i$, bounding box set $B = \varnothing$. Output: Bounding box set $B$.

1: for $m = 1$ to $M$ do 2: $a \gets$ The maximum value in $\mathbf{A}_i$; 3: $\mathbf{H}_i \gets$ Values greater than $a \cdot (0.9 - m \cdot 0.05)$ in $\mathbf{A}_i$ are 1, and other values are 0; * This step identifies high-activation regions in $\mathbf{A}_i$ by setting a threshold. The threshold decreases with each iteration $m$, allowing detection of less salient regions in subsequent steps. 4: $\bar{\mathbf{H}}_i \gets$ Calculate the largest connected component of $\mathbf{H}_i$; * This isolates the most significant contiguous highly activated area. 5: $[x_1, y_1, x_2, y_2] \gets$ Calculate the bounding box coordinates of $\bar{\mathbf{H}}_i$; * This extracts the bounding box for the identified region within the feature map space. 6: $[\hat{x}_1, \hat{y}_1, \hat{x}_2, \hat{y}_2] \gets$ Calculate the original image coordinates corresponding to $[x_1, y_1, x_2, y_2]$; * The bounding box coordinates are scaled back to the original image dimensions. 7: $\mathbf{A}_i \gets \mathbf{A}_i \cdot (1 - \mathbf{H}_i) + (\mathbf{H}_i \cdot a - \mathbf{H}_i \cdot \mathbf{A}_i)$; * This is a suppression step (similar to Non-Maximum Suppression - NMS). It effectively erases or suppresses the currently localized region in $\mathbf{A}_i$ by reducing its activation. $(1 - H_i)$ zeroes out the selected region. $(H_i * a - H_i * A_i)$ seems to set values in the $H_i$ region to $a - A_i$, effectively ensuring that the previous max value is gone, preventing re-selection of the same peak. A simpler interpretation would be that $A_i$ in the localized region is set to a very low value or zero, so the next iteration finds a new peak. 8: $B \gets B \cup \{ [\hat{x}_1, \hat{y}_1, \hat{x}_2, \hat{y}_2] \}$; * The bounding box of the current distinct region is added to the set $B$. 9: end for 10: return Bounding box set $B$.

After obtaining $M$ bounding boxes, $M$ patches $\{ \mathbf{I}_i^1, \dots, \mathbf{I}_i^M \}$ are cropped from image $\mathbf{I}_i$. Each patch is then processed similarly to the full image $\mathbf{I}_i$ to generate its own feature vectors $\{ \mathbf{f}_i^{m,j} \}$ and matching scores $\mathbf{M}_i^m$ and $\mathbf{M}_i^{m,con}$.

To promote distinctiveness among C-vectors, the max function replaces the sum function in the class vector computation for regions:

• $\hat{\mathbf{y}}_i^{\bar{m},0} = \alpha \operatorname{max} (\mathbf{M}_i^{m,con})$
• $\hat{\mathbf{y}}_i^{m,l} = \alpha \operatorname{max} (\mathbf{M}_i^m)$

  • Here, $\operatorname{max}(\cdot)$ is a row-wise max operation, meaning for each category, it takes the maximum matching score across all $k$ C-vectors for that category. This encourages each C-vector to specialize in representing a particular characteristic (as only the best matching C-vector contributes significantly), thereby promoting distinctiveness.

    The loss function associated with multi-region feature embedding is then defined by summing the classification losses and cross-layer transfer losses for each patch: $$L_{region} = \sum_{j=1}^M (L_{cls}^j + L_{trans}^j)$$

• $L_{cls}^j$: The classification loss (Equation 10) applied to the $j$-th patch.
• $L_{trans}^j$: The cross-layer transfer loss (Equation 11) applied to the $j$-th patch.

4.2.4. Training and Inference

The entire CMBH method is trained end-to-end using batch-based stochastic gradient descent and back-propagation. The overall loss function is a summation of all individual loss terms: $$\mathcal{L} = \mathcal{L}_{hash} + \mathcal{L}_{cls} + \mathcal{L}_{trans} + \mathcal{L}_{region}$$

• $\mathcal{L}_{hash}$: The hash codes learning loss (Equation 5).
• $\mathcal{L}_{cls}$: The cross-layer classification loss for the full image (Equation 10).
• $\mathcal{L}_{trans}$: The cross-layer semantic information transfer loss for the full image (Equation 11).
• $\mathcal{L}_{region}$: The multi-region loss (Equation 14).

Inference Procedure: After training, given a query image $\mathbf{I}_q$:

1. The backbone network extracts feature map $\mathbf{F}_q^l$ from the specified layer $l$.

2. $\mathbf{F}_q^l$ is converted to feature vector $\mathbf{f}_q^l$.

3. $\mathbf{f}_q^l$ is used to calculate the matching score matrix $\mathbf{M}_q$ (Equation 3).

4. The class vector $\hat{\mathbf{y}}_q^l$ is computed from $\mathbf{M}_q$.

5. Finally, the hash code $\mathbf{b}_q$ is obtained using the modified mapping (Equation 12).

   Crucially, during inference and testing, none of the cross-layer feature extraction (beyond the single layer $l$) nor the multi-region feature extraction is involved. This design ensures that the proposed method maintains its high retrieval efficiency at test time.

5. Experimental Setup

5.1. Datasets

The proposed method is evaluated on five widely-used fine-grained datasets to demonstrate its robustness and generalizability. The official training/testing data splitting is adopted for fair comparison.

The following are the results from Table 1 of the original paper:

• CUB-200-2011 [19]: A highly popular fine-grained dataset for bird species recognition, containing 200 categories of birds. It is known for its challenging intra-class variations (e.g., pose changes) and inter-class similarities.

• FGVC-Aircrafts [15]: Focuses on aircraft model recognition, with 100 categories. It presents challenges due to variations in viewpoint, lighting, and occlusions.

• Food101 [1]: A dataset for food recognition, comprising 101 food categories. It's often used to test models' ability to distinguish between visually similar food items.

• NABirds [8]: A larger and more challenging bird dataset than CUB-200-2011, featuring 555 categories of North American birds. Its larger number of categories and instances makes it suitable for evaluating scalability.

• VegFru [9]: A diverse fine-grained dataset containing images of 292 categories of vegetables and fruits. It is characterized by a large number of testing images, making efficiency particularly critical.

  These datasets are chosen because they are widely recognized benchmarks in fine-grained image analysis, covering various domains (animals, vehicles, food, plants) and offering diverse scales and levels of fine-grainedness. They are effective for validating the model's ability to capture subtle differences and its performance under different data complexities.

5.2. Evaluation Metrics

The performance of all models is evaluated using the Mean Average Precision (MAP@all), which is the most widely adopted metric for hashing-based retrieval.

• Mean Average Precision (MAP@all):
  • Conceptual Definition: MAP is a common metric used to evaluate the performance of ranking tasks, such as information retrieval or image retrieval. It measures the quality of retrieval by considering both precision and recall. For a single query, Average Precision (AP) calculates the average of precision values at each relevant item retrieved. MAP then averages these AP scores across all queries (and often across all categories in a multi-class setting). MAP@all implies that the entire retrieved list (up to the total number of items in the database) is considered for computing precision and recall, or effectively, that all relevant items are eventually retrieved. A higher MAP indicates better retrieval performance, meaning that relevant items are generally ranked higher in the retrieved list.
  • Mathematical Formula: The Average Precision (AP) for a single query $q$ is defined as: $ \mathrm{AP}q = \sum{k=1}^N P(k) \Delta r(k) $ where:
    • $N$: The total number of items in the ranked retrieval list.
    • P(k): The precision at cutoff $k$ in the ranked list (i.e., the proportion of relevant items among the top $k$ retrieved items).
    • $\Delta r(k)$: The change in recall from k-1 to $k$. If the item at rank $k$ is relevant, $\Delta r(k) = 1 / (\text{total number of relevant items for query } q)$; otherwise, $\Delta r(k) = 0$. A more common way to compute AP for a query $q$ is: $ \mathrm{AP}q = \frac{\sum{k=1}^N (P(k) \times \mathrm{rel}(k))}{\text{number of relevant items for query } q} $ where $\mathrm{rel}(k)$ is an indicator function, equal to 1 if the item at rank $k$ is relevant, and 0 otherwise. P(k) is calculated as: $ P(k) = \frac{\text{number of relevant items at rank } \le k}{k} $ The Mean Average Precision (MAP) is then the average of the AP scores across all $Q$ queries: $ \mathrm{MAP} = \frac{1}{Q} \sum_{q=1}^Q \mathrm{AP}_q $
  • Symbol Explanation:
    • $\mathrm{AP}_q$: Average Precision for query $q$.
    • $N$: Total number of retrieved items.
    • P(k): Precision at rank $k$.
    • $\Delta r(k)$: Change in recall at rank $k$.
    • $\mathrm{rel}(k)$: Indicator function for relevance at rank $k$.
    • $Q$: Total number of queries.

5.3. Baselines

To provide a comprehensive comparison and fully represent the state-of-the-art (SOTA) performance, the proposed method is compared against nearly all hashing-based FGIR methods published in the recent three years, as summarized in the paper's related work and results tables. These baselines are chosen because they represent the current leading approaches in fine-grained hashing.

The following methods are included as baselines:

• ExchNet [4]
• A$^2$-Net [21]
• FCAENet [28]
• SEMICON [18]
• FISH [3]
• A$^2$-Net$^{++}$ [23]
• AGMH [13]
• CNET [27]
• DLTH [12]
• sRLH [24]

Experimental Settings Common to All Methods:

• Backbone Network: For fair comparison, ResNet-50 and ResNet-18 (without the final pooling and classification layer) are chosen as the backbone networks.
• Hyper-parameters:
  • Last three stages of the backbone are used, i.e., $L=3$ in Equation (1).
  • The penultimate stage, $l=2$, is used for hash code generation.
  • Number of C-vectors for each category, $k=2$.
  • Number of cropped patches $M=4$.
• Image Preprocessing: All input images are resized to $255 \times 255$ and then random cropped to $224 \times 224$ for training, and center cropped to $224 \times 224$ for testing.
• Optimizer: Standard SGD optimizer with momentum 0.9 and weight decay 5e-4.
• Training Epochs: 100 epochs.
• Learning Rate: Initial learning rate of 0.001, divided by 10 at the 70th epoch.
• Batch Size: 16 for CUB-200-2011 and Aircrafts; 64 for Food101, NABirds, and VegFru.

6. Results & Analysis

6.1. Core Results Analysis

The experimental results demonstrate that the proposed CMBH method significantly outperforms state-of-the-art (SOTA) methods across various datasets and hash code lengths. This superiority is particularly pronounced for shorter hash code lengths, which is critical for real-world efficient retrieval scenarios.

The following are the results from Table 2 of the original paper:

Analysis of Table 2:

• Across all datasets and bit lengths, CMBH consistently achieves the highest MAP scores. This indicates a robust and superior performance compared to existing SOTA methods, particularly for ResNet-50 backbone.

• Significant Gains for Short Code Lengths (e.g., 12 bits): The most striking improvements are observed at 12 bits. For example, on CUB-200-2011, CMBH achieves 84.07% MAP, significantly higher than the next best (CNET at 77.10%). Similar large margins are seen on Aircrafts (89.11% vs. 86.15% for CNET), Food101 (87.85% vs. 83.06% for CNET), NABirds (74.42% vs. 68.42% for CNET), and VegFru (84.37% vs. 81.63% for CNET).

• Validation of Characteristics Matching: The superior performance, especially with short codes, strongly supports the paper's central hypothesis: the characteristics matching based hash code generation strategy is highly effective. It allows hash codes to efficiently capture fine-grained differences without requiring comprehensive direct feature embedding, thus effectively addressing the feature learning contradiction.

• Comparison to Feature-Heavy Methods: Methods like FISH often perform well at longer bit lengths, but CMBH still surpasses them. The substantial improvements, particularly for shorter hash codes, underscore that the CMBH paradigm is more efficient at abstracting and encoding discriminative information for hashing.

  The following are the results from Table 3 of the original paper:

  Method	Backbone	16bits	32bits	48bits	64bits
  DLTH	ResNet50	68.84	77.82	79.97	81.32
  sRLH	ResNet18	62.68	69.37	71.27	71.60
  FISH	ResNet18	76.03	77.14	78.06	78.34
  AGMH	ResNet18	59.68	76.71	80.73	81.43
  Ours	ResNet18	82.10	82.86	83.36	83.31

Analysis of Table 3:

• Consistent Superiority with ResNet18: Even with a lighter backbone like ResNet18, CMBH maintains its leading performance across all code lengths. For instance, at 16 bits, CMBH achieves 82.10% MAP, significantly higher than FISH (76.03%) and sRLH (62.68%).
• Scalability with Different Backbones: The results demonstrate that CMBH is effective not only with powerful backbones like ResNet50 but also with more lightweight ResNet18, indicating its general applicability and efficiency benefits.

6.2. Ablation Studies / Parameter Analysis

6.2.1. Ablation Study

The ablation study dissects the proposed CMBH into its core components to verify the effectiveness and necessity of each module.

The following are the results from Table 4 of the original paper:

Components:

• (B): Represents the basic hash code generation and training procedure (Section 3.2).
• (L): Represents C-vector optimization based on multi-layer semantic information integration (Section 3.3.1), excluding the cross-layer information transfer loss.
• (T): Represents the cross-layer information transfer supervised by Equation (11).
• (R): Represents the multi-region feature embedding (Section 3.3.2).

Analysis:

• Baseline (B alone): The basic component (B) alone performs poorly (e.g., 17.89% for 12 bits). This highlights that the characteristics matching idea's effectiveness critically depends on well-optimized C-vectors that can accurately describe fine-grained samples. Without proper C-vector optimization, the matching scores are not discriminative enough.

• Adding Multi-layer Integration (B + L): Introducing C-vector optimization via multi-layer semantic information integration (L) leads to a dramatic performance jump (from 17.89% to 77.56% for 12 bits). This demonstrates the significant role of C-vectors in representing fine-grained categories and the effectiveness of embedding multi-layer information into them.

• Adding Cross-layer Transfer (B + L + T): Further incorporating the cross-layer information transfer module (T) provides additional improvements (e.g., from 77.56% to 80.30% for 12 bits). This module, inspired by knowledge distillation, strengthens the semantic consistency across layers and ensures the main layer's C-vectors benefit from the integrated knowledge.

• Adding Multi-region Embedding (B + L + T + R / Complete CMBH): The final addition of the multi-region feature embedding module (R) yields the best performance (e.g., 84.07% for 12 bits). This module enhances the C-vectors' ability to capture local details and promotes their distinctiveness, further boosting retrieval accuracy.

  Each component incrementally contributes to the overall performance, confirming their necessity and effectiveness in the CMBH framework.

6.2.2. Hyper-parameter Analysis

The paper also conducts an analysis of key hyper-parameters: $k$ (number of C-vectors per category) and $\alpha$ (gradient magnification factor).

The following figure (Figure 2 from the original paper) illustrates the mean average precision (MAP) analysis under different parameters on the CUB-200-2011 dataset. The left subplot (a) shows the variation of MAP with different $k$ values, while the right subplot (b) presents the MAP performance with varying $\alpha$ values. Different colored lines represent different bit lengths: 12, 24, 32, and 48 bits, reflecting the impact of each parameter on retrieval performance.

Analysis of Figure 2a (Impact of $k$):

• Significance of $k > 1$: The performance generally improves when $k > 1$ compared to $k=1$, especially for relatively higher hash code lengths. This confirms the necessity of using multiple C-vectors to describe the intra-class diversity present within each fine-grained category. A single C-vector might not be enough to capture all subtle variations of a complex fine-grained class.
• Optimal $k$ and Overfitting: Excessively large values of $k$ can lead to a decline in performance. This is attributed to two main reasons:
  1. Limited Information Representation Capacity of Hash Codes: Even with multiple C-vectors, the ultimate hash code length is fixed. If $k$ is too large, the matching score matrix becomes very high-dimensional, and compressing this into a short hash code can still lead to information loss or redundancy if the hash code cannot adequately represent the increased characteristic diversity.
  2. Overfitting: A larger $k$ introduces more parameters in the C-vector set, which can increase the risk of overfitting to the training data, especially if the dataset size is not sufficiently large to support learning so many distinct characteristics per category.
• $k=1$ vs. SOTA: Notably, even with $k=1$, CMBH often surpasses most existing SOTA baselines (as seen in Table 2). This further emphasizes the core argument of the paper: the essence of fine-grained hashing is reflecting differences through matching, not necessarily comprehensive feature extraction and embedding, validating the paradigm even in its simplest form. For the experiments, $k=2$ was chosen.

Analysis of Figure 2b (Impact of $\alpha$):

• Gradient Magnification: $\alpha$ is introduced to magnify gradients during BP (backpropagation), countering the gradient shrinkage caused by L2 normalization in the matching score calculation.
• Stable Performance: Stable and effective results are achieved once $\alpha$ reaches a certain magnitude (e.g., around 8-16). This suggests that sufficient gradient scaling is necessary, but extremely high values don't necessarily lead to continuous improvement.
• Code Length Interaction:
  • Higher $\alpha$ for Longer Codes: Relatively higher values of $\alpha$ tend to yield slightly better performance for longer hash codes (e.g., 32 and 48 bits). This might be because longer codes have more capacity to encode the diverse features encouraged by magnified gradients.
  • Lower $\alpha$ for Shorter Codes: Relatively lower values of $\alpha$ (e.g., 8 or 12) lead to better performance for shorter hash codes (e.g., 12 bits). This could imply that for very compact hash codes, overly diverse or magnified features might lead to noise or less optimal compression.
• General Recommendation: For practical purposes, setting $\alpha$ to 12 or 16 provides relatively stable and effective results across various datasets. The paper suggests that adaptively adjusting $\alpha$ based on hash code length could be a future research direction.

6.2.3. Parameters and Computational Costs

A crucial aspect of hashing-based retrieval is efficiency. The paper compares the parameters and computational costs (FLOPs) of CMBH during inference against other SOTA methods.

The following are the results from Table 5 of the original paper:

Analysis:

• Significantly Fewer Parameters: CMBH consistently demonstrates a substantially lower number of parameters during inference compared to other methods, for both ResNet50 and ResNet18 backbones.

  • With ResNet50, CMBH has around 14.3M parameters, which is much less than SEMICON (42.3M), CNET (41.1M), and FISH (23.9M).
  • The difference is even more pronounced with ResNet18. CMBH uses approximately 4.2M parameters, while FISH and sRLH use around 11.2M. This reduction in parameters directly contributes to lower memory footprint and faster model loading.

• Comparable or Lower FLOPs: In terms of computational costs (FLOPs), CMBH is comparable to or even lower than SOTA methods during inference.

  • For ResNet50, CMBH has 4.3492G FLOPs, which is lower than SEMICON (7.0910G) and CNET (9.3546G), and slightly higher than FISH (4.1321G).
  • For ResNet18, CMBH achieves the lowest FLOPs at 1.6696G, compared to FISH (1.8236G) and sRLH (1.8235G).

• Resolution of Efficiency Contradiction: When combined with the superior MAP scores from Tables 2 and 3, these efficiency metrics conclusively prove that CMBH successfully addresses the efficiency contradiction. It achieves state-of-the-art retrieval accuracy with significantly reduced parameters and comparable/lower computational costs during inference. This is a direct result of its design philosophy where complex multi-layer and multi-region modules are used only for training to optimize C-vectors, thus not burdening the inference pipeline.

  In summary, the experimental results convincingly demonstrate that CMBH offers a compelling solution for efficient fine-grained image retrieval by simultaneously achieving high effectiveness and efficiency, particularly excelling in scenarios demanding compact hash codes.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper rigorously analyzes two fundamental and often overlooked contradictions in the design of hashing-based models for efficient fine-grained image retrieval (FGIR): the feature learning contradiction (limited information capacity of hash codes vs. detailed fine-grained features) and the efficiency contradiction (complex feature extraction models vs. efficient retrieval).

To address these, the authors propose a novel Characteristics Matching Based Hashing (CMBH) method. The core innovation of CMBH lies in generating hash codes by evaluating the degree of matching between an image's feature vector and a set of abstract Characteristic vectors (C-vectors), rather than directly embedding comprehensive image features. This approach simplifies the information that needs to be encoded in hash codes, making them more effective at capturing fine-grained differences even at very short lengths.

Furthermore, CMBH incorporates two auxiliary modules—a cross-layer semantic information transfer module and a multi-region feature embedding module—which are crucial for robust C-vector optimization but are designed to operate exclusively during the training phase. This modular design allows for rich, fine-grained feature learning to inform C-vectors without introducing additional parameters or computational costs during inference, thereby resolving the efficiency contradiction.

Extensive experiments on five widely-used fine-grained datasets demonstrate CMBH's superior performance in both effectiveness (MAP) and efficiency (parameters and FLOPs) compared to numerous state-of-the-art methods. The improvements are particularly notable for short hash code lengths, making CMBH a highly practical solution for large-scale FGIR systems.

7.2. Limitations & Future Work

The paper explicitly points out one direction for future work regarding the hyper-parameter $\alpha$:

• Adaptive Adjustment of $\alpha$: The analysis of $\alpha$ in Figure 2b suggests that different values of $\alpha$ might be optimal for different hash code lengths. The authors propose that adaptively adjusting the value of $\alpha$ based on the hash code lengths could lead to universal performance improvements. This implies that the current fixed $\alpha$ might not be globally optimal across all scenarios and that a more dynamic approach could further enhance the model's robustness and performance.

  While not explicitly stated as limitations, some implicit considerations for future work could include:

• Computational Cost of Training: Although the method is highly efficient during inference, the training phase involves multiple loss functions, multi-layer feature extraction, and multi-region processing. While these are crucial for C-vector optimization, the overall training time might be considerable, especially on very large datasets. Future work could explore ways to accelerate the training process without compromising the quality of C-vectors.

• Sensitivity to $k$: The hyper-parameter $k$ (number of C-vectors per category) impacts performance. While $k=2$ was chosen empirically, finding an optimal or dynamically determined $k$ for different datasets or levels of fine-grainedness could be beneficial.

• Generalizability of C-vectors: The C-vectors are specific to the trained categories. For open-set FGIR or zero-shot FGIR, adapting CMBH would require strategies to learn or generalize C-vectors for novel categories.

7.3. Personal Insights & Critique

The CMBH paper offers a refreshing and principled approach to fine-grained hashing by challenging the conventional wisdom of directly embedding complex features into binary codes.

• Innovation and Paradigm Shift: The concept of characteristics matching is highly innovative. By abstracting fine-grained differences into C-vectors and then basing hash codes on matching scores, the paper provides an elegant solution to the information bottleneck of hashing. This paradigm shift—from detailed description to abstract matching—is a significant conceptual leap that aligns well with how human cognition operates when distinguishing subtle differences. Instead of processing every detail, humans often rely on identifying key, salient characteristics. This intuitive alignment suggests a promising direction for future research in hashing and compact representation learning.

• Effectiveness and Efficiency: The experimental results are very strong, particularly the significant gains at short hash code lengths and the superior efficiency during inference. This dual achievement of both effectiveness and efficiency is precisely what the field of large-scale retrieval demands. The clever design of having auxiliary modules operate only during training is a key enabler for this.

• Potential for Transferability: The characteristics matching idea might be transferable beyond FGIR. For any task where subtle differences are important but low-dimensional representations are required, a similar approach could be adopted. For example, medical image retrieval (distinguishing between similar disease sub-types) or material science image analysis.

• Potential Improvement Areas:

  • Interpretability of C-vectors: While C-vectors are designed to represent "critical characteristics," their exact semantic meaning or visual manifestation might not be directly interpretable. Future work could explore methods to visualize or interpret what specific fine-grained attributes each C-vector has learned to capture, which could provide deeper insights and build more trustworthy FGIR systems.

  • Robustness to Adversarial Attacks: Fine-grained models can be vulnerable to adversarial attacks. It would be interesting to investigate how CMBH's characteristics matching approach performs under such attacks, especially given the abstraction in its hash code generation.

  • Online/Continual Learning: For dynamic real-world applications where new fine-grained categories emerge over time, the process of optimizing C-vectors for new categories within the CMBH framework would need to be investigated.

    Overall, CMBH is a well-argued, technically sound, and empirically validated contribution that advances the state-of-the-art in efficient fine-grained image retrieval and offers a novel perspective for tackling the effectiveness-efficiency trade-off in compact representation learning.