Initializing viewer…

English Analysis

Details

1. Bibliographic Information

Title: jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval
Authors: Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, and Han Xiao. All authors are affiliated with Jina AI GmbH, Berlin, Germany.
Journal/Conference: The paper is available on arXiv, which is a preprint server for academic papers in fields like physics, mathematics, and computer science. This means it has not yet undergone formal peer review for a conference or journal but is shared to disseminate findings quickly.
Publication Year: The paper ID 2506.18902v3 suggests a submission in June 2025 (version 3).
Abstract: The authors introduce jina-embeddings-v4, a large 3.8 billion parameter embedding model designed for both text and images. Its key features include a novel architecture that supports both traditional single-vector embeddings and more precise multi-vector (late interaction) embeddings. The model is enhanced with task-specific Low-Rank Adaptation (LoRA) adapters to excel in different retrieval scenarios like document search, semantic similarity, and code search. The paper claims state-of-the-art performance in both single-modal (e.g., text-to-text) and cross-modal (e.g., text-to-image) retrieval, with exceptional ability in understanding visually rich documents like tables and charts. To support this claim, they also introduce a new benchmark called Jina-VDR for this specific task.
Original Source Link:
- arXiv Page: https://arxiv.org/abs/2506.18902
- PDF Link: http://arxiv.org/pdf/2506.18902v3
- Status: Preprint on arXiv.

2. Executive Summary

Background & Motivation (Why):
- Core Problem: Modern AI applications require understanding and searching through diverse data types—text, images, code, and complex documents containing both. Traditionally, this required deploying multiple specialized models, one for each data type or task (e.g., one for text search, another for image search). This approach is inefficient, costly, and complex to maintain.
- Existing Gaps: Many multimodal models, especially older ones, suffer from a "modality gap," where embeddings for text and images live in separate regions of the vector space, hindering effective cross-modal search. Furthermore, existing benchmarks for visually rich documents (like screenshots or PDFs) are often limited in scope, focusing primarily on question-answering tasks.
- Innovation: This paper introduces jina-embeddings-v4 as a "universal" solution. It is a single, unified model that can process text, images, code, and visually rich content. It offers the flexibility of both fast single-vector search and high-precision multi-vector search. By using a modern Vision-Language Model (VLM) architecture, it aims to close the modality gap and provide state-of-the-art performance across a wide array of retrieval tasks.
Main Contributions / Findings (What):
1. A Unified Multimodal Model: They present jina-embeddings-v4, a 3.8B parameter model based on the Qwen2.5-VL architecture that projects text and images into a shared semantic space.
2. Dual Embedding Formats: The model uniquely supports two output types:
  - Single-vector: A standard dense embedding (2048 dimensions) for efficient, broad-strokes retrieval.
  - Multi-vector: A sequence of token-level embeddings for high-precision, late-interaction style retrieval (like ColBERT).
3. Task-Specific LoRA Adapters: The model includes small, interchangeable LoRA adapters (60M parameters each) that optimize it for specific tasks: asymmetric retrieval (query-document search), semantic similarity, and code retrieval.
4. A New Benchmark for Visually Rich Documents: They introduce Jina-VDR, a comprehensive and multilingual benchmark for retrieving visually complex documents like charts, tables, maps, and advertisements, going beyond the limitations of existing benchmarks.
5. State-of-the-Art Performance: The paper demonstrates through extensive evaluations that jina-embeddings-v4 achieves top-tier results on text, code, and cross-modal retrieval benchmarks, and significantly outperforms other models on visually rich document retrieval.
6. Reduced Modality Gap: The model's unified architecture is shown to substantially reduce the modality gap compared to dual-encoder models like CLIP, leading to better-aligned text and image representations.

Foundational Concepts:
- Semantic Embeddings: A method to represent data (like words, sentences, or images) as numerical vectors in a high-dimensional space. The key idea is that items with similar meanings will have vectors that are close to each other (e.g., measured by cosine similarity). This enables semantic search, clustering, and other AI tasks.
- Multimodal vs. Single-Modal Models: Single-modal models handle one type of data (e.g., text only). Multimodal models are designed to process and understand information from multiple data types, like text and images, simultaneously.
- Dual-Encoder Architecture (e.g., CLIP): A common design for multimodal models where separate neural networks (encoders) process each modality independently. A text encoder processes text, and an image encoder processes images. During training, the model learns to align the output vectors so that a picture of a cat and the text "a photo of a cat" produce similar embeddings. A key weakness is the "modality gap."
- Unified VLM Architecture (e.g., Qwen2.5-VL): A more recent approach where a single, powerful language model forms the core. Images are first processed by a vision encoder into a sequence of "image tokens," which are then fed into the same language model that processes text tokens. This shared processing pathway helps create a more unified semantic space.
- Single-vector vs. Multi-vector (Late Interaction) Retrieval:
  - Single-vector (Dense): The entire input (e.g., a document) is compressed into a single vector. Search is fast—just find the nearest vectors. However, some nuance can be lost.
  - Multi-vector (Late Interaction): The input is represented as a sequence of vectors, one for each token. To compare a query and a document, a more complex calculation (like MaxSim in ColBERT) is performed. This is slower and requires more storage but is much more precise as it captures finer-grained interactions.
- LoRA (Low-Rank Adaptation): An efficient fine-tuning technique for large language models. Instead of retraining all billions of parameters, LoRA freezes the original model and trains only a few small, additional matrices ("adapters"). This dramatically reduces computational cost while achieving performance comparable to full fine-tuning.
- Contrastive Learning: A training method where the model learns by comparing samples. It is given positive pairs (items that should be similar) and negative pairs (items that should be dissimilar). The model's goal is to learn a representation that pulls positive pairs closer together and pushes negative pairs farther apart in the embedding space. InfoNCE is a popular loss function for this.
- Matryoshka Representation Learning (MRL): A training technique that makes embedding vectors "truncatable." It nests representations of different dimensions within a single large vector, ordering the dimensions by importance. This allows users to shorten the vector (e.g., from 2048 to 256) at inference time to trade off a small amount of precision for significant gains in speed and storage efficiency.
Previous Works & Differentiation:
- CLIP and jina-clip: These are foundational dual-encoder models for text-image tasks. jina-embeddings-v4 differs by using a unified VLM architecture, which the paper shows is superior for reducing the modality gap and achieving better cross-modal alignment.
- ColBERT and ColPali: These are pioneers in late-interaction retrieval for text and visually rich documents, respectively. jina-embeddings-v4 builds on this by being the first major model to offer both single-vector and multi-vector outputs in a single unified architecture, giving users a choice between speed and precision.
- jina-embeddings-v3: The predecessor to this model, which was text-only but introduced the concept of task-specific LoRA adapters. Version 4 extends this paradigm to the multimodal domain.
- Other VLM-based Embedding Models: The paper contrasts itself with other VLM-based models by highlighting its multilingual training, its support for both single and multi-vector retrieval, and the fact that it does not require task-specific instructions at inference time.
- ViDoRe and MIEB Benchmarks: These are existing benchmarks for visually rich documents. The paper argues they are limited (ViDoRe focuses on QA; MIEB extends it but is still nascent). Jina-VDR is introduced to provide a more diverse and comprehensive evaluation suite.

4. Methodology (Core Technology & Implementation)

The core of jina-embeddings-v4 is its unified architecture and dual-phase training strategy.

Model Architecture:

The model's architecture is visualized in the paper's Figure 1.

Figure 1: Architecture of jina-embeddings-v4. This diagram shows how the model takes text or image inputs, processes them through a shared pathway (vision encoder + Qwen2.5 LM decoder), and uses task-specific LoRA adapters. The output can be either a single, mean-pooled vector or a sequence of multi-vectors for late interaction retrieval.
1. Backbone: The model is built on Qwen2.5-VL-3B-Instruct, a 3.8 billion parameter Vision-Language Model.
2. Input Processing:
  - Text: Text is tokenized and converted into vector representations.
  - Images: Images are first processed by a vision encoder, which transforms the image into a sequence of "image tokens."
3. Unified Processing: Both the text tokens and image tokens are fed into the same Qwen2.5 language model decoder. This shared processing path is crucial for creating a unified semantic space and minimizing the modality gap.
4. Dual Output Modes:
  - Single-vector Output: To get a single dense vector, the final layer's token embeddings are averaged (mean pooling). This produces a 2048-dimensional vector, which can be truncated (down to 128 dimensions) thanks to MRL training.
  - Multi-vector Output: To get multi-vector embeddings for late interaction, an additional projection layer is applied to the unpooled token embeddings from the base model, resulting in a 128-dimensional vector for each input token.
5. Task-Specific LoRA Adapters: The model integrates three LoRA adapters that modify the behavior of the frozen backbone for specific tasks:
  - retrieval: For asymmetric search (e.g., short query, long document).
  - text-matching: For symmetric search (e.g., finding similar documents).
  - code: For retrieving code snippets.
    
    A summary of the model specifications is provided in Table 1.
    
    | | | :--- | :--- | Model Parameters | 3.8 billion (3.8 × 10⁹) plus 60M per LoRA | Text Input Size | Up to 32,768 tokens | Image Input | All images resized to 20 megapixels | Single-vector Embedding Size | 2048 dimensions, truncatable down to 128 | Multi-vector Embedding Size | 128 dimensions per token
Note: This table is a transcription of Table 1 from the original paper.
Training Method:

The training is performed in two main phases, with the backbone model weights remaining frozen throughout. Only the LoRA adapters and the multi-vector projection layer are trained.

Phase 1: General Pair Training A single LoRA adapter is trained on a massive dataset of text-text and text-image pairs using a contrastive objective. The goal is to teach the model general semantic similarity for both modalities and both output types simultaneously.
- Loss Function: A key innovation is the joint loss function that co-trains for both single-vector and multi-vector similarity.
  - For multi-vector similarity, the late interaction score is calculated and normalized by the query length for training stability. The formula for the late interaction score $s_{\mathrm{late}}(q, p)$ is: $s _ { \mathrm { l a t e } } ( \boldsymbol { q } , \boldsymbol { p } ) = \sum _ { i = 1 } ^ { n } \operatorname* { m a x } _ { \boldsymbol { j } \in \{ 1 , \dots , m \} } \pmb { q } _ { i } \cdot \pmb { p } _ { j } ^ { T }$ Where $\boldsymbol{q}_i$ is the $i$ -th token embedding of the query and $\boldsymbol{p}_j$ is the $j$ -th token embedding of the document. The score is the sum of maximum cosine similarities for each query token against all document tokens.
  - The training uses the InfoNCE loss function ( $\mathcal{L}_{\mathrm{NCE}}$ ) to push positive pairs together and negative pairs apart. $\mathcal { L } _ { \mathrm { NCE } } ( \mathbf { S } ( \mathcal { B } ) , \tau ) : = - \sum _ { i , j = 0 } ^ { n } \mathrm { s o f t m a x } ( \mathbf { S } ( \mathcal { B } ) , \tau , i , i )$ Where $\mathbf{S}(\mathcal{B})$ is the matrix of similarity scores for a batch $\mathcal{B}$ , and $\tau$ is a temperature hyperparameter.
  - To balance the training between the more accurate multi-vector outputs and the single-vector outputs, knowledge distillation is used via the Kullback-Leibler (KL) divergence loss ( $\mathcal{L}_{D}$ ). This encourages the single-vector similarity distribution to match the multi-vector one. $\mathcal { L } _ { D } ( B , \tau ) : = D _ { \mathrm { K L } } ( \mathbf { S } _ { \mathrm { d e n s e } } ^ { \prime } ( \mathcal { B } ) \| \mathbf { S } _ { \mathrm { l a t e } } ^ { \prime } ( \mathcal { B } ) )$
  - The final joint loss function combines these components for both text (txt) and multimodal (multi) batches with weights $w_1, \dots, w_6$ : $\begin{array} { r l } & { \mathcal { L } _ { \mathrm { j o i n t } } ( \mathcal { B } _ { \mathrm { t x t } } , \mathcal { B } _ { \mathrm { m u l t i } } , \tau ) : = } \\ & { \quad \quad \quad w _ { 1 } \mathcal { L } _ { \mathrm { NCE } } ( \mathbf { S } _ { \mathrm { d e n s e } } ( \mathcal { B } _ { \mathrm { t x t } } ) , \tau ) } \\ & { \quad \quad \quad + w _ { 2 } \mathcal { L } _ { \mathrm { NCE } } ( \mathbf { S } _ { \mathrm { l a t e } } ( \mathcal { B } _ { \mathrm { t x t } } ) , \tau ) + w _ { 3 } \mathcal { L } _ { D } ( \mathcal { B } _ { \mathrm { t x t } } ) } \\ & { \quad \quad \quad + w _ { 4 } \mathcal { L } _ { \mathrm { NCE } } ( \mathbf { S } _ { \mathrm { d e n s e } } ( \mathcal { B } _ { \mathrm { m u l t i } } ) , \tau ) } \\ & { \quad \quad \quad + w _ { 5 } \mathcal { L } _ { \mathrm { NCE } } ( \mathbf { S } _ { \mathrm { l a t e } } ( \mathcal { B } _ { \mathrm { m u l t i } } ) , \tau ) + w _ { 6 } \mathcal { L } _ { D } ( \mathcal { B } _ { \mathrm { m u l t i } } ) } \end{array}$
  - The training data also includes hard negatives (items that are semantically close but incorrect matches) to improve the model's fine-grained discrimination ability.
Phase 2: Task-Specific Training The general-purpose LoRA adapter from Phase 1 is duplicated three times, and each copy is further fine-tuned for a specific task.

Task Name Description

retrieval Asymmetric retrieval for queries and documents

text-matching Semantic text similarity and symmetric retrieval

code Retrieving code snippets

Note: This table is a transcription of Table 2 from the original paper.
- Asymmetric Retrieval Adapter: Uses distinct prefixes (e.g., "query:", "passage:") to signal to the model whether it is encoding a query or a document, enabling it to produce specialized embeddings for each.
- Text Matching Adapter: For tasks requiring nuanced similarity scores (not just relevance), this adapter is trained with the CoSENT loss function ( $\mathcal{L}_{\mathrm{co}}$ ) on datasets with ground-truth similarity values. $\mathcal { L } _ { \mathrm { c o } } ( \mathbf { S } ( \boldsymbol { B } ) , \tau ) : = \ln \Big [ 1 + \sum _ { \begin{array} { c } { ( q _ { 1 } , p _ { 1 } ) , } \\ { ( q _ { 2 } , p _ { 2 } ) } \end{array} } \frac { e ^ { s ( q _ { 2 } , p _ { 2 } ) } - e ^ { s ( q _ { 1 } , p _ { 1 } ) } } { \tau } \Big ]$ Where $\zeta(q, p)$ is the ground truth similarity, and the model learns to order pairs correctly based on their scores.
- Code Adapter: Fine-tuned on triplets of (query, positive code snippet, negative code snippet) from datasets like CodeSearchNet to specialize in natural language-to-code retrieval.

5. Experimental Setup

Datasets:
- Jina-VDR: The new benchmark introduced in this paper for visually rich document retrieval. It extends the existing ViDoRe benchmark with 30 new tasks covering diverse domains (legal, historical, marketing), formats (charts, tables, maps), and languages.
- ViDoRe: An existing benchmark for vision document retrieval, focused on QA over charts, tables, and PDFs.
- MTEB & MMTEB: Standard and widely used benchmarks for evaluating English and multilingual text embedding models on various tasks (retrieval, clustering, STS).
- LongEmbed: A benchmark specifically for evaluating retrieval performance on long documents.
- CLIP Benchmark: A standard suite for evaluating text-to-image retrieval performance, including datasets like flickr30k and mscoco.
- MTEB-CoIR: A comprehensive benchmark for evaluating code retrieval models.
Evaluation Metrics:
1. nDCG@k (Normalized Discounted Cumulative Gain at k):
  - Conceptual Definition: A ranking quality metric used for information retrieval tasks. It evaluates how good a ranked list of search results is. It rewards placing highly relevant documents at the top of the list and penalizes placing them lower down. The score is normalized to be between 0 and 1, where 1 represents a perfect ranking.
  - Mathematical Formula: $\mathrm{DCG}_k = \sum_{i=1}^{k} \frac{rel_i}{\log_2(i+1)} \quad ; \quad \mathrm{nDCG}_k = \frac{\mathrm{DCG}_k}{\mathrm{IDCG}_k}$
  - Symbol Explanation:
    - $k$ : The number of top results to consider (e.g., nDCG@10).
    - $rel_i$ : The graded relevance of the result at position $i$ .
    - $IDCG_k$ : The Ideal Discounted Cumulative Gain, which is the DCG score of a perfect ranking.
2. Spearman's Rank Correlation Coefficient (ρ):
  - Conceptual Definition: Used for Semantic Textual Similarity (STS) tasks. It measures how well the model's similarity scores for pairs of sentences align with human judgments. A score of +1 indicates a perfect positive correlation, -1 a perfect negative correlation, and 0 no correlation.
  - Mathematical Formula: $\rho = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)}$
  - Symbol Explanation:
    - $d_i$ : The difference in ranks for the $i$ -th pair between the model's prediction and the ground truth.
    - $n$ : The total number of pairs.
3. Recall@k:
  - Conceptual Definition: Used for text-to-image retrieval. It measures the percentage of queries for which the correct image is found within the top $k$ retrieved results.
  - Mathematical Formula: $\mathrm{Recall}@k = \frac{\text{Number of queries with correct item in top } k}{\text{Total number of queries}}$
Baselines: The paper compares jina-embeddings-v4 against a wide range of state-of-the-art models, including:
- Text Embedding Models: OpenAI's text-embedding-3-large, bge-m3, multilingual-e5-large-instruct, voyage-3, Google's gemini-embedding-001.
- Multimodal Models: jina-clip-v2, nllb-clip-large-siglip, colpali-v1.2, dse-qwen2-2b-mrl-v1, voyage-multimodal-v3.
- Code Embedding Models: voyage-code, jina-embedings-v2-code.
- Previous Jina Model: jina-embeddings-v3.

6. Results & Analysis

The paper presents a comprehensive evaluation, demonstrating strong performance across all targeted areas.

Core Results:

Table 3 provides a high-level summary of average scores across different benchmark suites. jina-embeddings-v4 (both dense/single-vector and late/multi-vector versions) consistently ranks at or near the top.

Model	J-VDR	ViDoRe	CLIPB	MMTEB	MTEB-en	COIR	LEMB	STS-m	STS-en
jina-embeddings-v4 (dense)	73.98	84.11	84.11	66.49	55.97	71.59	67.11	72.70	85.89
jina-embeddings-v4 (late)	80.55	90.17
text-embedding-3-large				59.27	57.98	62.36	52.42	70.17	81.44
bge-m3				55.36			58.73
multilingual-e5-large-instruct				57.12			41.76
jina-embeddings-v3	47.82	26.02		58.58	53.47
					54.33	55.07	55.66	75.77	85.82
voyage-3				66.13	53.46	67.23	74.06	68.33	78.59
gemini-embedding-001				67.71	64.35	73.11		78.35	85.29
jina-embedings-v2-code						52.24
voyage-code						77.33
nllb-clip-large-siglip			83.19
jina-clip-v2	40.52	53.61	81.12

colpali-v1.2 (late)	63.80	83.90
dse-qwen2-2b-mrl-v1 (dense)	67.25	85.80
voyage-multimodal-v3 (dense)		84.24

Note: This table is a transcription of Table 3 from the original paper. Blank cells indicate data was not reported. Some model names were truncated in the original table.

Visually Rich Document Retrieval (J-VDR & ViDoRe): This is where jina-embeddings-v4 shows its greatest strength. It establishes a new state-of-the-art on both benchmarks, significantly outperforming previous models. The multi-vector (late) version achieves an impressive score of 80.55 on the diverse J-VDR benchmark and 90.17 on ViDoRe.
Text Retrieval (MTEB & MMTEB): The model shows very competitive performance, outperforming its predecessor jina-embeddings-v3 and holding its own against top models like gemini-embedding-001.
Semantic Textual Similarity (STS): The model is best-in-class for English STS tasks (STS-en score of 85.89) and highly competitive on multilingual STS tasks.
Code Retrieval (COIR): It performs well for a general-purpose model, though the specialized voyage-code model has a slight edge.
Cross-Modal Retrieval (CLIPB): jina-embeddings-v4 achieves the highest average score of 84.11, demonstrating strong text-to-image search capabilities.

Analysis of the Embedding Space:

The paper provides a compelling analysis of why its architecture is superior to traditional dual-encoder (CLIP-style) models.
- Modality Gap: Figure 2 shows the distribution of cosine similarities for matching pairs. In CLIP models (top, middle), there is a clear gap: text-text pairs have high similarity, while image-text pairs have much lower similarity. In jina-embeddings-v4 (bottom), the distributions for both pair types significantly overlap in the high-similarity region, indicating the modality gap has been dramatically reduced.
  
  Figure 2: Distribution of cosine similarities for matched pairs. The plots compare OpenAI CLIP (top), jina-clip-v2 (middle), and jina-embeddings-v4 (bottom). The reduced gap between the blue (Image-Text) and pink (Text-Text) distributions in the bottom plot shows the improved cross-modal alignment of jina-embeddings-v4.
- Cross-Modal Alignment: Table 4 quantifies this observation, showing that jina-embeddings-v4 has a much higher cross-modal alignment score (average similarity of matching image-text pairs) than CLIP-style models.
- Cone Effect: Figure 3 visualizes the separation between positive (correct) and negative (incorrect) image-text pairs. In CLIP (top), the distributions are very close, a symptom of the "cone effect" where all embeddings are clustered in a narrow cone, making discrimination difficult. In contrast, jina-embeddings-v4 (bottom) shows a clear and wide separation between the positive (blue) and negative (orange) distributions, indicating it uses the embedding space more effectively.
  
  Figure 3: Distribution of cosine similarities for positive vs. negative pairs. The plots show OpenAI CLIP (top), jina-clip-v2 (middle), and jina-embeddings-v4 (bottom). The clear separation of peaks for positive (blue) and negative (orange) samples in the bottom plot demonstrates the superior discriminative power of jina-embeddings-v4.

7. Conclusion & Reflections

Conclusion Summary: The paper successfully introduces jina-embeddings-v4 as a powerful, versatile, and state-of-the-art embedding model. Its unified VLM architecture, dual-output functionality, and task-specific LoRA adapters make it a highly practical tool for a wide range of multimodal and multilingual retrieval applications. Its key achievements are setting a new standard for visually rich document retrieval and demonstrating a concrete architectural solution to the long-standing modality gap problem in cross-modal systems.
Limitations & Future Work: The authors acknowledge the model's large size as a potential barrier and state their intention to explore smaller, more efficient variants in the future. They also plan to further enhance the model's multilingual capabilities.
Personal Insights & Critique:
- Strengths:
  - The paper's primary strength is its practicality and completeness. It doesn't just propose a novel architecture; it delivers a fully-fledged model with different adapters and output modes, addressing real-world trade-offs between speed and accuracy.
  - The introduction of the Jina-VDR benchmark is a significant contribution to the community. By providing a diverse and challenging testbed, it will likely drive further innovation in the field of visually rich document understanding.
  - The analysis of the embedding space (modality gap, cone effect) is rigorous and provides clear, intuitive evidence for the superiority of the unified VLM architecture over dual-encoders for retrieval tasks.
- Potential Weaknesses & Open Questions:
  - Computational Cost: At 3.8B parameters, the model is computationally demanding for inference, especially the multi-vector mode. While the authors mention future work on smaller models, the current version may be inaccessible for resource-constrained applications.
  - Dependence on Backbone Model: The model's capabilities, particularly its multilingual support and potential biases, are fundamentally tied to its Qwen2.5-VL backbone. As the authors note, its performance dips on languages not well-represented in the backbone's pre-training data.
  - Ablation Studies: While the overall results are strong, the paper could have benefited from more detailed ablation studies to quantify the individual contributions of different components, such as the D_KL distillation loss or the specific composition of the training data for visually rich documents.

Task Name	Description
`retrieval`	Asymmetric retrieval for queries and documents
`text-matching`	Semantic text similarity and symmetric retrieval
`code`	Retrieving code snippets