1. Bibliographic Information

1.1. Title

The central topic of this paper is the creation of a comprehensive dataset for parsing diverse structured texts in academic literature and the development of a multimodal model that leverages this dataset. The title AceParse: A Comprehensive Dataset with Diverse Structured Texts for Academic Literature Parsing clearly reflects this focus.

1.2. Authors

The authors of this paper are:

• Huawei Ji

• Cheng Deng

• Bo Xue

• Zhouyang Jin

• Jiaxin Ding (corresponding author, indicated by *)

• Xiaoying Gan

• Luoyi Fu

• Xinbing Wang

• Chenghu Zhou

  Their affiliations are:

• Shanghai Jiao Tong University, Shanghai, China (for authors Ji, Deng, Xue, Jin, Ding, Gan, Fu, Wang)

• IGSNRR (Institute of Geographic Sciences and Natural Resources Research), Chinese Academy of Sciences, Beijing, China (for Chenghu Zhou)

  The authors appear to be primarily affiliated with academic institutions in China, with Shanghai Jiao Tong University being a prominent research university known for its engineering and computer science programs. This suggests a research background in areas like artificial intelligence, natural language processing, and data science.

1.3. Journal/Conference

The paper is published as a preprint on arXiv (identified by the Original Source Link). While arXiv is not a peer-reviewed journal or conference, it is a widely used open-access repository for preprints in fields like physics, mathematics, computer science, and more. Publishing on arXiv allows for rapid dissemination of research findings and early feedback from the scientific community. The version $v2$ indicates that the paper has been revised at least once since its initial submission.

1.4. Publication Year

The publication date (UTC) is 2024-09-16T06:06:34.000Z, which indicates a publication in the year 2024.

1.5. Abstract

The paper addresses the challenge of parsing diverse structured texts within academic literature, which is predominantly stored in PDF format. Motivated by the shift towards data-centric AI, the authors highlight the lack of comprehensive datasets covering various text structures as a key limitation. To overcome this, they introduce AceParse, the first open-source dataset specifically designed for parsing a wide range of structured texts, including formulas, tables, lists, algorithms, and sentences with embedded mathematical expressions. Using AceParse, they fine-tuned a multimodal model named AceParser, which significantly outperforms the previous state-of-the-art methods. AceParser achieves a 4.1% improvement in F1 score and a 5% improvement in Jaccard Similarity, demonstrating the potential of multimodal models in this domain. The dataset is made publicly available.

1.6. Original Source Link

The original source link is https://arxiv.org/abs/2409.10016v2. This is a link to the arXiv preprint repository, indicating the paper is a preprint and has not yet undergone formal peer review or publication in a journal or conference proceedings.

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the challenging task of accurately parsing diverse structured texts from academic literature, which is typically stored in PDF (Portable Document Format) files. This problem is crucial because academic literature contains a wealth of scientific knowledge but is often in a non-machine-readable format, making it difficult for AI (Artificial Intelligence) systems to process and understand.

This problem is particularly important in the context of data-centric AI, a paradigm where the emphasis shifts from merely developing advanced models to improving the quality, diversity, and representativeness of the data used for training. High-quality data is foundational for building robust AI systems, and academic literature represents a critical, yet underutilized, data source due to parsing difficulties.

Specific challenges and gaps in prior research include:

• OCR-based (Optical Character Recognition) methods: These primarily focus on character recognition and often lose critical structural information (e.g., the hierarchical arrangement of a list or the relationships within a table).

• Modular approaches: While capable of handling specific content types like tables or formulas, they struggle with complex structures such as algorithms and lists, and often suffer from error accumulation due to a lack of integration between different modules.

• End-to-end parsing models (e.g., Nougat): These are often trained on limited, proprietary datasets that lack diversity in structured content, hindering their ability to generalize across various document structures.

• Existing open-source datasets: These are typically limited to character-level parsing or focus on only one specific type of structured content (e.g., tables or formulas), failing to cover the full spectrum of structured elements found in academic documents.

  The paper's entry point or innovative idea is to address this data gap by creating AceParse, the first comprehensive, open-source dataset that covers a wide array of structured text types in academic literature. This dataset then enables the training of a multimodal model, AceParser, which can handle this diversity effectively.

2.2. Main Contributions / Findings

The paper makes several primary contributions:

• AceParse Dataset: Introduction of AceParse, the first comprehensive, open-source dataset specifically designed for parsing diverse structured texts (including formulas, tables, lists, algorithms, and embedded mathematical expressions) from academic literature. This dataset comprises 500k parsed document pairs, annotated using LaTeX markup, which directly addresses the limitation of existing datasets that either lack diversity or structure-aware annotations.

• AceParser Model: Development and fine-tuning of AceParser, an end-to-end multimodal model based on the Florence-2 architecture. This model is capable of generating structured text in markup languages, effectively parsing the varied content types in academic literature.

• State-of-the-Art Performance: AceParser achieves superior parsing performance compared to previous state-of-the-art methods. It demonstrates a 4.1% improvement in F1 score and a 5% improvement in Jaccard Similarity over existing approaches.

• Comprehensive Comparison and Overview: The paper systematically compares current parsing methods and provides an extensive overview of existing parsing datasets, serving as a valuable reference for the document parsing community.

  The key conclusions and findings are:

• There is a critical need for datasets that cover diverse structured content to advance academic literature parsing.

• The data synthesis approach used to create AceParse is an effective method for generating high-quality image-annotation pairs for complex structured content.

• Multimodal models, when trained on sufficiently diverse and structured datasets like AceParse, show significant potential for accurately parsing academic literature, outperforming traditional OCR-based, modular, and even previous end-to-end methods.

• The LaTeX markup language is an effective way to represent the structural information of diverse texts for parsing tasks.

  These findings solve the problem of accurately extracting and representing the rich, structured information embedded within academic PDFs, which is a prerequisite for advanced AI applications in scientific research.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a reader should be familiar with the following concepts:

• Data-centric AI: This paradigm shifts the focus from optimizing AI models to optimizing the data itself. Instead of iteratively improving model architectures, data-centric AI emphasizes enhancing the quality, consistency, and representativeness of training data. The core idea is that better data leads to better AI performance, often more effectively than complex model changes.

• Academic Literature Parsing: This refers to the process of extracting structured information (text, tables, figures, formulas, references) from academic documents, typically in PDF format, and converting it into a machine-readable format. The goal is to preserve both the content and its underlying structure, enabling AI systems to understand and reason about scientific knowledge.

• PDF (Portable Document Format): A file format used to present documents, including text formatting and images, in a manner independent of application software, hardware, and operating systems. While PDFs are excellent for human readability and printing, extracting their underlying structure and content accurately for machine processing is challenging.

• Structured Texts: In the context of academic literature, these are text elements that have a defined logical or hierarchical organization beyond simple paragraphs. Examples include:

  • Formulas: Mathematical equations.
  • Tables: Data organized in rows and columns.
  • Lists: Ordered or unordered sequences of items.
  • Algorithms: Step-by-step procedures, often presented in a structured, pseudo-code format.
  • Sentences with embedded mathematical expressions: Regular sentences that include mathematical notation within them.

• LaTeX: A document preparation system widely used in academia for scientific and technical documents. It allows authors to define the structure and content of their documents using markup language commands, which are then compiled into PDF or other formats. LaTeX is highly effective for typesetting complex mathematical formulas, tables, and structured content, making it an ideal target format for structured text parsing.

• Multimodal Model: An AI model that can process and understand information from multiple modalities (types of data). In this paper, multimodal refers to models that can process both visual information (e.g., document images) and textual information (e.g., LaTeX annotations or text prompts) simultaneously, leveraging the strengths of each.

• F1 Score: A commonly used metric in classification tasks, particularly when evaluating performance on imbalanced datasets. It is the harmonic mean of precision and recall.

  • Precision: The proportion of true positive results among all positive results returned by the model. It answers: "Of all items the model identified as positive, how many are actually positive?"
  • Recall: The proportion of true positive results among all relevant (actual positive) results. It answers: "Of all actual positive items, how many did the model identify correctly?"
  • The F1 score balances precision and recall, providing a single measure that is high only if both are high.

• Jaccard Similarity (Jaccard Index): A statistic used for gauging the similarity and diversity of sample sets. It is defined as the size of the intersection divided by the size of the union of the sample sets. In text parsing, it can be used to compare the similarity between the predicted output text and the ground truth text, or between detected bounding boxes.

• Levenshtein Distance: Also known as edit distance, it measures the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one word or sequence into the other. A lower Levenshtein Distance indicates higher similarity between two strings.

• BLEU (Bilingual Evaluation Understudy): An algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. While originally for translation, it can be adapted to evaluate the similarity of generated text (e.g., parsed output) to a reference text, focusing on n-gram overlap.

• Vision-language model: A type of multimodal AI model specifically designed to understand and generate content that combines visual and linguistic information. These models can perform tasks like image captioning, visual question answering, and, in this case, parsing visual document layouts into structured text.

3.2. Previous Works

The paper discusses several categories of previous work, highlighting their limitations that AceParse and AceParser aim to address:

• OCR-based methods:

  • Tesseract [4] and PPOCR [5]: These are widely used Optical Character Recognition (OCR) engines.
  • Limitation: They primarily focus on recognizing characters and converting scanned images into plain text. While crucial for digitizing documents, they lack the ability to preserve structural information such as the layout of tables, the hierarchy of lists, or the logical connections within formulas. They treat all text as a flat sequence, leading to a loss of context essential for understanding scientific content. The paper mentions Tesseract and PPOCR showing lower performance because they are not structure-aware.

• Modular Approaches:

  • Mineru [7] and Pix2Text: These methods often involve separate modules for different content types (e.g., one module for tables, another for formulas).
  • Limitation: While they can handle predefined content types, they struggle with complex structures like algorithms or diverse lists. More critically, the lack of integration between these separate modules can lead to inconsistent outputs for interconnected content types, and error accumulation where errors from one module propagate to others. Pix2Text and Mineru are described as less competitive due to this.

• End-to-End Parsing Models:

  • Nougat [9]: A neural optical understanding model for academic documents.
  • Limitation: These models aim to parse documents in a single pass. However, existing end-to-end models like Nougat are often trained on narrow proprietary datasets with a limited diversity of structured content. This restricts their ability to generalize effectively across the wide range of structures found in diverse academic literature.

• Existing Open-Source Datasets:

  • DocLayNet [10] and DocBank [11]: These datasets primarily focus on character-level parsing or document layout analysis.
  • TableBank [12]: Focuses specifically on tables.
  • IM2LATEX [13]: Focuses specifically on formulas.
  • Limitation: They are limited to either character-level output without explicit structural markup, or they focus exclusively on a single type of structured text. None of them provide a comprehensive range of structured elements (formulas, tables, lists, algorithms, embedded math in sentences) annotated with a markup language like LaTeX that explicitly describes their structure. This gap is precisely what AceParse aims to fill.

3.3. Technological Evolution

The field of document parsing has evolved significantly:

1. Early OCR (e.g., Tesseract): Focused on character recognition from scanned images, converting pixels into text characters. The output was mostly flat text.
2. Layout Analysis and Structured OCR: Added capabilities to identify different document regions (e.g., paragraphs, headings, images) but still lacked deep understanding of complex internal structures.
3. Specialized Parsers (e.g., TableBank, IM2LATEX): Emergence of dedicated tools and datasets for specific structured elements like tables or mathematical formulas. These often used rule-based methods or specialized machine learning models.
4. Modular Deep Learning Approaches (e.g., Mineru, Pix2Text): Leveraged deep learning for individual parsing tasks, often chaining modules together. This improved performance for specific tasks but retained issues of integration and error propagation.
5. End-to-End Deep Learning Models (e.g., Nougat): Attempted to parse documents holistically using powerful neural networks, often vision-language models. These aimed to simplify the pipeline but were often limited by the diversity of their training data.
6. Multimodal, Data-Centric Approaches (AceParser): This paper's work fits into this latest stage, emphasizing the crucial role of diverse and comprehensive datasets in enabling multimodal end-to-end models (like AceParser fine-tuned from Florence-2) to achieve superior performance across a wide array of structured text types. The use of LaTeX as the target markup language is a key step towards rich, machine-readable representations.

3.4. Differentiation Analysis

Compared to the main methods in related work, AceParse and AceParser present several core differences and innovations:

• Comprehensive Structured Text Coverage: Unlike existing datasets that are limited to character-level parsing or focus on a single type of structured text (e.g., TableBank for tables, IM2LATEX for formulas), AceParse encompasses a broad range of structured texts, including formulas, tables, lists, algorithms, and sentences with embedded mathematical expressions. This diversity is its primary differentiator.
• LaTeX Annotations for Structure: AceParse annotates these diverse structured texts using the LaTeX markup language. This is a significant innovation as LaTeX explicitly describes the underlying structure, not just the raw text content, which is crucial for AI models to understand and regenerate these complex elements accurately. Previous general datasets often lack such rich, structure-aware annotations.
• Data Synthesis Approach: The dataset construction method relies on a data synthesis approach by randomly combining structured texts from LaTeX source code. This addresses the limitations of PDF page matching (which can lose structured text at page breaks) and the scalability challenges of collecting large volumes of academic literature. It allows for generating a large volume of high-quality image-annotation pairs efficiently.
• Multimodal, End-to-End Parsing: AceParser is an end-to-end multimodal model fine-tuned from Florence-2. Unlike OCR-based methods that lose structure or modular approaches that suffer from error accumulation, AceParser processes both visual (document image) and implicit textual information simultaneously to directly generate structured LaTeX output. This integrated approach leverages the power of modern vision-language models.
• Performance Leap: AceParser achieves state-of-the-art performance, outperforming previous methods (including Nougat) by 4.1% in F1 score and 5% in Jaccard Similarity, specifically on a dataset designed for diverse structured texts. This demonstrates the effectiveness of combining a comprehensive dataset with a powerful multimodal architecture.

4. Methodology

4.1. Principles

The core idea behind this paper's methodology is to enable unified and accurate parsing of diverse structured texts in academic literature by:

1. Creating a comprehensive, structure-aware dataset: This involves collecting a wide variety of structured text types and annotating them with LaTeX to explicitly capture their underlying structure.
2. Employing a data synthesis approach: To overcome the challenges of traditional data collection (e.g., PDF page matching issues, scalability), the authors synthesize new data by combining existing LaTeX components.
3. Leveraging a multimodal, end-to-end model: Utilizing a powerful vision-language model architecture, fine-tuned on the new dataset, to directly parse document images into structured LaTeX output. The theoretical basis is that modern transformer-based multimodal models can learn complex mappings between visual layouts and structured text representations if provided with sufficient and diverse training data.

4.2. Core Methodology In-depth (Layer by Layer)

The methodology consists of two main parts: Dataset Construction for AceParse and the AceParser Network Architecture.

4.2.1. Dataset Construction (AceParse)

The AceParse dataset is constructed using a novel data synthesis approach to overcome the limitations of previous methods, such as PDF page matching which often leads to loss of structured text at page ends. This approach ensures scalability and high-quality image-annotation pairs. The process is built upon three key dimensions:

4.2.1.1. Document Collection

The first step involves gathering raw LaTeX source files.

• Source: 10,000 open-access LaTeX source files were collected from 102 subfields within computer science on ArXiv. The selection was guided by ArXiv IDs listed in Papers with Code, following the method used in [15].
• Custom Parsing Scripts: To handle the inherent structural and formatting inconsistencies across various LaTeX files and subfields, custom parsing scripts were developed. These scripts were designed to normalize differing LaTeX conventions and ensure consistent content extraction.

4.2.1.2. Data Synthesis

This is the core of the AceParse dataset generation, aiming to create high-quality image-annotation pairs.

• Cleaning Source Code: A combination of rule-based techniques and domain-specific knowledge of academic writing and LaTeX syntax was applied to clean the collected LaTeX source files. This involved:

  • Filtering out irrelevant content.
  • Normalizing references and citations.
  • Standardizing or replacing non-standard commands to avoid issues like inconsistent citation formats, overly complex user-defined commands, and non-standard sectioning.

• Structured Text Extraction: After cleaning, the process focused on extracting specific diverse structured texts. This included:

  • Sentences with embedded structures (e.g., mathematical expressions within text).
  • Formulas.
  • Tables.
  • Lists.
  • Algorithms. Plain text sentences without embedded structures were excluded to focus specifically on complex structured content. This extraction resulted in over 700,000 structured items.

• Random Sampling and Combination: These 700,000+ structured items were then randomly sampled and combined to synthesize new LaTeX files.

• Compilation Challenge and Solution: A major challenge was ensuring that these randomly combined LaTeX files would compile successfully into PDFs despite the inherent randomness. The combination of different structures and custom commands can easily lead to compilation errors. To address this:

  • Strict Syntax Checks: Implemented pre-processing steps with strict syntax checks to catch potential issues before the final compilation stage.
  • Harmonizing Commands: Introduced a step to sanitize and harmonize conflicting user-defined commands from different sources.

• PDF Generation: After these adjustments, the synthesized TeX files were successfully compiled into PDFs using pdflatex1, ensuring that all structures were rendered correctly, despite the randomized content. This process generates the image-annotation pairs.

  The following figure (Figure 1 from the original paper) illustrates the overall workflow, including the data synthesis process:

  该图像是一个示意图，展示了AceParse数据集生成及AceParser模型结构流程，包括文档收集、数据合成、边界检测及多模态编码解码过程，展示了带公式的文本解析示例。

anbounydeteinThe etwork ritecurceParsVisaltokbedign ext toke i atenatetultoaokebdi whhprocese byBART-bamultalecoer

The provided figure description is malformed, likely due to OCR errors in the original abstract for the figure caption. Based on the image content, it depicts the AceParse dataset generation and AceParser model architecture.

The statistical attributes of the AceParse dataset are detailed in Figure 2. Each synthesized document image contains annotation text centered at 1,107 characters. The majority of images have dimensions centered at $974 \times 493$ pixels, with median values for width and height being 974 and 493 pixels, respectively. These dimensions balance resolution and file size for optimal training.

The following are the results from Table I of the original paper, comparing AceParse with other academic literature parsing datasets:

The following are the results from Table I of the original paper:

Note on Table I: "Modality" indicates the type of content the dataset is designed for. "Structured" refers to whether the labels are represented using markup languages. $X$ indicates no, ✓ indicates yes, and the blank or ; symbols likely indicate partial or unspecified structured labeling. $*AceParse$ includes various structured texts such as tables, formulas, lists, algorithms, etc.

4.2.1.3. Boundary Detection

This final step in dataset construction involves accurately extracting relevant image regions from the generated PDFs.

• PDF to Image Conversion: PDFs were first converted to images using $PyMuPDF^2$.

• Cropping: Pixel-level boundary detection was used to identify the corners of text areas, and images were cropped to focus on relevant content.

• Heuristic Rules: Heuristic rules were applied to detect page boundaries and discard samples with irregular layouts or distorted content. This ensures cleaner and more precise image extractions, critical for downstream AI tasks.

  The following figure (Figure 2 from the original paper) details the statistical attributes of the AceParse dataset:

  该图像是一个包含表格和两个图表的复合图。图(a)为AceParse数据集中结构化文本属性统计表，图(b)展示标签长度（字符数）分布直方图，图(c)为标签宽高（像素）密度热力图，反映文本区域尺寸特征。

SaarataTr cerc kernel density estimation of image dimensions.

4.2.2. AceParser Network Architecture

The AceParser model is an end-to-end multimodal academic literature parsing model that is fine-tuned based on the Florence-2 architecture [14]. Florence-2 is a robust multi-task multimodal model, pretrained on 5 billion data instances, and equipped with OCR capabilities. However, it lacks the inherent ability to parse structured texts into markup languages.

The Florence-2 architecture, which AceParser adopts, comprises two main components:

1. Vision Encoder: This component is DaViT [16] (Dual Attention Vision Transformers). Its role is to process the input image.

2. Multimodal Encoder-Decoder: This component is based on BART [17] (Bidirectional and Auto-Regressive Transformers). It integrates visual and textual information and generates the structured output.

   Here's how an input document image is processed:

• Input Image: A document image, denoted as $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$, where $H$ is height, $W$ is width, and 3 represents the RGB color channels.

• Vision Encoding: The input image $\mathbf{I}$ is divided into patches, and these patches are then embedded by the vision encoder (DaViT) into visual token embeddings $\mathbf{V} \in \mathbb{R}^{\bar{N_v} \times d}$.

  • $\bar{N_v}$: Represents the number of visual tokens generated from the image.
  • $d$: Denotes the dimension of the hidden layers (feature representation size).

• Task Prompt and Text Encoding: A task prompt (which guides the model on what to do, e.g., "parse document") is also provided. This prompt is embedded by a text embedding layer into text token embeddings $\mathbf{T} \in \mathbb{R}^{N_t \times d}$.

  • $N_t$: Represents the number of text tokens from the prompt.
  • $d$: Same hidden layer dimension.

• Multimodal Token Embeddings: The visual token embeddings $\mathbf{V}$ and text token embeddings $\mathbf{T}$ are then concatenated. Positional encoding is applied to these combined embeddings to retain spatial or sequential information. This results in the multimodal token embeddings $\mathbf{X} \in \mathbb{R}^{(N_v + \bar{N_t}) \times d}$.

• Multimodal Encoder Input: These multimodal token embeddings $\mathbf{X}$ serve as the input to the multimodal encoder (BART-based).

• Decoder and Fine-tuning: During training, AceParser is fine-tuned using teacher forcing and an autoregressive loss. Teacher forcing means that the actual previous output tokens from the ground truth annotation are fed as input to the decoder at each step, instead of the model's own predictions. The decoder then predicts the next token in the sequence. The autoregressive loss aims to maximize the probability of generating the correct sequence of LaTeX tokens given the input.

  The loss function used for training is: $$\mathcal { L } = - \sum _ { t = 1 } ^ { T } \log P ( y _ { t } | y _ { 1 : t - 1 } , x )$$ Where:

• $\mathcal{L}$: Represents the cross-entropy loss for a given sequence. The goal of training is to minimize this loss.

• $T$: Denotes the total number of tokens in the target LaTeX annotation sequence.

• $t$: Represents a specific time step in the sequence generation process (from 1 to $T$).

• $P(y_t | y_{1:t-1}, x)$: Represents the predicted probability of the actual token $y_t$ occurring at time step $t$, given the previous ground truth tokens $y_{1:t-1}$ (due to teacher forcing) and the input sequence $x$.

• $y_t$: Is the actual target LaTeX token at time step $t$.

• $y_{1:t-1}$: Are the sequence of actual target LaTeX tokens from the beginning up to the previous time step t-1.

• $x$: Represents the input sequence, which in this context refers to the multimodal token embeddings $\mathbf{X}$ (derived from the document image $\mathbf{I}$ and task prompt).

  This loss function essentially measures how well the model predicts each token in the LaTeX annotation sequence, given the visual input and the preceding correct tokens. The negative logarithm ensures that higher probabilities for the correct token result in lower loss, and the sum accumulates the loss over the entire output sequence.

The following are the results from Table II of the original paper, comparing AceParser with other academic literature parsing methods:

The following are the results from Table II of the original paper:

Note on Table II: "Params" refers to the number of model parameters. "Paradigm" describes whether the model uses a modular approach. "Structured" indicates whether the method can parse structured texts. $X$ indicates no, ✓ indicates yes, and $V$ indicates partial or ambiguous.

5. Experimental Setup

5.1. Datasets

The primary dataset used for experiments is the newly introduced AceParse dataset.

• Source: The AceParse dataset is synthesized from 10,000 open-access LaTeX source files collected from 102 subfields within computer science on ArXiv.

• Scale: The dataset includes 500,000 parsed document image-LaTeX annotation pairs. It contains over 700,000 structured items extracted during the data synthesis phase.

• Characteristics: It is comprehensive, covering a broad range of diverse structured texts including formulas, tables, lists, algorithms, and sentences with embedded mathematical expressions. The annotations are in LaTeX markup language, preserving structural information. The images in the dataset have dimensions centered at $974 \times 493$ pixels, optimizing for resolution and file size.

• Domain: Academic literature, specifically computer science subfields.

• Splitting: The AceParse dataset is divided into training, validation, and test sets with a ratio of 8:1:1, respectively. This means 80% for training, 10% for validation, and 10% for final evaluation. All reported comparison results are based on the held-out test set.

  These datasets were chosen because AceParse is specifically designed to address the limitations of existing datasets by providing a diverse and comprehensive collection of structured texts annotated with LaTeX. This makes it uniquely suited for validating methods aimed at unified academic literature parsing. The data synthesis approach ensures high quality and scalability, making the dataset effective for training and evaluating models like AceParser.

5.2. Evaluation Metrics

For every evaluation metric mentioned in the paper, here is a complete explanation:

5.2.1. Levenshtein Distance (LD)

• Conceptual Definition: Levenshtein Distance, also known as edit distance, quantifies the similarity between two strings (e.g., the predicted LaTeX output and the ground truth LaTeX annotation). It measures the minimum number of single-character edits (insertions, deletions, or substitutions) required to transform one string into the other. A lower Levenshtein Distance indicates greater similarity, meaning fewer edits are needed to match the predicted output to the ground truth.

• Mathematical Formula: Let $a$ and $b$ be two strings of lengths $|a|$ and $|b|$ respectively. The Levenshtein Distance lev(a, b) is calculated recursively: $ lev(a, b) = \begin{cases}

  |a| & \text{if } |b| = 0 \ |b| & \text{if } |a| = 0 \

  lev(\text{tail}(a), \text{tail}(b)) & \text{if } a[0] = b[0] \ 1 + \min \begin{cases} lev(\text{tail}(a), b) \ lev(a, \text{tail}(b)) \ lev(\text{tail}(a), \text{tail}(b)) \end{cases} & \text{if } a[0] \neq b[0] \end{cases} $ (The formula above is a common recursive definition, but in practice, it's typically computed using dynamic programming for efficiency, building up a matrix of distances between all prefixes of the two strings.)

• Symbol Explanation:

  • $a$: The first string (e.g., the model's predicted output).
  • $b$: The second string (e.g., the ground truth LaTeX annotation).
  • $|a|$, $|b|$: The lengths of strings $a$ and $b$, respectively.
  • a[0], b[0]: The first character of strings $a$ and $b$.
  • $\text{tail}(a)$, $\text{tail}(b)$: The strings $a$ and $b$ without their first character.
  • $\min$: The minimum function, selecting the smallest of its arguments.
  • $1 + \min(\dots)$: Represents the cost of one edit operation plus the minimum distance from the remaining substrings.
  • The paper reports LD ↓, indicating that a lower Levenshtein Distance is better.

5.2.2. BLEU (Bilingual Evaluation Understudy)

• Conceptual Definition: BLEU is a metric for automatically evaluating machine-generated text against a set of human-created reference texts. It primarily measures the precision of n-grams (contiguous sequences of $n$ items) in the candidate text compared to the reference texts, with a penalty for brevity. While originally developed for machine translation, it is adapted here to assess the similarity and quality of the generated LaTeX output compared to the ground truth, focusing on the overlap of token sequences. A higher BLEU score indicates greater similarity to the reference.
• Mathematical Formula: $ \text{BLEU} = \text{BP} \cdot \exp \left( \sum_{n=1}^{N} w_n \log p_n \right) $ Where: $ \text{BP} = \begin{cases} 1 & \text{if } c > r \ e^{(1 - r/c)} & \text{if } c \le r \end{cases} $ $ p_n = \frac{\sum_{\text{sentence} \in \text{corpus}} \sum_{\text{n-gram} \in \text{sentence}} \text{Count}{\text{clip}}(\text{n-gram})}{\sum{\text{sentence} \in \text{corpus}} \sum_{\text{n-gram} \in \text{sentence}} \text{Count}(\text{n-gram})} $
• Symbol Explanation:
  • $\text{BLEU}$: The BLEU score, ranging from 0 to 1 (or 0 to 100).
  • $\text{BP}$: The brevity penalty. This penalty factor is applied to penalize candidate sentences that are shorter than their reference translations.
  • $c$: The length of the candidate (predicted) text.
  • $r$: The effective reference corpus length (sum of the lengths of the reference sentences closest to the length of the candidate sentences).
  • $N$: The maximum n-gram order (e.g., typically 4 for 1-gram, 2-gram, 3-gram, 4-gram).
  • $w_n$: The weight for each n-gram (often uniform, e.g., $1/N$).
  • $p_n$: The modified n-gram precision for n-grams of length $n$.
  • $\text{Count}_{\text{clip}}(\text{n-gram})$: The count of an n-gram in the candidate text, clipped by its maximum count in any single reference text. This prevents overly high scores for candidates that simply repeat n-grams.
  • $\text{Count}(\text{n-gram})$: The count of an n-gram in the candidate text.
  • The paper reports BLEU ↑, indicating that a higher BLEU score is better.

5.2.3. F1 Score

• Conceptual Definition: The F1 score is a measure of a model's accuracy on a dataset. It is the harmonic mean of precision and recall, providing a single score that balances both. It is particularly useful when classes are unevenly distributed or when both false positives and false negatives are important to minimize. For text generation tasks, it can be applied token-wise or by comparing sets of n-grams.
• Mathematical Formula: $ \text{F1} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} $ Where: $ \text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}} $ $ \text{Recall} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}} $
• Symbol Explanation:
  • $\text{F1}$: The F1 score, typically ranging from 0 to 1 (or 0% to 100%).
  • $\text{Precision}$: The proportion of correctly identified positive cases among all cases predicted as positive.
  • $\text{Recall}$: The proportion of correctly identified positive cases among all actual positive cases.
  • $\text{True Positives}$: Instances correctly identified as positive.
  • $\text{False Positives}$: Instances incorrectly identified as positive (Type I error).
  • $\text{False Negatives}$: Instances incorrectly identified as negative (Type II error).
  • The paper reports F1 ↑, indicating that a higher F1 score is better.

5.2.4. Jaccard Similarity (JS)

• Conceptual Definition: Jaccard Similarity, or Jaccard Index, measures the similarity between two finite sample sets. It is defined as the size of the intersection of the sets divided by the size of their union. In the context of text parsing, this can be applied to sets of words, characters, or n-grams from the predicted output and the ground truth LaTeX annotation. A higher Jaccard Similarity value indicates greater overlap and thus higher similarity between the two texts.
• Mathematical Formula: Let $A$ and $B$ be two sets (e.g., sets of tokens from the predicted and ground truth texts). $ J(A, B) = \frac{|A \cap B|}{|A \cup B|} $
• Symbol Explanation:
  • J(A, B): The Jaccard Similarity between sets $A$ and $B$, typically ranging from 0 to 1.
  • $A$: The first set (e.g., tokens from the model's predicted output).
  • $B$: The second set (e.g., tokens from the ground truth LaTeX annotation).
  • $|A \cap B|$: The number of elements common to both sets $A$ and $B$ (the size of their intersection).
  • $|A \cup B|$: The total number of unique elements in both sets $A$ and $B$ (the size of their union). This can also be calculated as $|A| + |B| - |A \cap B|$.
  • The paper reports JS↑, indicating that a higher Jaccard Similarity is better.

5.3. Baselines

The paper compares AceParser against several existing methods, chosen to represent different approaches to academic literature parsing:

• Tesseract [4]: A traditional OCR engine. It represents methods focused solely on character recognition without explicit structural understanding.

• PPOCR [5]: An ultra-lightweight OCR system. Another OCR-based method, representing a more modern OCR approach, still primarily focused on character extraction.

• Pix2Text: A modular approach that is structure-aware (indicated by ✓ in Table II). It likely uses separate components to recognize different structured elements.

• Mineru [7]: Described as a "one-stop, open-source, high-quality data extraction tool," Mineru is a modular and structure-aware method (indicated by ✓). It represents sophisticated modular parsing systems.

• Nougat [9]: A neural optical understanding model for academic documents. This is an end-to-end method and is partially structure-aware (indicated by $V$). Nougat is often considered the previous state-of-the-art for academic document parsing and is a direct competitor in terms of paradigm.

  These baselines are representative because they cover the spectrum from basic OCR to more advanced modular and end-to-end deep learning approaches for document parsing, including methods that claim structure-awareness. Comparing against these diverse baselines allows the authors to thoroughly evaluate AceParser's performance and highlight its strengths in handling diverse structured texts.

5.4. Training Details

The training setup for the AceParser model is as follows:

• Initialization: The AceParser model is initialized with pre-trained weights from Florence-2 [14]. This leverages the extensive knowledge Florence-2 gained from its pre-training on 5 billion data instances across a variety of vision tasks.
• Optimizer: AdamW optimizer is used. AdamW is an optimization algorithm that is an extension of Adam and specifically designed to improve weight decay regularization, which can help prevent overfitting in deep learning models.
• Learning Rate: A learning rate of $1 \times 10^{-5}$ is set for the optimizer.
• Learning Rate Schedule: A linear learning rate schedule is employed, which includes a 10% warm-up phase.
  • Linear Learning Rate Schedule: The learning rate starts at a low value, linearly increases to the peak learning rate, and then linearly decays.
  • Warm-up Phase: During the initial 10% of training steps, the learning rate gradually increases from a very small value to the specified $1 \times 10^{-5}$. This warm-up helps stabilize the training at the beginning and often leads to better final performance.
• Hardware: Training is performed on four NVIDIA GeForce RTX 3090 GPUs. The NVIDIA GeForce RTX 3090 is a high-end consumer GPU suitable for intensive deep learning tasks, and using four of them enables distributed training to speed up the process.
• Batch Size: A batch size of 8 is used. This refers to the number of training examples processed together in one forward and backward pass during training.

6. Results & Analysis

6.1. Core Results Analysis

The experimental results demonstrate the superior performance of AceParser compared to existing academic literature parsing methods, particularly in handling diverse structured texts.

The following are the results from Table III of the original paper, comparing different parsing methods using various evaluation metrics:

The following are the results from Table III of the original paper:

Note on Table III: Metric abbreviations are used for clarity, such as LD for Levenshtein Distance. ↓ indicates that a lower value is better, while ↑ indicates that a higher value is better. Time is likely reported in seconds per sample. The "Improvements" row indicates the gain of AceParser over the previous best method for each metric.

Key observations and analysis from Table III:

• Performance of Non-Structure-Aware Methods:

  • Tesseract and PPOCR, which are OCR-based methods not designed for structured parsing, show the lowest performance across all metrics (e.g., F1 scores of 51.3 and 53.4, Jaccard Similarity of 37.2 and 39.4). Their Levenshtein Distance is high (0.52 and 0.53) and BLEU scores are low (19.3 and 18.4). This is expected, as they lack the ability to understand and output structural markup, confirming the paper's initial motivation about the limitations of OCR.

• Performance of Modular Approaches:

  • Pix2Text and Mineru, which are structure-aware modular approaches, perform better than OCR-only methods but are still less competitive than end-to-end models. Pix2Text has an F1 of 62.6 and JS of 47.2. Mineru achieves better results with F1 of 68.2 and JS of 53.4, demonstrating the value of structure-awareness. However, the paper attributes their sub-optimal performance to error accumulation across modules, which is a common drawback of modular systems. Mineru also has a significantly higher Time (984.9 seconds per sample), making it impractical for many applications.

• Performance of End-to-End Models:

  • Nougat, a previous end-to-end state-of-the-art model, achieves an F1 of 68.0 and JS of 53.4. This is comparable to Mineru in accuracy but with a much faster inference Time (11.24 seconds).

  • AceParser's Superiority: AceParser significantly outperforms all other methods. It achieves the highest F1 score of 72.3 and Jaccard Similarity of 58.4. It also has the lowest Levenshtein Distance of 0.34 and the highest BLEU score of 50.2.

  • Quantified Improvements: Compared to the previous best (Mineru or Nougat), AceParser demonstrates:

    • $+0.05$ improvement in Levenshtein Distance (from 0.39 to 0.34, lower is better).
    • $+4.6$ improvement in BLEU (from 45.6 to 50.2, higher is better).
    • $+4.1$ improvement in F1 score (from 68.2 to 72.3, higher is better).
    • $+5.0$ improvement in Jaccard Similarity (from 53.4 to 58.4, higher is better).

  • Parsing Speed: While AceParser is significantly more accurate, its parsing speed of 5.92 seconds per sample is faster than PPOCR and Nougat (and dramatically faster than Mineru), but slower than Tesseract and Pix2Text. The authors acknowledge this as a limitation and a focus for future optimization.

    These results strongly validate the effectiveness of AceParser and the AceParse dataset. The improvements across all structure-aware metrics (F1, Jaccard Similarity, BLEU) highlight AceParser's ability to accurately parse diverse structured texts into LaTeX representations. The comparison underscores the advantage of an end-to-end multimodal model trained on a comprehensive dataset like AceParse over OCR-only and modular approaches.

6.2. Case Study

The paper provides a case study to visually demonstrate AceParser's capabilities, specifically in parsing academic documents containing complex formulas.

The following figure (Figure 3 from the original paper) illustrates this case study:

该图像是图表，展示了某篇论文中的多幅热力图及公式示意。左侧包含一组矩阵形式的数学公式 $R_y(\theta)=e^{-i\frac{\theta}{2}\sigma_y}=\begin{pmatrix} \cos(\frac{\theta}{2}) & -\sin(\frac{\theta}{2}) \\ \sin(\frac{\theta}{2}) & \cos(\frac{\theta}{2}) \end{pmatrix}$，右侧为不同参数条件下的热力图对比，橙色虚线框标记了重点区域。

ct training. Structured text locations are highlighted with yellow ellipses.

The provided figure description is malformed, likely due to OCR errors. Based on the image content, it shows original image, feature map, and cross-modality attention matrices before and after training, focusing on a complex formula.

Analysis of the case study (Figure 3):

• Original Image and Feature Map (Left Side): The left side of Figure 3 shows the original image of an academic document containing a complex mathematical formula, alongside the feature map extracted by the image encoder.

  • Observation: It is noted that the image encoder (part of AceParser's Florence-2 base) not only captures areas with plain text but also focuses on special symbols and structures within the formulas. This indicates that the visual representation learned by the encoder is rich enough to differentiate and highlight complex mathematical notation, which is crucial for accurate parsing.

• Cross-Modality Attention Matrices (Right Side): The right side illustrates cross-modality attention matrices both before and after training on AceParse. These matrices are critical for understanding how the multimodal model aligns information between the visual input and the output LaTeX tokens. They show the relationships between input tokens (visual patches and task prompts) and corresponding parsed output tokens.

  • Before Training: The attention scores in the formula region are likely diffuse or less focused, indicating that the pre-trained Florence-2 model, while robust for general vision tasks, initially has difficulty parsing structured text into explicit LaTeX representations.

  • After Training: A substantial rise in attention scores within the formula region is observed after training on AceParse. This significant shift demonstrates that the model has learned to effectively align specific visual elements of the formula (e.g., symbols, superscripts, fractions) with their corresponding LaTeX markup during the fine-tuning process. This targeted attention is a direct result of being trained on the structure-rich annotations of AceParse, leading to a significant improvement in its ability to accurately parse such complex structures. The yellow ellipses highlight these regions of interest.

    This case study visually confirms the quantitative results, showing that AceParser effectively leverages its multimodal architecture and the AceParse dataset to achieve a deep understanding of document structure, particularly for intricate elements like mathematical formulas.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces AceParse, which is presented as the first comprehensive, open-source dataset specifically designed for parsing diverse structured texts within academic literature. It successfully addresses the critical gap in existing datasets by including a wide array of structured content, such as formulas, tables, lists, algorithms, and embedded mathematical expressions, all annotated in LaTeX markup. Leveraging this dataset, the authors developed AceParser, an end-to-end multimodal model fine-tuned from Florence-2, which achieves state-of-the-art performance in academic literature parsing. AceParser demonstrated significant improvements, with a 4.1% increase in F1 score and a 5% increase in Jaccard Similarity over previous methods. The work establishes a robust foundation for future research in academic literature parsing and the development of advanced end-to-end parsing models.

7.2. Limitations & Future Work

The authors acknowledge certain limitations and suggest directions for future work:

• Parsing Speed: A current limitation of AceParser is its relatively slow parsing speed (5.92 seconds per sample). This indicates that while accuracy is high, computational efficiency for large-scale processing still needs improvement.
• Dataset Quality Enhancement: Future work includes plans to further enhance the quality of the AceParse dataset. This could involve more granular annotations, broader diversity, or more rigorous cleaning.
• Increase Document Length: The current dataset documents might have certain length constraints. Increasing the document length in the dataset could improve the model's ability to handle longer, more complex real-world academic papers.
• Explore Smaller Models: To address the parsing speed limitation, the authors plan to explore the use of smaller, more efficient models. This could involve model compression techniques, knowledge distillation, or designing more lightweight architectures that maintain high accuracy while significantly reducing inference time.

7.3. Personal Insights & Critique

This paper makes a substantial contribution to the field of document AI and natural language processing, particularly for scientific literature.

• Innovation of Dataset Design: The data synthesis approach for AceParse is a particularly clever innovation. Relying on actual LaTeX source files and then synthesizing new combinations effectively bypasses the limitations of PDF-based extraction (like page break issues) and allows for generating a vast, high-quality, and structurally diverse dataset efficiently. This approach could be transferred to other domains where structured data can be programmatically generated.

• Leveraging Multimodal Models: The choice to fine-tune a powerful vision-language model like Florence-2 is strategic. These models have a strong foundation in understanding visual layouts and context, which is crucial for parsing complex academic documents. The demonstrated improvements validate the potential of multimodal learning in this domain.

• Practical Value: The AceParse dataset and AceParser have significant practical value. By enabling accurate parsing of diverse structured texts, they can unlock scientific knowledge currently locked in PDFs, facilitating downstream applications such as:

  • Knowledge Graph Construction: Automatically building knowledge graphs from scientific papers.
  • Information Retrieval: More accurate search and retrieval of specific information within academic databases.
  • Automated Summarization and Question Answering: Generating more sophisticated summaries or answering complex questions based on scientific content.
  • Scientific Discovery: Assisting researchers in identifying trends, gaps, and connections across vast bodies of literature.

• Critique on Speed: While AceParser achieves state-of-the-art accuracy, its parsing speed of 5.92 seconds per sample is a considerable bottleneck for processing very large corpora of documents (e.g., millions of papers). The mentioned future work on smaller models is critical. Further research might also explore batch processing optimizations or hardware-accelerated inference.

• Generalizability of LaTeX: While LaTeX is excellent for capturing structure, it's worth considering if the model's performance relies heavily on LaTeX-specific constructs. Real-world documents can have less formal or more varied structured representations (e.g., custom Word templates, scanned historical documents). The current dataset focuses on LaTeX-generated PDFs. Future work might investigate robustness to more varied PDF generation sources.

• Interpretability of Attention: The case study showing attention maps is insightful. Further analysis into what specific visual features the model attends to for different types of structured elements (e.g., distinguishing a table cell from a formula component) could provide deeper interpretability and reveal potential areas for model improvement or robustness.

  Overall, this paper presents a robust and impactful contribution, especially by providing a much-needed, comprehensive, and open-source dataset, which will undoubtedly accelerate research in academic literature parsing.