1. Bibliographic Information

• Title: SAM 2: Segment Anything in Images and Videos
• Authors: Nikhila Ravi, Valentin Gabeur, YuanT Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer.
• Affiliations: The authors are from Meta FAIR (Fundamental AI Research). Several are noted as core contributors and project leads, indicating a significant, large-scale research effort.
• Journal/Conference: The paper is available on arXiv, a preprint server for academic papers. This means it has not yet undergone formal peer review for a conference or journal but is shared to disseminate findings quickly.
• Publication Year: 2024
• Abstract: The paper introduces the Segment Anything Model 2 (SAM 2), a foundation model for promptable segmentation in both images and videos. To train this model, the authors developed a "data engine" that uses model-in-the-loop annotation to create the largest video segmentation dataset to date, named SA-V. The SAM 2 model itself is a transformer-based architecture that uses a streaming memory mechanism to handle videos in real-time. The results show that SAM 2 achieves higher accuracy with 3x fewer user interactions in video segmentation compared to previous methods. For image segmentation, it is more accurate and 6x faster than the original Segment Anything Model (SAM). The authors are releasing the model, dataset, and code to encourage further research.
• Original Source Link:
  • ArXiv: https://arxiv.org/abs/2408.00714v2
  • PDF: https://arxiv.org/pdf/2408.00714v2.pdf

2. Executive Summary

• Background & Motivation (Why): The original Segment Anything Model (SAM) revolutionized image segmentation by creating a foundation model that could segment any object in a static image based on simple user prompts (like clicks or boxes). However, the real world is dynamic, and a vast amount of visual data is in video format. Video segmentation presents unique and significant challenges not found in images, such as object motion, deformation, occlusion (disappearing and reappearing), and the need for computationally efficient processing of long frame sequences. Existing video segmentation models and datasets were either specialized for certain object types or lacked the "segment anything" generality and interactive capabilities of SAM. The core motivation was to create a unified, universal segmentation system that works seamlessly for both images and videos, addressing the limitations of prior work.

• Main Contributions / Findings (What): The paper makes three primary contributions:

  1. A New Task - Promptable Visual Segmentation (PVS): They formally define a task that extends promptable image segmentation to the video domain. In PVS, a user can provide prompts (clicks, boxes, masks) on any frame of a video to define an object, and the model must generate its spatio-temporal mask (a "masklet") across the entire video. The system allows for iterative refinement with more prompts on other frames.

  2. A New Model - SAM 2: They propose a novel model architecture that generalizes SAM for video. SAM 2 uses a streaming approach, processing frames one by one. Its key innovation is a memory mechanism that stores information from previously seen frames and user interactions. This memory allows the model to track objects consistently and correct errors effectively. When applied to a single image (a one-frame video), it behaves like the original SAM.

  3. A New Dataset - SA-V (Segment Anything Video): To train SAM 2, the authors built a highly efficient data engine—an annotation pipeline with the model in the loop. This system enabled the collection of the largest video segmentation dataset to date, SA-V, containing 35.5 million masks across over 50,000 videos. Unlike previous datasets, SA-V is not restricted to specific object categories and includes both whole objects and parts.

     The main finding is that SAM 2 significantly advances the state-of-the-art. It achieves better accuracy in video segmentation while requiring 3x fewer user interactions. For image segmentation, it surpasses the original SAM in accuracy while being 6x faster.

3. Prerequisite Knowledge & Related Work

To understand this paper, a beginner should be familiar with the following concepts.

• Foundational Concepts:

  • Image Segmentation: The task of partitioning a digital image into multiple segments or regions. The goal is to assign a label to every pixel in an image such that pixels with the same label share certain characteristics.
  • Video Object Segmentation (VOS): An extension of image segmentation to videos. The goal is to segment and track a specific object or objects across all frames of a video sequence. A common variant is semi-supervised VOS, where the ground-truth mask of the object is given only in the first frame.
  • Promptable Segmentation: A paradigm introduced by the original Segment Anything Model (SAM). Instead of segmenting pre-defined classes, the model segments an object specified by a user through "prompts" like points (clicks), bounding boxes, or text. This makes the model general and task-agnostic.
  • Foundation Model: A large-scale AI model trained on a vast quantity of broad data, designed to be adapted (or used zero-shot) for a wide range of downstream tasks. BERT in language and SAM in vision are prime examples.
  • Transformer Architecture: A neural network architecture that relies on a mechanism called self-attention. It excels at modeling long-range dependencies in sequential data. Originally developed for natural language processing, it has been successfully adapted for computer vision tasks, where it can relate different parts of an image or different frames in a video.

• Previous Works:

  • Segment Anything (SAM): The direct predecessor. SAM demonstrated that a promptable model trained on a massive and diverse image dataset (SA-1B) could achieve remarkable zero-shot generalization for segmenting objects in images. Its primary limitation was that it only operated on single, static images.
  • Interactive Video Object Segmentation (IVOS): Early methods used techniques like graph-based optimization. More recent approaches combine SAM with a separate video tracker (e.g., $SAM+XMem++$). The paper argues these hybrid methods are suboptimal because the tracker might fail, SAM might perform poorly on lower-quality video frames, and there is no integrated way to interactively correct tracking errors without restarting the process.
  • Video Object Segmentation (VOS) Models: Many models like XMem, Cutie, and STCN were designed for the semi-supervised VOS task. They are typically conditioned on the first frame's mask and propagate it forward. The authors note that creating this initial high-quality mask is time-consuming, and these models are not designed for interactive refinement on other frames.
  • Video Segmentation Datasets: Existing datasets like DAVIS, YouTube-VOS, and MOSE were crucial for advancing VOS research. However, the paper points out their limitations: they are orders of magnitude smaller than SA-V, often focus on a limited set of object categories (people, vehicles, animals), and do not extensively cover object parts.

• Differentiation: SAM 2 distinguishes itself from prior work in several key ways:

  • Unified Model vs. Hybrid Systems: Unlike SAM + tracker approaches, SAM 2 is a single, end-to-end model that directly handles both spatial prompting and temporal propagation. This is achieved through its integrated memory architecture.
  • Fully Interactive vs. First-Frame Prompt: Unlike traditional VOS models, SAM 2 allows for prompts on any frame, not just the first. This enables users to correct errors fluidly at any point in the video, making the process truly interactive.
  • "Anything" vs. "Something": By training on the massive and diverse SA-V dataset, SAM 2 inherits the "segment anything" capability of SAM and extends it to video, covering arbitrary objects and parts, not just predefined categories.

4. Methodology (Core Technology & Implementation)

The paper's methodology has two main pillars: the SAM 2 model architecture and the data engine used to create the SA-V dataset.

该图像是示意图，展示了SAM 2模型处理视频分割的整体流程，包括利用多帧中的提示（如框、点）输入，通过图像编码器、记忆注意模块和掩码解码器实现对视频中目标的分割和记忆更新，结合SA-V数据集进行训练。

Figure 1 provides a high-level overview of the work. The task (a) is promptable visual segmentation. The model (b), SAM 2, uses a streaming memory to process videos interactively. The data (c) is collected via a data engine.

The SAM 2 Model

SAM 2 is designed as a generalization of SAM for the video domain. It processes videos in a streaming fashion, one frame at a time, while maintaining a memory of past frames and interactions.

该图像是多个视频分割数据集上，SAM 2与其他方法在3次点击下的J&F指标随注释时间变化的折线图，展示了SAM 2在各数据集上显著优于其他两种方法的性能。

Figure 3 shows the detailed architecture of SAM 2.

• Principles: The core idea is to condition the segmentation of the current frame on a memory of past information. This allows the model to maintain object identity, handle occlusions, and incorporate user feedback over time.

• Steps & Procedures (Pipeline):

  1. Image Encoding: For each frame in the video, a Hiera image encoder (a hierarchical vision transformer) extracts feature embeddings. This is done only once per frame.
  2. Memory Attention: The features for the current frame are passed to the memory attention module. This module uses cross-attention to look at a memory bank containing information from previous frames. This conditions the current frame's features with relevant temporal context.
  3. Prompting: A user provides a prompt (e.g., a click) on a specific frame. The prompt encoder converts this sparse input into an embedding, similar to the original SAM.
  4. Mask Decoding: The mask decoder takes the memory-conditioned frame features and the prompt embeddings as input. It then predicts a segmentation mask for the object of interest on the current frame. It can predict multiple masks to handle ambiguity (e.g., a single click could refer to a whole person or just their shirt). It also has a head to predict if the object is even present in the frame (to handle occlusions).
  5. Memory Update: The predicted mask and frame features are processed by a memory encoder to create a new "memory" representation for this frame. This memory is then added to the memory bank.

• Key Architectural Details:

  • Image Encoder: A MAE-pretrained Hiera model is used. Its hierarchical nature provides multi-scale features, which is beneficial for segmenting objects of different sizes.
  • Memory Bank: This is a crucial component for temporal consistency. It consists of two First-In-First-Out (FIFO) queues:
    • A queue of memories from up to $N$ recent frames. These memories are encoded with temporal position information to help the model understand short-term motion.
    • A queue of memories from up to $M$ prompted frames. These are stored to retain user guidance over time. They are not encoded with temporal position to allow generalization, as prompts can come from any point in the video.
    • It also stores lightweight object pointers derived from the decoder's output tokens, which capture high-level semantic information about the target object.
  • Mask Decoder: It largely follows the SAM design with "two-way" transformer blocks that update both frame and prompt embeddings. A key novelty is the addition of skip connections from the hierarchical image encoder, which bypass the memory attention module. This allows high-resolution features to be directly used for decoding, improving mask boundary quality.
  • Training: The model is trained by simulating interactive sessions on 8-frame video clips. The training process randomly samples prompts (clicks, boxes, masks) and adds corrective clicks based on the model's errors compared to the ground-truth mask, teaching the model to refine its predictions.

The Data Engine and SA-V Dataset

A model like SAM 2 requires a massive, diverse dataset. The authors built a "data engine" to create SA-V.

• Data Engine Phases:

  1. Phase 1 (SAM per-frame): The initial phase used the original SAM to annotate each frame of a video individually. This was extremely slow (37.8s/frame) but produced high-quality annotations. This data was used to create the validation and test sets to ensure unbiased evaluation.
  2. Phase 2 (SAM + SAM 2 Mask): A basic version of SAM 2 (which only accepted mask prompts) was introduced. Annotators would create a mask on one frame using SAM, and SAM 2 Mask would propagate it through the video. Annotators could then correct errors on other frames by creating a new mask from scratch. This was ~5.1x faster (7.4s/frame).
  3. Phase 3 (SAM 2): The final, fully interactive SAM 2 model was used. Now, annotators could provide simple clicks to correct the model's propagated masks. Because SAM 2 has memory, a single click was often enough to fix an error. This was the most efficient phase, achieving an 8.4x speedup over Phase 1 (4.5s/frame).

• Automatic Masklet Generation: To increase data diversity and discover model weaknesses, the system automatically generated segmentation proposals by prompting SAM 2 with a grid of points on the first frame. These auto-generated "masklets" were then verified by human annotators. Correct ones were added to the dataset, and incorrect ones were sent back to annotators for refinement, creating a virtuous cycle of data collection and model improvement.

• The SA-V Dataset: The final dataset is massive and diverse.

  • Size: It contains 50.9K videos, 642.6K masklets (individual object tracks), and a total of 35.5 million masks. This is over 50x more masks than any previous public VOS dataset.

  • Content: The videos are "in-the-wild," covering diverse indoor and outdoor scenes. The annotations are class-agnostic ("anything") and include both whole objects and parts.

  • Challenge: The dataset is challenging, with a high "disappearance rate" (42.5%), meaning objects are frequently occluded and then reappear.

    The following manually transcribed table from the paper shows the scale of SA-V compared to existing datasets.

Table 3 (Transcribed): Comparison of our datasets with open source datasets in terms of number of videos, duration, number of masklets, masks, frames, and disappearance rate.

5. Experimental Setup

• Datasets:

  • Video Segmentation: A suite of 17 zero-shot datasets were used for evaluation, including 9 densely annotated ones for interactive experiments (DAVIS, YouTube-VOS, MOSE, etc.) and others for semi-supervised VOS. The custom-built SA-V val and SA-V test sets were used to measure "segment anything" capability on challenging, unseen videos.
  • Image Segmentation: 37 zero-shot datasets were used, including the 23 datasets from the original SAM paper (SA-23) and 14 new ones.

• Evaluation Metrics:

  1. J&F ($\mathcal{J}\&\mathcal{F}$):
     • Conceptual Definition: The primary metric for video object segmentation quality. It is the average of two scores: the Jaccard Index ($\mathcal{J}$), which measures the overlap between the predicted and ground-truth mask regions, and the Contour Accuracy ($\mathcal{F}$), which measures how well the boundary of the prediction matches the ground-truth boundary. A high $\mathcal{J}\&\mathcal{F}$ score indicates the prediction is accurate in both area and shape.
     • Mathematical Formula: $$\mathcal{J}\&\mathcal{F} = \frac{\mathcal{J} + \mathcal{F}}{2}$$
     • Symbol Explanation:
       • $\mathcal{J}$ is the Jaccard Index (or Intersection over Union, IoU): $\mathcal{J} = \frac{|M \cap G|}{|M \cup G|}$, where $M$ is the set of pixels in the predicted mask and $G$ is the set of pixels in the ground-truth mask.
       • $\mathcal{F}$ is the F-measure for contour accuracy, calculated from contour precision ($P_c$) and recall ($R_c$): $\mathcal{F} = \frac{2 \cdot P_c \cdot R_c}{P_c + R_c}$.
  2. mIoU (mean Intersection over Union):
     • Conceptual Definition: A standard metric for image segmentation. It computes the IoU (Jaccard Index) for each object or class and then averages these scores over all objects/classes in the dataset. It provides a single number to summarize overall segmentation quality.
     • Mathematical Formula: For a single class, IoU is as defined above. The mIoU over a dataset with $C$ classes is: $$\mathrm{mIoU} = \frac{1}{C} \sum_{i=1}^{C} \frac{\text{TP}_i}{\text{TP}_i + \text{FP}_i + \text{FN}_i}$$
     • Symbol Explanation: For class $i$, $\text{TP}_i$ is the number of true positive pixels, $\text{FP}_i$ is false positives, and $\text{FN}_i$ is false negatives.
  3. $\mathcal{G}$ (Overall Score for YouTube-VOS):
     • Conceptual Definition: The official metric for the YouTube-VOS challenge. It averages the $\mathcal{J}\&\mathcal{F}$ scores over object categories that were "seen" during training and those that were "unseen," providing a balanced measure of generalization.
     • Mathematical Formula: $$\mathcal{G} = \frac{1}{2} (\mathcal{G}_{\text{seen}} + \mathcal{G}_{\text{unseen}}) \quad \text{where} \quad \mathcal{G}_{\text{cat}} = \frac{1}{2} (\mathcal{J}_{\text{mean, cat}} + \mathcal{F}_{\text{mean, cat}})$$
     • Symbol Explanation: $\mathcal{J}_{\text{mean, cat}}$ and $\mathcal{F}_{\text{mean, cat}}$ are the mean Jaccard and Contour scores for a given category set (seen or unseen).

• Baselines:

  • SAM: The original Segment Anything Model, used as a baseline for image segmentation tasks.
  • $SAM+XMem++$ and $SAM+Cutie$: Strong baselines created by the authors to represent state-of-the-art hybrid approaches. They use SAM to generate a mask from a prompt, then feed this mask to a powerful VOS model ($XMem++$ or Cutie) for propagation.
  • State-of-the-Art VOS Models: A wide range of top-performing VOS models like Cutie, DEVA, XMem, STCN, etc., were used for comparison in the standard semi-supervised VOS setting.

6. Results & Analysis

The paper presents a comprehensive set of experiments demonstrating SAM 2's superior performance.

• Data Engine Effectiveness: The transcribed tables below show the impact of the data engine.

  Table 1 (Transcribed): Data engine analysis.

  	Model in the Loop	Time per Frame	Edited Frames	Clicks per Clicked Frame	Phase 1 Mask Alignment Score (IoU > 0.75)
  					All
  Phase 1	SAM only	37.8 s	100.00 %	4.80	-
  Phase 2	SAM + SAM 2 Mask	7.4 s	23.25 %	3.61	86.4 %
  Phase 3	SAM 2	4.5 s	19.04 %	2.68	89.1 %

  Analysis: Phase 3, using the full SAM 2 model, is 8.4x faster than the manual Phase 1, requires editing fewer frames, and needs fewer clicks for correction, while achieving comparable or even better quality.

  Table 2 (Transcribed): Segmentation accuracy ($\mathcal{J}\&\mathcal{F}$ metric) improvement from adding data from each data engine phase.

  Training data	SA-V val	9 zero-shot
  VOS + SA-1B	50.0	62.5
  + Phase 1	53.0	66.9
  + Phase 2	58.8	70.9
  + Phase 3+ Auto	63.2	71.5

  Analysis: This table clearly shows a "data flywheel" effect. As more data is collected with the increasingly powerful model (from Phase 1 to Phase 3), the resulting model trained on that data becomes progressively better, not just on the in-domain SA-V set but also on diverse zero-shot benchmarks.

• Promptable Video Segmentation:

  该图像是一个表格，展示了三种方法在多个视频分割数据集（如EndoVis 2018、ESD、LVOSv2等）上的性能对比，SAM 2在所有数据集上均表现最佳，平均得分为79.8，显著优于其他方法。

  Figure 5 Analysis: In both offline (multiple passes to find worst frames) and online (single forward pass) interactive settings, SAM 2 consistently outperforms the strong $SAM+XMem++$ and $SAM+Cutie$ baselines. It achieves higher accuracy ($\mathcal{J}\&\mathcal{F}$) with the same number of prompted frames, meaning a user gets a better result with less effort. For example, SAM 2 with 2-3 frame interactions reaches an accuracy level that baselines need 8 or more interactions to achieve, supporting the >3x fewer interactions claim.

• Semi-supervised Video Object Segmentation:

  Table 4 (Transcribed): Zero-shot VOS on 17 datasets, prompting on the first frame only.

  Method	1-click	3-click	5-click	bounding box	ground-truth mask‡
  SAM+XMem++	56.9	68.4	70.6	67.6	72.7
  SAM+Cutie	56.7	70.1	72.2	69.4	74.1
  SAM 2	64.7	75.3	77.6	74.4	79.3

  Analysis: Even in the classic VOS setting (where it is not its primary designed task), SAM 2 significantly outperforms the specialized baselines across all prompt types. The largest gains are seen with click prompts, but it also wins when given a perfect ground-truth mask, showing its core propagation mechanism is superior.

• Image Segmentation:

  Table 5 (Transcribed): Zero-shot accuracy on the Segment Anything (SA) task across 37 datasets.

  Model	Data	SA-23 All	SA-23 Image	SA-23 Video	14 new Video	FPS
  SAM	SA-1B	58.1 (81.3)	60.8 (82.1)	54.5 (80.3)	59.1 (83.4)	21.7
  SAM 2	SA-1B	58.9 (81.7)	60.8 (82.1)	56.4 (81.2)	56.6 (83.7)	130.1
  SAM 2	our mix	61.9 (83.5)	63.3 (83.8)	60.1 (83.2)	69.6 (85.8)	130.1

  Analysis: When trained on the same SA-1B data, SAM 2 is slightly more accurate than the original SAM but is ~6x faster (130.1 FPS vs 21.7 FPS), mainly due to a more efficient Hiera image encoder. When trained on the new data mix including SA-V, SAM 2's accuracy gets a significant boost, especially on video-derived image datasets.

• Comparison to State-of-the-Art (SOTA):

  Table 6 (Transcribed): VOS comparison to prior work (first-frame ground-truth mask prompt).

  Method	MOSE val	DAVIS 2017 val	LVOS val	SA-V val	SA-V test	YTVOS 2019 val
  STCN	52.5	85.4	-	61.0	62.5	82.7
  XMem	59.6	86.0	-	60.1	62.3	85.6
  Cutie-base+	71.7	88.1	-	61.3	62.8	87.5
  SAM 2 (Hiera-B+)	76.6	90.2	78.0	76.8	77.0	88.6
  SAM 2 (Hiera-L)	77.9	90.7	78.0	77.9	78.4	89.3

  Analysis: SAM 2 sets a new SOTA on all listed benchmarks, often by a large margin. The most dramatic gap is on the new SA-V benchmarks. Prior SOTA models cluster around ~62% $\mathcal{J}\&\mathcal{F}$, whereas SAM 2 scores ~77-78%. This demonstrates that previous models, while good on existing datasets, lack the "segment anything" capability that SAM 2 has acquired by training on the SA-V dataset. Using a larger backbone (Hiera-L) provides further accuracy gains.

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully presents a comprehensive solution for extending the "segment anything" paradigm from images to videos. The authors achieve this through three synergistic contributions: defining the Promptable Visual Segmentation (PVS) task, designing the memory-equipped SAM 2 model, and creating the massive SA-V dataset with a novel data engine. The results robustly demonstrate that SAM 2 is a significant step forward, offering a more accurate, efficient, and interactive experience for video segmentation, while also improving upon its predecessor in the image domain.

• Limitations & Future Work: The provided text cuts off before Appendix C, which is titled "Limitations." Therefore, the authors' own stated limitations are not available in the source document. However, based on the paper's scope, potential limitations could include:

  • Very Long Videos: The memory bank has a fixed size ($N$ recent frames, $M$ prompted frames). Its performance and computational cost on extremely long videos (e.g., hours of footage) are not explored.
  • Computational Resources: While faster than SAM, training and running such a large foundation model still requires significant computational power (e.g., A100 GPUs), limiting accessibility for some researchers.
  • Ambiguity Propagation: While the model predicts multiple masks to handle ambiguity on a single frame, how this ambiguity is managed or resolved over time (beyond picking the highest-scoring mask) could be a complex area for future work.

• Personal Insights & Critique:

  • The Data Engine is the Star: While the SAM 2 model is an elegant piece of engineering, the true game-changer is the data engine. The paper demonstrates a powerful, scalable methodology for creating massive, high-quality datasets. This "model-in-the-loop" annotation creates a virtuous cycle where better models lead to faster data collection, which in turn leads to better models. This concept is highly transferable to other domains in AI that are bottlenecked by data.
  • A Milestone for Video Understanding: SAM 2 marks a clear transition from specialized, task-specific video models to general-purpose, promptable foundation models for video. This is likely to have a similar impact on video-related applications (editing, AR/VR, robotics) as the original SAM had on image-based tasks.
  • Openness as a Catalyst: By releasing the model, the SA-V dataset (under a permissive license), and the code, the authors are providing an invaluable resource to the research community. This will undoubtedly accelerate progress in video segmentation and related fields.
  • Future Directions: An interesting open question is how to further integrate other modalities. Could text prompts be used more effectively to specify objects or their actions in video? Can the model's temporal understanding be extended from just segmentation to action recognition or video captioning? SAM 2 lays a strong foundation for exploring these multi-modal, spatio-temporal perception tasks.