1. Bibliographic Information

1.1. Title

The central topic of the paper is "Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists". This title indicates the paper introduces a new, large-scale, high-quality dataset designed for video editing tasks, specifically focusing on instruction-based methods, and highlighting its creation by "video specialists."

1.2. Authors

The authors of the paper are Bojia $\mathbf { Z _ { i } ^ { * } } ^ { 1 }$, Penghui Ruan* 2, Marco Chen 3, Xianbiao Qi† 4, Shaozhe Hao 5, Shihao Zhao 5, Youze Huang6, Bin Liang 1, Rong Xiao 4, and Kam-Fai Wong 1. The superscripts likely denote their affiliations, suggesting a collaborative effort across multiple institutions, though the specific institutions are not detailed in the provided text. The asterisks and dagger indicate equal contributions or corresponding authorship.

1.3. Journal/Conference

The paper is published as a preprint on arXiv, indicated by the "Original Source Link: https://arxiv.org/abs/2502.06734v3" and "Published at (UTC): 2025-02-10T17:58:22.000Z". arXiv is a widely respected open-access repository for preprints of scientific papers, particularly in physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. It allows researchers to share their work rapidly before or during peer review, making findings immediately accessible to the academic community.

1.4. Publication Year

The publication year, based on the provided UTC timestamp, is 2025.

1.5. Abstract

The paper addresses challenges in current video editing techniques, specifically inversion-based methods (slow inference, poor fine-grained editing, artifacts) and end-to-end methods (fast but suffer from a lack of high-quality training data). To fill this gap for end-to-end methods, the authors introduce Señorita-2M, a high-quality video editing dataset comprising approximately 2 million video editing pairs. This dataset is constructed using four specialized, state-of-the-art video editing models, developed and trained by the authors' team. A filtering pipeline is also proposed to ensure data quality by eliminating poorly edited video pairs. Furthermore, the paper explores various common video editing architectures to identify the most effective structures given current pre-trained generative models. Extensive experiments demonstrate that Señorita-2M significantly contributes to achieving remarkably high-quality video editing results. More details are available at their project website.

1.6. Original Source Link

The official source link for the paper is https://arxiv.org/abs/2502.06734v3. The PDF link is https://arxiv.org/pdf/2502.06734v3.pdf. This indicates the paper is an arXiv preprint.

2. Executive Summary

2.1. Background & Motivation

The field of video generation has witnessed rapid advancements, particularly with diffusion-based generative techniques like SORA, Kling, and Gen3 achieving impressive results. This surge in video generation capabilities naturally fuels the demand for sophisticated video editing techniques. However, current video editing methods face significant hurdles.

The paper identifies two main categories of video editing:

1. Inversion-based methods: These methods, while training-free and flexible, are computationally expensive, leading to time-consuming inference. They also struggle with fine-grained editing instructions and often produce undesirable artifacts and jitter (inconsistencies between frames), hindering their practical application.

2. End-to-end methods: These approaches promise faster inference speeds as they are trained directly on edited video pairs. However, their primary limitation is the lack of high-quality training video pairs. Without diverse and meticulously crafted data, end-to-end models often yield poor editing results.

   The core problem the paper aims to solve is this critical data shortage and quality issue for end-to-end video editing methods. This problem is important because end-to-end models hold the potential for efficient and high-quality video editing, which is essential for creative industries, content creation, and general user accessibility. The existing gap in high-quality instruction-based datasets prevents these methods from reaching their full potential, especially when compared to the more mature field of image editing.

The paper's innovative idea and entry point is to proactively address this data scarcity by constructing a massive, high-quality, instruction-based video editing dataset. They achieve this not through manual annotation or simple synthetic generation, but by leveraging a team of "video specialists" – specifically, by crafting and training several state-of-the-art (SOTA) specialized video editing models to act as "experts" for generating the dataset. This approach aims to create data that is both large-scale and of superior quality, enabling the development of truly effective end-to-end video editing solutions.

2.2. Main Contributions / Findings

The paper makes several significant contributions to the field of video editing:

1. Introduction of Señorita-2M Dataset: The primary contribution is Señorita-2M, the first truly large-scale instruction-based video editing dataset. It comprises approximately 2 million high-quality video editing pairs. Unlike existing datasets that focus on local edits (e.g., RACCooN, VIVID-10M) or are synthetically generated with insufficient quality (InsV2V), Señorita-2M offers both local and global editing tasks and is built from original video content sourced from the internet. This dataset directly addresses the data bottleneck for end-to-end video editing methods.

2. Development of Specialized Expert Models: To construct Señorita-2M, the authors crafted four high-quality, specialized video editing models (referred to as "experts"). Each expert is designed for a particular editing task and achieves state-of-the-art performance in its domain:

   • A global stylizer for applying overall style changes.
   • A local stylizer for editing specific regions.
   • A text-guided video inpainter for filling in masked areas.
   • A video remover for object removal. These experts, alongside other specialized models (like Grounded-SAM2 for segmentation), are instrumental in generating the diverse and high-quality data.

3. Robust Data Filtering Pipeline: A sophisticated filtering pipeline is introduced to ensure the high quality of the Señorita-2M dataset. This pipeline effectively removes failed video samples, poorly text-aligned videos, and subtle video pairs (those with minimal changes), which is crucial for training effective end-to-end models.

4. Exploration of Video Editing Architectures: The paper explores common video editing architectures and training strategies (e.g., InstructPix2Pix and ControlNet based models, with and without first-frame guidance, and dataset enhancement). This investigation helps identify the most effective structure for video editing models based on current pre-trained generative models.

5. Demonstrated Effectiveness: Extensive experiments demonstrate that Señorita-2M can train high-quality video editing models. The models trained on this dataset exhibit high visual quality, strong frame consistency, and excellent text alignment, significantly outperforming previous methods across various quantitative metrics (e.g., Ewarp, CLIPScore, Temporal Consistency) and user preference studies.

   In summary, the paper successfully addresses a critical challenge in video editing by providing a groundbreaking dataset and a robust methodology for its creation, paving the way for more advanced and practical end-to-end video editing solutions.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To fully understand this paper, a foundational grasp of several key concepts in artificial intelligence, particularly in computer vision and natural language processing, is essential.

3.1.1. Generative Models and Diffusion Models

At its core, this paper deals with generative models, which are types of AI models that can generate new data instances that resemble the training data. For example, generating new images or videos from text descriptions. Diffusion models are a class of generative models that have recently achieved state-of-the-art (SOTA) results in image and video synthesis. They work by progressively adding noise to data (forward diffusion process) until it becomes pure noise, and then learning to reverse this process (reverse diffusion process) to reconstruct data from noise. This iterative denoising process allows them to generate highly realistic and diverse outputs.

• Examples: Stable Diffusion, SORA, Kling.

3.1.2. UNet and Transformer Architectures

These are neural network architectures commonly used in diffusion models:

• UNet (U-shaped Network): Originally designed for image segmentation, the UNet architecture consists of an encoder path that captures context and a decoder path that enables precise localization. Its characteristic U-shape comes from skip connections that directly transfer feature maps from the encoder to the decoder, helping preserve fine-grained details during the generation process. Many early diffusion models (e.g., Stable Diffusion) use UNet as their backbone for noise prediction.
• DiT (Diffusion Transformers): DiT architectures replace the traditional convolutional layers of UNet with Transformer blocks. Transformers, initially designed for natural language processing, are excellent at modeling long-range dependencies using attention mechanisms. Applying Transformers to diffusion models allows them to handle higher resolutions and improve the quality and consistency of generated content, especially in text-to-image and text-to-video tasks. Pixart and Flux are examples of models using DiT.

3.1.3. Latent Space

In many generative models, especially diffusion models, data (like images or videos) is often transformed into a lower-dimensional representation called latent space (also known as latent vector or latent code). This transformation is typically done by an encoder (e.g., from a Variational Autoencoder (VAE)). Operating in latent space is computationally more efficient and can help the model learn more meaningful and abstract representations of the data, leading to better generation and editing capabilities. An decoder then transforms the latent representation back into the original data space.

3.1.4. Inversion-based vs. End-to-End Editing

These are two broad categories of editing approaches discussed in the paper:

• Inversion-based methods: These methods first invert an input image or video back into the latent space of a pre-trained generative model (e.g., a diffusion model). Once in the latent space, editing is performed by manipulating this latent representation or by guiding the denoising process with text prompts. The image/video is then regenerated from the modified latent code. They are training-free for new editing tasks but can be slow and prone to inconsistencies.
• End-to-end methods: These methods involve training a specific editing model directly on pairs of original and edited images/videos. The model learns to directly transform an input into an edited output based on an instruction. They are typically faster at inference but require large datasets of high-quality input-output pairs for training.

3.1.5. ControlNet

ControlNet is an neural network architecture that enables diffusion models (like Stable Diffusion) to be controlled with additional input conditions, such as edge maps (e.g., Canny), depth maps, skeletal poses, or segmentation masks. It works by taking a pre-trained diffusion model and adding a trainable copy (the control branch) that learns from these extra conditions, while keeping the original model's weights frozen. This allows for highly precise and diverse control over the generation process without compromising the quality of the pre-trained model.

3.1.6. Large Language Models (LLMs)

LLMs are advanced neural networks trained on vast amounts of text data, enabling them to understand, generate, and process human language. They are crucial for tasks like instruction generation, where they can translate high-level editing goals into specific, actionable prompts for generative models. LLAMA-3 is an example mentioned in the paper.

3.1.7. CLIP (Contrastive Language-Image Pre-training)

CLIP is a neural network trained on a massive dataset of image-text pairs to learn highly effective multimodal embeddings. This means it can represent both images and text in a common embedding space where similar images and texts are close together. CLIP is widely used for:

• Zero-shot image classification: Classifying images without being explicitly trained on those categories.
• Text-to-image relevance assessment: Measuring how well an image aligns with a given text prompt. This paper uses CLIP to filter text-misaligned video pairs and to measure text-video alignment during evaluation.

3.1.8. Classifier-Free Guidance (CFG)

Classifier-Free Guidance (CFG) is a technique used in diffusion models to improve the adherence of generated samples to a given condition (e.g., a text prompt). It works by performing two denoising steps at each iteration: one conditioned on the prompt and one unconditioned (or conditioned on an empty/null prompt). The unconditioned prediction is then subtracted from the conditioned prediction and scaled by a guidance scale (CFG weight) to push the generation further in the direction of the prompt. A higher CFG weight leads to stronger adherence but can sometimes reduce diversity or quality.

3.1.9. Object Detection and Segmentation (Grounded-SAM2)

Object detection is the task of identifying and localizing objects within an image or video, typically by drawing bounding boxes around them. Image segmentation goes a step further by identifying the precise pixel-level boundaries of objects. Grounded-SAM2 is a powerful model that combines GroundedDINO (for open-set object detection, i.e., detecting any object given a text prompt) with Segment Anything Model (SAM) (for high-quality mask generation from prompts or bounding boxes). Grounded-SAM2 allows for instruction-based object segmentation in images and videos, enabling precise mask generation for local editing tasks like inpainting or object removal.

3.2. Previous Works

The paper contextualizes its contributions by reviewing existing image and video editing techniques and datasets.

3.2.1. Image Editing

• Inversion-based Methods: These methods generally involve converting an image into a latent representation and then manipulating it with a prompt for regeneration. Examples include:
  • DDIM inversion: Focuses on inverting images to latent space, with research aiming to reduce discretization errors (Huberman-Spiegelglas et al., 2024; Lu et al., 2022; Wallace et al., 2023).
  • SDEdit (Meng et al., 2021): Introduces noise to images and denoises them based on a target text prompt.
  • Prompt-to-Prompt (Hertz et al., 2022): Modifies attention maps during diffusion steps to guide edits.
  • Null-Text Inversion (Mokady et al., 2023): Adjusts textual embeddings for classifier-free guidance.
  • Imagic (Kawar et al., 2023), Plug-and-Play Diffusion Features (Tumanyan et al., 2023), Zero-shot image-to-image translation (Parmar et al., 2023) are also mentioned as part of this category.
• End-to-end (Supervised) Methods: These are trained on image editing datasets to produce edits. Their effectiveness heavily relies on the quality and quantity of training pairs.
  • InstructPix2Pix (Brooks et al., 2023): Uses data generated by a diffusion model itself, filtered by CLIP-score, to enable instruction-following image editing.
  • MagicBrush (Zhang et al., 2024a): Improves data quality with human-annotated editing data using DALLE-2.
  • EmuEdit (Sheynin et al., 2024): Achieves high performance using smaller biases and higher-quality data, expanding its dataset to 10 million image pairs with free-form and local editing.
  • UltraEdit (Zhao et al., 2024a): Constructs a large-scale dataset (4 million samples) using an inpainting model and inversion methods, with LLM-generated instructions.
  • Omni-Edit (Wei et al., 2024): Improved UltraEdit by using multiple expert models and multimodal frameworks for higher-quality dataset generation.
  • HIVE (Zhang et al., 2024b) and Instructdiffusion (Geng et al., 2024) are also cited.

3.2.2. Video Editing

Video editing is recognized as more challenging than image editing, with frame consistency being a major hurdle.

• Inversion-based Methods: Most existing video editing techniques fall into this category. They often suffer from long editing durations and frame inconsistencies.
  • Tune-A-Video (Wu et al., 2023): Fine-tunes diffusion models on specific videos.
  • Pix2Video (Ceylan et al., 2023) and TokenFlow (Geyer et al., 2023): Focus on consistency through attention across frames or editing key frames.
  • AnyV2V (Ku et al., 2024): Generates edited videos by injecting features guided by the first frame.
  • New models like Gen3 and SORA perform style transfer by adding noise and regenerating.
  • StableV2V (Liu et al., 2024a) and Video-p2p (Liu et al., 2023b) are also referenced.
• End-to-end (Supervised) Methods: A smaller number of approaches exist due to data limitations.
  • InsV2V (Cheng et al., 2024): Trains an editing model using generated video pairs, but suffered from insufficient data quality.
  • EVE (Singer et al., 2025): Uses an SDS (Score Distillation Sampling) loss for distillation.
  • RACCooN (Yoon et al., 2024) and VIVID-10M (Hu et al., 2024): Use inpainting models and video annotations to produce local editing models, but primarily focus on local edits.
  • Propgen (Liu et al., 2024b): Used for local editing, propagates edits across frames using segmentation models.
  • Revideo (Mou et al., 2024): Utilizes motion and content to control output.

3.2.3. Image and Video Editing Datasets

• Image Editing Datasets: Often rely on synthetic data.
  • InstructPix2Pix (Brooks et al., 2023): Uses CLIP-score-based filtering.
  • MagicBrush (Zhang et al., 2024a): Human annotations from DALLE-2.
  • HQ-Edit (Hui et al., 2024): Uses DALLE3 for high-quality edited pairs.
  • EmuEdit (Sheynin et al., 2024): Expanded to 10 million image pairs, combining free-form and local editing.
  • UltraEdit (Zhao et al., 2024a): 4 million samples with LLM-generated instructions.
  • Omni-Edit (Wei et al., 2024): Diversified editing using multiple expert models.
• Video Editing Datasets: Few exist, and they have limitations.
  • RACCooN (Yoon et al., 2024) and VIVID-10M (Hu et al., 2024): Use inpainting models for video annotation, focusing on local edits.
  • InsV2V (Cheng et al., 2024): Builds dataset with generated original and target video pairs, but data quality was insufficient.

3.3. Technological Evolution

The evolution in generative AI has moved from basic Generative Adversarial Networks (GANs) to highly sophisticated diffusion models. Initially, these models excelled in image generation, with UNet and later Transformer-based (DiT) architectures pushing boundaries in fidelity and control. The success in image generation naturally extended to video, but video presents unique challenges, primarily temporal consistency (ensuring edits are smooth and coherent across frames) and the sheer computational cost of processing sequences of images.

Image editing has matured, offering both flexible inversion-based methods and high-quality end-to-end solutions, bolstered by large, diverse datasets. Video editing, however, lagged due to its inherent complexity. Most early video editing focused on inversion-based approaches, which, while flexible, suffered from slowness and inconsistencies. End-to-end video editing promises faster, more consistent results but was hampered by the scarcity of high-quality training data. Existing video datasets were either too small, focused only on local edits, or of poor quality.

This paper's work fits within this technological timeline by directly addressing the data bottleneck for end-to-end video editing. It leverages the advancements in LLMs for sophisticated instruction generation and harnesses the power of SOTA generative models and specialized computer vision experts (like Grounded-SAM2) to create a dataset that was previously unavailable. By doing so, it aims to accelerate the development of robust and general-purpose end-to-end video editing models, bringing video editing closer to the maturity seen in image editing.

3.4. Differentiation Analysis

Compared to the main methods in related work, Señorita-2M's approach has several core differences and innovations:

1. Scale and Scope of Dataset:

   • Differentiation: Señorita-2M provides 2 million instruction-based video editing pairs, making it the first truly large-scale dataset for general video editing. This is a significant leap from previous video editing datasets.
   • Contrast: InsV2V (0.06M pairs) was smaller and suffered from poor data quality. VIVID-10M (1.5M pairs) and RACCooN focused primarily on local editing (e.g., inpainting) and region-specific edits, not general instruction-based editing encompassing both local and global transformations. Most prior methods either work with small datasets or rely on synthetic data of lower quality.

2. Quality of Data Generation via Multiple Specialized Experts:

   • Differentiation: Instead of relying on a single, general model or simple synthetic processes, Señorita-2M is built by meticulously crafting and training four high-quality, specialized video editing models (global stylizer, local stylizer, inpainter, remover) by video specialists. Each expert achieves state-of-the-art performance in its specific task. This multi-expert approach ensures higher fidelity and diversity in the generated edited videos.
   • Contrast: Other datasets might use one general inpainting model (VIVID-10M, RACCooN) or rely on generated video pairs from a single diffusion model (InsV2V), which inherently limits quality and diversity. The paper highlights InsV2V's insufficient data quality as a key motivation for their work.

3. Instruction-based Generality:

   • Differentiation: Señorita-2M supports 18 distinct instruction-based video editing tasks, including both local and global edits (like object swap, local/global style transfer, object addition/removal, inpainting/outpainting, and conditional generation tasks). The use of Large Language Models (LLMs) to generate clear and effective instructions further enhances its utility for training instruction-following models.
   • Contrast: Datasets like VIVID-10M and RACCooN focus more on local, region-specific edits often derived from segmentation masks, rather than broad, natural language instructions applicable to various editing types.

4. Robust Filtering Pipeline:

   • Differentiation: The paper proposes a comprehensive three-stage filtering pipeline (quality filtering, text-alignment filtering using CLIP, and subtle change removal) to ensure that only high-quality, relevant, and sufficiently altered video pairs are included in the final dataset. This rigorous curation process is vital for the dataset's efficacy.

   • Contrast: While some image editing datasets (e.g., InstructPix2Pix, UltraEdit, Omni-Edit) utilize filtering, the specific challenges of video (e.g., temporal consistency needs to be inherently handled by the expert models) and the multi-stage, CLIP-based filtering adapted for video context represents a key improvement.

     In essence, Señorita-2M addresses the core limitations of previous video editing datasets by combining large-scale data generation with multiple specialized SOTA experts, LLM-driven instruction generation, and a rigorous filtering process, thereby creating a dataset that is significantly more comprehensive and higher quality than its predecessors for training end-to-end video editing models.

4. Methodology

This section details the meticulous process of constructing the Señorita-2M dataset, which involves the development of specialized video editing "experts" and a sophisticated data generation and filtering pipeline. The core idea is to generate a massive, high-quality dataset by leveraging highly capable models to perform various editing tasks, then curating the output rigorously.

4.1. Principles

The fundamental principles guiding the Señorita-2M methodology are:

1. Leveraging Expert Models: Instead of manual annotation, the dataset is synthesized by employing several state-of-the-art (SOTA), specialized video editing models. Each expert is tailored to a specific editing task (e.g., stylization, removal, inpainting) and trained to achieve high performance. This ensures high-quality raw edited video content.
2. Instruction-Based Data Generation: Large Language Models (LLMs) are used to generate natural language instructions for each editing pair. This ensures the dataset is suitable for training instruction-following video editing models.
3. Comprehensive Editing Tasks: The dataset covers a wide range of local and global editing tasks (18 distinct types), providing a diverse training resource for general video editing.
4. Rigorous Quality Control: A multi-stage filtering pipeline is implemented to discard low-quality, poorly text-aligned, or minimally edited video pairs. This is crucial for maintaining the dataset's high quality and effectiveness for model training.
5. Real-World Video Foundation: The dataset is built upon real-world videos sourced from the internet, ensuring ecological validity and diversity in content.

4.2. Core Methodology In-depth (Layer by Layer)

The construction of Señorita-2M is divided into two main phases: building the video experts and constructing the dataset itself, followed by a filtering pipeline.

4.2.1. The Construction of Video Experts

The paper trains four high-quality video editing experts based on CogVideoX (Yang et al., 2024), a powerful text-to-video diffusion model. These experts are a global stylizer, a local stylizer, a text-guided video inpainter, and a video remover.

4.2.1.1. The Training Data for Video Experts

The base training data for these experts is the Webvid-10M dataset (Bain et al., 2021), which contains a large collection of video-text pairs. To enrich this data for training the experts:

• CogVLM2 (Hong et al., 2024), a vision-language model, is used to generate detailed captions for each video (around 50 words) and to recognize objects within them.

• The recognized objects are then segmented and tracked across video frames using GroundedSAM2 (Liu et al., 2023a; Ravi et al., 2024). GroundedSAM2 combines GroundedDINO (for open-set object detection) with SAM (for high-quality mask generation), allowing for precise, instruction-based object segmentation.

  The overall data preparation for expert training involves these CogVLM2 and Grounded-SAM2 steps to get annotated information like masks and phrases.

  该图像是论文中的示意图，展示了Senorita-2M数据集的构建流程，包括通过模型识别视频中的对象生成掩码，利用多种检测方法形成控制条件，以及通过CogVlm2生成视频描述，最终整合形成带注释的训练数据集。

The figure above illustrates the process of generating masks and captions for the dataset. The initial video is processed by CogVLM2 for captioning and object recognition, and Grounded-SAM2 for precise object segmentation and tracking, which are then used to create control conditions (like depth maps) and video captions.

4.2.1.2. The Design and Training for Video Experts

Global Stylizer

The global stylizer is designed to apply a consistent style to an entire video, addressing the challenge that current video generation models often struggle to understand and apply style prompts effectively.

• Approach:

  1. The initial frame of a video is first edited using an image ControlNet (specifically, ControlNet-SD1.5 (Zhang et al., 2023a)) with a style prompt. This step ensures the style is correctly applied to at least one frame.
  2. A video ControlNet is then guided by this styled first frame to complete the stylization for the remaining frames of the video.
  3. To ensure robust style transfer and temporal consistency, the video ControlNet utilizes multiple control conditions. These conditions capture structural and textural information from the input video frames:
     • Canny edges: Outlines strong edges.
     • HED (Holistically-Nested Edge Detection): Detects both fine and coarse edges.
     • Depth maps: Captures geometric depth information.
  4. Each of these control conditions is transformed into latent space representations via a 3D-VAE (Variational Autoencoder), which helps the model work in a more compressed and semantically meaningful space.

• Training Details (Appendix B.2):

  • The global stylizer is built upon the CogVideoX model, leveraging its video generation capabilities.
  • Architecture: It integrates ControlNet by adding a control branch that takes the HED, Canny, and Depth latent representations as input (concatenated along the channel dimension, resulting in 48 channels). The control branch has N layers which generate control hidden states. These control hidden states are then added to corresponding K-th layers of the main branch (the CogVideoX backbone) at specific points.
  • Training Stages:
    • Phase 1: Trained with a resolution of $256 \times 448 \times 33$ (height x width x frames). A 10% null prompt is incorporated during training to enable classifier-free guidance (CFG), allowing for flexible control over adherence to the prompt during inference.
    • Phase 2: The model from Phase 1 is finetuned with an increased spatial resolution of $448 \times 896$.
  • Inference: The edited first frame (from image ControlNet) and the prompt are fed into the model along with the control conditions (Canny, HED, Depth) at a resolution of $336 \times 592$, producing 33 frames.

Local Stylizer

The local stylizer is designed to manipulate styles in specific regions of videos while preserving the original background, drawing inspiration from inpainting methods like AVID (Zhang et al., 2023b) and CoCoCo (Zi et al., 2024).

• Approach:

  1. The model uses the same three control conditions (Canny, HED, and Depth maps) as the global stylizer, which are fed into its ControlNet branch.
  2. Additionally, mask conditions (indicating the region to be styled) are fed into the main branch of the model. This allows the model to localize the editing effect.
  3. The pretrained model used as the backbone is CogVideoX-2B.

• Training Details (Appendix B.3):

  • Optimization: Trained using the AdamW optimizer (Loshchilov & Hutter, 2017).
  • Hyperparameters: Batch size of 16, learning rate of 1e-5, and weight decay of 1e-4.
  • Data: Videos consist of 33 frames at a resolution of $336 \times 592$.
  • Efficiency: To preserve generalization ability and accelerate training, FFN (Feed-Forward Network) layers (except for the first DiT block) are frozen.
  • Prompting: A sentence prefix "It's" is prepended to the detected object phrase, e.g., "It's a yellow house."
  • Inference: Takes 3 minutes on an Nvidia RTX 4090 for a $336 \times 592 \times 33$ video.

Text-guided Video Inpainter

This expert focuses on filling missing or masked regions in a video, guided by text, while addressing artifacts from older models and the lack of open-source alternatives.

• Approach:

  1. The inpainting model is based on CogVideoX-5B-I2V.
  2. It is guided by a first frame that has already been edited using Flux-Fill (Black Forest Labs, 2024), a well-performed image editor. This helps ensure high-quality initial conditions for inpainting.
  3. To prevent overfitting to specific mask shapes, the inpainter is trained with four types of masks, including random and precise shapes.
  4. The mask embedding is fed into the ControlNet encoder, while the mask itself is concatenated with the input frames in the main branch. The mask also influences text embeddings and time embeddings.

• Training Details (Appendix B.4):

  • Mask Generation: Random masks are generated using Grounded-SAM2 for objects, and then expanded from these initial masks.
  • Prompting: Each mask is paired with a text prompt, which consists of positive and negative instructions. The positive prompt is fed into the text embedders, while the negative prompt is fed into other channels that are zero-initialized.
  • Optimization: Trained using AdamW.
  • Hyperparameters: Batch size of 16, learning rate of 1e-5.
  • Data: Resolution $336 \times 592$, 33 frames, stride of 2.
  • Efficiency: FFN layers (except for the first DiT block) are frozen.
  • Inference: Can be finished within 2 minutes on an Nvidia RTX 4090 for 33 frames at $336 \times 592$.

Video Remover

The video remover aims to accurately remove objects from videos without introducing blur or other artifacts, a common issue in existing inpainting models like Propinater (Zhou et al., 2023).

• Approach:

  1. The model is based on CogVideoX2B.
  2. It employs a novel mask selection strategy during training to break the correlation between generated content and mask shape, ensuring the model learns to fill the masked region naturally, rather than simply blurring or using surrounding pixels.
  3. Instruction Types for Training:
     • 90% of masks are randomly sampled from unrelated videos with positive instructions like "Remove {object name}".
     • 10% of masks precisely cover objects with negative instructions like "Generate {object name}".
  4. During inference, classifier-free guidance (CFG) is used with both types of instructions. This guides the generation away from the negative condition (i.e., not generating the object within the mask) while still following the positive instruction (i.e., removing the object), resulting in content generation unrelated to the original mask shape.

• Training Details (Appendix B.5):

  • Optimization: Trained using AdamW.

  • Hyperparameters: Batch size of 16, learning rate of 1e-5, weight decay of 1e-4.

  • Mask Sampling: 90% task-irrelevant masks (randomly sampled) and 10% task-relevant masks (precisely covering objects).

  • Data: Videos sampled at 33 frames, stride of 2, resolution $336 \times 592$.

  • Efficiency: CogVideoX-mode is initialized with pretrained parameters. FFN layers (except for the first DiT block) are frozen.

  • Prompts: The positive prompt is "Remove {object name}", and the negative prompt is "Generate {object name}".

  • Inference: Can be finished within 1 minute on an Nvidia RTX 4090 for 33 frames at $336 \times 592$.

    该图像是图13，展示了去除器及子数据集构建流水线的框架。上部对比了传统填充器与本方法的关系断裂，中部为训练阶段的流程，底部为数据集构建阶段，涵盖视频输入、掩码生成及基于指令的视频去除与数据集生成过程。

The figure above illustrates the framework of the remover and its sub-dataset construction pipeline. The top part conceptually compares a traditional inpainter (which might struggle with blur) with the proposed method that aims to break the correlation between generated content and mask shape. The middle section shows the training stage: video with a mask and positive/negative prompts are fed into the CogVideoX backbone, with FFN layers frozen and ControlNet layers active. The bottom section depicts the dataset construction: input video, mask generation (Grounded-SAM2), prompt creation, and then using the trained remover to generate the edited video for the dataset.

4.2.2. The Construction of Señorita-2M

This section describes how the raw videos are collected, processed by the experts, and formatted into the final dataset.

该图像是一个示意图，展示了Señorita-2M数据集中视频编辑及筛选流程。包括多个视频编辑模型（Global Stylizer、Local Stylizer、Remover、Inpainter、SAM2、Depth Detector）对输入视频的处理，以及通过视觉质量过滤、文本对齐过滤和视觉相似性过滤的多阶段筛选，最终得到高质量的干净视频对。

The figure above provides an overview of the Señorita-2M dataset construction pipeline. It starts with an input video (sourced from Pexels). This video undergoes processing for object detection and tracking using Grounded-SAM2, and captioning using CogVLM2. Concurrently, control conditions (Canny, HED, Depth) are extracted. These processed inputs are then fed into the specialized video editing experts (Global Stylizer, Local Stylizer, Remover, Inpainter). The outputs from these experts, along with LLM-generated instructions, form edited video pairs. These pairs then pass through a filtering pipeline comprising visual quality filtering, text alignment filtering, and visual similarity filtering to ensure only high-quality data contributes to the final Señorita-2M dataset.

4.2.2.1. The Data Source in Señorita-2M

• Video Collection: Videos are crawled from Pexels.com, a video-sharing website known for high-resolution and high-quality content, using authenticated APIs. This ensures legal sourcing and diverse real-world footage. The initial collection consists of approximately 390,000 videos.
• Video Pre-processing: Videos longer than 500 frames are truncated to 500 frames to manage data size.
• Annotation:
  • BLIP-2 (Li et al., 2023) is used to generate brief captions for the videos, adhering to CLIP's length restrictions.
  • CogVLM2 (Hong et al., 2024) and Grounded-SAM2 (Liu et al., 2023a; Ravi et al., 2024) are employed to obtain mask regions for objects and their corresponding phrases. This process generates around 800,000 mask sequences.

4.2.2.2. Local Edit

This category includes 6 tasks that involve modifying specific regions or objects within a video. The input resolution for most local editing tasks is $336 \times 592$ with 33 frames, unless otherwise specified.

• Object Swap:

  1. LLaMA-3 (Dubey et al., 2024) (an LLM) is prompted to suggest a replacement object for an existing object in the video.
  2. FLUX-Fill (Black Forest Labs, 2024), a powerful image editor, performs the object swap on the first frame of the video.
  3. The trained inpainter expert then generates the remaining frames, guided by the edited first frame.
  4. Finally, an LLM generates instructions that refer to both the original and the new objects (e.g., "Swap the tiger for cat").

• Local Style Transfer:

  1. An LLM is used to construct a prompt by adding descriptive adjectives to the object name (e.g., "make the flower pink").
  2. This prompt is fed into the trained local stylizer expert to modify the masked region of the object.
  3. The LLM converts this into the final instruction.

• Object Removal:

  1. The trained remover expert is used to remove objects.
  2. Positive and negative instructions are generated, typically by adding "Remove" or "Generate" before the object name.
  3. The remover uses classifier-free guidance with both instruction types during inference.
  4. An LLM then generates the final instruction for the dataset.

• Object Addition:

  1. This task is conceptually the reverse of object removal. The source and target videos are effectively swapped in the pipeline (i.e., a video with an object is generated from a video without it).
  2. An LLM assists in rewriting and enhancing the instructions (e.g., "Add a flower").

• Video Inpainting and Outpainting:

  1. For inpainting, a region is removed (set to zeros) from the first frame. The position of this masked region is shifted over time to create dynamic masks for the video.
  2. Instructions are generated by prepending "inpaint" before the video's original caption.
  3. Outpainting is similar but typically involves expanding the video frame into a black background region, with instructions prefixed by "outpaint".
  4. The inpainting and outpainting processes have a higher resolution of $1280 \times 1984$ with 64 frames.

4.2.2.3. Global Edit

This category involves three components that apply changes across the entire video or deal with structural properties. The default resolution for object grounding and conditional generation tasks is $1120 \times 1984$ with 64 frames.

• Style Transfer:

  1. Style prompts (from Midjourney or custom lists, like those in Appendix E) are combined with BLIP-2 captions to create style-rich prompts.
  2. These prompts are input into ControlNet-SD1.5-HED to perform style transfer on the first video frame.
  3. The edited first frame is then integrated with control conditions (canny, depth, hed) and processed through the trained global stylizer expert to generate the rest of the frames.
  4. An LLM converts the style prompts into actionable instructions for further optimization.

• Object Grounding:

  1. This task generates video pairs where object localization is key, helping a video editor learn to accurately identify relevant regions based on instructions.
  2. Areas unrelated to the prompt are marked in black, while prompt-related instances are highlighted in distinct colors.
  3. Initial instructions are formed by prepending words like "Detect" or "Ground" before the object name.
  4. An LLM refines these instructions for clarity.

• Conditional Generation: This component comprises 10 tasks aimed at supporting video-to-video translation based on structural or aesthetic controls:

  • Deblur: Sharpening blurry video.
  • Canny-to-Video: Generating video from Canny edge maps.
  • Depth-to-Video: Generating video from depth maps.
  • Depth Detection: Generating depth maps from video.
  • Hed-to-Video: Generating video from HED edge maps.
  • Hed Detection: Generating HED edge maps from video.
  • Upscaling: Increasing video resolution.
  • FakeScribble-to-Video: Generating video from stylized scribbles.
  • FakeScribble Detection: Generating stylized scribbles from video.
  • Colorization: Adding color to grayscale video. These tasks generate controllable input and target videos.

4.2.2.4. DATA SELECTION (Filtering Pipeline)

To ensure the Señorita-2M dataset consists only of high-quality and meaningful video editing pairs, a multi-stage filtering pipeline is applied.

该图像是论文中用于构建高质量视频编辑数据集Señorita-2M的数据处理流程示意图，展示了从输入视频、特征提取、文本提示生成到视频编辑及数据集生成的全过程。

The figure above depicts the data selection process. Raw video clips are fed into various video editing experts. The edited videos are then paired with the original videos and their corresponding LLM-generated instructions. These pairs are subjected to a multi-stage filtering process to ensure quality control, resulting in a refined dataset of high-quality video editing pairs.

• Quality Filtering:

  • Objective: To identify and remove failed edits (e.g., distorted, incomplete, or corrupted outputs from experts).
  • Method: Classifiers are trained for this purpose.
    1. A manual annotation step creates a small dataset of successful vs. failed samples.
    2. Features are extracted from 17 frames per video using a frozen CLIP vision encoder (Radford et al., 2021). CLIP's vision encoder is a powerful feature extractor that captures semantic information from images.
    3. These features are then fed into MLP (Multi-Layer Perceptron) classifiers for classification. An MLP is a simple feed-forward neural network with multiple layers of neurons.
    4. To enhance robustness, an ensemble of MLP classifiers trained with various strategies is used.
    5. Different thresholds are applied for different editing tasks. For general quality filtering, a threshold of 0.6 is used (meaning samples with a classifier score below 0.6 are filtered out).

• Removing Poor Text-alignment Videos:

  • Objective: To ensure the edited video accurately reflects the text prompt.
  • Method: CLIP is used to measure the similarity between the edited video and its corresponding text prompt.
    1. For object swapping and local stylization, the inpainting prompt is compared with the edited video.
    2. For stylization (global), the style prompt is compared with the edited video.
    3. Object removal tasks are excluded from this filtering step as they lack a suitable target prompt for comparison.
    4. Different thresholds are used: 0.22 for object removal and local stylization, and 0.2 for global stylization and object addition. Videos with CLIP similarity below these thresholds are removed.

• Removing Subtle Video Pairs:

  • Objective: To eliminate video pairs where the edit is too minor or the edited video is nearly identical to the original, which could lead to overfitting or inefficient learning during training.

  • Method: CLIP's vision encoder is used to extract features from both the original and edited videos. The similarity between these feature vectors is then computed. Video pairs with a similarity score exceeding 0.95 (indicating very little change) are removed.

    LLM Prompts for Instruction Construction (Appendix C.4): The paper provides specific prompts used for LLaMA-3 to generate clear and concise instructions for different editing tasks. These demonstrate how generic prompts are transformed into specific editing commands.

• Global Stylization Prompt Example: Help me enhance the instruction <input>. Don't give useless information, such as "There be". For example, <input> is "Fox, a frame after frame style, vivid movie style, the art created by Fox", the answer is "make it a chic frame after frame style". Don't give me descriptions. Please give me answer directly. Now, the <input> is "{style_prompt}", the answer is: This prompt guides the LLM to extract the core stylistic instruction from a descriptive style prompt.

• Local Stylization Prompt Example: Help me enhance the instruction <input>. Don't give useless information, such as "There be". For example, <input> is "bird -> yellow bird", the answer is "make the bird yellow.". <input> is "chick -> green chick", the answer is "make the chick green.". <input> is "fox -> brown and furry fox", the answer is "make the fox brown and furry". The <input> is "pigeons -> gray pigeons", the answer is "make pigeons gray.". Don't give me descriptions. Please give me answer directly. Now, the <input> is "{object_name}" -> "{text_prompt}", the answer is: This prompt teaches the LLM to convert an object-to-edited-object description into a concise local stylization command.

• Object Removal Prompt Example: Help me enhance the <input>. Don't give useless information, such as "There be". For example, <input> is "Remove dog", the answer is "Delete the dog. <input> is "Remove dog", the answer is "Remove the dog from the video.". <input> is "Remove dog", the answer is "Discard the dog from videos.". <input> is "Remove dog", the answer is "Eliminate the dog." Don't give me descriptions. Please give me answer directly. Now, the <input> is "Remove a {object_name}", the answer is: This guides the LLM to generate various ways to phrase an object removal instruction.

• Object Addition Prompt Example: Help me enhance the <input>. Don't give useless information, such as "There be" For example, <input> is "Add dog", the answer is "Insert a dog.". <input> is "Add dog", the answer is "Place a dog.". <input> is "Add dog", the answer is "Add a dog to this video. I will give you negative instruction." <input> is "Add dog", then "Install a helicopter pad.". (That's wrong) Don't give me descriptions. Please give me answer directly. Now, the <input> is "Add a {object_name}", the answer is: This prompt helps the LLM generate object addition instructions, including learning from negative examples.

• Object Swap Prompt Example: Help me rewrite the <input>. Don't change meaning. Don't give useless information, such as "There be". For example, <input> is "replace cat with dog.", the answer is "turn cat into dog.". <input> is "replace cat with dog.", the answer is "change the cat to dog.". <input> is "replace cat with dog.", the answer is "Let there be a dog in the place of the cat.". Don't give me descriptions. Please give me answer directly. Now, the <input> is "Turn {target_name} into {object_name}", the answer is: This prompt enables the LLM to rephrase object swap instructions into concise commands.

  该图像是论文中的示意图，展示了视频编辑数据集Señorita-2M的构建流程，包括去背景处理、通过VAE编码、ControlNet-Inpainter模型编辑及基于LLM生成编辑指令的完整数据集构造过程。

The figure above illustrates the construction of the Local Stylizer sub-dataset. It shows an input video, a mask generated by GroundedSAM2, and the first frame of the video. The ControlNet-Inpainter is used along with a text prompt to edit the first frame and then propagate the local style to the entire video. The LLM then generates the editing instruction, forming an (original video, edited video, instruction) triplet.

该图像是论文中关于Senorita-2M数据集构建流程的示意图，详细展示了从输入视频、生成掩码、提示词构建，到视频编辑模型训练和数据集生成的全过程。

The figure above depicts the construction of the Text-guided Video Inpainter sub-dataset. An input video is provided, from which masks are extracted. The first frame is edited using Flux-Fill with a text prompt. This edited first frame, along with the masks and text prompt, guides the video inpainter (based on CogVideoX-5B-I2V) to generate the inpainted video. An LLM then creates the final instruction, forming a complete dataset triplet.

4.2.3. Training Details of Editing Model

The paper describes the training process for the video editing models that are evaluated using the Señorita-2M dataset.

• Base Model: CogVideoX-5B-I2V (Yang et al., 2024) is used as the base model.
• Architecture: It is integrated with ControlNet to leverage first-frame guidance for the editing process. This means the model can use a well-edited first frame to maintain temporal consistency and guide the edits across subsequent frames.
• Training Stages:
  • First Stage:
    • Batch size: 32
    • Learning rate: 1e-5
    • Weight decay: 1e-4
    • Epochs: 2
    • Resolution: $336 \times 592$
    • Frames: 33 frames are sampled from videos.
  • Second Stage (Finetuning):
    • Resolution: $448 \times 768$ (higher resolution)
    • Batch size: 16 (reduced due to higher resolution)
    • Epochs: 1 This second stage aims to help the model edit high-resolution videos effectively.

4.2.4. Inference of Experts

The inference process for the specialized experts to generate the dataset is detailed:

• Hardware: Nvidia 4090 GPUs were used.
• Local Stylizer: Classifier-Free Guidance (CFG) of 6, resolution $336 \times 592$, processing 33 frames.
• Global Stylizer: Initial resolution $256 \times 496$, then resized to $336 \times 592$, handling 33 frames.
• Inpainter: CFG of 6, resolution $336 \times 592$, processing 33 frames.
• Remover: CFG of 2, resolution $336 \times 592$.
• Auxiliary Techniques: Depth estimators, HED, Canny detectors, and other computer vision techniques were employed to generate video pairs for conditional generation tasks, all at a resolution of $1120 \times 1984$.

5. Experimental Setup

5.1. Datasets

Several datasets are utilized for training the expert models, constructing Señorita-2M, and evaluating the final video editing models.

• Webvid-10M (Bain et al., 2021): This is a large-scale video-text dataset used for training the base video editing experts. It provides a diverse set of real-world videos paired with text descriptions.

• Señorita-2M: This is the novel dataset introduced in the paper, comprising approximately 2 million high-quality instruction-based video editing pairs.

  • Source: Videos are crawled from Pexels.com using authenticated APIs, ensuring high resolution and quality. The dataset contains 388,909 real videos as its base.
  • Editing Types: It covers both local and global editing tasks, with 18 distinct types, including object swap, local/global style transfer, object addition/removal, inpainting/outpainting, object grounding, and various conditional generation tasks.
  • Frames: Videos in the dataset range from 33 to 64 frames.
  • Resolution: Resolutions vary from $336 \times 592$ to $1120 \times 1984$.
  • Construction: Built using 6 specialized expert models (four core trained experts plus CogVLM2 and Grounded-SAM2).
  • Open-Source: The dataset is planned to be open-sourced upon acceptance.
  • Example Data Sample (Conceptual, from Figure 3 of the paper abstract):
    • Instruction: "Remove the girl."
    • Original video: A video clip containing a girl.
    • Edited video: The same video clip with the girl removed.
    • Instruction: "Swap the tiger for cat."
    • Original video: A video clip with a tiger.
    • Edited video: The same video clip with the tiger replaced by a cat.
    • Instruction: "Make it watercolor style."
    • Original video: A regular video clip.
    • Edited video: The same video clip rendered in watercolor style.
  • Señorita-2M is chosen because it directly addresses the need for high-quality, diverse, and instruction-based video editing data, which was a critical gap in prior research. Its large scale and meticulous curation make it effective for validating the performance of end-to-end video editing models.

• DAVIS dataset (Pont-Tuset et al., 2017): This dataset is used for model evaluation.

  • Characteristics: DAVIS (Densely Annotated VIdeo Segmentation) is a popular dataset for video object segmentation. It provides high-quality pixel-accurate annotations for multiple objects in video sequences.
  • Usage: In this paper, DAVIS is used with randomly generated editing prompts to evaluate the stability and quality of the edited videos. This choice is logical because DAVIS provides complex, real-world video sequences with dynamic content, making it a good benchmark for assessing temporal consistency and editing quality.

• Omni-Edit dataset (Wei et al., 2024): This image editing dataset is used in an ablation study to enhance video editing models (specifically Ins-Edit* and Control-Edit*). It is a large and high-quality dataset built using multiple expert models, demonstrating the potential for cross-modal learning or transfer of data generation strategies.

• InsV2V dataset (Cheng et al., 2024): This synthetic video editing dataset is used as a baseline for comparison and in an ablation study. It consists of video pairs for training editors. The paper explicitly states that InsV2V suffered from insufficient data quality, making it a suitable contrasting dataset to highlight the superiority of Señorita-2M.

5.2. Evaluation Metrics

The paper uses a comprehensive set of quantitative metrics to evaluate the performance of video editing models, focusing on visual quality, text alignment, and temporal consistency.

5.2.1. Ewarp

• Conceptual Definition: Ewarp (Error Warp) quantifies the amount of unwanted distortion or warping present in the edited video, particularly related to the stability and consistency of object shapes and movements. A lower Ewarp value indicates less distortion and better preservation of the original geometry or smoother transitions.
• Mathematical Formula: The paper does not provide a formula for Ewarp. Ewarp is a metric often used in video processing to measure optical flow distortions or geometric inconsistencies. A common approach involves estimating optical flow between frames and then quantifying the divergence or curl of the flow field, or the deviation from a smooth, consistent transformation. Another interpretation can be based on comparing the transformed coordinates of keypoints/features in consecutive frames. Without the paper's specific definition, we'll use a general representation related to flow consistency. Let $F_t$ be the $t$-th frame of a video, and $V_{t \to t+1}$ be the optical flow field from frame $t$ to $t+1$. Ewarp often relates to the discrepancy in how points move according to the flow field. A simplified conceptual formula, without being specific to the paper's implementation, could be based on measuring the "warping energy" or "divergence" of the flow: $ \text{Ewarp} = \frac{1}{T-1} \sum_{t=1}^{T-1} ||V_{t \to t+1}(p) - \text{smooth_flow}(p)||2 $ Or, using a more standard warping error concept from related fields: $ \text{Ewarp} = \frac{1}{T-1} \sum{t=1}^{T-1} \sum_{p \in \text{image}} || \text{warp}(F_t, V_{t \to t+1})(p) - F_{t+1}(p) ||_2^2 $ where $\text{warp}(F_t, V_{t \to t+1})$ warps frame $F_t$ using flow $V_{t \to t+1}$. The paper's context of Ewarp (10-3) suggests it's a scaled value indicating small errors.
• Symbol Explanation:
  • $T$: Total number of frames in the video.
  • $t$: Current frame index.
  • $F_t$: The $t$-th frame of the video.
  • $V_{t \to t+1}$: Optical flow vector field from frame $t$ to frame $t+1$.
  • $p$: A pixel coordinate within the image.
  • $\text{smooth\_flow}(p)$: An idealized smooth optical flow field, often obtained by regularization or averaging.
  • $||\cdot||_2$: L2 norm (Euclidean distance).
  • $\text{warp}(F_t, V_{t \to t+1})$: A warping function that transforms pixels of frame $F_t$ according to the optical flow field $V_{t \to t+1}$.
  • The paper refers to it as Ewarp(10-3), implying the reported values are scaled by $10^{-3}$. Lower values are better.

5.2.2. CLIPScore

• Conceptual Definition: CLIPScore measures the semantic similarity or alignment between an image (or video frame) and a given text prompt. It leverages the CLIP model's ability to embed both modalities into a common space. A higher CLIPScore indicates that the visual content strongly matches the textual description. For video, this is typically averaged over frames or computed with a video-level aggregation.
• Mathematical Formula: The CLIPScore for a single image $I$ and text $T$ is typically calculated as the cosine similarity between their respective CLIP embeddings: $ \text{CLIPScore}(I, T) = \text{cosine_similarity}(\text{CLIP_Encoder}(I), \text{CLIP_Encoder}(T)) $ For a video $V$ with $N$ frames and a text prompt $T$, it can be computed by averaging the CLIPScore of individual frames with the text: $ \text{CLIPScore}(V, T) = \frac{1}{N} \sum_{i=1}^{N} \text{cosine_similarity}(\text{CLIP_Encoder}(F_i), \text{CLIP_Encoder}(T)) $ Alternatively, some implementations might use the highest similarity score or incorporate a video-specific CLIP model. The paper states CLIP is used to assess text-video alignment.
• Symbol Explanation:
  • $I$: An image.
  • $T$: A text prompt.
  • $V$: A video.
  • $F_i$: The $i$-th frame of the video $V$.
  • $\text{CLIP\_Encoder}(X)$: The CLIP model's encoder that converts an image or text $X$ into its embedding vector.
  • $\text{cosine\_similarity}(u, v)$: The cosine similarity between two vectors $u$ and $v$, defined as $\frac{u \cdot v}{||u|| \cdot ||v||}$. Higher values are better.

5.2.3. Temporal Consistency

• Conceptual Definition: Temporal Consistency assesses how smoothly and coherently visual elements, styles, or transformations are maintained across consecutive frames of a video. High temporal consistency means that edits do not suddenly jump or flicker, and objects maintain their identity and appearance over time. The paper states it computes similarity between adjacent frames using CLIP features.
• Mathematical Formula: Based on the paper's description, Temporal Consistency for a video $V$ with $N$ frames is calculated as the average cosine similarity between CLIP embeddings of adjacent frames: $ \text{Temporal Consistency}(V) = \frac{1}{N-1} \sum_{i=1}^{N-1} \text{cosine_similarity}(\text{CLIP_Encoder}(F_i), \text{CLIP_Encoder}(F_{i+1})) $
• Symbol Explanation:
  • $V$: A video.
  • $N$: Total number of frames in the video.
  • $F_i$: The $i$-th frame of the video $V$.
  • $\text{CLIP\_Encoder}(X)$: The CLIP model's encoder that converts an image $X$ into its embedding vector.
  • $\text{cosine\_similarity}(u, v)$: The cosine similarity between two vectors $u$ and $v$. Higher values are better.

5.2.4. User Preference

• Conceptual Definition: User Preference is a subjective human evaluation metric that measures how much human evaluators prefer the output of one method over others, based on various qualitative factors like visual quality, naturalness of edits, and fidelity to instructions. It is typically obtained through user studies where participants rate or choose preferred outputs.
• Mathematical Formula: There is no standard mathematical formula for User Preference. It's usually reported as a percentage of times a given method's output was preferred in a pairwise comparison or as an average score from a Likert scale rating. In this paper, it's reported as a percentage in a user study.
• Symbol Explanation: N/A, as it's a direct percentage from user feedback. Higher values are better.

5.2.5. PSNR (Peak Signal-to-Noise Ratio)

• Conceptual Definition: PSNR is a widely used objective metric to quantify the quality of reconstruction of an image or video compared to an original reference. It measures the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. It is typically expressed in decibels (dB). A higher PSNR value indicates better quality (less noise/distortion).
• Mathematical Formula: First, MSE (Mean Squared Error) is calculated: $ \text{MSE} = \frac{1}{MN} \sum_{i=0}^{M-1} \sum_{j=0}^{N-1} [I(i,j) - K(i,j)]^2 $ Then, PSNR is calculated using MSE: $ \text{PSNR} = 10 \cdot \log_{10} \left( \frac{\text{MAX}_I^2}{\text{MSE}} \right) $
• Symbol Explanation:
  • I(i,j): The pixel value at coordinates (i,j) in the original image/frame.
  • K(i,j): The pixel value at coordinates (i,j) in the edited (or noisy) image/frame.
  • M, N: Dimensions of the image (height and width).
  • $\text{MAX}_I$: The maximum possible pixel value of the image (e.g., 255 for 8-bit grayscale images).
  • $\text{log}_{10}$: The base-10 logarithm. Higher values are better.

5.2.6. SSIM (Structural Similarity Index Measure)

• Conceptual Definition: SSIM is another objective metric used to assess the perceived quality of distorted images or videos. Unlike PSNR which measures absolute errors, SSIM aims to model the human visual system's perception of structural information. It compares three key features: luminance, contrast, and structure. Its value ranges from -1 to 1, where 1 indicates perfect similarity.
• Mathematical Formula: For two image patches $x$ and $y$: $ \text{SSIM}(x, y) = \frac{(2\mu_x\mu_y + C_1)(2\sigma_{xy} + C_2)}{(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)} $
• Symbol Explanation:
  • $\mu_x, \mu_y$: The average (mean) of pixel values in image patches $x$ and $y$, respectively.
  • $\sigma_x, \sigma_y$: The standard deviation of pixel values in image patches $x$ and $y$, respectively.
  • $\sigma_{xy}$: The covariance between $x$ and $y$.
  • $C_1 = (K_1L)^2$, $C_2 = (K_2L)^2$: Small constants to avoid division by zero or instability when the denominators are very small.
  • $L$: The dynamic range of the pixel values (e.g., 255 for 8-bit images).
  • $K_1, K_2$: Small constants, typically $K_1 = 0.01$ and $K_2 = 0.03$. Higher values are better.

5.2.7. LPIPS (Learned Perceptual Image Patch Similarity)

• Conceptual Definition: LPIPS is a perceptual metric that measures the distance between two images, often used to evaluate the realism or perceptual similarity of generative models outputs. Unlike PSNR or SSIM, LPIPS uses features extracted from pre-trained deep neural networks (like VGG or AlexNet) to compare images, better aligning with human perception of similarity. A lower LPIPS score indicates higher perceptual similarity.
• Mathematical Formula: $ \text{LPIPS}(x, x_0) = \sum_l \frac{1}{H_l W_l} \sum_{h,w} ||w_l \odot (\phi_l(x){h,w} - \phi_l(x_0){h,w})||_2^2 $
• Symbol Explanation:
  • $x, x_0$: Two image patches (reference and distorted).
  • $l$: Index over different layers of the pre-trained neural network.
  • $\phi_l$: Feature maps from layer $l$ of the pre-trained network.
  • $H_l, W_l$: Height and width of the feature maps at layer $l$.
  • $w_l$: A learnable weight vector that scales the feature differences at each layer.
  • $\odot$: Element-wise multiplication.
  • $||\cdot||_2^2$: Squared L2 norm. Lower values are better.

5.2.8. MSE (Mean Squared Error)

• Conceptual Definition: MSE is a fundamental metric that measures the average of the squares of the errors, i.e., the average squared difference between the estimated values and the actual value. In image/video processing, it quantifies the average pixel-wise difference between a generated/edited image/frame and its ground truth. A lower MSE indicates higher fidelity to the original.
• Mathematical Formula: $ \text{MSE} = \frac{1}{MN} \sum_{i=0}^{M-1} \sum_{j=0}^{N-1} [I(i,j) - K(i,j)]^2 $
• Symbol Explanation:
  • I(i,j): The pixel value at coordinates (i,j) in the original image/frame.
  • K(i,j): The pixel value at coordinates (i,j) in the edited (or noisy) image/frame.
  • M, N: Dimensions of the image (height and width). Lower values are better.

5.2.9. Relevance

• Conceptual Definition: This metric is specifically used in the context of the remover expert evaluation. In the paper's Table 8, it appears as a quantitative measure, with lower values being better. It likely measures how well the removed object is not present in the edited video, or the success of the removal, potentially by comparing features of the removed region with the background. Without a specific definition in the paper, it's inferred to be an inverse measure of the presence or detectability of the "removed" content.
• Mathematical Formula: Not provided in the paper.
• Symbol Explanation: Not provided in the paper.

5.3. Baselines

The paper compares its approach and the models trained on Señorita-2M against several representative video editing methods and architectures:

• Tokenflow (Geyer et al., 2023): An inversion-based video editing method that focuses on consistency by using consistent diffusion features (attention across frames). It's a strong baseline for ensuring temporal coherence.
• Flatten: This method is mentioned in the quantitative comparisons but is not explicitly cited in the bibliography. It might refer to a common internal baseline or a generalized approach that flattens video editing into frame-by-frame image editing, which often struggles with temporal consistency.
• AnyV2V (Ku et al., 2024): A plug-and-play framework for video-to-video editing that generates edited videos by injecting features, guided by the first frame. This tests performance against methods that use first-frame guidance.
• InsV2V (Cheng et al., 2024): An end-to-end video editing method that trains on generated video pairs. This is a crucial baseline for direct comparison, as both InsV2V and the proposed approach rely on video pairs for training. InsV2V is noted for its insufficient data quality.
• Propainter (Zhou et al., 2023): Specifically used as a baseline for object removal evaluation. It's an inpainting model that aims for propagation and transformer-based improvements in video inpainting.
• Different Editing Architectures: For the ablation study on architectures, the paper investigates variants of two widely used image editing architectures adapted for video:
  • InstructPix2Pix-based (Ins-Edit) (Brooks et al., 2023): This architecture concatenates conditions (like text prompts) and input latents to output predicted noise.
  • ControlNet-based (Control-Edit) (Zhang et al., 2023a): This architecture uses a control branch for editing conditions and a main branch for input latents.
  • These architectures are further evaluated with:

    • * (e.g., Ins-Edit*, Control-Edit*): Denotes enhancement using the Omni-Edit dataset.

    • FF- (e.g., FF-Ins-Edit, FF-Control-Edit): Denotes the use of first-frame guidance.

      These baselines are representative because they cover both inversion-based and end-to-end video editing, various strategies for temporal consistency, and different architectural approaches for conditional image/video generation. This broad comparison allows the paper to effectively demonstrate the advantages of Señorita-2M and the models trained on it.

6. Results & Analysis

The experimental results validate the effectiveness of the Señorita-2M dataset in training high-quality video editing models. The analysis compares models trained on Señorita-2M with existing methods, conducts ablation studies on dataset quality and quantity, and explores the impact of different editing architectures.

6.1. Core Results Analysis

6.1.1. Quantitative Comparison (Main)

The main quantitative comparison (Table 2) evaluates the overall performance of the model trained on Señorita-2M against several state-of-the-art baselines.

The following are the results from Table 2 of the original paper:

• Analysis:

  • Overall Dominance: The model trained on Señorita-2M (Ours) consistently outperforms all other methods across all evaluated metrics.

  • Ewarp (↓): Ours achieves the lowest Ewarp of 9.42, indicating significantly less warping and higher geometric stability compared to Tokenflow (16.31), Flatten (16.31), AnyV2V (20.48), and InsV2V (16.50). This suggests superior temporal smoothness.

  • CLIPScore (↑): Ours leads with a CLIPScore of 0.2895, demonstrating the best text-video alignment. This is a notable improvement over AnyV2V (0.2723) and especially InsV2V (0.1675), highlighting the effectiveness of Señorita-2M in teaching models to follow instructions.

  • Temporal Consistency (↑): With a score of 0.9775, Ours shows excellent consistency across frames, surpassing all baselines, including Tokenflow (0.9752) which is designed for consistency.

  • User Preference (↑): The most striking result is the User Preference score of 53.17%, which is dramatically higher than the next best (AnyV2V at 19.40%). This indicates a strong qualitative advantage and superior perceived quality by human evaluators.

    The paper specifically emphasizes the comparison with InsV2V, as both are end-to-end methods that rely on video pairs for training. The model trained on Señorita-2M significantly surpasses InsV2V on all metrics (Ewarp: 9.42 vs. 16.50; CLIPScore: 0.2895 vs. 0.1675; Temporal Consistency: 0.9775 vs. 0.9727). This strongly validates the superior quality and effectiveness of Señorita-2M over previous datasets for end-to-end video editing.

6.1.2. Qualitative Results

The quantitative superiority is reinforced by qualitative results and a user study. The high User Preference score of 53.17% (Table 2) indicates that users find the edited videos from models trained on Señorita-2M to be significantly more appealing and relevant.

该图像是多组视频编辑结果的对比示意图，展示了在不同编辑指令下如去除人物、改变车轮颜色、添加帽子和转为二次元风格的效果变化。每组包括原始视频帧及对应的编辑后帧，体现了各编辑方法的性能差异。

The figure above provides a visual comparison of editing results from different methods. Each row presents an editing task (e.g., "Remove the man", "Change the wheel to yellow", "Add a hat", "Make it anime style"), showing the original video and the edited outputs from AnyV2V, InsV2V, and Ours. The results visually confirm the superior quality, naturalness, and adherence to instructions of the model trained on Señorita-2M compared to the baselines. For instance, in "Remove the man", Ours seamlessly removes the person with a natural background, while AnyV2V and InsV2V leave noticeable artifacts or distortions. Similarly, the style transfer and object edits appear more cohesive and high-fidelity with Ours.

6.1.3. Expert Model Quantitative Comparisons (Appendix B)

The paper also provides quantitative comparisons for each of the specialized expert models that generate the Señorita-2M dataset, demonstrating their individual state-of-the-art performance.

Global Stylization Expert

The following are the results from Table 5 of the original paper:

• Analysis: The Global Stylizer Expert achieves the best Ewarp (9.02), CLIPScore (0.3145), and Temporal Consistency (0.9781), indicating its effectiveness in performing global style transfer with high stability, text alignment, and temporal coherence. This confirms the high quality of the data generated for global stylization tasks within Señorita-2M.

Local Stylization Expert

The following are the results from Table 6 of the original paper:

• Analysis: The Local Stylizer Expert achieves the best Ewarp (6.50), CLIPScore (0.2944), Temporal Consistency (0.9828), PSNR (28.29), SSIM (0.9843), LPIPS (0.0346), and MSE (108.25). This comprehensive excellence indicates its ability to perform local stylization effectively while preserving the background and maintaining high temporal coherence and perceptual quality. InsV2V shows a competitive Ewarp but falls behind in other quality metrics.

Text-guided Video Inpainter (Object Swap) Expert

The following are the results from Table 7 of the original paper, specifically for object swap which utilizes the inpainter:

• Analysis: The Inpainter Expert (used for object swap) achieves the highest CLIPScore (0.3186) and Temporal Consistency (0.9782), indicating strong text alignment and frame coherence. While InsV2V has a lower Ewarp (8.80 vs. 12.06), its CLIPScore is significantly lower, suggesting its object swap results might not always align well with the prompt despite less warping. The Inpainter Expert also excels in PSNR, SSIM, LPIPS, and MSE, demonstrating superior perceptual quality and fidelity.

Video Remover Expert

The following are the results from Table 8 of the original paper:

• Analysis: The Remover Expert achieves the lowest Ewarp (4.21) and Relevance (0.1554), indicating superior geometric stability and the most successful object removal. It also has very high Temporal Consistency (0.9864), slightly surpassing Propainter. While Propainter shows exceptionally high PSNR, SSIM, and low LPIPS/MSE (suggesting high pixel-level fidelity, possibly due to blurring or very subtle changes), the Remover Expert's leading Ewarp and Relevance scores highlight its strength in clean, stable object removal without visual artifacts, which was the stated goal.

6.2. Ablation Studies / Parameter Analysis

6.2.1. Ablation Study on Dataset Quality and Quantity

The following are the results from Table 3 of the original paper:

• Analysis:
  • Impact of Dataset Quality (Ablation-1 vs. Ablation-2): By keeping the training samples (60K) and epochs (8) constant, switching from the InsV2V dataset to Señorita-2M significantly improves performance. CLIPScore increases from 0.2366 to 0.2596, and Temporal Consistency rises from 0.9712 to 0.9783. Ewarp also slightly improves (8.51 to 8.44). This clearly demonstrates that the high quality of the Señorita-2M dataset is crucial for training effective video editing models, leading to better text alignment and frame coherence.
  • Impact of Dataset Quantity (Ablation-2 vs. Ablation-3): Increasing the number of training samples from 60K to 120K from Señorita-2M (while reducing epochs to 4) further enhances results. CLIPScore improves to 0.2641, Temporal Consistency reaches 0.9785, and Ewarp further decreases to 7.95. This indicates that a larger and more diverse dataset enables the model to learn a broader range of editing capabilities, leading to reduced warping errors and improved consistency.

6.2.2. Ablation Study on Different Editing Architectures

The following are the results from Table 4 of the original paper:

• Analysis: This study investigates the impact of InstructPix2Pix (Ins-Edit) and ControlNet (Control-Edit) architectures, Omni-Edit dataset enhancement (*), and first-frame guidance (FF-) on video editing performance.
  • Base Architectures: Control-Edit (14.40% user preference) slightly outperforms Ins-Edit (3.87%) in user preference and CLIPScore (0.2882 vs 0.2648), despite Ins-Edit having higher Temporal Consistency (0.9797).
  • Dataset Enhancement (*): Incorporating the Omni-Edit dataset (Control-Edit*) significantly boosts user preference to 23.26% and achieves the highest Temporal Consistency (0.9802). This suggests that combining diverse, high-quality image editing data can benefit video editing. Interestingly, Ins-Edit* performs worse than Ins-Edit in some metrics, indicating that not all dataset enhancements are universally beneficial across architectures.
  • First-Frame Guidance (FF-): Models with first-frame guidance show substantial improvements. FF-Control-Edit achieves the highest User Preference at 37.12% and the highest CLIPScore (0.2895), confirming the importance of first-frame guidance for superior editing results. FF-Ins-Edit achieves the lowest Ewarp (8.44), indicating excellent geometric stability.
  • Conclusion on Architectures: The results highlight that first-frame guidance is a critical component for high-performance video editing models, drastically improving user preference and text alignment. ControlNet-based architectures generally show strong performance, especially when combined with first-frame guidance and potentially dataset enhancements.

6.3. General Training Parameters and Environment

• Data Preparation: Original videos are resized to $336 \times 592$ or $592 \times 336$. BLIP-2 descriptions and 810K masks with corresponding phrases are prepared.
• Instruction Generation: LLMs (Dubey et al., 2024) transform object names or editing prompts into clear instructions.
• Determining Source and Target Videos: For Object Swap and Object Addition, the edited video is the source and the original is the target (or vice-versa for addition). For Object Removal and Stylization, the edited video is the target.
• Filtering Pipeline Parameters:

  • Quality filter: Threshold of 0.6 to remove failure cases.

  • CLIP similarity (text-alignment): Threshold of 0.22 for object removal and local stylization; 0.2 for global stylization and object addition.

  • CLIP similarity (subtle changes): Remove pairs with similarity above 0.95 between original and edited videos.

    These results collectively demonstrate the efficacy and superiority of Señorita-2M as a training dataset, and the importance of both dataset quality and quantity, as well as suitable architectural choices like first-frame guidance, for achieving state-of-the-art video editing performance.

7. Conclusion & Reflections

7.1. Conclusion Summary

In this paper, the authors introduce Señorita-2M, a groundbreaking high-quality, instruction-based video editing dataset. This dataset is distinguished by its massive scale, comprising approximately 2 million video editing pairs, and its comprehensive coverage of 18 distinct video editing tasks (including local and global stylization, object removal/addition/swap, inpainting/outpainting, and various conditional generations).

The creation of Señorita-2M involved a novel methodology:

1. Development of Specialized Expert Models: Four state-of-the-art video editing models (global stylizer, local stylizer, text-guided inpainter, video remover) were meticulously crafted and trained to serve as high-quality data generators.

2. Sophisticated Data Pipeline: These experts were integrated with other computer vision tools (CogVLM2, Grounded-SAM2) and Large Language Models (LLMs) for robust video annotation and instruction generation, ensuring precise and diverse editing commands.

3. Rigorous Filtering: A multi-stage filtering pipeline was implemented to guarantee data quality by eliminating failed edits, poorly text-aligned videos, and subtly changed pairs.

   Extensive experiments confirmed the dataset's effectiveness. Models trained on Señorita-2M achieved remarkably high-quality video editing results, demonstrating significant improvements in visual quality, frame consistency, and text alignment compared to previous methods. Furthermore, an exploration of common video editing architectures highlighted the critical role of first-frame guidance and dataset quality in enhancing performance. The dataset and models are planned to be open-sourced, fostering further research and development in the field.

7.2. Limitations & Future Work

The paper explicitly discusses a crucial Impact Statement which implicitly outlines a limitation and direction for future work:

• Potential for Misuse: The authors acknowledge that models trained on Señorita-2M are capable of editing videos, which inherently carries the risk of generating deepfakes or other misleading content. This is a significant societal concern with advanced generative AI.

• Mitigation Strategy: The suggested mitigation is to rely on deepfake detection methods. This points towards a future research direction where robust deepfake detection becomes an essential counterpart to powerful video editing technologies.

  While the paper doesn't explicitly list a "Future Work" section, several avenues can be inferred:

• Architectural Optimization: The exploration of different video editing architectures (e.g., InstructPix2Pix, ControlNet, and first-frame guided variants) suggests that further optimization or novel architectural designs tailored specifically for end-to-end video editing on large datasets could be a promising direction.

• Beyond Pexels: While Pexels provides high-quality content, expanding the diversity of source videos (e.g., beyond professionally shot clips to user-generated content) could further generalize models.

• Real-time Editing: The current inference times for experts (e.g., 1-3 minutes) suggest there's still room for optimizing the editing process towards real-time applications, which would require more efficient models or specialized hardware.

• User Controllability: Further enhancing fine-grained user control and interactivity in video editing, possibly by integrating more sophisticated human-in-the-loop feedback mechanisms, could be explored.

• Ethical AI Development: Beyond mere detection, proactive measures to watermark generated content, develop responsible deployment guidelines, or embed ethical safeguards directly into generative models could be crucial for addressing the deepfake challenge more comprehensively.

7.3. Personal Insights & Critique

The Señorita-2M paper presents a highly impactful contribution to the field of video editing, addressing a critical bottleneck that has hindered the progress of end-to-end methods. My personal insights and critique are as follows:

• Strengths:

  • Addressing the Data Gap: The most significant strength is the creation of a large-scale, high-quality, instruction-based dataset. This directly tackles the primary challenge for end-to-end video editing, positioning Señorita-2M as a foundational resource for future research.
  • Innovative Data Generation: The strategy of using multiple specialized SOTA expert models (Global Stylizer, Local Stylizer, Inpainter, Remover) for data generation is highly effective. It allows for the creation of diverse and high-fidelity edited videos that would be impossible to obtain through manual annotation or simple synthetic methods. This expert-driven synthesis is a robust approach to dataset construction.
  • Rigorous Quality Control: The multi-stage filtering pipeline (quality, text-alignment, subtle change removal) is commendable. It ensures that the millions of generated pairs are genuinely useful, preventing model overfitting to low-quality or irrelevant data. The use of CLIP for both text alignment and visual similarity is a smart application of multimodal models.
  • Comprehensive Task Coverage: The 18 distinct editing tasks, encompassing both local and global transformations, demonstrate a commitment to building a general-purpose dataset. This breadth will enable the training of more versatile video editing models.
  • Demonstrated Impact: The extensive experiments and ablation studies clearly validate the dataset's superiority and highlight key architectural considerations, providing valuable insights for the community. The user preference study adds crucial qualitative validation.
  • Open-Sourcing Commitment: The promise to open-source the dataset and models is a significant strength, promoting reproducibility and accelerating research for the wider AI community.

• Potential Issues & Areas for Improvement:

  • Transparency of Expert Training Details: While the appendix provides some training details for the experts, a more comprehensive exposition of their individual architectures, specific losses, and how their "SOTA" claims are validated (e.g., against other specialized models for removal/inpainting, not just general video editing baselines) in the main paper could further strengthen the methodology section.
  • Threshold Justification: The specific thresholds used in the filtering pipeline (e.g., 0.6 for quality, 0.22/0.2 for CLIP text alignment, 0.95 for subtle changes) are given but not deeply justified. An ablation study on these thresholds could show their impact and provide more confidence in their selection.
  • Computational Cost: Generating 2 million video pairs with specialized SOTA models and a rigorous filtering process is computationally intensive. While this is a necessary cost for a high-quality dataset, a discussion on the total computational resources (e.g., GPU hours, energy consumption) would provide valuable context for sustainability and resource allocation in similar large-scale data generation efforts.
  • "Flatten" Baseline: The Flatten baseline is used in quantitative comparisons but lacks a citation. Clarifying its definition or providing a reference would improve reproducibility and understanding.
  • Refinement of Metrics: For Ewarp and Relevance, providing the exact mathematical formulations used in the paper (even if in an appendix) would enhance the rigor and clarity of the evaluation.
  • Ethical Considerations and Proactive Measures: While the Impact Statement is appreciated, the reliance on deepfake detection methods as the primary mitigation is a reactive approach. Future work could delve into embedding digital watermarks, provably uneditable regions, or user consent mechanisms within the editing process itself to proactively address ethical concerns.

• Transferability and Future Value:

  • The methodology for generating high-quality datasets using specialized expert models and LLM-driven instruction generation is highly transferable. This paradigm could be applied to other complex generative tasks where high-quality paired data is scarce, such as 3D asset generation, complex scene manipulation, or scientific data augmentation.
  • The Señorita-2M dataset itself will undoubtedly serve as a critical benchmark for the next generation of video editing models, driving advancements in temporal consistency, instruction following, and perceptual quality. It has the potential to accelerate the development of practical and versatile video editing tools for various applications, from professional content creation to everyday user-generated content.