1. Bibliographic Information

• Title: Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play
• Authors: Qinsi Wang, Bo Liu, Tianyi Zhou, Jing Shi, Yueqian Lin, Yiran Chen, Hai Helen Li, Kun Wang, Wentian Zhao.
• Affiliations: The authors are from Duke University, National University of Singapore, University of Maryland, and Adobe Inc. This collaboration brings together expertise from top academic institutions and a leading industry research lab.
• Journal/Conference: The paper is available as a preprint on arXiv. Preprints are common in fast-moving fields like AI for rapid dissemination of results, though they have not yet completed formal peer review.
• Publication Year: The paper was submitted to arXiv in 2025 (as per the citation format in the paper).
• Abstract: The paper addresses the high cost and scalability limitations of training Vision-Language Models (VLMs), which traditionally depend on manually created datasets. The authors propose Vision-Zero, a framework that enables VLMs to improve themselves through a competitive game called "Who Is the Spy?". This game can be generated from any pair of images with subtle differences. The key contributions are: 1) a strategic self-play framework where models learn by playing different roles, generating their own training data; 2) the ability to use arbitrary images (synthetic, charts, real-world), enhancing generalization; and 3) a novel training algorithm called Iterative Self-Play Policy Optimization (Iterative-SPO), which alternates between self-play and reinforcement learning to ensure sustained performance gains. Despite using no human-labeled data for the game, Vision-Zero achieves state-of-the-art results on various reasoning and vision tasks, outperforming methods that rely on expensive annotated datasets.
• Original Source Link: https://arxiv.org/pdf/2509.25541 (Note: The link provided in the prompt is a placeholder). The paper is currently available as a preprint.

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: Training powerful Vision-Language Models (VLMs) is incredibly expensive and slow. Current methods, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), rely heavily on large, diverse datasets that require massive human effort for creation, annotation, and verification.
  • Importance & Gaps: This dependency creates two major bottlenecks: a data scarcity problem, where the cost limits the scale and variety of training data, and a knowledge ceiling, where a model's capabilities are fundamentally limited by the quality and scope of human-provided examples. It cannot learn strategies or knowledge beyond what its human supervisors already know. The paper argues that extending self-play, a technique successful in games like Go and Dota 2, to VLMs could break these barriers, but designing a suitable game for multimodal reasoning is a non-trivial challenge.
  • Fresh Angle: Vision-Zero introduces a novel paradigm for VLM training that is completely independent of human-annotated data. It frames the learning process as a multi-agent social deduction game ("Who Is the Spy?") played using pairs of images. This gamified approach forces the VLM to develop sophisticated visual reasoning, strategic communication, and deception detection skills, all while generating its own training data from the game's outcomes.

• Main Contributions / Findings (What):

  • Vision-Zero Framework: The primary contribution is a zero-human-in-the-loop training framework for VLMs. It uses a "Who Is the Spy?" game to drive self-improvement. This framework is domain-agnostic, meaning it can generate games from any type of image pair (synthetic scenes, charts, real-world photos), making it highly versatile and cost-effective.
  • Iterative Self-Play Policy Optimization (Iterative-SPO): A novel training algorithm that alternates between two phases: self-play for strategic exploration (Clue Stage) and reinforcement learning with verifiable rewards (RLVR) for exploiting correct decision-making (Decision Stage). This hybrid approach prevents the model from getting stuck in a performance plateau (a common issue in pure self-play) and ensures stable, continuous improvement.
  • State-of-the-Art Performance without Labels: The paper demonstrates that despite being trained on automatically generated, label-free data, Vision-Zero significantly enhances the performance of base VLMs. It surpasses strong baselines trained on expensive, human-annotated datasets, particularly in reasoning, chart question-answering, and other vision-centric tasks.

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • Vision-Language Models (VLMs): These are AI models, like GPT-4o or Qwen-VL, that can understand and process information from both images and text simultaneously. They can perform tasks like describing an image, answering questions about it, or following text-based instructions to edit an image.
  • Reinforcement Learning (RL): A type of machine learning where an agent learns to make decisions by performing actions in an environment to maximize a cumulative reward. It learns through trial and error.
  • Reinforcement Learning from Human Feedback (RLHF): A technique used to align large language models with human preferences. Humans rank the model's outputs, and this feedback is used to train a "reward model" that guides the main model's learning process.
  • Reinforcement Learning with Verifiable Rewards (RLVR): A variant of RL where the reward for an action can be automatically and objectively verified without human judgment. For example, in a math problem, the reward is high if the final answer is correct and low otherwise.
  • Self-Play: A training paradigm where an AI agent learns by playing a game against itself (or copies of itself). As the agent improves, its opponent also becomes stronger, creating a curriculum of ever-increasing difficulty. This allows the agent to surpass human-level performance, as seen with AlphaGo.

• Previous Works & Differentiation:

  • The paper situates itself against traditional VLM training methods that are costly (e.g., COCO, Ego4D, Visual Genome datasets). It presents Vision-Zero as a solution to the data and knowledge bottlenecks of these methods.

  • It draws inspiration from recent work applying self-play to Large Language Models (LLMs) in games like Tic-Tac-Toe (SPIRAL) or math/coding puzzles (Absolute Zero). However, it notes that extending self-play to the more complex, multimodal domain of VLMs is largely unexplored.

  • The authors establish four criteria for an ideal self-play environment: (1) skills learned in the game must transfer to target tasks, (2) skill growth must be scalable, (3) the environment must be diverse, and (4) it should require little to no external data. They argue existing visual games fail to meet all four criteria simultaneously.

  • Vision-Zero is differentiated by its unique "Who Is the Spy?" game design, which naturally requires visual discrimination, logical reasoning, and strategic communication. Unlike other gamified frameworks that might use a single, narrow environment (like Go or Chess), Vision-Zero's ability to generate games from any image pair makes its environment incredibly diverse and generalizable.

    Image 2 illustrates the conceptual shift proposed by Vision-Zero.

    该图像为三张插图，分别示意了三种学习模式下人机交互中的视觉空间推理过程。(a) 监督学习场景中，人向机器人直接描述红色立方体相对于圆柱体的位置；(b) 强化学习场景中，人提问机器人“红色立方体相对于圆柱体在哪里？”，机器人思考后回答“左边”；(c) Vision-Zero场景中，两个机器人通过相互对话协作推理红色立方体的位置，表现出更自然且高效的互动推理能力。

• (a) Supervised Learning: A human explicitly tells the model the answer ("The red cube is to the left of the cylinder."). The model learns by mimicking this labeled data.

• (b) Reinforcement Learning: A human asks a question, and the model explores to find the answer. It receives a reward if its answer is correct. This still relies on human-created question-answer pairs.

• (c) Vision-Zero: Models interact with each other in a game. They generate their own data (clues, votes) and learn from the game's outcome, completely eliminating the need for direct human supervision.

4. Methodology (Core Technology & Implementation)

The Vision-Zero framework consists of two main components: the game environment and the Iterative-SPO training algorithm.

• Principles: The core idea is that a competitive game can create a natural curriculum for learning complex reasoning skills. In "Who Is the Spy?", players must ground their language in visual evidence, reason about inconsistencies between different players' descriptions, and think strategically to win. This process generates rich, implicit supervision signals that can be used for training.

4.1 Environment and Data

• Strategic Environment: "Who Is the Spy?" Game

  • Setup: A game involves multiple players, typically one Spy and several Civilians (e.g., $n_c=4$ civilians).
  • Image Distribution: All Civilians receive the same image ($I_c$), while the Spy receives a subtly different version ($I_s$). The goal for the Civilians is to identify the Spy, and the goal for the Spy is to remain hidden.
  • Clue Stage: Players take turns providing a single-sentence verbal clue about their image. Each player can see the clues provided by previous players.
    • Civilian Strategy: Provide accurate clues to help other civilians identify the common image, but without giving away too much information that the Spy could exploit.
    • Spy Strategy: Analyze other players' clues to infer the content of the "civilian" image and provide a clue that is vague enough or focuses on common elements to avoid suspicion.
  • Decision Stage: After all clues are given, the Civilians privately analyze all the clues and their own image to vote on who they think the Spy is. A player can also respond with "n/a" if they are uncertain. The player with the most votes is identified.

• Label-Free and Domain-Agnostic Data Input

  • Vision-Zero's key innovation is its ability to create this game from any pair of images ($I_c, I_s$) that have a difference. This makes it highly scalable and adaptable.

  • The paper demonstrates this using three data types, as shown in Image 4:

    该图像为三组示意图，分别展示了CLEVR合成数据、图表数据和真实世界数据的对比样例。CLEVR部分展示了不同物体的位置和形状变化，图表部分展示了属性数据的交换及各种图表形式，真实世界部分展示了物体或风格的小幅变化，如风景、人物和动物的对比图。整体体现了论文通过多样图像对进行游戏化训练的多领域适用性。

  • CLEVR Data: Synthetic images of simple 3D shapes. The "spy" image is created by programmatically changing the color and shape of two objects. This is cheap to generate (6 hours on one GPU for 2,000 pairs).

  • Chart Data: Real charts (bar, line, pie) from the ChartQA dataset. The "spy" image is generated by using an LLM (Gemini2.5-Flash) to swap numerical attributes and then re-rendering the chart.

  • Real-World Data: Natural images from the ImgEdit dataset. These pairs already contain an original and an edited version.

4.2 Iterative Self-Play Policy Optimization (Iterative-SPO)

This novel algorithm alternates between training the model on the two game stages to ensure continuous and stable improvement.

• Self-Play Policy Optimization in Clue Stage

  • Goal: Players aim to give clues that minimize the number of votes they receive in the decision stage.
  • Zero-Sum Reward: The reward system is designed as a zero-sum game between the Spy and the Civilians. The reward for the Spy ($r_s^{clue}$) and for each Civilian ($r_{c_j}^{clue}$) is defined based on the number of votes received. A player who receives more votes gets a lower reward.
  • Role-Advantage Estimation (RAE): The Spy has an informational disadvantage (they don't know what the civilian image looks like initially). RAE is used to adjust the rewards to account for this asymmetry, ensuring a fairer and more stable learning signal. The advantage $A_k^{clue}$ is the reward minus a baseline for that role.
  • Objective Function: The model's policy ($\pi_{\theta}$) is updated using a policy gradient method. The objective is to maximize the advantage-weighted log-probability of the generated clues, with a KL-divergence term to prevent the policy from deviating too far from a reference policy ($\pi_{\text{ref}}$), which stabilizes training.

• RLVR in the Decision Stage

  • Goal: Civilians aim to correctly identify the Spy.
  • Discrete Reward: The reward is verifiable and discrete: +1 for a correct vote, -1 for a wrong vote, and -0.5 for an uncertain ("n/a") vote. This encourages accurate identification but penalizes guessing less than being confidently wrong.
  • Group Normalization & Objective: Since some game rounds are inherently harder than others, the rewards within a round are normalized (subtracting the mean and dividing by the standard deviation). This creates a round-agnostic advantage signal, $A_{c_i}^{dec}$. The model is then trained to maximize the advantage-weighted log-probability of its votes, again with KL regularization.

• Iterative Stage Training

  • Problem: Pure self-play can lead to a performance plateau (local equilibrium), while pure RLVR can lead to overfitting on the decision task.
  • Solution: Iterative-SPO dynamically switches between training the Clue Stage and the Decision Stage based on performance metrics.
    • Switch from Decision to Clue: If the model gets very good at identifying the spy (high accuracy acc_t, low uncertainty na_t), it means the clues have become too easy to decipher. The system then switches to training the Clue Stage to make the players (especially the Spy) generate more sophisticated and deceptive clues.
    • Switch from Clue to Decision: If the spy becomes too good at hiding (low accuracy, high uncertainty), it means the decision-maker is not skilled enough. The system then switches to training the Decision Stage to improve the model's ability to detect subtle inconsistencies.
  • This alternating process ensures that the game's difficulty continuously adapts to the model's current skill level, driving sustained improvement.

5. Experimental Setup

• Datasets:

  • Training: The model is trained on the custom-generated datasets: CLEVR-based (2k pairs), Chart-based (1k pairs), and Real-World (1k pairs).
  • Evaluation: The trained models are evaluated on a wide range of 14 public benchmarks to test for generalization, categorized into:
    • Reasoning & Math: MathVista, MathVision, WeMath, MathVerse, LogicVista, DynaMath.
    • Chart/OCR: AI2D, ChartQA, OCRBench, SEED-Bench-2.
    • Vision-Centric: RealWorldQA, MMVP, BLINK, MuirBench.
    • General Multimodal: MMMU, MMMU-pro.

• Evaluation Metrics:

  • The primary metric across all benchmarks is Accuracy.
  • Conceptual Definition: Accuracy measures the proportion of correctly answered questions out of the total number of questions in a benchmark dataset. It is a straightforward measure of a model's correctness.
  • Mathematical Formula: $$\text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}$$
  • Symbol Explanation:
    • Number of Correct Predictions: The count of test instances where the model's output matches the ground-truth label.
    • Total Number of Predictions: The total number of instances in the test set.

• Baselines:

  • The base model for most experiments is Qwen2.5-VL-7B. The effectiveness is also shown on InternVL3-8B and InternVL3-14B.
  • The method is compared against several state-of-the-art (SOTA) post-training methods that use expensive human-labeled data:
    • R1-OneVision-7B, MM-Eureka-Qwen-7B, VLAA-Thinker-7B, OpenVLThinker-7B: These are all trained with RLVR-style methods on curated reasoning and math datasets.
    • ViGaL: This is another game-based training method, but it collects data offline from games like Snake and then trains on it, rather than using online, interactive self-play.

6. Results & Analysis

The experimental results robustly demonstrate the effectiveness of the Vision-Zero framework.

6.1 Core Results

• Strong Generalization to Reasoning and Math Tasks (Table 1):

  Method	MathVista	MathVision	WeMath	MathVerse	LogicVista	DynaMath	Avg.
  Proprietary Model
  GPT4o	61.4	30.4	40.0	50.2	45.9	32.3	43.4
  Gemini2.0-Flash	73.4	41.3	57.1	54.4	56.2	43.7	54.4
  Performance on Qwen2.5-VL-7B
  Qwen2.5-VL-7B	68.2	25.4	36.1	49.0	47.2	20.9	41.1
  R1-OneVision-7B	64.1	24.1	35.8	47.1	44.5	21.4	39.5
  MM-Eureka-Qwen-7B	73.0	26.9	36.2	50.3	42.9	24.2	42.9
  VLAA-Thinker-7B	68.0	26.4	36.0	51.7	47.2	21.9	41.9
  OpenVLThinker-7B	70.2	25.3	36.5	47.9	44.3	21.2	40.9
  ViGaL-Snake	70.7	26.5		51.1
  ViGaL-Rotation	71.2	26.3		50.4
  ViGaL-Snake+Rotation	71.9	27.5	36.9	52.4	46.5	22.9	43.0
  VisionZero-Qwen-7B (CLEVR)	72.6	28.1	39.8	51.9	50.1	22.3	44.1
  VisionZero-Qwen-7B (Chart)	72.2	27.6	39.2	52.1	50.6	21.9	43.9
  VisionZero-Qwen-7B (Real-World)	72.4	28.0	39.5	52.2	50.3	22.1	44.1

Note: This table is a transcription of the original data from Table 1, not the original image.

*   <strong>Analysis:</strong> `Vision-Zero` models consistently achieve the highest average score (44.1%) among all open-source models, outperforming the base model by 3% and the strongest baseline (`ViGaL`) by 1.1%. This is remarkable because `Vision-Zero` was not explicitly trained on any math or reasoning problems; the skills learned in the strategic game generalized effectively to these complex domains.

• Mitigation of Negative Transfer (Table 2):

  	Chart / OCR	Vision-Centric
  Model	AI2D	ChartQA
  Proprietary Model
  GPT-40	84.4	85.7
  Gemini2.0-Flash	87.2	79.3
  Performance on Qwen2.5-VL-7B
  Qwen2.5-VL-7B	84.7	86.1
  R1-OneVision-7B	82.2
  MM-Eureka-Qwen-7B	84.1	77.3
  VLAA-Thinker-7B	84.0	84.3
  OpenVLThinker-7B	81.8
  ViGaL-Snake+Rotation	84.5	79.9
  VisionZero-Qwen-7B (CLEVR)	84.5	86.3
  VisionZero-Qwen-7B (Chart)	85.8	87.2
  VisionZero-Qwen-7B (Real-World)	84.8	86.3

Note: This table is a transcription of the original data from Table 2, not the original image.

*   <strong>Analysis:</strong> Many baseline models, when trained on reasoning tasks, suffer performance drops on other tasks (e.g., `MM-Eureka` drops by ~9% on `ChartQA`). In contrast, `Vision-Zero` models not only improve on reasoning but also maintain or even improve performance on chart, OCR, and vision-centric tasks. For example, the model trained on `Chart` data improves across all chart/OCR tasks while also improving on vision-centric tasks. This demonstrates the framework's ability to foster holistic capability growth.

• Low Dataset Construction Cost (Table 3):

  	Data Cost				Training		Performance
  	Data Type	Num	Prepare Method	Cost	Method	Interact	MMMu	MMMupro
  Qwen2.5-VL-7B							54.3	37.0
  R1-OneVision-7B		155k	Programmatic construction with human checks.	A few months	SFT+GRPO	X	51.9	32.6
  VLAA-Thinker-7B	Real-World Data	25k			SFT+GRPO	X	48.2	31.9
  OpenVLThinker-7B	12k	SFT+GRPO			X	54.8	22.1
  MM-Eureka-Qwen-7B		15k			GRPO	X	55.8	36.9
  ViGaL-Snake	Synthetic Data	72k	Collected in game environment via PPO policy	A few weeks	RLOO	X	55.8	36.6
  ViGaL-Rotation							54.1	37.7
  ViGaL-Snake+Rotation							58.0	37.4
  VisionZero-Qwen-7B (CLEVR)	Synthetic	2k	Batch render scenes	≈6 GPUh	Alternating Self-play+GRPO	✓	58.8	37.7

Note: This table is a transcription of the original data from Table 3, not the original image. Some layout details have been simplified for clarity.

*   <strong>Analysis:</strong> This table highlights the massive efficiency gain of `Vision-Zero`. While baseline methods require tens of thousands of samples and months of human effort, `Vision-Zero` achieves superior results using only 2,000 automatically generated image pairs, which took just 6 GPU hours to create. This makes it an extremely economical and scalable training paradigm.

• Qualitative Analysis of Reasoning (Image 5):

  该图像为示意图，展示了视觉语言模型在“谁是间谍”游戏中训练前后的对比。左侧为训练前，模型间通过描述位置关系推断间谍，存在暴露差异；右侧为训练后，模型学会避免揭示差异，聚焦共同特征，实现策略优化，提升了推理能力和隐秘性。图中包含玩家对话和训练过程的策略分析。

  • This figure provides a concrete example of how the model's reasoning improves. Before training, the Spy gives a very specific clue ("The green cylinder is behind the metallic sphere.") that risks revealing the difference. Its reasoning is shallow. After training, the Spy's reasoning is far more strategic. It analyzes the civilians' clues, identifies the differences, and chooses to describe a feature that is likely common to both images ("The metallic sphere is to the left of the yellow sphere."). This shows a marked improvement in planning, strategy, and logical deduction.

• Sustainable Performance Growth (Image 6):

  该图像包含三个折线图，展示了不同模型（Qwen2.5-VL-7B、InternVL3-8B、InternVL3-14B）在自我博弈训练中随迭代次数变化的性能指标：(a) 胜率随迭代上升，InternVL3-8B和InternVL3-14B表现优于Qwen2.5-VL-7B；(b) 线索（Clue）平均生成标记长度随迭代增加，InternVL3-14B最高；(c) 决策（Decision）平均生成标记长度也随迭代增长，InternVL3-14B同样领先。整体显示较大模型在策略复杂度和表现上有提升趋势。

  • These plots show that over 100 training iterations, the models consistently improve.
    • (a) Winning Rate: The models' win rate against a fixed, untrained opponent steadily increases, showing they are learning effective strategies.
    • (b) & (c) Average Token Length: The length of the generated text in both the Clue and Decision stages increases. This suggests the model is developing more complex and detailed reasoning chains ("thinking out loud") to arrive at its conclusions, a sign of more sophisticated cognitive processing.

6.2 Ablation Studies

• Model Generalizability (Table 4):

  Model	MathVista	MathVision	WeMath	MathVerse	LogicVista	DynaMath	Avg.
  Performance on InternVL3-8B
  InternVL3-8B	60.4	21.3	26.8	32.2	40.5	26.8	34.7
  VisionZero-InternVL3-8B	62.2	24.2	28.7	32.9	41.8	29.2	36.5
  Performance on InternVL3-14B
  InternVL3-14B	74.1	33.8	42.3	43.3	51.6	30.1	45.8
  VisionZero-InternVL3-14B	75.4	34.8	44.9	45.1	53.1	31.3	47.4

Note: This table is a transcription of the original data from Table 4, not the original image.

*   <strong>Analysis:</strong> `Vision-Zero` is not specific to one model architecture. When applied to the `InternVL3` family of models, it again yields consistent performance improvements (1.8% average gain for 8B, 1.6% for 14B), proving its general applicability.

• Superiority of Iterative-SPO (Image 7):

  该图像包含两个折线图。左图显示随着训练迭代次数（Iteration）增加，三种策略（Alternately、Pure Decision、Pure Clue）的胜率（Win Rate）变化趋势，显示Alternately策略胜率最高。右图展示相同迭代过程中三种策略的准确率（Accuracy）变化，Alternately策略准确率优于其他两种。整体反映不同训练策略在模型性能上的表现差异。

  • This is a crucial ablation that validates the novel training algorithm.
    • Pure Clue (only self-play) shows the slowest improvement and plateaus early. This is because without an effective decision-maker, the reward signal for giving good clues is weak and noisy.
    • Pure Decision (only RLVR) improves but also plateaus, as it can only get as good as the clues it is given. It cannot push the clue-givers to become more strategic.
    • Alternately (Iterative-SPO) consistently outperforms both. By alternating, it creates a co-evolutionary dynamic where a better decision-maker forces the clue-givers to be more clever, and more clever clue-givers provide a richer training signal for the decision-maker. This leads to the highest win rates and accuracy.

7. Conclusion & Reflections

• Conclusion Summary: The paper introduces Vision-Zero, a pioneering self-improvement framework for VLMs that operates with zero human supervision. By leveraging a strategic "Who Is the Spy?" game generated from domain-agnostic image pairs and a novel Iterative-SPO algorithm, it successfully overcomes the data cost and knowledge ceiling limitations of traditional training paradigms. The method is shown to be highly cost-effective, generalizable across different model architectures, and capable of producing SOTA results on a wide array of reasoning and vision tasks.

• Limitations & Future Work:

  • The paper does not extensively discuss limitations. However, one could infer some potential areas for future work:
  • Game Complexity: The current game involves simple clues and a single vote. More complex social deduction games (e.g., with multiple rounds of debate, special roles, or deception over longer horizons) could foster even more advanced reasoning skills.
  • Scalability of Roles: The experiments use a small number of players (e.g., 5). Scaling the game to many more players could introduce new challenges in communication and coordination.
  • Generation of Image Pairs: While the framework is domain-agnostic, the quality of the learned skills may depend on the nature of the differences in the image pairs. Future work could explore how to automatically generate maximally informative or challenging image pairs.

• Personal Insights & Critique:

  • Vision-Zero represents a significant conceptual leap for VLM training. The shift from data consumption to data generation via self-play is a powerful and sustainable direction for scaling AI capabilities. The idea is elegant and effectively turns the model's own growing intelligence into its primary training resource.
  • The Iterative-SPO algorithm is a clever solution to the stability and plateauing problems often seen in multi-agent RL and self-play. The dynamic switching mechanism is a practical and well-justified engineering choice.
  • The most impressive aspect is the strong out-of-domain generalization. That a model playing a simple visual discrimination game can improve its performance on formal mathematics problems is a compelling demonstration that the framework is teaching a fundamental, transferable form of logical and comparative reasoning.
  • This work opens up exciting possibilities for creating highly specialized VLMs at low cost. For example, one could train a VLM for medical image analysis by feeding it pairs of medical scans with subtle pathological differences, allowing it to learn fine-grained discrimination skills without needing massive, expert-annotated datasets. Vision-Zero could be a key enabler for democratizing the development of high-performance, specialized AI.