1. Bibliographic Information

• Title: LMAgent: A Large-scale Multimodal Agents Society for Multi-user Simulation
• Authors: Yijun Liu, Wu Liu, Xiaoyan Gu, Xiaodong He, Yong Rui, and Yongdong Zhang. The authors are affiliated with prominent institutions in China, including the University of Science and Technology of China, the Chinese Academy of Sciences, JD AI Research, and Lenovo Research. Their backgrounds span academia and industry, with expertise in information science, AI, and e-commerce.
• Journal/Conference: The paper is available on arXiv, a repository for electronic preprints of scientific papers.
• Publication Year: 2024 (Preprint submitted in December).
• Abstract: The paper addresses the challenge of creating believable simulations of multi-user behavior, which is crucial for understanding complex social systems. While Large Language Model (LLM)-based AI agents have shown promise, they often lack the scale and multimodal interaction capabilities of real human societies. The authors introduce LMAgent, a large-scale (over 10,000 agents) multimodal agent society, using e-commerce as a case study. Agents in LMAgent can chat, browse, purchase, review products, and even participate in live streaming e-commerce. To achieve this, the paper proposes two key technical innovations: (1) a self-consistency prompting mechanism to improve agents' multimodal decision-making, and (2) a fast memory mechanism combined with a small-world network model to enhance system efficiency for large-scale simulations. Experiments show that the agents' behavior is comparable to humans and that the system can exhibit emergent social phenomena like herd behavior, demonstrating its potential for credible social simulation.
• Original Source Link:

  • arXiv Page: https://arxiv.org/abs/2412.09237

  • PDF Link: https://arxiv.org/pdf/2412.09237v2.pdf

  • Publication Status: This is a preprint and has not yet undergone formal peer review for a conference or journal publication.

    ---

2. Executive Summary

• Background & Motivation (Why):

  • Core Problem: Simulating complex human social behavior realistically is a long-standing goal in AI. Such simulations can help us understand and predict phenomena in areas like economics, sociology, and online user dynamics.
  • Existing Gaps: Recent advancements use Large Language Models (LLMs) to create AI agents with human-like intelligence. However, existing systems suffer from two major limitations:
    1. Limited Modality: Most systems are text-only, ignoring the rich multimodal (text, image, video) nature of human interactions, especially in online environments like e-commerce.
    2. Limited Scale: They typically simulate only a small number of agents (dozens or hundreds), which is insufficient to capture complex, large-scale social dynamics that emerge from the interactions of thousands or millions of individuals.
  • Fresh Angle: LMAgent aims to bridge these gaps by creating a simulation environment that is both very large-scale (over 10,000 agents) and multimodal. It focuses on the e-commerce domain, a rich setting for studying complex user behavior involving social influence, visual product assessment, and purchasing decisions.

• Main Contributions / Findings (What):

  • A Large-Scale Multimodal Agent Society (LMAgent): The paper presents a system capable of simulating over 10,000 agents that interact using both text and images. This is a significant increase in scale compared to prior work.

  • Self-Consistency Prompting Mechanism: To improve agents' decision-making with multimodal information (e.g., product images and descriptions), this novel prompting technique breaks down the decision process into steps. It first reasons about the agent's internal state (persona, preferences) and then combines this with external environmental information, leading to more consistent and believable actions.

  • Efficient System Architecture: To make a 10,000-agent simulation feasible, the paper introduces two efficiency-boosting techniques:

    • A fast memory mechanism that caches common, simple behaviors to reduce expensive calls to the LLM, improving efficiency by approximately 40%.
    • A small-world network model to structure the agents' social relationships, mimicking real-world social networks and enabling efficient information spread.

  • Realistic Behavioral Simulation: Experiments demonstrate that LMAgent agents achieve purchase behavior accuracy comparable to real humans and significantly better than existing systems. The system also replicates complex social phenomena, such as herd behavior (agents converging on popular products) and realistic co-purchase patterns, validating its potential as a credible tool for social science research.

    ---

3. Prerequisite Knowledge & Related Work

• Foundational Concepts:

  • AI Agent: An artificial entity that can perceive its environment, make independent decisions, and take actions to achieve its goals. In this paper, agents are designed to mimic human consumers.
  • Multi-Agent System (MAS): A system composed of multiple interacting AI agents. MAS are used to study how individual behaviors lead to collective, system-level phenomena.
  • Large Language Model (LLM): A type of AI, like OpenAI's GPT series, trained on vast amounts of text data. LLMs excel at understanding context, generating human-like text, reasoning, and planning, making them ideal "brains" for AI agents. A Multimodal LLM (MLLM) is an extension that can process information from multiple modalities, such as text and images, simultaneously.
  • Small-World Network: A type of mathematical graph that has properties found in many real-world social networks. Key features include high clustering (friends of a person are likely to be friends with each other) and a short average path length (any two people are connected by a short chain of acquaintances), famously known as the "six degrees of separation" concept. This structure is more realistic than simple random or regular grid networks.
  • Herd Behavior: A social phenomenon where individuals in a group act collectively without centralized direction. In a consumer context, it refers to people buying a product primarily because many others are also buying it.
  • Chain-of-Thought (CoT) Prompting: A technique used to improve the reasoning ability of LLMs. Instead of asking for a direct answer, the LLM is prompted to first generate a series of intermediate reasoning steps, which helps guide it to a more accurate final conclusion. The paper's self-consistency prompting is inspired by this idea.

• Previous Works:

  • Rule-Based and Reinforcement Learning Agents: Early agents were often built with hand-crafted rules (finite-state machines) or trained with reinforcement learning (e.g., RecSim, AlphaStar). These methods are effective when goals and rewards are clearly defined but struggle in open-ended social simulations where rewards are ambiguous.
  • LLM-based Agent Systems: Recent systems leverage LLMs as the core decision-making engine.
    • GenerativeAgent (Park et al., 2023): A landmark paper that created a small sandbox environment ("Smallville") with 25 agents who planned their days, interacted, and formed memories. It demonstrated the potential for believable, emergent social behaviors but was small-scale and text-only.
    • AgentVerse and ChatDEV: These systems use LLM agents for collaborative tasks like software development, but not for large-scale social simulation.
    • RecAgent: An agent-based recommendation system that simulates user behavior, but on a smaller scale and with a primary focus on recommendation algorithms rather than emergent social phenomena.

• Differentiation: LMAgent differentiates itself from prior work in three key ways:

  1. Scale: It simulates over 10,000 agents, an order of magnitude larger than most previous LLM-based agent societies.

  2. Modality: It is inherently multimodal, allowing agents to process and react to both text and images, which is crucial for realism in e-commerce.

  3. Efficiency: It introduces specific mechanisms (fast memory, small-world network) to make such a large-scale simulation computationally tractable.

     ---

4. Methodology (Core Technology & Implementation)

The core of LMAgent is its agent architecture and the sandbox environment they inhabit. The paper uses e-commerce consumers as a concrete example.

Image 2: This diagram illustrates the overall architecture of LMAgent. (a) shows a single agent interacting with the multimodal environment (containing products with images and text), driven by an MLLM. Key mechanisms like self-consistency prompting and the small-world social network are highlighted. (b) provides a detailed view of the agent's fast memory system, which includes sensor memory, a memory bank for efficiency, and short-term and long-term memory modules that evolve over time.

• A. Multimodal Agent Architecture: Each agent in LMAgent has both internal and external behaviors, all powered by a multimodal LLM.

• B. Internal Behavior: These are the cognitive processes of an agent.

  1. Persona: Each agent is given a unique identity defined by attributes like name, gender, age, occupation, personal traits, and purchasing preferences. These are generated by an LLM to ensure diversity and realism.
  2. Fast Memory: To handle the massive amount of information in a large-scale simulation efficiently, LMAgent uses a sophisticated memory system inspired by cognitive neuroscience.
     • Sensor Memory: Processes raw observations ($o_i$) at a given time, like seeing a product. It immediately forgets details, but key information is compressed into a concise sentence ($c_i^s$). $$c _ { i } ^ { s } = f _ { c } ( o _ { i } )$$
       • $c_i^s$: The compressed, concise sentence summarizing the observation.
       • $o_i$: The observation, which can be text and/or images.
       • $f_c$: A prompt function that instructs the LLM to perform this compression.
     • Short-Term Memory: Stores these compressed observations as formatted records: $m_i^s = <c_i^s, e_i, I_i, t_i>$, where $e_i$ is an embedding (a vector representation) of the memory, $t_i$ is the timestamp, and $I_i$ is an importance score. The importance is rated by the LLM: $$I _ { i } = f _ { r } ( c _ { i } ^ { s } )$$
       • $I_i$: The importance score of the memory.
       • $f_r$: A prompt function that asks the LLM to rate the memory's significance.
     • Long-Term Memory: When a memory is deemed important (e.g., similar memories appear multiple times), it is moved to long-term memory. These memories can be forgotten over time based on their age and importance, using the following formula: $$f \left( m _ { i } ^ { l } \right) = 1 - \frac { \hat { t } _ { i } + I _ { i } } 2 * \operatorname* { m a x } \left( I _ { i } ^ { \beta } , \delta \right)$$
       • $f(m_i^l)$: The probability of forgetting a long-term memory $m_i^l$. A higher value means more likely to be forgotten.
       • $\hat{t}_i$: Recency score (normalized from 0 for oldest to 1 for newest).
       • $I_i$: Importance score of the memory.
       • $\beta, \delta$: Hyperparameters that control the forgetting curve. This formula ensures that older, less important memories are more likely to be forgotten.
     • Memory Bank: This is the key innovation for efficiency. Many agent actions are simple and repetitive (e.g., "enter shopping mall"). Instead of calling the LLM every time for these basic behaviors, the system pre-computes and caches their importance scores and embeddings in a memory bank. When an agent performs a basic action, this information is retrieved directly, saving LLM calls. The paper states this accounts for over 60% of actions and improves efficiency by about 40%.
  3. Planning and Reflection: Agents periodically perform high-level thinking. Planning involves setting long-term goals. Reflection involves reviewing past memories to derive higher-level insights (e.g., "I seem to be buying a lot of electronics lately"), which are then stored in long-term memory to guide future behavior.

• C. External Behavior: These are the observable actions an agent can take.

  1. Shopping and Social Interaction:
     • Shopping Actions: Browsing, Searching, Paging, Viewing Details, Purchasing.
     • Social Actions: Chatting with friends, Posting messages to all friends, and Live streaming (for select "superstar" agents).
  2. Self-consistency Prompting: This mechanism improves decision-making in complex, multimodal scenarios. It works in two stages:
     • Stage 1 (Internal Focus): The LLM first generates a summary ($\mathcal{P}_1$) based on the agent's persona ($\mathcal{C}_i$) and its last observation ($o_i$). This step forces the model to consider "who" the agent is before deciding "what" to do. $$\mathcal { P } _ { 1 } = f _ { s } ( \mathcal { C } _ { i } , o _ { i } )$$
       • $\mathcal{P}_1$: A summary of the agent's internal state and perspective.
       • $f_s$: A prompt function guiding the LLM to create this summary.
     • Stage 2 (External Focus): The summary $\mathcal{P}_1$ is combined with external environmental information $\mathcal{E}$ (e.g., product images and descriptions) to make the final action decision ($a$). $$a = f _ { e } ( \mathcal { P } _ { 1 } , \mathcal { E } )$$
       • $a$: The agent's next action.
       • $f_e$: A prompt function guiding the LLM to make a final decision. By decoupling the task, the LLM can make more credible and self-consistent choices.

• D. Sandbox Environment:

  1. Small-world Topology Networks: To create a realistic social structure, the agents' relationships are initialized using a small-world network model.

     

     Image 3: This figure compares different network structures. (a) a Regular Network is highly ordered, (b) a Random Network is chaotic, and (c) a Small-world Network balances order and randomness, featuring local clusters and long-range connections, much like real social networks.

     The construction follows Algorithm 1:

     • Step 1: Arrange $N$ agents in a ring.
     • Step 2: Connect each agent to its $k$ nearest neighbors. This creates high local clustering.
     • Step 3: For each edge, with a small probability $p$, "rewire" it to a random agent in the network. This introduces "shortcuts" that drastically reduce the average path length. This approach is computationally efficient (linear time complexity $O(kN)$) and creates a more realistic social fabric for the simulation.

  2. Multi-user Simulator: As outlined in Algorithm 2, the simulation proceeds in discrete time steps. In each step, every agent:

     • Performs planning and reflection if needed.
     • Selects and executes its next action using the self-consistency prompting mechanism.
     • Updates its memory using the fast memory mechanism. All actions are logged for later analysis.

5. Experimental Setup

• Datasets:

  • Amazon Review Dataset: A massive dataset containing 233.1 million purchases and reviews from over 20 million users. It includes product metadata like names, prices, and images, and is used to initialize the virtual shopping environment and user purchase histories.
  • JD User Behavior Data: Real-world user co-purchase data from JD.com (a major Chinese e-commerce platform) is used as a ground truth to validate the co-purchase patterns generated by the simulation.

• Evaluation Metrics:

  • Purchase Accuracy ($p$): Measures how accurately the agent's simulated purchases match a user's real, held-out purchase history.
    • Conceptual Definition: This metric calculates the percentage of correctly "predicted" items. An agent is shown a list of products containing some items the real user actually bought (ground truth) and some random distractor items. The metric measures how many of the ground truth items the agent chooses to "buy".
    • Mathematical Formula: $$p = \sum _ { u \in U } \frac { | T _ { u } \cap S _ { u } | } { | T _ { u } | } \times 1 0 0 \%$$
    • Symbol Explanation:
      • $U$: The set of all simulated users.
      • $T_u$: The set of ground truth items that user $u$ actually purchased.
      • $S_u$: The set of items the agent simulating user $u$ chose to purchase.
      • $|T_u \cap S_u|$: The number of items correctly predicted (the intersection of the two sets).
      • $|T_u|$: The total number of ground truth items. The metric is evaluated in different settings, denoted as $a @ (a+b)$, where $a = |T_u|$ and $a+b$ is the total number of items in the recommendation list.
  • Qualitative Behavioral Metrics: Human annotators and GPT-4 scored agent and human behaviors on a 1-5 scale across several dimensions: Believability, Knowledge, Personalization, Social Norms, Social Influence, Naturalness, and Expressiveness.
  • Pointwise Mutual Information (PMI): Used to measure the association between pairs of products in co-purchase data.
    • Conceptual Definition: PMI compares the probability of two items appearing together versus the probability of them appearing independently. A high PMI means the items are purchased together more often than by chance.
    • Mathematical Formula: $$\mathrm{PMI}(x, y) = \log_2 \frac{P(x, y)}{P(x)P(y)}$$
    • Symbol Explanation:
      • P(x, y): The probability that a user purchases both item $x$ and item $y$.
      • P(x): The probability that a user purchases item $x$.
      • P(y): The probability that a user purchases item $y$.
  • Randolph's Kappa ($\kappa$): A statistic used to measure the agreement between multiple annotators when rating qualitative data. A score of 0.573 indicates "moderate" agreement.

• Baselines: The paper compares LMAgent against several models for the user purchase behavior task:

  • Random: Randomly selects items to purchase.

  • Embedding [37]: A model that uses product embeddings to recommend similar items.

  • Collaborative Filtering [38]: A classic recommendation technique that finds users with similar tastes.

  • Recsim [22]: A reinforcement learning-based simulation environment for recommendations.

  • RecAgent [6]: A more recent LLM-based agent system for recommendation.

    ---

6. Results & Analysis

• Core Results: User Purchase Behavior Evaluation

  This is a manual transcription of Table I from the paper. TABLE I: THE RESULTS OF DIFFERENT MODELS ON USER PURCHASE SIMULATION

  Model	1@6	1@10	3@6	3@10	AVG
  Random	16.00	11.20	51.07	28.67	26.74
  Embedding [37]	37.60	23.20	65.47	48.53	43.70
  Collaborative Filtering [38]	52.80	32.40	67.87	52.67	51.44
  Recsim [22]	48.40	43.60	75.33	57.73	56.27
  RecAgent [6]	52.40	46.00	73.87	61.47	58.44
  LMAgent	70.40	63.60	82.67	75.47	73.04

  Analysis: LMAgent dramatically outperforms all baseline models across all settings. Its average accuracy of 73.04 is a ~14.6 point improvement over the next best model, RecAgent. The improvement is most significant in the harder "1-out-of-N" tasks (1@6 and 1@10), where LMAgent is over 32% better on average. This strongly suggests that its multimodal capabilities and self-consistency prompting are highly effective for simulating realistic purchase decisions.

• Agent Behavior Analysis

  This is a manual transcription of Table II from the paper. TABLE II: EVALUATION OF BEHAVIOR CHAINS OF AGENTS AND HUMANS

  Dim	Random		LMAgent		Human
  	H	G	H	G	H	G
  Believability	2.70	3.17	4.24	3.72	4.80	3.33
  Knowledge	2.75	3.22	4.05	3.89	4.20	2.83
  Personalization	2.68	4.10	4.20	4.46	4.53	3.77
  Social Norms	4.33	3.10	4.59	3.64	4.87	3.53
  Social Influence	2.93	3.83	4.43	4.11	4.60	3.67
  Average	3.08	3.48	4.30	3.96	4.60	3.43

  Analysis: In human evaluations (H), LMAgent's behavior chains are rated very close to actual human behavior (average score of 4.30 vs. 4.60), and far superior to a random agent. Interestingly, GPT-4 (G) rates LMAgent higher than humans, which the authors attribute to LLM self-bias (favoring outputs similar to its own).

  This is a manual transcription of Table III from the paper. TABLE III: EVALUATION OF BEHAVIOR CONTENT FOR AGENTS AND HUMANS

  Dim	LMAgent		Human
  	H	G	H	G
  Naturalness	4.45	4.90	4.53	3.33
  Expressiveness	4.49	4.04	4.50	3.27
  Average	4.47	4.47	4.52	3.30

  Analysis: The social content (chats, posts) generated by LMAgent is almost indistinguishable from human-generated content in terms of naturalness and expressiveness, with an average human rating of 4.47 versus 4.52 for real humans.

• Social Influence Analysis

  This is a manual transcription of Table IV from the paper. TABLE IV: EVALUATION OF SIMULATED USER PURCHASE BEHAVIOR UNDER VARYING KINDS OF SOCIAL INFLUENCE

  Influence	1@6	3@6	Average
  None	70.40	82.67	76.54
  Negative	32.80	37.33	35.17 ($\downarrow$41.37)
  Positive	78.00	88.40	83.17 ($\uparrow$6.63)
  Positive (live-stream)	80.00	86.67	83.33 ($\uparrow$6.79)

  Analysis: Social influence has a strong and realistic effect on agent behavior. Negative information drastically reduces purchase probability (down 41.37%), while positive information from friends or a live-streamer provides a significant boost (up ~6.7%). This demonstrates that the agents are not just making decisions in a vacuum but are responsive to their social context.

• Ablation Study

  

  Image 4: This figure demonstrates the efficiency gain from the fast memory mechanism. The line chart on the left shows that total token consumption (a proxy for computational cost) with fast memory is consistently about 40% lower than without it. The pie charts on the right confirm that the distribution of tokens between input and output remains similar.

  This is a manual transcription of Table V from the paper. TABLE V: RESULTS OF THE ABLATION STUDIES

  Fast Memory	Multimodal	SCP	1@6	3@6	Average
  -	✓	✓	65.30	79.23	72.27
  ✓	-	-	66.10	77.87	71.99 ($\downarrow$0.28)
  ✓	✓	-	68.20	81.27	74.74 ($\uparrow$2.47)
  ✓	✓	✓	70.40	82.67	76.54 ($\uparrow$4.27)

  Analysis:

  • Fast Memory: Using fast memory results in a negligible performance drop of only 0.28%, while Figure 4 shows it reduces cost by ~40%. This confirms it is a highly effective optimization.
  • Multimodality: Adding multimodal information (images) improves performance by 2.47% over a text-only version.
  • Self-Consistency Prompting (SCP): Adding SCP on top of multimodality provides a further boost of 1.8% (from 74.74 to 76.54), for a total improvement of 4.27% over the text-only baseline. This confirms that both multimodality and the specialized prompting are critical components.

• Large-scale Consumer Simulation Analysis

  

  Image 5: This figure presents results from the large-scale simulation. (a) and (b) are PMI heatmaps showing the co-purchase correlations for real JD.com users and LMAgent, respectively. The patterns are strikingly similar (e.g., high correlation within video games, negative correlation between industrial supplies and art). (c) is a line chart showing that as the number of agents increases, the purchase share of the top-ranked product (Top-1) grows significantly, demonstrating emergent herd behavior.

  

  Image 6: This figure analyzes different network topologies. The line chart on the left shows that the small-world network has a fast information dissemination rate, close to a random network initially. The bar charts on the right confirm that the small-world network has a high clustering coefficient (like a regular network) and a low average path length (like a random network), which are hallmarks of real-world social networks.

  Analysis:

  • Realistic Group Behavior: The co-purchase patterns generated by 10,000 agents closely mirror real-world data (Figure 5), validating the simulation's authenticity.

  • Emergent Herd Behavior: As the agent population grows, their purchasing behavior becomes more concentrated on a few popular items. This is not explicitly programmed but emerges naturally from social interactions, demonstrating the power of large-scale simulation.

  • Network Topology: The small-world network is shown to be the best choice, as it balances realistic social structure (high clustering) with efficient information flow (low path length), as seen in Figure 6.

    ---

7. Conclusion & Reflections

• Conclusion Summary: The paper successfully introduces LMAgent, a groundbreaking framework for creating very large-scale (10,000+ agents) and multimodal agent-based simulations. Through novel techniques like self-consistency prompting and a fast memory mechanism, it achieves a new level of realism and scale in simulating human social behavior, particularly in the complex domain of e-commerce. The system not only reproduces individual behaviors with high fidelity but also captures emergent, large-scale social phenomena like herd behavior, demonstrating its significant potential as a tool for research in social sciences, economics, and AI.

• Limitations & Future Work:

  • Domain Specificity: While demonstrated in e-commerce, the framework's adaptability to other complex domains (e.g., finance, urban planning, political opinion dynamics) is yet to be proven, though the authors suggest it is versatile.
  • Cost and Accessibility: The simulation relies on powerful, proprietary MLLMs like GPT-4, making it computationally expensive and potentially inaccessible for researchers without significant funding or API access.
  • Depth of Sociality: The social interactions are still relatively simple (chat, post, live stream). Real human social dynamics involve more nuanced relationships, trust-building, and long-term memory effects that could be explored further.
  • Future Work: The authors conclude by stating that as LLMs continue to improve, future versions of this work could create even more realistic simulations, further advancing the field of computational social science.

• Personal Insights & Critique:

  • LMAgent represents a significant engineering and conceptual leap in agent-based modeling. Moving from dozens of text-only agents to 10,000 multimodal agents is a non-trivial achievement, and the efficiency mechanisms proposed are practical and impactful.
  • The concept of self-consistency prompting is a clever and transferable idea. It addresses a common weakness in LLMs—making decisions that are inconsistent with a given persona—by explicitly breaking the reasoning process into "self-reflection" and "action" stages.
  • Ethical Considerations: The ability to create such believable simulations of human societies raises important ethical questions. These "digital twins" of society could be used for manipulative purposes, such as testing viral marketing strategies or political propaganda, without real-world consequences for the simulator. As these tools become more powerful, a framework for their ethical use will be essential.
  • The paper's validation against real-world data (JD.com co-purchase patterns) is a major strength, lending significant credibility to the simulation's outputs. It moves beyond simply being a "cool demo" and shows potential as a scientifically valid instrument. LMAgent is a powerful demonstration of the potential for LLMs to not just mimic individual intelligence, but to simulate the complex, emergent intelligence of an entire society.