1. Bibliographic Information

1.1. Title

EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Models

1.2. Authors

The paper is authored by Li Zhou, Lutong Yu, You Lv, Yihang Lin, Zefeng Zhao, Junyi Ao, Yuhao Zhang, Benyou Wang, and Haizhou Li. All authors are affiliated with The Chinese University of Hong Kong, Shenzhen, and Haizhou Li is also affiliated with the Shenzhen Research Institute of Big Data. Their research backgrounds generally lie in speech processing, natural language understanding, and artificial intelligence, with a focus on areas like Speech Language Models (SLMs) and empathetic AI.

1.3. Journal/Conference

The paper is published at (or intended for) "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" as indicated by several references (e.g., [2], [16], [33], [37], [56]) within the paper's bibliography, dated July 2025 in Vienna, Austria. This suggests it is a publication for a prominent conference in computational linguistics.

1.4. Publication Year

2025

1.5. Abstract

Speech Language Models (SLMs) have made significant advancements in spoken language understanding. However, their ability to fully perceive non-lexical vocal cues and respond empathetically, considering both emotional and contextual factors, remains largely unexplored. Existing benchmarks typically evaluate individual capabilities like linguistic, acoustic, reasoning, or dialogue abilities in isolation, failing to integrate these crucial skills for emotionally intelligent conversation. This paper introduces EchoMind, the first interrelated, multi-level benchmark designed to simulate the cognitive process of empathetic dialogue. It achieves this through sequential, context-linked tasks: spoken-content understanding, vocal-cue perception, integrated reasoning, and response generation. A unique feature of EchoMind is its use of identical, semantically neutral scripts with controlled variations in vocal style, allowing for the isolation and study of delivery effects independent of the transcript. The benchmark is structured around an empathy-oriented framework comprising 3 coarse and 12 fine-grained dimensions, encompassing 39 vocal attributes. Evaluation utilizes both objective and subjective metrics. Testing 12 advanced SLMs revealed that even state-of-the-art models struggle with highly expressive vocal cues, which in turn limits the quality of their empathetic responses. Further analyses on prompt strength, speech source, and ideal vocal cue recognition exposed persistent weaknesses in instruction-following, resilience to natural speech variability, and effective utilization of vocal cues for empathy. The findings emphasize the critical need for SLMs to integrate linguistic content with diverse vocal cues to achieve truly empathetic conversational abilities.

1.6. Original Source Link

Official Source: https://arxiv.org/abs/2510.22758 PDF Link: https://arxiv.org/pdf/2510.22758.pdf Publication Status: Preprint (as it's on arXiv, and the publication date is in the future relative to the current time, indicating it's a submission or accepted paper for 2025).

2. Executive Summary

2.1. Background & Motivation

2.1.1. Core Problem

The core problem the paper addresses is the inability of current Speech Language Models (SLMs) to fully perceive and integrate non-lexical vocal cues (like prosody, emotion, physiological signals, and environmental sounds) alongside spoken words to generate truly empathetic responses. While SLMs have advanced spoken language understanding, their capacity for emotionally intelligent conversation, which requires understanding how something is said and under what circumstances, is limited.

2.1.2. Importance and Gaps in Prior Research

Empathetic understanding and response generation are crucial for natural, trustworthy, and human-like human-computer interaction, intelligent assistants, and empathetic companions. Prior research and existing benchmarks typically evaluate SLM capabilities (linguistic, acoustic, reasoning, dialogue) in isolation. They often rely on repurposing existing corpora or narrowly targeted datasets, which lack shared contextual grounding across tasks. This isolated evaluation fails to capture the intricate interplay of understanding, reasoning, and response generation necessary for empathetic dialogue, particularly the crucial role of non-lexical vocal cues.

2.1.3. Paper's Entry Point and Innovative Idea

The paper's innovative idea is to create EchoMind, the first interrelated, multi-level benchmark that explicitly simulates the cognitive process of empathetic dialogue. It uniquely focuses on isolating the impact of vocal delivery by using semantically neutral scripts with controlled vocal-style variations. This allows for a systematic evaluation of how SLMs perceive, reason about, and respond to non-lexical acoustic cues, thereby filling the gap left by benchmarks that treat these capabilities in isolation.

2.2. Main Contributions / Findings

2.2.1. Primary Contributions

The paper makes four primary contributions:

1. Empathy-Oriented Evaluation Framework: It proposes a novel framework spanning 3 coarse and 12 fine-grained dimensions, covering 39 vocal attributes. It also constructs high-quality dialogue scripts with controlled vocal-style variations to isolate the impact of vocal expression.
2. Multi-Level, Interrelated Tasks: It designs multi-level tasks (understanding, reasoning, conversation) aligned with empathetic cognition, each with dedicated quantitative and qualitative evaluation, including joint assessment of textual and acoustic expressiveness in open-ended conversation. These tasks are grounded in the same set of audio instances, ensuring contextual consistency.
3. Benchmarking of Advanced SLMs: It benchmarks 12 advanced SLMs on EchoMind, revealing that even state-of-the-art models struggle significantly with highly expressive vocal cues, which in turn limits the quality of their empathetic responses.
4. In-depth Behavioral Analyses: It conducts detailed behavioral analyses of SLMs, investigating prompt sensitivity, performance gaps between synthetic and human speech, and the upper-bound empathetic response capability under ideal vocal-cue recognition. These analyses uncover persistent weaknesses in instruction-following, resilience to natural speech variability, and effective utilization of vocal cues for empathy.

2.2.2. Key Conclusions and Findings

The key conclusions and findings include:

• State-of-the-art SLMs exhibit strong performance in content understanding (ASR) but show considerable variability and struggle with voice-related information in both understanding and reasoning tasks.

• The closed-source GPT-4o-Audio generally outperforms open-source models.

• While many SLMs can generate contextually appropriate, natural, and colloquial responses, they are significantly limited in leveraging vocal cues to produce truly empathetic replies (C4 and VES scores are low across models).

• There is a general positive correlation between stronger voice understanding and reasoning ability, and conversational response quality (particularly voice-cue-oriented dimensions). However, some models are outliers, suggesting issues beyond mere perception, such as instruction-following.

• SLMs show sensitivity to prompting strategies, with some performing better with explicit instructions to consider vocal cues, while others perform best with no prompt at all, indicating limited instruction-following capabilities.

• Human-recorded speech poses greater challenges than TTS-generated data across all evaluation levels, particularly in conversation, due to higher acoustic variability and prosodic nuance. This highlights a need for greater robustness to real-world speech.

• When models are provided with ideal vocal-cue information, their empathetic response quality (C1, C4, VES) improves, indicating a potential ceiling for empathetic responses if vocal cues are perfectly recognized, but also underscoring that current models often fail to adequately recognize these cues themselves.

  These findings collectively underscore that current SLMs lack the comprehensive integration of linguistic content with diverse vocal cues necessary for truly empathetic conversational ability, highlighting crucial areas for future research and development.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand EchoMind, a reader needs to be familiar with several core concepts in Speech Language Models (SLMs) and Artificial Intelligence (AI):

• Speech Language Models (SLMs): These are AI models that process and generate human speech. They go beyond simple Automatic Speech Recognition (ASR) or Text-to-Speech (TTS) by integrating language understanding and generation capabilities directly with speech. The paper discusses their evolution from cascade pipelines (separate ASR, LLM, TTS modules) to unified end-to-end architectures that directly map speech input to speech output, aiming for more natural and coherent speech interactions.
• Empathetic AI: This refers to AI systems designed to understand and respond to human emotions and states in a way that is perceived as empathetic. In the context of EchoMind, it means not just understanding the literal words but also the underlying emotional and contextual factors conveyed through vocal cues, and then generating responses that acknowledge and align with these.
• Non-Lexical Vocal Cues (Paralinguistic Features): These are aspects of speech that are not related to the literal meaning of words but convey significant information about a speaker's state, emotion, or context. Examples include:
  • Prosody: The rhythm, stress, and intonation of speech. This includes volume, speech rate (pace), and pitch.
  • Emotion: The emotional state conveyed through voice (e.g., happiness, sadness, anger, fear).
  • Physiological Vocal Signals: Sounds related to the speaker's physical state (e.g., breathing, coughing, sobbing, vocal fatigue, hoarseness).
  • Non-Verbal Expressions (NVEs): Sounds like sighs, laughs, yawns, moans.
  • Environmental Sounds: Background noises that provide contextual information (e.g., wind, thunderstorm, sea beach, chatter, music).
• Automatic Speech Recognition (ASR): The process by which spoken language is converted into text. ASR is a fundamental component of SLMs and is evaluated in EchoMind for its ability to transcribe speech even under challenging conditions (expressive delivery, environmental noise).
• Large Language Models (LLMs): Powerful AI models trained on vast amounts of text data, capable of understanding, generating, and processing human language. They are often integrated into SLMs for their text understanding and generation capabilities. GPT-4o is a prominent LLM mentioned in the paper, used for script generation and as a Model-as-a-Judge.
• Text-to-Speech (TTS): The technology that converts written text into spoken audio. TTS systems are essential for SLMs to generate spoken responses. The paper uses Doubao TTS and GPT-4o-mini-TTS for audio synthesis.
• Model-as-a-Judge: An evaluation paradigm where a powerful LLM is used to assess the quality of responses generated by other models, often in subjective dimensions like naturalness, coherence, or empathy. EchoMind uses GPT-4o and Gemini-2.5-Pro as Model-as-a-Judge for subjective evaluations.

3.2. Previous Works

The paper discusses various categories of existing audio-based benchmarks for SLMs, highlighting their limitations before introducing EchoMind.

• Multi-task and Comprehensive Capability Benchmarks:

  • Examples: AudioBench [17], Dynamic-SUPERB [15, 41], AIR-Bench [42], MMSU [23], MMAU [22], MSU-Bench [45].
  • Focus: These benchmarks assess a wide range of abilities, including ASR, speaker identification, emotion classification, environmental sound recognition, and music understanding.
  • Limitation: While comprehensive in scope, they typically evaluate these abilities in isolation and often do not focus on empathetic dialogue or the integration of these skills for conversational responsiveness.

• Knowledge-Oriented QA Benchmarks:

  • Examples: VoxEval [16], Voicebench [21].
  • Focus: Question answering from spoken input, emphasizing factual knowledge.
  • Limitation: Offer limited assessment of paralinguistic or environmental information, which are crucial for empathetic interaction.

• Reasoning-Focused Benchmarks:

  • Examples: Audio Entailment [18], SAKURA [19], MMAR [44], SpeechR [43].
  • Focus: Deductive, multi-hop, or deep reasoning by combining linguistic content with specific acoustic features.
  • Limitation: While addressing reasoning, they may not fully integrate this with conversational response generation or broader empathetic considerations.

• Dialogue-Centered Benchmarks:

  • Examples: SD-Eval [11], VoxDialog [12], EChat-eval [14], URO-Bench [13], MTalk-bench [20].
  • Focus: Incorporate speaker, paralinguistic, and environmental cues into conversational contexts to approximate interactive use cases.
  • Limitation: Many still lack the interrelated, multi-level structure that simulates the cognitive process of empathetic dialogue from understanding to reasoning to generation, especially with controlled vocal variations to isolate cue impact. Some may focus on single expressive styles for the same script, unlike EchoMind's multiple expressive styles.

Core Concept: Attention Mechanism (Relevant to many SLMs and LLMs) Many of the SLMs and LLMs mentioned, such as GPT-4o, Qwen2.5-Omni-7B, and those in Audio Flamingo 3 series, heavily rely on the transformer architecture which uses the attention mechanism. While not explicitly detailed in the EchoMind paper, understanding attention is foundational to how these models process and integrate multimodal information.

The attention mechanism allows a model to weigh the importance of different parts of the input sequence when processing a specific part of the output. For multimodal models, this extends to weighing information from different modalities (e.g., text and audio features).

The core Self-Attention formula is: $ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $ Where:

• $Q$ (Query), $K$ (Key), $V$ (Value) are matrices representing different linear transformations of the input embeddings.

  • $Q$ allows the model to "query" for relevant information.
  • $K$ provides the "keys" against which queries are matched.
  • $V$ holds the actual "values" (information) to be extracted.

• $Q K^T$ calculates the dot product similarity between queries and keys, indicating how relevant each key is to each query.

• $\sqrt{d_k}$ is a scaling factor, where $d_k$ is the dimension of the keys. This scaling helps prevent the dot products from becoming too large, which can push the softmax function into regions with very small gradients.

• $\mathrm{softmax}(\cdot)$ normalizes the scores, turning them into attention weights that sum to 1, indicating the probability distribution over the input elements.

• The final multiplication by $V$ produces a weighted sum of the value vectors, where values corresponding to highly attended keys contribute more to the output.

  In SLMs, this mechanism can be extended to cross-modal attention, allowing text embeddings to attend to audio features, and vice versa, facilitating the integration of spoken content and vocal cues.

3.3. Technological Evolution

The field of SLMs has evolved from cascade pipelines to unified end-to-end architectures.

• Cascade Pipelines (Early SLMs): These systems typically consisted of separate modules:

  1. An ASR module to transcribe speech into text.
  2. A Large Language Model (LLM) to process the text and generate a textual response.
  3. A Text-to-Speech (TTS) module to synthesize the textual response into audio.
  • Limitation: While functional, cascade designs often lose vocal cue information between stages. For example, ASR primarily focuses on converting speech to text, often discarding prosodic or emotional cues that could be vital for an empathetic LLM response or a nuanced TTS output. The recognition and reasoning steps are separate from synthesis, limiting how vocal cues can inform conversational planning and emotional expressiveness.

• End-to-End Architectures (Current SLMs): These models integrate speech understanding and generation within a single framework. They aim to directly map speech input to speech output.

  • Approaches:
    • Serial text-then-speech token generation: Models first generate textual tokens, then speech tokens, but within a more integrated architecture.
    • Parallel token decoding: More advanced models generate text and speech tokens in parallel or with sophisticated cross-modal alignment to reduce latency and preserve semantic-prosodic coherence.
  • Advancements: These systems employ advanced audio tokenization, cross-modal alignment, and streaming/full-duplex decoding to support timbre control, emotional expressiveness, and real-time interaction.
  • Gap: Despite these advancements, the EchoMind paper argues that even these end-to-end models still struggle to effectively perceive and integrate non-lexical vocal cues for truly empathetic responses, which is the specific gap EchoMind aims to address.

3.4. Differentiation Analysis

Compared to the existing audio-based benchmarks summarized in Table 1, EchoMind introduces several core innovations:

• Interrelated and Multi-Level Task Structure (Corr. column in Table 1): Unlike many benchmarks that evaluate capabilities in isolation ($Corr. = X$), EchoMind's tasks (understanding, reasoning, conversation) are context-linked and sequential, mirroring the human cognitive process of empathetic dialogue. This allows for a holistic evaluation of how different skills interact.

• Controlled Vocal-Style Variations ($Style = M$ in Table 1): EchoMind explicitly designs identical, semantically neutral scripts rendered in multiple expressive styles (target, alternative, neutral). This is a crucial distinction from benchmarks that use single expressive styles ($Style = S$) or don't control for this, as it allows EchoMind to isolate the impact of vocal delivery independent of linguistic content. This direct control is vital for assessing vocal-aware speaker-state inference.

• Comprehensive Empathy-Oriented Framework: EchoMind is grounded in a detailed empathy-oriented framework with 3 coarse and 12 fine-grained dimensions, covering 39 vocal attributes. This systematic categorization and control of vocal cues (Speaker information, Paralinguistic information, Environmental information) is more extensive and purpose-built for empathy evaluation than many existing benchmarks that might only include paralinguistic features ($Para. = ✓$) or environmental sounds ($Env. = ✓$) without such a structured framework for empathetic response.

• Unified Contextual Grounding: All tasks in EchoMind are built upon the same set of audio instances, ensuring contextual consistency across understanding, reasoning, and conversation stages. Many other benchmarks construct narrowly targeted datasets that lack this shared grounding.

• Integrated Response Generation Evaluation ($Output = text, audio$): EchoMind explicitly evaluates both textual and acoustic expressiveness in open-ended conversation, including audio output ($Output = text, audio$) and subjective metrics for emotional alignment (EmoAlign, VES). Many benchmarks primarily focus on text output ($Output = text$).

• Focus on Beyond-Words Information: EchoMind's design explicitly targets the perception and interpretation of information beyond the literal transcript, which is central to high emotional intelligence, a dimension often under-evaluated or indirectly assessed in other benchmarks.

  The following are the results from Table 1 of the original paper:

  Benchmark	Voice Character			Data Character			Task			Corr.
  	Spk.	Para.	Env.	Input	Output	Style	Understanding		Reas.		Conv.
  							Content	Voice
  AudioBench [17]			(only)	text, audio	text	S	✓		✓	X	X
  Dynamic-SUPERB [15, 41]	*		(only)	text, audio	text	S	✓				X
  AIR-Bench [42]			(only)	text, audio	text	-	✓	*		*x	×
  Audio Entailment [18]			(only)	text, audio	text	-			✓		-
  SAKURA [19]			(only)	text, audio	text	S	✓		✓	X	X
  MMAR [44]			(only)	text, audio	text	S	✓		✓	X	X
  MMSU [23]			(only)	text, audio	text	S	✓
  MMAU [22]			(only)	text, audio	text	S	✓			x	x
  MSU-Bench [45]	✓	✓	✓	text, audio	text	S	✓	✓	✓	X	X
  SD-Eval [11]	✓	✓	✓	text, audio	text	M	✓	✓	X	✓
  VoxDialog [12]	✓	✓	✓	text, audio	text, audio	S	✓	✓	✓
  EChat-eval [14]	✓	✓	✓	text, audio	text, audio	S	20	✗	✓
  URO-Bench [13]	✓	✓	(only)	text, audio	text, audio	S	✓	✓	✓	✓
  EchoMind (Ours)	✓	✓	✓	text, audio	text, audio	M	✓	✓	✓	✓	✓

Notes for Table 1:

• Spk.: Presence of speaker information. Para.: Presence of paralinguistic features. Env.: Presence of environmental sounds. (only) means environmental sounds alone.

• $S$: Single expressive style for the same script. $M$: Multiple expressive styles.

• Reas.: Reasoning tasks. Conv.: Conversation tasks.

• Corr.: Whether different types of tasks in the benchmark are interrelated.

  As seen in the table, EchoMind stands out by having Spk., Para., Env., supporting text, audio input and output, using Multiple (M) expressive styles, including all Understanding (Content & Voice), Reasoning, and Conversation tasks, and crucially, having interrelated (✓) tasks.

4. Methodology

4.1. Principles

The core idea behind EchoMind's design is to simulate the cognitive process of empathetic dialogue in humans. This involves a sequence of steps:

1. Understanding: Perceiving both the literal spoken content and non-lexical vocal cues.

2. Reasoning: Integrating this understood information to infer the speaker's state and intent.

3. Conversation: Generating a response that is contextually, emotionally, and prosodically aligned with the speaker's input.

   To rigorously evaluate this, EchoMind adheres to two key principles:

• Isolation of Vocal Cues: By using semantically neutral scripts with controlled vocal-style variations, the benchmark ensures that any observed differences in model behavior or empathetic response can be directly attributed to the perception and processing of non-lexical acoustic cues, rather than explicit linguistic signals.
• Interrelated Multi-Level Evaluation: All tasks are context-linked and grounded in the same set of audio instances, allowing for a unified and comprehensive assessment of how understanding, reasoning, and response generation interact.

4.2. Core Methodology In-depth (Layer by Layer)

The EchoMind benchmark is designed with several integrated components:

4.2.1. Empathy-Oriented Framework

The benchmark is structured around an empathy-oriented framework that categorizes vocal cues into three coarse-grained dimensions, which are further refined into 12 fine-grained categories, encompassing a total of 39 specific vocal attributes. This framework guides the data generation and task design.

• Coarse-Grained Dimensions:

  1. Speaker Information: Attributes related to the speaker's identity.
  2. Paralinguistic Information: Attributes related to how speech is delivered (e.g., emotions, physiological state).
  3. Environmental Information: Attributes related to the surrounding context.

• Fine-Grained Categories and Vocal Attributes (Summarized from Table 2):

  • Speaker information:
    • Gender: Male, Female
    • Age: Child, Elderly
  • Paralinguistic Information:
    • Physiological State: Hoarse, Breath, Vocal fatigue, Sobbing
    • Emotion: Happy, Sad, Surprised, Angry, Fear, Disgust
    • Volume: Shout, Whisper
    • Speed: Fast, Slow
    • NVE (Non-Verbal Expression): Cough (keke), Sigh (ai), Laughter (haha), Yawn (ah~), Moan (uh)
  • Environmental Information:

    • Weather: Wind, Thunderstorm, Raining

    • Location: Sea Beach, Basketball Court, Driving (Bus), Subway

    • Human sounds: Applause, Cheering, Chatter, Children's Voice (play, speak)

    • Sudden Event: Alarm, Ringtone, Vehicle horn

    • Others: Music (Happy, Funny, Exciting, Angry), Dog bark

      The following are the results from Table 2 of the original paper:

      Speaker information
      Gender	Male, Female
      Age	Child, Elderly
      Paralinguistic Information
      Physiological State	Hoarse, Breath, Vocal fatigue, Sobbing
      Emotion	Happy, Sad, Surprised, Angry,
      Volume	Fear, Disgust Shout, Whisper
      Speed	Fast, Slow
      NVE	Cough (keke), Sigh(ai), Laugh- ter (haha), Yawn (ah~), Moan (uh)
      Environmental Information
      Weather	Wind, Thunderstorm, Raining
      Location	Sea Beach, Basketball Court, Driving (Bus), Subway
      Human sounds	Applause, Cheering, Chatter, Children's Voice (play, speak),
      Sudden Event	Alarm, Ringtone, Vehicle horn
      Others	Music (Happy, Funny, Exciting, Angry) , Dog bark

4.2.2. Audio Dataset Construction

The dataset construction ensures semantically neutral scripts and controlled vocal variations.

• Dialogue Script Synthesis:

  • GPT-4o [47] is used to generate one-turn dialogues, where a User initiates and a System responds.
  • User utterances are designed to be semantically neutral, avoiding explicit emotional or contextual cues in the text. This is crucial for isolating the impact of vocal cues.
  • For each user utterance, GPT-4o generates three types of responses:
    1. A highEQ response conditioned on both the content and a specified vocal cue.
    2. A cue-agnostic response (text-only), focusing purely on the semantic meaning.
    3. An alternative empathetic response under a different vocal attribute expression.
  • Diversity: 17 topics (e.g., work, health, travel) are defined for non-environmental attributes, with five scripts generated per topic. For environmental sounds, five scripts are generated without topic constraints.
  • Quality Control: All generated user utterances undergo manual review by three authors to mitigate LLM hallucinations [48], ensuring coherence and appropriateness. Only unanimously approved scripts are retained, resulting in 1,137 final scripts.
  • Reference Responses: Each of the three response types is expanded to five reference responses to support robust, multi-reference evaluation.

• Dialogue Audio Synthesis:

  • For each user-level utterance, three vocal-style speech variations are generated: target, neutral, and alternative expressiveness.
  • Tailored Strategy: Different synthesis methods are applied based on the vocal attribute's dimension:
    • Speaker Information: Doubao TTS API (a commercial Text-to-Speech service) is used.
    • Paralinguistic Cues: A multi-method approach:
      • Cough and Vocal fatigue are generated by guiding a Doubao conversational agent.
      • Hoarse is synthesized using Doubao's voice cloning.
      • Other vocal cues are generated using GPT-4o-mini-TTS with attribute-specific prompts.
    • Environmental Context: Clean speech is generated with Doubao TTS and mixed with background sounds from AudioCaps [49].
  • Balancing: Male and female voices are balanced across synthesis conditions.
  • Response Audio Generation: GPT-4o generates a voice-aware profile (specifying affect, tone, emotion, personality) for responses, which then guides GPT-4o-mini-TTS in audio generation, ensuring contextual and emotional alignment.

• EchoMind-Human Version:

  • To address potential artifacts or biases from fully TTS-generated data, a subset of 491 scripts (balanced across vocal attributes) is recorded by two professional human voice actors (one male, one female) with excellent English proficiency. This EchoMind-Human version serves to test model robustness against natural speech variability.

4.2.3. Multi-Level Tasks Formulation

EchoMind is structured into three interrelated levels, simulating human dialogue cognition, and anchored by the same set of audio instances for contextual consistency.

The following figure (Figure 1 from the original paper) illustrates the EchoMind framework and examples:

该图像是图示，展示了EchoMind框架中关于同理心对话的多层次认知过程的模拟。图中分为两部分：第一部分包括理解、推理和开放式对话三层级的任务；第二部分则展示了相同剧本下的不同声调变化的响应示例，强调了声音表达对回应的影响。

• Level 1: Understanding

  • Objective: Evaluate models' ability to perceive content and vocal cues.
  • Tasks:
    • Content Understanding (ASR): Measures transcription accuracy under challenging acoustic conditions (expressive delivery, environmental noise).
    • Voice Understanding (MCQs): Focuses on recognizing vocal cues through Multiple-Choice Questions.
      • Coarse-Grained: "Which of the following best describes this audio?" with options from different vocal dimensions, designed to have a unique correct answer by avoiding similarity, implication, and inclusion conflicts.
      • Fine-Grained: Focus on a single vocal dimension (e.g., "What is the most likely non-verbal expression the speaker made during the audio?").

• Level 2: Reasoning

  • Objective: Assess higher-order comprehension that requires integrating linguistic content and acoustic features to infer speaker state, intent, or situational context.
  • Task: Integrated Reasoning (MCQs).
    • Complexity: More challenging than voice understanding MCQs as they demand deeper inference beyond surface-level perception.
    • Question Types (10 total): Examples include Multiple People Detection, Laughter Sentiment Detection, Audio-Text Sentiment Consistency, Response Style Matching, Personalized Recommendation Matching, Contextual Suggestion Generation, Preceding Event Inference, and Speaker Intent Recognition. For instance, Personalized Recommendation Matching requires inferring speaker attributes from voice and applying this knowledge to make appropriate recommendations.
    • Construction: Relevant vocal attributes are defined, and questions/answers are constructed through manual design and GPT-4o generation, followed by a two-stage human verification process to ensure distinctiveness and a unique correct answer.

• Level 3: Conversation

  • Objective: Evaluate the model's ability to produce contextually coherent, socially appropriate, and empathetic open-ended responses to spoken input. This reflects the integration of perception and reasoning into natural dialogue.
  • Task: Dialogue (Open-domain Response).

    • Output: Models generate both textual and audio responses.

      The following are the results from Table 3 of the original paper:

      Task	Count
      Level 1: Understanding
      Content Understanding (ASR)	3356 (1137)
      Voice Understanding (MCQs)	4576 (2274)
      - Coarse-Grained	2338 (1137)
      - Gender Recognition	110 (55)
      - Age Group Classification	192 (64)
      - Voice Style Detection	348 (290)
      - Speech Emotion Recognition	794 (298)
      - Speaking Pace Classification	144 (34)
      - NVE Recognition	336 (239)
      - Background Sound Detection	314 (157)
      Level 2: Reasoning
      Integrated Reasoning (MCQs)	4747 (3612)
      - Multiple People Detection	248 (101)
      - Laughter Sentiment Detection	29 (29)
      - Shouting Sentiment Detection	32 (32)
      - Audio-Text Sentiment Consistency	244 (99)
      - Response Style Matching	368 (368)
      - Personalized Recommendation Matching	1473 (630)
      - Contextual Suggestion Generation	450 (450)
      - Preceding Event Inference	399 (399)
      - Speaker Intent Recognition	370 (370)
      - Empathy-Aware Response Selection	1134 (1134)
      Level 3: Conversation
      Dialogue (Open-domain Response)	3356 (1137)

Notes for Table 3: Numbers in parentheses show target expression audio inputs. Count indicates the total number of instances for each task or sub-task.

5. Experimental Setup

5.1. Datasets

The experiments primarily use the EchoMind benchmark dataset, which has two main versions: a TTS-generated version and a human-recorded version.

• EchoMind (TTS-generated version):

  • Source: Generated using GPT-4o for scripts and various TTS engines (Doubao TTS, GPT-4o-mini-TTS, AudioCaps for mixing environmental sounds) for audio.
  • Scale: Contains 1,137 unique scripts. Each script is presented in three vocal-style variations (target, neutral, alternative expressiveness), leading to 3,356 total audio inputs for Content Understanding and Dialogue tasks, and 4,576 for Voice Understanding and Reasoning MCQs (as some MCQs might have more combinations).
  • Characteristics:
    • Semantically neutral dialogue scripts to isolate vocal cue impact.
    • Covers 3 coarse and 12 fine-grained dimensions across 39 vocal attributes (Table 2).
    • Controlled vocal-style variations for the same script.
    • Reference responses are provided for evaluation.
  • Domain: One-turn dialogues across 17 diverse topics (e.g., work, health, travel, entertainment).
  • Example Data Sample: As seen in Table 12, a user utterance like "I finally managed to finish the science project last night!" can be delivered with a Happy voice attribute, a Neutral voice attribute, or a Sigh (ai) voice attribute, leading to different System responses and voice styles for the response.

• EchoMind-Human Version:

  • Source: A subset of 491 scripts from the TTS-generated version are manually recorded by professional human voice actors.

  • Scale: 491 scripts, each with target, neutral, and alternative expressiveness. Total 1,453 audio inputs (1,473 as per Table 10, likely due to additional processing for some tasks or a slight discrepancy).

  • Characteristics: Provides greater acoustic variability and prosodic nuance inherent in natural human speech, used to assess model robustness against real-world speech.

  • Choice Rationale: These datasets are chosen because EchoMind's design directly addresses the limitations of existing benchmarks by providing a multi-level, interrelated, and vocal-cue-controlled environment to evaluate empathetic SLM capabilities. The human-recorded subset ensures that evaluations are not solely reliant on potentially idealized TTS data.

    The following are the results from Table 8 of the original paper, showing detailed statistics for all audio inputs in EchoMind (TTS-generated):

    Voice Dimensions	Voice Attributes	Count	Hours	Dur.	Words/sec
    Neutral		1082	1.21	4.03	2.43
    Speaker information
    Gender	Male, Female	110	0.12	3.99	2.84/2.43
    Age	Child, Elderly	128	0.15	4.12	2.32/2.62
    Paralinguistic Information
    Physiological State	Hoarse, Breath, Vocal fatigue, Sobbing	258	0.44	6.17	2.57/1.57/1.74/1.01
    Emotion	Happy, Sad, Surprised, Angry, Fear, Disgust	794	0.99	4.5	2.36/1.73/2.46/2.48/1.76/1.43
    Volume	Shout, Whisper	90	0.12	4.68	2.49/1.85
    Speed NVE	Fast, Slow	244	0.50	7.42	3.05/1.06
    	Cough (keke), Sigh (ai), Laughter (ha), Yawn (ah∼), Moan (uh)	336	0.69	7.16	1.68/1.16/1.49/1.13/1.10
    Environmental Information
    Weather Location	Wind, Thunderstorm, Raining Driving (Bus), Subway, Sea Beach, Basketball Court
    Human sounds	Applause, Cheering, Chatter, Children's Voice (play, speak) Alarm, Ringtone, Vehicle horn
    Sudden Event
    Others	Music (Happy, Funny, Exciting, Angry) , Dog bark
    Overall		3356	4.51	4.84	2.03

The following are the results from Table 9 of the original paper, showing detailed statistics for target expression audio inputs in EchoMind (TTS-generated):

The following are the results from Table 10 of the original paper, showing detailed statistics for all audio inputs in EchoMind-Human:

The following are the results from Table 11 of the original paper, showing detailed statistics for target expression audio inputs in EchoMind-Human:

The following are the results from Table 12 of the original paper, showing examples of synthesized dialogue data from EchoMind:

The following are the results from Table 13 of the original paper, showing examples of MCQ tasks:

Question: What is the most likely age group of the speaker in the audio?

5.2. Evaluation Metrics

The evaluation metrics in EchoMind are comprehensive, covering objective and subjective measures across text and audio modalities for each task level.

The following are the results from Table 4 of the original paper:

5.2.1. Understanding Level

• Content Understanding (ASR):

  • Word Error Rate (WER):
    • Conceptual Definition: WER is a common metric for ASR systems, measuring the difference between a hypothesized (transcribed) sequence of words and a reference (ground truth) sequence. It quantifies the number of errors (substitutions, insertions, deletions) relative to the total number of words in the reference. Lower WER indicates better performance.
    • Mathematical Formula: $ \mathrm{WER} = \frac{S + D + I}{N} $
    • Symbol Explanation:
      • $S$: Number of substitutions (a word in the reference is replaced by a different word).
      • $D$: Number of deletions (a word in the reference is missing from the hypothesis).
      • $I$: Number of insertions (an extra word is present in the hypothesis that is not in the reference).
      • $N$: Total number of words in the reference (ground truth) transcript.
  • Semantic Similarity (SemSim):
    • Conceptual Definition: SemSim measures how semantically close two pieces of text (the gold reference transcript and the predicted transcript) are. It captures whether the meaning is preserved, even if the exact words differ. Higher SemSim indicates better performance.
    • Calculation: Computed by encoding both transcripts using Qwen3-Embedding-0.5B and measuring cosine similarity between their respective embeddings.
    • Mathematical Formula (Cosine Similarity): $ \mathrm{cosine_similarity}(A, B) = \frac{A \cdot B}{||A|| \cdot ||B||} = \frac{\sum_{i=1}^{n} A_i B_i}{\sqrt{\sum_{i=1}^{n} A_i^2} \sqrt{\sum_{i=1}^{n} B_i^2}} $
    • Symbol Explanation:
      • $A$: The embedding vector of the reference transcript.
      • $B$: The embedding vector of the predicted transcript.
      • $A_i$, $B_i$: The $i$-th component of vectors $A$ and $B$, respectively.
      • $n$: The dimension of the embedding vectors.
      • $\cdot$: Dot product.
      • $||A||$: Euclidean norm (magnitude) of vector $A$.

• Voice Understanding (MCQs):

  • Accuracy (ACC):
    • Conceptual Definition: Accuracy measures the proportion of correctly answered multiple-choice questions. It is a straightforward metric for classification tasks. Higher ACC indicates better performance.
    • Mathematical Formula: $ \mathrm{ACC} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}} $
    • Symbol Explanation:
      • Number of Correct Predictions: The count of questions where the model's chosen answer matches the ground truth.
      • Total Number of Predictions: The total number of questions attempted.

5.2.2. Reasoning Level

• Integrated Reasoning (MCQs):
  • Accuracy (ACC): Same definition and formula as for Voice Understanding (MCQs).

5.2.3. Conversation Level

This level involves both text-level and audio-level evaluations, combining objective (reference-based) and subjective (Model-as-a-Judge and human) metrics.

5.2.3.1. Text-level Evaluation

• Objective Metrics (Reference-based):

  • BLEU (Bilingual Evaluation Understudy) [50]:
    • Conceptual Definition: BLEU measures the n-gram overlap between a generated text and a set of reference texts. It also includes a brevity penalty to discourage overly short outputs. Higher BLEU scores indicate closer resemblance to human references.
    • Mathematical Formula: $ \mathrm{BLEU} = \mathrm{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right) $ where $ \mathrm{BP} = \begin{cases} 1 & \text{if } c > r \ e^{(1-r/c)} & \text{if } c \le r \end{cases} $
    • Symbol Explanation:
      • $\mathrm{BP}$: Brevity Penalty, which penalizes generated translations that are too short compared to the reference.
      • $c$: Length of the candidate (generated) translation.
      • $r$: Effective reference corpus length.
      • $N$: Maximum n-gram order (typically 4).
      • $w_n$: Weight for each n-gram precision $p_n$ (usually $1/N$).
      • $p_n$: Precision for $n$-grams, calculated as the count of $n$-grams in the candidate that also appear in any reference, divided by the total number of $n$-grams in the candidate.
  • ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence) [51]:
    • Conceptual Definition: ROUGE-L measures the longest common subsequence (LCS) between a generated text and a reference text. It is particularly useful for summarization tasks but also applies to dialogue by focusing on the sequence of words that are common, regardless of contiguity. Higher ROUGE-L indicates more shared content with the reference.
    • Mathematical Formula: $ \mathrm{R_{LCS}} = \frac{\mathrm{LCS}(X, Y)}{m} $ $ \mathrm{P_{LCS}} = \frac{\mathrm{LCS}(X, Y)}{n} $ $ \mathrm{ROUGE-L} = F_{LCS} = \frac{(1 + \beta^2) R_{LCS} P_{LCS}}{R_{LCS} + \beta^2 P_{LCS}} $ (Typically, $\beta$ is set to 1, making it the F-measure: $F_{LCS} = \frac{2 R_{LCS} P_{LCS}}{R_{LCS} + P_{LCS}}$)
    • Symbol Explanation:
      • $\mathrm{LCS}(X, Y)$: Length of the longest common subsequence between candidate sentence $X$ and reference sentence $Y$.
      • $m$: Length of the reference sentence $Y$.
      • $n$: Length of the candidate sentence $X$.
      • $\mathrm{R_{LCS}}$: LCS-based recall.
      • $\mathrm{P_{LCS}}$: LCS-based precision.
      • $\beta$: A parameter that controls the relative importance of $\mathrm{R_{LCS}}$ and $\mathrm{P_{LCS}}$.
  • METEOR (Metric for Evaluation of Translation with Explicit Ordering) [52]:
    • Conceptual Definition: METEOR addresses some limitations of BLEU by considering synonymy and stemming in addition to exact word matching. It computes a harmonic mean of precision and recall based on unigram alignments between the generated text and references, with a penalty for fragmentation. Higher METEOR indicates better quality.
    • Mathematical Formula: $ \mathrm{METEOR} = (1 - \mathrm{Penalty}) \cdot F_{mean} $ where $ F_{mean} = \frac{10 \cdot P \cdot R}{P + 9 \cdot R} $ $ \mathrm{Penalty} = 0.5 \cdot \left(\frac{\text{Number of Chunks}}{\text{Number of Unigrams Matched}}\right)^3 $
    • Symbol Explanation:
      • $P$: Unigram precision.
      • $R$: Unigram recall.
      • Number of Chunks: The number of contiguous sequences of matched unigrams.
      • Number of Unigrams Matched: Total count of unigrams that are matched between the candidate and reference(s).
  • BERTScore [53]:
    • Conceptual Definition: BERTScore leverages contextual embeddings from pre-trained BERT models to measure semantic similarity between candidate and reference sentences. Instead of discrete n-gram matching, it computes cosine similarity between BERT embeddings of tokens and uses optimal transport to find the best matching pairs, yielding a more robust semantic comparison. Higher BERTScore indicates greater semantic similarity.
    • Mathematical Formula: BERTScore is calculated by computing precision, recall, and an F1-score based on cosine similarities between token embeddings. $ P = \frac{1}{|{\hat{x}}|} \sum_{i=1}^{|{\hat{x}}|} \max_{j \in {1,...,|x|}} \mathrm{cos}(x_j, \hat{x}i) $ $ R = \frac{1}{|x|} \sum{j=1}^{|x|} \max_{i \in {1,...,|{\hat{x}}|}} \mathrm{cos}(x_j, \hat{x}_i) $ $ F_1 = 2 \frac{P \cdot R}{P + R} $
    • Symbol Explanation:
      • $x$: Reference sentence.
      • $\hat{x}$: Candidate (generated) sentence.
      • $|x|$, $|{\hat{x}}|$: Lengths of the reference and candidate sentences, respectively.
      • $x_j$, $\hat{x}_i$: BERT embedding of the $j$-th token in the reference and $i$-th token in the candidate, respectively.
      • $\mathrm{cos}(\cdot, \cdot)$: Cosine similarity between two embedding vectors.
      • $P$: Precision, measuring how much of the candidate is covered by the reference.
      • $R$: Recall, measuring how much of the reference is covered by the candidate.
      • $F_1$: F1-score, the harmonic mean of precision and recall.

• Subjective Metrics (Model-as-a-Judge): These metrics are rated on a 5-point scale, implemented using GPT-4o as a Model-as-a-Judge, and also used for human evaluation.

  • C1 (Context Fit):
    • Conceptual Definition: Assesses how relevant the response is to the conversation and how appropriately it addresses the elements of the spoken content and vocal cues implied by the scenario.
  • C2 (Response Naturalness):
    • Conceptual Definition: Evaluates how smoothly the response flows within the dialogue, reflecting whether it sounds like something a real person would say.
  • C3 (Colloquialism Degree):
    • Conceptual Definition: Measures the extent to which the response uses natural, everyday conversational language, reflecting its informality and conversational style.
  • C4 (Speech Information Relevance):
    • Conceptual Definition: Quantifies the degree to which the response incorporates speaker-related vocal attributes from the input speech. This is a crucial empathy-oriented metric.

5.2.3.2. Audio-level Evaluation

• Objective Metrics (Quality):

  • NISQA (Non-Intrusive Speech Quality Assessment) [54]:
    • Conceptual Definition: NISQA is a neural network-based model that predicts speech quality (Mean Opinion Score - MOS) in a non-intrusive manner, meaning it only requires the speech signal itself, without a clean reference signal. It assesses various aspects of quality, including overall quality, noisiness, coloration, discontinuity, and loudness. Higher scores indicate better quality.
    • Mathematical Formula: Not a single formula, but a deep CNN-self-attention model that outputs a MOS prediction based on speech features.
  • UTMOS (Utokyo-SARULab MOS prediction system) [55]:
    • Conceptual Definition: UTMOS is another non-intrusive MOS prediction system for evaluating speech quality, often used in challenges for Voice MOS. Similar to NISQA, it provides an objective estimate of speech naturalness and overall audio quality. Higher scores indicate better quality.
    • Mathematical Formula: Also based on complex deep learning models, not a single analytical formula.

• Subjective Metrics (Model-as-a-Judge and Human):

  • EmoAlign (Emotional Alignment):
    • Conceptual Definition: A reference-based measure that compares the gold reference emotions (predicted by GPT-4o from dialogue content and vocal cues) with the emotions inferred from the generated audio response using emotion2vec [56]. It assesses how well the generated audio's emotional content matches the intended emotional target. Higher scores indicate better alignment.
  • Vocal Empathy Score (VES):
    • Conceptual Definition: Uses Gemini-2.5-Pro [57], a state-of-the-art voice understanding model, to assess whether a generated audio response mirrors the interlocutor's vocal style and emotional state. This metric directly measures the prosodic appropriateness and emotional expressiveness of the generated speech in an empathetic context. It's rated on a 5-point scale.

5.2.3.3. Human-only Subjective Metrics (Additional)

These are used in the human evaluation subset.

• Audio-Quality:
  • Conceptual Definition: Assesses the clarity and overall perceptual quality of the generated response audio by human evaluators.
• Response Difference:

  • Conceptual Definition: Measures the average variation in responses (on a 5-point scale) when the same script is rendered in different vocal styles. This assesses how well the model adapts its response to different vocal inputs.

    The following are the results from Table 14 of the original paper, showing specific scoring definitions of metrics:

    Metric Name	Specific Scoring Definition
    		Definition Reflects how well the re- sponse fits within the context Context Fit of the scenario (i.e., topic, and speaker A's utterance).	5 points: The reply fully matches the dialogue background; it is smooth and natural, perfectly fitting the context and situation. 4 points: The reply adapts well to the dialogue background; the content is coherent and relevant, with minor room for improvement. 3 points: The reply basically adapts to the dialogue background and is
    	Response	Focus on whether the re- sponse seems relevant to the conversation and addresses the elements in the case ap- propriately. Reflects how naturally the response flows within the conversation. It considers	generally on-topic, but parts feel unnatural or slightly off-topic. 2 points: The reply partially fits the dialogue background, but the content is not fully relevant and feels somewhat unnatural or lacks fluency. 1 point: The reply does not adapt to the dialogue background at all; it is unrelated to the topic or context and feels abrupt or unnatural. 5 points: The response is exceptionally natural, fully capturing the flow and authenticity of real conversation; it sounds like a genuine exchange between two people. 4 points: The response is very natural, with a tone that fits casual dia-
    C2	Naturalness	whether the response sounds like something a real person would say in the given con- text. Evaluates how informal or	logue; there are no noticeable awkward or unnatural elements. 3 points: The response is generally natural, though somewhat formulaic; overall, it matches the rhythm and tone of everyday conversation. 2 points: The response has some naturalness, but the tone or phrasing still feels slightly unnatural, with a rigid structure. 1 point: The response feels stiff or robotic, lacking conversational flu- ency; it sounds like pre-written lines. 5 points: The response is fully colloquial, using the relaxed, authentic language of everyday dialogue; it feels effortless and natural.
    C3	Colloquialism Degree	conversational the response content looks like. Checks if the response uses natural, everyday language, particu- larly in spoken or informal settings.	4 points: The response is largely colloquial—warm, natural, and well- suited to informal exchanges, with only a trace of formality. 3 points: The response strikes a moderate balance: it mixes formal and colloquial expressions, making it suitable for daily conversation but still slightly reserved. 2 points: The response contains some colloquial elements, yet its overall tone remains fairly formal, lacking lived-in, natural phrasing. 1 point: The response is entirely non-colloquial—overly formal or aca- demic—and completely mismatched with everyday spoken language. 5 points: The response is entirely grounded in the speech information ,
    C4	Speech Infor- mation Rele- vance	Evaluates how the response should be formulated based on the provided speech infor- mation. The score should re- flect how accurately the sen- tence addresses or incorpo- rates the speech information into this response.	accurately reflecting its relevant content and achieving a high degree of alignment with speech information. 4 points: The response takes the speech information into account and shows some awareness of , yet it does not fully integrate it into the conversation, making the reply somewhat stiff and leaving room for more natural expression. 3 points: The response somewhat overlooks the speech information, failing to fully incorporate its characteristics, resulting in a reply that feels imprecise or biased. 2 points: The response barely acknowledges the speech information and instead presents content that is either contradictory or inconsistent with. 1 point: The response is completely unrelated to the provided speech
    VES	Vocal Empa- thy Score	Measures how well the re- sponder's speech expresses an appropriate emotional tone and vocal style to match the speaker's described state.	information; it offers no content that reflects or addresses in any way. 5 points: Perfect empathy: The responder's vocal emotional intensity, pitch, rhythm, and tone highly match the speaker's state, conveying appropriate care or emotional resonance. 4 points: Basic empathy: The vocal style of the responder generally matches the speaker's state, but there are minor deficiencies, such as the emotional intensity being slightly weaker or missing subtle pauses. 3 points: Weak empathy: The direction is correct, with some resonance, but the emotional expression is insufficient or lacks key vocal features. 2 points: Incorrect empathy: Most of the style doesn't match the speaker's state, even opposite to it. 1 point: No empathy: The vocal style shows no emotional expression at

The following are the results from Table 15 of the original paper, showing specific scoring definitions of metrics used for human evaluation only:

5.3. Baselines

The paper evaluates 12 advanced end-to-end SLMs, covering a mix of closed-source and open-source models:

• Closed-source: GPT-4o-Audio [3] (OpenAI)
• Open-source:

  • Audio Flamingo 3 series [30] (Base, Base + $Φ$ Thinking, and Chat version)

  • DeSTA2.5-Audio [58]

  • VITA-Audio [31]

  • LLaMA-Omni2 [37]

  • Baichuan-Omni-1.5 [5]

  • GLM-4-Voice [4]

  • OpenS2S [38]

  • Qwen2.5-Omni-7B [7]

  • Kimi-Audio [59]

  • Step-Audio [35]

  • EchoX [34]

    These baselines represent the current state-of-the-art in Speech Language Models, including leading models from major research institutions and companies. They are representative because they cover various architectural approaches (e.g., unified, end-to-end), different scales, and both publicly available and proprietary systems, allowing for a broad assessment of current capabilities in empathetic dialogue.

5.4. Prompts Setup

The paper employs different prompting strategies depending on the task:

• ASR Task:

  • Default prompts are used if available for the SLM.
  • Otherwise, a generic instruction: "Please transcribe the speech in the input audio into text".

• MCQs Task (Voice Understanding & Reasoning):

  • The task inputs include the input audio, the question, and the provided options.
  • Instructions specify the expected output format (e.g., selecting option A, B, C, or D).

• Conversation Task (Three-Tier Prompting Strategy): This systematic approach examines model performance under different levels of instruction to understand prompt sensitivity.

  • P1 (Zero-prompt): Models process the audio input directly without any system prompt. This tests the model's inherent ability to infer conversational intent and context.
  • P2 (Basic Prompt): Models are given a general instruction: "provide a direct and concise response". This provides minimal guidance for conversational behavior.
  • P3 (Enhanced Prompt): This builds upon the basic prompt by explicitly instructing models to consider both the spoken content and the vocal cues when generating responses. The prompt is tailored based on the vocal cue dimension:

    • For Speaker Information: "Your task is to provide a direct and concise response, considering both the spoken content and any personal information present in the user's voice."

    • For Paralinguistic Information: "Your task is to provide a direct and concise response in a customer service setting, considering both the spoken content and any paralinguistic information present in the user's voice."

    • For Environment Information: "Your task is to provide a direct and concise response, considering both the spoken content and any background sounds present."

      The following are the results from Table 16 of the original paper, showing system prompt settings for the conversation task:

      P2 Basic

I will provide a specific topic/scenario along with the user's input. Your task is to provide a direct and concise response, simulating a one-turn interaction.

P3 Enhance

Speaker Information: I will provide a specific topic/scenario along with the user's input. Your task is to provide a direct and concise response, considering both the spoken content and any personal information present in the user's voice.

Paralinguistic Information: I will provide a specific topic/scenario along with the user's input. Your task is to provide a direct and concise response in a customer service setting, considering both the spoken content and any paralinguistic information present in the user's voice.

Environment Information: I will provide a specific topic/scenario along with the user's input. Your task is to provide a direct and concise response, considering both the spoken content and any background sounds present.

5.5. Audio Inputs Setup

Across all tasks, evaluations are primarily conducted on target expression audio inputs. This ensures that the evaluation is strictly relevant to the specific vocal attribute being tested and facilitates inter-task correlation analysis. Alternative and neutral inputs serve as controlled variables to understand the impact of varying vocal expressions on model performance.

6. Results & Analysis

6.1. Core Results Analysis

The evaluation of 12 advanced SLMs on EchoMind reveals consistent strengths in content understanding but significant weaknesses in vocal-cue processing and empathetic response generation.

The following are the results from Table 5 of the original paper, showing overall performance of SLMs across all EchoMind tasks:

Notes for Table 5:

• WER ↓: Word Error Rate (lower is better).
• SemSim ↑: Semantic Similarity (higher is better).
• ACC ↑: Accuracy (higher is better).
• NISQA ↑: Non-Intrusive Speech Quality Assessment (higher is better).
• DNMOS↑: A speech quality metric, likely related to Degradation Non-Intrusive Mean Opinion Score (higher is better).
• EmoAlign ↑: Emotional Alignment (higher is better).
• VES ↑: Vocal Empathy Score (higher is better).
• BLEU ↑, ROUGE-L ↑, METEOR ↑, BERTScore ↑: Text generation metrics (higher is better).
• C1 ↑, C2 ↑, C3 ↑, C4 ↑: Subjective metrics for conversational text (higher is better).
• Bold and underline indicate the best and second-best performance.
• - in WER/SemSim indicates no native ASR capability or results not directly comparable.
• Empty cells in Response (Audio) mean the model cannot directly produce speech output.

6.1.1. Performance Overview

• Content Understanding (ASR): Most SLMs demonstrate strong performance in SemSim (e.g., Qwen2.5-Omni-7B at 99.27%, Audio-Flamingo3 at 99.18%), indicating good linguistic content transcription. WER varies, with Audio-Flamingo3 showing the lowest (2.93).
• Voice Understanding (MCQs): Performance varies considerably. GPT-4o-Audio leads with 66.25% ACC, followed by Audio-Flamingo3+Think (65.16%) and Audio-Flamingo3 (64.29%). Many open-source models struggle, with VITA-Audio and GLM-4-voice scoring very low (25.24% and 25.54% respectively), suggesting difficulty in perceiving non-lexical cues.
• Reasoning (MCQs): This task requires integrating spoken content and vocal cues. GPT-4o-Audio again performs best (68.04% ACC), with DeSTA2.5-Audio close behind (63.04%). The general trend follows voice understanding, but scores are often lower, highlighting the challenge of multimodal inference. GLM-4-voice (22.28%) and VITA-Audio (27.69%) show significant struggles here.
• Conversation (Text-level):
  • Objective Metrics: GPT-4o-Audio achieves the highest scores across BLEU (2.54), ROUGE-L (19.91), METEOR (18.37), and BERTScore (82.70), demonstrating superior text generation quality compared to references.
  • Subjective Metrics (C1-C4): GPT-4o-Audio generally excels in C1 (context fit), C2 (response naturalness), and C3 (colloquialism degree). However, a critical finding emerges for C4 (speech information relevance). No model exceeds an average score of 4, and most hover around 3. This indicates that while SLMs can generate natural and relevant text, they consistently struggle to effectively leverage vocal cues to inform their textual responses in an empathetic way.
• Conversation (Audio-level):
  • Quality (NISQA, DNMOS): Most models capable of generating speech produce high-quality audio (NISQA scores often above 4.5), indicating good TTS capabilities. VITA-Audio and GPT-4o-Audio show strong NISQA scores.
  • Emotional Alignment (EmoAlign, VES): Subjective metrics for emotional alignment (especially VES) reveal persistent challenges. GPT-4o-Audio leads in VES (3.34) and EmoAlign (51.31%), but these scores are still relatively low compared to human expectations for empathetic responses, reinforcing the models' limitations in adapting vocal delivery to match the interlocutor's state. EchoX shows a particularly low VES of 1.40.

6.1.2. The Vocal-Cue Gap

The results consistently demonstrate a "vocal-cue gap". SLMs are proficient at transcribing spoken content and producing high-quality speech, but their ability to deeply perceive, reason about, and integrate non-lexical vocal cues into both the textual content and vocal delivery of their responses remains limited. This is particularly evident in the lower scores for Voice Understanding, Reasoning, C4 (Speech Information Relevance), and VES (Vocal Empathy Score).

6.2. Task Correlations - General Positive Association in Vocal-Cue-Aware Performance

The following figure (Figure 2 from the original paper) presents the correlations between model performance in vocal-cue-aware understanding, reasoning, and conversational response quality:

该图像是图表，展示了模型在声调感知理解、推理和对话响应质量的相关性。图中包含三个散点图，分别对应不同的变量组合，数据点表现了各种模型的性能差异。

The correlation analysis reveals:

• Understanding-Reasoning Correlation (Left Plot): A general positive correlation is observed. Models with stronger voice understanding abilities tend to achieve higher reasoning accuracy. This suggests that accurate perception of vocal cues is a prerequisite for effective multimodal inference (integrating content and voice). However, the plot also shows that strong understanding doesn't always guarantee equally high voice-based reasoning, as some SLMs (e.g., Audio-Flamingo3+Think) deviate, implying that the reasoning step itself poses additional challenges beyond mere perception.
• Understanding-Conversation and Reasoning-Conversation Correlations (Middle and Right Plots): A broadly similar upward trend is seen in both plots. Models that are better at understanding vocal cues and reasoning with them tend to produce higher quality conversational responses (as measured by C4 and VES).
• Outliers and Instruction-Following: Notably, GLM-4-voice and VITA-Audio emerge as clear outliers. They exhibit relatively high conversational response quality (particularly VES) despite low scores in vocal-cue understanding and reasoning. This discrepancy is attributed to potential weaker instruction-following capability. The understanding and reasoning tasks are MCQ-formatted, demanding precise compliance with instructions. As Table 5 and further analysis (RQ1) show, these models sometimes achieve their best conversational performance without any system prompt, suggesting that explicit instructions might even hinder their "natural" empathetic response if their instruction-following is weak. This implies that the underlying empathetic capacity might be present, but not effectively elicited or measured by instruction-heavy tasks.

6.3. Human Evaluation — Alignment with Model-based Automatic Metrics

To validate the Model-as-a-Judge paradigm, a human evaluation was conducted on a sampled subset (6 cases per vocal-cue type) for three representative SLMs: Qwen2.5-Omni-7B, Step-Audio, and GPT-4o-Audio.

The following are the results from Table 6 of the original paper, showing comparison of human and Model-as-a-judge scores for three representative SLMs on the conversation task:

Notes for Table 6: Bold and underline indicate the best and second-best performance.

• Overall Alignment: The human and Model-as-a-Judge scores (GPT-4o for C1-C4, Gemini for VES, NISQA for Audio-Quality) show a consistent relative ranking between the models, generally supporting the validity of the automatic evaluation protocol. For instance, Step-Audio generally performs well across both evaluations, while Qwen2.5-Omni-7B shows slightly lower scores, and GPT-4o-Audio often leads.
• Discrepancies for GPT-4o-Audio: Two notable divergences occur for GPT-4o-Audio:
  1. C2 (Response Naturalness): Human ratings (3.73) are significantly lower than GPT-4o's automatic score (4.74).
  2. VES (Vocal Empathy Score): Human ratings (2.49) are also considerably lower than Gemini's automatic score (3.34).
  • Reasons for Discrepancy: Human evaluators attributed these lower scores to GPT-4o-Audio generating "overly long, formally structured responses" that sounded less natural for dialogue, and its "synthesized voice being more formal in timbre" compared to other models' softer, warmer tones, which are perceived as more empathetic. This highlights a subtle but important gap between LLM-as-a-judge perceptions and nuanced human preferences for empathetic conversational style.
• Response Difference: All models achieved Response Difference scores above 3.0 (with GPT-4o-Audio highest at 3.50). This indicates that the models do exhibit some adaptation to vocal-cue variations despite identical content. However, none surpassed 4.0, underscoring that there is substantial room for improvement in generating truly distinct and appropriately varied responses based on vocal style.

6.4. RQ1: Prompt Sensitivity of Vocal-Cue-Aware Conversational Responses

The following figure (Figure 3 from the original paper) illustrates the sensitivity of conversational responses under three prompt settings:

该图像是一个散点图，展示了不同模型在两个评估标准（C4和VES）下的表现，且响应敏感度在三种提示设置（P1、P2和P3）中有所变化。通过对比，可以看到不同模型在精准度评分上的差异。

• P1: Zero-prompt (no system instructions).

• P2: Basic prompt ("provide a direct and concise response").

• P3: Enhanced prompt (explicitly instructs to consider spoken content and vocal cues, tailored by dimension).

  The analysis of C4 (speech information relevance) and VES (vocal empathy score) under these three prompt configurations reveals:

• General Sensitivity: Most models exhibit sensitivity to prompt variation, meaning their performance in leveraging vocal cues changes based on the instructions. Step-Audio shows the largest performance differences across settings, indicating high sensitivity.

• Effectiveness of Enhanced Prompts: Seven out of 12 SLMs achieve their highest C4 scores with the P3 enhanced prompt. This suggests that explicit instructions to attend to vocal cues can be effective in guiding some models to generate more vocal-cue-aware textual responses.

• Limited Instruction-Following: Conversely, some models perform best with P1 (zero-prompt). This implies that their instruction-following capability remains limited. For these models, adding explicit instructions (even well-intentioned ones like P3) can sometimes degrade performance, perhaps by interfering with their inherent processing or by misinterpreting the specific nuances of the prompt. This finding aligns with the observation that GLM-4-voice and VITA-Audio were outliers in the correlation analysis, performing well conversationally despite low MCQ scores, possibly because they struggled with the MCQ format's strict instruction demands.

  The following are the results from Table 17 of the original paper, showing best-response prompt for each SLM, corresponding to the best scores reported in Table 5:

  Model Prompt

  Audio-Flamingo3 P1

  Audio-Flamingo3+Think P1

  Audio-Flamingo3-Chat P1

  DeSTA2.5-Audio P3

  Vita-Audio P1

  LLaMA-Omni2 P3

  Baichuan-Omni-1.5 P3

  GLM-4-voice P1 P3

  OpenS2S Qwen2.5-Omni-7B P3

  Kimi-Audio P1

  P3

  Step-Audio EchoX P2

  GPT-4o-Audio P3

Notes for Table 17: This table shows which prompt (P1, P2, or P3) led to the best conversational response performance for each model in Table 5, as selected based on voice-cue-related metrics (C4 and VES). Several models (Audio-Flamingo3, Audio-Flamingo3+Think, Audio-Flamingo3-Chat, VITA-Audio, GLM-4-voice, Kimi-Audio) achieved their best with P1 (zero-prompt), confirming the observation about instruction-following limitations. Others (DeSTA2.5-Audio, LLaMA-Omni2, Baichuan-Omni-1.5, OpenS2S Qwen2.5-Omni-7B, GPT-4o-Audio) benefited from P3 (enhanced prompt).

6.5. RQ2: Impact of Speech Source on Vocal-Cue Processing Performance

The following figure (Figure 4 from the original paper) compares the performance differences of the three top-performing models on the EchoMind-Human version and the corresponding TTS-generated version of the same scripts, focusing on metrics assessing vocal-cue processing:

该图像是一个柱状图，展示了不同模型在理解（语音）与推理以及对话（C1、C4 和 VES）任务中的表现差异。左侧的柱状图比较了人类与合成语音（TTS）在理解和推理上的差异，右侧展示了在对话任务上的表现。

• General Trend: The comparison for the three top-performing models (Qwen2.5-Omni-7B, Step-Audio, GPT-4o-Audio) on the EchoMind-Human (human-recorded) versus TTS-generated versions shows that human-recorded speech consistently poses greater challenges.
• Impact Across Levels: The performance gap is observed across all three evaluation levels (Understanding (Voice), Reasoning, and Conversation as measured by C1, C4, and VES).
• Most Pronounced Impact in Conversation: The most significant performance drop is typically seen in the conversation task (particularly C4 and VES), indicating that models struggle more to generate empathetically aligned responses when processing the nuanced and variable nature of human speech.
• Reasons for the Gap: This performance gap likely reflects the greater acoustic variability (e.g., natural pauses, disfluencies, subtle emotional inflections) and prosodic nuance present in human speech compared to the often more "clean" and predictable TTS-generated data.
• Implication: This finding underscores the critical need to enhance model robustness for real-world, human-machine interaction. Current SLMs, even advanced ones, are not yet fully adept at handling the complexities of natural human vocal expression.

6.6. RQ3: Upper Bound of Empathetic Response Quality Under Ideal Vocal-Cue Recognition

To understand the potential maximum empathetic response capability of SLMs, an idealized scenario is simulated in the conversation task. In this scenario, models are provided with both the audio input AND the corresponding vocal-cue information (i.e., perfect vocal cue recognition is assumed).

The following are the results from Table 7 of the original paper, showing upper-bound performance evaluation:

Notes for Table 7: Values in parentheses indicate gains over the baseline without vocal-cue input (from Table 5).

• Improved Performance: Under this ideal condition (perfect vocal cue recognition), all three representative models (Qwen2.5-Omni-7B, Step-Audio, GPT-4o-Audio) achieve higher scores across C1 (context fit), C4 (speech-information relevance), and VES (vocal empathy score).
• Gains in Vocal-Cue-Aware Metrics: The gains are particularly significant for C4 and VES, the metrics directly related to vocal-cue integration and empathetic vocal delivery. For instance, Step-Audio shows the largest gain in C4 (+1.10), while GPT-4o-Audio shows the largest gain in VES (+1.08).
• Potential Ceiling: GPT-4o-Audio reaches the highest absolute values (C1: 4.83, C4: 4.45, VES: 4.42) under this ideal setting. These results reflect the potential ceiling of current SLMs' empathetic response capability if they could perfectly recognize and utilize vocal-cue information.
• Persistent Gap: Even with ideal vocal-cue information, the scores, while improved, are not perfect (e.g., VES maxes out at 4.42 out of 5). This suggests that empapthetic response generation involves more than just perfect recognition; it also requires sophisticated reasoning and generation capabilities to translate that recognition into a truly human-like empathetic response, a challenge that still exists. The primary takeaway, however, is that a major bottleneck for current SLMs is their ability to adequately recognize and extract these crucial vocal cues from the raw audio.

7. Conclusion & Reflections

7.1. Conclusion Summary

This work introduces EchoMind, a groundbreaking interrelated, multi-level benchmark specifically designed to evaluate the empathetic capabilities of Speech Language Models (SLMs). By simulating the cognitive process of empathetic dialogue through sequential, context-linked tasks (understanding, reasoning, and response generation), EchoMind offers a comprehensive assessment framework. Its unique design incorporates 3 coarse and 12 fine-grained dimensions, covering 39 vocal attributes, and crucially employs semantically neutral scripts with controlled vocal-style variations to isolate the impact of non-lexical cues.

Benchmarking 12 advanced SLMs on EchoMind revealed a significant vocal-cue gap: even state-of-the-art models struggle to perceive and effectively utilize highly expressive vocal cues, severely limiting their ability to generate responses that are both contextually appropriate and emotionally aligned. Detailed behavioral analyses highlighted persistent weaknesses in instruction-following, robustness to natural speech variability (human vs. TTS performance gaps), and the effective use of vocal attributes for empathy. While models show potential when ideal vocal-cue information is provided, their inherent recognition capabilities remain a bottleneck. The findings underscore a critical need for SLMs that can seamlessly integrate linguistic content with diverse vocal cues to achieve truly human-like, emotionally intelligent conversational ability.

7.2. Limitations & Future Work

The paper implicitly points to several limitations of current SLMs and, by extension, areas for future work:

• Robustness to Natural Speech Variability: The observed performance drop on EchoMind-Human data indicates that SLMs need to become more robust to the inherent acoustic variability and prosodic nuance of real human speech. Future work should focus on training models with more diverse and naturalistic datasets.
• Instruction-Following Capabilities: The prompt sensitivity analysis revealed that many models struggle with instruction-following, sometimes performing better with zero prompts than with explicit instructions to use vocal cues. This suggests a need for models that can better understand and incorporate complex, multi-faceted instructions into their generation process.
• Effective Vocal Cue Recognition and Integration: The "vocal-cue gap" is the central finding. Future SLMs must improve their ability to:
  1. Accurately perceive and extract a wide range of non-lexical vocal cues (speaker information, paralinguistics, environmental context).
  2. Integrate these cues deeply with linguistic content during the reasoning phase.
  3. Translate this integrated understanding into empathetically aligned textual responses and prosodically appropriate audio delivery.
• Beyond "Perfect" Vocal Cue Recognition: Even with ideal vocal-cue information, SLMs did not achieve perfect empathetic responses, suggesting that the generation component itself requires further sophistication in rendering nuanced emotional and stylistic alignment.
• Multimodal Reasoning Complexity: The reasoning task scores, while correlated with understanding, were often lower, highlighting that complex multimodal reasoning itself is a challenging area for current SLMs.

7.3. Personal Insights & Critique

EchoMind represents a significant leap forward in evaluating the emotional intelligence of SLMs. The benchmark's multi-level, interrelated design is particularly insightful, as it moves beyond isolated skill assessment to simulate a more holistic cognitive process, which is essential for human-like interaction. The meticulous construction of semantically neutral scripts with controlled vocal variations is a brilliant methodological choice, allowing researchers to precisely pinpoint the role of non-lexical cues.

Insights:

• The "Vocal-Cue Gap" is Critical: The paper effectively identifies a fundamental limitation of current SLMs. While large models excel at linguistic understanding and text generation, their inability to truly "hear" and "feel" the emotional and contextual nuances in a speaker's voice is a major barrier to genuine empathy. This highlights that simply scaling up LLMs or TTS capabilities in isolation won't achieve the desired empathetic dialogue.
• Value of Multimodal Integration: The correlations between understanding, reasoning, and conversation underscore that empathy is not a single skill but an emergent property of integrated multimodal processing. Future SLM architectures must be designed from the ground up to deeply intertwine linguistic and acoustic information.
• Human vs. Synthetic Data for Robustness: The EchoMind-Human dataset and the performance drop it reveals are crucial. It's a reminder that AI systems trained predominantly on synthetic or "clean" data may perform poorly in the messy reality of human interaction. This emphasizes the need for domain adaptation and robustness training using diverse, naturalistic data.
• Beyond Model-as-a-Judge: The subtle discrepancies between GPT-4o as a judge and human evaluators for naturalness and vocal empathy are fascinating. It suggests that while LLMs are powerful evaluation tools, the nuances of human perception, especially for qualities like "warmth" or "naturalness" in voice, might still require human oversight or more sophisticated AI-based perceptual models.

Critique/Areas for Improvement:

• Generalizability of Vocal Attributes: While 39 vocal attributes are extensive, human vocal expression is infinitely nuanced. The current framework, while strong, still relies on predefined categories. Future work might explore models that can learn to interpret and respond to novel or complex combinations of cues not explicitly defined.

• Long-Term Dialogue: EchoMind focuses on one-turn dialogues. Real empathetic conversation often unfolds over multiple turns, requiring memory, consistency, and adaptation to evolving emotional states. Extending the benchmark to multi-turn interactions would introduce new complexities and more realistic challenges.

• Ethical Considerations: The development of truly empathetic AI raises significant ethical questions (e.g., manipulation, emotional labor, misuse). While not the focus of this technical paper, it's a broader context that should be considered as these models improve.

• Interpretability: Understanding why an SLM responds empathetically or fails to do so could be critical. Integrating interpretability methods into the evaluation framework could provide deeper insights into model decision-making processes regarding vocal cues.

  Overall, EchoMind provides a robust foundation for driving research towards SLMs that can engage in genuinely empathetic conversations, moving the field closer to truly intelligent and human-aware AI.