1. Bibliographic Information

1.1. Title

SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing

Central topic: A multimodal model that aligns spatial audio with natural language by decomposing audio into semantic and spatial embeddings, enabling cross-modal retrieval, zero-shot direction understanding, and text-guided spatial audio editing.

1.2. Authors

Jinbo Hu (Institute of Acoustics, Chinese Academy of Sciences; MiLM Plus, Xiaomi Inc.), Yin Cao (Xi’an Jiaotong Liverpool University), Ming Wu (Institute of Acoustics, Chinese Academy of Sciences), Zhenbo Luo (MiLM Plus, Xiaomi Inc.), Jun Yang (Institute of Acoustics, Chinese Academy of Sciences; University of Chinese Academy of Sciences).

Research background and affiliations:

• The team spans academia and industry in China, with strengths in acoustics, machine listening, and audio-language modeling. Prior work includes PSELDNets, a pre-trained SELD (sound event localization and detection) foundation model emphasizing spatial audio.

1.3. Journal/Conference

arXiv preprint (no explicit peer-reviewed venue listed). arXiv is a preprint server widely used in machine learning and signal processing for rapid dissemination; results are not peer-reviewed unless otherwise noted.

1.4. Publication Year

2025 (posted July 22, 2025 UTC).

1.5. Abstract

The paper introduces SALM, a Spatial Audio-Language Model that aligns spatial audio with language via contrastive learning. SALM uses a text encoder and a dual-branch audio encoder: a semantic branch (content) and a spatial branch (direction), connected via soft parameter sharing and projected into structured 512-D embeddings. Key capabilities include: aligned audio-text representations, separate and joint spatial/semantic extraction, zero-shot direction classification, and text-guided spatial audio editing by replacing spatial embeddings with directional text embeddings. Experiments on synthetic spatial versions of AudioCaps and Clotho (plus real-scene SRIR augmentation) show improved retrieval, well-structured embeddings, low localization error, and flexible editing.

1.6. Original Source Link

• arXiv page: https://arxiv.org/abs/2507.16724v2

• PDF: https://arxiv.org/pdf/2507.16724v2.pdf

  Status: Preprint on arXiv (v2).

2. Executive Summary

2.1. Background & Motivation

• Core problem: Existing audio-language models (ALMs like CLAP) excel at aligning monophonic audio with text but largely ignore spatial attributes (e.g., direction of arrival, room acoustics). Conversely, SELD systems (SELD: sound event localization and detection) localize and detect predefined classes but do not use natural language and lack cross-modal alignment.
• Importance: Spatial hearing (what and where) is essential for human-like audio perception and downstream tasks (robotics, AR/VR, hearing aids). Bridging audio semantics and spatial attributes with language unlocks search, understanding, and editing tasks that require both content and location.
• Gap:
  • ALMs trained on mono audio cannot perceive spatial scenes.
  • SELD models are task-specific, category-bound, and not language-aligned.
  • Lack of paired spatial audio–language datasets.
• Entry point/innovation: SALM proposes a dual-branch audio encoder (semantic and spatial branches) aligned with a text encoder through multi-modal contrastive learning. It further structures embeddings so that spatial editing can be performed by substituting spatial embeddings with directional text embeddings.

2.2. Main Contributions / Findings

• A new model: SALM, a spatial audio-language model that:
  • Decomposes spatial audio into Audio Semantic Embeddings and Audio Spatial Embeddings, then fuses them via a learnable weighted sum into Joint Audio Embeddings.
  • Aligns audio and text using contrastive losses, and adds a DOA (direction-of-arrival) loss from FOA-based DOA labels (synthetic and measured SRIRs).
• A paired spatial audio-language dataset creation pipeline:
  • Spatial audio simulation (FOA) from mono clips using synthetic/measured SRIRs.
  • Spatial caption generation using LLaMA-derived rephrasing with explicit directional tokens (8-direction set).
• Capabilities demonstrated:
  • Strong cross-modal spatial semantic retrieval on synthetic datasets (sClotho, sAudioCaps) and robust generalization to measured SRIR test sets (sClotho-R, sAudioCaps-R).
  • Near-perfect zero-shot direction classification (8 classes) using spatial or joint embeddings.
  • Spatial audio editing: changing direction by replacing spatial embeddings with normalized directional text embeddings—preserving semantics while altering direction.
• Results:
  • Significant improvements over LAION-CLAP baselines on spatial retrieval.
  • Low localization error (down to ~1.2–1.8 degrees in synthetic evaluation; larger but improved errors on real-scene SRIR sets).

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

• Spatial audio: Audio that preserves directional and spatial cues, enabling perception of where sounds originate in 3D space.
• FOA (First-Order Ambisonics): A spatial audio format with four channels (W, X, Y, Z) encoding the sound field, independent of the microphone array geometry, commonly used due to its flexibility.
• SRIR (Spatial Room Impulse Response): A room’s impulse response measured or simulated at specific positions in 3D space, used to convolve dry (mono) audio for realistic spatialization.
• DOA (Direction of Arrival): The 3D direction from which a sound source reaches the microphone; can be represented as azimuth/elevation angles or a 3D Cartesian unit vector.
• Contrastive learning (multimodal): Learning representations that bring matching pairs (e.g., audio-text) closer and push non-matching pairs apart in an embedding space. CLAP and CLIP are archetypal examples aligning audio/image with text.
• Audio-language models (ALMs): Models like CLAP trained on large audio-text pairs for retrieval and zero-shot classification; typically lack spatial conditioning.
• SELD (Sound Event Localization and Detection): A task jointly detecting sound categories and localizing their DOA. Two key paradigms:
  • ACCDOA: Activity-coupled Cartesian DOA representation fuses detection and localization into a single vector per class (magnitude = activity, direction = DOA).
  • EINV2: Decouples detection from localization via two sub-networks, mitigating entanglement.
• Intensity vectors: Spatial features derived from FOA channels estimating acoustic intensity direction, aiding DOA estimation.
• Transformers:
  • RoBERTa: A bidirectional transformer for language modeling used here as the base of the text encoder (as in LAION-CLAP).
  • Swin Transformer: A hierarchical vision transformer with shifted windows; HTS-AT adapts a Swin-like architecture to audio, providing strong time-frequency modeling.

3.2. Previous Works

• CLAP and LAION-CLAP: Built audio-text alignment with contrastive losses, achieving robust zero-shot classification/retrieval on mono audio. Limitations: no explicit modeling of spatial cues.
• Audio-aware LLMs: E.g., Pengi, LLaMA-based models, Qwen2-Audio—improve audio understanding but not designed as general spatially-aware embedding learners.
• Spatial audio LLMs / reasoning (BAT; spatial QA efforts): Task-specific spatial reasoning, less focused on broad, task-agnostic embedding alignment.
• ELSA (NeurIPS 2024): A CLAP-like approach aligning spatially-augmented audio with text, pioneering spatially-aware multimodal embeddings.
• SELD families:
  • ACCDOA, Multi-ACCDOA: Coupling detection+DOA, allowing overlapping same-class sources via permutation strategies.
  • EINV2 and variants: Decoupling detection from DOA estimation for better flexibility and robustness.
  • PSELDNets: Pre-trained SELD foundation models showing gains from pretraining on large synthetic spatial data.

3.3. Technological Evolution

• From mono audio-text alignment (CLAP) → initial spatial audio-text alignment (ELSA) → SALM adds structured embeddings (explicit semantic/spatial branches), soft parameter sharing, joint-and-decoupled losses, and text-driven spatial editing within embeddings. It also leverages pre-trained checkpoints from both LAION-CLAP (audio semantic branch) and PSELDNets (audio spatial branch) to bootstrap spatial and semantic capacities.

3.4. Differentiation Analysis

• Compared to CLAP/LAION-CLAP: SALM inputs all FOA channels and explicitly separates spatial and semantic representation paths while still allowing joint fusion—addressing spatial understanding absent in prior ALMs.
• Compared to ELSA: SALM’s architectural decoupling (dual-branch with soft parameter sharing) plus a DOA supervision term capitalizes on explicit spatial labels from SRIRs; SALM also demonstrates spatial audio editing by swapping in directional text embeddings.
• Compared to SELD models: SALM is task-agnostic and language-aligned, not limited to predefined categories, and supports cross-modal retrieval and editing.

4. Methodology

4.1. Principles

Core idea: Learn a shared embedding space where:

• Textual descriptions (with or without explicit spatial cues) and spatial audio map to aligned embeddings.
• Spatial audio is decomposed into semantic and spatial components via a dual-branch audio encoder. These can be:
  • Used independently (e.g., spatial-only or semantic-only tasks),
  • Or fused into a joint embedding through a learned, dimension-wise weighting.
• Contrastive losses enforce cross-modal alignment; an auxiliary DOA loss leverages SRIR-derived spatial labels to strengthen spatial grounding.
• Editing is enabled by directly manipulating the structured embedding: replace the spatial embedding with a normalized directional text embedding to change perceived direction while preserving semantics.

4.2. Architecture and Data Flow

The following figure (Figure 1 from the original paper) shows the system architecture:

该图像是 Spatial Audio Language Model (SALM) 的网络架构示意图，展示了音频输入和文本输入的编码过程，包括音频编码器的空间和语义分支以及相应的多层感知器层（MLP Layer）。该结构用于提取音频和文本的嵌入，并计算对比损失以实现跨模态对齐。

4.2.1. Inputs and Encoders

• Text input:

  • Original captions → Text Embeddings.
  • Spatially enriched captions → Spatial Text Embeddings.
  • Text encoder: based on LAION-CLAP’s RoBERTa text encoder (initialized from its pre-trained checkpoint).

• Audio input:

  • Spatial audio in FOA (WXYZ) format.

  • Semantic branch input: omnidirectional channel (W) only.

  • Spatial branch input: all FOA channels (WXYZ), along with intensity-vector features.

  • Both branches use HTS-AT (a Swin-Transformer-based audio transformer).

    Initialization:

• Audio Semantic Branch: initialized from the LAION-CLAP audio encoder checkpoint (monophonic semantics).

• Audio Spatial Branch: initialized from the DOA branch in the EINV2 variant of PSELDNets (spatial localization).

  Soft parameter sharing:

• The two branches are connected by learnable parameters (dotted lines in Figure 1), allowing information exchange while maintaining functional decoupling.

4.2.2. Embedding Projections

• Each encoder (Text, Audio Spatial, Audio Semantic) produces a 768-dimensional embedding.
• Each is projected via its own two-layer MLP to a unified 512-dimensional space (so all embeddings are dimensionally consistent).

4.2.3. Joint Embedding Construction

The audio encoder outputs:

• Audio Spatial Embeddings: $E_{\mathtt{ASp}} \in \mathbb{R}^{512}$

• Audio Semantic Embeddings: $E_{\mathtt{ASe}} \in \mathbb{R}^{512}$

  They are merged into Joint Audio Embeddings $E_{\mathtt{JA}} \in \mathbb{R}^{512}$ using the paper’s exact equation: $ E _ { \mathtt { J A } } = E _ { \mathtt { A S e } } + \mathbf { s } \odot E _ { \mathtt { A S p } } , $ where:

• $E_{\mathtt{ASe}}$ is the semantic embedding,

• $E_{\mathtt{ASp}}$ is the spatial embedding,

• $\mathbf{s} \in \mathbb{R}^{512}$ is a vector of learnable weights,

• $\odot$ denotes the Hadamard (element-wise) product.

  Intuition: The model learns per-dimension weights to balance the contributions of spatial and semantic information into a single, comprehensive representation.

4.3. Training Objectives (Cross-Modal Alignment + DOA Supervision)

Following LAION-CLAP and ELSA, SALM uses batched contrastive losses to align corresponding audio-text samples and repel mismatched pairs. Given synthetic rooms with known source positions, a DOA loss further supervises spatial grounding.

The overall loss (exactly as in the paper) is: $ \mathcal { L } _ { \mathrm { S A L M } } = \frac { 1 } { 2 } ( \mathcal { L } _ { \mathrm { C L } } + \mathcal { L } _ { \mathrm { s C L } } ) + \mathcal { L } _ { \mathrm { D O A } } , $ where:

• $\mathcal{L}_{\mathrm{CL}}$: contrastive loss between Audio Semantic Embeddings and Text Embeddings (aligns content),

• $\mathcal{L}_{\mathrm{sCL}}$: contrastive loss between Joint Audio Embeddings and Spatial Text Embeddings (aligns spatial semantics),

• $\mathcal{L}_{\mathrm{DOA}}$: cosine distance loss between the predicted and target DOA vectors.

  Implementation details described in the paper:

• A two-layer MLP takes the 512-D Audio Spatial Embeddings and predicts Cartesian DOA (3D unit vector).

• $\mathcal{L}_{\mathrm{DOA}}$ is computed as the cosine distance between predicted and ground-truth DOA vectors.

• The combined objective contains one aggregated term $\mathcal{L}_{\mathrm{sCL}}$ and two decoupled terms $\mathcal{L}_{\mathrm{CL}}$ and $\mathcal{L}_{\mathrm{DOA}}$, allowing the model to enhance semantic understanding and spatial perception jointly and independently.

4.4. Structured Embeddings and Text-Guided Editing

Beyond representation learning, SALM introduces a mechanism to edit spatial audio embeddings using text-derived direction embeddings.

Text-driven direction embedding:

• Prepare a directional sentence such as “The sound is coming from the southwest.”

• Encode it via the text encoder to get a direction embedding $E_{\mathrm{TDi}} \in \mathbb{R}^{512}$.

  Editing operation (exactly as in the paper): $ \tilde { E _ { \mathtt { J A } } } = E _ { \mathtt { A S e } } + \mathbf { s } \odot \frac { | E _ { \mathtt { A S p } } | \cdot E _ { \mathtt { T D i } } } { | E _ { \mathtt { T D i } } | } , $ where:

• $\tilde{E}_{\mathtt{JA}}$ is the edited joint embedding,

• $\| \cdot \|$ is the L2 norm (Euclidean norm),

• $E_{\mathtt{ASp}}$ sets the scale for the directional substitution to keep magnitude consistent,

• The fraction normalizes $E_{\mathrm{TDi}}$ and scales it to match $\|E_{\mathtt{ASp}}\|$.

  Intuition:

• Preserve audio semantics via $E_{\mathtt{ASe}}$.

• Replace spatial content by inserting a normalized directional text embedding, then fuse using the same weight vector $\mathbf{s}$.

• This yields a new joint embedding whose spatial interpretation corresponds to the text-specified direction while keeping the underlying content stable.

5. Experimental Setup

5.1. Datasets

• Base caption datasets:

  • AudioCaps: 50,956 audio clips with human captions (from AudioSet/Freesound).
  • Clotho: 5,929 audio clips with 5 captions per clip (Freesound-based).

• Spatial augmentation pipeline (paired spatial audio–language):

  1. FOA spatialization:
     • Convolve mono audio with FOA-format SRIRs.
     • Synthetic SRIRs: simulated shoebox rooms with varied geometries and frequency-dependent surface absorption; a virtual microphone at room center captures SRIRs from multiple locations.
     • Measured SRIRs: TAU-SRIR DB to approximate real acoustic environments.
  2. Spatial caption generation:
     • Map numeric azimuths to 8 direction classes at 45° intervals (e.g., north, northeast, east, …).
     • Prompt LLaMA 3.2-3B (temperature 0.2) to generate concise spatially enriched captions. Prompt: The sound: is coming from the . Rephrase the sentence in English to concisely describe the sound detail and the direction of its source.

• Resulting datasets:

  • sClotho: 17,787 spatial audio clips (~111 hours), derived by creating three spatial variants per original clip.

  • sAudioCaps: 152,868 spatial audio clips (~419 hours), similarly augmented.

  • Real-scene counterparts (measured SRIRs): sClotho-R, sAudioCaps-R (constructed at equivalent scale), used for testing generalization and for additional training augmentation in some runs.

    Example data form:

• Audio: 10-second FOA (WXYZ) segments at 24 kHz sampling rate.

• Text: Original and spatially enriched captions (e.g., “A bell rings from the northeast.”).

  Rationale: Creating paired spatial audio–language data addresses the absence of public datasets with explicit spatial annotations, enabling training and evaluation of spatial audio–language alignment and spatial understanding.

5.2. Features and Preprocessing

• Audio features:
  • 64-dimensional log mel spectrograms extracted from FOA signals with a 1024-point Hann window and 240-sample hop.
  • Intensity vectors computed from FOA used in the spatial branch (as in PSELDNets).
• Segment handling:
  • Fixed 10 s segments for training and inference.
  • Clips <10 s: repeated then zero-padded.
  • Clips >10 s: randomly cropped to 10 s.

5.3. Training Protocol

• Optimizer: Adam.
• Batch size: 64.
• Learning rate schedule: linear warm-up for 3 epochs, then cosine decay for 7 epochs.
• Base learning rate: $10^{-4}$.
• Encoders initialized from pre-trained checkpoints as described in Section 4 (text from LAION-CLAP, audio semantic from LAION-CLAP, audio spatial from PSELDNets DOA branch).

5.4. Evaluation Metrics

5.4.1. Recall@K for Cross-Modal Retrieval

• Concept: Measures how often the correct match appears within the top-K retrieved items for each query (text-to-audio and audio-to-text).
• Formula (standard): $ \mathrm{R@K} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\left[\mathrm{rank}(y_i \mid q_i) \le K\right], $ where:
• $N$: number of queries,
• $q_i$: the $i$-th query (text or audio),
• $y_i$: the correct target (audio or text) for $q_i$,
• $\mathrm{rank}(y_i \mid q_i)$: rank position of $y_i$ among all candidates sorted by similarity to $q_i$,
• $\mathbf{1}[\cdot]$: indicator function (1 if the condition is true, else 0).

5.4.2. Localization Error (Angular Distance)

• Concept: Measures the average angular deviation between predicted and ground-truth directions of arrival (DOA).

• Formula (standard 3D): $ \theta = \arccos\left(\frac{\hat{\mathbf{d}} \cdot \mathbf{d}}{|\hat{\mathbf{d}}|,|\mathbf{d}|}\right), $ where:

• $\hat{\mathbf{d}} \in \mathbb{R}^3$: predicted Cartesian DOA vector,

• $\mathbf{d} \in \mathbb{R}^3$: ground-truth Cartesian DOA vector,

• $\cdot$: dot product,

• $\|\cdot\|$: L2 norm.

• Reported as an average over samples, typically in degrees.

  Similarity computation for retrieval: cosine similarity between embeddings, consistent with CLAP-style evaluation.

5.5. Baselines and Model Variants

• LAION-CLAP: Uses only an omnidirectional channel, no explicit spatial modeling. Serves as a strong monophonic audio-text alignment baseline, but expected to underperform on spatial tasks.
• SALM-s: A SALM variant using only the Audio Spatial Branch (all FOA channels) without the dual-branch fusion—helps isolate the benefit of spatial branch vs. CLAP.
• SALM (full): Dual-branch (semantic+spatial) with soft parameter sharing and combined losses. Several configurations differ by which loss terms are included (as reported in Table 1).

6. Results & Analysis

6.1. Core Results Analysis

• On synthetic spatial datasets (sClotho, sAudioCaps), SALM improves substantially over LAION-CLAP for both text-to-audio and audio-to-text retrieval at R@1/5/10, and yields very low localization errors when DOA supervision is used.
• The SALM-s variant (spatial branch only) already surpasses LAION-CLAP, emphasizing the importance of FOA spatial cues even without semantic fusion.
• Adding the semantic branch and the decoupled contrastive term $\mathcal{L}_{\mathrm{CL}}$ further improves both retrieval and localization (lower degrees), showing the benefit of decoupling and jointly training spatial and semantic embeddings with appropriate supervision.
• Generalization to measured SRIR datasets (sClotho-R, sAudioCaps-R):
  • Training only on simulated SRIRs yields reasonable performance on real-scene SRIR tests.
  • Adding measured SRIRs to training further boosts recalls and significantly reduces localization error (e.g., from ~12.6–12.7° down to ~7.5–7.6°), indicating robust transfer.
• Zero-shot direction classification (8 classes) achieves near-perfect accuracy (>99.9%) with spatial or joint embeddings; semantic-only embeddings cannot classify direction (as expected), validating the decoupling of spatial content.
• Spatial audio editing:
  • Two operations: “Swap” (replace spatial embedding with same-direction text embedding, i.e., direction-invariant editing) and “Change” (replace with a different-direction text embedding).
  • Retrieval performance remains largely stable after editing, indicating that joint embeddings are well-structured: semantic content is preserved while direction is controlled textually.

6.2. Data Presentation (Tables)

The following are the results from Table 1 of the original paper:

Interpretation:

• LAION-CLAP has low recalls; missing spatial info.

• SALM-Ss (spatial-only) dramatically increases retrieval and achieves low localization error.

• Full SALM configurations further improve both retrieval and localization; including the decoupled contrastive term appears to provide regularization benefits aligning semantic and textual content.

  The following are the results from Table 2 of the original paper:

  Training Datasets	sClotho-R							sAudioCaps-R
  	Text-to-Audio			Audio-to-Text			Local.	Text-to-Audio			Audio-to-Text			Local.
  	R@1	R@5	R@10	R@1	R@5	R@10	Error	R@1	R@5	R@10	R@1	R@5	R@10	Error
  sAudioCaps, sClotho	6.3%	18.2%	26.3%	6.1%	17.7%	25.1%	12.7°	13.5%	36.3%	48.7%	13.7%	33.5%	44.4%	12.6
  + sAudioCaps-R, sClotho-R	7.2%	22.2%	32.2%	7.8%	21.9%	30.7%	7.6°	16.4%	42.6%	55.9%	21.4%	46.4%	56.7%	7.5°

Interpretation:

• Training solely on simulated SRIRs gives reasonable generalization to measured SRIR test sets.

• Adding measured SRIRs to the training set substantially improves both recall and localization error on real scenes, underlining the importance of domain coverage.

  The following are the results from Table 3 of the original paper:

  Feature	sClotho	sAudioCaps
  Audio Sem. Embed.	8.2%	12.2%
  Audio Spat. Embed.	99.9%	100.0%
  Joint Audio Embed.	99.9%	100.0%

Interpretation:

• Spatial and joint embeddings are highly aligned with directional text queries (8 classes), whereas semantic embeddings alone (W-channel only) do not encode directionality.

6.3. Spatial Audio Editing Results (Qualitative)

As can be seen from the results in Figure 2 of the original paper, the authors evaluate how editing affects retrieval performance:

该图像是图表，展示了在编辑空间音频时的空间检索性能。图表分为两个部分：左侧是 sClotho 数据集，右侧是 sAudioCaps 数据集。'None'、'Swap' 和 'Change' 代表不进行编辑、方向不变的编辑和方向修改，Y轴显示了性能值（%）。

• “None”: baseline, no editing.

• “Swap”: replace the spatial embedding with a text embedding derived from the same direction (direction-invariant editing).

• “Change”: replace the spatial embedding with a text embedding for a different direction (direction modification).

  Observation: Both “Swap” and “Change” minimally degrade retrieval performance, implying that the joint embeddings maintain semantic integrity while enabling controllable spatial directionality.

6.4. Ablations and Component Effects

• SALM-s vs LAION-CLAP: Using all FOA channels and a spatially focused branch significantly enhances retrieval, showing that spatial cues are critical.
• Dual-branch vs single-branch: Adding the semantic branch and soft parameter sharing reduces localization error by ~60% (from 4.2° to ~1.6–1.8° in sClotho), underscoring the benefit of jointly modeling semantics and spatial features.
• Loss terms: Including the decoupled $\mathcal{L}_{\mathrm{CL}}$ serves as a regularizer aligning semantic audio and text; combined with $\mathcal{L}_{\mathrm{sCL}}$ and $\mathcal{L}_{\mathrm{DOA}}$, it yields the best overall performance.

7. Conclusion & Reflections

7.1. Conclusion Summary

• SALM introduces structured spatial audio-language embeddings via a dual-branch audio encoder (semantic + spatial) aligned with a text encoder using contrastive learning and DOA supervision.
• The approach achieves:
  • Strong spatial audio–text retrieval,
  • Very low localization error on synthetic datasets and strong generalization to measured SRIRs,
  • Near-perfect zero-shot direction classification (8 classes),
  • Text-driven spatial audio editing by replacing the spatial embedding with normalized directional text embeddings.
• Conceptually, SALM provides a foundation for spatially aware multimodal understanding and flexible editing in embedding space.

7.2. Limitations & Future Work

Author-indicated or implied limitations:

• Dataset construction relies on synthetic spatialization and measured SRIRs; although results generalize well, further validation on fully real, in-the-wild spatial recordings would strengthen claims.

• Directional text labels discretized into 8 directions for caption generation and evaluation; finer-grained elevation/azimuth control and continuous direction text may require expanded prompts and evaluation.

• The editing is performed in embedding space; end-to-end waveform-level spatial editing or generative resynthesis is beyond the scope here.

• Training/inference on fixed 10-second segments may limit temporal granularity or real-time applications; streaming or variable-length handling could be explored.

• Overlapping multi-source scenes and complex spatial layouts are not deeply analyzed; future work could extend to multi-source, dynamic scenes with richer labels.

  Potential future directions:

• Integrate generative models to synthesize spatial audio waveforms from edited embeddings (text-guided spatial audio rendering).

• Expand direction vocabularies and continuous spatial text control, including elevation and distance descriptors.

• Broaden to other spatial audio formats (higher-order Ambisonics, binaural), sensor arrays, and device-specific HRTFs.

• Evaluate on real-world, multi-source recordings with annotated scene graphs and extend to spatial question answering at scale.

• Incorporate richer environmental attributes (reverberation, occlusions) into textual descriptions and embedding manipulation.

7.3. Personal Insights & Critique

• Strengths:
  • Clear architectural decoupling of semantics and spatial content with a principled fusion mechanism.
  • Effective use of pre-trained checkpoints (semantic from LAION-CLAP, spatial from PSELDNets) to bootstrap learning.
  • Practical editing recipe with normalization to maintain scale consistency.
  • Strong empirical evidence across synthetic and measured SRIR settings.
• Points to examine:
  • The reliance on 8-direction discretization for captions may limit nuanced spatial language understanding (e.g., “slightly behind-left,” elevation changes). A continuous spatial-text mapping would be compelling.
  • While localization errors are low in synthetic settings, real-scene errors remain higher; more diverse measured SRIR training or domain adaptation might further close the gap.
  • The work assumes availability of clean single-source spatialization per sample; scenarios with multiple simultaneous sources and occlusions need deeper exploration.
  • The DOA loss uses cosine distance; future work might test alternative geodesic losses or uncertainty-aware DOA outputs, especially in reverberant or low-SNR conditions.
• Transferability:

  • The structured embedding approach (separate spatial and semantic embeddings with learnable fusion) could benefit other multimodal domains where content and context should be decoupled and recomposed (e.g., vision-language with spatial relationships, robotics navigation with natural language instructions).

  • Text-driven spatial editing in embedding space suggests analogous mechanisms for controlling other nuisance or context variables (e.g., style, timbre, or environment) via targeted text embeddings.

    In sum, SALM is a solid step toward general-purpose spatial audio–language alignment with practical editing capabilities, and it sets a strong foundation for future spatial reasoning, generation, and interactive applications.