多模态情感和情感识别通常通过早期融合,（在分类, 之前连接模态）或后期融合,（结合独立训练的单模态预测器）来解决。早期融合可以是准确的，但整体,，而后期融合是模块化的，但可能会失去跨模式交互。本文重新审视了 XAI 引导的自适应融合(\xgaf),，这是一种基于树的单模态和跨模态专家的混合，其样本级权重源自 TreeSHAP 归因量值。我们重点关注当专家具有不相等的特征维度时 SHAP 归因减少的效果。在这种情况下，, 均值-abs 和中值-abs 减少可以抑制高维跨模态专家,，而总和-abs 减少则保留总归因质量。在 MELD 7 类情感识别, sum-abs \xgaf{} 上几乎匹配三个面部序列聚合器; 的早期融合，Transformer 变体达到 0.5983 \wf{},，而早期融合为 0.6018，后期融合概率平均为 0.4598。 McNemar 测试显示 sum-abs \xgaf{} 与 MELD ($p=1.000$), 的早期融合之间没有显着差异，而 \xgaf{} 仍然明显优于晚期融合 ($p<0.0001$)。在CMU-MOSEI 3级情感识别, sum-abs \xgaf{}上达到0.6519 \wf{},略超过早期融合(0.6485)和后期融合(0.5696)。消融研究表明，主要收益来自添加跨模态专家,，尤其是三模态专家,，而不是复杂的每样本路由。诊断进一步表明，平均腹肌和中值腹肌权重几乎一致,，而总和腹肌权重集中在三峰专家身上。因此,的主要贡献是对SHAP缩减,专家维度,和跨模式专家设计如何影响模块化多模式融合的透明实证分析。

Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors. Early fusion can be accurate but monolithic, while late fusion is modular but may lose cross-modal interactions. This paper revisits XAI-guided adaptive fusion (\xgaf), a tree-based mixture of unimodal and cross-modal experts whose sample-level weights are derived from TreeSHAP attribution magnitudes. We focus on the effect of SHAP attribution reduction when experts have unequal feature dimensionalities. In this setting, mean-abs and median-abs reductions can suppress high-dimensional cross-modal experts, whereas sum-abs reduction preserves total attribution mass. On MELD 7-class emotion recognition, sum-abs \xgaf{} nearly matches early fusion across three face-sequence aggregators; the Transformer variant reaches 0.5983 \wf{}, compared with 0.6018 for early fusion and 0.4598 for probability-average late fusion. McNemar testing shows no significant difference between sum-abs \xgaf{} and early fusion on MELD ($p=1.000$), while \xgaf{} remains significantly better than late fusion ($p<0.0001$). On CMU-MOSEI 3-class sentiment recognition, sum-abs \xgaf{} reaches 0.6519 \wf{}, slightly exceeding early fusion (0.6485) and late fusion (0.5696). Ablation studies show that the main gain comes from adding cross-modal experts, especially the trimodal expert, rather than from complex per-sample routing. Diagnostics further show that mean-abs and median-abs weights are nearly uniform, while sum-abs weights concentrate on the trimodal expert. Thus, the main contribution is a transparent empirical analysis of how SHAP reduction, expert dimensionality, and cross-modal expert design affect modular multimodal fusion.

科目:人工智能(cs.AI)

Subjects: Artificial Intelligence (cs.AI)