校准将模型'的预测不确定性与其经验结果的频率保持一致，对于理解和信任报告的概率非常重要。最近的工作表明，在单个预测变量的水平上实施校准可以提高整体精度和校准,，专家混合(MoE)模型显示出很强的经验改进，特别是;，但是,校准帮助MoE的条件还没有得到很好的理解。在这项工作, 中，我们研究 MoE 模型在分布偏移, 下的表现，重点关注路由机制如何与专家级校准交互。我们表明，专家校准足以确保在硬路由模型, 的广泛分布变化下校准整个模型，但不足以校准软路由模型。为了解决这个,，我们提出了一种对抗性重新加权，它可以惩罚分布偏移,下路由聚合的校准误差，并且我们证明它可以改善跨模型类,预测任务,和分布偏移的数据,的平均和困难子集上的精度校准权衡。
Calibration aligns a model的 predictive uncertainty with the frequencies of its empirical outcomes and is important for understanding and trusting reported probabilities. Recent work shows that enforcing calibration at the level of individual predictors can improve ensemble accuracy and calibration, with mixture-of-experts (MoE) models showing strong empirical improvements in particular; however, the conditions under which calibration helps MoE are not well understood. In this work, we study how MoE models behave under distribution shift, focusing on how routing mechanisms interact with expert-level calibration. We show that expert calibration is sufficient to ensure calibration of the overall model under a broad class of distribution shifts in hard-routed models, but is insufficient for calibrating soft-routed models. To address this, we propose an adversarial reweighting that penalizes calibration errors of the routed aggregate under distribution shift, and we demonstrate that it improves the accuracy-calibration tradeoff both on average and on difficult subsets of the data, across model classes, prediction tasks, and distribution shifts.
科目: 人工智能 (cs.AI); 机器学习 (cs.LG)
Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)