训练后量化已广泛用于压缩大型语言模型，使其可部署在资源受限的设备上。然而,量化方法的评估主要使用准确性和困惑度,，无法捕获量化变体中的行为变化。在这项工作,中，我们提出了正确性协议,，这是一种决策级指标，可以测量基本模型与其量化变体之间正确预测的交集。我们在多个模型和量化位级别(8位到2位),中使用此指标，我们发现即使保留了准确性和困惑度，基本变体和量化变体通常也会发生行为变化。为了解释这种效应,，我们使用统计和分布测量来研究量化对注意力权重结构的影响。结果揭示了低位宽处的断点，并表明查询和键投影比值和输出投影对量化更敏感。这些结果证明了基础模型和量化模型之间等效性的错觉，并激发了超越量化方法的困惑和准确性的行为评估。
Post-Training Quantization has become widely used to compress large language models to make them deployable on resource-constrained devices. However, the evaluation of quantization methods mainly uses accuracy and perplexity, which cannot capture the behavioral changes in the quantized variants. In this work, we propose Correctness Agreement, a decision-level metric that can measure the intersection of correct predictions between the base model and its quantized variant. We use this metric across multiple models and quantization bit levels (8-bit to 2-bit), and we find that the base and quantized variants usually have a shift in behavior even when accuracy and perplexity are preserved. In order to explain this effect, we study the effect of quantization on the structure of the attention weights using statistical and distributional measures. The results reveal a breakpoint at low bit widths and show that query and key projections are more sensitive to quantization than the value and output projections. These results prove the illusion of equivalency between the base and quantized models and inspire behavioral evaluation beyond perplexity and accuracy for quantization methods.
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)