FP4 训练承诺大幅降低 LLM 预训练, 的内存和计算成本，但当前的 FP4 硬件路径和配方,（包括 NVIDIA Blackwell/Rubin 级系统和 AMD MI350 系列 GPU,）仍然以 E2M1 数据元素为中心。在这项研究中,，我们确定了该选择的基本限制: 非均匀格式（例如 E2M1）本质上会受到收缩偏差, 的影响，这是由其可表示箱的几何不对称性引起的系统负舍入误差。我们表明，这种偏差在各层之间乘法累积，并通过随机 Hadamard 变换 (RHT), 放大，为现有基于 E2M1 的 FP4 配方中观察到的训练不稳定性提供了统一的解释。相比之下，, 均匀网格(E1M2/INT4) 绕过了此网格几何误差，并更好地将 RHT 改进的桶利用率转换为更高的量化质量。基于这一发现,，我们提出 UFP4, 一种统一的 4 位训练方案，将 RHT 应用于所有三个训练 GEMM，同时将随机舍入限制为仅 dY。在密集 1.5B, MoE 7.9B, 和 MoE 124B 长期预训练, UFP4 上，与基于 E2M1 的强基线, 相比，UFP4 始终实现较低的 BF16 相对损失退化，并由缩放律分析和消融研究支持。我们的结果表明，未来的加速器应该支持 E1M2/INT4 风格的统一 4 位网格，作为与 E2M1 一样的一流训练原语。

FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. Our results suggest that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1.

科目:人工智能(cs.AI)

Subjects: Artificial Intelligence (cs.AI)