自动语音识别主要由一次发出一个标记的自回归解码器主导。我们询问离散扩散语言模型是否可以转录语音，而不是,通过少量的去噪步骤并行精炼整个转录。我们为 DiffusionGemma, 训练一个音频原生接口，这是一个 26B 专家混合模型，该模型通过统一的, 随机令牌离散扩散而不是最近扩散语言模型常见的吸收掩模方案来生成文本。冻结的 Whisper 编码器提供声学特征,，轻型投影仪将它们映射到模型嵌入空间,，低阶适配器让冻结的骨干网参与新的模态。训练了大约 42M 参数,，占主干网的 0.16%。我们发现自然训练目标无法使音频接地，因为它们的梯度只能通过已经忽略它的注意力到达投影仪。通过冻结输出头应用的联结主义时间分类损失打破了这一僵局。生成的模型在 LibriSpeech test-clean, 上达到 6.6% 的单词错误率，无论话语长度如何，都以大约八个并行步骤进行转录,，并使用在六种语言, 上训练的单个适配器，我们在此对英语, 印地语, 和普通话进行评估。
Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.
科目: 人工智能 (cs.AI); 声音 (cs.SD)
Subjects: Artificial Intelligence (cs.AI); Sound (cs.SD)