长上下文 LLM 推理越来越受到 KV 缓存, 的内存和带宽成本的限制，但积极的压缩可以消除检索和多步推理所需的特定于层的证据。我们引入了 FreqDepthKV, 一种推理时间缓存压缩方法，它将相邻层 KV 状态分解为共享低频深度分量和稀疏高频残差。轻量级在线探针根据注意力头对重建敏感注意力日志,的贡献将注意力头分配给共享深度,残差深度,或精确缓存模式，从而允许压缩策略适应提示结构而无需重新训练。在长上下文问答, 针检索, 总结, 和代码生成基准, FreqDepthKV 中，在大幅减少缓存预算的情况下保持任务准确性。使用 32k 令牌预填充窗口, FreqDepthKV 达到 58.3 Exact Match, 63.0 F1, 32.5 ROUGE-L, 和 48.1 pass@1, 紧密匹配完整 KV，同时优于以前的压缩缓存方法。它还将解码吞吐量提高到 70.4 token/s,，将 TTFT 降低到 2.06 秒,，并将峰值 KV 内存降低到 6.2 GB,，实现 3.9 倍的有效压缩比。
Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning. We introduce FreqDepthKV, an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes according to their contribution to reconstruction-sensitive attention logits, allowing the compression policy to adapt to prompt structure without retraining. Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets. With a 32k-token prefill window, FreqDepthKV reaches 58.3 Exact Match, 63.0 F1, 32.5 ROUGE-L, and 48.1 pass@1, closely matching full KV while outperforming prior compressed-cache methods. It also improves decoding throughput to 70.4 tokens/s, reduces TTFT to 2.06 seconds, and lowers peak KV memory to 6.2 GB, achieving a 3.9x effective compression ratio.
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)