扩展推理时间推理的标准技术是自我一致性,，其中从法学硕士中抽取多个候选答案，并选择最常见的答案。最近, 已表明加权多数投票(e.g. Confidence-Informed Self Consistency (CISC)), 为每个候选答案分配一个置信值，并选择具有最大累积分数, 的答案，在各种流行的基准测试中往往更准确。实际上，, 加权多数投票需要对每个候选人的推理轨迹调用批评者法学硕士，以产生答案的置信度分数。尽管具有潜在的性能优势，但 LLM 调用的第二系列极大地增加了加权多数投票, 的开销和成本。为了减少这种费用,，我们提出了 VecCISC, 一个轻量级的, 自适应框架，该框架使用语义相似性度量来过滤语义上与其他, 退化, 或幻觉, 等效的推理痕迹，从而减少评论家必须评估的候选答案的数量。为了确保足够的实验彻底性,，我们在五个具有挑战性的,广泛采用的数据集上评估VecCISC，这些数据集涵盖数学,化学,生物学,常识推理,和人文学科。我们的结果表明，VecCISC 将总代币使用量减少了 47%,，同时保持或超过 CISC 的准确性。
A standard technique for scaling inference-time reasoning is Self-Consistency, whereby multiple candidate answers are sampled from an LLM and the most common answer is selected. More recently, it has been shown that weighted majority voting (e.g. Confidence-Informed Self Consistency (CISC)), which assigns a confidence value to each candidate answer and chooses the answer with the largest accumulated score, tends to be more accurate on a wide range of popular benchmarks. In practice, weighted majority voting necessitates calling a critic LLM on each candidate的 reasoning trace to produce the answer的 confidence score. This secondary series of LLM calls greatly increases the overhead and cost of weighted majority voting, despite its potential performance benefits. To reduce this expense, we propose VecCISC, a lightweight, adaptive framework that uses a measure of semantic similarity to filter reasoning traces that are semantically equivalent to others, degenerate, or hallucinated, thus decreasing the number of candidate answers that must be evaluated by the critic. To ensure adequate experimental thoroughness, we evaluate VecCISC on five challenging, widely-adopted datasets spanning the domains of mathematics, chemistry, biology, commonsense reasoning, and the humanities. Our results demonstrate that VecCISC reduces the total token usage by 47%, while maintaining or exceeding the accuracy of CISC.
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)