自动驾驶汽车通常由深度学习模型控制，有时会在意外情况下失败。例如,，汽车可能会莫名其妙地刹车并挡住迎面而来的紧急车辆的路径。人类驾驶员或乘客可能需要快速反应以防止碰撞。
Self-driving cars are often controlled by deep learning models that sometimes fail in unexpected situations. For instance, the car might inexplicably brake and block the path of an oncoming emergency vehicle. A human driver or passenger may need to react rapidly to prevent a collision.
为了帮助人类更好地预测车辆的 错误,，麻省理工学院和自动驾驶汽车技术公司 Motional 的研究人员开发了一种新方法，可以为底层模型的 决策提供清晰的解释。
To help humans better anticipate a vehicle的 mistakes, researchers from MIT and autonomous vehicle technology company Motional developed a new method that provides clear explanations of the underlying model的 decisions.
通常,深度学习模型的内部推理过程是不透明且难以理解的。但称为概念包装网络 (CW-Net), 的新方法, 将该推理过程转化为概念，忠实地描述自动驾驶车辆的决策，而不改变其驾驶性能。
Usually, the internal reasoning process of a deep learning model is opaque and difficult to understand. But the new method, called the Concept-Wrapper Network (CW-Net), translates that reasoning process into concepts that faithfully describe the autonomous vehicle的 decisions without altering its driving performance.
CW-Net 使用可理解的概念, 解释基于机器学习的规划人员的决策，例如“ 接近停止的车辆” 或“ 接近骑车人。” 这些解释可以纠正驾驶员和乘客对车辆行为的误解并提高他们的态势感知能力。
CW-Net explains the decisions of machine learning-based planners using understandable concepts, like “approaching stopped vehicle” or “close to cyclist.” These explanations can correct misconceptions drivers and passengers have about vehicle behavior and improve their situational awareness.
在私人赛道的道路测试中, CW-Net 解释帮助安全驾驶员更准确地预测车辆行为; 与非专家用户进行的更大规模的模拟研究得出了类似的结果。这些实验展示了 CW-Net 如何在工程师对车载人工智能系统进行故障排除时提供重要的反馈。从长远来看,，这项技术可以提高自动驾驶汽车,的安全性和透明度，同时建立对驾驶员和乘客的适当信任。
In road tests on a private track, CW-Net explanations helped safety drivers more accurately predict vehicle behavior; a larger simulation study with nonexpert users yielded similar results. These experiments show how CW-Net can provide important feedback for engineers as they troubleshoot in-vehicle artificial intelligence systems. In the longer term, this technique could boost the safety and transparency of autonomous vehicles, while building appropriate trust in drivers and passengers.
该论文的共同作者包括：第一作者 Eoin Kenny,（前麻省理工学院博士后，现任摩根大通高级人工智能研究员;）、共同高级作者 Momchil Tomov,（Motional; 的研究员科学家）以及 Motional 团队成员 Akshay Dharmavaram, Sang Uk Lee, Tung Phan-Minh, Shreyas Rajesh, Yunqing Hu, 和Laura Major, Motional 总裁兼首席执行官。这项研究今天发表在《自然》杂志上。
She is joined on the paper by lead author Eoin Kenny, a former MIT postdoc who is now a senior AI researcher at J.P. Morgan Chase; co-senior author Momchil Tomov, a staff research scientist at Motional; as well as Motional team members Akshay Dharmavaram, Sang Uk Lee, Tung Phan-Minh, Shreyas Rajesh, Yunqing Hu, and Laura Major, president and CEO of Motional. The research appears today in Nature.
基于机器学习的规划器充当自动驾驶汽车的“brain”。这些强大的深度学习架构处理来自车辆的 摄像头和激光雷达传感器, 的数据，生成车辆的 环境, 的高级摘要，决定汽车接下来应该做什么, 并输出其遵循的轨迹。
Machine-learning-based planners act as the “brain” of a self-driving car. These powerful deep-learning architectures process data from the vehicle的 cameras and lidar sensors, generate a high-level summary of the vehicle的 environment, decide what the car should do next, and output a trajectory for it to follow.
规划者通常是黑盒模型,，这意味着他们的内部决策过程非常复杂，很难理解。这可能会让科学家和安全驾驶员一头雾水，不明白为什么自动驾驶汽车会做出像幻影制动这样的意外决定,。
The planners are usually black-box models, which means their internal decision-making process is so complex it is difficult to understand. This can leave scientists and safety drivers in the dark about why an autonomous vehicle made an unexpected decision, like phantom braking.
研究人员设计 CW-Net 来使用可理解的概念, 解释车辆的决策，同时确保这些解释准确反映其行为背后的真正原因。
The researchers designed CW-Net to explain a vehicle的 decisions using understandable concepts, while ensuring those explanations accurately reflect the true reasons behind its behavior.
“特别是在自动驾驶汽车等高风险环境中,，’重要的是这些解释不会产生潜在的误导。因为 CW-Net 在做出决策时忠实于因果关系,，因此围绕肯尼所说的解释提供了一定的保证,”。
“Especially in high-stakes settings like self-driving cars, it的 important that the explanations are not potentially misleading. Because CW-Net is causally faithful in how it makes decisions, that provides certain guarantees around the explanations,” Kenny says.
CW-Net 是一个 “ 概念分类器,” 一种 AI 算法，经过训练可以预测输入数据中存在的高级概念。研究人员将 CW-Net 模块插入自动驾驶汽车 现有机器学习规划器架构的中间。
CW-Net is a “concept classifier,” an AI algorithm that has been trained to predict the high-level concepts that exist within input data. The researchers plug the CW-Net module into the middle of an autonomous vehicle的 existing machine-learning planner architecture.
它将模型的的内部推理过程转换为可理解的概念,，例如“接近停止的车辆”或“接近骑自行车的人。”然后，它强制规划模型架构的最后一部分在决定车辆下一步应该做什么时使用这些概念。通过这种方式，, CW-Net 确保概念忠实地解释车辆的 的动作。
It translates the model的 internal reasoning process into understandable concepts, like “approaching stopped vehicle” or “close to cyclist.” Then it forces the final piece of the planning model architecture to use those concepts when it decides what the vehicle should do next. In this way, CW-Net ensures the concepts faithfully explain the vehicle的 actions.
同时, CW-Net 使用它分类的概念来生成清晰的解释，并与车辆轨迹, 一起实时输出。
At the same time, CW-Net uses the concepts it classified to generate clear explanations that are output along with the vehicle trajectory, in real-time.
“不仅仅是想知道汽车为何停下,，实时数据提供的反馈可让您在部署期间测试系统。您还可以将该数据提供给工程师，以潜在地改进系统,” Kenny 说。
“Instead of just wondering why the car stopped, having real-time data provides feedback that lets you test the system during deployment. You could also give that data to an engineer to potentially improve the system,” Kenny says.
研究人员使用来自自动驾驶汽车, 的 1.3 亿个场景示例数据集训练 CW-Net 来预测概念，每个场景中都有多个标记概念。使用如此大的,标记数据集使其能够在各种设置中准确识别概念。
The researchers trained CW-Net to predict concepts using a dataset of 130 million examples of scenes from self-driving cars, with multiple labeled concepts in each scene. Using such a large, labeled dataset enables it to identify concepts accurately in a wide range of settings.
他们还设计了 CW-Net 来模仿基于机器学习的规划器, 的驾驶决策，因此该模块不会对车辆性能产生负面影响。
They also designed CW-Net to mimic the driving decisions of machine-learning-based planners, so the module would not negatively impact vehicle performance.
最终, CW-Net在不改变原始深度学习模型的情况下生成了准确的,可理解的解释。
In the end, CW-Net generates accurate, understandable explanations without altering the original deep learning model.
为了测试 CW-Net,，研究人员将该模块部署在私人赛道上的真实自动驾驶测试车辆 (a Motional Robotaxi) 上，并配有安全驾驶员。他们发现 CW-Net 可以帮助安全驾驶员更好地预测车辆在意外情况下的表现。
To test CW-Net, the researchers deployed the module on a real autonomous driving test vehicle (a Motional robotaxi) on a private track with a safety driver. They found that CW-Net helped the safety driver better predict how the vehicle would behave in surprising situations.
例如,，车辆在接近骑自行车者, 时始终停下来，安全驾驶员认为它这样做是因为它检测到了骑自行车者。但 CW-Net 的解释显示，该模型没有正确配置’t 来检测骑自行车的人，并选择了一条可能导致碰撞的轨迹。相反,它停了下来，因为它的紧急制动程序在距离太近时启动。
For instance, the vehicle consistently stopped when it approached a cyclist, and the safety driver assumed it did so because it detected that cyclist. But CW-Net explanations revealed that the model wasn’t properly configured to detect the cyclist and chose a trajectory that would have caused a collision. Instead, it stopped because its emergency braking procedure kicked in when it got too close.
有了有关模型的错误,的信息，安全驾驶员可以在类似情况下降低速度或更快地进入手动驾驶模式。这也可以帮助工程师修复模型，以避免将来出现这种故障。
Armed with this information about the model的 mistake, the safety driver could reduce speed or engage manual driving mode sooner in similar situations. This could also help engineers fix the model to avoid this failure in the future.
在使用拉斯维加斯, 道路上捕获的真实驾驶情况进行的大型在线模拟研究中，研究人员看到了类似的结果。 CW-Net 的解释显着提高了参与者 预测自动驾驶汽车行为的能力。
In larger online simulation studies using real driving situations captured on the roads of Las Vegas, the researchers saw similar results. CW-Net explanations significantly improved participants abilities to predict how an autonomous vehicle will behave.
在未来,，研究人员可以扩展 CW-Net，以便该模块可以涵盖更多概念并探索不同的训练和设计技术，从而提高性能并提高可解释性。
In the future, the researchers could extend CW-Net so the module can cover more concepts and explore different training and design techniques that could boost performance and improve interpretability.
“我们的研究表明，可解释性对于这些高风险环境是多么重要,，以及人们在未来为自动驾驶汽车或其他安全关键环境制造人工智能时应该如何考虑这一点,” Kenny 说。
“Our study shows how crucial interpretability can be to these high-stakes environments, and how it should be on the mind of people as they are making AI in the future, for self-driving cars or other safety-critical environments,” Kenny says.