Jianbiao Mei1,2,*, Yukai Ma1,2,*, Xuemeng Yang2, Licheng Wen2, Xinyu Cai2, Xin Li2,4, Daocheng Fu2, Bo Zhang2, Pinlong Cai2, Min Dou2, Botian Shi2,†, Liang He3, Yong Liu1,†, Yu Qiao2
1 Zhejiang University 2 Shanghai Artificial Intelligence Laboratory 3 East China Normal University 4 Shanghai Jiao Tong University
* Equal Contribution † Corresponding Authors
-
[2024-9-26]Our LeapAD is accepted by NeurIPS 2024! -
[2024-5-27]The paper can be accessed at arxiv. -
[2024-5-22]We released our project website here.
We introduce LeapAD, a novel paradigm for autonomous driving inspired by the human cognitive process. Specifically, LeapAD emulates human attention by selecting critical objects relevant to driving decisions, simplifying environmental interpretation, and mitigating decision-making complexities. Additionally, LeapAD incorporates an innovative dual-process decision-making module, which consists of an Analytic Process (System-II) for thorough analysis and reasoning, along with a Heuristic Process (System-I) for swift and empirical processing.
The scene understanding module analyzes surrounding images and provides descriptions of critical objects that may influence driving decisions. These scenario descriptions are then fed into the dual-process decision module for reasoning and decision-making. The generated decisions are then transmitted to action executor, where they are converted into control signals for interaction with the simulator. The Analytic Process then uses an LLM to accumulate driving analysis and decision-making experience and conduct reflection on accidents. The experience is stored in the memory bank and transferred to a lightweight language model, forming our Heuristic Process for quick responses and continuous learning.
When Heuristic Process encounters traffic accidents, the Analytic Process intervenes, analyzing historical frames to pinpoint errors and provide corrected samples. These corrected samples are then integrated into the memory bank to facilitate continuous learning.
- Please follow INSTALL.md to setup the environment, make sure you are using a machine with at least a mid-end GPU to run CARLA.
- To run the LeapAD within CARLA, please refer to the QUICK_START.md .
case1.mp4
case2.mp4
case3.mp4
We conduct closed-loop tests in CARLA. It can be seen that LeapAD can make informed decisions using the Heuristic Process with only 1.8B parameters while driving.
We utilized the following repos during development:
Thanks for their Awesome open-sourced work!
This project is released under the Apache 2.0 license.
If you find our project useful, please kindly cite us via:
@article{mei2024continuously,
title={Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving},
author={Mei, Jianbiao and Ma, Yukai and Yang, Xuemeng and Wen, Licheng and Cai, Xinyu and Li, Xin and Fu, Daocheng and Zhang, Bo and Cai, Pinlong and Dou, Min and others},
journal={arXiv preprint arXiv:2405.15324}