Adequately exercising the behaviors of autonomous vehicles is fundamental to their validation. However, quantifying an autonomous vehicle's testing adequacy is challenging as the system's behavior is influenced both by its state as well as its physical environment. To address this challenge, our work builds on two insights. First, data sensed by the autonomous vehicle provides a unique spatial signature of the physical environment inputs.Second, given its current state, inputs residing outside the autonomous vehicle's physically reachable regions are less relevant to its behavior. Building on those insights, we introduce an abstraction that enables the computation of a physical environment-based coverage metric, PhysCov. The abstraction combines the sensor readings with a physical reachability analysis based on the vehicle's state and dynamics to determine the region of the environment that may affect the autonomous vehicle. It then characterizes that region through a parameterizable geometric approximation that can trade quality for cost. Tests with the same characterizations are deemed to have had similar internal states and exposed to similar environments, and thus likely to exercise the same set of behaviors, while tests with distinct characterizations will increase PhysCov. A study on two simulated and one real system examines PhysCovs's ability to quantify an autonomous vehicle's test suite, showcases its characterization cost and precision, and investigates its correlation with failures found and potential for test selection.
We show an example of our approach working on a real world autonomous system below. Here you can see a car driving down a road, the sensor readings, the reduced reachable set, the reachable set vectorization, and finally the RRS values.
The artifact contains the code to run three studies:
- HighwayEnv
- BeamNG
- Waymo
We recommend starting with HighwayEnv README.MD. Then we recommend downloading the provided data using this README.MD. Once you have downloaded the data, we recommend trying to generate some of the results in the paper using this README.MD.
All notes in this document are related to everything other than BeamNG. To run BeamNG please refer to the BeamNG Environment README.
This software was developed, run and test on the following machines. Note: BeamNG requires the Windows operating system, and was run on a different machine. For more details on BeamNG please refer to the BeamNG README.MD.
| Computer | CPU | CPU Cores | RAM | Operating System |
|---|---|---|---|---|
| Computer 1 | AMD Ryzen Threadripper 3990X | 128 | 128 GB | Ubuntu 20.04 |
| Computer 2 | Intel Core i7-10750H | 12 | 16 GB | Ubuntu 20.04 |
We require several python packages to run our software. To install them you can use the package manager pip. We tested this using Python 3.7.
sudo apt install python3-pipFinally install a local copy of the highway_env environment. To do that make sure you are in this directory and run:
python3 -m pip install ./environments/highway/highway_env_v2Next install the following packages:
python3 -m pip install --upgrade pip
python3 -m pip install gym==0.18
python3 -m pip install matplotlib==3.6.0
python3 -m pip install coverage==5.5
python3 -m pip install shapely==2.0.1
python3 -m pip install numpy==1.19.5
python3 -m pip install pandas==1.2.0
python3 -m pip install ordered_set==4.1.0
python3 -m pip install tqdm==4.65.0
python3 -m pip install matplotlib-venn==0.11.9
python3 -m pip install scikit-learn
python3 -m pip install tensorflow==2.11.0
python3 -m pip install waymo-open-dataset-tf-2-11-0
python3 -m pip install protobuf==3.20.0| Component | Description | Link |
|---|---|---|
| Highway-Env | This will describe how to run the HighwayEnv environment, and how to generate sample runs. | README.MD |
| BeamNG | This will describe how to run the BeamNG environment, and how to generate sample runs. (Requires windows) | README.MD |
| Waymo | This will describe how to run the Waymo Open Perception Dataset, and how to extract the data. | README.MD |
| RRS Pipeline | This will show you how to run the RRS Pipeline on example data, as well as your own. | README.MD |
| Data Analysis | This will show you how to generate the graphs presented in our paper. | README.MD |
| Provided Data | This will provide a subset of the data used in the paper. The data used in the paper was over 2TB. | README.MD |
| Paper | Includes the paper published at ISSTA 2023 | Paper |
We have provided the following data. For more information of the details of the data please refer to the data subset README.MD.