-
Notifications
You must be signed in to change notification settings - Fork 40
Setup and Operation
This page details the steps for installing the DeepPicar software, and how to run the car both manually and autonomously.
It is assumed that a Raspberry Pi 3/4 is available with an operating system already installed. If this isn't the case, it can be done by following the steps in Installing Operating System Images. We have used both Raspbian and Ubuntu MATE on the Pi 3 Model B, but would recommend Raspbian Buster for newer Pi models such as the Pi 3 A+/B+ or the Pi 4.
The DeepPicar code can be downloaded by issuing the following terminal command:
$ git clone --depth=1 https://github.com/mbechtel2/DeepPicar-v2
Before running the code, though, some additional Python libraries need to be installed. The first of these is the python-dev package which can installed with the following command:
$ sudo apt-get install python-dev
Other packages are then needed for communicating with and operating the camera sensor and the motorhat. For the camera, we use OpenCV which can be installed in python with:
$ sudo apt-get install python-opencv
Another required package is PIL (for Image and ImageDraw), which is used for displaying driving statistics in first-person view (FPV) videos. It can be installed with:
$ sudo pip install pillow
$ sudo apt-get install python-imaging
For the motorhat, we use the Pololu DRV8835 dual motor driver kit. The Adafruit MotorHAT or an Arduino can also be used for the motorhat, in which case, the Adafruit MotorHAT Python Library or the python-serial package should be installed instead.
Finally, TensorFlow needs to be installed in order to train and run the DeepPicar's neural network (while the version shouldn't matter, 1.12.0 and 1.13.1 are broken, so we recommend 1.11.0 or earlier). Since the Raspberry Pi 3/4 is officially supported by Google, TensorFlow can be installed by running:
$ sudo apt-get install libhdf5-dev #Needed for h5py
$ sudo pip install Cython==0.29.16 #Needed for h5py, Cython 3.0a1 results in an installation error
$ sudo pip install --no-cache-dir tensorflow==1.11.0
This will install the CPU-only version of TensorFlow. The GPU version cannot be used as the Pi 3 and 4 don't have a CUDA compatible GPUs. You can refer to https://www.tensorflow.org/install/pip for more in-depth TensorFlow installation instructions and options.
If all the prerequisite packages were installed successfully, then you can now control the DeepPicar manually by invoking the picar-mini-kbd-common.py file from the DeepPicar code. From a terminal, you can input
$ sudo python picar-mini-kbd-common.py
or
$ sudo ./picar-mini-kbd-common.py
After the program starts, performance statistics will be printed to the terminal and a new OpenCV window will be opened. Make sure to select this new window, or inputs will not work.
The commands for controlling the DeepPicar are as follows:
- 'a': Assuming the car is initially stationary, this will start moving the car forward at a constant speed. By default, this speed is 50% of the car's max throttle, but this can be changed as will be detailed in the command line options. If the car is already moving forward, inputting this command again will have no effect.
- 'z': This performs the same action as 'a', but will instead move the car in the reverse direction.
- 's': This command will stop the car, regardless of its current direction. (Note that the car can't be stopped by issuing the command for the opposite direction. For example, if the car is moving backwards, issuing the 'a' command will not stop the car but have it start moving forwards).
- 'j': Turns the wheels of the car to the left.
- 'l': Turns the wheels of the car to the right.
- 'k': Centers the wheels of the car. (Note that the wheels of the car will remain turned until this command is given. Also, the wheels cannot be centered by issuing the 'j' or 'l' commands when the car is turning right or left, respectively. Doing so will instead turn the wheels entirely in the opposite direction).
- 't': This command will populate the OpenCV window with a first-person view from the DeepPicar as it is driving. However, this comes at a major performance loss and shouldn't be enabled when collecting data that will be used for training a neural network.
If prefereable, all of the above keybindings can be changed by modifying the settings in picar-mini-kbd-common.py starting at line 137. For example, to use the 'h' key to turn the wheels to the left, you would modify the code as follows
137 if ch == ord('h'):
138 actuator.left()
139 print("left")
140 angle = deg2rad(-30)
141 btn = ord('h')
When running picar-mini-kbd-common-py, there are 3 command line options that can be used to further control the car:
- '-t' or '--throttle': This sets the car's speed for when it is moving, and should be input as a percentage value. By default, this is set to 50%.
- '-d' or '--dnn': This controls whether the DeepPicar should operated manually or autonomously. For manual control, do not list this option.
- '-n' or '--ncpu': When running autonomously, this option changes the number of cores used by the neural network inferencing operation. By default, this is set to 2.
- '-f' or '--fpvvideo': When running autonomously, this option has the car record a FPV video. Each frame also has its number and the car's steering angle at that frame printed in the top left corner.
For example, to manually run the DeepPicar at 100% throttle, you would run
$ sudo python picar-mini-kbd-common.py -t 100
To run autonomously at full throttle with all cores available to the neural network, you would instead run
$ sudo python picar-mini-kbd-common.py -t 100 -n 4 -d
Before collecting data, a datafiles directory should be created by issuing mkdir directories in the DeepPicar directory. This folder will be used in the training process for gathering the data that will be used to optimize the DeepPicar network.
In order to collect data while manually controlling the DeepPicar, there is an additional command in the 'r' key. When this command is given for the first time, the DeepPicar will begin collecting every frame that it sees as well as the car's steering angle (i.e. left, center, or right). When given for the second time, or after 1000 frames have passed, the frames and steering angles collected are written to the following files:
- 'out-video.avi': A video file that shows a first-person view of the DeepPicar while it was driving. It is created from all of the frames collected while the car was operating.
- 'out-key.csv': A CSV file that contains each frame's information. Specifically, it has the frame number and the car's steering angle at that frame.
In order to keep the collected data, both files need to be moved to the epochs folder and renamed with their unique dataset number. This can be done with the create-epoch.sh script:
$ ./create-epoch.sh #
This will move the out-video.avi and out-key.csv to the 'epochs' folder with the names out-video-#.avi and out-key-#.csv, respectively. The same can be done with by running:
$ mv out-video.avi epochs/out-video-#.avi
$ mv out-key.csv epochs/out-key-#.csv
Replace # above with the number of the dataset. For example, for the first dataset, # should be 1. These steps need to be completed before the next dataset is recorded. If not, the collected data will be overwritten.
We highly recommend copying all data to a remote desktop with a CUDA compatible GPU and the tensorflow-gpu pip package installed. This will greatly reduce the time it takes to train the neural network (from ~4 hours on the Pi 3 to a few minutes).
In order to train the DeepPicar's neural network, some preliminary steps need to be taken in the params.py file. Specifically, 3 values need to be set:
- 'model_load_file': The name of the files created when the model is being trained. Effectively speaking, this is the model's name.
- 'epochs['train']': The datasets being used for training the model.
- 'epochs['val']': The datasets being used for validating the model.
All trained models will be saved to a directory named with the 'saved_dir' value. Three additional model files (model_load_file2,model_load_file3, and model_load_file4) are used by the evaluation scripts in the 'scripts' folder. For normal use, these files aren't necessary.
For example, to train a model name "DeepPicarModel" using dataset 1, 2, 3 and 4 for training and datasets 5, 6, 7 and 8 for validation, you would make the following changes:
30 model_load_file = os.path.abspath('DeepPicarModel')
.
.
.
62 epochs['train'] = [1,2,3,4]
63 epochs['val'] = [5,6,7,8]
Note that the dataset numbers for the training and validation sets must be valid. For example, if 1 is given as a training dataset, but out-video-1.avi or out-key-1.csv don't exist, the training process will stop.
Once the above changes have been made, the model can be trained by running
$ python train.py
This will then run through 2000 training steps, and, when finished, will result in an optimized neural network model. If trained on a remote desktop, the model files can be copied over to the Pi 3 and placed in the models directory.
Now that a neural network model has been trained, it can be used to control the DeepPicar autonomously. This can be done by using the -d command line option when running picar-mini-kbd-common.py:
$ sudo python picar-mini-kbd-common.py -d
After doing so, there will be a delay (due the TensorFlow library being imported), after which performance statistics will again be printed out to the terminal. The same commands for manual control will work, however the car will steer itself after it has been given the command to move forward.
If multiple trained neural network models are available, you can switch between them by changing the model_load_file value in the params.py folder as seen above. In addition to creating the model's files, it is also used for importing the model for autonomous operation.
When the car is stopped (i.e. the 'q' command), a video from the car's POV will be generated (by default, it is called "fpv-video.mp4"). For each frame in the video, there will also be a small box in the top left corner that displays the frame's number as well as the steering angle of the car at that frame.