- Please install Open in Colab extension in Google Chrome in order to open a Github-hosted notebook in Google Colab with one-click.
- Personally, I recommend using Kaggle kernel directly to explore the dataset, instead of downloading the dataset to local machine, if you do not have a powerful GPU.
Self-driving vehicles is the most hottest and interesting topics of research and business nowadays. More and more giant companies have jumped into this area. In this project, I have implemented the CNN model put forward by Nvidia Research in this paper using PyTorch to extract features from 3 cameras in the vehicle simulation software to predict steering angle. This is an end-to-end approach to apply to autonomous driving.
We will use Python as the primary programming language and PyTorch as the Deep Learning framework. Other resources / software / library could be found as follows.
- Self-driving car simulator developed by Udacity with Unity. Download here
- Install PyTorch environment (latest version the best) in your local machine.
- Log in Google Colab (if you do not have GPU and would love to utilize the power of GPU, please try this and be sure to enable
GPU
as accelerator)
I have already uploaded training images in Kaggle and Floydhub. Feel free to download:
- Kaggle together with starter code and pre-trained models
- Floydhub
Left | Center | Right |
---|---|---|
Download this repo.
git clone https://github.com/Zhenye-Na/self-driving-vehicles-sim-with-ml.git
cd self-driving-vehicles-sim-with-ml/src
usage: main.py [-h] [--dataroot DATAROOT] [--ckptroot CKPTROOT] [--lr LR]
[--weight_decay WEIGHT_DECAY] [--batch_size BATCH_SIZE]
[--num_workers NUM_WORKERS] [--train_size TRAIN_SIZE]
[--shuffle SHUFFLE] [--epochs EPOCHS]
[--start_epoch START_EPOCH] [--resume RESUME]
[--model_name MODEL_NAME]
Main pipeline for self-driving vehicles simulation using machine learning.
optional arguments:
-h, --help show this help message and exit
--dataroot DATAROOT path to dataset
--ckptroot CKPTROOT path to checkpoint
--lr LR learning rate
--weight_decay WEIGHT_DECAY
weight decay (L2 penalty)
--batch_size BATCH_SIZE
training batch size
--num_workers NUM_WORKERS
# of workers used in dataloader
--train_size TRAIN_SIZE
train validation set split ratio
--shuffle SHUFFLE whether shuffle data during training
--epochs EPOCHS number of epochs to train
--start_epoch START_EPOCH
pre-trained epochs
--resume RESUME whether re-training from ckpt
--model_name MODEL_NAME
model architecture to use [nvidia, lenet]
An example of training usage is shown as follows:
python3 main.py --epochs=50 --resume=True
if you prefer to use Colab as training platform, feel free to use train.ipynb
script. Make sure you have already uploaded training data to Google Drive.
Kaggle also provides GPU support in its own kernel. Feel free to use this as the start code.
Training images are loaded in BGR colorspace using
cv2
whiledrive.py
load images in RGB to predict the steering angles.
After training process, use the saved model and drive.py
file to test your model performance in the simulator. Remeber to select AUTONOMOUS MODE
. Click Allow
to accept incoming network connections for python scripts.
usage: drive.py [-h] model [image_folder]
Remote Driving
positional arguments:
model Path to model h5 file. Model should be on the same path.
image_folder Path to image folder. This is where the images from the run
will be saved.
optional arguments:
-h, --help show this help message and exit
An example of test usage is shown as follows:
python3 drive.py model.h5 runs1/
Terrain 1 | Terrain 2 | Terrain 3 |
---|---|---|
usage: video.py [-h] [--fps FPS] image_folder
Create driving video.
positional arguments:
image_folder Path to image folder. The video will be created from these
images.
optional arguments:
-h, --help show this help message and exit
--fps FPS FPS (Frames per second) setting for the video.
An example of creating video usage is shown as follows:
python3 video.py runs1/ --fps 48
Once you have installed the self-driving car simulator, you will find there are 2 different tracks in the simulator. The second one is much harder to deal with. Please choose the terrain you like and make sure that you select TRAINING MODE
.
Click RECORD
button on the right corner and select a directory as the folder to save your training image and driving log information.
Click RECORD
again and move your car smoothly and carefully. After you have completed recording your move, the training data will be stored in the folder you selected. Here I suggest you record at least 3 laps of the race. The first lap of race, please try best to stay at the center of the road, the rest could be either on the left hand side and right hand side of the road separately.
/IMG/
- recorded images from cneter, left and right cameras.driving_log.csv
- saved the image information and associated information like steer angle, current speed, throttle and brake.
.
βββ RcCarDataset.py # Customed Dataset for Self-driving car simulator (Simple CNN as well as Nvidia paper's CNN model)
βββ drive.py # Test script
βββ main.py # Main pipeline
βββ model.py # CNN model declaration
βββ train_in_colab.ipynb # Colab training script
βββ trainer.py # Trainer
βββ utils.py # Helper functions
βββ video.py # Create videos
NVIDIA model used
Image normalization to avoid saturation and make gradients work better.
Convolution: 5x5, filter: 24, strides: 2x2, activation: ELU
Convolution: 5x5, filter: 36, strides: 2x2, activation: ELU
Convolution: 5x5, filter: 48, strides: 2x2, activation: ELU
Convolution: 3x3, filter: 64, strides: 1x1, activation: ELU
Convolution: 3x3, filter: 64, strides: 1x1, activation: ELU
Drop out (0.5)
Fully connected: neurons: 100, activation: ELU
Fully connected: neurons: 50, activation: ELU
Fully connected: neurons: 10, activation: ELU
Fully connected: neurons: 1 (output)
Hyper-parameters | Description |
---|---|
lr=1e-4 | learning rate |
weight_decay=1e-5 | weight decay (L2 penalty) |
batch_size=32 | training batch size |
num_workers=8 | # of workers used in dataloader |
train_size=0.8 | train-validation set split ratio |
shuffle=True | whether shuffling data during training |
This error means that you are trying to load a newer version of model checkpoint in an older version of PyTorch.
Check PyTorch version in your local machine:
import pytorch
print(torch.__version__)
For Google Colab users, please try to first get your PyTorch version in your local machine and install the same version on Colab virtual machine using the following snippets, PyTorch 0.4.1
, for example.
# http://pytorch.org/
from os.path import exists
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\([0-9]*\)$/cu\1\2/'
accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'
!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.1-{platform}-linux_x86_64.whl torchvision
Thanks for @quliang93 to point this out
- Make sure Unity is installed correctly. Udacity self-driving simulator is build with unity, so you have to install it.
- Make sure the version of socket is compatible
pip install Flask-SocketIO==4.3.1
pip install python-engineio==3.13.2
pip install python-socketio==4.6.0
Belows are the steps to run drive.py
in Windows:
- run command:
python drive.py model.h5
- open the Udacity simulator, and choose autonomous mode
If everything is configured properly, you would see the car move.
In case if you still have the problem, feel free to start a thread in #3
Applied a Savitzky-Golay filter to the steering angle column. This helps reduce the training loss because original signal for steering angle input is via keyboard which is very dicrete.
Track 1 | Track 2 | Track 3 |
---|---|---|
- Edge detection pre-processing
- Use Joystick instead of Keyboard to create more "continuous" dataset (denoising input instead)
- Predict
$\Delta_\text{steering angle}$ instead of$\theta_{\text{steering angle}}$ - Take "velocity" into account
- Add "autonomous mode" images into training set
[1] Nvidia research, End to End Learning for Self-Driving Cars
[2] Self-driving car simulator developed by Udacity with Unity