Risk-sensitive-RL

Code base for paper: Adaptive Risk Tendency: Nano Drone Navigation in Cluttered Environments with Distributional Reinforcement Learning

Dependencies

• python 3.6+
• gym 0.10.5
• numpy 1.19.5
• pytorch 1.9.0

Installation of Crazyflie Training Environment

git clone https://github.com/tudelft/risk-sensitive-rl.git
cd risk-sensitive-rl/crazyflie-env
python3 -m pip install -e '.[crazyflie-env]'

Import Crazyflie Environment

import gym, crazyflie_env
env = gym.make('CrazyflieEnv-v0') # env id

Note that the interface for this environment is slightly different than OpenAI gym. By calling env.step(<action>), the returned state is a object that need to be transferred to numpy arrays. This is implemented by a transfer function series in art-iqn/utils/util.py.

For the details of using this environment, refer to art-iqn/train.py or art-iqn/tactical.py.

Reproduction

To reproduce our real-world experiments, one needs a crazyflie drone system equiped with multiranger deck and an global position system like OptiTrack to locate the drone's absolute position. Onboard position estimation system can also be used but there maybe exist large drift.

IQN policy with fixed CVaR values

python3 art-iqn/run_cyber_zoo.py --cvar=<your_cvar_value> --exp_id='<your_experiment_id>'

ART-IQN

python3 art-iqn/run_cyber_zoo_art.py 

Video

Supplementary video for our real-world experiment can be found in art-iqn/experiments/videos.