Double IQN

[prabhatnagarajan]

This PR adds Double IQN. This PR includes a new example script for IQN based agents (since it does not fall under "reproducibility". It includes a new Agent DoubleIQN, as well as a test.

[prabhatnagarajan]

Game	Double 3-Step IQN	ChainerRL IQN
Asterix	402856.2	454846.7
BeamRider	25382.4	38111.4
Berzerk	2013.2	138167.9
Bowling	90.4	84.3
Breakout	393.2	658.6
Enduro	2362.5	2363.3
Freeway	34.0	34.0
Hero	36409.1	27639.9
MsPacman	6924.1	5786.5
Pong	21.0	21.0
PrivateEye	200.0	289.9
Qbert	27683.3	24950.3
Seaquest	9944.8	27045.5
SpaceInvaders	53172.5	27810.9
Venture	1520.0	289.0
VideoPinball	774232.6	664013.5

Out of 16 domains, Double 3-Step IQN vs. ChainerRL's IQN:

• 2 ties
• 7 domains where Double 3-Step IQN beats ChainerRL
• 7 domains where CRL beats Double 3-Step IQN.

[prabhatnagarajan]

Game	CRL IQN	CRL DoubleIQN
Asterix	454846.7	483300
Asteroids	3885.9	3860.71
Atlantis	946912.5	978200
Berzerk	138167.9	2339.0
Breakout	658.6	608.3
Enduro	2363.3	2359.25
Freeway	34.0	34.0
Hero	27639.9	28818.65
SpaceInvaders	27810.9	54967.5

Double IQN wins on 4 domains
IQN wins on 4 domains
Tie on 1 domain

[prabhatnagarajan]

Full fledged IQN Results

Game	DoubleIQN	ChainerRL IQN
Asterix	507353.8	454846.7
Asteroids	3915.47	3885.9
BeamRider	33666.2	38111.4
Berzerk	2339.0	138167.9
Bowling	80.33	84.3
Breakout	666.28	658.6
Enduro	2359.25	2363.3
Freeway	34.0	34.0
Hero	28564.58	27639.9
MontezumaRevenge	5.55	0.6
Pong	21.0	21.0
PrivateEye	295.65	289.9
Qbert	29531.1	24950.3
Riverraid	22519.31	20716.1
Seaquest	30870.0	27045.5
SpaceInvaders	55127.5	27810.9
Venture	719.51	289.0
VideoPinball	731942.25	664013.5

Double IQN wins on 12 domains!
There are 2 ties.
IQN wins on 4 domains.

[muupan]

I think it is better to explicitly add train_double_iqn.py rather than train_iqn_ale.py because it will be clear from the filename what the file is for. IQN already has examples/atari/reproduction/iqn.

[muupan]

I don't think Use primary network for target computation is a good description of the algorithm. Can you just say something like IQN with DoubleDQN-like target computation in the first line and add a detailed explanation after a blank line (as suggested by the google style https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings)?