Double IQN

[cross32768]

This PR adds Double IQN as in ChainerRL as suggested in #4.
This PR includes new agent DoubleIQN and it's training example as well as tests, based on ChainerRL implementation and IQN in PFRL .

[prabhatnagarajan]

/test

[pfn-ci-bot]

Successfully created a job for commit 453a4f8:

• Dashboard for commit 453a4f8

[cross32768]

/test

[pfn-ci-bot]

  [NOT_FOUND] API failed: /a/github_check_membership: HTTP error: 404 Not Found: https://api.github.com/orgs/pfnet/members/cross32768
  2020-09-24 19:44:32.596771 call.go:280] API failed: /a/github_check_membership
  
  Stack trace:
    github.com/pfnet/flexci/internal/common/api.callInternal (call.go:280)
    github.com/pfnet/flexci/internal/common/api.Call (call.go:128)
    github.com/pfnet/flexci/internal/common/api.CallWithRetry (call.go:311)
    github.com/pfnet/flexci/internal/common/api.GithubCheckMembership (call.go:514)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*githubWebhookIssueCommentFlow).triggerTest (github_issue_comment.go:213)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*githubWebhookIssueCommentFlow).Do (github_issue_comment.go:99)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.githubIssueCommentHandler (github_issue_comment.go:47)
    runtime.call64 (asm_amd64.s:523)
    reflect.Value.call (value.go:447)
    reflect.Value.Call (value.go:308)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).registerHandler.func1.1 (handler.go:178)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).callHandler (handler.go:466)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).doInternal (handler.go:318)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).Do (handler.go:277)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).registerHandler.func1 (handler.go:175)
    github.com/pfnet/flexci/internal/frontend/core.(*handlerFlow).Do (handler.go:713)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).Register.func1 (handler.go:116)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.(*ServeMux).ServeHTTP (server.go:2361)
    github.com/pfnet/flexci/internal/common/api.callInternal.func2 (call.go:204)
    github.com/pfnet/flexci/internal/common/api.callInternal (call.go:212)
    github.com/pfnet/flexci/internal/common/api.Call (call.go:128)
    github.com/pfnet/flexci/internal/common/api.GithubIssueComment (call.go:500)
    github.com/pfnet/flexci/internal/frontend/handler/xternalhandler.(*githubHookFlow).doInternal (github_webhook.go:146)
    github.com/pfnet/flexci/internal/frontend/handler/xternalhandler.(*githubHookFlow).Do (github_webhook.go:39)
    github.com/pfnet/flexci/internal/frontend/handler/xternalhandler.githubWebhookHandler (github_webhook.go:29)
    github.com/pfnet/flexci/internal/frontend/core.(*handlerFlow).Do (handler.go:713)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).Register.func1 (handler.go:116)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.(*ServeMux).ServeHTTP (server.go:2361)
    google.golang.org/appengine/internal.executeRequestSafely (api.go:165)
    google.golang.org/appengine/internal.handleHTTP (api.go:124)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.serverHandler.ServeHTTP (server.go:2741)
    net/http.(*conn).serve (server.go:1847)
    runtime.goexit (asm_amd64.s:1333)
  
  Cause: [NOT_FOUND] HTTP error: 404 Not Found: https://api.github.com/orgs/pfnet/members/cross32768
  2020-09-24 19:44:32.592168 github_create_comment.go:91] HTTP error: 404 Not Found: https://api.github.com/orgs/pfnet/members/cross32768
  
  Stack trace:
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.callGithubAPI (github_create_comment.go:91)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.callGithubAPIWithRetry (github_create_comment.go:115)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.githubCheckMembershipHandler (github_check_membership.go:29)
    runtime.call64 (asm_amd64.s:523)
    reflect.Value.call (value.go:447)
    reflect.Value.Call (value.go:308)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).registerHandler.func1.1 (handler.go:178)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).callHandler (handler.go:466)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).doInternal (handler.go:318)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).Do (handler.go:277)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).registerHandler.func1 (handler.go:175)
    github.com/pfnet/flexci/internal/frontend/core.(*handlerFlow).Do (handler.go:713)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).Register.func1 (handler.go:116)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.(*ServeMux).ServeHTTP (server.go:2361)
    github.com/pfnet/flexci/internal/common/api.callInternal.func2 (call.go:204)
    github.com/pfnet/flexci/internal/common/api.callInternal (call.go:212)
    github.com/pfnet/flexci/internal/common/api.Call (call.go:128)
    github.com/pfnet/flexci/internal/common/api.CallWithRetry (call.go:311)
    github.com/pfnet/flexci/internal/common/api.GithubCheckMembership (call.go:514)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*githubWebhookIssueCommentFlow).triggerTest (github_issue_comment.go:213)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.(*githubWebhookIssueCommentFlow).Do (github_issue_comment.go:99)
    github.com/pfnet/flexci/internal/frontend/handler/apihandler.githubIssueCommentHandler (github_issue_comment.go:47)
    runtime.call64 (asm_amd64.s:523)
    reflect.Value.call (value.go:447)
    reflect.Value.Call (value.go:308)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).registerHandler.func1.1 (handler.go:178)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).callHandler (handler.go:466)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).doInternal (handler.go:318)
    github.com/pfnet/flexci/internal/frontend/core.(*apiHandlerFlow).Do (handler.go:277)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).registerHandler.func1 (handler.go:175)
    github.com/pfnet/flexci/internal/frontend/core.(*handlerFlow).Do (handler.go:713)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).Register.func1 (handler.go:116)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.(*ServeMux).ServeHTTP (server.go:2361)
    github.com/pfnet/flexci/internal/common/api.callInternal.func2 (call.go:204)
    github.com/pfnet/flexci/internal/common/api.callInternal (call.go:212)
    github.com/pfnet/flexci/internal/common/api.Call (call.go:128)
    github.com/pfnet/flexci/internal/common/api.GithubIssueComment (call.go:500)
    github.com/pfnet/flexci/internal/frontend/handler/xternalhandler.(*githubHookFlow).doInternal (github_webhook.go:146)
    github.com/pfnet/flexci/internal/frontend/handler/xternalhandler.(*githubHookFlow).Do (github_webhook.go:39)
    github.com/pfnet/flexci/internal/frontend/handler/xternalhandler.githubWebhookHandler (github_webhook.go:29)
    github.com/pfnet/flexci/internal/frontend/core.(*handlerFlow).Do (handler.go:713)
    github.com/pfnet/flexci/internal/frontend/core.(*registerHandlerFlow).Register.func1 (handler.go:116)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.(*ServeMux).ServeHTTP (server.go:2361)
    google.golang.org/appengine/internal.executeRequestSafely (api.go:165)
    google.golang.org/appengine/internal.handleHTTP (api.go:124)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.serverHandler.ServeHTTP (server.go:2741)
    net/http.(*conn).serve (server.go:1847)
    runtime.goexit (asm_amd64.s:1333)

[muupan]

/test

[pfn-ci-bot]

Successfully created a job for commit a98d6f5:

• Dashboard for commit a98d6f5

[muupan]

@cross32768 Sorry for the delayed response! Do you have time/resource for running Atari experiments like chainer/chainerrl#503 (comment) to verify its relative performance to IQN?

[cross32768]

I have some computing resource and conducted some experiments to verify correctness of implementation. But to conduct experiment in all Atari environments is difficult for my computing power. Is it okay to verify performance in some parts of Atari environments?

[muupan]

Yes, it is fine to run only the subset. It would be perfect if you run the same set of chainer/chainerrl#503 (comment) so that we can compare scores.

[cross32768]

I'll add results of experiments on same environments as in chainer/chainerrl#503 (comment) by editing this comment.

All parameter for experiments is default parameter of training code, and reported scores are "mean" of evaluation after 5e7 training steps. Each score is average of 3 experiments with different seeds.

Game	Double IQN	IQN
Asterix	351356.3	376688.3
Asteroids	2750.2	3346.2
Beamrider	26850.8	27659.5

[cross32768]

I will withdraw the pull request once and try to investigate the cause of problem because all scores observed is lower than PFRL original scores of IQN (https://github.com/pfnet/pfrl/tree/master/examples/atari/reproduction/iqn ).