Atlantis crashes when hitting GitHub rate limit

[adubeniuk]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

GitHub user access token requests are limited to 5,000 requests per hour and per authenticated user. When using one heavily-used GitHub account for Atlantis it is possible to hit a rate limit. When that happens Atlantis throws an error and crashes.

Reproduction Steps

Use GitHub account, create 5000 test requests to API then trigger any Atlantis command.

Logs

Logs

{"level":"error","ts":"2023-09-04T15:08:54.599Z","caller":"vcs/instrumented_client.go:142","msg":"Unable to react to comment, error: POST https://api.github.com/repos/[REDACTED]/issues/comments/[REDACTED]/reactions: 403 API rate limit exceeded for user ID [REDACTED]. [rate reset in 34s]","json":{},"stacktrace":"github.com/runatlantis/atlantis/server/events/vcs.(*InstrumentedClient).ReactToComment\n\tgh.neting.cc/runatlantis/atlantis/server/events/vcs/instrumented_client.go:142\ngh.neting.cc/runatlantis/atlantis/server/events/vcs.(*ClientProxy).ReactToComment\n\tgh.neting.cc/runatlantis/atlantis/server/events/vcs/proxy.go:68\ngh.neting.cc/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleCommentEvent\n\tgh.neting.cc/runatlantis/atlantis/server/controllers/events/events_controller.go:570\ngh.neting.cc/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).HandleGithubCommentEvent\n\tgh.neting.cc/runatlantis/atlantis/server/controllers/events/events_controller.go:318\ngh.neting.cc/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleGithubPost\n\tgh.neting.cc/runatlantis/atlantis/server/controllers/events/events_controller.go:177\ngh.neting.cc/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).Post\n\tgh.neting.cc/runatlantis/atlantis/server/controllers/events/events_controller.go:104\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2122\ngh.neting.cc/gorilla/mux.(*Router).ServeHTTP\n\tgh.neting.cc/gorilla/mux@v1.8.0/mux.go:210\ngh.neting.cc/urfave/negroni/v3.Wrap.func1\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:59\ngh.neting.cc/urfave/negroni/v3.HandlerFunc.ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:33\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP\n\tgh.neting.cc/runatlantis/atlantis/server/middleware.go:70\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/urfave/negroni/v3.(*Recovery).ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/recovery.go:210\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/urfave/negroni/v3.(*Negroni).ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:111\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2936\nnet/http.(*conn).serve\n\tnet/http/server.go:1995"}
{"level":"warn","ts":"2023-09-04T15:08:54.599Z","caller":"events/events_controller.go:572","msg":"Failed to react to comment: POST https://api.github.com/repos/[REDACTED]/issues/comments/[REDACTED]/reactions: 403 API rate limit exceeded for user ID [REDACTED]. [rate reset in 34s]","json":{"gh-request-id":"X-Github-Delivery=[REDACTED]"},"stacktrace":"github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleCommentEvent\n\tgh.neting.cc/runatlantis/atlantis/server/controllers/events/events_controller.go:572\ngh.neting.cc/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).HandleGithubCommentEvent\n\tgh.neting.cc/runatlantis/atlantis/server/controllers/events/events_controller.go:318\ngh.neting.cc/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleGithubPost\n\tgh.neting.cc/runatlantis/atlantis/server/controllers/events/events_controller.go:177\ngh.neting.cc/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).Post\n\tgh.neting.cc/runatlantis/atlantis/server/controllers/events/events_controller.go:104\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2122\ngh.neting.cc/gorilla/mux.(*Router).ServeHTTP\n\tgh.neting.cc/gorilla/mux@v1.8.0/mux.go:210\ngh.neting.cc/urfave/negroni/v3.Wrap.func1\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:59\ngh.neting.cc/urfave/negroni/v3.HandlerFunc.ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:33\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP\n\tgh.neting.cc/runatlantis/atlantis/server/middleware.go:70\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/urfave/negroni/v3.(*Recovery).ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/recovery.go:210\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/urfave/negroni/v3.(*Negroni).ServeHTTP\n\tgh.neting.cc/urfave/negroni/v3@v3.0.0/negroni.go:111\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2936\nnet/http.(*conn).serve\n\tnet/http/server.go:1995"}
{"level":"error","ts":"2023-09-04T15:08:54.599Z","caller":"vcs/instrumented_client.go:72","msg":"Unable to get pull number for repo, error: GET https://api.github.com/repos/[REDACTED]/pulls/3374: 403 API rate limit of 5000 still exceeded until 2023-09-04 15:09:29 +0000 UTC, not making remote request. [rate reset in 34s]","json":{"repository":"[REDACTED]","pull-num":"3374"},"stacktrace":"github.com/runatlantis/atlantis/server/events/vcs.(*InstrumentedGithubClient).GetPullRequest\n\tgh.neting.cc/runatlantis/atlantis/server/events/vcs/instrumented_client.go:72\ngh.neting.cc/runatlantis/atlantis/server/events.(*DefaultCommandRunner).getGithubData\n\tgh.neting.cc/runatlantis/atlantis/server/events/command_runner.go:311\ngh.neting.cc/runatlantis/atlantis/server/events.(*DefaultCommandRunner).ensureValidRepoMetadata\n\tgh.neting.cc/runatlantis/atlantis/server/events/command_runner.go:371\ngh.neting.cc/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgh.neting.cc/runatlantis/atlantis/server/events/command_runner.go:263"}
{"level":"error","ts":"2023-09-04T15:08:54.600Z","caller":"events/command_runner.go:387","msg":"making pull request API call to GitHub: GET https://api.github.com/repos/[REDACTED]/pulls/3374: 403 API rate limit of 5000 still exceeded until 2023-09-04 15:09:29 +0000 UTC, not making remote request. [rate reset in 34s]","json":{"repo":"[REDACTED]","pull":"3374"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).ensureValidRepoMetadata\n\tgh.neting.cc/runatlantis/atlantis/server/events/command_runner.go:387\ngh.neting.cc/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgh.neting.cc/runatlantis/atlantis/server/events/command_runner.go:263"}
{"level":"error","ts":"2023-09-04T15:08:54.600Z","caller":"events/command_runner.go:429","msg":"PANIC: runtime error: invalid memory address or nil pointer dereference\nruntime/panic.go:260 (0x44e11c)\nruntime/signal_unix.go:837 (0x44e0ec)\ngh.neting.cc/runatlantis/atlantis/server/events/command_runner.go:388 (0xf70451)\ngh.neting.cc/runatlantis/atlantis/server/events/command_runner.go:263 (0xf6dfb7)\nruntime/asm_amd64.s:1598 (0x46b2e0)\n","json":{"repo":"[REDACTED]","pull":"3374"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\tgh.neting.cc/runatlantis/atlantis/server/events/command_runner.go:429\nruntime.gopanic\n\truntime/panic.go:890\nruntime.panicmem\n\truntime/panic.go:260\nruntime.sigpanic\n\truntime/signal_unix.go:837\ngh.neting.cc/runatlantis/atlantis/server/events.(*DefaultCommandRunner).ensureValidRepoMetadata\n\tgh.neting.cc/runatlantis/atlantis/server/events/command_runner.go:388\ngh.neting.cc/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgh.neting.cc/runatlantis/atlantis/server/events/command_runner.go:263"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xf70e83]

goroutine 100 [running]:
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics(0xc0006024e0, {{0xc000430fe0, 0x19}, {0xc000430fe0, 0x9}, {0xc000430fea, 0xf}, {0xc000088460, 0x64}, {0xc00020d680, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/events/command_runner.go:430 +0x203
panic({0x1173080, 0x1dae110})
	runtime/panic.go:890 +0x263
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).ensureValidRepoMetadata(_, {{0xc000430fe0, 0x19}, {0xc000430fe0, 0x9}, {0xc000430fea, 0xf}, {0xc000088460, 0x64}, {0xc00020d680, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/events/command_runner.go:388 +0x871
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand(0xc0006024e0, {{0xc000430fe0, 0x19}, {0xc000430fe0, 0x9}, {0xc000430fea, 0xf}, {0xc000088460, 0x64}, {0xc00020d680, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/events/command_runner.go:263 +0x718
created by github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleCommentEvent
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:594 +0x750

Environment details

If not already included, please provide the following:

• Atlantis version: v0.25.0
• Deployment method: eks
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: yes
• Atlantis flags:

Atlantis server-side config file:

atlantis-url: [REDACTED]
automerge: false
autoplan-file-list:  [REDACTED]
disable-autoplan: false
disable-repo-locking: true
enable-diff-markdown-format: true
hide-prev-plan-comments: true
repo-allowlist:  [REDACTED]
repo-config: /etc/atlantis/repos.yaml
silence-no-projects: true
silence-vcs-status-no-plans: true
vcs-status-name: "atlantis-dev"
executable-name: "atlantis-dev"
write-git-creds: true

Any other information you can provide about the environment/deployment (efs/nfs, aws/gcp, k8s/fargate, etc)

Additional Context