Skip to content

Commit

Permalink
Update Serf and memberlist (#4511)
Browse files Browse the repository at this point in the history
This includes fixes that improve gossip scalability on very large (> 10k node) clusters.

The Serf changes:
 - take snapshot disk IO out of the critical path for handling messages hashicorp/serf#524
 - make snapshot compaction much less aggressive - the old fixed threshold caused snapshots to be constantly compacted (synchronously with request handling) on clusters larger than about 2000 nodes! hashicorp/serf#525

Memberlist changes:
 - prioritize handling alive messages over suspect/dead to improve stability, and handle queue in LIFO order to avoid acting on info that 's already stale in the queue by the time we handle it. hashicorp/memberlist#159
 - limit the number of concurrent pushPull requests being handled at once to 128. In one test scenario with 10s of thousands of servers we saw channel and lock blocking cause over 3000 pushPulls at once which ballooned the memory of the server because each push pull contained a de-serialised list of all known 10k+ nodes and their tags for a total of about 60 million objects and 7GB of memory stuck. While the rest of the fixes here should prevent the same root cause from blocking in the same way, this prevents any other bug or source of contention from allowing pushPull messages to stack up and eat resources. hashicorp/memberlist#158
  • Loading branch information
banks authored and mkeeler committed Aug 9, 2018
1 parent c88900a commit 9ce1076
Show file tree
Hide file tree
Showing 9 changed files with 234 additions and 71 deletions.
2 changes: 1 addition & 1 deletion vendor/github.com/hashicorp/memberlist/Makefile

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion vendor/github.com/hashicorp/memberlist/README.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

36 changes: 23 additions & 13 deletions vendor/github.com/hashicorp/memberlist/memberlist.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

94 changes: 72 additions & 22 deletions vendor/github.com/hashicorp/memberlist/net.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions vendor/github.com/hashicorp/serf/serf/delegate.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion vendor/github.com/hashicorp/serf/serf/keymanager.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion vendor/github.com/hashicorp/serf/serf/serf.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 9ce1076

Please sign in to comment.