Support Horizontal Scaling #1048

kate-osborn · 2023-09-08T19:34:40Z

Proposed changes

Problem: NKG cannot be scaled horizontally because all replicas will write statuses to the Gateway API resources.

Solution: Add leader election to the status updater so that only one replica of NKG will write statuses to the Gateway API resources. Leader election is enabled by default but can be disabled via a cli arg --leader-election-disable. The lock name used for leader election can be configured via the cli arg --leader-election-lock-name.

Testing: Verified leader election works with multiple replicas.
Closes #637

Checklist

Before creating a PR, run through this checklist and mark each as complete.

I have read the CONTRIBUTING doc
I have added tests that prove my fix is effective or that my feature works
I have checked that all unit tests pass after adding my changes
I have updated necessary documentation
I have rebased my branch onto main
I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

internal/framework/status/updater.go

Makefile

internal/framework/status/updater.go

pleshakov

more comments:
(1) Would it make sense to add something about leader election to the architecture doc?

(2) What removes the leader election lock resource? Do we need to update our deinstallation instructions (helm and manifests) if it is not removed automatically?

cmd/gateway/commands.go

conformance/provisioner/static-deployment.yaml

internal/mode/static/manager.go

internal/framework/status/updater.go

internal/framework/status/updater_test.go

internal/mode/static/manager.go

internal/mode/static/leader.go

kate-osborn · 2023-09-14T17:17:36Z

more comments: (1) Would it make sense to add something about leader election to the architecture doc?

(2) What removes the leader election lock resource? Do we need to update our deinstallation instructions (helm and manifests) if it is not removed automatically?

(1) Sure, I can add something to the architecture doc.
(2). It's removed when you delete the namespace.

Makefile

ciarams87

🚀

docs/architecture.md

cmd/gateway/commands.go

internal/framework/status/updater.go

internal/mode/static/handler.go

internal/mode/static/config/config.go

internal/mode/static/manager.go

pleshakov

👍

bjee19 · 2023-09-18T20:56:49Z

https://github.com/nginxinc/nginx-kubernetes-gateway/blob/main/design/control-data-plane-separation/design.md

There is a line here that is: "Once we add leader election, the control plane will be able to scale.", would we like to change that now?

kate-osborn · 2023-09-18T21:32:25Z

https://github.com/nginxinc/nginx-kubernetes-gateway/blob/main/design/control-data-plane-separation/design.md

There is a line here that is: "Once we add leader election, the control plane will be able to scale.", would we like to change that now?

No, that's in an old design document. We don't need to keep those up-to-date.

bjee19

🚀🚀🚀

Co-authored-by: bjee19 <139261241+bjee19@users.noreply.github.com>

github-actions bot added documentation Improvements or additions to documentation enhancement New feature or request labels Sep 8, 2023

kate-osborn marked this pull request as ready for review September 8, 2023 21:38

kate-osborn requested a review from a team as a code owner September 8, 2023 21:38

bjee19 reviewed Sep 8, 2023

View reviewed changes

internal/framework/status/updater.go Show resolved Hide resolved

sjberman reviewed Sep 11, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

internal/framework/status/updater.go Outdated Show resolved Hide resolved

pleshakov reviewed Sep 12, 2023

View reviewed changes

kate-osborn requested review from sjberman, bjee19 and pleshakov September 14, 2023 19:00

sjberman reviewed Sep 14, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

sjberman approved these changes Sep 14, 2023

View reviewed changes

ciarams87 approved these changes Sep 15, 2023

View reviewed changes

pleshakov reviewed Sep 15, 2023

View reviewed changes

docs/architecture.md Show resolved Hide resolved

cmd/gateway/commands.go Show resolved Hide resolved

pleshakov reviewed Sep 15, 2023

View reviewed changes

internal/framework/status/updater.go Outdated Show resolved Hide resolved

kate-osborn force-pushed the feat/horizontal-scaling branch from 50e907a to d4b4423 Compare September 18, 2023 18:17

sjberman reviewed Sep 18, 2023

View reviewed changes

internal/mode/static/handler.go Outdated Show resolved Hide resolved

bjee19 reviewed Sep 18, 2023

View reviewed changes

internal/mode/static/config/config.go Outdated Show resolved Hide resolved

bjee19 reviewed Sep 18, 2023

View reviewed changes

internal/mode/static/config/config.go Outdated Show resolved Hide resolved

bjee19 reviewed Sep 18, 2023

View reviewed changes

internal/mode/static/manager.go Show resolved Hide resolved

pleshakov approved these changes Sep 18, 2023

View reviewed changes

kate-osborn mentioned this pull request Sep 18, 2023

Refactor Status Updater to be resource agnostic #1071

Closed

kate-osborn requested a review from bjee19 September 18, 2023 20:55

bjee19 approved these changes Sep 18, 2023

View reviewed changes

kate-osborn force-pushed the feat/horizontal-scaling branch from f0f0ed8 to ed875d2 Compare September 18, 2023 21:45

kate-osborn added 2 commits September 18, 2023 15:56

Add env var GOOS to makefile

75074e9

Add leader election

c179d66

kate-osborn and others added 24 commits September 18, 2023 15:56

Update helm README

ece0e64

Remove comment about leader election limitation

04c60a5

ARCH -> GOARCH

69c9683

Remove unnecessary context var name

ffd2a18

Group flag values

2a40805

Improve leader election lock name description

047f6ea

MY_* -> POD_*

c61e65b

LeaderElectorLogger

baca3a6

Make leader elector runnable by the manager

2437d29

Update architecture doc

7516e6b

GO_ARCH -> GOARCH

47dc941

Refactor status updater interface

e940333

Add comment about default values

de1af02

Update architecture doc

b30ce2d

Update cli-help doc

29fc37b

Add Status interface

972b745

Update the lock desc in helm values and readme

2ca2c30

Fix rebase error

536a6b4

Be consistent with disable var names

07d4ce2

Fix another rebase error

d0fbca8

Lowercase

89e9360

Update internal/mode/static/config/config.go

404d048

Co-authored-by: bjee19 <139261241+bjee19@users.noreply.github.com>

Update internal/mode/static/config/config.go

ff71d88

Co-authored-by: bjee19 <139261241+bjee19@users.noreply.github.com>

Be consistent with leader logs

28cd3b1

kate-osborn force-pushed the feat/horizontal-scaling branch from ed875d2 to 28cd3b1 Compare September 18, 2023 21:56

kate-osborn changed the title ~~Feat/horizontal scaling~~ Allow for Horizontal Scaling Sep 18, 2023

kate-osborn changed the title ~~Allow for Horizontal Scaling~~ Support Horizontal Scaling Sep 18, 2023

kate-osborn merged commit 43c1100 into nginxinc:main Sep 18, 2023
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Horizontal Scaling #1048

Support Horizontal Scaling #1048

kate-osborn commented Sep 8, 2023 •

edited

Loading

pleshakov left a comment

kate-osborn commented Sep 14, 2023

ciarams87 left a comment

pleshakov left a comment

bjee19 commented Sep 18, 2023

kate-osborn commented Sep 18, 2023

bjee19 left a comment

Support Horizontal Scaling #1048

Support Horizontal Scaling #1048

Conversation

kate-osborn commented Sep 8, 2023 • edited Loading

Proposed changes

Checklist

pleshakov left a comment

Choose a reason for hiding this comment

kate-osborn commented Sep 14, 2023

ciarams87 left a comment

Choose a reason for hiding this comment

pleshakov left a comment

Choose a reason for hiding this comment

bjee19 commented Sep 18, 2023

kate-osborn commented Sep 18, 2023

bjee19 left a comment

Choose a reason for hiding this comment

kate-osborn commented Sep 8, 2023 •

edited

Loading