From 806e3346989ac8c941550f4844f0a344d93a242f Mon Sep 17 00:00:00 2001 From: Patrik Cyvoct Date: Sat, 7 Nov 2020 10:58:55 +0100 Subject: [PATCH 1/4] Propose KEP 1860 as implementable Signed-off-by: Patrik Cyvoct --- .../1860-kube-proxy-IP-node-binding/README.md | 65 +++++++++++++++---- .../1860-kube-proxy-IP-node-binding/kep.yaml | 24 +++++-- 2 files changed, 71 insertions(+), 18 deletions(-) diff --git a/keps/sig-network/1860-kube-proxy-IP-node-binding/README.md b/keps/sig-network/1860-kube-proxy-IP-node-binding/README.md index 0eb80248fda..23c255984af 100644 --- a/keps/sig-network/1860-kube-proxy-IP-node-binding/README.md +++ b/keps/sig-network/1860-kube-proxy-IP-node-binding/README.md @@ -16,17 +16,20 @@ - [Risks and Mitigations](#risks-and-mitigations) - [Drawbacks](#drawbacks) - [Alternatives](#alternatives) +- [Design Details](#design-details) +- [Test Plan](#test-plan) +- [Graduation Criteria](#graduation-criteria) ## Release Signoff Checklist Items marked with (R) are required *prior to targeting to a milestone / release*. -- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) -- [ ] (R) KEP approvers have approved the KEP status as `implementable` -- [ ] (R) Design details are appropriately documented -- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input -- [ ] (R) Graduation criteria is in place +- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [x] (R) KEP approvers have approved the KEP status as `implementable` +- [x] (R) Design details are appropriately documented +- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input +- [x] (R) Graduation criteria is in place - [ ] (R) Production readiness review completed - [ ] Production readiness review approved - [ ] "Implementation History" section is up-to-date for milestone @@ -66,31 +69,31 @@ Currently there is some hacky workaround that set the `Hostname` on the `Service ## Proposal -The solution would be to add a new field in the `loadBalancer` field of a `Service`'s `status`, like `ingressMode`. This new field will be used by kube-proxy in order to not bind the Load Balancer's External IP to the node (in both IPVS and iptables mode). The value `VIP` would be the default one (if not set for instance), keeping the current behaviour. The value `Proxy` would be used in order to disable the the shortcut. +The solution would be to add a new field in the `loadBalancer` field of a `Service`'s `status`, like `ipMode`. This new field will be used by kube-proxy in order to not bind the Load Balancer's External IP to the node (in both IPVS and iptables mode). The value `VIP` would be the default one (if not set for instance), keeping the current behaviour. The value `Proxy` would be used in order to disable the the shortcut. -Since the `EnsureLoadBalancer` returns a `LoadBalancerStatus`, the Cloud Controller Manager can optionally set the `ingressMode` field before returning the status. +Since the `EnsureLoadBalancer` returns a `LoadBalancerStatus`, the Cloud Controller Manager can optionally set the `ipMode` field before returning the status. ### User Stories #### Story 1 -User 1 is a Managed Kubernetes user. There cluster is running on a cloud provider where the LB's behaviour matches the `VIP` `ingressMode`. +User 1 is a Managed Kubernetes user. There cluster is running on a cloud provider where the LB's behaviour matches the `VIP` `ipMode`. Nothing changes for them since the cloud provider manages the Cloud Controller Manager. #### Story 2 -User 2 is a Managed Kubernetes user. There cluster is running on a cloud provider where the LB's behaviour matches the `Proxy` `ingressMode`. +User 2 is a Managed Kubernetes user. There cluster is running on a cloud provider where the LB's behaviour matches the `Proxy` `ipMode`. Almost nothing changes for them since the cloud provider manages the Cloud Controller Manager. The only difference is that pods using the load balancer IP may observe a higher response time since the datapath will go through the load balancer. #### Story 3 User 3 is a developer working on a cloud provider Kubernetes offering. -On the next version of `k8s.io/cloud-provider`, the cloud provider can optionally set the `ingressMode` field as part of `LoadBalancerStatus`, and reflect the cloud provider load balancer behaviour. +On the next version of `k8s.io/cloud-provider`, the cloud provider can optionally set the `ipMode` field as part of `LoadBalancerStatus`, and reflect the cloud provider load balancer behaviour. ### Risks and Mitigations -1. The first risk is when using the `Proxy` `ingressMode` for pod using the load balancer IP. In this case the packets will not bypass the load balancer anymore. +1. The first risk is when using the `Proxy` `ipMode` for pod using the load balancer IP. In this case the packets will not bypass the load balancer anymore. However if a user wants to to keep using the in cluster datapath, he can still use the ClusterIP of the service. ## Drawbacks @@ -102,3 +105,43 @@ However if a user wants to to keep using the in cluster datapath, he can still u A viable alternative may be to use a flag directly on `kube-proxy`. When running on a cloud provider managed Kubernetes this solution is viable since the cloud provider will be able to set the right value on `kube-proxy`. When running a self hosted cluster, the user needs to be aware of how the cloud's load balancers works and need to set the flag on `kube-proxy` accordingly. + +## Design Details + +API changes to Service: +- Add a new field `status.loadBalancer.ingress[].ipMode: VIP | Proxy`. +- `ipMode` defaults to VIP if the feature gate is enabled, `nil` otherwise, preserving existing behaviour for Service Type=LoadBalancer. +- On create and update, it fails if `ipMode` is set without the `ip` field. + +## Test Plan + +Unit tests: +- unit tests for the ipvs and iptables rules +- unit tests for the validation +- unit tests for a new util in pkg/proxy + +E2E tests: +- The default behavior for `ipMode` does not break any existing e2e tests +- The default `VIP` value is already tested + +## Graduation Criteria + +### Alpha + +Adds new field `ipMode` to Service, which is used when `LoadBalancerIPMode` feature gate is enabled, allowing for rollback. + +### Beta/GA + +`LoadBalancerIPMode` is enabled by default. + +### Upgrade / Downgrade Strategy + +On upgrade, while the feature gate is disabled, nothing will change. Once the feature gate is enabled, all the previous LoadBalancer service will get an `ipMode` of `VIP`. +If `kube-proxy` was not yet upgraded: the field will simply be ignored. +If `kube-proxy` was upgraded, and the feature gate enabled, it will stil behave as before if the `ipMode` is `VIP`, and will behave accordingly if the `ipMode` is `Proxy`. + +On downgrade, the feature gate will simply be disabled, and as long as `kube-proxy` was downgraded before, nothing should be impacted. + +### Version Skew Strategy + +Version skew from the control plane to `kube-proxy` should be trivial since `kube-proxy` will simply ignore the `ipMode` field. diff --git a/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml b/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml index c56c71d0c4a..ecf87197e93 100644 --- a/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml +++ b/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml @@ -5,6 +5,7 @@ authors: owning-sig: sig-network participating-sigs: - sig-cloud-provider +status: implementable reviewers: - "@thockin" - "@cheftako" @@ -12,10 +13,19 @@ reviewers: approvers: - "@thockin" - "@andrewsykim" -editor: "@Sh4d1" -creation-date: 2019-12-05 -last-updated: 2020-06-18 -status: provisional -replaces: -superseded-by: -see-also: + +stage: "alpha" + +latest-milestone: "v1.21" + +milestone: + alpha: "v1.21" + beta: "v1.22" + stable: "v1.22" + +feature-gates: + - name: LoadBalancerIPMode + components: + - kube-apiserver + - kube-proxy +disable-supported: true From 70fd12eba2f5de5ed879add5be1ee6118f1e37cd Mon Sep 17 00:00:00 2001 From: Patrik Cyvoct Date: Sat, 7 Nov 2020 11:03:32 +0100 Subject: [PATCH 2/4] fix toc Signed-off-by: Patrik Cyvoct --- keps/sig-network/1860-kube-proxy-IP-node-binding/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/keps/sig-network/1860-kube-proxy-IP-node-binding/README.md b/keps/sig-network/1860-kube-proxy-IP-node-binding/README.md index 23c255984af..0ffcae16914 100644 --- a/keps/sig-network/1860-kube-proxy-IP-node-binding/README.md +++ b/keps/sig-network/1860-kube-proxy-IP-node-binding/README.md @@ -19,6 +19,10 @@ - [Design Details](#design-details) - [Test Plan](#test-plan) - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Beta/GA](#betaga) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) ## Release Signoff Checklist From a6f75768280d3d3f1cb3a50fa59eb325c56afaa9 Mon Sep 17 00:00:00 2001 From: Patrik Cyvoct Date: Mon, 9 Nov 2020 07:27:42 +0100 Subject: [PATCH 3/4] set to implemented Signed-off-by: Patrik Cyvoct --- keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml b/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml index ecf87197e93..2c0a96ef2fc 100644 --- a/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml +++ b/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml @@ -5,7 +5,7 @@ authors: owning-sig: sig-network participating-sigs: - sig-cloud-provider -status: implementable +status: implemented reviewers: - "@thockin" - "@cheftako" From 0e987cdf859d790118c2bc706315b9aa2ed2e15c Mon Sep 17 00:00:00 2001 From: Patrik Cyvoct Date: Wed, 11 Nov 2020 10:50:36 +0100 Subject: [PATCH 4/4] set to implementable Signed-off-by: Patrik Cyvoct --- keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml b/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml index 2c0a96ef2fc..ecf87197e93 100644 --- a/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml +++ b/keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml @@ -5,7 +5,7 @@ authors: owning-sig: sig-network participating-sigs: - sig-cloud-provider -status: implemented +status: implementable reviewers: - "@thockin" - "@cheftako"