Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make Kubernetes aware of the LoadBalancer behaviour #1392

Merged
merged 1 commit into from
Jun 18, 2020
Merged

Make Kubernetes aware of the LoadBalancer behaviour #1392

merged 1 commit into from
Jun 18, 2020

Conversation

Sh4d1
Copy link
Member

@Sh4d1 Sh4d1 commented Dec 5, 2019

First draft of KEP to allow an option to change kube proxy behaviour regarding the LoadBalancer IPs

Signed-off-by: Patrik Cyvoct patrik@ptrk.io

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 5, 2019
@k8s-ci-robot
Copy link
Contributor

Welcome @Sh4d1!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot
Copy link
Contributor

Hi @Sh4d1. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Dec 5, 2019
@Sh4d1 Sh4d1 mentioned this pull request Dec 5, 2019
Closed
@k8s-ci-robot k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/network Categorizes an issue or PR as relevant to SIG Network. labels Dec 5, 2019
@Sh4d1
Copy link
Member Author

Sh4d1 commented Dec 5, 2019

/assign @caseydavenport

jcodybaker
jcodybaker reviewed Dec 5, 2019
View reviewed changes
Copy link

@jcodybaker jcodybaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm an engineer with DigitalOcean (called out in the KEP's motivation section), and added some clarification in comments on the motivation.

keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated
1. Some cloud provider (DigitalOcean, Scaleway, ...) are using the LB's external IP as egress IP. This is a probem in the ipvs mode of kube proxy since the IP is bouded to an interface and healtchecks from the LB are never coming back.
2. Some cloud provider are also having features at the LB level (TLS termination, PROXY protocol, ...) and bypassing the LB means missing these features when the packet arrives to the service (leading to protocols errors).

So giving an options to these cloud to disable the actual beahviour would be very valuable.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One additional motivation:

Load-balancers which set status.loadBalancer.ingress[*].hostname instead of setting status.loadBalancer.ingress[*].ip don't exhibit this kube-proxy behavior. Because iptables/ipvs don't support rules based upon potentially dynamic hostnames, they leave the traffic untouched.

DigitalOcean has used this behavioral difference as a workaround the issue for customers who want to hairpin traffic through the LB (for example to get the ProxyProtocol headers appended). But it's hacky, and we much prefer the idea of addressing this functionality as the KEP proposes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep AWS does the same! Forgot to add it 😅

@Sh4d1 Sh4d1 changed the title First draft of the KEP Give the ability to the LoadBalancer provisioner to disable the LB IP binding Dec 5, 2019
@timoreimann timoreimann mentioned this pull request Dec 16, 2019
Open
@Sh4d1
Copy link
Member Author

Sh4d1 commented Dec 21, 2019

@caseydavenport @thockin the first draft looks good to me

@Sh4d1
Copy link
Member Author

Sh4d1 commented Jan 24, 2020

@thockin any updates on this?

@thockin thockin self-assigned this Feb 13, 2020
@munnerz munnerz mentioned this pull request Feb 20, 2020
Closed
@sftim
Copy link
Contributor

sftim commented Feb 22, 2020

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 22, 2020
@Sh4d1 Sh4d1 force-pushed the kube-proxy-disable-bind branch from b013220 to 46d6d37 Compare February 23, 2020 09:55
@Sh4d1 Sh4d1 mentioned this pull request Mar 13, 2020
Closed
MorrisLaw
MorrisLaw reviewed Mar 31, 2020
View reviewed changes
Copy link
Member

@MorrisLaw MorrisLaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! I've found some typos. I've crossed out and highlighted my suggestions to hopefully make it easier to make the changes. LGTM after you go through the comments 👍

keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
keps/sig-network/20191205-kube-proxy-IP-node-binding.md Outdated Show resolved Hide resolved
@Sh4d1 Sh4d1 force-pushed the kube-proxy-disable-bind branch from 0d81559 to f63cb1d Compare March 31, 2020 20:02
@Sh4d1
Copy link
Member Author

Sh4d1 commented Mar 31, 2020

Thanks for the review @MorrisLaw 😄 I think it's all resolved!

@Sh4d1
Copy link
Member Author

Sh4d1 commented Mar 31, 2020

@thockin any updates? I have a working code that seems to work 😄 Just need to choose the right naming!

@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 13, 2020
@Sh4d1
Copy link
Member Author

Sh4d1 commented Jun 13, 2020 •
edited
Loading

KEP is updated with a bit more detail. Not sure what else I can add, open to suggestions 😄

If this were a flag on kube-proxy, I would encourage (for example) GKE to set
the flag correctly and configure nodes to not do this bypass (or to only do it
sometimes).

Devil's advocate: If we are adding a flag to kube-proxy, why not just make that the ONLY mechanism?

As stated at the end of the KEP, it works for managed solutions. If a user uses a self hosted cluster with a cloud's CCM, he then needs to be aware of the LB behaviour and act accordingly. That's why I tend to prefer the cloud controller manager solution.
That said, in fact an additional flag to kube proxy (maybe not in the scope of this kep though?) might be useful as well.

@danwinship danwinship mentioned this pull request Jun 15, 2020
Closed
thockin
thockin reviewed Jun 15, 2020
View reviewed changes
Copy link
Member

@thockin thockin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am OK with this - @uablrek can you say whether this speaks to your need?

/approve

keps/sig-network/1860-kube-proxy-IP-node-binding/README.md Outdated Show resolved Hide resolved
keps/sig-network/1860-kube-proxy-IP-node-binding/README.md Outdated Show resolved Hide resolved
keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml Outdated Show resolved Hide resolved
keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml Outdated Show resolved Hide resolved
keps/sig-network/1860-kube-proxy-IP-node-binding/kep.yaml Outdated Show resolved Hide resolved
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Sh4d1, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 15, 2020
@andrewsykim andrewsykim mentioned this pull request Jun 18, 2020
Closed
@andrewsykim
Copy link
Member

@Sh4d1 can you squash your commits please?

@Sh4d1
Make Kubernetes aware of the LoadBalancer behaviour
615efc2 
Signed-off-by: Patrik Cyvoct <patrik@ptrk.io>
@Sh4d1 Sh4d1 force-pushed the kube-proxy-disable-bind branch from 0778bfa to 615efc2 Compare June 18, 2020 17:05
@Sh4d1
Copy link
Member Author

Sh4d1 commented Jun 18, 2020

@andrewrynhard squashed!

@andrewrynhard
Copy link

@andrewsykim ^ 🙂

@Sh4d1
Copy link
Member Author

Sh4d1 commented Jun 18, 2020

Oh 🙄 totally sorry for that HL 😅

@MorrisLaw
Copy link
Member

MorrisLaw commented Jun 18, 2020 •
edited
Loading

This should fit DigitalOcean's use case, lgtm 👍

@k8s-ci-robot k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jun 18, 2020
@MorrisLaw
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 18, 2020
@k8s-ci-robot k8s-ci-robot merged commit 8b9b994 into kubernetes:master Jun 18, 2020
@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone Jun 18, 2020
@Sh4d1 Sh4d1 mentioned this pull request Jun 19, 2020
Merged
@thockin
Copy link
Member

thockin commented Jun 26, 2020

@tgraf You might want to track this, since you have a kube-proxy replacement.

@MatthiasLohr MatthiasLohr mentioned this pull request Jul 6, 2020
Closed
@meyskens meyskens mentioned this pull request Oct 8, 2020
Closed
@Sh4d1 Sh4d1 mentioned this pull request Jan 4, 2021
Closed
@process0 process0 mentioned this pull request Jun 22, 2021
Closed
@RyanAoh RyanAoh mentioned this pull request Jun 29, 2023
Merged
@simon-wessel simon-wessel mentioned this pull request Mar 11, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@uablrek uablrek uablrek left review comments

@andrewsykim andrewsykim andrewsykim left review comments

@imroc imroc imroc left review comments

@thockin thockin thockin left review comments

@jcodybaker jcodybaker jcodybaker left review comments

@satyasm satyasm satyasm left review comments

@MorrisLaw MorrisLaw MorrisLaw approved these changes

@caseydavenport caseydavenport Awaiting requested review from caseydavenport

Assignees

@caseydavenport caseydavenport

@thockin thockin

@MorrisLaw MorrisLaw

@andrewsykim andrewsykim

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/network Categorizes an issue or PR as relevant to SIG Network. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
v1.19
Development

Successfully merging this pull request may close these issues.