Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not return error if EndpointSlicesServiceKey func errors #1733

Merged

Conversation

swetharepakula
Copy link
Member

  • Service key may not be possible if EndpointSlice does not have all
    the expected labels set, namely the service label. This is a valid
    EPS and should be ignored when indexing. Returning an error causes
    the store to panic. Instead return empty list of keys and no error,
    so that the EPS is ignored.

Fixes #1730

@k8s-ci-robot k8s-ci-robot added do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 16, 2022
 * Service key may not be possible if EndpointSlice does not have all
   the expected labels set, namely the service label. This is a valid
   EPS and should be ignored when indexing. Returning an error causes
   the store to panic. Instead return empty list of keys and no error,
   so that the EPS is ignored.
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 16, 2022
@swetharepakula swetharepakula force-pushed the fix-eps-indexer-panic branch from a6ff573 to 41aa75c Compare June 16, 2022 02:35
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Jun 16, 2022
@swetharepakula
Copy link
Member Author

/assign @bowei

@swetharepakula
Copy link
Member Author

/assign @cezarygerard

@bowei
Copy link
Member

bowei commented Jun 16, 2022

It's strange that we are allowed to return an error given that it leads to a panic() in all cases?

@swetharepakula
Copy link
Member Author

I haven't found documentation on how the error should be used for IndexFunc, but an error from it will always cause the thread safe store to panic.

@kl52752
Copy link
Contributor

kl52752 commented Jun 17, 2022

can you point where this crash happend?

@swetharepakula
Copy link
Member Author

swetharepakula commented Jun 17, 2022

The panic happens in client-go in the store implementation used for all of the resources we watch. For reference: https://github.com/kubernetes/client-go/blob/master/tools/cache/thread_safe_store.go#L264-L275. For EndpointSlice we use our own indexer which triggers the panic. The issue #1730 mentions that this occurs in GKE and is reproducible when running the cluster manually.

I have manually testing with a local instance of ingress-gce that it panics without this change when it encounters an EndpointSlice without the service label and that this fix causes the controller to properly ignore that EndpointSlice.

@kl52752
Copy link
Contributor

kl52752 commented Jun 17, 2022

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 17, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kl52752, swetharepakula

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [kl52752,swetharepakula]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 2918b8a into kubernetes:master Jun 17, 2022
k8s-ci-robot added a commit that referenced this pull request Jun 21, 2022
[Cherrypick #1733 into release-1.15] Do not return error if EndpointSlicesServiceKey func errors
k8s-ci-robot added a commit that referenced this pull request Jun 21, 2022
[Cherrypick #1733 into release-1.14] Do not return error if EndpointSlicesServiceKey func errors
k8s-ci-robot added a commit that referenced this pull request Jun 21, 2022
[Cherrypick #1733 into release-1.16] Do not return error if EndpointSlicesServiceKey func errors
k8s-ci-robot added a commit that referenced this pull request Jun 21, 2022
[Cherrypick #1733 in release-1.17] Do not return error if EndpointSlicesServiceKey func errors
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ingress controller crash if endpoint slice missing service label
5 participants