controller: Don't panic when ready condition in a endpointslice is missing

[schoentoon]

While I was setting up an EndpointSlice it wasn't instantly obvious that conditions->ready was actually required as shown in https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
Rather than the ingress that pointed to the service that used this endpointslice just not working, I found ingress-nginx in a CrashBackoffLoop instead. With a panic in the logs. Stacktrace being the following:

goroutine 212 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1809060?, 0x293cf10})
        k8s.io/apimachinery@v0.25.3/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x173a860?})
        k8s.io/apimachinery@v0.25.3/pkg/util/runtime/runtime.go:49 +0x75
panic({0x1809060, 0x293cf10})
        runtime/panic.go:884 +0x212
k8s.io/ingress-nginx/internal/ingress/controller.getEndpointsFromSlices(0xc0008b16b0, 0xc00053b648, {0x1a4bc03, 0x3}, 0xc00053b630)
        k8s.io/ingress-nginx/internal/ingress/controller/endpointslices.go:115 +0xede
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).serviceEndpoints(0xc0001ce3c0, {0xc000045030, 0x10}, {0xc000045040, 0x4})
        k8s.io/ingress-nginx/internal/ingress/controller/controller.go:1110 +0x5ec
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).createUpstreams(0xc0001ce3c0, {0xc00058e580, 0x10, 0x10?}, 0xc0006d11e0)
        k8s.io/ingress-nginx/internal/ingress/controller/controller.go:1016 +0x19e5
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).getBackendServers(0xc0001ce3c0, {0xc00058e580?, 0x10, 0x10})
        k8s.io/ingress-nginx/internal/ingress/controller/controller.go:608 +0x8e
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).getConfiguration(0xc0001ce3c0, {0xc00058e580, 0x10, 0x0?})
        k8s.io/ingress-nginx/internal/ingress/controller/controller.go:513 +0x4e
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress(0xc0001ce3c0, {0x174d640, 0x1})
        k8s.io/ingress-nginx/internal/ingress/controller/controller.go:155 +0x9a
k8s.io/ingress-nginx/internal/task.(*Queue).worker(0xc00043e3f0)
        k8s.io/ingress-nginx/internal/task/queue.go:129 +0x446
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
        k8s.io/apimachinery@v0.25.3/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x1cce3a0, 0xc0004d5440}, 0x1, 0xc00024cc00)
        k8s.io/apimachinery@v0.25.3/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        k8s.io/apimachinery@v0.25.3/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        k8s.io/apimachinery@v0.25.3/pkg/util/wait/wait.go:92
k8s.io/ingress-nginx/internal/task.(*Queue).Run(0x0?, 0x0?, 0x0?)
        k8s.io/ingress-nginx/internal/task/queue.go:61 +0x45
created by k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).Start
        k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:306 +0x3bc
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15a627e]

What this PR does / why we need it:

This will make it so that invalid EndpointSlice (or at least, the once missing ready) are handled just like handle: false rather than it crashing.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• CVE Report (Scanner found CVE and adding report)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation only

Which issue/s this PR fixes

I was going to make an issue for this, then I looked at the source and realized how easy the fix would be.

How Has This Been Tested?

I quite frankly didn't test this as the change is so trivial.

Checklist:

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I've read the CONTRIBUTION guide
• I have added unit and/or e2e tests to cover my changes.
• All new and existing tests passed.
• Added Release Notes.

Does my pull request need a release note?

Any user-visible or operator-visible change qualifies for a release note. This could be a:

• CLI change
• API change
• UI change
• configuration schema change
• behavioral change
• change in non-functional attributes such as efficiency or availability, availability of a new platform
• a warning about a deprecation
• fix of a previous Known Issue
• fix of a vulnerability (CVE)

No release notes are required for changes to the following:

• Tests
• Build infrastructure
• Fixes for unreleased bugs

For more tips on writing good release notes, check out the Release Notes Handbook

NONE

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: schoentoon / name: Toon Schoenmakers (ea2fec0c5b6dc3ea7e1ca32dfe4716095134bdfe)

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

Welcome @schoentoon!

It looks like this is your first PR to kubernetes/ingress-nginx 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/ingress-nginx has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @schoentoon. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[strongjz]

/ok-to-test

[tombokombo]

@schoentoon I was implementing slices, and yes, ready is optional according docs, sorry for problem. But read the docs
https://pkg.go.dev/k8s.io/api/discovery/v1#EndpointConditions

 A nil value indicates an unknown state. In most cases consumers should interpret this unknown state as ready.

so the fix shoud be

if (ep.Conditions.Ready != nil) && !(*ep.Conditions.Ready) {

Now, you are skipping this unknown state endpoints which is against docs ( in most cases <- whatever this means). Could you test it, share results and better describe your setup? Thx

[schoentoon]

You're absolutely right and I must have read over that. I modified my pull request accordingly.

[tombokombo]

You're absolutely right and I must have read over that. I modified my pull request accordingly.

perfect, but Im still wondering about your setup and actual backends map from nginx in both cases.

[schoentoon]

So, as I mentioned in the other issue. I manually create an EndpointSlice pointing to an IP address outside my cluster. Below is basically a direct copy from my ansible playbook to set that up. I initially ran into this crash when setting up the EndpointSlice without any conditions at all. The only reason I went this route to begin with, was that my previous setup (a Service with type ExternalName) kept giving a certain warning in I believe ingress-nginx (I would have to double check that) when in use with an IP address rather than a domain name.

- name: Create a Service
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: Service
      metadata:
        name: "jellyfin"
        namespace: "{{ k8s_namespace }}"
        labels:
          app: "jellyfin"
      spec:
        type: ClusterIP
        clusterIP: None
        ports:
          - protocol: TCP
            port: 8096

- name: Create a EndpointSlice
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: EndpointSlice
      metadata:
        name: "jellyfin"
        namespace: "{{ k8s_namespace }}"
        labels:
          app: "jellyfin"
          kubernetes.io/service-name: "jellyfin"
      addressType: IPv4
      ports:
        - protocol: TCP
          port: 8096
      endpoints:
        - addresses:
            - "10.0.0.137"
          conditions:
            ready: true

- name: Create an Ingress
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: "jellyfin"
        namespace: "{{ k8s_namespace }}"
        annotations:
          kubernetes.io/ingress.class: "nginx"
          nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
          nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
          nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
      spec:
        rules:
        - host: "jellyfin.<snip>"
          http:
            paths:
            - pathType: Prefix
              path: "/"
              backend:
               service:
                 name: "jellyfin"
                 port:
                   number: 8096

[tombokombo]

ok, and why not to use nginx.ingress.kubernetes.io/service-upstream annotion https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#service-upstream ? Just create ingress+service, nothing more. Because creating edp slices by hand is too low level as there is endpointslices-controller as part of dataplane which is taking care of them.

[schoentoon]

I simply didn't run into that option really. But thanks, I'll have a look at that tonight.

[tombokombo]

I've check the code, service-upstream will just use svc ClusterIp as upstream backend address,this will not help. But you can create svc and endpoint, endpointslices are autogenereted for you by controller from endpoint. Endpoint api is far more simpler then slices, and there is no mention that it's going to be deprecated/removed.

apiVersion: v1
kind: Endpoints
metadata:
  name: test-ext
  namespace: test-ns
subsets:
- addresses:
  - ip: YOUR.TARGET.IP
  ports:
  - name: http
    port: 80
    protocol: TCP
 

[strongjz]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: schoentoon, strongjz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [strongjz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment