service-upstream sending traffic to unhealthy/terminating pods

[evandam]

What happened:

Ingress-nginx sends traffic to pods with failing readiness probes or a terminating state when using nginx.ingress.kubernetes.io/service-upstream: "true". Trying to access the same services directly from another pod (ex: curl podinfo.test.svc.cluster.local results in the expected behavior of not routing traffic to an unhealthy pod.

What you expected to happen:

When a pod is failing readiness probes or enters the terminating state, requests should not be sent to that pod.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.3.0
  Build:         2b7b74854d90ad9b4b96a5011b9e8b67d20bfb8f
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.10-eks-84b4fe6", GitCommit:"cc6a1b4915a99f49f5510ef0667f94b9ca832a8a", GitTreeState:"clean", BuildDate:"2022-06-09T18:24:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS EKS
• OS (e.g. from /etc/os-release):

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

• Kernel (e.g. uname -a): Linux ip-10-22-21-148.us-west-2.compute.internal 5.4.196-108.356.amzn2.x86_64 #1 SMP Thu May 26 12:49:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
  • EKS 1.22, Ingress-nginx installed with Helm chart version 4.2.3
• Basic cluster related info:
  • kubectl version

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.10-eks-84b4fe6", GitCommit:"cc6a1b4915a99f49f5510ef0667f94b9ca832a8a", GitTreeState:"clean", BuildDate:"2022-06-09T18:24:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

• kubectl get nodes -o wide

NAME                                         STATUS   ROLES    AGE    VERSION               INTERNAL-IP    EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-22-21-148.us-west-2.compute.internal   Ready    <none>   24h    v1.22.9-eks-810597c   10.22.21.148   <none>        Amazon Linux 2   5.4.196-108.356.amzn2.x86_64   docker://20.10.13
ip-10-22-30-61.us-west-2.compute.internal    Ready    <none>   6d2h   v1.22.9-eks-810597c   10.22.30.61    <none>        Amazon Linux 2   5.4.196-108.356.amzn2.x86_64   docker://20.10.13
ip-10-22-47-48.us-west-2.compute.internal    Ready    <none>   26h    v1.22.9-eks-810597c   10.22.47.48    <none>        Amazon Linux 2   5.4.196-108.356.amzn2.x86_64   docker://20.10.13
ip-10-22-6-195.us-west-2.compute.internal    Ready    <none>   16d    v1.22.9-eks-810597c   10.22.6.195    <none>        Amazon Linux 2   5.4.196-108.356.amzn2.x86_64   docker://20.10.13

• How was the ingress-nginx-controller installed:
  • If helm was used then please show output of helm ls -A | grep -i ingress

ingress-nginx               	ingress-nginx	15      	2022-08-24 14:07:01.595264 -0700 PDT	deployed	ingress-nginx-4.2.3                	1.3.0      

• If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>

USER-SUPPLIED VALUES:
controller:
  config:
    custom-http-errors: "503"
    http-snippet: |
      server {
        listen 18080;

        location /nginx_status {
          allow 127.0.0.1;
          allow ::1;
          allow 10.0.0.0/8;
          deny all;
          stub_status on;
        }

        location / {
          return 404;
        }
      }
    service-upstream: true
    use-forwarded-headers: true
  ingressClassResource:
    controllerValue: k8s.io/ingress-nginx
    default: false
    enabled: true
    name: nginx
  metrics:
    enabled: true
  podAnnotations:
    log.config.scalyr.com/attributes.parser: ingress-nginx
    prometheus.io/port: "10254"
    prometheus.io/scrape: "true"
  replicaCount: 1
  service:
    enableHttps: false
    externalTrafficPolicy: Local
    type: NodePort
  topologySpreadConstraints:
  - labelSelector:
      matchLabels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
    maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
defaultBackend:
  enabled: true
  extraEnvs:
  - name: DEBUG
    value: "true"
  extraVolumeMounts:
  - mountPath: /www
    name: custom-error-pages
  extraVolumes:
  - configMap:
      name: custom-error-pages
    name: custom-error-pages
  image:
    image: ingress-nginx/nginx-errors
    registry: k8s.gcr.io
    tag: 1.2.0
  replicaCount: 3

• If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

• if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses

Name:         alb
Labels:       app.kubernetes.io/instance=aws-load-balancer-controller
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=aws-load-balancer-controller
              app.kubernetes.io/version=v2.4.1
              helm.sh/chart=aws-load-balancer-controller-1.4.1
Annotations:  meta.helm.sh/release-name: aws-load-balancer-controller
              meta.helm.sh/release-namespace: kube-cloudops
Controller:   ingress.k8s.aws/alb
Parameters:
  APIGroup:  elbv2.k8s.aws
  Kind:      IngressClassParams
  Name:      alb
Events:      <none>


Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.3.0
              helm.sh/chart=ingress-nginx-4.2.3
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

• kubectl -n <ingresscontrollernamespace> get all -A -o wide

NAME                                                READY   STATUS    RESTARTS   AGE    IP             NODE                                         NOMINATED NODE   READINESS GATES
pod/ingress-nginx-controller-6c96574cb-kkmj9        1/1     Running   0          38m    10.22.32.220   ip-10-22-47-48.us-west-2.compute.internal    <none>           2/2
pod/ingress-nginx-defaultbackend-5fd9bff589-n4dn8   1/1     Running   0          23h    10.22.16.103   ip-10-22-21-148.us-west-2.compute.internal   <none>           <none>
pod/ingress-nginx-defaultbackend-5fd9bff589-swtkz   1/1     Running   0          6d1h   10.22.23.214   ip-10-22-30-61.us-west-2.compute.internal    <none>           <none>
pod/ingress-nginx-defaultbackend-5fd9bff589-t2sd2   1/1     Running   0          9d     10.22.5.88     ip-10-22-6-195.us-west-2.compute.internal    <none>           <none>

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/ingress-nginx-controller             NodePort    172.22.15.224    <none>        80:32662/TCP   82d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission   ClusterIP   172.22.103.165   <none>        443/TCP        82d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-metrics     ClusterIP   172.22.252.30    <none>        10254/TCP      68d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-defaultbackend         ClusterIP   172.22.181.225   <none>        80/TCP         82d   app.kubernetes.io/component=default-backend,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                      IMAGES                                                                                                                    SELECTOR
deployment.apps/ingress-nginx-controller       1/1     1            1           82d   controller                      registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
deployment.apps/ingress-nginx-defaultbackend   3/3     3            3           82d   ingress-nginx-default-backend   k8s.gcr.io/ingress-nginx/nginx-errors:1.2.0                                                                               app.kubernetes.io/component=default-backend,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME                                                      DESIRED   CURRENT   READY   AGE   CONTAINERS                      IMAGES                                                                                                                    SELECTOR
replicaset.apps/ingress-nginx-controller-56684f67         0         0         0       76d   controller                      k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185        app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=56684f67
replicaset.apps/ingress-nginx-controller-57d4f94b4c       0         0         0       20d   controller                      k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185        app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=57d4f94b4c
replicaset.apps/ingress-nginx-controller-6c96574cb        1         1         1       38m   controller                      registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6c96574cb
replicaset.apps/ingress-nginx-controller-6dcdd64b99       0         0         0       81d   controller                      k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185        app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6dcdd64b99
replicaset.apps/ingress-nginx-controller-774bdffc94       0         0         0       82d   controller                      k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185        app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=774bdffc94
replicaset.apps/ingress-nginx-controller-7bd8b8848c       0         0         0       76d   controller                      k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185        app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=7bd8b8848c
replicaset.apps/ingress-nginx-controller-7c5777474f       0         0         0       20d   controller                      k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185        app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=7c5777474f
replicaset.apps/ingress-nginx-controller-c866f7666        0         0         0       68d   controller                      k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185        app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=c866f7666
replicaset.apps/ingress-nginx-defaultbackend-5fd9bff589   3         3         3       82d   ingress-nginx-default-backend   k8s.gcr.io/ingress-nginx/nginx-errors:1.2.0                                                                               app.kubernetes.io/component=default-backend,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5fd9bff589

• kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>: See above

• kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>: See above

• Current state of ingress object, if applicable:

  • kubectl -n <appnnamespace> get all,ing -o wide

NAME                          READY   STATUS    RESTARTS   AGE     IP             NODE                                         NOMINATED NODE   READINESS GATES
pod/podinfo-cdf8fc69d-62mjc   1/1     Running   0          32m     10.22.35.201   ip-10-22-47-48.us-west-2.compute.internal    <none>           <none>
pod/podinfo-cdf8fc69d-t7tmv   0/1     Running   0          35m     10.22.24.160   ip-10-22-21-148.us-west-2.compute.internal   <none>           <none>
pod/redis-6bc5f4d85c-46ccj    1/1     Running   0          3h11m   10.22.44.227   ip-10-22-47-48.us-west-2.compute.internal    <none>           <none>
pod/ubuntu-647674d858-l4mn4   1/1     Running   0          8d      10.22.14.43    ip-10-22-6-195.us-west-2.compute.internal    <none>           <none>

NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE    SELECTOR
service/podinfo   ClusterIP   172.22.210.53    <none>        80/TCP     82d    app.kubernetes.io/name=podinfo
service/redis     ClusterIP   172.22.143.217   <none>        6379/TCP   5d6h   app=redis

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                        SELECTOR
deployment.apps/podinfo   1/2     2            1           82d    podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo
deployment.apps/redis     1/1     1            1           5d6h   redis        redis:6                       app=redis
deployment.apps/ubuntu    1/1     1            1           61d    ubuntu       ubuntu:20.04                  name=ubuntu

NAME                                 DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                        SELECTOR
replicaset.apps/podinfo-564497c5c4   0         0         0       5d6h    podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=564497c5c4
replicaset.apps/podinfo-5c6d6649d7   0         0         0       4h32m   podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=5c6d6649d7
replicaset.apps/podinfo-5fc67cb588   0         0         0       82d     podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=5fc67cb588
replicaset.apps/podinfo-65b69dff74   0         0         0       80m     podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=65b69dff74
replicaset.apps/podinfo-6769d9f54c   0         0         0       4h36m   podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=6769d9f54c
replicaset.apps/podinfo-699cc7cfd9   0         0         0       77m     podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=699cc7cfd9
replicaset.apps/podinfo-784bc4bff5   0         0         0       2d4h    podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=784bc4bff5
replicaset.apps/podinfo-7c86ff6654   0         0         0       90m     podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=7c86ff6654
replicaset.apps/podinfo-7df899f4cc   0         0         0       82m     podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=7df899f4cc
replicaset.apps/podinfo-87bd97c48    0         0         0       79d     podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=87bd97c48
replicaset.apps/podinfo-cdf8fc69d    2         2         1       48m     podinfo      stefanprodan/podinfo:latest   app.kubernetes.io/name=podinfo,pod-template-hash=cdf8fc69d
replicaset.apps/redis-6bc5f4d85c     1         1         1       5d6h    redis        redis:6                       app=redis,pod-template-hash=6bc5f4d85c
replicaset.apps/ubuntu-647674d858    1         1         1       61d     ubuntu       ubuntu:20.04                  name=ubuntu,pod-template-hash=647674d858

NAME                                CLASS   HOSTS                            ADDRESS         PORTS   AGE
ingress.networking.k8s.io/podinfo   nginx   cloudops-podinfo.example.com   172.22.15.224   80      82d

• kubectl -n <appnamespace> describe ing <ingressname>

Name:             podinfo
Labels:           <none>
Namespace:        cloudops
Address:          172.22.15.224
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                            Path  Backends
  ----                            ----  --------
  cloudops-podinfo.example.com
                                  /   podinfo:http (10.22.35.201:9898)
Annotations:                      external-dns.alpha.kubernetes.io/target: dev-ingress.example.com
                                  external-dns.alpha.kubernetes.io/ttl: 60
                                  nginx.ingress.kubernetes.io/affinity-mode: balanced
                                  nginx.ingress.kubernetes.io/service-upstream: true
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    58m (x3 over 16d)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    38m (x2 over 42m)  nginx-ingress-controller  Scheduled for sync

• If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

curl -sv https://cloudops-podinfo.example.com/
*   Trying 54.185.239.110:443...
* connect to 54.185.239.110 port 443 failed: Connection refused
*   Trying 52.35.220.84:443...
* Connected to cloudops-podinfo.example.com/ (52.35.220.84) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=example.com/
*  start date: Mar 17 00:00:00 2022 GMT
*  expire date: Apr 15 23:59:59 2023 GMT
*  subjectAltName: host "cloudops-podinfo.example.com/" matched cert's "*.example.com/"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x15a812c00)
> GET / HTTP/2
> Host: cloudops-podinfo.example.com
> user-agent: curl/7.79.1
> accept: */*
> cache-control: no-cache
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Wed, 24 Aug 2022 21:52:07 GMT
< content-type: application/json; charset=utf-8
< content-length: 342
< x-content-type-options: nosniff
<
{
  "hostname": "podinfo-cdf8fc69d-62mjc",
  "version": "6.2.0",
  "revision": "",
  "color": "#34577c",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "greetings from blue",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.18.5",
  "num_goroutine": "13",
  "num_cpu": "4"
* Connection #0 to host cloudops-podinfo.example.com left intact
}%

• Others:
  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Install minikube/kind

• Minikube https://minikube.sigs.k8s.io/docs/start/
• Kind https://kind.sigs.k8s.io/docs/user/quick-start/

Install the ingress controller

Install ingress-controller with Helm values provided above.

Install an application that will act as default backend (is just an echo app)

kubectl apply -f https://gist.githubusercontent.com/evandam/05d15d72645827f18ef627096e129ea4/raw/fc40b527cf9d09f6af7b87eabcc5d8aa21b2b9e9/podinfo.yaml

Create an ingress (please add any additional annotation required)

echo "
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: podinfo
  namespace: cloudops
  annotations:
    external-dns.alpha.kubernetes.io/target: dev-ingress.example.com
    external-dns.alpha.kubernetes.io/ttl: "60"
    nginx.ingress.kubernetes.io/service-upstream: "true"
spec:
  ingressClassName: nginx
  rules:
    - host: cloudops-podinfo.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: podinfo
                port:
                  name: http
" | kubectl apply -f -

make a request

POD_NAME=$(k get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -o NAME)
kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H 'Host: cloudops-podinfo.example.com/' localhost

Cause the readiness probe to start failing:

kubectl exec -n cloudops deploy/podinfo -- curl -v -X POST localhost:9898/readyz/disable

Note that ingress-nginx continues to send traffic to the unhealthy pod, but accessing the service directly works as expected

kubectl exec -n ingress-nginx deploy/ingress-nginx-controller -- curl podinfo.cloudops.svc.cluster.local/

Anything else we need to know:

Our environment has an AWS ALB ingress in front of ingress-nginx, so configs may be slightly different but the issue is the same.

[k8s-ci-robot]

@evandam: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/remove-kind bug
@evandam I think your expectation is invalid but I am not an expert.
My comment is based on the fact a single upstream was chosen in nginx.conf (as a consequence of using this annotation). Without needing to state it implies that there is no alternative upstream to use https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#service-upstream . This is already hinted in a logical sense in the comments in the docs that I linked to. It says on error the request will not be dispatched to another. I think its useful for zero-downtime deployment as stated as traffic will always go to upstream but outside of zero-time usecase, this kind of problem is expected.

But lets wait for other comments as I am not an expert.

[scalp42]

The issue is that switching to using endpoints creates other issues like being out of sync and needing the hooks with sleep as workaround 😅

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Ghilteras]

can someone remove the stale tag and reapply the bug one please?

[longwuyuan]

• The comment from @scalp42 has a lot of relevance
• Some other users have reported unexpected results around this topic
• Nobody has clarified or elucidated the step by step process and logic of decision making to choose routing destination in the scene of using (and thus forcing a specific upstream)
• This data is not enough to illustrate a bug or demonstrate a problem that needs to be worked on by the project developers
• Please post data with tests, configs, resource states, curl requests, logs and all such info that a developer will need to analyze and debug the perceived problem here
• Until such data is posted, this issue is just adding to the tally of open issues without any perceivable action item for anyone or a trajectory of this discussion
• So I will close this issue for now with the message that the origin createor of the issue can repost data that makes sense to a developer in the context of proof of a bug or explanation for a problem that needs to be solved by the project. Please provide data from tests on the recent release of the controller
  /close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

• The comment from @scalp42 has a lot of relevance
• Some other users have reported unexpected results around this topic
• Nobody has clarified or elucidated the step by step process and logic of decision making to choose routing destination in the scene of using (and thus forcing a specific upstream)
• This data is not enough to illustrate a bug or demonstrate a problem that needs to be worked on by the project developers
• Please post data with tests, configs, resource states, curl requests, logs and all such info that a developer will need to analyze and debug the perceived problem here
• Until such data is posted, this issue is just adding to the tally of open issues without any perceivable action item for anyone or a trajectory of this discussion
• So I will close this issue for now with the message that the origin createor of the issue can repost data that makes sense to a developer in the context of proof of a bug or explanation for a problem that needs to be solved by the project. Please provide data from tests on the recent release of the controller
  /close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[scalp42]

It's a shame, a lot of information was provided and reproducible steps. Thanks for the feedback either way, cheers!

[longwuyuan]

@scalp42 sorry you feel that way. But where do we go from here. The docs clearly indicate that that annotation helps in use case of zero-downtime-deployments https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#service-upstream . The reference discussion is here #257

So it looks like this issue description

• is precisely or explicitly about a zero-downtime deployment use-case implying that one known upstream will be reliably available regardless
• is about knowingly choosing one specific upstream
• is about facing a possibility of the chosen upstream unhealthy
• and then expecting the controller to dynamically reconfigure itself away of the existing explicitly configured behaviour to route to a single dedicated upstream

While this does not seem like a K8S KEP Ingress API spec, its possible it can be done. But it requires developers time. The first step to that is triaging the issue to completion. The second step is a expert to opine on available options for action.

Since there is no action item being tracked here, adding it to the tally of 480 open issues can be avoided by closing it. In case the triaging data and the expert discussions related data get posted here, a relevant tag like bug or feature etc can be set by the creator of the issue, when they re-open the issue