Sharing TLS config between ingresses

[itaysk]

Is this a request for help?: yes

What keywords did you search in NGINX Ingress controller issues before filing this one?: share tls secret

I have multiple ingress resources matching the same host, each matching a different path. I'm now trying to add TLS to the application, but wouldn't want to add the same TLS configuration to every ingress (that happen to be owned by different people in the organization).
Is there a way to share the TLS configuration between ingresses?

In my trial and error I did reach a state where one of my ingresses that didn't have TLS did HTTPS because another ingress rule for the same host existed with the correct TLS config (I confirmed that by deleting the other ingress observed my TLS-less ingress starts to fail on HTTPS). I though this is by design but couldn't reproduce this behavior, perhaps it worked only because the order of ingress creation worked for the benefit of that scenario.

I am aware of --default-ssl-certificate flag but would prefer a solution that didn't involve modifying the ingress controller deployment (for non-technical reasons - separation of duties within the organization)

[aledbf]

@itaysk that's correct. If one of the ingresses in the same hostname has a tls section with a valid secret, the other ingresses will "inherit" the SSL configuration.

[aledbf]

I am aware of --default-ssl-certificate flag but would prefer a solution that didn't involve modifying the ingress controller deployment (for non-technical reasons - separation of duties within the organization)

This is useful only when you have a wildcard SSL certificate valid for all the hosts you want/need SSL

[itaysk]

The way you describe it didn't work for me, perhaps you could help me troubleshoot?
on my local cluster, I have the nginx-controller 0.24.1, an ingress with TLS configuration (and default backend) and some workload (including deployment and service) with ingress matching a path with no TLS config. All ingresses match the same host.

The workload ingress works on HTTP but for HTTPS logs:

[error] 39#39: *536 [lua] certificate.lua:77: call(): failed to convert certificate chain from PEM to DER: PEM_read_bio_X509_AUX() failed, context: ssl_certificate_by_lua*, client: 192.168.65.3, server: 0.0.0.0:443

The TLS secret must be valid because if I added a rule to the TLS ingress, that path would work on HTTPS.

full manifests:

TLS ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingtls
  namespace: ingress-nginx
  annotations:
    kubernetes.io/ingress.class: "kubernetes-nginx"
spec:
  tls:
    - hosts:
      - itaysk.test
      secretName: nginx-kube-tls
  backend:
    serviceName: yo-itaysk-test
    servicePort: 80

workload ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: kubernetes-nginx
  name: yo-itaysk-test-kube
  namespace: ingress-nginx
spec:
  rules:
  - host: itaysk.test
    http:
      paths:
      - backend:
          serviceName: yo-itaysk-test
          servicePort: 80
        path: /fff

[aledbf]

failed to convert certificate chain from PEM to DER

Please check the contents of the secret that contains the SSL certificate is valid.

[itaysk]

@aledbf as I mentioned above, the secret is valid and works for termination. If I just added a rule with a path to the ingtls ingress, then I could access that path using HTTPS. That's a good enough proof that the secret is not the issue right?

[aledbf]

as I mentioned above, the secret is valid and works for termination

How are you testing this?

That's a good enough proof that the secret is not the issue right?

No, because the error I posted in my previous comment if from openssl in nginx, not the ingress controller.

Just in case, we have two parts here, we validate the contents of the secret in the controller (Go code) but actually, until you try to use in nginx, there is no way to validate is ok from nginx side.

[itaysk]

How are you testing this?

adding a rule with a path to the ingtls ingress, then I can access that path using HTTPS.
meaning nginx (not the controller) manages to use this secret for TLS termination

[itaysk]

Hi @aledbf , I am reproducing this in a fresh Kubernetes cluster, dummy simple workload, and latest version of ingress. not sure what's missing :

• installed a sample workload (pod+service) in namespace default, called myapp

• ingress-nginx is installed using the chart from stable into namespace ingress

• created a cert and a kube secret, called server-secret in both namespaces default and ingress

  • the cert is valid. I can serve https traffic using it.

• created an ingress in ingress namespace for the sake of shared tls coniguration, called ingress-tls

  • it contains just tls and backend configurations (no rules).
  • in tls it declares the host, and references the secret server-secret

• I create an ingress, called myapp in default namespace

  • it has http rules to match a path to the service myapp on port 80

I can access the service via the ingress over http
I cannot access the service via ingress over https. here are some attempts I've made for the value of the tls section in the app ingress:

1. no tls at all:
   fails to start TLS, no server certificate is presented.
   from ingress logs:

[error] 262#262: *6844 [lua] certificate.lua:77: call(): failed to convert certificate chain from PEM to DER: PEM_read_bio_X509_AUX() failed, context: ssl_certificate_by_lua*, client: 192.168.65.3, server: 0.0.0.0:443

2. tls with host (same host as the host in ingress-tls) but without secret specified: certificate validation failed on the client.
   wrong server certificate was presented - for CN = Kubernetes Ingress Controller Fake Certificate

3. tls with host and secret specified: works but this is what I'm trying to avoid

also tried to create ingress-tls in default namespace as well, didn't affect the results above

[nr17]

I was also able to reproduce this issue with 0.24.1. It works with 0.23.0. Seems like a regression.

[rajatjindal]

to add up on comment from Nitin: we had a integration testcase which was failing and while debugging we reached this ticket here. downgrading the ingress controller to 0.23.0 works.

[aledbf]

If someone can provide a failing e2e test or concrete kubectl commands that reproduces this issue, it would be appreciated.

[nr17]

I created the issue on my minikube setup using the following steps.
Also, I have a temporary fix for it, will create a PR for it.

1. Install tiller
   helm init --history-max 200

2. Create self-signed cert
   openssl req -x509 -newkey rsa:4096 -sha256 -days 36 -nodes -keyout example.key -out example.crt -extensions san -config <(echo "[req]"; echo distinguished_name=req; echo "[san]"; echo subjectAltName=DNS:*.nginx-minikube-test.com) -subj /CN=nginx-minikube-test.com

3. Create secret in namespace kube-system
   kubectl create secret tls mycert --key example.key --cert example.crt

4. Install nginx-ingress helm chart
   customize values

   • change versions (Try 0.23.0 and 0.24.0)
   • opt for default cert:
     extraArgs:
     default-ssl-certificate: "kube-system/mycert"
     #extraArgs: {}

   helm upgrade --install --debug minikube ./nginx-ingress

5. Create ingress for the test:
   k apply -f ingress-non-tls.yaml

6. Test: 192.168.99.102 is minikube ip
   30469 is host port for svc minikube-nginx-ingress-controller:
   minikube-nginx-ingress-controller LoadBalancer 10.109.111.11 80:31405/TCP,443:30469/TCP 6m51s

   curl -kv --resolve tc.nginx-minikube-test.com:30469:192.168.99.102 'https://tc.nginx-minikube-test.com:30469/echo' -H'Host: tc.nginx-minikube-test.com'

[itaysk]

I have a question: given this scenario:
namespace-a/ingress-a: tls:hello.com, secret: mysecret
namespace-b/ingress-b: tls:hello.com
I'm putting the secret in namespace-a, is that correct?

[itaysk]

From my initial tests looks like this is fixed - Thanks!!
Is there a plan for an upcoming release soon?

[itaysk]

I saw in a response to another issue that it's targeted for end of the week, thanks
#4231 (comment)

[jmart1]

I have a question: given this scenario: namespace-a/ingress-a: tls:hello.com, secret: mysecret namespace-b/ingress-b: tls:hello.com I'm putting the secret in namespace-a, is that correct?

What is the answer to this question?