Bad ingress annotation crashes all nginx-controllers once restarted

[garyhuang123]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): NO

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version:
0.16.2

Kubernetes version (use kubectl version):
1.11.5

Environment:

• Cloud provider or hardware configuration: On-premise VMware
• OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 (Core)
• Kernel (e.g. uname -a): Linux 3.10.0-957.5.1.el7.x86_64
• Install tools: rancher
• Others: rancher 2.1.3

What happened:

When user put '$' sign into ingress annotations,

• If nginx-controller is already running, it does not crash. As it rejects to reload with bad rules. But all of the future coming healthy rules are all rejected.
• If nginx-controller restarts, it fails to reload all rules when there are bad things (like, with ${variable}) in there. The /etc/nginx/nginx.conf becomes with no upstreams and /healthz location. Then Kubernetes probes pod is unhealthy with no /healthz location served. So it kills nginx-contoller and tries to restart it. But that repeats because of the continuous conf syntax check failure. And finally all nginx-controllers fall into the crashloopbackoff state.

What you expected to happen:

Nginx-controller should be as robust/stable enough as a gateway. Shouldn't be that easy to crash.

• If there are illegal signs inside ingress annotations, it should be rejected before applying into Kubernetes
• Even if there are some bad rules generated from ingress specs, nginx-controller should be able to filter them out and discard them, also should log down what rules are discarded for track.
• If there are bad rules there, nginx-controller should not just ignore all of the future good ones.

How to reproduce it (as minimally and precisely as possible):

1. Put ${variable} into the Ingress annotations, like,
   ‘nginx.ingress.kubernetes.io/upstream-vhost: mc.${teamServer}.dev.com’

   

2. Delete nginx-controller pods, so Kubernetes will create new ones, and you'll find all pods fail with below log
   

@aledbf, @ElvinEfendi

[aledbf]

Closing. Duplicated #3588