ingress controller is picking up a change in nginx config map, but it takes a while to apply it to nginx.conf

[mwasilew2]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):
configmap
nginx
reload
whitelist

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

NGINX Ingress controller version:
0.15.0

Kubernetes version (use kubectl version):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.8-gke.0", GitCommit:"6e5b33a290a99c067003632e0fd6be0ead48b233", GitTreeState:"clean", BuildDate:"2018-02-16T18:26:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: GKE
• OS (e.g. from /etc/os-release):

cat /etc/*-release

CHROMEOS_AUSERVER=https://tools.google.com/service/update2
CHROMEOS_BOARD_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_RELEASE_BOARD=lakitu-signed-mpkeys
CHROMEOS_RELEASE_BRANCH_NUMBER=12
CHROMEOS_RELEASE_BUILDER_PATH=lakitu-release/R65-10323.12.0
CHROMEOS_RELEASE_BUILD_NUMBER=10323
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_DESCRIPTION=10323.12.0 (Official Build) beta-channel lakitu
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=0
CHROMEOS_RELEASE_TRACK=beta-channel
CHROMEOS_RELEASE_VERSION=10323.12.0
DEVICETYPE=OTHER
GOOGLE_RELEASE=10323.12.0
HWID_OVERRIDE=LAKITU DEFAULT
BUILD_ID=10323.12.0
NAME="Container-Optimized OS"
KERNEL_COMMIT_ID=2d7de0bde20ae17f934c2a2e44cb24b6a1471dec
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=65
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME="Container-Optimized OS from Google"
VERSION=65
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/"
ID=cos

• Kernel (e.g. uname -a): 4.4.111+
• Install tools:
  nginx ingress controller deployed with:
  helm install stable/nginx-ingress --name ingress --namespace kube-system --version 0.19.0 -f my_values.yaml

my_values.yaml:

controller:
image:
repository: quay.io/kubernetes-ingress-controller/nginx-ingress-controller
tag: "0.9.0-beta.15"
pullPolicy: IfNotPresent

my_values2.yaml:

controller:
image:
repository: quay.io/kubernetes-ingress-controller/nginx-ingress-controller
tag: "0.15.0"
pullPolicy: IfNotPresent

(the difference is in the image version)

• Others:

What happened:
if I use 0.9.0-beta.15, it takes <5s for the change in nginx config map to be reflected in the behavior of nginx.

if I use anything above 0.9.0-beta.15 (tried with 0.10.2, 0.11.0, 0.13.0, 0.15.0), controller logs that it detected a change in the configmap and reloads the backend, but nginx.conf doesn't change. After 2-5 mins, controller decides to reload the backend again and nginx.conf has the desired values

What you expected to happen:
I expected the 0.15.0 to react to changes in config as quickly as 0.9.0-beta.15 did

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

the change I was trying to make was adjusting the whitelist-source-range

is it possible this is a misconfiguration on my side?

[aledbf]

@mwasilew2 please check the prefix of the annotations contains nginx

[aledbf]

Closing. Please check the --annotations-prefix flag https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/
Please reopen if the issue persists if the flag and the annotations are correct.

[mwasilew2]

I'm not sure what annotations you're reffering to. I'm guessing you're talking about annotations added to ingress k8s objects ( https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/ ) and the flag that tells controller "how to find them in the ingress objects" (the doc you linked above).

I'm using the configmap for providing a whitelist cause I want the rules to be applied to all ingresses (I want the whitelist to be global) https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/

I've looked at the configmap in the repo: https://github.com/kubernetes/ingress-nginx/blob/master/deploy/configmap.yaml

and compared it with the one in the helm chart: https://github.com/kubernetes/charts/blob/master/stable/nginx-ingress/templates/controller-configmap.yaml

and the one that's present in my cluster:

apiVersion: v1
data:
server-tokens: "false"
whitelist-source-range: <my_list>
kind: ConfigMap
metadata:
creationTimestamp: null
labels:
app: nginx-ingress
chart: nginx-ingress-0.19.0
component: controller
heritage: Tiller
release: ingress
name: ingress-nginx-ingress-controller
selfLink: /api/v1/namespaces/kube-system/configmaps/ingress-nginx-ingress-controller

and none of them have any annotations. The documentation also doesn't say anything about annotations in the configmap object.

it works as expected (the whitelist is picked up in the end), but it takes 5 min compared to 5s in the old version.

I'm happy to provide more debug info or write a patch. Could you point me in the right direction? There has to be something translating the configmap into nginx.conf, I see in the repo it used to be done with a template file. Is this still the case?

[mwasilew2]

just an FYI, I can't reopen the issue

[antoineco]

Could you please provide the controller logs, especially the part supporting this:

controller logs that it detected a change in the configmap and reloads the backend

(and surround code snippets with ```(your_code)``` to improve readability 🙂)

[antoineco]

Usual suspect: an Equal() function returning true when it should not.

[antoineco]

/assign @antoineco

[antoineco]

@mwasilew2 I can reproduce what you described.

Procedure (on the current master):

---

1. Edit the ConfigMap and add the mentioned setting
   a. kubectl edit cm nginx-configuration
   b. set whitelist-source-range: 127.0.0.1 and save
   c. the following log entries are printed

I0612 14:43:25.298234       6 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"nginx", Name:"nginx-configuration", UID:"75927380-6e41-11e8-86c0-025000000001", APIVersion:"v1", ResourceVersion:"82157", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap nginx/nginx-configuration
I0612 14:43:25.304823       6 controller.go:168] backend reload required
I0612 14:43:25.478106       6 controller.go:178] ingress backend successfully reloaded...

---

2. Display the NGINX config
   a. kubectl nginx-ingress-controller-7dbfc54c45-mt2kx -- cat nginx.conf
   b. whitelisting geo maps are not set

---

3. Edit some existing Ingress object
   a. kubectl edit ing storefront
   b. the following log entries are printed

I0612 14:44:57.293875       6 controller.go:168] backend reload required
I0612 14:44:57.295805       6 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"nginx", Name:"storefront", UID:"faea7ba5-6e4c-11e8-86c0-025000000001", APIVersion:"extensions", ResourceVersion:"82268", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress nginx/storefront
I0612 14:44:57.405939       6 controller.go:178] ingress backend successfully reloaded...

---

4. Display the NGINX config (again)
   a. kubectl nginx-ingress-controller-7dbfc54c45-mt2kx -- cat nginx.conf
   b. notice the expected geo map is now present

        # Deny for storefront.127.0.0.1.xip.io/
        geo $the_real_ip $deny_sVIHBncfBbenOkmIeagnHsOvuoHgQCmO {
                default 1;

                127.0.0.1 0;
        }

[antoineco]

/kind bug

[mwasilew2]

yup, that's it, I was getting the same behaviour and the exact same logs (I assume you no longer need mine)

[antoineco]

Note, that bug is amplified by #2634.