nginx-ingress-controller 0.9.0-beta.10 not updating configuration #945

stibi · 2017-07-10T21:51:42Z

Hi guys,
I have just deployed the 0.9.0-beta.10 nginx controller on my cluster, but I see the same problem as was described before here.

When I create a new ingress, I can see that the controller notice the event:

[kindled-rabbit-nginx-ingress-controller-f6mxf] I0710 21:29:49.433953       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hosting", Name:"hosting-eswqklnm-custom", UID:"caf35bfc-65b4-11e7-90
ed-06a0d248057a", APIVersion:"extensions", ResourceVersion:"16928428", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress hosting/hosting-eswqklnm-custom
[kindled-rabbit-nginx-ingress-controller-8x848] I0710 21:29:49.442867       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hosting", Name:"hosting-eswqklnm-custom", UID:"caf35bfc-65b4-11e7-90
ed-06a0d248057a", APIVersion:"extensions", ResourceVersion:"16928428", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress hosting/hosting-eswqklnm-custom
[kindled-rabbit-nginx-ingress-controller-13tdd] I0710 21:29:49.441041       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hosting", Name:"hosting-eswqklnm-custom", UID:"caf35bfc-65b4-11e7-90
ed-06a0d248057a", APIVersion:"extensions", ResourceVersion:"16928428", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress hosting/hosting-eswqklnm-custom

But no ingress backend successfully reloaded... message is logged and the new host block is not present in the nginx.conf.

I have to kill all the nginx-controller pods to get the ingress reflected in the nginx.conf.

I have the controller deployed as daemonset, if that is useful information.

I'm not sure if RBAC is involved in this somehow? I have not configured that for nginx controller yet, I have to take a look what it is about, quite a few new things here since I last visited :)

Thanks for help! 💯

The text was updated successfully, but these errors were encountered:

aledbf · 2017-07-10T21:53:38Z

@stibi do you see a CREATE event? (update only applies to existing Ingress rules)

Edit: updates not necessarily require a reload.

stibi · 2017-07-10T21:59:48Z

you are right that the log snippet was taken when updating an ingress.

I have deleted it (btw, ingress backend successfully reloaded... now)

[kindled-rabbit-nginx-ingress-controller-jm524] I0710 21:55:38.009918       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hosting", Name:"hosting-eswqklnm-custom", UID:"caf35bfc-65b4-11e7-90
ed-06a0d248057a", APIVersion:"extensions", ResourceVersion:"16931353", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress hosting/hosting-eswqklnm-custom                                                      
[kindled-rabbit-nginx-ingress-controller-88591] I0710 21:55:38.008559       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hosting", Name:"hosting-eswqklnm-custom", UID:"caf35bfc-65b4-11e7-90
ed-06a0d248057a", APIVersion:"extensions", ResourceVersion:"16931353", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress hosting/hosting-eswqklnm-custom                                                      
[kindled-rabbit-nginx-ingress-controller-9hvhj] I0710 21:55:38.011033       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hosting", Name:"hosting-eswqklnm-custom", UID:"caf35bfc-65b4-11e7-90
ed-06a0d248057a", APIVersion:"extensions", ResourceVersion:"16931353", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress hosting/hosting-eswqklnm-custom                                                      
[kindled-rabbit-nginx-ingress-controller-88591] I0710 21:55:38.034628       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"kube-lego-nginx", UID:"048b0b3e-fe6a-11e6-9814-0
60439e40a77", APIVersion:"extensions", ResourceVersion:"16931357", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress kube-system/kube-lego-nginx                                                              
[kindled-rabbit-nginx-ingress-controller-jm524] I0710 21:55:38.035637       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"kube-lego-nginx", UID:"048b0b3e-fe6a-11e6-9814-0
60439e40a77", APIVersion:"extensions", ResourceVersion:"16931357", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress kube-system/kube-lego-nginx                                                              
[kindled-rabbit-nginx-ingress-controller-9hvhj] I0710 21:55:38.037294       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"kube-lego-nginx", UID:"048b0b3e-fe6a-11e6-9814-0
60439e40a77", APIVersion:"extensions", ResourceVersion:"16931357", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress kube-system/kube-lego-nginx                                                              
[kindled-rabbit-nginx-ingress-controller-jm524] 10.1.58.1 - [10.1.58.1] - - [10/Jul/2017:21:55:39 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 200 16 "-" "Go-http-client/1.1" 135 0.002 [kube-system
-kube-lego-nginx-8080] 10.1.45.14:8080 16 0.002 200                                                                                                                                                                 
[kindled-rabbit-nginx-ingress-controller-9hvhj] I0710 21:55:39.958392       5 controller.go:427] backend reload required                                                                                            
[kindled-rabbit-nginx-ingress-controller-9hvhj] I0710 21:55:40.079660       5 controller.go:437] ingress backend successfully reloaded...                                                                           
[kindled-rabbit-nginx-ingress-controller-88591] I0710 21:55:40.291195       5 controller.go:427] backend reload required                                                                                            
[kindled-rabbit-nginx-ingress-controller-88591] I0710 21:55:40.403311       5 controller.go:437] ingress backend successfully reloaded...                                                                           
[kindled-rabbit-nginx-ingress-controller-jm524] I0710 21:55:40.919051       5 controller.go:427] backend reload required                                                                                            
[kindled-rabbit-nginx-ingress-controller-jm524] I0710 21:55:41.038105       5 controller.go:437] ingress backend successfully reloaded...

and then created it again:

kindled-rabbit-nginx-ingress-controller-jm524] I0710 21:56:04.584486       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hosting", Name:"hosting-eswqklnm-custom", UID:"91ff1d57-65ba-11e7-90ed-06a0d248057a", APIVersion:"extensions", ResourceVersion:"16931409", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress hosting/hosting-eswqklnm-custom                                                      
[kindled-rabbit-nginx-ingress-controller-88591] I0710 21:56:04.584850       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hosting", Name:"hosting-eswqklnm-custom", UID:"91ff1d57-65ba-11e7-90ed-06a0d248057a", APIVersion:"extensions", ResourceVersion:"16931409", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress hosting/hosting-eswqklnm-custom                                                      
[kindled-rabbit-nginx-ingress-controller-9hvhj] I0710 21:56:04.586851       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hosting", Name:"hosting-eswqklnm-custom", UID:"91ff1d57-65ba-11e7-90ed-06a0d248057a", APIVersion:"extensions", ResourceVersion:"16931409", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress hosting/hosting-eswqklnm-custom

But no reload for the CREATE event :(

Btw, reload is not triggered for UPDATE ?

aledbf · 2017-07-10T22:07:38Z

Btw, reload is not triggered for UPDATE ?

Only if you change the host, path or the endpoints (of the service) are updated

juliohm1978 · 2017-07-11T16:29:28Z

I'm not sure what could be causing it, but my first guess is to review pod permissions given to nginx-ingress-controller.

There are examples here: https://github.com/kubernetes/ingress/tree/master/examples/rbac/nginx. I'll admit it took me a while to get this sorted out to get it all working.

Maybe watch for log messages that indicate any lack of permissions?

stibi · 2017-07-11T16:32:00Z

Yes, RBAC is a thing I will look at today. Thanks for the hint!

aledbf · 2017-07-12T18:32:28Z

@stibi please update the image to quay.io/aledbf/nginx-ingress-controller:0.160

stibi · 2017-07-12T19:03:12Z

no change :( Still the same problem.

aledbf · 2017-07-12T19:30:43Z

@stibi please update to quay.io/aledbf/nginx-ingress-controller:0.161

stibi · 2017-07-12T19:56:07Z

no luck

aledbf · 2017-07-12T19:58:07Z

@stibi can you send the complete log? (please check my message in slack)

stibi · 2017-07-13T15:19:06Z

How likely is it a bug or a problem with configuration on my side?

I'll try to take a look on the code, to find out what is going on…

aledbf · 2017-07-13T15:21:55Z

@stibi I cannot reproduce this issue. Can you try to create an example that I can follow?

stibi · 2017-07-14T09:40:44Z

I'm trying to realiably reproduce this, right now I think the problem is when an ingress contains more than one host rule in spec.rules.

A new ingress with one rule triggers reload, a modification of an ingres with just one rule too…but once there are more rules, it doesn't do the reload.

I'm spinning up a clean local k8s cluster, to verify this in a different environment than my dev and prod clusters.

stibi · 2017-07-15T15:45:56Z

Could you try these steps to reproduce the problem?

create a new ingress with one host rule, for example:

spec:
  rules:
  - host: somedomain.cz
    http:
      paths:
      - backend:
          serviceName: some-service
          servicePort: 80
        path: /

nginx controller should reload correctly and the domain is working, accessible through the controller
add another host rule, with different domain, example:

spec:
  rules:
  - host: somedomain.cz
    http:
      paths:
      - backend:
          serviceName: some-service
          servicePort: 80
        path: /
  - host: anotherdomain.cz
    http:
      paths:
      - backend:
          serviceName: some-service
          servicePort: 80
        path: /

now here the reload is not happening on my side, you can add as much rules as you want, but no reload
the only way to trigger the sync & reload is to modify the first domain

I was digging in the code, I have some idea how it all works now, so I did this patch: stibi@299d4c3

It fixed the problem, the reload is triggered as expected…but I'm not really sure I have fixed the root cause. I'm trying to cover this with tests, because I still don't have the "aha" moment :)

aledbf · 2017-07-15T17:12:27Z

@stibi thank you for the report and finding the issue

aledbf · 2017-07-16T19:49:22Z

@stibi please check 0.9.0-beta.11

katopz · 2017-12-25T07:42:43Z

@aledbf Is this fixed in 0.9.0? I still got this issue for some reason.

Config map seem OK, it appear client-max-body-size "20m"

But after, apply, replace configmap, even delete pod
and I also try annotations nothing work for me, it's still client_max_body_size "1m" there

Do I miss something?

katopz · 2017-12-25T08:02:10Z

Also proxy-body-size didn't do the trick

katopz · 2017-12-25T10:50:20Z

For workaround, I've to edit deployment.yaml and redeploy to trigger configmap to be update. 🎄

aledbf · 2017-12-25T17:42:58Z

@katopz please check the documentation
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/configmap.md#proxy-body-size
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/annotations.md#custom-max-body-size

stibi mentioned this issue Jul 11, 2017

New TLS certificates are not reloaded by nginx-ingress-controller #947

Closed

stibi mentioned this issue Jul 15, 2017

check number of servers in configuration comparator #973

Merged

aledbf closed this as completed in #973 Jul 15, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nginx-ingress-controller 0.9.0-beta.10 not updating configuration #945

nginx-ingress-controller 0.9.0-beta.10 not updating configuration #945

stibi commented Jul 10, 2017

aledbf commented Jul 10, 2017 •

edited

Loading

stibi commented Jul 10, 2017 •

edited

Loading

aledbf commented Jul 10, 2017

juliohm1978 commented Jul 11, 2017 •

edited

Loading

stibi commented Jul 11, 2017 •

edited

Loading

aledbf commented Jul 12, 2017

stibi commented Jul 12, 2017

aledbf commented Jul 12, 2017

stibi commented Jul 12, 2017

aledbf commented Jul 12, 2017

stibi commented Jul 13, 2017

aledbf commented Jul 13, 2017

stibi commented Jul 14, 2017

stibi commented Jul 15, 2017

aledbf commented Jul 15, 2017

aledbf commented Jul 16, 2017

katopz commented Dec 25, 2017

katopz commented Dec 25, 2017

katopz commented Dec 25, 2017

aledbf commented Dec 25, 2017

nginx-ingress-controller 0.9.0-beta.10 not updating configuration #945

nginx-ingress-controller 0.9.0-beta.10 not updating configuration #945

Comments

stibi commented Jul 10, 2017

aledbf commented Jul 10, 2017 • edited Loading

stibi commented Jul 10, 2017 • edited Loading

aledbf commented Jul 10, 2017

juliohm1978 commented Jul 11, 2017 • edited Loading

stibi commented Jul 11, 2017 • edited Loading

aledbf commented Jul 12, 2017

stibi commented Jul 12, 2017

aledbf commented Jul 12, 2017

stibi commented Jul 12, 2017

aledbf commented Jul 12, 2017

stibi commented Jul 13, 2017

aledbf commented Jul 13, 2017

stibi commented Jul 14, 2017

stibi commented Jul 15, 2017

aledbf commented Jul 15, 2017

aledbf commented Jul 16, 2017

katopz commented Dec 25, 2017

katopz commented Dec 25, 2017

katopz commented Dec 25, 2017

aledbf commented Dec 25, 2017

aledbf commented Jul 10, 2017 •

edited

Loading

stibi commented Jul 10, 2017 •

edited

Loading

juliohm1978 commented Jul 11, 2017 •

edited

Loading

stibi commented Jul 11, 2017 •

edited

Loading