Question: lua-resty-balancer and upstream-hash-by

[JordanP]

Hi guys,
So call me a fool but I'd like to give the enable-dynamic-configuration=true a shot in production here.

But some of my Ingresses run with ingress.kubernetes.io/upstream-hash-by: "$1" where the $1 is a capturing group of rewrite regex (injected with ingress.kubernetes.io/configuration-snippet)

Question is: would that work with lua-resty-balancer ? I could give it a try but release 0.15 does not have commit 7ac4e1d and I think I need that commit.

@ElvinEfendi what do you think ?

Thanks

[ElvinEfendi]

Hi @JordanP,

yes, it should work. You don't need that commit unless you're using session affinity annotation.

By the way, we at Shopify run ingress-nginx in dynamic mode for most of our services.

[JordanP]

Great that it should !

I am not using session affinity but running with 0.15 doesn't work. All my requests are always sent to the same upstream server. You sure this function https://github.com/kubernetes/ingress-nginx/blob/nginx-0.15.0/rootfs/etc/nginx/lua/util.lua#L7 would work if ngx_var is literally $1. With lua-resty-balancer, the rewrite rule will be evaluated before the selection of the upstream server right ?

Good to know that Shopify is running in dynamic mode, it gives confidence.

[ElvinEfendi]

I've not tried regex group capturing myself but according to https://github.com/openresty/lua-nginx-module#ngxvarvariable

Nginx regex group capturing variables $1, $2, $3, and etc, can be read by this interface as well, by writing ngx.var[1], ngx.var[2], ngx.var[3], and etc.

therefore I said it should work. If you can post here the exact configuration you use I can try to regenerate locally.

With lua-resty-balancer, the rewrite rule will be evaluated before the selection of the upstream server right ?

I'd assume so. It'd be easier to talk on a specific config you have though.

[JordanP]

So I manage to come up with a nice and easy reproducer :)

Take this file here: https://github.com/kubernetes/ingress-nginx/blob/master/images/echoheaders/echo-app.yaml (bump the replicas to 2 or 3) and add the following ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tototest
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/upstream-hash-by: "$1"
    ingress.kubernetes.io/configuration-snippet: |
      rewrite ^/toto/(.*) /$1 break;
spec:
  rules:
  - host: tototest.XXX.XX
    http:
      paths:
      - backend:
          serviceName: echoheaders
          servicePort: 8080

Now without --enable-dynamic-configuration if you curl --resolve tototest.XXX.XX:80:PUB_IP http://tototest.XXX.XX/toto/$i/foobar several times with different values for $i you'll see in the response (the pod name line) that several different pods replied.

If you do the same with --enable-dynamic-configuration only one pod gets all the request.

I know you may not have a lot of time to debug/support this, but if this could work for me I'd be a thorough tester and user of --enable-dynamic-configuration and that's something, right ? :p

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ElvinEfendi]

/remove-lifecycle stale

[ElvinEfendi]

@JordanP this issue has slipped from my radar. Thanks for the reproducer. Let me know if index.docker.io/shopify/nginx-ingress-controller:pull_2964 fixes the issue for you. Here's the patch if you wanna review: #2964

[JordanP]

@ElvinEfendi I've tested the fix, it works ! Thanks you 👍