Ingress for multiple paths not working on Raspberry Pi 4, Microk8s 1.19

[sonttran]

Ubuntu 20.04.1
Raspberry Pi 4B 4Gb
Microk8s 1.19

Hi, I have a simple nginx ingress setup on Mircok8s 1.19 running on Raspberry Pi 4B. But the multiple paths won't work. However, if I delete all paths and keep the root one, it will work. My backend pods can receive requests on both configs. But for multiple paths config ingress, the full path won't be passed down to backend. I.e.

With this config:

      - path: /moon
        pathType: Prefix
        backend:
          service:
            name: moon
            port:
              number: 3000
      - path: /earth
        pathType: Prefix
        backend:
          service:
            name: earth
            port:
              number: 3000

Expected: /moon/and/anything/esle will hit moon backend. But in reality, looking at the pod log, I see / as the request path.
Expected: /earth/and/anything/esle will hit earth backend. But in reality, looking at the pod log, I see / as the request path.

However, if I remove all paths and only keep the root one /. Ingress will pass down request with path correctly. I.e.
With this config:

      - path: /
        pathType: Prefix
        backend:
          service:
            name: earth
            port:
              number: 3000

/moon/api/login will hit moon backend correctly. And logs in moon pod show request path as /moon/api/login.

Full non-working Ingress config with multiple paths:

### INGRESS
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: all-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/enable-cors: "true"
  labels:
    app: ingress
spec:
  defaultBackend:
    service:
      name: nginx
      port:
        number: 80
  rules:
  - http:
      paths:
      - path: /moon
        pathType: Prefix
        backend:
          service:
            name: moon
            port:
              number: 3000
      - path: /earth
        pathType: Prefix
        backend:
          service:
            name: earth
            port:
              number: 3000

Full working Ingress config with single path.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: all-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/enable-cors: "true"
  labels:
    app: ingress
spec:
  defaultBackend:
    service:
      name: nginx
      port:
        number: 80
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: earth
            port:
              number: 3000

Reference: docs on official Kubernetes Ingress config.
Inspect tarball attached.

Any help is very much appreciated. Thank you!
inspection-report-20200919_233343.tar.gz

[balchua]

Hi @sonttran thanks for using MicroK8s.
Quick question, did you deploy your own ingress or you used the addon?

Try removing the nginx rewrite annotation. nginx.ingress.kubernetes.io/rewrite-target

[sonttran]

Hi @balchua , I use the addon. I enabled it by running microk8s.kubectl enable ingress. I have tried to remove rewrite annotation. But ingress won't work at all without it.

[balchua]

Does your error look similar to this?
kubernetes/ingress-nginx#3762

[sonttran]

No, @balchua, I tried that too but it would not work for me.

[balchua]

@sonttran
Do you mind trying this?

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - host: test.com
    http:
      paths:
      - path: /foo/.*
        backend:
          serviceName: test
          servicePort: 80

By using the v1beta1 of ingress version?

[thiller2018]

I had the same issue. My server was on UTC, my client on CEST. I changed the timezone on the server and it worked.

[sonttran]

Thanks for the suggestion @balchua. Tried that and it does not work.
@thiller2018 : I'm right next to my server so there's no time difference here. I think this is a flaky bug.

[balchua]

@sonttran do you mind pasting the v1beta1 ingress? Thanks

[reddeppas]

hi did you try the example mentioned here https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/rewrites

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: cafe-ingress
annotations:
nginx.org/rewrites: "serviceName=tea-svc rewrite=/;serviceName=coffee-svc rewrite=/beans/"
spec:
rules:

• host: cafe.example.com
  http:
  paths:
  • path: /tea/
    backend:
    serviceName: tea-svc
    servicePort: 80
  • path: /coffee/
    backend:
    serviceName: coffee-svc
    servicePort: 80

[sonttran]

@balchua @reddeppas: thanks for the suggestions. This config now works for me. I need to comment out both nginx and rewrite annotations.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: all-ingress
  annotations:
    # kubernetes.io/ingress.class: nginx # strangely, rm this make multiple paths working
    # nginx.ingress.kubernetes.io/rewrite-target: / # strangely, rm this make multiple paths working
    nginx.ingress.kubernetes.io/enable-cors: "true"
  labels:
    app: ingress
spec:
  defaultBackend:
    service:
      name: nginx
      port:
        number: 80
  rules:
  - http:
      paths:
      - path: /moon
        pathType: Prefix
        backend:
          service:
            name: moon
            port:
              number: 3000
      - path: /earth
        pathType: Prefix
        backend:
          service:
            name: earth
            port:
              number: 3000
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

[balchua]

Hi @sonttran im going to close this issue now that you have resolved it. Thanks!