callerIpAddress only ever shows the pod IP of the self hosted gateway

[ericlanderson]

Report

We are trying to capture the client IP of the requester in the self hosted gateway logs and expected the log field callerIpAddress to contain the client IP. We have tested this both with a LoadBalancer (MetalLB) and NodePort service. Both services were configured with externalTrafficPolicy: Local. In our case, the IP is always the IP of the self hosted gateway Pod. The only other case where the callerIpAddress was different was when we kubectl port-forward... to the service. The callerIpAddress when testing via port-forward is 127.0.0.1.

Expected Behavior

I was expecting the callerIpAddress to be the actual IP of the client making the request (especially when externalTrafficPolicy: Local).

Actual Behavior

callerIpAddress log field only ever shows the pod IP of the self hosted gateway.

Steps to Reproduce the Problem

1. Install self-hosted gateway using Helm.
2. Edit the self-hosted gateway service and set externalTrafficPolicy: Local.
3. curl service via NodePort or LoadBalancer IP.

Logs from self-hosted gateway

{"isRequestSuccess":false,"totalTime":9,"category":"GatewayLogs","callerIpAddress":"10.42.27.182","timeGenerated":"2024-01-16T17:53:35.9919859Z","region":"<private-region>","correlationId":"78afa01a-2ebb-48e5-b24b-fdcf0a70b6e2","method":"POST","url":"https://<private-api>/","backendResponseCode":401,"responseCode":401,"responseSize":364,"cache":"none","backendTime":7,"apiId":"dune-buggy","operationId":"health","clientProtocol":"HTTP/2","backendProtocol":"HTTP/1.1","apiRevision":"1","clientTlsVersion":"1.2","backendMethod":"POST","backendUrl":"http://<private-api>/","backendRequestHeaders":{"X-Forwarded-For":"10.42.27.182,10.42.27.182"}}

Self-hosted Gateway Version

2.4.0

Deployment

Self-Managed (YAML/Helm)

Platform

Other

Kubernetes Version

1.24

Anything else?

On prem Kubernetes.

[tomkerkhove]

Hey, would you mind trying with SHGW v2.5.0 please?

[ericlanderson]

I updated to v2.5.0 but that did not change the behavior. I did some additional testing and realized this is issue we have encountered is due to our use of Linkerd service mesh. If I run the SHGW without the service mesh then callerIpAddress contains the expected IP.

[ericlanderson]

We can close this issue and I will follow up with the Linkerd folks. If I find a solution, I will update this issue with a solution.

[tomkerkhove]

No worries and thanks for letting me know!

[ericlanderson]

Turns out this was simpler than expected. When using Linkerd with the self hosted gateway, just add the following annotation for your deployment or namespace config.linkerd.io/skip-inbound-ports: 8080,8081. This configures Linkerd to skip processing on inbound connections but still allows the gateway to participate with meshed traffic downstream.

[tomkerkhove]

Thanks for sharing!