Unable to use Forward Proxy in SSO Auth for oauth token

[saithejareddy]

We are running buzzfeed-sso in a private cluster which don't have access to internet.
For Applications which will communicate to internet facing services we are using a forward proxy(Squid) hosted in same network.

Using below Environment variables to connect with Proxy:

HTTPS_PROXY=http://squid.local:1234
HTTP_PROXY=http://squid.local:1234
no_proxy=cluster.local (For internal communication)

We are getting below error from the sso-authenticator application.

{"error":"Post https://www.googleapis.com/oauth2/v4/token: dial tcp 142.250.183.42:443: i/o timeout","level":"error","msg":"error redeeming authentication code","remote_address":"<redacted>","service":"sso-authenticator","time":"2021-05-23 16:31:43.135"}
Is there any other way to achieve this?

[saithejareddy]

We were able to hit the same endpoint using cURL library from the same container with same environment variables.

[saithejareddy]

Any luck here?

[saithejareddy]

@Jusshersmith I tried with your commit. It fixed sso_auth issue, now we i am seeing issue with sso_proxy. Attaching logs here.

sso proxy is calling its self domain with /google/redeem (where google is oauth provider here)and then its failing.

Note: sso-proxy also using same image and it contains the proxy environment variables.

sso-proxy-7c489b86d4-2r9vq sso-proxy {"level":"info","msg":"starting OAuth flow","service":"sso-proxy","sign_in_url":{"Scheme":"https","Opaque":"","User":null,"Host":"sso-auth.xxx.yyy.zzz","Path":"/google/sign_in","RawPath":"","ForceQuery":false,"RawQuery":"client_id="<redacted>","Fragment":""},"time":"2021-06-02 18:12:52.982"} sso-proxy-7c489b86d4-2r9vq sso-proxy {"action":"proxy","http_status":302,"level":"info","msg":"","remote_address":"<redacted>","request_duration":1.0428119999999999,"request_method":"GET","request_uri":"app-1.sso.xxx.yyy.zzz/?","service":"sso-proxy","time":"2021-06-02 18:12:52.982","user":"","user_agent":"<redacted>"} sso-auth-ddbdb8ff7-tkrdx sso-auth {"level":"info","msg":"authentication: user passed validation","remote_address":"<redacted>","service":"sso-authenticator","time":"2021-06-02 18:12:53.139","user":"username@domain.com"} sso-auth-ddbdb8ff7-tkrdx sso-auth {"action":"sign_in","http_status":302,"level":"info","msg":"","proxy_host":"app-1.sso.xxx.yyy.zzz","remote_address":"<redacted>","request_duration":72.031412,"request_method":"GET","request_uri":"/google/sign_in?client_id=<redacted>","service":"sso-authenticator","time":"2021-06-02 18:12:53.140","user":"","user_agent":"<redacted>"} sso-proxy-7c489b86d4-2r9vq sso-proxy {"error":"Post https://sso-auth.xxx.yyy.zzz/google/redeem: dial tcp <APP_PUBLIC_IP>:443: i/o timeout","level":"error","msg":"error redeeming authorization code","remote_address":"<redacted>","service":"sso-proxy","time":"2021-06-02 18:12:55.302"} sso-proxy-7c489b86d4-2r9vq sso-proxy {"http_status":500,"level":"info","msg":"error page","page_message":"Internal Error","page_title":"Internal Error","remote_address":"<redacted>","service":"sso-proxy","time":"2021-06-02 18:12:55.302"} sso-proxy-7c489b86d4-2r9vq sso-proxy {"action":"callback","http_status":500,"level":"info","msg":"","remote_address":"<redacted>","request_duration":2000.4191389999999,"request_method":"GET","request_uri":"app-1.sso.xxx.yyy.zzz/oauth2/callback?<redacted>","service":"sso-proxy","time":"2021-06-02 18:12:55.302","user":"","user_agent":"<redacted>"}

[saithejareddy]

@Jusshersmith Any luck here?

[Jusshersmith]

Hey @saithejareddy,

Thanks for testing that change, and for the extra detail you've sent over.

I'm sorry, I didn't have as much time yesterday to look into your response as I would have liked. I'll try to find some time today, but to be realistic, it will likely end up being next week I'm afraid.

[saithejareddy]

Hey @Jusshersmith Thank you so much.

[saithejareddy]

@Jusshersmith Any luck here? :(

[Jusshersmith]

Hi @saithejareddy,

Might the Squid proxy provide any logging or metrics that could prove useful here? As far as I can tell, by the time you see the error redeeming authorization code error, sso_proxy has already sent successful requests to sso_auth, so it would appear they can both reach each other 🤔.

[saithejareddy]

Hi @Jusshersmith

If you look at the below error, It's clear that sso-proxy is not able to communicate with the sso-auth (It says TCP i/o timeout). This is possible if sso-proxy is not using HTTP/HTTPS_PROXY for this call because it don't have internet to communicate a public IP resolvable DNS. I had verified with squid logs as well. I am seeing requests coming from sso-auth only but there's no requests coming from sso-proxy here.

sso-proxy-7c489b86d4-2r9vq sso-proxy {"error":"Post https://sso-auth.xxx.yyy.zzz/google/redeem: dial tcp <APP_PUBLIC_IP>:443: i/o timeout","level":"error","msg":"error redeeming authorization code","remote_address":"<redacted>","service":"sso-proxy","time":"2021-06-02 18:12:55.302"}

[saithejareddy]

Hi @Jusshersmith

I had tested adding ProxyFromEnvironment here as mentioned in the above PR as well. It works for me..! Thanks for the help. 🤝

[saithejareddy]

Hi @Jusshersmith

I am seeing same above issue when upgraded to v3.0.0. Unable to figure out why this issue started coming up again. Can you please help here?