HTTP(S) proxy not used when CA_BUNDLE is set

[gtoullier]

Is this a bug report or feature request?

• Bug Report

Describe the bug
AuthService cannot connect to OIDC provider through a proxy if a CA_BUNDLE is set.
Error: connection timed out.
HTTPS_PROXY and HTTP_PROXY env vars are set.

How to Reproduce
Steps to reproduce the behavior:

1. Deploy AuthService with the env var CA_BUNDLE set to a custom CA certificates file.
2. Container is starting and an error is raised in the log:

OIDC provider setup failed, retrying in 10 seconds: Get \"https://<OIDC PROVIDER>/.well-known/openid-configuration\": dial tcp XXX.XXX.XXX.XXX:443: connect: connection timed out

Expected behavior
The connection should be established

Config Files
Here the manifest:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: authservice
spec:
  template:
    spec:
      containers:
      - name: authservice
        image: gcr.io/arrikto/kubeflow/oidc-authservice:e236439
        env:
        - name: http_proxy
          value: http://<PROXY>:3128
        - name: https_proxy
          value: http://<PROXY>:3128
        - name: no_proxy
          value: XXX
        - name: HTTP_PROXY
          value: http://<PROXY>:3128
        - name: HTTPS_PROXY
          value: http://<PROXY>:3128
        - name: NO_PROXY
          value: XXX
        - name: CA_BUNDLE
          value: <PATH>/certificates.crt
        resources:
          requests:
            memory: 64Mi
            cpu: 100m
        volumeMounts:
          - mountPath: <PATH>
            name: custo-ca
      volumes:
        - name: custo-ca
          configMap:
            name: custo-ca

Logs

time="2022-02-17T10:15:47Z" level=info msg="Config: &{ProviderURL:https://<OIDC PROVIDER>/dex ClientID:xxx ClientSecret:xxx OIDCAuthURL:/dex/auth RedirectURL:/login/oidc OIDCScopes:[openid profile email groups[] StrictSessionValidation:false OIDCStateStorePath:/var/lib/authservice/data.db AuthserviceURLPrefix:/dex/ SkipAuthURLs:[/dex/] AuthHeader:Authorization Audiences:[istio-ingressgateway.istio-system.svc.cluster.local[] HomepageURL:/dex/site/homepage AfterLoginURL: AfterLogoutURL:/dex/site/after_logout UserIDHeader:kubeflow-userid GroupsHeader:kubeflow-groups UserIDPrefix: UserIDTransformer:{rules:[]} UserIDClaim:email UserIDTokenHeader: GroupsClaim:groups IDTokenHeader:Authorization Hostname: Port:8080 WebServerPort:8082 ReadinessProbePort:8081 CABundlePath:<PATH>/certificates.crt SessionStorePath:/var/lib/authservice/data.db SessionMaxAge:86400 SessionSameSite:Lax ClientName:AuthService ThemesURL:themes Theme:kubeflow TemplatePath:[web/templates/default] UserTemplateContext:map[] GroupsAllowlist:[*]}"
time="2022-02-17T10:15:47Z" level=info msg="Starting readiness probe at 8081"
time="2022-02-17T10:15:47Z" level=info msg="Starting server at :8080"
time="2022-02-17T10:15:47Z" level=info msg="Starting web server at :8082"
time="2022-02-17T10:17:54Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: Get \"https://<OIDC PROVIDER>/.well-known/openid-configuration\": dial tcp <OIDC PROVIDER IP>:443: connect: connection timed out"
time="2022-02-17T10:20:11Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: Get \"https://<OIDC PROVIDER>/.well-known/openid-configuration\": dial tcp <OIDC PROVIDER IP>:443: connect: connection timed out"

Environment:

• AuthService version: e236439
• Platform: custom
• Kubernetes version: 1.19

Additional context
I built a custom image from gcr.io/arrikto/kubeflow/oidc-authservice where I put the custom CA certificates into /usr/local/share/ca-certificates/
When I deploy it, i don't set the CA_BUNDLE.
In this case AuthService works well, the connection with OIDC provider is done through proxy as expected.
So it seems when the CA_BUNDLE is set, the HTTP client does not use the PROXY env vars.

On the AuthService container, in the netstat result, we see the connection is done directly without proxy.

~ $ netstat -apn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address              State       PID/Program name
tcp        0      1 192.168.128.47:57590    <OIDC PROVIDER IP>:443       SYN_SENT    1/oidc-authservice
tcp        0      0 :::8080                 :::*                         LISTEN      1/oidc-authservice
tcp        0      0 :::8081                 :::*                         LISTEN      1/oidc-authservice
tcp        0      0 :::8082                 :::*                         LISTEN      1/oidc-authservice
...
~ $

[vijay-nag]

Hi @gtoullier,

Is there an insecure option to ignore ssl verification?

[dbg-raghulkrishna]

same is there a insecure option to ignore ssl verification?