Getting 302 when talking to InferenceService

[kimwnasptd]

When trying out the manifests from KFServing 0.6 and Knative 0.22 #1957 #1965 I could not manage to successfully send a request to a deployed InferenceService. My cluster was deployed using the example manifests which use Dex and AuthService.

Here's the output of a request to an InferenceService from a Pod inside the cluster:

curl -v -d @./data.json \
    -H "Content-type: application/json" \
    -H "Host: serving-0-l5tzp.kubeflow-user.svc.cluster.local" \
    http://knative-local-gateway.knative-serving/v1/models/serving-0-l5tzp:predict

* Connected to knative-local-gateway.istio-system (10.100.145.211) port 80 (#0)
> POST /v1/models/serving-0-l5tzp:predict HTTP/1.1
> Host: serving-0-l5tzp.kubeflow-user.svc.cluster.local
> User-Agent: curl/7.58.0
> Accept: */*
> Content-type: application/json
> Content-Length: 210730
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 302 Found
< location: /dex/auth?client_id=authservice&redirect_uri=%2Fauthservice%2Foidc%2Fcallback&response_type=code&scope=openid+profile+email+groups&state=MTYyODg1NjU3N3xOd3dBTkU1TlEwSkhTelJEVWt0YVJraFlWbEUxTWxoV1MxcEJTVTB5VmpaT1JVOU9Sa3MwTlVORlFWTkxVVVpMVFVvek0xWkhVRkU9fB4L_P4o1oE7KW1zcfBOSHmp52w-OipJBAXz8KjfUHue
< set-cookie: oidc_state_csrf=MTYyODg1NjU3N3xOd3dBTkU1TlEwSkhTelJEVWt0YVJraFlWbEUxTWxoV1MxcEJTVTB5VmpaT1JVOU9Sa3MwTlVORlFWTkxVVVpMVFVvek0xWkhVRkU9fB4L_P4o1oE7KW1zcfBOSHmp52w-OipJBAXz8KjfUHue; Path=/; Expires=Fri, 02 Feb 2052 13:03:06 GMT; Max-Age=1200000000000
< date: Fri, 13 Aug 2021 12:09:37 GMT
< x-envoy-upstream-service-time: 4
< server: envoy
< content-length: 0

cc @pvaneck @yuzisun

[kimwnasptd]

After inspection I figured out that the reason for this is:

1. the knative-local-gateway/knative-serving Gateway doesn't have its own envoy Pod and binds/configures the istio-ingressgateway Gateway/Pod. The cluster-local-gateway/istio-system Gateway had it's own Deployment/Service, which was used from the previous version of Knative [0.17]
2. the EnvoyFilter for the authservice is forwarding all requests from the istio-ingressgateway to the Authservice, to ensure they are authenticated

Because of the above 2 all incluster traffic that is send to an InferenceService, which passes from the istio-ingressgateway, will be sent to the authservice as well. But this is not a desired behavior since the AuthService should be filtering requests that come from outside the cluster.

[kimwnasptd]

A workaround for this would be to edit the EnvoyFilter to only apply to the 8080 port of the istio-ingressgateway. This means that only the requests from that port would be subject to the AuthService for ensuring user authentication.

spec:
  configPatches:
  - applyTo: HTTP_FILTER
    ...
    match:
      context: GATEWAY
      listeners:
        portNumber: 8080
...

This should only affect users who are using the AuthService as an OIDC provider, with the provided EnvoyFilter from the manifests.

[Hmr-ramzi]

@kimwnasptd Thanks for opening this. I get instead a 404 error. I am unable to figure out where to start debugging this from istio or knative. All services (inferenceservice and knativeservice) are with status ready

[yhwang]

@kimwnasptd do you hit the RBAC: access denied after adding portNumber: 8080 to authn-filter EnvoyFilter?
The inferenceservice pod is running on user's namespace and it still needs proper AuthorizationPolicy to access the predictor API in my case.

[kimwnasptd]

Thanks for taking a look @yhwang!

@kimwnasptd do you hit the RBAC: access denied after adding portNumber: 8080 to authn-filter EnvoyFilter?

In my case the requests were coming from the same namespace. So the Profile's AuthorizationPolicy allowed the requests from my Pod [Notebook] to hit the InferenceService, since everything was living in the same namespace.

[kimwnasptd]

We also tackled this problem in a different way for MiniKF. We didn't like the approach of having Knative's incluster-gateway to use the Ingress Gateway's Pod, since we wanted isolation between ingress and in-cluster traffic.

In the end we configured the Knative manifests to have the knative-local-gateway Service/Gateway objects to use Istio's cluster-local-gateway. I'll make a PR to add an overlay in the existing Knative manifests for this workaround.