Fix: Properly re-raise non-grpc exceptions during refreshing of proxy-auth credentials in auth interceptor

[fg91]

Why are the changes needed?

Flytekit's grpc client for flyteadmin has a grpc channel interceptor for lazy authentication which does the following:

class AuthUnaryInterceptor(grpc.UnaryUnaryClientInterceptor, grpc.UnaryStreamClientInterceptor):
    ...
    def intercept_unary_unary(...):
        """
        Intercepts unary calls and adds auth metadata if available. On Unauthenticated, resets the token and refreshes
        and then retries with the new token
        """
        ...
        fut: grpc.Future = continuation(updated_call_details, request)
        e = fut.exception()
        if e:
            if e.code() == grpc.StatusCode.UNAUTHENTICATED ...:
                self._authenticator.refresh_credentials()
                ...
                return continuation(updated_call_details, request)

1. Try request without refreshing credentials
2. If this fails (401), refresh credentials
3. Try again

---

Since flyteorg/flyte#3965 / #1787, the grpc client can be configured to additionally generate and send a 2nd token as proxy-authorization header for a proxy in front of flyteadmin.

In this case, two such AuthUnaryInterceptors are configured sequentially, the first lazily refreshing tokens for the proxy, the second lazily refreshing tokens for flyteadmin itself.

The process then looks as follows:

1. Try request without refreshing credentials for proxy
2. If the proxy returns 401, refresh credentials for proxy
3. Try again
4. If flyteadmin returns 401, refresh credentials for flyteadmin
5. Try again

The following can currently go wrong:

Refreshing credentials for the proxy (step 2) happens by configuring the proxyCommand arg in the config .flyte/config.yaml with a command which is invoked by the CommandAuthenticator and which is supposed to return a token.

Let's say invoking this command in step 2 fails.

In this case, in step 4 in the AuthUnaryInterceptor, the exception e

class AuthUnaryInterceptor(grpc.UnaryUnaryClientInterceptor, grpc.UnaryStreamClientInterceptor):
    ...
    def intercept_unary_unary(...):
        ...
        e = fut.exception()
        if e:
            if e.code() == grpc.StatusCode.UNAUTHENTICATED ...

is not a grpc exception that has a .code attribute but is simply a python exception raised by the CommandAuthenticator.

The users then sees this error:

Failed with Exception <class 'RuntimeError'> Reason:
[...], reason: SYSTEM:Unknown: error=None, cause='AuthenticationError' object has no attribute 'code'

The actual exception (which the user would need to see to understand why the refreshing of the token failed) is shadowed by this exception.

What changes were proposed in this pull request?

Before checking if e.code() is 401, we need to check if e has a .code attribute. If not, re-raise e.

With this fix, the user sees:

Failed with Exception <class 'RuntimeError'> Reason:
[...], reason: SYSTEM:Unknown: error=None, cause=Problems refreshing token with command: Command '['some', 'failing', 'command']' returned non-zero exit status 1.

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.