Bad credentials file causes Pub/Sub topic creation to hang

[meltsufin]

See: spring-attic/spring-cloud-gcp#962.
I suspect that the root cause is in the client library.

Note that when the same credentials file is used to publish a message to a topic, an appropriate exception is thrown saying java.io.IOException: Error getting access token for service account and 401 Unauthorized.

[jlheard]

This repeats over and over in the logs when running an Integration test after the key has been deleted:

15:28:04.944 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] HANDSHAKEN: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
15:28:04.944 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] OUTBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=32768}
15:28:04.944 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
15:28:04.946 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] INBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=16384}
15:28:04.946 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] OUTBOUND SETTINGS: ack=true
15:28:04.946 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
15:28:05.011 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] INBOUND SETTINGS: ack=true
15:32:04.945 [grpc-default-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler - [id: 0x464293d0, L:/172.21.104.17:50735 - R:spanner.googleapis.com/172.217.5.74:443] INBOUND GO_AWAY: lastStreamId=0 errorCode=0 length=17 bytes=73657373696f6e5f74696d65645f6f7574

[ajaaym]

@chingor13 after 3757 merged in grpc, all IOException is getting retried. google-auth-library-java throws IOException every where and almost all are not retriable. Does it make sense to change to something other than IOException?

[ktandon-informatica-com]

@ajaaym can you let me know when this bug #3573 or rather the original #5025 will be fixed? I wanted to know the rough timelines for the same for internal tracking.

[sduskis]

This isn't something we can fix in this repository. The grpc team is working on a retry framework, and their framework may deal with this issue.

[dzou]

Any new updates on this issue?

[AmaraSamar]

Do we have any update on the retry framework that the grpc team is working on ?

[kamalaboulhosn]

It looks like this does ultimately return now, though it ends up waiting for the deadline of the RPC to pass because it considers it a retryable error (UNAVAILABLE). Given that UNAVAILABLE errors are generally retryable, I don't think we want to turn all of those errors to non-retryable.

The default timeout is 10 minutes, though one can override this:

      TopicAdminSettings.Builder topicAdminClientBuilder = TopicAdminSettings.newBuilder();
      topicAdminClientBuilder
          .createTopicSettings()
          .setRetrySettings(
              topicAdminClientBuilder.createTopicSettings().getRetrySettings().toBuilder()
                  .setTotalTimeout(Duration.ofSeconds(30))
                  .build());
     TopicAdminClient topicAdminClient = TopicAdminClient.create(topicAdminClientBuilder.build());

If one calls createTopic on this topicAdminClient, it will return an error in 30 seconds if the credentials file is invalid.

[dzou]

@kamalaboulhosn - I see, thank you for your research on this.

I don't feel 100% confident about this though; it doesn't feel quite right to ask users to change the timeout setting just so it won't hang if they accidentally use an incorrect credentials file or delete it.

I'll dig deeper to ask GRPC if they throw a different error than UNAVAILABLE. I'm gonna reopen this and see if I can get to the bottom of this.

[dzou]

Earlier in the year I filed an issue to see if we can fix this problem: grpc/grpc-java#6808

From what it looks like I don't think the problem will be fixed due to certain technical reasons... I think we'll just have to live with the 10-minute timeout.

Returning this to closed.