Persistent Subscriptions - SubscribeAsync and SubscribeToStreamAsync calls hang after reconnecting multiple times

[rmfaheem]

Describe the bug
When reconnecting a subscriber the server/client, the handshake stops prematurely.

To Reproduce

1. Start the server.
2. Start the client with only one persistent subscriber. There should be logic to reconnect the subscriber (dispose and try to connect a new instance of the same subscriber) if the connection is dropped.
3. Killed the server a couple of times in which the subscriber reconnects.
4. In the last reconnection attempt, the client seemed to believe it's connected (no exception was thrown). However, the subscription from the server's point of view does not have the client connected to it and therefore no messages are flowing to the client.

This happens only if a cancellation token is not passed to the EventStorePersistentSubscriptionsClient.SubscribeAsync call.

Expected behavior
Regardless of the optional cancellation token, I'd expect the call to EventStorePersistentSubscriptionsClient.SubscribeAsync to yield either a value or to throw.

Actual behavior
Connection hangs with unfinished handshake.

Config/Logs/Screenshots

static async Task ConnectPersistentSubscriber(Action<PersistentSubscription, SubscriptionDroppedReason, Exception> onSubscriptionDropped) 
{
   
    var eventStoreClientSettings = EventStoreClientSettings.Create("esdb://admin:changeit@127.0.0.1:2113?tls=true&tlsVerifyCert=false");
    eventStoreClientSettings.LoggerFactory = LoggerFactory.Create(builder => {
            builder.AddSerilog();
        });
    var eventStoreClient = new EventStorePersistentSubscriptionsClient(eventStoreClientSettings);
    try
    {
        await eventStoreClient.SubscribeAsync(
            "$id-id12334",
            "placeholder",
            delegate { return Task.CompletedTask; },
            (s, r, ex) =>
            {
                eventStoreClient.Dispose();
                onSubscriptionDropped(s, r, ex);
            }

            // Have to pass a cancellation token otherwise the call to "SubscribeAsync" will eventually hang with an incomplete handshake.
            ,cancellationToken: new CancellationTokenSource(2000).Token
            
            );
    }
    catch
    {
        eventStoreClient.Dispose();
        throw;
    }
}

Normal Connection:

[15:04:40 DBG] Starting gRPC call. Method type: 'DuplexStreaming', URI: 'https://127.0.0.1:2113/event_store.client.persistent_subscriptions.PersistentSubscriptions/Read'.
[15:04:40 DBG] Sending message.
[15:04:40 VRB] Serialized 'EventStore.Client.PersistentSubscriptions.ReadReq' to 62 byte message.
[15:04:40 VRB] Message sent.
[15:04:40 VRB] Response headers received.
[15:04:40 DBG] Reading message.
[15:04:40 VRB] Deserializing 54 byte message to 'EventStore.Client.PersistentSubscriptions.ReadResp'.
[15:04:40 VRB] Received message.
SubscriptionId: $id-12334::placeholder
[15:04:40 DBG] Reading message.
[15:04:40 VRB] Deserializing 911 byte message to 'EventStore.Client.PersistentSubscriptions.ReadResp'.
[15:04:40 VRB] Received message.
[15:04:41 INF] Inside EchoService.Handle
[15:04:41 DBG] Append to stream - message-5183f522-95cf-4708-a9fe-9ef91009a17f@NoStream.
[15:04:41 DBG] Starting gRPC call. Method type: 'ClientStreaming', URI: 'https://127.0.0.1:2113/event_store.client.streams.Streams/Append'.
[15:04:41 VRB] Starting deadline timeout. Duration: 00:00:04.9956674.
[15:04:41 DBG] Sending message.
[15:04:41 VRB] Serialized 'EventStore.Client.Streams.AppendReq' to 52 byte message.
[15:04:41 VRB] Message sent.
[15:04:41 VRB] Appending event to stream - { "streamName": "abc" }@9b8b5d7c-a214-5277-8bbe-608220f0dab1 placeholder.
[15:04:41 DBG] Sending message.
[15:04:41 VRB] Serialized 'EventStore.Client.Streams.AppendReq' to 408 byte message.
[15:04:41 VRB] Message sent.
[15:04:41 DBG] Completing client stream.
[15:04:41 VRB] Response headers received.
[15:04:41 DBG] Reading message.
[15:04:41 VRB] Deserializing 16 byte message to 'EventStore.Client.Streams.AppendResp'.
[15:04:41 VRB] Received message.
[15:04:41 DBG] Finished gRPC call.

Bugged Connection:

[14:59:10 DBG] Starting gRPC call. Method type: 'DuplexStreaming', URI: 'https://127.0.0.1:2113/event_store.client.persistent_subscriptions.PersistentSubscriptions/Read'.
[14:59:11 DBG] Sending message.
[14:59:11 VRB] Serialized 'EventStore.Client.PersistentSubscriptions.ReadReq' to 62 byte message.
[14:59:11 VRB] Message sent.

EventStore details

• EventStore server version: 21.6.0

• Operating system: Windows Server 2016

• EventStore client version: 20.10.0, 21.2.0

Originally from support ticket: 109806