Get server crash signal for java server streaming RPC

[adiguzel]

Hi,

We are using Armeria for gRPC communication. We encountered an issue about detecting/catching on the client when a stream is abandoned by a server (due to shutdown). We have a few questions.

Background

We are writing a server-streaming RPC in java and client side uses blocking gRPC.

The client nodes start a streaming gRPC with a server. Server keeps the stream observer open and sends messages to relevant open stream observers when it has events. We don't want the streams to timeout or be closed (unless there are network issues) neither on the client side nor on the client side, unless server or client nodes needs to shutdown.

Problem

Currently we are trying to figure out how to get a signal on client side when the server crashes, so that dangling streams can be cleaned-up on client side.

In a realistic environment, we observed that the client doesn't always get an exception when the server is killed (we tested actually graceful shutdown). When server is shutdown, the client gets an exception only when the server has sent a message via the stream before. The exception received on that case is io.grpc.StatusRuntimeException: UNKNOWN , cause: com.linecorp.armeria.common.ClosedSessionException. We couldn't reproduce it in a test.

Question-1: Is this working as intended? Do we have to set a client or server option to cover all the cases?

Question-2: We asked the same question in grpc-io forum and got this answer but i am not sure if it is relevant "Is gRPC keepalive enabled? It should help detect and close dead connections.". Should we set child channel option in this use case serverBuilder.childChannelOption(ChannelOption.SO_KEEPALIVE, true)?

Question-2: Is blocking server side gRPC calls suitable for this use case? Should we use bi-directional streaming instead? Would it be more performant (not blocking threads on the client side). Would the above problem avoided in bi-directional streaming?

Question-3: Are there any client factory, stub or server options we should be aware of to avoid the streams getting closed too

I added more details about client and server side below.

Thank you,
Ugur

Details

Versions:

armeria: 1.14.1
grpc-api:1.45.1 (et. al)

Service definition looks like this:

service MyService {
  rpc GetStream(GetRequest) returns (stream GetResponse) {}
}

The client code looks like this:

public void get() {
   try {
        var iter = blockingStub.GetStream(request);
        while(iter.hasNext()) {
             GetResponse resp = iter.next();
        }
    } catch (StatusRuntimeException e) {
        logInfo("RPC failed: {0}", e.getStatus());
    }
}

How we create Armeria server and bind services:

ServerBuilder serverBuilder = Server.builder();
serverBuilder.maxNumConnections(5000);
serverBuilder.workerGroup(EventLoopGroups.newEventLoopGroup(Flags.numCommonWorkers(), "<name>", true), true);
serverBuilder.serverBuilder.idleTimeoutMillis(1_800_00); // 30 minites

for(HttpService httpService: <httpServices built from service definition>) {
 ServiceBindingBuilder serviceBuilder =
  ServiceBindingBuilder serviceBuilder = serverBuilder.route().pathPrefix(path).// some decorators etc.;
  serviceBuilder.blockingTaskExecutor(<some custom scheduled executor>, true).build(service);
}

// We don't set any child options. We also don't override any other potentially relevant server connection options such as 
// `pingIntervalMillis` (default value 0), `maxConnectionAgeMillis`  (default value 0, means infinite?) ,
// `connectionDrainDurationMicros` (default value: 1000000L).
serverBuilder.build

We don't set any child options. We also don't override any other potentially relevant server connection options such as pingIntervalMillis (default value 0), maxConnectionAgeMillis (default value 0, means infinite?) , connectionDrainDurationMicros (default value: 1000000L).

On the client side ClientFactory creation:

   ClientFactoryBuilder builder = ClientFactory.builder();
    builder.connectionPoolListener(ConnectionPoolListener.logging());
    builder.pingIntervalMillis(2_000)); // 2 seconds
    
    // Some TLS stuff
   
    // No other settings, defaults used
   return builder.build();

On the client side stub creation:

ClientBuilder builder =
          Clients.builder(Scheme.of(GrpcSerializationFormats.PROTO, protocol), endpointGroup, path)
              .factory(<clientFactoryAbove>) ;

// By doing this we avoided the stream to close too early on the client or server side.
// Is this the right approach?  
 builder.responseTimeout(<Duration of 356 DAYS>);
// some decorators

    
MyServiceBlockingStub stub =  builder.build(MyServiceBlockingStub.class);

// set logging interceptors

// No other settings, defaults used
return  stub;

[minwoox]

we observed that the client doesn't always get an exception when the server is killed (we tested actually graceful shutdown).

I believe that the client gets an exception, which is ClosedSessionException, if the server is killed because the connection is closed.
Would you mind elaborating on the situation that the client doesn't get an exception more specifically?

Question-1: Is this working as intended? Do we have to set a client or server option to cover all the cases?

Yes, ClosedSessionException is raised when the connection is closed by the server crash. To recover this situation, you might want to use RetryingClient with CSLB. (I assumed that you have multiple servers so that one of the servers is crashed you can keep continuing serving.)

Question-2: We asked the same question in grpc-io forum and got this answer but i am not sure if it is relevant "Is gRPC keepalive enabled? It should help detect and close dead connections.". Should we set child channel option in this use case serverBuilder.childChannelOption(ChannelOption.SO_KEEPALIVE, true)?

Let me answer this question after I receive the answer to my first question.

Is blocking server side gRPC calls suitable for this use case?

I don't think so.

Should we use bi-directional streaming instead? Would it be more performant (not blocking threads on the client side).

If you don't have to stream the request, you don't have to use bi-directional streaming. I believe that your case is just sending stream of data from the server to client when an event occurs so it's not the case.

Question-3: Are there any client factory, stub or server options we should be aware of to avoid the streams getting closed too

The stream is closed if the underlying connection is closed. I think you need RetryingClient with CSLB instead if the problem occurs by the server crash.

[adiguzel]

Thank you for the quick answer!

I believe that the client gets an exception, which is ClosedSessionException, if the server is killed because the connection is closed. Would you mind elaborating on the situation that the client doesn't get an exception more specifically?

In a local test case i wrote, it behaved like this as well. But in a realistic environment for the mentioned service, we observed client side doesn't always get StatusRuntimeException with ClosedSessionException. Situation looks like this:

1. [Stream 1] Node A starts a streaming request to Node S, stub returns an iterator: periodically checks the returned iterator
2. [Stream 2] Node A starts a streaming request to Node S, stub returns an iterator: periodically checks the returned iterator
3. [Stream 3] Node B starts a streaming request to Node S, stub returns an iterator: periodically checks the returned iterator
4. Server S sends a messages to the Stream 2.
5. Server S sends a messages to the Stream 3.
6. [Expected] Client side code listening iterator for stream 2 and 3 see StatusRuntimeException with ClosedSessionException
7. [Unexpected] Client side code listening iterator for stream 1 doesn't fail with an exception at all.

Q: Is blocking server side gRPC calls suitable for this use case?
A: I don't think so.

Do you recommend FutureStub or async stub then? Is the issue with blocking stub due to thread blocking (bad system resource utilization?)

If you don't have to stream the request, you don't have to use bi-directional streaming. I believe that your case is just sending stream of data from the server to client when an event occurs so it's not the case.

That is true. Though I was thinking that it may be more flexible in the future we may want client side to push an update to client as well. And that it may lead to better system resource utilisation. Let me know whether my assessment isn't correct.

Question-3: Are there any client factory, stub or server options we should be aware of to avoid the streams getting closed too

We plan to introduce retry logic on the client side on getting StatusRuntimeException. The issue was that we realize that we don't always/consistently get StatusRuntimeException for all cases (including server shutting down or crashing a) before sending any message to the open stream or b) after pushing any message to the open stream).
What does CSLB stand for?

Excuse my gap of knowledge here, how does Armeria handle temporary network issues. I know there is Channel concept.
Q: Does the channel close (and any open stream/request) in case the connection between client and server has issues?
Q: Does Armeria re-establish a new channel transparently (and doesn't fail a stream if it worked) or does it always throw exception?
Q: Does Armeria still keep sending keep-alive frames through the channels even through client/server doesn't write data to a stream? Does that mean as long as Armeria is able to send these frames, the stream won't close due to timeout?
Q: Is there any deep dive document / page where we can learn more details . We look at Armeria docs regularly but it is more high level.

[jrhee17]

When server is shutdown, the client gets an exception only when the server has sent a message via the stream before

By stream, I assume you mean a gRPC stream. (not HTTP/2 stream)

If a server shutdown is initiated, the expected behavior is:

1. Each HTTP/2 connection sends a GOAWAY frame
2. Once a graceful shutdown period has passed, the connections are shut down from the server-side.
3. Once a connection is shut down, the HTTP/2 streams for each connection will be closed (along with the corresponding requests)

Logs on whether a GOAWAY frame has been sent/received can be verified by enabling debug logs for the following class:
https://github.com/line/armeria/blob/main/core/src/main/java/com/linecorp/armeria/internal/common/Http2GoAwayHandler.java

[Unexpected] Client side code listening iterator for stream 1 doesn't fail with an exception at all.

Essentially, I think we need to know if the underlying connection isn't being closed, or just the request isn't being closed.
When checking the logs from ConnectionPoolListener.logging(), were you able to check if the connection for the request was actually closed?
Any reproducer would help a lot as well

Does the channel close (and any open stream/request) in case the connection between client and server has issues?

If a timeout is reached or either client/server initiates a channel close, the connection will be closed.
For instance, if the jvm crashes and the server isn't shut down cleanly I guess the underlying tcp connections may not have a chance to be closed. In this case, adding keepalive options (pingIntervalMillis) may help like you already did.

Q: Does Armeria re-establish a new channel transparently (and doesn't fail a stream if it worked) or does it always throw exception?

There isn't any automatic retry logic so an exception will be thrown. You would need to add a RetryingClient to retry on failure to establish a connection.

Q: Does Armeria still keep sending keep-alive frames through the channels even through client/server doesn't write data to a stream? Does that mean as long as Armeria is able to send these frames, the stream won't close due to timeout?

Some timeouts like idleTimeout depend on whether there was a read/write on the connection.
For other timeouts like requestTimeout, a hard limit is applied regardless of whether keep-alive frames are being sent.
Armeria doesn't send keep-alive frames by default, but by setting pingIntervalMillis like you did ping frames will be sent periodically.

Q: Is there any deep dive document / page where we can learn more details . We look at Armeria docs regularly but it is more high level.

Unfortunately, the docs at Armeria docs are all we have for now 😅

[minwoox]

adding keepalive options (pingIntervalMillis) may help like you already did.

Because of that, the connection will always be closed even if the server is unexpectedly closed.
We have to find out why the stream 1 doesn't get any exception.
Could you follow what @jrhee17 said and see if there's something that doesn't work as expected?
For the ping, we can enable the logger and see if it works correctly:

armeria/core/src/main/java/com/linecorp/armeria/internal/common/AbstractKeepAliveHandler.java

Line 303 in 20af195

logger.debug("{} PING write successful", channel);

armeria/core/src/main/java/com/linecorp/armeria/internal/common/AbstractKeepAliveHandler.java

Lines 260 to 273 in 20af195

	private void closeChannelAndLog() {
	if (pingState == PingState.SHUTDOWN) {
	return;
	}
	logger.debug("{} Closing an idle channel", channel);
	pingState = PingState.SHUTDOWN;
	channel.close().addListener(future -> {
	if (future.isSuccess()) {
	logger.debug("{} Closed an idle channel", channel);
	} else {
	logger.debug("{} Failed to close an idle channel", channel, future.cause());
	}
	});
	}

[minwoox]

@adiguzel What is the server crash you meant by the way?

[minwoox]

Also, could you enable Blockhound to see if there's any blocked event loop?
https://github.com/reactor/BlockHound

[jrhee17]

Do you recommend FutureStub or async stub then? Is the issue with blocking stub due to thread blocking (bad system resource utilization?)

I'm not really familiar with blocking stubs from server-side 😅 Blocking stubs from client-side should not matter though.

What does CSLB stand for?

Client Side Load Balancing.
I guess what @minwoox wanted to say was you can use a HealthCheckedEndpointGroup for retries in case a server is unhealthy.
If however you have your own health check mechanism built into your proxy, just retrying may be enough