Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

idle: decrement active call count for streaming RPCs only when the call completes #6610

Merged
merged 6 commits into from
Sep 11, 2023

Conversation

easwars
Copy link
Contributor

@easwars easwars commented Sep 7, 2023

RELEASE NOTES:

  • idle: fix a bug that was decrementing active RPC count too early for streaming RPCs; leading to channel moving to IDLE even though it had open streams

@easwars easwars requested a review from dfawley September 7, 2023 23:22
@easwars easwars added this to the 1.59 Release milestone Sep 7, 2023
@easwars easwars changed the title idle: decrement active RPC count for streaming RPCs only when the call completes idle: decrement active call count for streaming RPCs only when the call completes Sep 7, 2023
@arvindbr8 arvindbr8 assigned easwars and unassigned dfawley Sep 8, 2023
@easwars easwars assigned dfawley and unassigned easwars Sep 8, 2023
@dfawley dfawley assigned easwars and unassigned dfawley Sep 8, 2023
@easwars easwars assigned dfawley and unassigned easwars Sep 8, 2023
// Tests the case where channel idleness is enabled by passing a small value for
// idle_timeout. Verifies that a READY channel with an ongoing streaming RPC
// stays READY.
func (s) TestChannelIdleness_Enabled_OngoingCall_Streaming(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be merged with the other test? The difference is:

  1. The server needs to implement something for the streaming handler. It can be done the same way as unary did, though: block on a channel and return nil.
  2. Instead of just calling the RPC method, the client needs to create the stream and then call Recv(). This could be a little func() in a testCases struct.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrote this to be a single test with two subtests for unary and streaming cases.

the client needs to create the stream and then call Recv()

Why is calling Recv() required?

internal/idle/idle_e2e_test.go Outdated Show resolved Hide resolved
stream.go Outdated Show resolved Hide resolved
@dfawley dfawley assigned easwars and unassigned dfawley Sep 8, 2023
internal/idle/idle_e2e_test.go Show resolved Hide resolved
internal/idle/idle_e2e_test.go Outdated Show resolved Hide resolved
Comment on lines 261 to 269
// Verify that there are no idleness related channelz events.
if err := channelzTraceEventNotFound(ctx, "entering idle mode"); err != nil {
errCh <- err
return
}
if err := channelzTraceEventNotFound(ctx, "exiting idle mode"); err != nil {
errCh <- err
return
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel super great about these checks, by the way, because if we decided to change the string we log, then the test case is no longer valid but continues to pass. And someone reworking some log strings isn't necessarily going to assume there are test cases relying upon them.

Unless there's corresponding tests that use the same string (ideally it would be a const then) that confirm it went into idle mode by looking for the trace events?

Not something necessarily for this PR but something to think about.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. I added a TODO. Will do some thinking to see how we can improve this.

Comment on lines +256 to +259
if cc.WaitForStateChange(sCtx, connectivity.Ready) {
errCh <- fmt.Errorf("state changed from %q to %q when no state change was expected", connectivity.Ready, cc.GetState())
return
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't you simplify this test a good amount, remove the errCh, and go back to the old testutils.AwaitNoStateChange, by performing the RPC in a goroutine instead of doing this block in a goroutine?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we move the RPC to a goroutine and perform the rest of the checks here, there seems to be a race between how the RPC terminates:

  • the blockCh getting closes, and thereby a normal termination
  • the context getting cancelled

So, I was having to do way too many checks in different places. The way it currently exists seems easier for error handling.

@easwars easwars assigned dfawley and unassigned easwars Sep 8, 2023
@@ -259,15 +264,14 @@ func (s) TestChannelIdleness_Enabled_OngoingCall(t *testing.T) {
}

// Verify that there are no idleness related channelz events.
//
// TODO: Improve the checks here. If these log strings are
// changed in the code , these checks will continue to pass.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the space before the comma please. (Never know what other linters will think later unfortunately.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@easwars easwars merged commit 254bccb into grpc:master Sep 11, 2023
9 of 10 checks passed
@easwars easwars deleted the idleness_with_open_stream branch September 11, 2023 15:39
easwars added a commit to easwars/grpc-go that referenced this pull request Sep 13, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants