Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixing multinode state transition logic by register polling subscription in rpc client #14534

Merged

Conversation

huangzhen1997
Copy link
Contributor

@huangzhen1997 huangzhen1997 commented Sep 23, 2024

Description

SubscribeToFinalizedHeads and SubscribeToHeads won’t abort polling subscription after Close
It could cause clients like HeadTracker to keep using subscriptions to unhealthy RPC despite MultiNode's transitioning to an unhealthy state.
DoD: polling subscription must be registered

Also added a patch for subscribeNewHead when polling is enabled.

Tickets:

BCFR-946(https://smartcontract-it.atlassian.net/browse/BCFR-946

@huangzhen1997 huangzhen1997 marked this pull request as ready for review September 24, 2024 13:08
@huangzhen1997 huangzhen1997 requested review from a team as code owners September 24, 2024 13:08
@huangzhen1997 huangzhen1997 requested review from reductionista and removed request for a team September 24, 2024 13:08
@huangzhen1997 huangzhen1997 changed the title Register polling subscription in rpc client fixing multinode state transition logic by register polling subscription in rpc client Sep 24, 2024
"chainlink": patch
---

- register polling subscription logic in rpc client so when node unhealthy new susbcription will be used
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- register polling subscription logic in rpc client so when node unhealthy new susbcription will be used
- register polling subscription to avoid subscription leaking when rpc client gets closed.

require.NoError(t, rpc.Dial(ctx))

ch := make(chan *evmtypes.Head)
_, err := rpc.SubscribeNewHead(tests.Context(t), ch)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you check that the returned subscription is closed?

@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"math/big"
_ "net/http/pprof"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

debug artifact

rpc.DisconnectAll()

// ensure sub is closed
_, ok := <-errCh
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use

select {
<-errCh: // ok
default: fail
}

To return more explicit error instead of tests timeout

@@ -129,7 +129,8 @@ type rpcClient struct {
ws rawclient
http *rawclient

stateMu sync.RWMutex // protects state* fields
stateMu sync.RWMutex // protects state* fields
subsSliceMu sync.RWMutex // protects subscription slice
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a deadlock was caught by my new unit tests, and this fixes it.

Copy link
Collaborator

@dhaidashenko dhaidashenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@reductionista reductionista added this pull request to the merge queue Sep 30, 2024
Merged via the queue into develop with commit de268e9 Sep 30, 2024
128 checks passed
@reductionista reductionista deleted the BCFR-946/register-polling-subscription-in-rpc-client branch September 30, 2024 15:41
AnieeG pushed a commit that referenced this pull request Sep 30, 2024
…ion in rpc client (#14534)

* polling subscription to be registered

* adding changeset

* fix Subscribe new head

* add test

* update changeset

* fix lint

* fix deadlock and add unit test for http polling sub

* update changeset

* adding sub closed check and remove import

* add unit test coverage for http polling subscribeToHeads

* update test

* address comments part 1

* clean

* part 2

* fix lint

* fix lint
matYang pushed a commit to smartcontractkit/ccip that referenced this pull request Nov 14, 2024
Optional WS URL for RPCs to unblock new chain integration.
Charry-picks from:
smartcontractkit/chainlink#14354
smartcontractkit/chainlink#14373
smartcontractkit/chainlink#14534
smartcontractkit/chainlink#14364
smartcontractkit/chainlink#14929

---------

Co-authored-by: Joe Huang <joe.huang@smartcontract.com>
matYang added a commit to smartcontractkit/ccip that referenced this pull request Nov 18, 2024
Optional WS URL for RPCs to unblock new chain integration.
Charry-picks from:
smartcontractkit/chainlink#14534
smartcontractkit/chainlink#14364
smartcontractkit/chainlink#14929

---------

Co-authored-by: Joe Huang <joe.huang@smartcontract.com>
Co-authored-by: Adam Hamrick <adam.hamrick@smartcontract.com>
Co-authored-by: Chunkai Yang <matYang@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants