Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forkchoice external locks v2 #12036

Merged
merged 25 commits into from
Mar 2, 2023
Merged

Forkchoice external locks v2 #12036

merged 25 commits into from
Mar 2, 2023

Conversation

potuz
Copy link
Contributor

@potuz potuz commented Feb 23, 2023

A different take on #12014

We remove all locks from forkchoice and lock in chain_info.go so that RPC calls do not need to lock. Writers of blockchain package core functions need to access directly forkchoice instead of chain_info functions.

  • Check recursive write locks
  • Check for calls to forkchoice interface directly
  • Check for calls to chain_info
  • Check for recursive locks
  • Check for double locks with blockchain.headLock
  • Check for missing locks

@potuz potuz requested a review from a team as a code owner February 23, 2023 12:44
@potuz potuz requested review from kasey, saolyn and nisdas February 23, 2023 12:44
Comment on lines +49 to +50
s.cfg.ForkChoiceStore.Lock()
defer s.cfg.ForkChoiceStore.Unlock()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s.cfg.ForkChoiceStore.IsOptimistic is also called within onBlock

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same with finalized and justified epochs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not a problem, the problem is calls to chaininfo.IsOptimistic any access to forkchoice directly is not a problem

Copy link
Member

@terencechain terencechain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some missing locks from my quick audit:

Blockchain pkg:
func (s *Service) saveGenesisData(ctx context.Context, genesisState state.BeaconState) error
func (s *Service) StartFromSavedState(saved state.BeaconState)
func (s *Service) ReceiveAttesterSlashing(ctx context.Context, slashing *ethpb.AttesterSlashing) // Can be triggered via p2p

Debug rpc pkg
(ds *Server) GetForkChoice(ctx context.Context, _ *emptypb.Empty) (*ethpbv1.ForkChoiceDump, error

Proposer RPC pkg
(vs *Server) circuitBreakBuilder(s primitives.Slot) (bool, error)

Stategen pkg
New(beaconDB db.NoHeadAccessDatabase, fc forkchoice.ForkChoicer, opts ...StateGenOption) *State

@@ -122,7 +122,7 @@ func (s *Service) saveHead(ctx context.Context, newHeadRoot [32]byte, headBlock
reorgDistance.Observe(float64(dis))
reorgDepth.Observe(float64(dep))

isOptimistic, err := s.IsOptimistic(ctx)
isOptimistic, err := s.ForkChoicer().IsOptimistic(newHeadRoot)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function saveHead calls later the function notifyNewHeadEvent which acquires a lock for IsOptimistic. It would be nice to check the safety of this

Copy link
Member

@terencechain terencechain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add locks around ProposerBoost(). It's only used in spec tests. But spec tests run all the tests in parallel so we might see dead locks / flakeness there

terencechain
terencechain previously approved these changes Mar 1, 2023
@potuz potuz added Blocked Blocked by research or external factors Ready For Review A pull request ready for code review labels Mar 1, 2023
@potuz
Copy link
Contributor Author

potuz commented Mar 1, 2023

Blocking it for some more eyes cc. @nisdas

@@ -277,6 +285,8 @@ func (s *Service) CurrentFork() *ethpb.Fork {

// IsCanonical returns true if the input block root is part of the canonical chain.
func (s *Service) IsCanonical(ctx context.Context, blockRoot [32]byte) (bool, error) {
s.ForkChoicer().RLock()
defer s.ForkChoicer().RUnlock()
// If the block has not been finalized, check fork choice store to see if the block is canonical
if s.cfg.ForkChoiceStore.HasNode(blockRoot) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really related to this PR, but can we stick to either just using Forkchoicer or ForkChoiceStore ? Using both gives the impression they are different things.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In part I am guilty of this since I never liked the overhead of an extra call to ForkChoicer(), it would be nice to clean this up and actually not export the field and only use the getter.

@@ -52,6 +52,7 @@ type head struct {

// This saves head info to the local service cache, it also saves the
// new head root to the DB.
// Requires a lock on forkchoice.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Requires a lock on forkchoice.
// Caller of the method MUST acquire a lock on forkchoice.

reportSlotMetrics(blockCopy.Block().Slot(), s.HeadSlot(), s.CurrentSlot(), finalized)
}

if err := s.cfg.BeaconDB.SaveBlocks(ctx, s.getInitSyncBlocks()); err != nil {
return err
}
finalized := s.FinalizedCheckpt()
finalized := s.ForkChoicer().FinalizedCheckpoint()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not do the copy here ?

finalized := &ethpb.Checkpoint{Epoch: cp.Epoch, Root: bytesutil.SafeCopyBytes(cp.Root[:])}

func (s *Service) checkSaveHotStateDB(ctx context.Context) error {
currentEpoch := slots.ToEpoch(s.CurrentSlot())
// Prevent `sinceFinality` going underflow.
var sinceFinality primitives.Epoch
finalized := s.FinalizedCheckpt()
finalized := s.ForkChoicer().FinalizedCheckpoint()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same thing here:

finalized := &ethpb.Checkpoint{Epoch: cp.Epoch, Root: bytesutil.SafeCopyBytes(cp.Root[:])}

@potuz potuz merged commit 0a87210 into develop Mar 2, 2023
@delete-merged-branch delete-merged-branch bot deleted the forkchoice-external-locks-v2 branch March 2, 2023 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Blocked Blocked by research or external factors Ready For Review A pull request ready for code review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants