Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(dot/state): replace sync.Map with map+mutex #2286

Merged
merged 4 commits into from
Mar 3, 2022

Conversation

qdm12
Copy link
Contributor

@qdm12 qdm12 commented Feb 9, 2022

Changes

Tests

go test ./dot/state

Issues

Primary Reviewer

dot/state/hashtoblock.go Outdated Show resolved Hide resolved
dot/state/hashtoblock.go Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Feb 11, 2022

Codecov Report

Merging #2286 (7136ee9) into development (a1d7269) will decrease coverage by 2.77%.
The diff coverage is 88.37%.

Impacted file tree graph

@@               Coverage Diff               @@
##           development    #2286      +/-   ##
===============================================
- Coverage        60.08%   57.30%   -2.78%     
===============================================
  Files              211      212       +1     
  Lines            27759    27768       +9     
===============================================
- Hits             16678    15912     -766     
- Misses            9357    10223     +866     
+ Partials          1724     1633      -91     
Flag Coverage Δ
unit-tests 57.30% <88.37%> (-2.78%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
dot/state/offline_pruner.go 0.00% <0.00%> (ø)
dot/state/block_finalisation.go 44.64% <75.00%> (+0.66%) ⬆️
dot/state/block.go 42.45% <83.33%> (-2.48%) ⬇️
dot/state/hashtoblockmap.go 100.00% <100.00%> (ø)
dot/state/initialize.go 42.34% <100.00%> (+1.60%) ⬆️
dot/state/service.go 51.78% <100.00%> (-0.22%) ⬇️
dot/state/storage.go 49.72% <100.00%> (-2.35%) ⬇️
dot/state/tries.go 100.00% <100.00%> (ø)
dot/utils.go 0.00% <0.00%> (-80.65%) ⬇️
dot/build_spec.go 0.00% <0.00%> (-77.89%) ⬇️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67a9bbb...7136ee9. Read the comment docs.

t.Parallel()

var startWg, endWg sync.WaitGroup
ctx, cancel := context.WithCancel(context.Background())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this context?

Copy link
Contributor Author

@qdm12 qdm12 Feb 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To cancel all runInLoop goroutines all at once.

		for ctx.Err() == nil {
			f()
		}

and the cancel is called by the timer.

To be fair, all this feels like over-engineering but also like the right way to test for data races quickly, since it has to be 'tight' and as event-driven as possible. Probably a topic I should share in go-guild 😉

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really see the need to use a context to handle cancellation. Could you not just close some channel to signify cancellation?

Copy link
Contributor Author

@qdm12 qdm12 Mar 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we can use a struct{} channel that gets closed, it's just that it then becomes:

stop := false
for {
	select {
	case <-stopGoroutines:
		stop = true
	default:
	}
	if stop {
		break
	}
	f()
}

Which is longer/more confusing I think. (and no I don't want to use goto 😄)

@qdm12 qdm12 merged commit 666795b into development Mar 3, 2022
@qdm12 qdm12 deleted the qdm12-dot-state-syncmap branch March 3, 2022 15:26
rrtti pushed a commit to polkadot-fellows/seeding that referenced this pull request Sep 29, 2022
I am Quentin Mc Gaw, a software engineer working the Go Polkadot host **Gossamer**.
I have been working full time on Gossamer since October 2021, mostly on the state trie and storage.
I have also made a [few minor pull requests](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12) to the Polkadot specification repository.

I am requesting to join the Fellowship at rank 1.

## Main contributions

### Gossamer

- Fix memory leaks
  - Trie encoding buffer pools usage fixed [#2009](ChainSafe/gossamer#2009)
  - Fix state map of tries memory leak [#2286](ChainSafe/gossamer#2286)
  - Fix sync benchmark [#2234](ChainSafe/gossamer#2234)
- Trie proof fixes ([#2604](ChainSafe/gossamer#2604), [#2661](ChainSafe/gossamer#2661))
- Fix end to end tests orchestration ([#2470](ChainSafe/gossamer#2470), [#2452](ChainSafe/gossamer#2452), [#2385](ChainSafe/gossamer#2385), [#2370](ChainSafe/gossamer#2370))
- State trie statistics ([#2378](ChainSafe/gossamer#2378), [#2310](ChainSafe/gossamer#2310), [#2272](ChainSafe/gossamer#2272))
- State trie fixes and improvements
  - Only deep copy nodes when mutation is certain [#2352](ChainSafe/gossamer#2352) and [#2223](ChainSafe/gossamer#2223)
  - Only deep copy necessary fields of a node [#2384](ChainSafe/gossamer#2384)
  - Use Merkle values for database keys instead of always hash [#2725](ChainSafe/gossamer#2725)
  - Opportunistic parallel Merkle value commputing [#2081](ChainSafe/gossamer#2081)
- Grandpa capped number of tracked messages ([#2490](ChainSafe/gossamer#2490), [#2485](ChainSafe/gossamer#2485))
- Add pprof HTTP service for profiling [#1991](ChainSafe/gossamer#1991)

Ongoing work:

- State trie lazy loading and caching
- State trie v1 support ([#2736](ChainSafe/gossamer#2736), [#2747](ChainSafe/gossamer#2747), [#2687](ChainSafe/gossamer#2687), [#2686](ChainSafe/gossamer#2686), [#2685](ChainSafe/gossamer#2685), [#2673](ChainSafe/gossamer#2673), [#2611](ChainSafe/gossamer#2611), [#2530](ChainSafe/gossamer#2530))

### Polkadot specification

➡️ [Pull requests from qdm12](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants