[Execution] Shadow Execution node implementation #5411

zhangchiqing · 2024-02-16T23:46:24Z

Adding observer mode to execution node so that a unstaked EN can sync blocks and collections from a staked public access node and execute blocks. It's useful for testing execution node changes, as it can run and execute on its own with new changes.

How to test:

# start the localnet with test execution node running 
cd integration/localnet
make stop; make bootstrap-test-en; make start;

# sending transaction to validate the execution
cd integration/benchmark/cmd/manual
go run ./main.go

# expect to see `block executed` log from the test execution node
docker logs -f localnet-test_execution_1-1

codecov-commenter · 2024-02-16T23:51:09Z

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 59.90%. Comparing base (1726b7b) to head (f265dc9).

Files	Patch %	Lines
engine/execution/provider/engine.go	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5411      +/-   ##
==========================================
+ Coverage   55.97%   59.90%   +3.93%     
==========================================
  Files        1022      543     -479     
  Lines       99705    51998   -47707     
==========================================
- Hits        55807    31150   -24657     
+ Misses      39598    18532   -21066     
+ Partials     4300     2316    -1984

Flag	Coverage Δ
unittests	`59.90% <0.00%> (+3.93%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

yhassanzadeh13 · 2024-02-22T19:10:52Z

network/noop.go

Please move this to network/underlay sub-package.

peterargue · 2024-02-22T22:22:15Z

cmd/execution_builder.go

+	error,
+) {
+	if !node.ObserverMode {
+		return &module.NoopReadyDoneAware{}, nil


can you add a log indicating whether or not this module was run

peterargue · 2024-02-22T22:28:24Z

cmd/execution_builder.go

+	// Execution Data cache that uses a blobstore as the backend (instead of a downloader)
+	// This ensures that it simply returns a not found error if the blob doesn't exist
+	// instead of attempting to download it from the network.


this comment isn't right. it's actually using a downloader (correctly).

OK, it was copied from here tho

https://github.com/onflow/flow-go/blob/leo/localnet-test-en/cmd/access/node_builder/access_node_builder.go#L559-L561

peterargue · 2024-02-22T22:31:00Z

cmd/execution_builder.go

+	exeNode.followerDistributor.AddOnBlockFinalizedConsumer(r.OnBlockFinalized)
+
+	execDataDistributor.AddOnExecutionDataReceivedConsumer(func(data *execution_data.BlockExecutionDataEntity) {
+		if len(data.BlockExecutionData.ChunkExecutionDatas) == 0 {


did you find that this case was encountered? There should always be at least one chunk because of the system chunk

peterargue · 2024-02-22T22:46:57Z

cmd/utils.go

+) (
+	p2p.IDTranslator,
+	func() module.IdentifierProvider,
+	error) {


Suggested change

error) {

error,

) {

janezpodhostnik · 2024-03-01T14:32:59Z

integration/testnet/util.go

@@ -97,13 +103,13 @@ func WriteFile(path string, data []byte) error {
 	return err
 }

-func WriteObserverPrivateKey(observerName, bootstrapDir string) error {
+func WriteObserverPrivateKey(observerName, bootstrapDir string) (crypto.PrivateKey, error) {


this new return is never used.

zhangchiqing force-pushed the leo/localnet-test-en branch 2 times, most recently from f42a417 to c4ac5bd Compare February 22, 2024 01:40

zhangchiqing marked this pull request as ready for review February 22, 2024 01:40

zhangchiqing requested review from peterargue, yhassanzadeh13 and ramtinms as code owners February 22, 2024 01:40

zhangchiqing requested review from janezpodhostnik, sideninja, kc1116 and yhassanzadeh13 and removed request for ramtinms and yhassanzadeh13 February 22, 2024 01:40

yhassanzadeh13 approved these changes Feb 22, 2024

View reviewed changes

network/noop.go Outdated

Copy link

Contributor

yhassanzadeh13 Feb 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move this to network/underlay sub-package.

peterargue approved these changes Feb 22, 2024

View reviewed changes

zhangchiqing force-pushed the leo/localnet-test-en branch 3 times, most recently from 5e08a8f to 023479f Compare February 28, 2024 20:47

zhangchiqing enabled auto-merge February 28, 2024 20:47

zhangchiqing disabled auto-merge February 28, 2024 20:47

zhangchiqing force-pushed the leo/localnet-test-en branch from 023479f to 29d62a4 Compare February 28, 2024 23:30

zhangchiqing added 9 commits February 28, 2024 15:33

test en

b658f2c

add observer mode

b592e80

implement observer model

5cf82ff

update sync engine id provider

3e0228a

update nodeid generation

4f41e90

refactor key loading

2655d90

fix key

a2774ba

use different network type

d56ae78

udpate collection requester

e06f70a

zhangchiqing added 17 commits February 28, 2024 15:34

restricting role for observer mode

d550950

clean up

5a3c902

add comments and clean up

b05437a

refactor collection requester

18b5c0f

refactor noop conduit

9df6cb7

process local collection

f4bb1f3

update address

cb4e828

fix get libp2p node

8f9d035

add comments

eacc666

make tidy

db88349

fix lint

541399f

fix lint

ac6acc9

fix tests

eb31fe0

revert change

d30ca3e

update comments

ba80525

move noop to underlay

9b15bdc

init libp2pnode

e8db78b

zhangchiqing force-pushed the leo/localnet-test-en branch from 29d62a4 to e8db78b Compare February 28, 2024 23:34

zhangchiqing added 4 commits February 28, 2024 18:19

revert observer build changes

1d469a3

reuse BootstrapIdentities

9d74f20

use 5001 instead of 5000 for observer node in localnet

7103608

add log

f265dc9

zhangchiqing enabled auto-merge March 1, 2024 00:48

janezpodhostnik approved these changes Mar 1, 2024

View reviewed changes

zhangchiqing added this pull request to the merge queue Mar 1, 2024

Merged via the queue into master with commit 057d89b Mar 1, 2024
51 checks passed

zhangchiqing deleted the leo/localnet-test-en branch March 1, 2024 15:50

zhangchiqing mentioned this pull request Mar 6, 2024

Backport v0.33 localnet test en #5518

Merged

zhangchiqing mentioned this pull request Apr 25, 2024

Investigate Slow catchup speed for Execution Test environment #5782

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Execution] Shadow Execution node implementation #5411

[Execution] Shadow Execution node implementation #5411

zhangchiqing commented Feb 16, 2024 •

edited

Loading

codecov-commenter commented Feb 16, 2024 •

edited

Loading

yhassanzadeh13 Feb 22, 2024

peterargue Feb 22, 2024

peterargue Feb 22, 2024

zhangchiqing Feb 23, 2024

peterargue Feb 22, 2024

peterargue Feb 22, 2024

janezpodhostnik Mar 1, 2024

[Execution] Shadow Execution node implementation #5411

[Execution] Shadow Execution node implementation #5411

Conversation

zhangchiqing commented Feb 16, 2024 • edited Loading

codecov-commenter commented Feb 16, 2024 • edited Loading

Codecov Report

yhassanzadeh13 Feb 22, 2024

Choose a reason for hiding this comment

peterargue Feb 22, 2024

Choose a reason for hiding this comment

peterargue Feb 22, 2024

Choose a reason for hiding this comment

zhangchiqing Feb 23, 2024

Choose a reason for hiding this comment

peterargue Feb 22, 2024

Choose a reason for hiding this comment

peterargue Feb 22, 2024

Choose a reason for hiding this comment

janezpodhostnik Mar 1, 2024

Choose a reason for hiding this comment

zhangchiqing commented Feb 16, 2024 •

edited

Loading

codecov-commenter commented Feb 16, 2024 •

edited

Loading