[Access] Add support for indexing execution data on Observers #5256

AndriiDiachuk · 2024-01-19T15:00:18Z

This PR includes implementations for the following issue:

Add execution state checkpoint and indexing support for the Observer node (#4848).

This branch was merged with #5253 as it needs some parts for the issue above.

… execution state

codecov-commenter · 2024-01-22T21:59:16Z

Codecov Report

Attention: Patch coverage is 36.73469% with 93 lines in your changes are missing coverage. Please review.

Project coverage is 55.79%. Comparing base (1656e98) to head (615bd99).

Files	Patch %	Lines
...chronization/indexer/collection_executed_metric.go	14.63%	68 Missing and 2 partials ⚠️
...dule/state_synchronization/indexer/indexer_core.go	64.28%	12 Missing and 3 partials ⚠️
engine/access/ingestion/engine.go	75.00%	5 Missing ⚠️
module/metrics/noop.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5256      +/-   ##
==========================================
- Coverage   55.84%   55.79%   -0.06%     
==========================================
  Files        1030     1031       +1     
  Lines      100843   100860      +17     
==========================================
- Hits        56320    56274      -46     
- Misses      40186    40255      +69     
+ Partials     4337     4331       -6

Flag	Coverage Δ
unittests	`55.79% <36.73%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…o AndriiDiachuk/access-add-support-for-indexing-execution-data

…://github.com/UlyanaAndrukhiv/flow-go into AndriiDiachuk/access-add-support-for-indexing-execution-data

peterargue

I didn't review all of the changes since I know some are duplicated from #5253.

Added a couple comments about where to place the collection handler, but this looks pretty close

cmd/observer/node_builder/observer_builder.go

…://github.com/UlyanaAndrukhiv/flow-go into AndriiDiachuk/access-add-support-for-indexing-execution-data

…es now

cmd/observer/node_builder/observer_builder.go

engine/access/ingestion/engine.go

module/state_synchronization/indexer/indexer_core.go

engine/access/ingestion/engine.go

cmd/observer/node_builder/observer_builder.go

…-data' of https://github.com/AndriiDiachuk/flow-go into AndriiDiachuk/access-add-support-for-indexing-execution-data

…ing-execution-data

AndriiDiachuk · 2024-03-04T14:45:17Z

@peterargue Ready for a new round of review.

…ing-execution-data

peterargue

added a couple small comments, but otherwise this looks ready to go

module/state_synchronization/indexer/collection_executed_metric.go

… constructor

peterargue

looks good. can you add indexing to the default localnet config for observers here:
https://github.com/onflow/flow-go/blob/master/integration/localnet/builder/bootstrap.go#L473

"--execution-data-indexing-enabled=true",
"--execution-state-dir=/data/execution-state",

AndriiDiachuk · 2024-03-09T09:21:33Z

looks good. can you add indexing to the default localnet config for observers here: https://github.com/onflow/flow-go/blob/master/integration/localnet/builder/bootstrap.go#L473
"--execution-data-indexing-enabled=true",
"--execution-state-dir=/data/execution-state",

@peterargue It was added couple if lines above. But I moved all together now.

…ing-execution-data

peterargue · 2024-03-09T13:14:22Z

@peterargue It was added couple if lines above. But I moved all together now.

Thanks. Not sure how I missed that

…support-for-indexing-execution-data [Access] Add support for indexing execution data on Observers

AndriiDiachuk added 5 commits January 9, 2024 16:58

Added BuildExecutionSyncComponents method for support for loading the…

525c22d

… execution state

Linted and sorted imports

f83e29d

Added related cmd flag for data indexer

984c16c

Added flag for new method in bootstrap

4531721

Merged

342cb6b

AndriiDiachuk marked this pull request as ready for review January 22, 2024 12:47

AndriiDiachuk requested review from yhassanzadeh13 and peterargue as code owners January 22, 2024 12:47

AndriiDiachuk added 7 commits January 23, 2024 10:31

Merge branch 'master' of https://github.com/AndriiDiachuk/flow-go int…

cfead2d

…o AndriiDiachuk/access-add-support-for-indexing-execution-data

Added missing implementation parts for observer

8c28ce2

Merge branch 'UlyanaAndrukhiv/2795-add-execution-state-sync' of https…

73b7bf9

…://github.com/UlyanaAndrukhiv/flow-go into AndriiDiachuk/access-add-support-for-indexing-execution-data

Linted

cbe9f32

Added missing implementation

9d72de7

Added handler for collections in observer builder

c455281

Removed part that left from ingestion engine

23a028d

peterargue reviewed Jan 30, 2024

View reviewed changes

cmd/observer/node_builder/observer_builder.go Outdated Show resolved Hide resolved

cmd/observer/node_builder/observer_builder.go Outdated Show resolved Hide resolved

AndriiDiachuk added 4 commits February 2, 2024 18:05

Merge branch 'UlyanaAndrukhiv/2795-add-execution-state-sync' of https…

acaad72

…://github.com/UlyanaAndrukhiv/flow-go into AndriiDiachuk/access-add-support-for-indexing-execution-data

Removed async register store

ae48b7f

Moved handle collection to ingestion engine and using it in both plac…

478b02e

…es now

Merged

e29fad6

AndriiDiachuk requested a review from peterargue February 2, 2024 17:30

AndriiDiachuk mentioned this pull request Feb 7, 2024

[Access] Add standard Access API implementations on Observer #5358

Merged

UlyanaAndrukhiv mentioned this pull request Feb 9, 2024

[Access] Add state streaming API to observers #5368

Merged

Merged master

abaf8e2

peterargue reviewed Feb 15, 2024

View reviewed changes

cmd/observer/node_builder/observer_builder.go Show resolved Hide resolved

cmd/observer/node_builder/observer_builder.go Outdated Show resolved Hide resolved

engine/access/ingestion/engine.go Outdated Show resolved Hide resolved

AndriiDiachuk added 4 commits February 16, 2024 17:30

Added missing flags to observer builder

30d3e5d

Fixed tests, moved HandleCollection inside indexer_core

d99834a

Removed commented code

59892a9

Linted

e9ea0c7

peterargue reviewed Feb 28, 2024

View reviewed changes

AndriiDiachuk added 6 commits March 4, 2024 16:10

Moved collection metrics to own file

2a50202

Merge branch 'AndriiDiachuk/access-add-support-for-indexing-execution…

d86dde5

…-data' of https://github.com/AndriiDiachuk/flow-go into AndriiDiachuk/access-add-support-for-indexing-execution-data

Removed space

03bebd0

Generated mocks:

d6e4d12

Merge branch 'master' into AndriiDiachuk/access-add-support-for-index…

c26c876

…ing-execution-data

Added godoc

85a5374

AndriiDiachuk requested a review from peterargue March 4, 2024 14:45

Merge branch 'master' into AndriiDiachuk/access-add-support-for-index…

7acc58f

…ing-execution-data

peterargue reviewed Mar 8, 2024

View reviewed changes

module/state_synchronization/indexer/collection_executed_metric.go Outdated Show resolved Hide resolved

module/state_synchronization/indexer/collection_executed_metric.go Outdated Show resolved Hide resolved

Renamed functions in interface, added collections and blocks to impls…

e3a78dc

… constructor

AndriiDiachuk requested review from ramtinms and janezpodhostnik as code owners March 8, 2024 15:09

AndriiDiachuk requested a review from peterargue March 8, 2024 15:10

AndriiDiachuk added 3 commits March 8, 2024 19:40

Fixed refactoring error

f01a9fb

Generated mocks

aa5e845

Fixed blockExecoted refactoring

9ebf13a

peterargue approved these changes Mar 8, 2024

View reviewed changes

Added command to bootstrap

de57eb5

Merge branch 'master' into AndriiDiachuk/access-add-support-for-index…

615bd99

…ing-execution-data

peterargue enabled auto-merge March 9, 2024 13:14

peterargue added this pull request to the merge queue Mar 11, 2024

Merged via the queue into onflow:master with commit 50761d4 Mar 11, 2024
55 checks passed

peterargue mentioned this pull request Mar 15, 2024

[Access] Add support for indexing execution data on Observers #4848

Closed

peterargue added a commit that referenced this pull request Mar 15, 2024

Merge pull request #5256 from AndriiDiachuk/AndriiDiachuk/access-add-…

47ca164

…support-for-indexing-execution-data [Access] Add support for indexing execution data on Observers

peterargue mentioned this pull request Mar 15, 2024

[Access] Add support for indexing execution data and streaming on Observers #5553

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Access] Add support for indexing execution data on Observers #5256

[Access] Add support for indexing execution data on Observers #5256

AndriiDiachuk commented Jan 19, 2024 •

edited

Loading

codecov-commenter commented Jan 22, 2024 •

edited

Loading

peterargue left a comment

AndriiDiachuk commented Mar 4, 2024

peterargue left a comment

peterargue left a comment

AndriiDiachuk commented Mar 9, 2024

peterargue commented Mar 9, 2024

[Access] Add support for indexing execution data on Observers #5256

[Access] Add support for indexing execution data on Observers #5256

Conversation

AndriiDiachuk commented Jan 19, 2024 • edited Loading

codecov-commenter commented Jan 22, 2024 • edited Loading

Codecov Report

peterargue left a comment

Choose a reason for hiding this comment

AndriiDiachuk commented Mar 4, 2024

peterargue left a comment

Choose a reason for hiding this comment

peterargue left a comment

Choose a reason for hiding this comment

AndriiDiachuk commented Mar 9, 2024

peterargue commented Mar 9, 2024

AndriiDiachuk commented Jan 19, 2024 •

edited

Loading

codecov-commenter commented Jan 22, 2024 •

edited

Loading