Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Access] Implement gRPC streaming endpoint SubscribeBlocks #5307

Merged
Merged
Show file tree
Hide file tree
Changes from 99 commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
8e24fde
Merged with master
UlyanaAndrukhiv Dec 19, 2023
a2f4e9c
Replaced protobuf version to new one
UlyanaAndrukhiv Dec 21, 2023
7f8356f
Implemented sealed blocks retriving
Guitarheroua Dec 27, 2023
118361f
Updated protobuf, added tests for finalized subscribe blocks
UlyanaAndrukhiv Dec 27, 2023
f6f2a16
Merged with stream-sealed
UlyanaAndrukhiv Dec 27, 2023
9f5e399
Refactored structs and naming
UlyanaAndrukhiv Dec 27, 2023
20e7d23
Make universal functions in for block watcher
Guitarheroua Dec 27, 2023
31d3f1e
Added checks for block status
UlyanaAndrukhiv Dec 27, 2023
69b04f1
Merge branch 'merge-stream-sealed' of github.com:Guitarheroua/flow-go…
UlyanaAndrukhiv Dec 27, 2023
ab097a6
Added test for subscribe blocks backend
UlyanaAndrukhiv Jan 3, 2024
c93df5b
Added more comments to subscribe blocks impl
UlyanaAndrukhiv Jan 4, 2024
d84b1d4
Updated subscriptionOnFinalizedBlock process
UlyanaAndrukhiv Jan 29, 2024
7ba6f9d
Updated protobuf
UlyanaAndrukhiv Jan 29, 2024
aa09fb1
Updated protobuf module
UlyanaAndrukhiv Jan 29, 2024
a7358b0
Reverted back unnecessary changes
UlyanaAndrukhiv Jan 29, 2024
5399dd3
Updated according to comments
UlyanaAndrukhiv Jan 30, 2024
44ebeb8
Added godoc for GetStartHeight
UlyanaAndrukhiv Jan 30, 2024
a9d99a1
Removed default block status for subscribe blocks
UlyanaAndrukhiv Jan 30, 2024
8d3af7e
Merged with master
UlyanaAndrukhiv Jan 30, 2024
98daa77
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Jan 30, 2024
579416a
Merged with master
UlyanaAndrukhiv Jan 30, 2024
ceebdcb
Fixed rpc backend tests, mocked seal head state when rpc backend creates
UlyanaAndrukhiv Jan 31, 2024
a168107
Renamed BlockWatcher to ChainStateTracker, added interface for it, ad…
UlyanaAndrukhiv Feb 6, 2024
79c9386
Updated generate-mocks
UlyanaAndrukhiv Feb 6, 2024
ecfa3bb
Updated version of flow-emulator
UlyanaAndrukhiv Feb 7, 2024
33d2526
Merged with master
UlyanaAndrukhiv Feb 14, 2024
e66c9cc
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Feb 14, 2024
0f2c249
Fixed dependencies
UlyanaAndrukhiv Feb 14, 2024
d06382b
Added updated protobuf for insecure
UlyanaAndrukhiv Feb 14, 2024
5a6c4d0
refactored creating ChainStateTracker, updated tests
UlyanaAndrukhiv Feb 16, 2024
536509a
Reverted back changes in backend_test
UlyanaAndrukhiv Feb 16, 2024
d07bc45
Updated last commit
UlyanaAndrukhiv Feb 16, 2024
52a1579
Merged with master
UlyanaAndrukhiv Feb 16, 2024
2bd3a79
Updated test
UlyanaAndrukhiv Feb 16, 2024
254aa51
Reverted back unnecessary changes
UlyanaAndrukhiv Feb 16, 2024
e2460d0
Implemented SubscribeBlockHeaders and SubscribeBlockDigests
UlyanaAndrukhiv Feb 20, 2024
ae56423
Refactored block and block headers subscribes
UlyanaAndrukhiv Feb 20, 2024
022e50e
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Feb 20, 2024
6c04830
Added more comments, updated backendSubscribeBlocks
UlyanaAndrukhiv Feb 20, 2024
2fdcbb7
Updated protobuf version
UlyanaAndrukhiv Feb 22, 2024
9c6ac0b
Moved index package to engine/access/index
UlyanaAndrukhiv Feb 22, 2024
cd8e6f9
Updated godoc, logs, errors andtests according to comments
UlyanaAndrukhiv Feb 22, 2024
03052b6
Added getBlockHeader method to avoid extra db lookup
UlyanaAndrukhiv Feb 22, 2024
6746499
Merged with master
UlyanaAndrukhiv Feb 22, 2024
c194f8d
Added irrecoverable handling errors for GetStartHeight
UlyanaAndrukhiv Feb 23, 2024
d19ed31
Removed multierror according to comment
UlyanaAndrukhiv Feb 23, 2024
cc46ed2
Updated errors
UlyanaAndrukhiv Feb 23, 2024
4557080
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Feb 23, 2024
1578064
Updated comment in chain_state_tracker.go
UlyanaAndrukhiv Feb 27, 2024
2c2ca8c
Renamed LightweightBlock structure to BlockDigest
UlyanaAndrukhiv Feb 27, 2024
76645bf
Updated handling errors on backend stream blocks
UlyanaAndrukhiv Feb 27, 2024
91f1663
Updated access handler according to commits
UlyanaAndrukhiv Feb 27, 2024
3489349
Merge branch 'master' of github.com:UlyanaAndrukhiv/flow-go into Ulya…
UlyanaAndrukhiv Feb 27, 2024
b629916
Merged with master
UlyanaAndrukhiv Feb 27, 2024
f7388fb
Split chain state tracker into multiple trackers, updated tests
UlyanaAndrukhiv Feb 27, 2024
2c19e2e
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Feb 27, 2024
422329e
Updated handler and protobuf
UlyanaAndrukhiv Feb 28, 2024
f1e66ac
Updated flow-go emulator version
UlyanaAndrukhiv Feb 28, 2024
0c6aa2e
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Feb 28, 2024
c1530f2
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Feb 28, 2024
1676f44
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Feb 28, 2024
b76bdbb
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
franklywatson Feb 28, 2024
f193119
Merged with master, moved transaction result index to index folder
UlyanaAndrukhiv Feb 28, 2024
528f267
Linted
UlyanaAndrukhiv Feb 28, 2024
0698b7c
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Feb 29, 2024
7e0bd39
Updated protobuf version
UlyanaAndrukhiv Feb 29, 2024
a3bbe3b
Updated godoc and errors according to comments
UlyanaAndrukhiv Feb 29, 2024
3b8d311
Added more godoc, updated according to comments
UlyanaAndrukhiv Feb 29, 2024
0d913b8
Updated namimg in tests, updated block tracker constructor
UlyanaAndrukhiv Feb 29, 2024
c7eb4d1
Refactored trackers
UlyanaAndrukhiv Feb 29, 2024
1115cfa
Removed unused variable
UlyanaAndrukhiv Feb 29, 2024
d5d812a
Updated godoc to trackers, added fixed according to comments
UlyanaAndrukhiv Feb 29, 2024
61323b8
Updated godoc
UlyanaAndrukhiv Feb 29, 2024
7f9b5bc
Added rest of updates according to comments
UlyanaAndrukhiv Mar 1, 2024
0548595
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Mar 1, 2024
593ff54
Updated flow-emulator version
UlyanaAndrukhiv Mar 1, 2024
ae3b494
Merge branch 'UlyanaAndrukhiv/subscribe-blocks' of github.com:UlyanaA…
UlyanaAndrukhiv Mar 1, 2024
139f082
Updated protobuf for integration tests
UlyanaAndrukhiv Mar 1, 2024
2b7b592
First batch of updates according to comments
UlyanaAndrukhiv Mar 5, 2024
4f592a7
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Mar 5, 2024
0b658f0
Moved broadcaster from state_stream engine to executionDataTracker ac…
UlyanaAndrukhiv Mar 5, 2024
af480e7
Updated protobuf and access handler according to comments
UlyanaAndrukhiv Mar 5, 2024
509238e
Refactoredd handler for subscriptions, updated BlockDigest according …
UlyanaAndrukhiv Mar 5, 2024
11c944f
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
franklywatson Mar 5, 2024
915da03
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Mar 6, 2024
3cb9be2
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Mar 6, 2024
3f1e02e
Updated flow-emulator version
UlyanaAndrukhiv Mar 7, 2024
365b642
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Mar 7, 2024
3e6563e
Added batch first of updates according to comments
UlyanaAndrukhiv Mar 8, 2024
d4fcbc6
Refactored handler.go according to suggested comment
UlyanaAndrukhiv Mar 8, 2024
852246a
Updated access api, according to comment, updated tests, moved common…
UlyanaAndrukhiv Mar 11, 2024
41248d5
Splitted handling the start block, updated godoc, updated tests
UlyanaAndrukhiv Mar 11, 2024
9020ed7
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Mar 11, 2024
cf65415
Generated mocks
UlyanaAndrukhiv Mar 11, 2024
2112249
Merge branch 'UlyanaAndrukhiv/subscribe-blocks' of github.com:UlyanaA…
UlyanaAndrukhiv Mar 11, 2024
fa2a1eb
Updated flow-emulator version
UlyanaAndrukhiv Mar 11, 2024
4f20305
Merged with master, added execution data tracker to observer
UlyanaAndrukhiv Mar 13, 2024
4bed022
Updated according to comments
UlyanaAndrukhiv Mar 13, 2024
a1241bd
Removed OnExecutionData from state stream engine
UlyanaAndrukhiv Mar 13, 2024
2f8254d
Moved fatal log inside execution data tracker, removed unnecessary Ge…
UlyanaAndrukhiv Mar 13, 2024
9675f59
Fixed error code in godoc
UlyanaAndrukhiv Mar 13, 2024
ffe008c
Merge branch 'master' into UlyanaAndrukhiv/subscribe-blocks
UlyanaAndrukhiv Mar 13, 2024
39d6307
Updated according to last comment
UlyanaAndrukhiv Mar 13, 2024
4b23c0a
Updated flow-emulator
UlyanaAndrukhiv Mar 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,8 @@ generate-mocks: install-mock-generators
mockery --name 'API' --dir="./access" --case=underscore --output="./access/mock" --outpkg="mock"
mockery --name 'API' --dir="./engine/protocol" --case=underscore --output="./engine/protocol/mock" --outpkg="mock"
mockery --name '.*' --dir="./engine/access/state_stream" --case=underscore --output="./engine/access/state_stream/mock" --outpkg="mock"
mockery --name 'BlockTracker' --dir="./engine/access/subscription" --case=underscore --output="./engine/access/subscription/mock" --outpkg="mock"
mockery --name 'ExecutionDataTracker' --dir="./engine/access/subscription" --case=underscore --output="./engine/access/subscription/mock" --outpkg="mock"
mockery --name 'ConnectionFactory' --dir="./engine/access/rpc/connection" --case=underscore --output="./engine/access/rpc/connection/mock" --outpkg="mock"
mockery --name 'Communicator' --dir="./engine/access/rpc/backend" --case=underscore --output="./engine/access/rpc/backend/mock" --outpkg="mock"

Expand Down
142 changes: 142 additions & 0 deletions access/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"github.com/onflow/flow/protobuf/go/flow/access"
"github.com/onflow/flow/protobuf/go/flow/entities"

"github.com/onflow/flow-go/engine/access/subscription"
"github.com/onflow/flow-go/engine/common/rpc/convert"
"github.com/onflow/flow-go/model/flow"
)
Expand Down Expand Up @@ -52,6 +53,147 @@ type API interface {

GetExecutionResultForBlockID(ctx context.Context, blockID flow.Identifier) (*flow.ExecutionResult, error)
GetExecutionResultByID(ctx context.Context, id flow.Identifier) (*flow.ExecutionResult, error)

// SubscribeBlocks

// SubscribeBlocksFromStartBlockID subscribes to the finalized or sealed blocks starting at the requested
// start block id, up until the latest available block. Once the latest is
// reached, the stream will remain open and responses are sent for each new
// block as it becomes available.
//
// Each block is filtered by the provided block status, and only
// those blocks that match the status are returned.
//
// Parameters:
// - ctx: Context for the operation.
// - startBlockID: The identifier of the starting block.
// - blockStatus: The status of the block, which could be only BlockStatusSealed or BlockStatusFinalized.
//
// If invalid parameters will be supplied SubscribeBlocksFromStartBlockID will return a failed subscription.
SubscribeBlocksFromStartBlockID(ctx context.Context, startBlockID flow.Identifier, blockStatus flow.BlockStatus) subscription.Subscription
// SubscribeBlocksFromStartHeight subscribes to the finalized or sealed blocks starting at the requested
// start block height, up until the latest available block. Once the latest is
// reached, the stream will remain open and responses are sent for each new
// block as it becomes available.
//
// Each block is filtered by the provided block status, and only
// those blocks that match the status are returned.
//
// Parameters:
// - ctx: Context for the operation.
// - startHeight: The height of the starting block.
// - blockStatus: The status of the block, which could be only BlockStatusSealed or BlockStatusFinalized.
//
// If invalid parameters will be supplied SubscribeBlocksFromStartHeight will return a failed subscription.
SubscribeBlocksFromStartHeight(ctx context.Context, startHeight uint64, blockStatus flow.BlockStatus) subscription.Subscription
// SubscribeBlocksFromLatest subscribes to the finalized or sealed blocks starting at the latest sealed block,
// up until the latest available block. Once the latest is
// reached, the stream will remain open and responses are sent for each new
// block as it becomes available.
//
// Each block is filtered by the provided block status, and only
// those blocks that match the status are returned.
//
// Parameters:
// - ctx: Context for the operation.
// - blockStatus: The status of the block, which could be only BlockStatusSealed or BlockStatusFinalized.
//
// If invalid parameters will be supplied SubscribeBlocksFromLatest will return a failed subscription.
SubscribeBlocksFromLatest(ctx context.Context, blockStatus flow.BlockStatus) subscription.Subscription

// SubscribeHeaders

// SubscribeBlockHeadersFromStartBlockID streams finalized or sealed block headers starting at the requested
// start block id, up until the latest available block header. Once the latest is
// reached, the stream will remain open and responses are sent for each new
// block header as it becomes available.
//
// Each block header are filtered by the provided block status, and only
// those block headers that match the status are returned.
//
// Parameters:
// - ctx: Context for the operation.
// - startBlockID: The identifier of the starting block.
// - blockStatus: The status of the block, which could be only BlockStatusSealed or BlockStatusFinalized.
//
// If invalid parameters will be supplied SubscribeBlockHeadersFromStartBlockID will return a failed subscription.
SubscribeBlockHeadersFromStartBlockID(ctx context.Context, startBlockID flow.Identifier, blockStatus flow.BlockStatus) subscription.Subscription
// SubscribeBlockHeadersFromStartHeight streams finalized or sealed block headers starting at the requested
// start block height, up until the latest available block header. Once the latest is
// reached, the stream will remain open and responses are sent for each new
// block header as it becomes available.
//
// Each block header are filtered by the provided block status, and only
// those block headers that match the status are returned.
//
// Parameters:
// - ctx: Context for the operation.
// - startHeight: The height of the starting block.
// - blockStatus: The status of the block, which could be only BlockStatusSealed or BlockStatusFinalized.
//
// If invalid parameters will be supplied SubscribeBlockHeadersFromStartHeight will return a failed subscription.
SubscribeBlockHeadersFromStartHeight(ctx context.Context, startHeight uint64, blockStatus flow.BlockStatus) subscription.Subscription
// SubscribeBlockHeadersFromLatest streams finalized or sealed block headers starting at the latest sealed block,
// up until the latest available block header. Once the latest is
// reached, the stream will remain open and responses are sent for each new
// block header as it becomes available.
//
// Each block header are filtered by the provided block status, and only
// those block headers that match the status are returned.
//
// Parameters:
// - ctx: Context for the operation.
// - blockStatus: The status of the block, which could be only BlockStatusSealed or BlockStatusFinalized.
//
// If invalid parameters will be supplied SubscribeBlockHeadersFromLatest will return a failed subscription.
SubscribeBlockHeadersFromLatest(ctx context.Context, blockStatus flow.BlockStatus) subscription.Subscription

// Subscribe digests

// SubscribeBlockDigestsFromStartBlockID streams finalized or sealed lightweight block starting at the requested
// start block id, up until the latest available block. Once the latest is
// reached, the stream will remain open and responses are sent for each new
// block as it becomes available.
//
// Each lightweight block are filtered by the provided block status, and only
// those blocks that match the status are returned.
//
// Parameters:
// - ctx: Context for the operation.
// - startBlockID: The identifier of the starting block.
// - blockStatus: The status of the block, which could be only BlockStatusSealed or BlockStatusFinalized.
//
// If invalid parameters will be supplied SubscribeBlockDigestsFromStartBlockID will return a failed subscription.
SubscribeBlockDigestsFromStartBlockID(ctx context.Context, startBlockID flow.Identifier, blockStatus flow.BlockStatus) subscription.Subscription
// SubscribeBlockDigestsFromStartHeight streams finalized or sealed lightweight block starting at the requested
// start block height, up until the latest available block. Once the latest is
// reached, the stream will remain open and responses are sent for each new
// block as it becomes available.
//
// Each lightweight block are filtered by the provided block status, and only
// those blocks that match the status are returned.
//
// Parameters:
// - ctx: Context for the operation.
// - startHeight: The height of the starting block.
// - blockStatus: The status of the block, which could be only BlockStatusSealed or BlockStatusFinalized.
//
// If invalid parameters will be supplied SubscribeBlockDigestsFromStartHeight will return a failed subscription.
SubscribeBlockDigestsFromStartHeight(ctx context.Context, startHeight uint64, blockStatus flow.BlockStatus) subscription.Subscription
// SubscribeBlockDigestsFromLatest streams finalized or sealed lightweight block starting at the latest sealed block,
// up until the latest available block. Once the latest is
// reached, the stream will remain open and responses are sent for each new
// block as it becomes available.
//
// Each lightweight block are filtered by the provided block status, and only
// those blocks that match the status are returned.
//
// Parameters:
// - ctx: Context for the operation.
// - blockStatus: The status of the block, which could be only BlockStatusSealed or BlockStatusFinalized.
//
// If invalid parameters will be supplied SubscribeBlockDigestsFromLatest will return a failed subscription.
SubscribeBlockDigestsFromLatest(ctx context.Context, blockStatus flow.BlockStatus) subscription.Subscription
}

// TODO: Combine this with flow.TransactionResult?
Expand Down
Loading