Sync master with feature-flip-204 (#5057)

* Update cadence to v0.42.5

* use random string

* update godoc

* add in memory register store

* update comments

* add comment and update GetUpdatedRegister

* refactor IsErrPruned

* update tests

* prefix with tests InMemoryRegister

* use t.Run

* Apply suggestions from code review

Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>

* address review comments

* Apply suggestions from code review

Co-authored-by: Gregor G. <75445744+sideninja@users.noreply.github.com>

* fix linter

* fix linter

* fix lint

* use RegisterEntries

* add chainhash to integration

* add chainhash to insecure

* add chainhash to integration

* adds duration to the startup log

* update mocks

* adds TestNewSubscriptionRecordCache

* adds TestGetSubscribedTopics

* adds TestDuplicateTopics

* adds test move update cycle

* adds TestSubscriptionValidator_Integration

* adds TestMoveUpdateCycleWithDifferentPeers

* update emulator version

* updates mocks

* fixes build errors in insecure package

* adds TestSubscriptionProvider_GetSubscribedTopics_SkippingUnknownPeers

* minor fixes

* add comment explainin idProvider mock expectations

* use unittest random string

* add comment about continue statement

- remove obsolete continue statement

* Update fixtures.go

* small fixes

* fixes merge conflicts

* remove unnecessary variable declaration

* add support for testnet chain

* generalize support for all chains

* lint fix

* restoore gating EVM setup behind feature flag

* lint fix

* consolidates PeerIdFixtureB with PeerIdFixture

* adds lint and tidy to insecure package

* lint fix

* Revert "lint fix"

This reverts commit 9b1a4cc.

* lint fix

* revert the account not found error on withdraw

* define direct call types

* update docs

* remove unnecessary event types

* .

* add more contract types for testing

* fix tests

* skip test, requires funding (implemented in follow up PR)

* unskip test

* check error

* • extended compile-time checks for telemetry consumer enforcing that all happy-path interfaces are implemented
• extended subscriptions of telemetry consumer to receive missing events

* reduced notifications consumer interface in PaceMaker, as it only emits events from `hotstuff.ParticipantConsumer`

* update tests

* Refactor event emmision code

* Update network/p2p/unicast/README.MD

Co-authored-by: Khalil Claybon <khalil.claybon@dapperlabs.com>

* expose more test utility

* lint

* add register store

* update LastFinalizedAndExecutedHeight

* convert storage.ErrNotFound to nil

* fix lint

* fixes the logger

* moving ready to the select-case

* moving ready to select-case

* revises the error by update loop to make it irrecoverable

* clean up benchmarking

* adds select case for startup of subscription provider

* Update network/p2p/scoring/subscription_provider.go

Co-authored-by: Khalil Claybon <khalil.claybon@dapperlabs.com>

* adds documentation to subscription record cache

* fixes a bug with subscription record id

* adds more documentation for current cycle

* fix epoch event docs

* [Access] Handle script canceled and timeout errors

* [Access] Make script exec configurable

* Make computation limit configurable on ENs

* set limit setting once

* undo whitespace changes

* add storehouse loader

* add loaders

* add comments update log

* add comments

* update loader

* add extending block snapshot

* fix tests

* fix lint

* update interface

* update tests

* fix lint

* update committer

* update mocks

* update committer tests

* refactor collector

* update mocks

* update bootstrap

* fix noop committer

* fix computer_test

* fix tests

* refactor test

* update committer tests

* fix tests

* Update engine/access/rpc/backend/backend_scripts_test.go

Co-authored-by: Gregor G. <75445744+sideninja@users.noreply.github.com>

* fix committer tests

* add comment

* update tests

* Added implementation for execution api engine

* Added tests for execution api engine

* Fixed issues based on comments

* Added accidentally removed code

* Updated implementations following protobuf changes,updated mocks

* Linted

* [Access] Update websockets event streaming to return JSON-CDC encoded events

* Added tests for sentinel errors

* Update engine/execution/rpc/engine.go

Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>

* update NewStorageSnapshot

* update mocks

* update tests

* refactor script engine

* update mocks

* fix tests

* fix tests

* remove scripts engine tests

* update errors

* update tests

* refactor tests

* test CreateSnapshot

* add executing block snapshot

* reuse convert functions

* add block_end_snapshot tests

* remove changes

* Apply suggestions from code review

Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>

* add todo

* Update engine/execution/storehouse/block_end_snapshot_test.go

Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>

* Update engine/execution/storehouse/block_end_snapshot_test.go

Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>

* fix lint

* update comment

* add back interface

* update getFromStorage

* handle not found case

* add IsBlockExcuted

* update state commmitment by block id

* update mocks

* fix lint

* update unexecuted_loader

* fix ingestion tests

* fix stop control

* use IsParentExecuted

* fix loader tests

* Fix bug in reencoding and update tests

* update core-contracts version and state commitment constants

This is just to get tests to pass, all these changes will be overwritten
with the update to version v0.15.0 in #5027

---------

Co-authored-by: Yahya Hassanzadeh <yhassanzadeh13@ku.edu.tr>
Co-authored-by: Supun Setunga <supun.setunga@gmail.com>
Co-authored-by: Khalil Claybon <khalil.claybon@dapperlabs.com>
Co-authored-by: Gregor G <75445744+sideninja@users.noreply.github.com>
Co-authored-by: Misha <15269764+gomisha@users.noreply.github.com>
Co-authored-by: Leo Zhang (zhangchiqing) <zhangchiqing@gmail.com>
Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
Co-authored-by: Bastian Müller <bastian@turbolent.com>
Co-authored-by: Gregor Gololicic <gregor.gololicic@dapperlabs.com>
Co-authored-by: Ramtin M. Seraj <ramtinms@users.noreply.github.com>
Co-authored-by: ramtinms <ramtin.seraj@dapperlabs.com>
Co-authored-by: Yahya Hassanzadeh, Ph.D <yhassanzadeh@ieee.org>
Co-authored-by: Khalil Claybon <kclaybon1@gmail.com>
Co-authored-by: Alexander Hentschel <alex.hentschel@axiomzen.co>
Co-authored-by: Janez Podhostnik <67895329+janezpodhostnik@users.noreply.github.com>
Co-authored-by: Janez Podhostnik <janez.podhostnik@gmail.com>
Co-authored-by: Jan Bernatik <jan.bernatik@dapperlabs.com>
Co-authored-by: Andrii <andriy.dyachuk95@gmail.com>
Co-authored-by: Yurii Oleksyshyn <yuraolex@gmail.com>
Co-authored-by: Kan Zhang <kan@dapperlabs.com>

SECURITY.md

@@ -9,4 +9,4 @@ If you care about making a difference, please follow the guidelines below.
 
 # **Guidelines For Responsible Disclosure**
 
-We ask that all researchers adhere to these guidelines [here](https://docs.onflow.org/bounties/responsible-disclosure/)
+We ask that all researchers adhere to these guidelines [here](https://flow.com/flow-responsible-disclosure)

access/handler.go

@@ -24,6 +24,16 @@ type Handler struct {
 	me                   module.Local
 }
 
+// TODO: this is implemented in https://github.com/onflow/flow-go/pull/4957, remove when merged
+func (h *Handler) GetProtocolStateSnapshotByBlockID(ctx context.Context, request *access.GetProtocolStateSnapshotByBlockIDRequest) (*access.ProtocolStateSnapshotResponse, error) {
+	panic("implement me")
+}
+
+// TODO: this is implemented in https://github.com/onflow/flow-go/pull/4957, remove when merged
+func (h *Handler) GetProtocolStateSnapshotByHeight(ctx context.Context, request *access.GetProtocolStateSnapshotByHeightRequest) (*access.ProtocolStateSnapshotResponse, error) {
+	panic("implement me")
+}
+
 // HandlerOption is used to hand over optional constructor parameters
 type HandlerOption func(*Handler)
 

cmd/access/node_builder/access_node_builder.go

@@ -146,6 +146,7 @@ type AccessNodeConfig struct {
 	executionDataIndexingEnabled bool
 	registersDBPath              string
 	checkpointFile               string
+	scriptExecutorConfig         query.QueryConfig
 }
 
 type PublicNetworkConfig struct {
@@ -232,6 +233,7 @@ func DefaultAccessNodeConfig() *AccessNodeConfig {
 		executionDataIndexingEnabled: false,
 		registersDBPath:              filepath.Join(homedir, ".flow", "execution_state"),
 		checkpointFile:               cmd.NotSet,
+		scriptExecutorConfig:         query.NewDefaultConfig(),
 	}
 }
 
@@ -771,6 +773,7 @@ func (builder *FlowAccessNodeBuilder) BuildExecutionSyncComponents() *FlowAccess
 					query.NewProtocolStateWrapper(builder.State),
 					builder.Storage.Headers,
 					builder.ExecutionIndexerCore.RegisterValue,
+					builder.scriptExecutorConfig,
 				)
 				if err != nil {
 					return nil, err
@@ -929,6 +932,11 @@ func (builder *FlowAccessNodeBuilder) extraFlags() {
 
 		// Script Execution
 		flags.StringVar(&builder.rpcConf.BackendConfig.ScriptExecutionMode, "script-execution-mode", defaultConfig.rpcConf.BackendConfig.ScriptExecutionMode, "mode to use when executing scripts. one of (local-only, execution-nodes-only, failover, compare)")
+		flags.Uint64Var(&builder.scriptExecutorConfig.ComputationLimit, "script-execution-computation-limit", defaultConfig.scriptExecutorConfig.ComputationLimit, "maximum number of computation units a locally executed script can use. default: 100000")
+		flags.IntVar(&builder.scriptExecutorConfig.MaxErrorMessageSize, "script-execution-max-error-length", defaultConfig.scriptExecutorConfig.MaxErrorMessageSize, "maximum number characters to include in error message strings. additional characters are truncated. default: 1000")
+		flags.DurationVar(&builder.scriptExecutorConfig.LogTimeThreshold, "script-execution-log-time-threshold", defaultConfig.scriptExecutorConfig.LogTimeThreshold, "emit a log for any scripts that take over this threshold. default: 1s")
+		flags.DurationVar(&builder.scriptExecutorConfig.ExecutionTimeLimit, "script-execution-timeout", defaultConfig.scriptExecutorConfig.ExecutionTimeLimit, "timeout value for locally executed scripts. default: 10s")
+
 	}).ValidateFlags(func() error {
 		if builder.supportsObserver && (builder.PublicNetworkConfig.BindAddress == cmd.NotSet || builder.PublicNetworkConfig.BindAddress == "") {
 			return errors.New("public-network-address must be set if supports-observer is true")
@@ -1563,6 +1571,7 @@ func (builder *FlowAccessNodeBuilder) initPublicLibp2pNode(networkKey crypto.Pri
 			UpdateInterval:    builder.FlowConfig.NetworkConfig.PeerUpdateInterval,
 			ConnectorFactory:  connection.DefaultLibp2pBackoffConnectorFactory(),
 		},
+		&builder.FlowConfig.NetworkConfig.GossipSubConfig.SubscriptionProviderConfig,
 		&p2p.DisallowListCacheConfig{
 			MaxSize: builder.FlowConfig.NetworkConfig.DisallowListNotificationCacheSize,
 			Metrics: metrics.DisallowListCacheMetricsFactory(builder.HeroCacheMetricsFactory(), network.PublicNetwork),

cmd/consensus/main.go

@@ -594,6 +594,8 @@ func main() {
 			)
 
 			notifier.AddParticipantConsumer(telemetryConsumer)
+			notifier.AddCommunicatorConsumer(telemetryConsumer)
+			notifier.AddFinalizationConsumer(telemetryConsumer)
 			notifier.AddFollowerConsumer(followerDistributor)
 
 			// initialize the persister

cmd/execution_builder.go

@@ -540,16 +540,10 @@ func (exeNode *ExecutionNode) LoadProviderEngine(
 			"cannot get the latest executed block id: %w",
 			err)
 	}
-	stateCommit, err := exeNode.executionState.StateCommitmentByBlockID(
-		ctx,
-		blockID)
+	blockSnapshot, _, err := exeNode.executionState.CreateStorageSnapshot(blockID)
 	if err != nil {
-		return nil, fmt.Errorf(
-			"cannot get the state commitment at latest executed block id %s: %w",
-			blockID.String(),
-			err)
+		return nil, fmt.Errorf("cannot create a storage snapshot at block %v: %w", blockID, err)
 	}
-	blockSnapshot := exeNode.executionState.NewStorageSnapshot(stateCommit)
 
 	// Get the epoch counter from the smart contract at the last executed block.
 	contractEpochCounter, err := getContractEpochCounter(
@@ -868,7 +862,7 @@ func (exeNode *ExecutionNode) LoadIngestionEngine(
 	}
 
 	fetcher := fetcher.NewCollectionFetcher(node.Logger, exeNode.collectionRequester, node.State, exeNode.exeConf.onflowOnlyLNs)
-	loader := loader.NewLoader(node.Logger, node.State, node.Storage.Headers, exeNode.executionState)
+	loader := loader.NewUnexecutedLoader(node.Logger, node.State, node.Storage.Headers, exeNode.executionState)
 
 	exeNode.ingestionEng, err = ingestion.New(
 		exeNode.ingestionUnit,
@@ -905,7 +899,6 @@ func (exeNode *ExecutionNode) LoadScriptsEngine(node *NodeConfig) (module.ReadyD
 
 	exeNode.scriptsEng = scripts.New(
 		node.Logger,
-		node.State,
 		exeNode.computationManager.QueryExecutor(),
 		exeNode.executionState,
 	)

cmd/execution_config.go

@@ -11,6 +11,7 @@ import (
 	"github.com/onflow/flow-go/engine/common/provider"
 	"github.com/onflow/flow-go/engine/execution/computation/query"
 	exeprovider "github.com/onflow/flow-go/engine/execution/provider"
+	"github.com/onflow/flow-go/fvm"
 	"github.com/onflow/flow-go/model/flow"
 	"github.com/onflow/flow-go/module/mempool"
 	"github.com/onflow/flow-go/utils/grpcutils"
@@ -89,6 +90,8 @@ func (exeConf *ExecutionConfig) SetupFlags(flags *pflag.FlagSet) {
 		"threshold for logging script execution")
 	flags.DurationVar(&exeConf.computationConfig.QueryConfig.ExecutionTimeLimit, "script-execution-time-limit", query.DefaultExecutionTimeLimit,
 		"script execution time limit")
+	flags.Uint64Var(&exeConf.computationConfig.QueryConfig.ComputationLimit, "script-execution-computation-limit", fvm.DefaultComputationLimit,
+		"script execution computation limit")
 	flags.UintVar(&exeConf.transactionResultsCacheSize, "transaction-results-cache-size", 10000, "number of transaction results to be cached")
 	flags.BoolVar(&exeConf.extensiveLog, "extensive-logging", false, "extensive logging logs tx contents and block headers")
 	flags.DurationVar(&exeConf.chunkDataPackQueryTimeout, "chunk-data-pack-query-timeout", exeprovider.DefaultChunkDataPackQueryTimeout, "timeout duration to determine a chunk data pack query being slow")

cmd/observer/node_builder/observer_builder.go

@@ -722,6 +722,7 @@ func (builder *ObserverServiceBuilder) initPublicLibp2pNode(networkKey crypto.Pr
 		&builder.FlowConfig.NetworkConfig.ResourceManager,
 		&builder.FlowConfig.NetworkConfig.GossipSubConfig.GossipSubRPCInspectorsConfig,
 		p2pconfig.PeerManagerDisableConfig(), // disable peer manager for observer node.
+		&builder.FlowConfig.NetworkConfig.GossipSubConfig.SubscriptionProviderConfig,
 		&p2p.DisallowListCacheConfig{
 			MaxSize: builder.FlowConfig.NetworkConfig.DisallowListNotificationCacheSize,
 			Metrics: metrics.DisallowListCacheMetricsFactory(builder.HeroCacheMetricsFactory(), network.PublicNetwork),

config/default-config.yml

@@ -33,18 +33,6 @@ network-config:
   # retry a unicast stream to a remote peer 3 times, the peer will give up and will not retry creating a unicast stream to that remote peer.
   # When it is set to zero it means that the peer will not retry creating a unicast stream to a remote peer if it fails.
   unicast-max-stream-creation-retry-attempt-times: 3
-  # The number of seconds that the local peer waits since the last successful dial to a remote peer before resetting the unicast dial retry budget from zero to the maximum default.
-  # If it is set to 3600s (1h) for example, it means that if it has passed at least one hour since the last successful dial, and the remote peer has a zero dial retry budget,
-  # the unicast dial retry budget for that remote peer will be reset to the maximum default.
-  unicast-dial-zero-retry-reset-threshold: 3600s
-  # The maximum number of retry attempts for dialing a remote peer before giving up. If it is set to 3 for example, it means that if a peer fails to dial a remote peer 3 times,
-  # the peer will give up and will not retry dialing that remote peer.
-  unicast-max-dial-retry-attempt-times: 3
-  # The backoff delay used in the exponential backoff for consecutive failed unicast dial attempts to a remote peer.
-  unicast-dial-backoff-delay: 1s
-  # The backoff delay used in the exponential backoff for backing off concurrent create stream attempts to the same remote peer
-  # when there is no available connections to that remote peer and a dial is in progress.
-  unicast-dial-in-progress-backoff-delay: 1s
   # The size of the dial config cache used to keep track of the dial config for each remote peer. The dial config is used to keep track of the dial retry budget for each remote peer.
   # Recommended to set it to the maximum number of remote peers in the network.
   unicast-dial-config-cache-size: 10_000
@@ -146,6 +134,15 @@ network-config:
   gossipsub-rpc-sent-tracker-workers: 5
   # Peer scoring is the default value for enabling peer scoring
   gossipsub-peer-scoring-enabled: true
+  # The interval for updating the list of subscribed peers to all topics in gossipsub. This is used to keep track of subscriptions
+  # violations and penalize peers accordingly. Recommended value is in the order of a few minutes to avoid contentions; as the operation
+  # reads all topics and all peers subscribed to each topic.
+  gossipsub-subscription-provider-update-interval: 10m
+  # The size of cache for keeping the list of all peers subscribed to each topic (same as the local node). This cache is the local node's
+  # view of the network and is used to detect subscription violations and penalize peers accordingly. Recommended to be big enough to
+  # keep the entire network's size. Otherwise, the local node's view of the network will be incomplete due to cache eviction.
+  # Recommended size is 10x the number of peers in the network.
+  gossipsub-subscription-provider-cache-size: 10000
 
   # Gossipsub rpc inspectors configs
   # The size of the queue for notifications about invalid RPC messages

consensus/hotstuff/notifications/telemetry.go

@@ -38,7 +38,10 @@ type TelemetryConsumer struct {
 	noPathLogger zerolog.Logger
 }
 
+// Telemetry implements consumers for _all happy-path_ interfaces in consensus/hotstuff/notifications/telemetry.go:
 var _ hotstuff.ParticipantConsumer = (*TelemetryConsumer)(nil)
+var _ hotstuff.CommunicatorConsumer = (*TelemetryConsumer)(nil)
+var _ hotstuff.FinalizationConsumer = (*TelemetryConsumer)(nil)
 var _ hotstuff.VoteCollectorConsumer = (*TelemetryConsumer)(nil)
 var _ hotstuff.TimeoutCollectorConsumer = (*TelemetryConsumer)(nil)
 

consensus/hotstuff/pacemaker/pacemaker.go

@@ -31,7 +31,7 @@ type ActivePaceMaker struct {
 
 	ctx            context.Context
 	timeoutControl *timeout.Controller
-	notifier       hotstuff.Consumer
+	notifier       hotstuff.ParticipantConsumer
 	viewTracker    viewTracker
 	started        bool
 }

engine/access/apiproxy/access_api_proxy.go

@@ -42,6 +42,16 @@ func (h *FlowAccessAPIRouter) log(handler, rpc string, err error) {
 	logger.Info().Msg("request succeeded")
 }
 
+// TODO: this is implemented in https://github.com/onflow/flow-go/pull/4957, remove when merged
+func (h *FlowAccessAPIRouter) GetProtocolStateSnapshotByBlockID(ctx context.Context, request *access.GetProtocolStateSnapshotByBlockIDRequest) (*access.ProtocolStateSnapshotResponse, error) {
+	panic("implement me")
+}
+
+// TODO: this is implemented in https://github.com/onflow/flow-go/pull/4957, remove when merged
+func (h *FlowAccessAPIRouter) GetProtocolStateSnapshotByHeight(ctx context.Context, request *access.GetProtocolStateSnapshotByHeightRequest) (*access.ProtocolStateSnapshotResponse, error) {
+	panic("implement me")
+}
+
 // Ping pings the service. It is special in the sense that it responds successful,
 // only if all underlying services are ready.
 func (h *FlowAccessAPIRouter) Ping(context context.Context, req *access.PingRequest) (*access.PingResponse, error) {