Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Message validation optimization #6462

Open
wants to merge 34 commits into
base: feat/equivalent-messages
Choose a base branch
from
+1,275 −634
Open

Message validation optimization #6462

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
10afdba
initial commit.
cristure Sep 12, 2024
aaa9d4b
added test for processing message.
cristure Sep 13, 2024
603bd69
added message validation cache.
cristure Sep 16, 2024
57f009e
fix some tests.
cristure Sep 16, 2024
f5a6f2e
fixed missing topic.
cristure Sep 16, 2024
8eec330
fix some more tests.
cristure Sep 17, 2024
a733097
fix nil pointer dereferences in tests.
cristure Sep 17, 2024
bab3b05
more tests fixed.
cristure Sep 17, 2024
c3f7120
add map for cacher in more unit tests.
cristure Sep 17, 2024
b62e37b
fix nil map for more tests.
cristure Sep 17, 2024
c00f8ee
fix process tests.
cristure Sep 17, 2024
94a64c1
Merge branch 'feat/equivalent-messages' into message_validation_optim…
cristure Sep 17, 2024
d77e367
Merge remote-tracking branch 'origin/feat/equivalent-messages' into m…
cristure Sep 18, 2024
d59c5bd
fix conflicts with target branch.
cristure Sep 18, 2024
396cd8e
refactored message validation to a different component.
cristure Sep 19, 2024
a44129f
cosmetic changes.
cristure Sep 20, 2024
afdceb1
added intercepted data verifier stub in multi data tests.
cristure Sep 20, 2024
d8d8113
fix single data interceptor tests.
cristure Sep 20, 2024
fe2082c
commit debug strings for CI.
cristure Sep 20, 2024
c6b7a0f
moved map of cachers in node processor.
cristure Sep 20, 2024
f59fc85
cosmetic changes.
cristure Sep 20, 2024
d00c7ab
fix integration tests.
cristure Sep 20, 2024
f21e24f
fix some more tests.
cristure Sep 20, 2024
f40d222
fix some unit tests.
cristure Sep 23, 2024
3a327c4
fix nil map in tests.
cristure Sep 23, 2024
a756a1f
Merge branch 'feat/equivalent-messages' into message_validation_optim…
cristure Sep 23, 2024
ff8eaa9
Merge remote-tracking branch 'origin/feat/equivalent-messages' into m…
cristure Sep 25, 2024
cde6c80
cosmetic changes.
cristure Sep 25, 2024
e0cf24b
Merge remote-tracking branch 'origin/message_validation_optimization'…
cristure Sep 25, 2024
9504263
refactored map of caches for intercepted data into a new component.
cristure Sep 25, 2024
b622085
added mock for interceptedDataVerifierFactory and injected into tests.
cristure Sep 26, 2024
f0b34ef
more test fixes.
cristure Sep 26, 2024
b05c650
increase wait time for CI run.
cristure Sep 26, 2024
5dc6bba
bring back rw mutex on interceptedDataVerifier.
cristure Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions epochStart/bootstrap/factory/epochStartInterceptorsContainerFactory.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"github.com/multiversx/mx-chain-core-go/core"
"github.com/multiversx/mx-chain-core-go/core/check"
"github.com/multiversx/mx-chain-core-go/data/typeConverters"

"github.com/multiversx/mx-chain-go/common"
"github.com/multiversx/mx-chain-go/config"
"github.com/multiversx/mx-chain-go/dataRetriever"
Expand All @@ -16,6 +17,7 @@ import (
"github.com/multiversx/mx-chain-go/process"
"github.com/multiversx/mx-chain-go/process/factory/interceptorscontainer"
"github.com/multiversx/mx-chain-go/sharding"
"github.com/multiversx/mx-chain-go/storage"
"github.com/multiversx/mx-chain-go/storage/cache"
"github.com/multiversx/mx-chain-go/update"
)
Expand Down Expand Up @@ -108,6 +110,7 @@ func NewEpochStartInterceptorsContainer(args ArgsEpochStartInterceptorContainer)
FullArchivePeerShardMapper: fullArchivePeerShardMapper,
HardforkTrigger: hardforkTrigger,
NodeOperationMode: args.NodeOperationMode,
ProcessedMessagesCacheMap: make(map[string]storage.Cacher),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check all intializations of this map, as we might end up with multiple versions of this.

}

interceptorsContainerFactory, err := interceptorscontainer.NewMetaInterceptorsContainerFactory(containerFactoryArgs)
Expand Down
3 changes: 2 additions & 1 deletion epochStart/bootstrap/process.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ import (
"github.com/multiversx/mx-chain-core-go/data"
"github.com/multiversx/mx-chain-core-go/data/block"
"github.com/multiversx/mx-chain-core-go/data/typeConverters/uint64ByteSlice"
logger "github.com/multiversx/mx-chain-logger-go"

"github.com/multiversx/mx-chain-go/common"
disabledCommon "github.com/multiversx/mx-chain-go/common/disabled"
"github.com/multiversx/mx-chain-go/common/ordering"
Expand Down Expand Up @@ -52,7 +54,6 @@ import (
"github.com/multiversx/mx-chain-go/trie/storageMarker"
"github.com/multiversx/mx-chain-go/update"
updateSync "github.com/multiversx/mx-chain-go/update/sync"
logger "github.com/multiversx/mx-chain-logger-go"
)

var log = logger.GetOrCreate("epochStart/bootstrap")
Expand Down
3 changes: 2 additions & 1 deletion epochStart/bootstrap/storageProcess_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (
"github.com/multiversx/mx-chain-core-go/data"
"github.com/multiversx/mx-chain-core-go/data/block"
"github.com/multiversx/mx-chain-core-go/data/endProcess"
"github.com/stretchr/testify/assert"

"github.com/multiversx/mx-chain-go/config"
"github.com/multiversx/mx-chain-go/dataRetriever"
"github.com/multiversx/mx-chain-go/epochStart"
Expand All @@ -23,7 +25,6 @@ import (
dataRetrieverMock "github.com/multiversx/mx-chain-go/testscommon/dataRetriever"
"github.com/multiversx/mx-chain-go/testscommon/economicsmocks"
"github.com/multiversx/mx-chain-go/testscommon/genesisMocks"
"github.com/stretchr/testify/assert"
)

func createMockStorageEpochStartBootstrapArgs(
Expand Down
19 changes: 11 additions & 8 deletions epochStart/bootstrap/syncEpochStartMeta.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"github.com/multiversx/mx-chain-core-go/data"
"github.com/multiversx/mx-chain-core-go/hashing"
"github.com/multiversx/mx-chain-core-go/marshal"

"github.com/multiversx/mx-chain-go/common"
"github.com/multiversx/mx-chain-go/config"
"github.com/multiversx/mx-chain-go/epochStart"
Expand All @@ -17,6 +18,7 @@ import (
"github.com/multiversx/mx-chain-go/process/interceptors"
interceptorsFactory "github.com/multiversx/mx-chain-go/process/interceptors/factory"
"github.com/multiversx/mx-chain-go/sharding"
"github.com/multiversx/mx-chain-go/storage"
)

var _ epochStart.StartOfEpochMetaSyncer = (*epochStartMetaSyncer)(nil)
Expand Down Expand Up @@ -91,14 +93,15 @@ func NewEpochStartMetaSyncer(args ArgsNewEpochStartMetaSyncer) (*epochStartMetaS

e.singleDataInterceptor, err = interceptors.NewSingleDataInterceptor(
interceptors.ArgSingleDataInterceptor{
Topic: factory.MetachainBlocksTopic,
DataFactory: interceptedMetaHdrDataFactory,
Processor: args.MetaBlockProcessor,
Throttler: disabled.NewThrottler(),
AntifloodHandler: disabled.NewAntiFloodHandler(),
WhiteListRequest: args.WhitelistHandler,
CurrentPeerId: args.Messenger.ID(),
PreferredPeersHolder: disabled.NewPreferredPeersHolder(),
Topic: factory.MetachainBlocksTopic,
DataFactory: interceptedMetaHdrDataFactory,
Processor: args.MetaBlockProcessor,
Throttler: disabled.NewThrottler(),
AntifloodHandler: disabled.NewAntiFloodHandler(),
WhiteListRequest: args.WhitelistHandler,
CurrentPeerId: args.Messenger.ID(),
PreferredPeersHolder: disabled.NewPreferredPeersHolder(),
ProcessedMessagesCacheMap: make(map[string]storage.Cacher),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should not be allocated here, but received through the arguments.

},
)
if err != nil {
Expand Down
8 changes: 8 additions & 0 deletions factory/processing/processComponents.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1501,6 +1501,8 @@ func (pcf *processComponentsFactory) newInterceptorContainerFactory(
nodeOperationMode = common.FullArchiveMode
}

processedMessagesCacheMap := make(map[string]storage.Cacher)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should not be initialized here, but received in the arguments so we can also mock it in tests.


shardCoordinator := pcf.bootstrapComponents.ShardCoordinator()
if shardCoordinator.SelfId() < shardCoordinator.NumberOfShards() {
return pcf.newShardInterceptorContainerFactory(
Expand All @@ -1513,6 +1515,7 @@ func (pcf *processComponentsFactory) newInterceptorContainerFactory(
fullArchivePeerShardMapper,
hardforkTrigger,
nodeOperationMode,
processedMessagesCacheMap,
)
}
if shardCoordinator.SelfId() == core.MetachainShardId {
Expand All @@ -1526,6 +1529,7 @@ func (pcf *processComponentsFactory) newInterceptorContainerFactory(
fullArchivePeerShardMapper,
hardforkTrigger,
nodeOperationMode,
processedMessagesCacheMap,
)
}

Expand Down Expand Up @@ -1665,6 +1669,7 @@ func (pcf *processComponentsFactory) newShardInterceptorContainerFactory(
fullArchivePeerShardMapper *networksharding.PeerShardMapper,
hardforkTrigger factory.HardforkTrigger,
nodeOperationMode common.NodeOperation,
processedMessagesCacheMap map[string]storage.Cacher,
) (process.InterceptorsContainerFactory, process.TimeCacher, error) {
headerBlackList := cache.NewTimeCache(timeSpanForBadHeaders)
shardInterceptorsContainerFactoryArgs := interceptorscontainer.CommonInterceptorsContainerFactoryArgs{
Expand Down Expand Up @@ -1698,6 +1703,7 @@ func (pcf *processComponentsFactory) newShardInterceptorContainerFactory(
FullArchivePeerShardMapper: fullArchivePeerShardMapper,
HardforkTrigger: hardforkTrigger,
NodeOperationMode: nodeOperationMode,
ProcessedMessagesCacheMap: processedMessagesCacheMap,
}

interceptorContainerFactory, err := interceptorscontainer.NewShardInterceptorsContainerFactory(shardInterceptorsContainerFactoryArgs)
Expand All @@ -1718,6 +1724,7 @@ func (pcf *processComponentsFactory) newMetaInterceptorContainerFactory(
fullArchivePeerShardMapper *networksharding.PeerShardMapper,
hardforkTrigger factory.HardforkTrigger,
nodeOperationMode common.NodeOperation,
processedMessageCacheMap map[string]storage.Cacher,
) (process.InterceptorsContainerFactory, process.TimeCacher, error) {
headerBlackList := cache.NewTimeCache(timeSpanForBadHeaders)
metaInterceptorsContainerFactoryArgs := interceptorscontainer.CommonInterceptorsContainerFactoryArgs{
Expand Down Expand Up @@ -1751,6 +1758,7 @@ func (pcf *processComponentsFactory) newMetaInterceptorContainerFactory(
FullArchivePeerShardMapper: fullArchivePeerShardMapper,
HardforkTrigger: hardforkTrigger,
NodeOperationMode: nodeOperationMode,
ProcessedMessagesCacheMap: processedMessageCacheMap,
}

interceptorContainerFactory, err := interceptorscontainer.NewMetaInterceptorsContainerFactory(metaInterceptorsContainerFactoryArgs)
Expand Down
3 changes: 3 additions & 0 deletions process/factory/interceptorscontainer/args.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,15 @@ package interceptorscontainer

import (
crypto "github.com/multiversx/mx-chain-crypto-go"

"github.com/multiversx/mx-chain-go/common"
"github.com/multiversx/mx-chain-go/dataRetriever"
"github.com/multiversx/mx-chain-go/heartbeat"
"github.com/multiversx/mx-chain-go/process"
"github.com/multiversx/mx-chain-go/sharding"
"github.com/multiversx/mx-chain-go/sharding/nodesCoordinator"
"github.com/multiversx/mx-chain-go/state"
"github.com/multiversx/mx-chain-go/storage"
)

// CommonInterceptorsContainerFactoryArgs holds the arguments needed for the metachain/shard interceptors factories
Expand Down Expand Up @@ -43,4 +45,5 @@ type CommonInterceptorsContainerFactoryArgs struct {
FullArchivePeerShardMapper process.PeerShardMapper
HardforkTrigger heartbeat.HardforkTrigger
NodeOperationMode common.NodeOperation
ProcessedMessagesCacheMap map[string]storage.Cacher
}
Loading
Loading