Configurable size of validate queue #255
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
validateWorker() reads from validateQ and invokes validate function that performs validation of the message. Signature validation is performed synchronously. The number of validate workers defaults to the number of CPUs and can be updated with WithValidateWorkers function. With no additional user validators, signature validation is the bottleneck when receiving new messages. Increasing the number of validating workers does not help given the context switching and bottleneck nature of this spot. As stated in WithValidateWorkers documentation, this function should be used rather to limit the number of workers to devote less CPU time for synchronous validation. On the other hand, with the default size of `validateQ`, some applications built on a top of libp2p may experience throttled validation and lost messages. This problem is addressed by WithValidateQueueSize allowing to configure the buffer size for synchronous validation. Application developers knowing the nature of their protocols can set this value to minimise the possibility of throttled synchronous validation and dropped messages. Configurable buffer size allows to gracefully handle peaks of messages and, from the other side, the number of concurrent synchronous workers is still limited by validateWorkers property so the receiver should not get congested.
|
Can you |
|
Ready! I see the build is failing but I think it could be some timeout. Unfortunately, I don't see an option to rerun it. |
pdyraga
added a commit
to keep-network/keep-core
that referenced
this pull request
Jan 31, 2020
We had to switch to our fork because of the PR we submitted to libp2p allowing to configure size of validation queue with WithValidateQueueSize function. libp2p/go-libp2p-pubsub#255 This PR is merged so we can go back to the libp2p/go-libp2p-pubsub. Since there was no official release from the time our PR was merged, we are attached to a commit in go-libp2p-pubsub master.
lukasz-zimnoch
added a commit
to keep-network/keep-core
that referenced
this pull request
Jan 31, 2020
Switch back to libp2p/go-libp2p-pubsub We had to switch to our fork because of the PR we submitted to libp2p allowing to configure size of validation queue with WithValidateQueueSize function: libp2p/go-libp2p-pubsub#255 This PR is merged so we can go back to the libp2p/go-libp2p-pubsub. Since there was no official release from the time our PR was merged, we are attached to a commit in go-libp2p-pubsub master.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
validateWorker()reads fromvalidateQand invokesvalidatefunction that performs validation of the message. Signature validation is performed synchronously. The number of validate workers defaults to the number of CPUs and can be updated withWithValidateWorkersfunction. With no additional user validators, signature validation is the bottleneck when receiving new messages.Increasing the number of validating workers does not help given the context switching and bottleneck nature of this spot. As stated in
WithValidateWorkersdocumentation, this function should be used rather to limit the number of workers to devote less CPU time for synchronous validation. On the other hand, with the default size ofvalidateQ, some applications built on a top of libp2p may experience throttled validation and lost messages.This problem is addressed in this PR with
WithValidateQueueSizeallowing to configure the buffer size for synchronous validation. Application developers knowing the nature of their protocols can set this value to minimise the possibility of throttled synchronous validation and dropped messages. Configurable buffer size allows to gracefully handle peaks of messages and, from the other side, the number of concurrent synchronous workers is still limited byvalidateWorkersproperty so the receiver should not get congested.