storage: avoid per-kv allocations during consistency checks #29419
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I noticed in an `alloc_objects` heap profile on a 3-node cluster restoring tpcc that more than 46% of all allocations were made in `computeChecksumPostApply`. Specifically, these allocations were all made in `Replica.sha512`. 28% of allocations were due to protobuf marshaling of `hlc.LegacyTimestamp`. The other 18% was in `encoding/binary.Write`. This removes both of these sources of per-key allocations. The first allocation was avoided by sharing a byte buffer across protobuf marshals. The second was avoided by removing the call to `binary.Write` (see golang/go#27403). I confirmed that this is no longer an issue by looking at heap profiles from before and after in a test that performed a consistency check. I plan to follow up on golang/go#27403 and search for any other offenders in our codebase. I already see a few. Release note: None
|
This change is |
|
|
petermattis
approved these changes
Aug 31, 2018
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (and 1 stale)
|
Excellent! I've noticed these too - glad you figured out how to zap them. |
|
TFTRs. bors r+ |
craig bot
pushed a commit
that referenced
this pull request
Aug 31, 2018
29419: storage: avoid per-kv allocations during consistency checks r=nvanbenschoten a=nvanbenschoten I noticed in an `alloc_objects` heap profile on a 3-node cluster restoring tpcc that more than 46% of all allocations were made in `computeChecksumPostApply`. Specifically, these allocations were all made in `Replica.sha512`. 28% of allocations were due to protobuf marshaling of `hlc.LegacyTimestamp`. The other 18% was in `encoding/binary.Write`. This removes both of these sources of per-key allocations. The first allocation was avoided by sharing a byte buffer across protobuf marshals. The second was avoided by removing the call to `binary.Write` (see golang/go#27403). I confirmed that this is no longer an issue by looking at heap profiles from before and after in a test that performed a consistency check. I plan to follow up on golang/go#27403 and search for any other offenders in our codebase. I already see a few. Release note: None Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Build succeeded |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I noticed in an
alloc_objectsheap profile on a 3-node cluster restoring tpcc that more than 46% of all allocations were made incomputeChecksumPostApply. Specifically, these allocations were all made inReplica.sha512. 28% of allocations were due to protobuf marshaling ofhlc.LegacyTimestamp.The other 18% was in
encoding/binary.Write.This removes both of these sources of per-key allocations. The first allocation was avoided by sharing a byte buffer across protobuf marshals. The second was avoided by removing the call to
binary.Write(see golang/go#27403). I confirmed that this is no longer an issue by looking at heap profiles from before and after in a test that performed a consistency check.I plan to follow up on golang/go#27403 and search for any other offenders in our codebase. I already see a few.
Release note: None