Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix corruption checks v3.5 #14282

Merged
merged 29 commits into from
Sep 7, 2022
Merged

Fix corruption checks v3.5 #14282

merged 29 commits into from
Sep 7, 2022

Conversation

serathius
Copy link
Member

@serathius
Copy link
Member Author

serathius commented Jul 27, 2022 •
edited
Loading

Seems like pretty big change for backport (adds 1800 lines), however over 1500 of lines is just tests.

@serathius serathius changed the title Fix checks v3.5 Fix corruption checks v3.5 Jul 27, 2022
@serathius serathius mentioned this pull request Jul 27, 2022
Closed
16 tasks
@ahrtr ahrtr mentioned this pull request Jul 28, 2022
Open
@serathius
Copy link
Member Author

Want to validate Compact performance before merging.

@serathius
server: Extract triggerCorruptAlarm to function
d32de2c 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Extract corruption detection to dedicated struct
35cbdf3 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Cover corruptionMonitor with tests
78a6f38 
Get 100% coverage on InitialCheck and PeriodicCheck functions to avoid
any mistakes.

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Test HashByRev values to make sure they don't change
336fef4 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Extract unsafeHashByRev function
bc592c7 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Move unsafeHashByRev to new hash.go file
3f26995 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Extract kvHash struct
f5ed371 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Refactor hasher
679e327 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Return error from scheduleCompaction
22d3e4e 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Move reading KV index inside scheduleCompaction function
ea684db 
Makes it easier to test hash match between scheduleCompaction and
HashByRev.

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Fix range in mock not returning same number of keys and values
f1a759a 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Calculate hash during compaction
21e5d5d 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Pass revision as int
2b8dd0d 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Move adjusting revision to hasher
991b429 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Store real rv range in hasher
1ff5992 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Return revision range that hash was calcualted for
a3f609d 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Remove duplicated compaction revision
6311072 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Extract hasher to separate interface
7358362 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Cache compaction hash for HashByRev API
1200b10 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
tests: Add integration tests for compact hash
d3db3bc 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
tests: Add tests for HashByRev HTTP API
00bc8da 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
tests: Unify TestCompactionHash and extend it to also Delete keys and…
037a898 
… Defrag

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
tests: Rename corruptHash to CorruptBBolt
a8020a0 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
tests: Move CorruptBBolt to testutil
8d4ca10 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Refactor compaction checker
4a75e3d 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
tests: Cover periodic check in tests
a56ec0b 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Implement compaction hash checking
21fb173 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
server: Make corrtuption check optional and period configurable
5660bf0 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
tests: Fix member id in CORRUPT alarm
2ddb9e0 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
@serathius
Copy link
Member Author

Verified performance and didn't find any significant difference with different configuration of auto-compaction and compact-hash-check.

@serathius serathius merged commit ba52d5a into etcd-io:release-3.5 Sep 7, 2022
@ahrtr
Copy link
Member

ahrtr commented Sep 16, 2022

Fixed #14282 in release-3.5.

It took me a couple of minutes to find this PR, because you got the issue link instead of this PR included in the changelog-3.5

@serathius serathius deleted the fix-checks-v3.5 branch June 15, 2023 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ahrtr ahrtr ahrtr approved these changes

@ptabor ptabor ptabor approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@serathius @ahrtr @ptabor