Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support debug dumping of WAL entries #26

Merged
merged 3 commits into from
Mar 31, 2023
Merged

Support debug dumping of WAL entries #26

merged 3 commits into from
Mar 31, 2023

Conversation

banks
Copy link
Member

@banks banks commented Mar 28, 2023

This PR adds library support for dumping WAL segment contents without full metadata. It includes a simple CLI example of how it can be used.

Why

This is a debugging tool. One issue with BoltDB-based log stores (and the Meta store in this library) is that the writing program holds an exclusive lock on the file while it is open meaning no other tool can read logs on disk while the writing process is still running.

While not a substitute for more fully integrated observability options in the product itself. These changes provide a simple and effective way for operators (or developers) to peek at what is inside WAL logs, even while they are still being written.

It sacrifices efficiency (it always scans whole segments since we don't know where the index is or if there is one) and perfect consistency (concurrent truncations could cause arbitrary inconsistency in the output including missed logs, deleted logs showing, out-of-order logs and more) but provides a quick and simple way to view log contents for manual inspection.

What

Mostly this is a code change in the segment package which deals with the encoded format of the segment files. A new method is added to allow dumping file contents even if the metadata is not known.

This shares most of it's code path with the tail recovery code which is refactored into a shared helper.

This is a no-op change as far as Raft usage goes - the code path is changed but behaviour is not and tests all validate that.

There is a simple cmd package added which provides a simple CLI tool that can be used to inspect WAL log files.

@banks banks marked this pull request as ready for review March 28, 2023 13:24
@banks banks requested a review from jmurret March 28, 2023 14:36
@banks
Copy link
Member Author

banks commented Mar 29, 2023

I ran all 4 workloads of the ALICE tests against this branch locally as an additional check that the refactor didn't change anything in the core workflow and all pass with no detected issues still.

  • ALICE tests ✅

Copy link
Member

@jmurret jmurret left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work. I trust the test coverage and did not see any nits or anything related to the functionality or doc strings. 🎉

@banks banks merged commit 742eb42 into main Mar 31, 2023
@banks banks deleted the waldump branch March 31, 2023 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants