Support debug dumping of WAL entries #26
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds library support for dumping WAL segment contents without full metadata. It includes a simple CLI example of how it can be used.
Why
This is a debugging tool. One issue with BoltDB-based log stores (and the Meta store in this library) is that the writing program holds an exclusive lock on the file while it is open meaning no other tool can read logs on disk while the writing process is still running.
While not a substitute for more fully integrated observability options in the product itself. These changes provide a simple and effective way for operators (or developers) to peek at what is inside WAL logs, even while they are still being written.
It sacrifices efficiency (it always scans whole segments since we don't know where the index is or if there is one) and perfect consistency (concurrent truncations could cause arbitrary inconsistency in the output including missed logs, deleted logs showing, out-of-order logs and more) but provides a quick and simple way to view log contents for manual inspection.
What
Mostly this is a code change in the
segment
package which deals with the encoded format of the segment files. A new method is added to allow dumping file contents even if the metadata is not known.This shares most of it's code path with the tail recovery code which is refactored into a shared helper.
This is a no-op change as far as Raft usage goes - the code path is changed but behaviour is not and tests all validate that.
There is a simple
cmd
package added which provides a simple CLI tool that can be used to inspect WAL log files.