Add support for --format flag for 'keys' #306
Conversation
|
@ptabor could you please rebase this PR? |
|
[scheduling this on my todo list] |
ptabor
force-pushed
the
20211220-bbolt_cmd_format
branch
2 times, most recently
from
2a359bc
to
3da2606
Compare
This allows to print keys as 'hex', especially useful in context of etcd revisions: ./bbolt keys --format=hex ./foo/db key Signed-off-by: Piotr Tabor <ptab@google.com>
Signed-off-by: Piotr Tabor <ptab@google.com>
Signed-off-by: Piotr Tabor <ptab@google.com>
ptabor
force-pushed
the
20211220-bbolt_cmd_format
branch
from
3da2606
to
5495c63
Compare
|
A quick comment: the test is broken https://github.com/etcd-io/bbolt/actions/runs/3643030691/jobs/6150802557 |
|
@serathius FYI |
cmd/bbolt/main.go
Outdated
| overflowN := p.overflow | ||
| if overflowN > 100*1024 { |
There was a problem hiding this comment.
It means bbolt doesn't support saving object > 400MB (assuming pageSize is 4K)?
There was a problem hiding this comment.
Addresses in the other PR: #354
db.go
Outdated
| const magic uint32 = 0xED0CDAED | ||
| const Magic uint32 = 0xED0CDAED |
There was a problem hiding this comment.
The Magic should be invisible to users/client applications.
There was a problem hiding this comment.
Addressed in the other PR: #354
In general we are missing an internal library that bbolt cli can use and share with the core implementation instead of having copy-paste of the data-structures and constants.
cmd/bbolt/main.go
Outdated
| if m.magic != bolt.Magic { | ||
| return 0, fmt.Errorf("the page has wrong (unexpected) magic") | ||
| } |
There was a problem hiding this comment.
Since the command is supposed to be executed by users. They need to use check command to validate the DB. In other words, even the db is corrupted, they should also run some command, such as dumping page items.
| if m.magic != bolt.Magic { | |
| return 0, fmt.Errorf("the page has wrong (unexpected) magic") | |
| } |
There was a problem hiding this comment.
In #354.
We need safe-guards even with bbolt tool. Allocating 1TBs of memory becouse it consider random garbage from meta-page as block-size is not good and risk --> especially if diagnostic needs to be performed within production environment in order to avoid moving customer's data.
If this would become a real problem -> we would need a 'force' flag to ignore such safe-guards.
|
This PR mentions |
|
Marking as draft. I used this branch to keep some changes to diagnose data corruption, and the last commit incidentally reverted the main changes. Some of the changes will go to a separate branch. |
ptabor
force-pushed
the
20211220-bbolt_cmd_format
branch
from
927b99a
to
b8c67fd
Compare
|
I brought back original scope of this request. Moving the discussion about 'Magic' to a separate PR i will follow with. |
| if val == nil { | ||
| return ErrKeyNotFound | ||
| return fmt.Errorf("Error %w for key: %q hex: \"%x\"", ErrKeyNotFound, key, string(key)) |
There was a problem hiding this comment.
Suggest to output the key in exactly the same format as inputted by user.
There was a problem hiding this comment.
I wanted to provide very explicit feedback. In particular usually we (etcd) want to pass \x00\x00... starting strings... and char-0 does not passes well through linux's vexec command so the strings are truncated.
Both format's allow to easily cross check against bbolt page db 5
$ echo $'aaa\x00aaa' | hexdump
0000000 6161 0a61
0000004
| var parseFormat string | ||
| var format string | ||
| fs.StringVar(&parseFormat, "parse-format", "ascii-encoded", "Output format. One of: ascii-encoded|hex") | ||
| fs.StringVar(&format, "format", "bytes", "Output format. One of: ascii-encoded|hex|bytes (default: bytes)") |
There was a problem hiding this comment.
Please also update (cmd *GetCommand) Usage()
Signed-off-by: Piotr Tabor <ptab@google.com>
ptabor
force-pushed
the
20211220-bbolt_cmd_format
branch
from
b8c67fd
to
4acf9d9
Compare
This allows to print keys as 'hex', especially useful in context of etcd revisions:
./bbolt keys --format=hex ./foo/db key