Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] Implement metadata storage in daily directory names and fast path for list target #313

Merged

Conversation

fako1024
Copy link
Collaborator

@fako1024 fako1024 commented May 9, 2024

Features:

  • Directory-global metadata information (everything needed for the list target, both simple and detailed) is encoded via custom string-compression (OS-directory name and length restriction safe, probably would even work under Windows 95 🤣), e.g. (first one without metadata, second one with):
└─ $ ▶ ls -al testdb/eth1/2016/02/
total 0
drwxr-xr-x 1 fako fako 192 Feb 24 14:28 1456358400
drwxr-xr-x 1 fako fako 192 May  9 13:17 1456444800_oX-1-aVB1-8PILs4-AK9rM2-0Gt8-05he
  • 100% backwards-compatible reading (with one small caveat for local queries: the goQuery version has to be at least the same as the goProbe version since the old goQuery version isn't aware of the suffix (and will not ignore it). This will not affect global-query or the API though since that one the DBWorkManager inside goProbe itself to read from the DB and hence will automatically receive valid data no matter what format the respective local DB will be in. So this in reality isn't a restriction, because it's rather unlikely that you update goProbe on your local system but not goQuery...
  • Zero-conversion transition for goProbe: Once updated, the DB writer will simply write the metadata for the current directory - old ones are untouched by default. Manual conversion can be done with the analyze-meta tool for now (not the final solution, but it was a simple addition, so I opted for not writing yet another tool) in case someone wants to benefit from the additional performance for the whole history.
  • Performance for the list target is 10-15 fold, depending on the storage / partition type.
  • Drive-By: Added list target query log in this commit.

Closes #282

@fako1024 fako1024 added feature New feature or request performance Performance / optimization related topics labels May 9, 2024
@fako1024 fako1024 requested a review from els0r May 9, 2024 11:33
@fako1024 fako1024 self-assigned this May 9, 2024
@fako1024 fako1024 linked an issue May 9, 2024 that may be closed by this pull request
2 tasks
Copy link
Owner

@els0r els0r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpicks:

  • viper in function call not part of the cobra command
  • consts

cmd/goQuery/cmd/list.go Outdated Show resolved Hide resolved
cmd/goQuery/cmd/root.go Outdated Show resolved Hide resolved
pkg/goDB/storage/gpfile/gpdir.go Outdated Show resolved Hide resolved
pkg/goDB/storage/gpfile/gpdir.go Outdated Show resolved Hide resolved
pkg/goDB/storage/gpfile/gpfile_test.go Outdated Show resolved Hide resolved
@fako1024 fako1024 merged commit 14f3155 into main May 10, 2024
5 checks passed
@fako1024 fako1024 deleted the 282-goquery-list-target-is-slow-on-sensors-with-many-interfaces branch May 10, 2024 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request performance Performance / optimization related topics
Projects
None yet
Development

Successfully merging this pull request may close these issues.

goQuery 'list' target is slow on sensors with many interfaces
2 participants