Implement memlog store operations #19533
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
urso
added
review
libbeat
needs_backport
|
Pinging @elastic/integrations-services (Team:Services) |
botelastic
bot
added
needs_team
urso
commented
Jun 30, 2020
| @@ -54,6 +54,9 @@ type Settings struct { | |||
| // Checkpoint predicate that can trigger a registry file rotation. If not | |||
| // configured, memlog will automatically trigger a checkpoint every 10MB. | |||
| Checkpoint CheckpointPredicate | |||
|
|
|||
| // If set memlog will not check the version of the meta file. | |||
| IgnoreVersionCheck bool | |||
There was a problem hiding this comment.
This is currently required by the filebeat migration (PR not open yet). When migrating the old store we create a temporary store in the same directory and convert all entries before updating the meta-data file (migration supports 3 different formats, which made the process a little hairy).
urso
commented
Jun 30, 2020
go.mod
Outdated
| @@ -163,7 +163,7 @@ require ( | |||
| golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5 | |||
| golang.org/x/text v0.3.2 | |||
| golang.org/x/time v0.0.0-20191024005414-555d28b269f0 | |||
| golang.org/x/tools v0.0.0-20200630154851-b2d8b0336632 | |||
| golang.org/x/tools v0.0.0-20200630223951-c138986dd9b9 | |||
There was a problem hiding this comment.
yay, go modules and version pinning :/
❕ Build Aborted
Expand to view the summary
Build stats
Test stats 🧪
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
zube
bot
added
[zube]: Inbox
[zube]: In Review
and removed
[zube]: In Review
[zube]: Inbox
labels
added 2 commits
July 1, 2020 13:54
Implement store operation that get/set key value pairs from the in-memory store, and log update operations. This change also include unit tests and store compliance tests. The addition of the statestore package is split up into multiple changeset to ease review. The final version of the package can be found [here](https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore). Once finalized, the libbeat/statestore package contains: - The statestore frontend and interface for use within Beats - Interfaces for the store backend - A common set of tests store backends need to support - a storetest package for testing new features that require a store. The testing helpers use map[string]interface{} that can be initialized or queried after the test run for validation purposes. - The default memlog backend + tests
faec
approved these changes
Jul 1, 2020
| @@ -110,7 +113,7 @@ func (r *Registry) Access(name string) (backend.Store, error) { | |||
| home := filepath.Join(r.settings.Root, name) | |||
| fileMode := r.settings.FileMode | |||
| bufSz := r.settings.BufferSize | |||
| store, err := openStore(logger, home, fileMode, bufSz, r.settings.Checkpoint) | |||
| store, err := openStore(logger, home, fileMode, bufSz, !r.settings.IgnoreVersionCheck, r.settings.Checkpoint) | |||
There was a problem hiding this comment.
Any reason for the parity flip between Settings and the function parameter? All else being equal I'd prefer if the parameter was ignoreVersionCheck also instead of mustCheckMeta (or at least that they were both the same)
There was a problem hiding this comment.
Bad habits :)
Update to ignoreVersionCheck.
| t.Fatal(err) | ||
| } | ||
|
|
||
| cases := list[:0] |
There was a problem hiding this comment.
I'm not sure I understand this construction -- this should be equivalent to cases := []os.FileInfo{}, right? Is this a type inference shortcut in the spirit of c++ auto, or something else?
There was a problem hiding this comment.
this should be equivalent to cases := []os.FileInfo{}, right?
kinda. As we have slices the construct creates a slice of length 0, but with the old capacity.
It's an in-place filtering loop for removing files from the list (e.g. the Readme.md). See: https://github.com/golang/go/wiki/SliceTricks#filtering-without-allocating
But yeah, this is me being lazy :)
| return nil, err | ||
| } | ||
| } else if !fi.Mode().IsDir() { | ||
| return nil, fmt.Errorf("'%v' is no directory", home) |
| // Error indicates the log file was incomplete or corrupted. | ||
| // Anyways, we already have the table in a valid state and will | ||
| // continue opening the store from here. | ||
| logp.Warn("Incomplete or corrupted log file in %v. Continue with last known complete and consistend state. Reason: %v", home, err) |
|
Filebeat tests are all green. CI failed notifying github |
v1v
added a commit
to v1v/beats
that referenced
this pull request
Jul 3, 2020
…ne-beats * upstream/master: (35 commits) [ci] fix env variable name for xpack filebeats (elastic#19617) Cache error responses for cloudfoundry apps metadata (elastic#19181) ci: user fixed type of agent (elastic#19625) Input v2 cursor testing (elastic#19573) Update Jenkinsfile to not inspect removed vendor (elastic#19610) Fix ordering and duplicate configs on autodiscover (elastic#19317) Prepare input/file for changes in the registrar (elastic#19516) Cursor input and manager implementation (elastic#19571) [Filebeat] Fix tls mapping in suricata module (elastic#19494) [Ingest Manager] Make Agent beta and Constraints experimental (elastic#19586) Accept prefix as metric_types for stackdriver metricset in GCP (elastic#19345) Implement memlog store operations (elastic#19533) introduce journalbeat/pkg in order to provide reusable shared code (elastic#19581) Add descriptions to HAProxy fields in Metricbeat (elastic#19561) ci: apm-server-update trigered only on upstream, comments, and manual triggered (elastic#19590) ci: enable upstream triggering on the packaging job (elastic#19589) ci: some jjbb improvements (elastic#19588) [MetricBeat] set tags correctly if the dimension value is ARN (elastic#19433) [Filebeat] Add default_fields: false to fields.yml in aws module (elastic#19568) Add publisher implementation for stateful inputs (elastic#19530) ...
2 tasks
urso
pushed a commit
to urso/beats
that referenced
this pull request
Jul 7, 2020
Implement store operation that get/set key value pairs from the in-memory store, and log update operations. This change also include unit tests and store compliance tests. The addition of the statestore package is split up into multiple changeset to ease review. The final version of the package can be found [here](https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore). Once finalized, the libbeat/statestore package contains: - The statestore frontend and interface for use within Beats - Interfaces for the store backend - A common set of tests store backends need to support - a storetest package for testing new features that require a store. The testing helpers use map[string]interface{} that can be initialized or queried after the test run for validation purposes. - The default memlog backend + tests
urso
pushed a commit
that referenced
this pull request
Jul 8, 2020
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
Implement store operation that get/set key value pairs from the in-memory store, and log update operations. This change also include unit tests and store compliance tests. The addition of the statestore package is split up into multiple changeset to ease review. The final version of the package can be found [here](https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore). Once finalized, the libbeat/statestore package contains: - The statestore frontend and interface for use within Beats - Interfaces for the store backend - A common set of tests store backends need to support - a storetest package for testing new features that require a store. The testing helpers use map[string]interface{} that can be initialized or queried after the test run for validation purposes. - The default memlog backend + tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Implement store operation that get/set key value pairs from
the in-memory store, and log update operations.
This change also include unit tests and store compliance tests.
The addition of the statestore package is split up into multiple
changeset to ease review. The final version of the package can be found
here.
Once finalized, the libbeat/statestore package contains:
testing helpers use map[string]interface{} that can be initialized or
queried after the test run for validation purposes.
Why is it important?
The statestore introduces a simple key-value store to Beats. The statestore will be used to replace the registry in filebeat in the future.
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added an entry inCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Author's Checklist
How to test this PR locally
go testRelated issues