[Filebeat][httpjson] Make httpjson use cursor input when using date cursor #20751
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
marc-gr
added
enhancement
in progress
|
Pinging @elastic/siem (Team:SIEM) |
botelastic
bot
added
needs_team
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
marc-gr
force-pushed
the
httpjson_cursor
branch
2 times, most recently
from
aea7847 to
c0560a3
Compare
marc-gr
force-pushed
the
httpjson_cursor
branch
from
c0560a3 to
b9c5f10
Compare
marc-gr
added
review
and removed
in progress
marc-gr
force-pushed
the
httpjson_cursor
branch
from
b9c5f10 to
60a2d38
Compare
urso
reviewed
Sep 14, 2020
urso
reviewed
Sep 14, 2020
| @@ -67,7 +67,8 @@ func (o *OAuth2) IsEnabled() bool { | |||
|
|
|||
| // Client wraps the given http.Client and returns a new one that will use the oauth authentication. | |||
| func (o *OAuth2) Client(ctx context.Context, client *http.Client) (*http.Client, error) { | |||
There was a problem hiding this comment.
Where do we need this type? If the implementation is internal to the httpjson package, why not keep it in the httpjson package and unexport it (implementation detail). If this type (and corresponding types/values) are more generic, consider to move it to the libbeat/common/transport package (or a sub-package).
urso
reviewed
Sep 14, 2020
urso
reviewed
Sep 14, 2020
| @@ -23,10 +25,10 @@ type dateCursor struct { | |||
| dateFormat string | |||
|
|
|||
| value string | |||
| valueTpl *Template | |||
| valueTpl *template.Template | |||
There was a problem hiding this comment.
this type mixes the computation/extraction of the cursor and the actual data. In case we want to store (or pass around) the data, consider to split the actual data (in the case 'value'), such that the 'computation' becomes immutable.
Edit: looks like cursorState is already the state type we want to have separated.
marc-gr
force-pushed
the
httpjson_cursor
branch
3 times, most recently
from
5a66004 to
113f3e3
Compare
marc-gr
force-pushed
the
httpjson_cursor
branch
from
113f3e3 to
dbac6cc
Compare
marc-gr
force-pushed
the
httpjson_cursor
branch
from
dbac6cc to
ba92901
Compare
adriansr
approved these changes
Sep 29, 2020
4 tasks
v1v
added a commit
to v1v/beats
that referenced
this pull request
Sep 30, 2020
…ci-build-label-support * upstream/master: [JJBB] Set shallow cloning to 10 (elastic#21409) docs: add link to release notes for 7.9.2 (elastic#21405) (elastic#21419) docs: Prepare Changelog for 7.9.2 (elastic#21229) (elastic#21403) fix: mark flaky tests (elastic#21300) fix: use a fixed version of setuptools (elastic#21393) Move Kubernetes events metricset to its own block in reference config (elastic#21407) [libbeat] Enable WriteAheadLimit in the disk queue (elastic#21391) docs: fix apt/yum formatting (elastic#21362) Fix shutdown tracking in s3 input (elastic#21380) [libbeat] Fix position writing in the disk queue Add UBI 8 image to the dependencies report (elastic#21374) Fix debug message to show actual SQS message ID (elastic#20614) [Elastic Agent] Rename *ConfigChange to PolicyChange (elastic#20779) [Elastic Agent] Add install/uninstall sub-command (elastic#21206) [Filebeat][httpjson] Make httpjson use cursor input when using date cursor (elastic#20751) feat: prepare release pipelines (elastic#21238) Add IP validation to Security module (elastic#21325)
marc-gr
added a commit
to marc-gr/beats
that referenced
this pull request
Oct 1, 2020
…ursor (elastic#20751) * Fix duplicate import * Move config to its own package * Minor improvements * Fix tests * Create input manager * Change requester to accept and store a cursor * Modify input to be embedded * Create stateless and cursor inputs * Initialize new input manager on publish * Add changelog entry and format files * Move test data folder * Change tests * Apply requested changes (cherry picked from commit 8f9d54b)
marc-gr
added a commit
that referenced
this pull request
Oct 2, 2020
…ursor (#20751) (#21384) * Fix duplicate import * Move config to its own package * Minor improvements * Fix tests * Create input manager * Change requester to accept and store a cursor * Modify input to be embedded * Create stateless and cursor inputs * Initialize new input manager on publish * Add changelog entry and format files * Move test data folder * Change tests * Apply requested changes (cherry picked from commit 8f9d54b)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Creates a custom input manager that will initialize a cursor input manager whenever a date cursor is configured.
Why is it important?
It is a recurrent requirement to keep state between beat restarts for the httpjson input. With this change, whenever a cursor is set up, the state will be kept.
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Related issues
Relates to #19071 #19486