Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #2565 to 7.8: Fix haproxy index template #18484

Closed
wants to merge 1,115 commits into from

Conversation

nchaulet
Copy link
Member

Cherry-pick of PR #2565 to 7.8 branch. Original message:

  • Add test case for haproxy schema.
  • Remove haproxy stats_addr option because hosts is the standard option
    used across modules.

Closes #2561 #2563

kuisathaverat and others added 30 commits April 13, 2020 10:37
* feat: packaging pipeline

* chore: refactor
* init commit, try to add asciidoc with link

* fix misc errors

* add extra newline to fix asciidoc

Co-Authored-By: DeDe Morton <dede.morton@elastic.co>

* Fix conditional section

Co-authored-by: DeDe Morton <dede.morton@elastic.co>
* fix: login into the docker registry

* test: make a pull after login to test

* docs: add note to the workaround
* Add static mapping for dynamodb metricset

Signed-off-by: chrismark <chrismarkou92@gmail.com>

* Fix field types

Signed-off-by: chrismark <chrismarkou92@gmail.com>
…7694)

We should always include these optional config values, as not using them
creates continuity issues in the Uptime app. The `id` attribute is the
only way we can consistently identify a monitor over time when its
config changes, and the `name` attribute is used for display purposes in
the Uptime app.

This change makes our docs examples reflect best practices.
This adds the aarch64 (arm64) architecture to the default list Linux package targets.

This will add three new artifacts to each beat project. For example:

- filebeat-$version-arm64.deb
- filebeat-$version-aarch64.rpm
- filebeat-$version-linux-arm64.tar.gz

I had to modify dockerlogbeat to honor the PLATFORMS selector. It was always trying
to build its linux/amd64 docker image even if that platform was not selected.
This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related elastic#16907 elastic#17682
* add support for role arn in aws config
* Reduce dependencies in Crawler

The crawler creates active inputs for static configuration, starts
config file reloading, and starts the module loader.
With this change the crawler has no direct dependency (well, reduced) on
input.Input anymore, but will use the `Runner` interface, even for
statically configured inputs.
This also reduces dependencies, as most plumbing is already done by the
inputs.RunnerFactory and must not be duplicated by the crawler anymore.

The input.Runner used to compute a 'ID' by hashing the inputs
configuration. The ID was public, to be used by the crawler only.
Instead of having the input compute the ID, it is the crawler who will
compute the input ID now.

Note: the cfgfile RunnerList maintains its own set of IDs. The crawler
and RunnerList each used to use the ID to check for 'duplicate'
configurations, but because the IDs are not 'shred' duplication
detection is not across the Beat.
ID detection is actively used by input config file reloading and auto
discvovery only, in order to check if an input still needs to be
running, are shall shut down.

* fix import formatting in crawler

* Update log message test looks for
* feat: filter changes using go list output

* fix: auditbeat trigger
…lastic#17550)

* Improve ECS categorization field mappings for nats module

- event.kind
- event.type
- related.ip

Closes elastic#16173
* Handle ECS-compatible server logs emitted by ES 8.0.0+

* Adding CHANGELOG entry

* Adding a couple more log entries
Product name in module should be CrowdStrike instead of Crowdstrike.
…ring (elastic#17609)

* Add ability to reconfigure a module

* Reconfigure Logstash module with required metricsets for xpack.enabled

* Replace assert with require

* Adding CHANGELOG entry

* Update default configuration files

* Auto-configure beat module metricsets when xpack.enabled = true

* Refactoring common code into helper function

* Adding tests for ReConfigure() / making it part of Module interface

* Moving comments

* Fixing infinite recursion 🤦

* Implement for kibana module

* Implementing for elasticsearch module

* Moving ReConfigure method to BaseModule from Module

* Fixing test function name

* Use errors.Wrapf

* Logging config change

* Adding comment about intent of use.

* s/ReConfigure/Reconfigure/

* Don't pass registry

* Return copy of reconfigured module

* Updating module docs to clarify auto-configuration

* Fixing test

* Trying out docs for `beat` module

* Fixing tests

* Adding tests for ReConfigure() / making it part of Module interface

* Moving comments

* Fixing infinite recursion 🤦

* Implement for kibana module

* Implementing for elasticsearch module

* Moving ReConfigure method to BaseModule from Module

* Logging config change

* Return copy of reconfigured module

* Updating module docs to clarify auto-configuration

* Fixing test

* Trying out docs for `beat` module

* Update metricbeat/docs/modules/beat.asciidoc

Co-Authored-By: DeDe Morton <dede.morton@elastic.co>

* Uppercasing start of log message

* Updating all stack modules' docs

* Reodering imports

* Fixing rebase error

Co-authored-by: DeDe Morton <dede.morton@elastic.co>
* Adding ECS-compatible sample slowlogs

* Handle ECS-compatible slowlogs emitted by ES 8.0.0+

* Adding CHANGELOG entry
* fix: disable workaround on macos

* Update Jenkinsfile

Co-Authored-By: Victor Martinez <victormartinezrubio@gmail.com>

* Update Jenkinsfile

Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
[Agent] added test which covers fleet to router (elastic#17746)
Fix some overflows on Prometheus histogram rate calculations.
They could be caused by:
* New buckets added to existing histograms on runtime, this happens at
  least with CockroachDB (see elastic#17736).
* Buckets with bigger upper limits have lower counters. This is wrong and
  has been only reproduced this on tests, but handling it just in case to avoid
  losing other data if this happens with some service.

Rate calculation methods return now also a boolean to be able to differenciate
if a zero value is caused because it was the first call, or because it the rate is
actually zero.
Add validation to histogram subfields in a way that they don't need to
be added to the mappings, but they can still be checked. This is helpful
to avoid having to add exceptions to all the modules based on Prometheus
when Elasticsearch types are used.
…17728)

* Adding sample logs

* Handle ECS-compatible deprecation logs emitted by ES 8.0.0+

* Adding CHANGELOG entry
jsoriano and others added 21 commits May 7, 2020 22:39
…d to a GH issue (elastic#18390)

* chore: add a bot comment whenever the "request-discuss" label is added to a GH issue

The comment will be removed if the label is removed.

* fix: rename bot descriptor
…8366)

* Remove cmd/elastic-agent/elastic-agent.go as its not used.

* Remove the usage of cmd/elastic-agent.
* install jq in separate steps

* changelog

* move up

* comment

* embarrasing commit

* tidy

* Update dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl

Co-authored-by: Blake Rouse <blake.rouse@elastic.co>

Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
…cloud (elastic#18398)

* remove validate region/zone

* add stackdriver into googlecloud default config
…lastic#18361)

* handle errors

* update changelog

* fix tests

* address review
…astic#18393)

* [Libbeat] Add more complete tests for opt parameters in ES output

This PR add additionnal test over the usager of the `parameters` options
in the Elasticsearch output:

- When preconfigured params are set without local params
- When preconfigured params are set with local params
- When no preconfigured params are configured but local are.
- When no preconfigured or local params are set.

The merge is also done close to the actual calls and will not be
executed if anything fails before the bulk request.

The test assertion is now more solid and take into consideration any
errors.

See discussion in elastic#18318 and elastic#18326
agent.hostname is not part of ECS so this removes the field. Instead the
agent.name field can be used for this purpose and it it part of ECS.
This adds an alias pointing from agent.hostname to agent.name.

Closes elastic#16377
Sometimes the DNS IP addresses from Sysmon in `winlog.event_data.QueryResults` are truncated.
The leads to mapping exceptions since the value is not of type `ip` in Elasticsearch.

To fix this the module will now filter any results that are not valid IP addresses.

Fixes elastic#18432
The shared pipeline for Cisco ASA and FTD creates temporary fields under
the _temp_ object. If a failure happens in the middle of the pipeline,
all those fields would be indexed, causing the index mapping to grow too
big.

Fixes elastic#18391
@nchaulet nchaulet requested review from a team as code owners May 13, 2020 16:22
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 13, 2020
@botelastic
Copy link

botelastic bot commented May 13, 2020

This issue doesn't have a Team:<team> label.

@nchaulet nchaulet closed this May 13, 2020
@nchaulet nchaulet deleted the backport_2565_7.8 branch May 13, 2020 16:23
@nchaulet nchaulet restored the backport_2565_7.8 branch May 13, 2020 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport needs_team Indicates that the issue/PR needs a Team:* label review
Projects
None yet
Development

Successfully merging this pull request may close these issues.