Automatic merge from master to 7.x branch #12224
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…2033) * Remove convertPeriodToDuration and use duration directly in GetStartTimeEndTime * Pass period in time.Duration type
* Fix memory leak in Filebeat pipeline acker Before this change acker goroutine was kept forever as processed events count was not correctly updated. Filebeat sends an empty event to update file states, this event is not published, but treated as dropped, without updating counters. This change makes sures that `a.events` count gets updated for dropped events also, so the acker gets closed after all ACKs happen.
The `convert` processor converts a field in the event to a different type, such as converting a string to an integer. For a full description of the processor's capabilities see the included documentation. Closes elastic#8124
* Update docs.asciidoc Added in a short note on a few of the other configurable variables. * Make update
…lastic#12086) On metricbeat, when the host autodiscover hint is used, and it includes the port, one of the exposed ports has to match with the one in the hint. If not, no configuration should be generated. If it is generated, it will have empty hosts, what would lead to unexpected errors as the seen in elastic#8264.
Rewrite test for more clarity ensuring that the server and listeners are stopped before the test function returns (cause for the case condition). Each test has it's very own socks5 server now. Tests not requiring a proxy don't spin up one.
Refactor and add more tests related to setting up template and ilm policy. Ensure template and ilm-policy is exported despite enabled=false setting.
The `login` dataset is not using the previous file offset when reading new entries in a utmp file. As a result, whenever a new login event occurs, all records are re-read. Also expands the documentation, moves test files to testdata/, and adds a test case that adds a utmp record to the test file and re-reads it to make sure this bug does not happen again.
…12094) This updates github.com/lib/pq to the latest version to fix a socket leak when SSL is not enabled on the PostgreSQL server. Fixes elastic#11393
* Change image references to use block syntax not inline * Fix format for inline links
Ensures behavior on running the `export` cmd does not change compared to last released version.
So far expected files in Filebeat tests were only generated and compared when a file exists. This changes to create a generated for all example logs. This will add a few more files to the repository but I think there the benefits outweight the costs as it means the modules are tested in more detail. Also minor changes will be detected easier.
elastic#11924 duplicated some code from go-sysinfo that is affected by a memory leak (fixed in elastic#12100) In this case, only master is affected as the PR that introduced the leaky gosigar wasn't backported.
A function in go-sysinfo used under Windows to split a command-line into arguments was leaking memory. This code was used in various places: add_process_metadata processor. (enabled by default in all Beats) Packetbeat's process monitor. (disabled by default). Auditbeat's system/process metricset. This PR updates vendored go-sysinfo to the most recent version, which fixes the leak.
This adds a new processor, extract_array, that allows accessing values inside arrays and copying them to target fields.
The setup command until now contained all the possible options from the other Beats. As Heartbeat does not ship anymore with dashboards, the --dashboards command is not needed anymore and is only confusing. I also removed all the other commands except `--ilm-policy` and `--template`. I'm not aware that `--pipelines` or `--machine-learning` would be used.
Here the comparison between `./heartbeat setup -h` from before and after.
Before:
```
This command does initial setup of the environment:
* Index mapping template in Elasticsearch to ensure fields are mapped.
* Kibana dashboards (where available).
* ML jobs (where available).
* Ingest pipelines (where available).
* ILM policy (for Elasticsearch 6.5 and newer).
Usage:
heartbeat setup [flags]
Flags:
--dashboards Setup dashboards
-h, --help help for setup
--ilm-policy Setup ILM policy
--machine-learning Setup machine learning job configurations
--pipelines Setup Ingest pipelines
--template Setup index template
```
After:
```
This command does initial setup of the environment:
* Index mapping template in Elasticsearch to ensure fields are mapped.
* ILM Policy
Usage:
heartbeat setup [flags]
Flags:
-h, --help help for setup
--ilm-policy Setup ILM policy
--template Setup index template
```
In this PR I did not include a check for the config option `setup.dashboards` to make sure they are not there like apm-server does (https://github.com/elastic/apm-server/blob/2baefab778fdfe70c47bc2fb488677b2e43e4635/beater/beater.go#L60) as I don't think it's necessary.
Changes the jenkins_ci.ps1 script to skip testing when magefile.go does not exist. This will allow us to add projects like x-pack/winlogbeat to the test matrix because not all branches have an x-pack/winlogbeat/magefile.go file.
…2125) Outlets are created during log input initialization, and if it fails they were never freed. Handle this case.
…from Docker (elastic#12039) Update instructions for system/socket metricset on Docker. And base permission checks on capabilities rather than on the effective uid. Running a process as root doesn't mean that it has all privileges, specially when run as container.
* Change type from scaled_float to long and add format
…#12044) These tests set up server listeners and create libbeat clients with varying proxy settings, and verify that the clients ping the correct target URL. This is a preparation for elastic#11713, since most of the logic (and work) is in testing the proxy settings; the much simpler PR adding the proxy-disable flag will be a followup to this one, to keep the functional changes isolated in case of rollbacks etc.
…c#12004) * manage leader metricset so that followers don't report errors nor events * add debug message when skipping leader events from non leader members
* Add package libbeat/common/cleanup
The cleanup package adds helpers for deferred optional cleanup on
errors.
For example:
```
ok := False
defer cleanup.IfNot(&ok, func() { ... })
// continue initialization
ok = True
return // some value
```
* Add changelog entry
…ic#12089) * schema conv option for ignore all errors Co-Authored-By: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Add `container` input, deprecate `docker` in favor of it
This change adds a new container input for better support of CRI based
scenarios.
`docker` input was acting as a catch all input for all container related
cases, but its config options were very opinionated towards Docker, with
some issues:
* `containers.ids` setting was good to abstract logs path, but we have
seen many cases were logs are not under default location, or follow a
different path pattern (ie CRI logs).
* `containers.*` settings have shown counter intuitive for many users,
in many cases we have seen people writing `container.*` instead, ending
up in a config error.
* Some existing settings (`combine_partials`, `cri.parse_flags`) were
introduced as a way to offer a backwards compatible upgrades, but it
doesn't really make sense to disable them, as they handle actual
format behaviors.
This new `container` input offers the same wrapper to read log files
from containers with the following changes:
* It exposes `paths` as the `log` input, instead of `containers.ids`
and `containers.path`.
* `parse_flags` and `combine_partials` are hardcoded, as there is no
good reason to disable them.
* `stream` selector is still available, under root settings.
* It allows to select the log format (also atodetect it), giving room
for future format changes. `format` can be `auto` (default), `docker`
and `CRI`.
Example configurations:
Get Docker logs:
```
filebeat.inputs:
- type: container
paths:
- /var/lib/docker/containers/*/*.log
```
Get Kubernetes logs:
```
filebeat.inputs:
- type: container
paths:
- /var/log/pods/*/*/*.log
# this could also be used:
#- /var/log/containers/*.log
```
Previous `docker` input is deprecated in favor of this, to be removed in 8.0
…tic#12148) Fixes formatting of auditbeat.yml and auditbeat.reference.yml across platforms.
…tic#11661) * fix json bom + testing
* Reset beat ID in registries in case loaded from meta file * Set, not reset * Adding CHANGELOG entry
The CoreDNS and Envoyproxy dashboard were in the 8 instead of 7 directory. This PR fixes this.
houndci-bot
reviewed
May 21, 2019
houndci-bot
reviewed
May 21, 2019
houndci-bot
reviewed
May 21, 2019
houndci-bot
reviewed
May 21, 2019
houndci-bot
reviewed
May 21, 2019
houndci-bot
reviewed
May 21, 2019
|
I've marked hound's comment as resolved these comments should have been resolved in the review process. |
ph
commented
May 21, 2019
| @@ -196,6 +224,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d | |||
|
|
|||
| *Filebeat* | |||
|
|
|||
| - `docker` input is deprecated in favour `container`. {pull}12162[12162] | |||
|
|
|||
| *Heartbeat* | |||
|
|
|||
| *Journalbeat* | |||
|
Fix the make check issue. |
|
@simitt merged ^ 🎉 |
ph
added a commit
to ph/beats
that referenced
this pull request
May 22, 2019
urso
pushed a commit
to urso/beats
that referenced
this pull request
Jun 13, 2019
This reverts commit 3d0d64e.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
no conflicts