Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade go-ucfg to version 0.8.2 #16199

Merged
merged 5 commits into from
Feb 11, 2020

Conversation

blakerouse
Copy link
Contributor

What does this PR do?

Upgrades go-ucfg to version 0.8.2. (Currently this is pointing to master to get a full test run before releasing go-ucfg at version 0.8.2).

Why is it important?

Includes the following fixes:

  • Fixed nonzero validator to not fail on nil array or slice.
  • Fixed nonzero validator to validate maps.
  • Fixed required validator to validate maps.

Checklist

  • My code follows the style guidelines of this project
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@blakerouse blakerouse changed the title Upgrade go-ucfg to master, for testing before 0.8.2 release. Upgrade go-ucfg to version 0.8.2 Feb 7, 2020
@blakerouse blakerouse marked this pull request as ready for review February 10, 2020 22:00
@blakerouse blakerouse requested a review from a team as a code owner February 10, 2020 22:00
@@ -83,7 +83,7 @@ func TestModuleConfig(t *testing.T) {
{
name: "missing required field",
Copy link
Contributor

@ycombinator ycombinator Feb 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Maybe update the test name too?

Copy link
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a nit, but otherwise LGTM.

@ycombinator ycombinator self-assigned this Feb 11, 2020
@blakerouse blakerouse merged commit f9403ce into elastic:master Feb 11, 2020
@blakerouse blakerouse added the needs_backport PR is waiting to be backported to other branches. label Feb 11, 2020
@blakerouse blakerouse deleted the upgrade-go-ucfg-0.8.2 branch February 11, 2020 21:13
@blakerouse blakerouse added v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Feb 11, 2020
kvch added a commit that referenced this pull request Feb 12, 2020
* [Filebeat] move create-[module,fileset,fields] to mage (#15836)

- move create-[module,fileset,fields] to mage
- make mage create commands available in x-pack/filebeat
- change Makefile to use mage for create commands

* Elasticsearch index must be lowercase (#16081)

* Index names must be lowercase

When indexing into Elasticsearch index names must always be lowercase.
If the index or indices setting are configured to produce non-lowercase
strings (e.g. by extracting part of the index name from the event
contents), we need to normalize them to be lowercase.

This change ensure that index names are always converted to lowercase.
Static strings are converted to lowercase upfront, while dynamic strings
will be post-processed.

* update kafka/redis/LS output to guarantee lowercase index
* add godoc

* Regenerate expected files after changes in date parsing (#16139)

Elasticsearch has modified the behaviour on date parsing when the date
doesn't include timezone data. Regenerate a couple of golden files that
are affected by this change.

* Add autodiscover for aws_ec2 (#14823)

* Add autodiscover for aws_ec2
* Add aws.ec2.* to autodiscover template

* Fix a connection error in httpjson input (#16123)

* Fix a connection error in httpjson input

* Include document_id in decode_json_fields allowed fields (#16156)

* ci: run test on Windows (#15570)

* feat: run test on Windows

* chore: parameter to enable/disable windows test

* deleteDir before of the checkout

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Apply suggestions from code review

* feat: apply dependency hierarchies

* fit: use isChanged for all matches, and add the libbeat match where it is needed

* feat: add x-pack/winlogbeat windows unit tests

* fix: duplicate when condition

* Update Jenkinsfile

Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

* improve kubernetes.pod.cpu.usage.limit.pct field description (#16128)

* upgrade github.com/gogo/protobuf/... to v1.3.1 (#16138)

* [Filebeat] Add ECS tls & categorization fields to apache module (#16121)

* Add ECS tls & categorization fields to apache module

- tls.cipher (access)
- tls.protocol (access)
- tls.protocol_version (access)
- event.kind (access)
- event.category (access)
- event.outcome (access)
- lowercase http.request.method for ECS compliance (access)
- event.kind (error)
- event.category (error)
- event.type (error)

Closes #16032

* [Metricbeat] Add Overview dashboard to Tomcat module

* [Metricbeat] Fix PostgreSQL Dashboard (#16132)

* [Metricbeat] Fix PostgreSQL Dashboard

* Update version

* Fix: imports order (#16207)

* [Metricbeat]kube-state-metrics: add storage class support (#16145)

* add ksm storage class support

* [Journalbeat] Improve parsing of syslog.pid in journalbeat to strip the username when present (#16116)

* Improve parsing of syslog.pid in journalbeat to strip the username in pid when present.

* Add entry to changelog with pull ID.

* Improve the comment on the username strip.

* [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (#16019)

* Add a sha256 pin for the CA Certificate

When multiples CA are presents on the system we cannot ensure that a
specific one was used to validates the chains exposer by the server.
This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all
the code that has to create a TCP client with TLS support.

When the option is set, it will hook a new callback in the validation
chains that will inspect the verified and validated chains by Go to
ensure that a lets a certificate in the chains match the provided
sha256.

Usage example for the Elasticsearch output.

```
output.elasticsearch:
  hosts: [127.0.0.1:9200]
  ssl.ca_sha256: <base64_encoded_sha1>
```

You can generate the pin using the **openssl** binary with the
following command:

```
openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
```

OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html)

You will need to start Elasticsearch with the following options

```yaml
xpack.security.enabled: true
indices.id_field_data.enabled: true
xpack.license.self_generated.type: trial
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key"
xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt"
xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt"
```

This pull request also include a new service in the docker-compose.yml
that will start a new Elasticsearch server with TLS and security
configured.

* [docs] Add 7.6 breaking changes and release highlights (#16202)

* [docs] Add early draft of Elastic Log Driver docs (#15799)

* Index template will only be loaded if the configured output is Elasticsearch or Elastic Cloud (#16124) (#16225)

Minor update to be more explicit on the index template loading requirement.

Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>

* Remove spaces in prometheus commented out option (#16233)

* Fix: don't miss address scheme (#16205)

* Fix: don't miss address scheme

* Add unit test

* Adjust source after code review

* Add comment to method

* Freeze virtualenv version until issue with CI is resolved (#16235)

* [docs] Fix install command to match instructions on docker hub (#16249)

* [docs] Add link to observability release blog (#16246)

* ci(jenkins): enable fix-permissions to be executed without running make too (#16130)

* ci(jenkins): enable fix-permissions to be executed without running make too

* ci(jenkins): go modules are stored in the HOME path

* ci(jenkins): fix permissions should run only if docker is enabled

* Upgrade go-ucfg to version 0.8.2 (#16199)

* Upgrade go-ucfg to master, for testing before 0.8.2 release.

* Update notice.

* Fix tests.

* Update to the v0.8.2 release tag and remake NOTICE.txt.

* Improve test name.

* Add ingress nginx controller fileset (#16197)

* update notice

Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Lei Qiu <lei.qiu@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Co-authored-by: Chris Mark <chrismarkou92@gmail.com>
Co-authored-by: Gil Raphaelli <g@raphaelli.com>
Co-authored-by: Mario Castro <mariocaster@gmail.com>
Co-authored-by: Dimitri Mazmanov <sorantis@gmail.com>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
Co-authored-by: Pablo Mercado <pablo.mercado@elastic.co>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
blakerouse added a commit to blakerouse/beats that referenced this pull request Feb 13, 2020
* Upgrade go-ucfg to master, for testing before 0.8.2 release.

* Update notice.

* Fix tests.

* Update to the v0.8.2 release tag and remake NOTICE.txt.

* Improve test name.

(cherry picked from commit f9403ce)
blakerouse added a commit that referenced this pull request Feb 13, 2020
* Upgrade go-ucfg to master, for testing before 0.8.2 release.

* Update notice.

* Fix tests.

* Update to the v0.8.2 release tag and remake NOTICE.txt.

* Improve test name.

(cherry picked from commit f9403ce)
@simitt
Copy link
Contributor

simitt commented Feb 20, 2020

@blakerouse do you have any objections against backporting this to 7.6? There's a bugfix in APM Server that should be backported to 7.6 relying on this fix.

@blakerouse
Copy link
Contributor Author

@simitt I don't see a big issue with it being backported, but the 0.8.1 PR also would need to be backported.

@simitt
Copy link
Contributor

simitt commented Feb 21, 2020

Thanks for pointing that out @blakerouse. I talked to @urso about this, and since 0.8.1 would introduce a breaking change I'm not going to backport to 7.6.

kvch added a commit that referenced this pull request Feb 26, 2020
* [Filebeat] move create-[module,fileset,fields] to mage (#15836)

- move create-[module,fileset,fields] to mage
- make mage create commands available in x-pack/filebeat
- change Makefile to use mage for create commands

* Elasticsearch index must be lowercase (#16081)

* Index names must be lowercase

When indexing into Elasticsearch index names must always be lowercase.
If the index or indices setting are configured to produce non-lowercase
strings (e.g. by extracting part of the index name from the event
contents), we need to normalize them to be lowercase.

This change ensure that index names are always converted to lowercase.
Static strings are converted to lowercase upfront, while dynamic strings
will be post-processed.

* update kafka/redis/LS output to guarantee lowercase index
* add godoc

* Regenerate expected files after changes in date parsing (#16139)

Elasticsearch has modified the behaviour on date parsing when the date
doesn't include timezone data. Regenerate a couple of golden files that
are affected by this change.

* Add autodiscover for aws_ec2 (#14823)

* Add autodiscover for aws_ec2
* Add aws.ec2.* to autodiscover template

* Fix a connection error in httpjson input (#16123)

* Fix a connection error in httpjson input

* Include document_id in decode_json_fields allowed fields (#16156)

* ci: run test on Windows (#15570)

* feat: run test on Windows

* chore: parameter to enable/disable windows test

* deleteDir before of the checkout

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Apply suggestions from code review

* feat: apply dependency hierarchies

* fit: use isChanged for all matches, and add the libbeat match where it is needed

* feat: add x-pack/winlogbeat windows unit tests

* fix: duplicate when condition

* Update Jenkinsfile

Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

* improve kubernetes.pod.cpu.usage.limit.pct field description (#16128)

* upgrade github.com/gogo/protobuf/... to v1.3.1 (#16138)

* [Filebeat] Add ECS tls & categorization fields to apache module (#16121)

* Add ECS tls & categorization fields to apache module

- tls.cipher (access)
- tls.protocol (access)
- tls.protocol_version (access)
- event.kind (access)
- event.category (access)
- event.outcome (access)
- lowercase http.request.method for ECS compliance (access)
- event.kind (error)
- event.category (error)
- event.type (error)

Closes #16032

* [Metricbeat] Add Overview dashboard to Tomcat module

* [Metricbeat] Fix PostgreSQL Dashboard (#16132)

* [Metricbeat] Fix PostgreSQL Dashboard

* Update version

* Fix: imports order (#16207)

* [Metricbeat]kube-state-metrics: add storage class support (#16145)

* add ksm storage class support

* [Journalbeat] Improve parsing of syslog.pid in journalbeat to strip the username when present (#16116)

* Improve parsing of syslog.pid in journalbeat to strip the username in pid when present.

* Add entry to changelog with pull ID.

* Improve the comment on the username strip.

* [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (#16019)

* Add a sha256 pin for the CA Certificate

When multiples CA are presents on the system we cannot ensure that a
specific one was used to validates the chains exposer by the server.
This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all
the code that has to create a TCP client with TLS support.

When the option is set, it will hook a new callback in the validation
chains that will inspect the verified and validated chains by Go to
ensure that a lets a certificate in the chains match the provided
sha256.

Usage example for the Elasticsearch output.

```
output.elasticsearch:
  hosts: [127.0.0.1:9200]
  ssl.ca_sha256: <base64_encoded_sha1>
```

You can generate the pin using the **openssl** binary with the
following command:

```
openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
```

OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html)

You will need to start Elasticsearch with the following options

```yaml
xpack.security.enabled: true
indices.id_field_data.enabled: true
xpack.license.self_generated.type: trial
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key"
xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt"
xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt"
```

This pull request also include a new service in the docker-compose.yml
that will start a new Elasticsearch server with TLS and security
configured.

* [docs] Add 7.6 breaking changes and release highlights (#16202)

* [docs] Add early draft of Elastic Log Driver docs (#15799)

* Index template will only be loaded if the configured output is Elasticsearch or Elastic Cloud (#16124) (#16225)

Minor update to be more explicit on the index template loading requirement.

Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>

* Remove spaces in prometheus commented out option (#16233)

* Fix: don't miss address scheme (#16205)

* Fix: don't miss address scheme

* Add unit test

* Adjust source after code review

* Add comment to method

* Freeze virtualenv version until issue with CI is resolved (#16235)

* [docs] Fix install command to match instructions on docker hub (#16249)

* [docs] Add link to observability release blog (#16246)

* ci(jenkins): enable fix-permissions to be executed without running make too (#16130)

* ci(jenkins): enable fix-permissions to be executed without running make too

* ci(jenkins): go modules are stored in the HOME path

* ci(jenkins): fix permissions should run only if docker is enabled

* Upgrade go-ucfg to version 0.8.2 (#16199)

* Upgrade go-ucfg to master, for testing before 0.8.2 release.

* Update notice.

* Fix tests.

* Update to the v0.8.2 release tag and remake NOTICE.txt.

* Improve test name.

* Add ingress nginx controller fileset (#16197)

* update notice

Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Lei Qiu <lei.qiu@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Co-authored-by: Chris Mark <chrismarkou92@gmail.com>
Co-authored-by: Gil Raphaelli <g@raphaelli.com>
Co-authored-by: Mario Castro <mariocaster@gmail.com>
Co-authored-by: Dimitri Mazmanov <sorantis@gmail.com>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
Co-authored-by: Pablo Mercado <pablo.mercado@elastic.co>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
kvch added a commit that referenced this pull request Feb 27, 2020
* [Filebeat] move create-[module,fileset,fields] to mage (#15836)

- move create-[module,fileset,fields] to mage
- make mage create commands available in x-pack/filebeat
- change Makefile to use mage for create commands

* Elasticsearch index must be lowercase (#16081)

* Index names must be lowercase

When indexing into Elasticsearch index names must always be lowercase.
If the index or indices setting are configured to produce non-lowercase
strings (e.g. by extracting part of the index name from the event
contents), we need to normalize them to be lowercase.

This change ensure that index names are always converted to lowercase.
Static strings are converted to lowercase upfront, while dynamic strings
will be post-processed.

* update kafka/redis/LS output to guarantee lowercase index
* add godoc

* Regenerate expected files after changes in date parsing (#16139)

Elasticsearch has modified the behaviour on date parsing when the date
doesn't include timezone data. Regenerate a couple of golden files that
are affected by this change.

* Add autodiscover for aws_ec2 (#14823)

* Add autodiscover for aws_ec2
* Add aws.ec2.* to autodiscover template

* Fix a connection error in httpjson input (#16123)

* Fix a connection error in httpjson input

* Include document_id in decode_json_fields allowed fields (#16156)

* ci: run test on Windows (#15570)

* feat: run test on Windows

* chore: parameter to enable/disable windows test

* deleteDir before of the checkout

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Apply suggestions from code review

* feat: apply dependency hierarchies

* fit: use isChanged for all matches, and add the libbeat match where it is needed

* feat: add x-pack/winlogbeat windows unit tests

* fix: duplicate when condition

* Update Jenkinsfile

Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

* improve kubernetes.pod.cpu.usage.limit.pct field description (#16128)

* upgrade github.com/gogo/protobuf/... to v1.3.1 (#16138)

* [Filebeat] Add ECS tls & categorization fields to apache module (#16121)

* Add ECS tls & categorization fields to apache module

- tls.cipher (access)
- tls.protocol (access)
- tls.protocol_version (access)
- event.kind (access)
- event.category (access)
- event.outcome (access)
- lowercase http.request.method for ECS compliance (access)
- event.kind (error)
- event.category (error)
- event.type (error)

Closes #16032

* [Metricbeat] Add Overview dashboard to Tomcat module

* [Metricbeat] Fix PostgreSQL Dashboard (#16132)

* [Metricbeat] Fix PostgreSQL Dashboard

* Update version

* Fix: imports order (#16207)

* [Metricbeat]kube-state-metrics: add storage class support (#16145)

* add ksm storage class support

* [Journalbeat] Improve parsing of syslog.pid in journalbeat to strip the username when present (#16116)

* Improve parsing of syslog.pid in journalbeat to strip the username in pid when present.

* Add entry to changelog with pull ID.

* Improve the comment on the username strip.

* [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (#16019)

* Add a sha256 pin for the CA Certificate

When multiples CA are presents on the system we cannot ensure that a
specific one was used to validates the chains exposer by the server.
This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all
the code that has to create a TCP client with TLS support.

When the option is set, it will hook a new callback in the validation
chains that will inspect the verified and validated chains by Go to
ensure that a lets a certificate in the chains match the provided
sha256.

Usage example for the Elasticsearch output.

```
output.elasticsearch:
  hosts: [127.0.0.1:9200]
  ssl.ca_sha256: <base64_encoded_sha1>
```

You can generate the pin using the **openssl** binary with the
following command:

```
openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
```

OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html)

You will need to start Elasticsearch with the following options

```yaml
xpack.security.enabled: true
indices.id_field_data.enabled: true
xpack.license.self_generated.type: trial
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key"
xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt"
xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt"
```

This pull request also include a new service in the docker-compose.yml
that will start a new Elasticsearch server with TLS and security
configured.

* [docs] Add 7.6 breaking changes and release highlights (#16202)

* [docs] Add early draft of Elastic Log Driver docs (#15799)

* Index template will only be loaded if the configured output is Elasticsearch or Elastic Cloud (#16124) (#16225)

Minor update to be more explicit on the index template loading requirement.

Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>

* Remove spaces in prometheus commented out option (#16233)

* Fix: don't miss address scheme (#16205)

* Fix: don't miss address scheme

* Add unit test

* Adjust source after code review

* Add comment to method

* Freeze virtualenv version until issue with CI is resolved (#16235)

* [docs] Fix install command to match instructions on docker hub (#16249)

* [docs] Add link to observability release blog (#16246)

* ci(jenkins): enable fix-permissions to be executed without running make too (#16130)

* ci(jenkins): enable fix-permissions to be executed without running make too

* ci(jenkins): go modules are stored in the HOME path

* ci(jenkins): fix permissions should run only if docker is enabled

* Upgrade go-ucfg to version 0.8.2 (#16199)

* Upgrade go-ucfg to master, for testing before 0.8.2 release.

* Update notice.

* Fix tests.

* Update to the v0.8.2 release tag and remake NOTICE.txt.

* Improve test name.

* Add ingress nginx controller fileset (#16197)

* update notice

Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Lei Qiu <lei.qiu@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Co-authored-by: Chris Mark <chrismarkou92@gmail.com>
Co-authored-by: Gil Raphaelli <g@raphaelli.com>
Co-authored-by: Mario Castro <mariocaster@gmail.com>
Co-authored-by: Dimitri Mazmanov <sorantis@gmail.com>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
Co-authored-by: Pablo Mercado <pablo.mercado@elastic.co>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
kvch added a commit that referenced this pull request Feb 28, 2020
* [Filebeat] move create-[module,fileset,fields] to mage (#15836)

- move create-[module,fileset,fields] to mage
- make mage create commands available in x-pack/filebeat
- change Makefile to use mage for create commands

* Elasticsearch index must be lowercase (#16081)

* Index names must be lowercase

When indexing into Elasticsearch index names must always be lowercase.
If the index or indices setting are configured to produce non-lowercase
strings (e.g. by extracting part of the index name from the event
contents), we need to normalize them to be lowercase.

This change ensure that index names are always converted to lowercase.
Static strings are converted to lowercase upfront, while dynamic strings
will be post-processed.

* update kafka/redis/LS output to guarantee lowercase index
* add godoc

* Regenerate expected files after changes in date parsing (#16139)

Elasticsearch has modified the behaviour on date parsing when the date
doesn't include timezone data. Regenerate a couple of golden files that
are affected by this change.

* Add autodiscover for aws_ec2 (#14823)

* Add autodiscover for aws_ec2
* Add aws.ec2.* to autodiscover template

* Fix a connection error in httpjson input (#16123)

* Fix a connection error in httpjson input

* Include document_id in decode_json_fields allowed fields (#16156)

* ci: run test on Windows (#15570)

* feat: run test on Windows

* chore: parameter to enable/disable windows test

* deleteDir before of the checkout

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Apply suggestions from code review

* feat: apply dependency hierarchies

* fit: use isChanged for all matches, and add the libbeat match where it is needed

* feat: add x-pack/winlogbeat windows unit tests

* fix: duplicate when condition

* Update Jenkinsfile

Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

* improve kubernetes.pod.cpu.usage.limit.pct field description (#16128)

* upgrade github.com/gogo/protobuf/... to v1.3.1 (#16138)

* [Filebeat] Add ECS tls & categorization fields to apache module (#16121)

* Add ECS tls & categorization fields to apache module

- tls.cipher (access)
- tls.protocol (access)
- tls.protocol_version (access)
- event.kind (access)
- event.category (access)
- event.outcome (access)
- lowercase http.request.method for ECS compliance (access)
- event.kind (error)
- event.category (error)
- event.type (error)

Closes #16032

* [Metricbeat] Add Overview dashboard to Tomcat module

* [Metricbeat] Fix PostgreSQL Dashboard (#16132)

* [Metricbeat] Fix PostgreSQL Dashboard

* Update version

* Fix: imports order (#16207)

* [Metricbeat]kube-state-metrics: add storage class support (#16145)

* add ksm storage class support

* [Journalbeat] Improve parsing of syslog.pid in journalbeat to strip the username when present (#16116)

* Improve parsing of syslog.pid in journalbeat to strip the username in pid when present.

* Add entry to changelog with pull ID.

* Improve the comment on the username strip.

* [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (#16019)

* Add a sha256 pin for the CA Certificate

When multiples CA are presents on the system we cannot ensure that a
specific one was used to validates the chains exposer by the server.
This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all
the code that has to create a TCP client with TLS support.

When the option is set, it will hook a new callback in the validation
chains that will inspect the verified and validated chains by Go to
ensure that a lets a certificate in the chains match the provided
sha256.

Usage example for the Elasticsearch output.

```
output.elasticsearch:
  hosts: [127.0.0.1:9200]
  ssl.ca_sha256: <base64_encoded_sha1>
```

You can generate the pin using the **openssl** binary with the
following command:

```
openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
```

OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html)

You will need to start Elasticsearch with the following options

```yaml
xpack.security.enabled: true
indices.id_field_data.enabled: true
xpack.license.self_generated.type: trial
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key"
xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt"
xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt"
```

This pull request also include a new service in the docker-compose.yml
that will start a new Elasticsearch server with TLS and security
configured.

* [docs] Add 7.6 breaking changes and release highlights (#16202)

* [docs] Add early draft of Elastic Log Driver docs (#15799)

* Index template will only be loaded if the configured output is Elasticsearch or Elastic Cloud (#16124) (#16225)

Minor update to be more explicit on the index template loading requirement.

Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>

* Remove spaces in prometheus commented out option (#16233)

* Fix: don't miss address scheme (#16205)

* Fix: don't miss address scheme

* Add unit test

* Adjust source after code review

* Add comment to method

* Freeze virtualenv version until issue with CI is resolved (#16235)

* [docs] Fix install command to match instructions on docker hub (#16249)

* [docs] Add link to observability release blog (#16246)

* ci(jenkins): enable fix-permissions to be executed without running make too (#16130)

* ci(jenkins): enable fix-permissions to be executed without running make too

* ci(jenkins): go modules are stored in the HOME path

* ci(jenkins): fix permissions should run only if docker is enabled

* Upgrade go-ucfg to version 0.8.2 (#16199)

* Upgrade go-ucfg to master, for testing before 0.8.2 release.

* Update notice.

* Fix tests.

* Update to the v0.8.2 release tag and remake NOTICE.txt.

* Improve test name.

* Add ingress nginx controller fileset (#16197)

* update notice

Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Lei Qiu <lei.qiu@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Co-authored-by: Chris Mark <chrismarkou92@gmail.com>
Co-authored-by: Gil Raphaelli <g@raphaelli.com>
Co-authored-by: Mario Castro <mariocaster@gmail.com>
Co-authored-by: Dimitri Mazmanov <sorantis@gmail.com>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
Co-authored-by: Pablo Mercado <pablo.mercado@elastic.co>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
kvch added a commit that referenced this pull request Mar 2, 2020
* [Filebeat] move create-[module,fileset,fields] to mage (#15836)

- move create-[module,fileset,fields] to mage
- make mage create commands available in x-pack/filebeat
- change Makefile to use mage for create commands

* Elasticsearch index must be lowercase (#16081)

* Index names must be lowercase

When indexing into Elasticsearch index names must always be lowercase.
If the index or indices setting are configured to produce non-lowercase
strings (e.g. by extracting part of the index name from the event
contents), we need to normalize them to be lowercase.

This change ensure that index names are always converted to lowercase.
Static strings are converted to lowercase upfront, while dynamic strings
will be post-processed.

* update kafka/redis/LS output to guarantee lowercase index
* add godoc

* Regenerate expected files after changes in date parsing (#16139)

Elasticsearch has modified the behaviour on date parsing when the date
doesn't include timezone data. Regenerate a couple of golden files that
are affected by this change.

* Add autodiscover for aws_ec2 (#14823)

* Add autodiscover for aws_ec2
* Add aws.ec2.* to autodiscover template

* Fix a connection error in httpjson input (#16123)

* Fix a connection error in httpjson input

* Include document_id in decode_json_fields allowed fields (#16156)

* ci: run test on Windows (#15570)

* feat: run test on Windows

* chore: parameter to enable/disable windows test

* deleteDir before of the checkout

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Apply suggestions from code review

* feat: apply dependency hierarchies

* fit: use isChanged for all matches, and add the libbeat match where it is needed

* feat: add x-pack/winlogbeat windows unit tests

* fix: duplicate when condition

* Update Jenkinsfile

Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

* improve kubernetes.pod.cpu.usage.limit.pct field description (#16128)

* upgrade github.com/gogo/protobuf/... to v1.3.1 (#16138)

* [Filebeat] Add ECS tls & categorization fields to apache module (#16121)

* Add ECS tls & categorization fields to apache module

- tls.cipher (access)
- tls.protocol (access)
- tls.protocol_version (access)
- event.kind (access)
- event.category (access)
- event.outcome (access)
- lowercase http.request.method for ECS compliance (access)
- event.kind (error)
- event.category (error)
- event.type (error)

Closes #16032

* [Metricbeat] Add Overview dashboard to Tomcat module

* [Metricbeat] Fix PostgreSQL Dashboard (#16132)

* [Metricbeat] Fix PostgreSQL Dashboard

* Update version

* Fix: imports order (#16207)

* [Metricbeat]kube-state-metrics: add storage class support (#16145)

* add ksm storage class support

* [Journalbeat] Improve parsing of syslog.pid in journalbeat to strip the username when present (#16116)

* Improve parsing of syslog.pid in journalbeat to strip the username in pid when present.

* Add entry to changelog with pull ID.

* Improve the comment on the username strip.

* [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (#16019)

* Add a sha256 pin for the CA Certificate

When multiples CA are presents on the system we cannot ensure that a
specific one was used to validates the chains exposer by the server.
This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all
the code that has to create a TCP client with TLS support.

When the option is set, it will hook a new callback in the validation
chains that will inspect the verified and validated chains by Go to
ensure that a lets a certificate in the chains match the provided
sha256.

Usage example for the Elasticsearch output.

```
output.elasticsearch:
  hosts: [127.0.0.1:9200]
  ssl.ca_sha256: <base64_encoded_sha1>
```

You can generate the pin using the **openssl** binary with the
following command:

```
openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
```

OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html)

You will need to start Elasticsearch with the following options

```yaml
xpack.security.enabled: true
indices.id_field_data.enabled: true
xpack.license.self_generated.type: trial
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key"
xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt"
xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt"
```

This pull request also include a new service in the docker-compose.yml
that will start a new Elasticsearch server with TLS and security
configured.

* [docs] Add 7.6 breaking changes and release highlights (#16202)

* [docs] Add early draft of Elastic Log Driver docs (#15799)

* Index template will only be loaded if the configured output is Elasticsearch or Elastic Cloud (#16124) (#16225)

Minor update to be more explicit on the index template loading requirement.

Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>

* Remove spaces in prometheus commented out option (#16233)

* Fix: don't miss address scheme (#16205)

* Fix: don't miss address scheme

* Add unit test

* Adjust source after code review

* Add comment to method

* Freeze virtualenv version until issue with CI is resolved (#16235)

* [docs] Fix install command to match instructions on docker hub (#16249)

* [docs] Add link to observability release blog (#16246)

* ci(jenkins): enable fix-permissions to be executed without running make too (#16130)

* ci(jenkins): enable fix-permissions to be executed without running make too

* ci(jenkins): go modules are stored in the HOME path

* ci(jenkins): fix permissions should run only if docker is enabled

* Upgrade go-ucfg to version 0.8.2 (#16199)

* Upgrade go-ucfg to master, for testing before 0.8.2 release.

* Update notice.

* Fix tests.

* Update to the v0.8.2 release tag and remake NOTICE.txt.

* Improve test name.

* Add ingress nginx controller fileset (#16197)

* update notice

Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Lei Qiu <lei.qiu@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Co-authored-by: Chris Mark <chrismarkou92@gmail.com>
Co-authored-by: Gil Raphaelli <g@raphaelli.com>
Co-authored-by: Mario Castro <mariocaster@gmail.com>
Co-authored-by: Dimitri Mazmanov <sorantis@gmail.com>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
Co-authored-by: Pablo Mercado <pablo.mercado@elastic.co>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
kvch added a commit that referenced this pull request Mar 3, 2020
* [Filebeat] move create-[module,fileset,fields] to mage (#15836)

- move create-[module,fileset,fields] to mage
- make mage create commands available in x-pack/filebeat
- change Makefile to use mage for create commands

* Elasticsearch index must be lowercase (#16081)

* Index names must be lowercase

When indexing into Elasticsearch index names must always be lowercase.
If the index or indices setting are configured to produce non-lowercase
strings (e.g. by extracting part of the index name from the event
contents), we need to normalize them to be lowercase.

This change ensure that index names are always converted to lowercase.
Static strings are converted to lowercase upfront, while dynamic strings
will be post-processed.

* update kafka/redis/LS output to guarantee lowercase index
* add godoc

* Regenerate expected files after changes in date parsing (#16139)

Elasticsearch has modified the behaviour on date parsing when the date
doesn't include timezone data. Regenerate a couple of golden files that
are affected by this change.

* Add autodiscover for aws_ec2 (#14823)

* Add autodiscover for aws_ec2
* Add aws.ec2.* to autodiscover template

* Fix a connection error in httpjson input (#16123)

* Fix a connection error in httpjson input

* Include document_id in decode_json_fields allowed fields (#16156)

* ci: run test on Windows (#15570)

* feat: run test on Windows

* chore: parameter to enable/disable windows test

* deleteDir before of the checkout

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Apply suggestions from code review

* feat: apply dependency hierarchies

* fit: use isChanged for all matches, and add the libbeat match where it is needed

* feat: add x-pack/winlogbeat windows unit tests

* fix: duplicate when condition

* Update Jenkinsfile

Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

* improve kubernetes.pod.cpu.usage.limit.pct field description (#16128)

* upgrade github.com/gogo/protobuf/... to v1.3.1 (#16138)

* [Filebeat] Add ECS tls & categorization fields to apache module (#16121)

* Add ECS tls & categorization fields to apache module

- tls.cipher (access)
- tls.protocol (access)
- tls.protocol_version (access)
- event.kind (access)
- event.category (access)
- event.outcome (access)
- lowercase http.request.method for ECS compliance (access)
- event.kind (error)
- event.category (error)
- event.type (error)

Closes #16032

* [Metricbeat] Add Overview dashboard to Tomcat module

* [Metricbeat] Fix PostgreSQL Dashboard (#16132)

* [Metricbeat] Fix PostgreSQL Dashboard

* Update version

* Fix: imports order (#16207)

* [Metricbeat]kube-state-metrics: add storage class support (#16145)

* add ksm storage class support

* [Journalbeat] Improve parsing of syslog.pid in journalbeat to strip the username when present (#16116)

* Improve parsing of syslog.pid in journalbeat to strip the username in pid when present.

* Add entry to changelog with pull ID.

* Improve the comment on the username strip.

* [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (#16019)

* Add a sha256 pin for the CA Certificate

When multiples CA are presents on the system we cannot ensure that a
specific one was used to validates the chains exposer by the server.
This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all
the code that has to create a TCP client with TLS support.

When the option is set, it will hook a new callback in the validation
chains that will inspect the verified and validated chains by Go to
ensure that a lets a certificate in the chains match the provided
sha256.

Usage example for the Elasticsearch output.

```
output.elasticsearch:
  hosts: [127.0.0.1:9200]
  ssl.ca_sha256: <base64_encoded_sha1>
```

You can generate the pin using the **openssl** binary with the
following command:

```
openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
```

OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html)

You will need to start Elasticsearch with the following options

```yaml
xpack.security.enabled: true
indices.id_field_data.enabled: true
xpack.license.self_generated.type: trial
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key"
xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt"
xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt"
```

This pull request also include a new service in the docker-compose.yml
that will start a new Elasticsearch server with TLS and security
configured.

* [docs] Add 7.6 breaking changes and release highlights (#16202)

* [docs] Add early draft of Elastic Log Driver docs (#15799)

* Index template will only be loaded if the configured output is Elasticsearch or Elastic Cloud (#16124) (#16225)

Minor update to be more explicit on the index template loading requirement.

Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>

* Remove spaces in prometheus commented out option (#16233)

* Fix: don't miss address scheme (#16205)

* Fix: don't miss address scheme

* Add unit test

* Adjust source after code review

* Add comment to method

* Freeze virtualenv version until issue with CI is resolved (#16235)

* [docs] Fix install command to match instructions on docker hub (#16249)

* [docs] Add link to observability release blog (#16246)

* ci(jenkins): enable fix-permissions to be executed without running make too (#16130)

* ci(jenkins): enable fix-permissions to be executed without running make too

* ci(jenkins): go modules are stored in the HOME path

* ci(jenkins): fix permissions should run only if docker is enabled

* Upgrade go-ucfg to version 0.8.2 (#16199)

* Upgrade go-ucfg to master, for testing before 0.8.2 release.

* Update notice.

* Fix tests.

* Update to the v0.8.2 release tag and remake NOTICE.txt.

* Improve test name.

* Add ingress nginx controller fileset (#16197)

* update notice

Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Lei Qiu <lei.qiu@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Co-authored-by: Chris Mark <chrismarkou92@gmail.com>
Co-authored-by: Gil Raphaelli <g@raphaelli.com>
Co-authored-by: Mario Castro <mariocaster@gmail.com>
Co-authored-by: Dimitri Mazmanov <sorantis@gmail.com>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
Co-authored-by: Pablo Mercado <pablo.mercado@elastic.co>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
* [Filebeat] move create-[module,fileset,fields] to mage (elastic#15836)

- move create-[module,fileset,fields] to mage
- make mage create commands available in x-pack/filebeat
- change Makefile to use mage for create commands

* Elasticsearch index must be lowercase (elastic#16081)

* Index names must be lowercase

When indexing into Elasticsearch index names must always be lowercase.
If the index or indices setting are configured to produce non-lowercase
strings (e.g. by extracting part of the index name from the event
contents), we need to normalize them to be lowercase.

This change ensure that index names are always converted to lowercase.
Static strings are converted to lowercase upfront, while dynamic strings
will be post-processed.

* update kafka/redis/LS output to guarantee lowercase index
* add godoc

* Regenerate expected files after changes in date parsing (elastic#16139)

Elasticsearch has modified the behaviour on date parsing when the date
doesn't include timezone data. Regenerate a couple of golden files that
are affected by this change.

* Add autodiscover for aws_ec2 (elastic#14823)

* Add autodiscover for aws_ec2
* Add aws.ec2.* to autodiscover template

* Fix a connection error in httpjson input (elastic#16123)

* Fix a connection error in httpjson input

* Include document_id in decode_json_fields allowed fields (elastic#16156)

* ci: run test on Windows (elastic#15570)

* feat: run test on Windows

* chore: parameter to enable/disable windows test

* deleteDir before of the checkout

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Update Jenkinsfile

* Apply suggestions from code review

* feat: apply dependency hierarchies

* fit: use isChanged for all matches, and add the libbeat match where it is needed

* feat: add x-pack/winlogbeat windows unit tests

* fix: duplicate when condition

* Update Jenkinsfile

Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

* improve kubernetes.pod.cpu.usage.limit.pct field description (elastic#16128)

* upgrade github.com/gogo/protobuf/... to v1.3.1 (elastic#16138)

* [Filebeat] Add ECS tls & categorization fields to apache module (elastic#16121)

* Add ECS tls & categorization fields to apache module

- tls.cipher (access)
- tls.protocol (access)
- tls.protocol_version (access)
- event.kind (access)
- event.category (access)
- event.outcome (access)
- lowercase http.request.method for ECS compliance (access)
- event.kind (error)
- event.category (error)
- event.type (error)

Closes elastic#16032

* [Metricbeat] Add Overview dashboard to Tomcat module

* [Metricbeat] Fix PostgreSQL Dashboard (elastic#16132)

* [Metricbeat] Fix PostgreSQL Dashboard

* Update version

* Fix: imports order (elastic#16207)

* [Metricbeat]kube-state-metrics: add storage class support (elastic#16145)

* add ksm storage class support

* [Journalbeat] Improve parsing of syslog.pid in journalbeat to strip the username when present (elastic#16116)

* Improve parsing of syslog.pid in journalbeat to strip the username in pid when present.

* Add entry to changelog with pull ID.

* Improve the comment on the username strip.

* [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (elastic#16019)

* Add a sha256 pin for the CA Certificate

When multiples CA are presents on the system we cannot ensure that a
specific one was used to validates the chains exposer by the server.
This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all
the code that has to create a TCP client with TLS support.

When the option is set, it will hook a new callback in the validation
chains that will inspect the verified and validated chains by Go to
ensure that a lets a certificate in the chains match the provided
sha256.

Usage example for the Elasticsearch output.

```
output.elasticsearch:
  hosts: [127.0.0.1:9200]
  ssl.ca_sha256: <base64_encoded_sha1>
```

You can generate the pin using the **openssl** binary with the
following command:

```
openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
```

OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html)

You will need to start Elasticsearch with the following options

```yaml
xpack.security.enabled: true
indices.id_field_data.enabled: true
xpack.license.self_generated.type: trial
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key"
xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt"
xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt"
```

This pull request also include a new service in the docker-compose.yml
that will start a new Elasticsearch server with TLS and security
configured.

* [docs] Add 7.6 breaking changes and release highlights (elastic#16202)

* [docs] Add early draft of Elastic Log Driver docs (elastic#15799)

* Index template will only be loaded if the configured output is Elasticsearch or Elastic Cloud (elastic#16124) (elastic#16225)

Minor update to be more explicit on the index template loading requirement.

Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>

* Remove spaces in prometheus commented out option (elastic#16233)

* Fix: don't miss address scheme (elastic#16205)

* Fix: don't miss address scheme

* Add unit test

* Adjust source after code review

* Add comment to method

* Freeze virtualenv version until issue with CI is resolved (elastic#16235)

* [docs] Fix install command to match instructions on docker hub (elastic#16249)

* [docs] Add link to observability release blog (elastic#16246)

* ci(jenkins): enable fix-permissions to be executed without running make too (elastic#16130)

* ci(jenkins): enable fix-permissions to be executed without running make too

* ci(jenkins): go modules are stored in the HOME path

* ci(jenkins): fix permissions should run only if docker is enabled

* Upgrade go-ucfg to version 0.8.2 (elastic#16199)

* Upgrade go-ucfg to master, for testing before 0.8.2 release.

* Update notice.

* Fix tests.

* Update to the v0.8.2 release tag and remake NOTICE.txt.

* Improve test name.

* Add ingress nginx controller fileset (elastic#16197)

* update notice

Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Lei Qiu <lei.qiu@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Co-authored-by: Chris Mark <chrismarkou92@gmail.com>
Co-authored-by: Gil Raphaelli <g@raphaelli.com>
Co-authored-by: Mario Castro <mariocaster@gmail.com>
Co-authored-by: Dimitri Mazmanov <sorantis@gmail.com>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
Co-authored-by: Pablo Mercado <pablo.mercado@elastic.co>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants