…acking (#40651)

…_cluster metricset (#40634)

* initial commit for datastore cluster

* mage update

* update:
- data.json
- field.yml

* doc updates

* datastore_cluster -> datastorecluster

* resolve review comments

* fix UTs

* update URL for UTs

* update UTs

* fix field mapping

* update data.json

…0661)

* log total time taken for ListTimeSeries requests

* add changelog entry

* fix PR id

* log total time taken for AggregatedList request

* update changelog entry

* remove log

* Update x-pack/metricbeat/module/gcp/metrics/compute/metadata.go

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>

* Update x-pack/metricbeat/module/gcp/metrics/metrics_requester.go

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>

* Update x-pack/metricbeat/module/gcp/metrics/compute/metadata.go

Co-authored-by: Harnish Chavda <118714680+harnish-elastic@users.noreply.github.com>

* Update x-pack/metricbeat/module/gcp/metrics/metrics_requester.go

Co-authored-by: Harnish Chavda <118714680+harnish-elastic@users.noreply.github.com>

---------

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>
Co-authored-by: Harnish Chavda <118714680+harnish-elastic@users.noreply.github.com>

…of groups (#40681)

To fix the failing linter rule that requires OSS code to not depend on x-pack
code, this change removes the dependency on x-pack/dockerlogbeat/pipelinemock
from the Filebeat filestream input. It does this by creating a mock client
that is local to filestream test code ("A little copying is better than a
little dependency").

Fixes #40293

* [Packetbeat] Fix Packetbeat parsing mongodb OP_MSG

* Fixes handling OP_MSG based request/response, missing "end" timestamp
  and "duration" field for the event

Adding required dependencies to run Synthetics to Wolfi-based heartbeat images, from wolfi-os package manager.

* refactor fleet mode detection and storage

CEL programs used in the CEL input are specific to return an object that
includes an array. This is tested for, but currently the check is
indirect and only shows that the type is not expected, rather than a
more informative message that the events array is missing. This changes
that so that when no events field is present, an error is returned
directly. This will flow on to a status update in the handling of the
returned error by the input Run method, alerting the user more promptly
to the cause. Since a CEL input program is not valid if it returns no
events field, a hard error (fleet status failed rather than degraded)
seems appropriate.

* Fix windows test pipeline by reflecting browser monitor build constraints in the include file

* Add support of snapshot in virtualmachine

* mage update and add fields.yml

* Update data_test.go

* address comments, mage update

* Lint error

* Add createtime

* Update CHANGELOG.next.asciidoc

* Add state and id for snapshot

* update changelog

* lint change

* lint change

Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>
(cherry picked from commit fff09dc)

Co-authored-by: elastic-vault-github-plugin-prod[bot] <150874479+elastic-vault-github-plugin-prod[bot]@users.noreply.github.com>
Co-authored-by: Julien Lind <julien.lind@elastic.co>

* New images with `-wolfi` suffix get created
* Wolfi is used for preparing the file structure for Docker images

…_cluster metricset (#40694)

* initial commit

* add childEntity

* add changelog entry

* resolve review comments

* Remove extra debug messages

---------

Signed-off-by: constanca <constanca.manteigas@elastic.co>

* refactor: consistently use one uuid library

the codebase is using a mix of github.com/google/uuid and github.com/gofrs/uuid
Update the codebase to use github.com/gofrs/uuid consistently and bump to the
current major version.

* lint: fix linting issues

* refactor: move to gofrs/uuid

* lint: add gomodguard setting

* lint: fix more linter issues

* lint: more linting errors

* lint: completely remove gofrs/uuid v4

* lint: more linting errors

* lint: more linting errors

* lint: fix more linting errors

* fix: replace uuid.MustParse with uuid.Must(uuid.FromString())

* refactor: update uuid.Parse to uuid.FromString

* lint: fix linting errors

* lint: fix linting errors

* docs: fix nolint comments

* lint: fix linting errors

* docs: update nolint comments

* test: update uuid parsing err

* lint: fix linter issues

* lint: fix linter issues

* lint: run go mod tidy

* lint: regenerate notice

* build: regenerate notice

* build: regenerate notice

* refactor: cleanup double uuid conversion

* test: gracefully log the error

…0692)

* Add option to prevent new session creation and pre-create cached sessions

Co-authored-by: Julien Lind <julien.lind@elastic.co>

* Automatically label PRs to `main` with `backport-8.x`

When the `main` branch switches over to `9.0`, we will want most PRs that are merged into
`main` to also be backported to `8.x`. This PR makes this process easier by automatically
adding the `backport-8.x` label to all PRs targeting the `main` branch.

The exceptions will be PRs for changes that should _only_ be present in `9.0`.  For such PRs,
authors should simply remove the `backport-8.x` label from their PR.

* Fix typo in comment

* Add missing v prefix to label for consistent with backport PR rule

…config api requests (#40086)


* updating with config_kubeAdm and setbool option

* Convert fields to mapstr

* lint changes

* Address comments, update changelog

* lint change

* Address CI failure

Adjust heartbeat pipeline matrix for Windows, to correctly scope unit tests.

- Fix custom fields not being applied when provided through configuration
- Add regression test

…ance API (#40678)

* initial commit for intervalId supports for performance metrics

* update docs and fix CI

* Add changelog entry

* fix CI

* resolve review comments

* fix loggers

* resolved review comments

* update versions

* update UTs

* update integration tests

* 10s -> 20s

* Update CHANGELOG.next.asciidoc

Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com>

* Update metricbeat/docs/modules/vsphere.asciidoc

Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com>

* make update

* add recover for ToMetricSeries panic

* return error instead just logging it.

* remove restriction of interval IDs

* remove unnecessary validations

* remove recover and add empty condition

* update changelog entry

* Fix wrapping of errors in loggers

* update data.json

* update data.json

* fix CI and loggers

* update changelog entries

* make update

* fix changelog entries

* update changelog entry

---------

Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com>

The current user must be used instead of `root`.

The old way of specifying maintainers in a Dockerfile was to use the
MAINTAINER instruction. This is now deprecated and the maintainer label was
used instead.

Nowadays the recommended way to set a maintainer is through the
`org.opencontainers.image.authors` label.

See https://docs.docker.com/reference/build-checks/maintainer-deprecated/.

If journalctl exits unexpectedly the journald input will restart it and set the cursor to the last know position. Any error/non zero return code is logged at level error. There is an exponential backoff that caps at 1 restart every 2s.

The bitSize argument in strconv.ParseFloat should be either 32 or 64.
Any value other than 32 will default to 64 in the current implementation,
though this behavior is undocumented, making it undefined.

See https://cs.opensource.google/go/go/+/refs/tags/go1.23.0:src/strconv/atof.go;l=704-708.

…updates (#40759)

Bumps the github-actions group with 3 updates in the / directory: [updatecli/updatecli-action](https://github.com/updatecli/updatecli-action), [slackapi/slack-github-action](https://github.com/slackapi/slack-github-action) and [docker/login-action](https://github.com/docker/login-action).


Updates `updatecli/updatecli-action` from 2.65.0 to 2.67.0
- [Release notes](https://github.com/updatecli/updatecli-action/releases)
- [Commits](updatecli/updatecli-action@a0c478c...6b8881a)

Updates `slackapi/slack-github-action` from 1.26.0 to 1.27.0
- [Release notes](https://github.com/slackapi/slack-github-action/releases)
- [Commits](slackapi/slack-github-action@70cd7be...37ebaef)

Updates `docker/login-action` from 3.2.0 to 3.3.0
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](docker/login-action@0d4c9c5...9780b0c)

---
updated-dependencies:
- dependency-name: updatecli/updatecli-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
- dependency-name: slackapi/slack-github-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

This commit removes a call to conn.cancelReqs() that causes the
Connection to be unusable after Close() is called, leading to the bug
described by #40705.

…`metrics_count` to Prometheus module if `metrics_count` is `true` (#40411)

Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com>

see #40524

Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.4 to 43.0.1.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](pyca/cryptography@42.0.4...43.0.1)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…40751)

The change in #40047 breaks the use of the replace processor to do
s/<pattern>// replacement. So make the validation on absence, not zero state.

…... (#40799)

... iscovery/updatecli" Updatecli version policy

Made with ❤️️ by updatecli

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…ng metricsets (#40766)

* initial commit

* update field mappings

* update virtual machine data.json

* format data.json

* resolve review comments

* add changelog entry

* testing: Fix Docker deprecation warnings

I stumbled upon some warnings when trying to start the integration testing
environment. The errors don't cause any harm but they pollute the output of
the tool. Errors include:

	WARN[0000] /home/mauri870/git/elastic/beats/testing/environments/snapshot.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avo
id potential confusion
	WARN[0000] /home/mauri870/git/elastic/beats/testing/environments/local.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid
potential confusion

	WARN: MaintainerDeprecated: Maintainer instruction is deprecated in favor of using label (line 4)

* add alert names

* update host metricset

* add alert metrics to all the metricsets

* add changelog entry

* update data_test

* add triggerd alarms

* added entity name

* mage check

---------

Co-authored-by: Niraj Rathod <niraj.rathod@crestdatasys.com>
Co-authored-by: Kush Rana <kush.rana@elastic.co>
Co-authored-by: harnish-elastic <harnish.chavda@elastic.co>

Adding code to OOTB support AWS Eventbridge generated events for S3 changes, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html

…ed s3 object (#40775)

* add debug histograms for s3 object size and events per object

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* changelog entry

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* move changelog to developer next

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

---------

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* chore: deps(updatecli/policy): bump "ghcr.io/elastic/oblt-updatecli-p...

... olicies/ironbank/templates" Updatecli version policy

Made with ❤️️ by updatecli

* chore: deps(updatecli/policy): bump "ghcr.io/updatecli/policies/autod...

... iscovery/updatecli" Updatecli version policy

Made with ❤️️ by updatecli

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…40764)

* tests: fix check to enable the race detector in supported platforms

This code is almost 4 years old, the support for the race detector was
lacking at that point and Elastic had its first steps with arm64 builds of Beats.

I updated the check to allow tests to run with RACE_DETECTOR=true in supported
platforms, this includes linux/arm64 and darwin/arm64.

* add log if race is enabled for an unsupported platform

Co-authored-by: Niraj Rathod <niraj.rathod@crestdatasys.com>

- Rework the input's invalid JSON sanitization feature to support configurable sanitizers
- Migrate existing sanitizers to the new format
- Add a new `replace_all` sanitizer
- Add a new `invalid_json_messages_total` input metric to count the number of messages that contain invalid JSON strings.

The `replace_all` sanitizer replaces all the occurrences of substring matching the regex expression `pattern` with a fixed literal string `replacement`.

Here is a sample configuration:

```yaml
sanitizers:
- type: replace_all
  spec:
    pattern: '\[\s*([^\[\]{},\s]+(?:\s+[^\[\]{},\s]+)*)\s*\]'
    replacement: "{}"
```

For example, if the diagnostic settings send the following message:

```json
{
    "AppImage": "orcas/postgres_standalone_16_u18:38.1.240825",
    "AppType": "PostgreSQL",
    "properties": [
        218 B blob data
    ]
}
```

With the previous sample configuration, the input will replace the invalid JSON, updating the message to the following version:

```json
{
    "AppImage": "orcas/postgres_standalone_16_u18:38.1.240825",
    "AppType": "PostgreSQL",
    "properties": {}
}
```

The `replace_all` sanitizer aims to restore JSON syntax validity by replacing invalid, unfixable JSON with literal values that make sense for the context.

The invalid JSON sanitization became a thing when we faced Azure services producing malformed JSON documents.

Here is a list of previous occurrences of this problem:

- #34092
- elastic/azuremarketplacedev#190

In all cases, we first contacted Azure support, but they didn't fix the invalid JSON.

---------

Co-authored-by: Gabriel Pop <94497545+gpop63@users.noreply.github.com>

* Osquerybeat: Upgrade to osquery 5.13.1

* Update CHANGELOG with PR number

* Revert version.go checked in by mistake

* support OwniningAccount filtering

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* document the new paramter

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* add changelog

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* Update CHANGELOG.next.asciidoc

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update x-pack/metricbeat/modules.d/aws.yml.disabled

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* add missing docuemntation of owning_account parameter

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

---------

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

These paths do not exist so they should not be part of the
codeowners file.

/x-pack/metricbeat/module/gcp/dns
/x-pack/metricbeat/module/gcp/firewall
/x-pack/metricbeat/module/gcp/vpcflow

* inital meraki module and metricsets

* initial refactor for single meraki metricset device_health

* added tunnel support aka VPN support by Device

* adding interfaces aka switch ports and switch port status

* processing review comments

* refactored for comments

* fixing default metricset

* Removing unused variables and adding text to required variables

* add go module deps

* fixing for loop and starting on fields yml

* remove %d from loop condition

* fixing review comments and loss latency

* update dashboard-api lib to version with MIT licensed ratelimiter

* remove unused fields in metricset struct

* remove device details for now

* Major refactor to improve overall readability and device status/metric associations.

This commit removes a couple of data points from the original code, namely:
 - network appliance port settings - these settings are not defined per-device,
   but per-network, and as such don't fit into the current naming taxonomy.
 - site-to-site VPN settings - similarly to the above, these are not per-device attibutes.

In addition there is a limitation included on including only a single metric bucket (as returned
from the API) per metricbeat collection loop. There needs to be some better logging/config
validation on this, which will come later; for now, just run with a maximum 5 minute collection
interval.

This commit includes the field mappings too, but it currently causes events to get dropped due
to the index field limit getting hit (max 10k fields). You can get around this by setting
'setup.template.settings.index.mapping.total_fields.limit' to a higher value. We maybe can't
include all these mappings in the metricbeat module, but for now they are here to allow testing
with standalone metricbeat.

* Add maximum collection interval check to avoid missing uplink loss and latency metrics.

Remove 'percent' units, because the data does not fit the 0-1 elastic interpretation of percent.

* revert accidental changes to reference config

* add example in data.json

* review comments: license headers, missing/incorrect mapping

* result of 'go mod tidy' ... trying to unblock the CI builds

* fix linter issues

* update NOTICE.txt

* add release tag to fields.yml, update docs

* review feedback: keyword -> dat

* commit results of 'make update' in x-pack metricbeat for CI

* remove mappings to avoid hitting field limits in global metricbeat index template

* add one field back to please the docs CI

* more linter errors guarding against shared loop var pointers

* fix error introduced in previous commit

* add exception for golangci-lint G601 which is no longer required in go1.22+

* remove the guards against taking addresses of loop vars

* i can't get the golanglint-ci config to get picked up, so i'm reverting that change and putting back the annoying index loop code

* Update go.mod to match toolchain version to .go-version

Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>

---------

Co-authored-by: Dan Hiebert <dan@semplicityinc.com>
Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>

* initial module creation

* panos.system metricset running

* remove testing data

* panos.disk metricset working

* rename metricset

* change metricset name

* bgp_peers metricset working

* temperature metricset

* more metricsets

* use MetricSetFields

* license notices

* update fields.yml

* added doc

* refactor down to 4 metricsets

* more cleanup

* cleanup field names

* remove yml

* panos.yml.disabled

* PR comment fixes

* more PR comments addressed. Still to do: tests

* Changes to:
	- move tunnels from vpn to interfaces metricset
	- address PR comments for field names in field.yml
	- split local/peer addresses into host and port for bgp
	- handle license expires of "never"

* Fixes for PR comments

* add license header

* add pango package

* mage check && mage update

* remove mappings & make update

* make linter happy

* add the untracked docs

* update the fields.yml

* update the fields.yml with example fields to make python integ tests happy

* make docs check happy and update codeowners

* add result of 'mage update' in x-pack/metricbeat

---------

Co-authored-by: subham sarkar <subham.sarkar@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>

)

…... (#40874)

... iscovery/updatecli" Updatecli version policy

Made with ❤️️ by updatecli

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* update field name

* update changelog

* update changelog

---------

Co-authored-by: Niraj Rathod <niraj.rathod@crestdatasys.com>

* Expose output disk queue settings when running under agent

…over) (#40301)

* disable kubeadm for filebeat autodiscover

* Update filebeat/_meta/test/docs/01_playground/filebeat.yaml

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

* Update libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

* Update dev-tools/kubernetes/filebeat/Dockerfile.debug

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

* fixing docs and adding mericbeat references

* fixing docs and adding mericbeat references

* Update filebeat/_meta/config/filebeat.autodiscover.reference.yml.tmpl

Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

---------

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>
Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* Add NewBeatReceiver function for beats as otel receivers

…wlogs (#40578)

Add Redis replication role to slowlogs

---------

Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>

… cloud labels (#40461)

* issue fix

* add change.log

* formatting

* change.log

* error-fix

* resolved comments

* resolved comments

* context added

* resolve comments

* error handling

* doc changes

* change.log

* Update gcp.asciidoc

* Update CHANGELOG.asciidoc

* doc changes

…... (#40898)

... olicies/ironbank/templates" Updatecli version policy

Made with ❤️️ by updatecli

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

For example, `filebeat-wolfi-oss` -> `filebeat-oss-wolfi`

Co-authored-by: dparkerelastic <david.parker@elastic.co>

…ental API (#40684)

* [kibana] support api_key

* changes username and password to generic values

* make check formatting changes

* removes skip-go-installation option

* fixes: naked return in func

* adds skip-go-installation back

* only sets header if apikey is not empty

---------

Co-authored-by: VihasMakwana <121151420+VihasMakwana@users.noreply.github.com>

…40915)

…s on Windows (#40909)

* [filebeat] Fix long filepaths in diagnostics exceeding max path limits on Windows

* Update changelog with PR number

* Adjust utz

* Quiet the linter false positive

* More linter pacification

* Fix linter complains for errors checks in okta.go

* Let the linter complain again

* Revert revert

* clean up old Jenkins resources

* clean up groovy scripts and other .ci resources

---------

Co-authored-by: julienlind <vincent.lind@elastic.co>

…ection period (#40870)

This PR is to record the previous endTime from the previous collection period to use for calculating the next collection start time and end time. This PR also changes log.logger from cloudwatch to aws.cloudwatch to match the other logger.

HTTP functions were previously conditional on the resource URL being an
HTTP/HTTPS URL. In hindsight, this offers no benefits and can result in
confusing errors when the end user enters an invalid URL for an HTTP-based
integration. Instead of a URL-related or HTTP-related error, the user is
given a compilation error indicating that the HTTP-related function that
are being called (and they may not be aware of in any way) are not valid
references. So let's just always make them available.

…ent (#40918)

* override endpoint to etric client

* config and changelog

* doc

* resolved comments

* lint fix

)

Added MODULE env var definition within mage

the used version was a fork of an old version with a fix for windows 7.
We dropped support for windows 7 a long time ago so we can switch to the
upstream version and stay up to date.

drop go-winio replace statement