Reduce agent logs by default

[pchila]

What does this PR do?

This PR drops the Non-zero metrics after 30s... logs before they get sent to Elasticsearch.

Why is it important?

The goal is to reduce the amount of elastic-agent monitoring events ingested in Elasticsearch in order to reduce index disk size.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Author's Checklist

• [ ]

How to test this PR locally

Run an elastic agent with and without this change on a deployment making a note of the start times and elastic-agent id of both.
Using the included ES script we then extract the documents ingested during those runs and store them in dedicated indices.

In my tests I ran 2 elastic agents managed by fleet with a default policy including system integration, first without this change and then including this change, then sliced the first 10 minutes of logs and metrics from startup using the included script.

This is a screenshot from the Index management page were we can see the disk sizes and document count for each index:

The measured impact on *logs-elastic_agent.filebeat* and *logs-elastic_agent.metricbeat* is:

• reduction of ~27% of document count for *logs-elastic_agent.filebeat* with a disk size reduction of ~18%
• reduction of ~21% of document count for *logs-elastic_agent.metricbeat* with a disk size reduction of ~16%

Related issues

• Relates Reduce the amount the agent logs by default #4252

Use cases

Screenshots

Logs

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request does not have a backport label. Could you fix it @pchila? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[amitkanfer]

@pchila does this really close #4252 ? or only address this suggestion by @cmacknz ?

[pchila]

@pchila does this really close #4252 ? or only address this suggestion by @cmacknz ?

At this stage it addresses @cmacknz 's suggestion and removes some unnecessary logs, not sure if we want to stop here for this PR or we are going deeper, it depends on the measurement of how much data we are saving as it is and general 8.14 timeframe... Will change the "Closes" to "Relates"

[pierrehilbert]

I agree here, we shouldn't try to close #4252 but try to get as many improvement as possible for 8.14 and then continue for the next versions what we won't have time to do in that timeframe.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
9.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[cmacknz]

I enrolled an agent built from this branch and observed 30s metrics in the local agent logs but not in Fleet as expected.

[pierrehilbert]

@cmacknz any objections with backporting this in 8.14?

[cmacknz]

No objections to backporting.