Make additional OTel Collector components available in Agent running in OTel mode

[ycombinator]

What does this PR do?

This PR makes the following additional OpenTelemetry Collector components available when Elastic Agent is running in otel mode:

• hostmetrics receiver
• filter processor
• elasticsearch exporter

Why is it important?

To allow users of Elastic Agent running as as OpenTelemetry Collector to configure pipelines containing the aforementioned components.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[ycombinator]

As requested by @strawgate for 8.15.0.

[strawgate]

We may also want to add resourcedetection and filter

[michalpristas]

maybe not in this PR but we will need to replace memorylimiterprocessor with memorylimiterextension.
difference is behavior is that processors drops the events and extensions acts as a pushback mechanism stopping/pausing processing on receiver side

[michalpristas]

go version is specified as 1.21.9 maybe we should match and update "bump go version" to update go.mod as well

[ycombinator]

Fixed (manually) in 176df27.

Put up #4636 to automatically keep the toolchain version here in sync with the contents of .go-version.

[ycombinator]

maybe not in this PR but we will need to replace memorylimiterprocessor with memorylimiterextension. difference is behavior is that processors drops the events and extensions acts as a pushback mechanism stopping/pausing processing on receiver side

@strawgate thoughts on this? Are you okay with us switching to the memorylimiterextension?

[michalpristas]

there will be a conflict: #4638

[ycombinator]

Let's wait to get #4638 merged first since that's a more important change IMO and then I'll rebase this PR on main.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-mode-es-exporter upstream/otel-mode-es-exporter
git merge upstream/main
git push upstream otel-mode-es-exporter

[strawgate]

maybe not in this PR but we will need to replace memorylimiterprocessor with memorylimiterextension. difference is behavior is that processors drops the events and extensions acts as a pushback mechanism stopping/pausing processing on receiver side

@strawgate thoughts on this? Are you okay with us switching to the memorylimiterextension?

@ycombinator I don't think i have any concerns so if you dont have any concerns then let's proceed

[strawgate]

If we can get hostmetrics and elasticsearch exporter for 8.14, that's a win in my book.

So 2 with a big hope that it's not one of those two

[ycombinator]

Thanks @strawgate. Trying out just hostmetricsreceiver and elasticsearchexporter in a separate PR: #4664. Not seeing the dependency conflict in go.mod with just these two 👍.

Will layer on the other components from this PR one by one so we can get as many in as possible without the dependency conflict.

[ycombinator]

maybe not in this PR but we will need to replace memorylimiterprocessor with memorylimiterextension. difference is behavior is that processors drops the events and extensions acts as a pushback mechanism stopping/pausing processing on receiver side

Filed #4665 to track this change.

[ycombinator]

It was resourcedetectionprocessor that was causing the k8s dependencies conflicts with the github.com/elastic/elastic-agent-autodiscover dependency. I've removed that from this PR now.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[ycombinator]

We have flaky integration tests failing on Windows: #4686 and #4687. @jlind23 @cmacknz since we're trying to get this change into 8.14.0 and the failing flaky tests are unrelated to the changes in this PR, are you okay to merge this PR as-is?

[ycombinator]

@cmacknz pointed out (in Slack) that the "The service did not respond to the start or control request in a timely fashion" error in CI isn't happening on other branches so we need to be sure one of the changes in this PR isn't causing this error.

I'm going to keep this PR open for now but create three additional PRs, one for each of the OTel Collector components being added to Agent. That way we can hopefully narrow down which component is causing the error and also get the other components in.

[ycombinator]

I'm going to keep this PR open for now but create three additional PRs, one for each of the OTel Collector components being added to Agent. That way we can hopefully narrow down which component is causing the error and also get the other components in.

• hostmetricsreceiver: Make hostmetricsreceiver available in Agent running in OTel mode #4706
• filterprocessor: Make filterprocessor available in Agent running in OTel mode #4708
• elasticsearchexporter: Make elasticsearchexporter available in Agent running in OTel mode #4707

[ycombinator]

Looks like it might be the hostmetricsreceiver that's the cause for the CI errors (isolated via #4706).

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-mode-es-exporter upstream/otel-mode-es-exporter
git merge upstream/main
git push upstream otel-mode-es-exporter

[ycombinator]

Closing this PR in favor of individual PRs for the three components being added in this PR:

• hostmetricsreceiver: Make hostmetricsreceiver available in Agent running in OTel mode #4706
• filterprocessor: Make filterprocessor available in Agent running in OTel mode #4708
• elasticsearchexporter: Make elasticsearchexporter available in Agent running in OTel mode #4707