Fix component control protocol to allow checkin to be chunked across multiple messages

[blakerouse]

What does this PR do?

Fixes component control protocol to allow checkin to be chunked across multiple messages. It is backwards compatible as well, so only clients that inform the Elastic Agent that they support chunking will chunking be used when the message is larger than that max message size.

Why is it important?

Fixes a core issue where when running large amounts of inputs in a single component that the Elastic Agent was unable to communicate the information because the amount of data being sent between the Elastic Agent and the component was larger than the max message size.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test (tested in a unit test)

Author's Checklist

• Enables chunking support when communicating with the Elastic Agent.

How to test this PR locally

$ cd ../beats
$ git checkout blakerouse/agent-client-chunking
$ cd ../elastic-agent
$ mage package  # not using `EXTERNAL=true`

Related issues

• Closes Control protocol checkin payloads can exceed the gRPC maximum message size when using autodiscovery #2460

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[blakerouse]

go.mod replace will be removed once elastic/elastic-agent-client#89 is merged

[blakerouse]

go.mod replace has been removed now that v7.6.0 of the elastic-agent-client has been released.

[cmacknz]

One other thought to minimize risk when we release this, and give us an option to isolate whether chunking is causing problems: can we add a configuration option in the policy to disable chunking? Perhaps by having the agent declare that it doesn't support chunking?

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b control-protocol-chunking upstream/control-protocol-chunking
git merge upstream/main
git push upstream control-protocol-chunking

[blakerouse]

One other thought to minimize risk when we release this, and give us an option to isolate whether chunking is causing problems: can we add a configuration option in the policy to disable chunking? Perhaps by having the agent declare that it doesn't support chunking?

I added this.

[cmacknz]

Latest changes LGTM, will approve once we get passing tests.

[pierrehilbert]

The CI failure seems unrelated (#4076)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b control-protocol-chunking upstream/control-protocol-chunking
git merge upstream/main
git push upstream control-protocol-chunking

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
76.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube