Add integration tests using a proxy with mTLS for control plane with Elastic Defend installed

[AndersonQ]

What does this PR do?

Add integration tests for the Elastic Agent running Elastic Defend with a mTLS proxy
See #5716 for tje covered test cases

Why is it important?

To have automated tests covering mTLS and Elastic Defend

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the [changelog tool](https://github.com/elastic/elastic-agent#changelog)
• I have added an integration test or an E2E test

Disruptive User Impact

• N/A

How to test this PR locally

In short, manually reproduce the tests described on #5716.

The long version:

• run 3 proxies:

  • mTLScli -> a proxy with mTLS to be passed as cli arguments
  • mTLSpolicy -> a proxy with mTLS configured through the policy
  • oneWayTLSpolicy -> a proxy with simple/one way TLS which defines its CA in the policy
  • all the certificates must be RSA because of an Elastic Defend restriction

• set up the proxies:

  • add a test to testing/integration/endpoint_security_test.go to configure and run the needed proxies:

func TestDebugTestProxies(t *testing.T) {
	prepareProxies(t, &url.URL{Host: "a.wrong.fleet.host"}, "the.real.fleet.host")

	t.Logf("CTRL + C to exit")
	<-make(chan struct{})
}

• run the test with:

go test -v -timeout 0 -tags integration -run TestDebugTestProxies ./testing/integration/

=== RUN   TestDebugTestProxies
    endpoint_security_test.go:1333: [mtlsCLI] certificates saved on: /tmp/TestDebugTestProxies-mtlsCLI2475114678
    endpoint_security_test.go:1333: [mtlsPolicy] certificates saved on: /tmp/TestDebugTestProxies-mtlsPolicy1317235685
    endpoint_security_test.go:1333: [oneWayTLSPolicy] certificates saved on: /tmp/TestDebugTestProxies-oneWayTLSPolicy327431283

• check the logs like [%s] certificates saved on: %s. they will show where the certificates are.
• set up the necessary fleet hosts and proxies on fleetUI
• install/enroll the agent using the configured fleet hosts and proxies. Again, check the test cases on Add an integration test using a proxy with mTLS for control plane and Elastic Defend installed #5716

Related issues

• Closes Add an integration test for --elastic-agent-cert-key-passphrase with Elastc Defend installed #5491
• Closes Add an integration test using a proxy with mTLS for control plane and Elastic Defend installed #5716

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 5716-5491-mtls-integration-test upstream/5716-5491-mtls-integration-test
git merge upstream/main
git push upstream 5716-5491-mtls-integration-test

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[AndersonQ]

folks (@blakerouse, @pchila, @kaanyalti), all the tests are passing now. In case you were waiting the tests to pass to review it

[blakerouse]

Overall this looks good.

I do question the change from returning errors to using require.NoError directly. Why was that change made?

[AndersonQ]

I do question the change from returning errors to using require.NoError directly. Why was that change made?

to follow the convention we have on our tests. The majority of the tests use this approach

[pchila]

A few comments here and there, mostly about readability

[pchila]

Nit: To help my brain tokenize the string maybe we can write it this way?

Suggested change

	func TestInstallDefendWithMTLSandEncCertKey(t *testing.T) {
	func TestInstallDefendWithmTlsAndEncCertKey(t *testing.T) {

(I know that TLS is an acronym but I think that in this case the go linter will let it slide 😄 )

[AndersonQ]

what about mTLS -> TestInstallDefendWithmTLSAndEncCertKey?

[pchila]

buff is really the last error encountered in the assert.Eventually, maybe we can define and treat it as such:

Suggested change

	buff := &strings.Builder{}
	assert.Eventuallyf(t, func() bool {
	buff.Reset()
	got, err := f.ExecInspect(ctx)
	if err != nil {
	buff.WriteString(fmt.Sprintf("error running inspect cmd: %v", err))
	return false
	}
	return proxyPolicymTLS.URL == got.Fleet.ProxyURL
	}, time.Minute, time.Second, "inspect never showed proxy from policy: %s", buff)
	// feel free to change the variable name
	var buff error
	assert.Eventuallyf(t, func() bool {
	got, err := f.ExecInspect(ctx)
	if err != nil {
	buff = fmt.Errorf("error running inspect cmd: %w", err)
	return false
	}
	// (optional) reset the error
	buff = nil
	return proxyPolicymTLS.URL == got.Fleet.ProxyURL
	}, time.Minute, time.Second, "inspect never showed proxy from policy: %s", buff)

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[pchila]

LGTM