TestFleetAgentWithoutTLS/ES_data_should_pass_validations flaky in 8-0-1

[thbkrkr]

cloud-on-k8s-operator-nightly/builds/219

🐞 TestFleetAgentWithoutTLS/ES_data_should_pass_validations ~ stack-8-0-1

=== RUN   TestFleetAgentWithoutTLS/ES_data_should_pass_validations
Retries (30m0s timeout): ....................................
  step.go:51: 
    Error Trace:	/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/utils.go:94
    Error:      	Received unexpected error:
      elasticsearch client failed for http://test-fleet-agent-notls-69s4-es-internal-http.e2e-zczhn-mercury.svc:9200/_data_stream/logs-elastic_agent.filebeat-default: 
      404 Not Found: Reason:no such index [logs-elastic_agent.filebeat-default] Type:index_not_found_exception 

[thbkrkr]

Seems caused by Error: [Elastic Agent on ECK policy] could not be added. in Kibana.

Pod log:

> grep "ERROR.*plugins" eckdiag/e2e-zczhn-mercury/pod/test-fleet-agent-notls-kcvx-kb-6bdf5d4f6f-hncrc/logs.txt

[2023-06-23T22:45:13.186+00:00][ERROR][plugins.fleet] uninstalling kubernetes-1.17.3 after error installing: 
  [Error: Encountered 2 errors creating saved objects: [
      {"type":"index-pattern","id":"logs-*","error":{"type":"conflict"}},
      {"type":"index-pattern","id":"metrics-*","error":{"type":"conflict"}}]]

[2023-06-23T22:45:15.311+00:00][ERROR][plugins.fleet] failed to uninstall or rollback package after installation error Error: 
  Saved object [epm-packages/kubernetes] not found

[2023-06-23T22:45:37.043+00:00][ERROR][plugins.fleet] Error: [Elastic Agent on ECK policy] could not be added. 
  [kubernetes] could not be installed due to error: [Error: Encountered 2 errors creating saved objects: [
      {"type":"index-pattern","id":"logs-*","error":{"type":"conflict"}},
      {"type":"index-pattern","id":"metrics-*","error":{"type":"conflict"}}]]

[thbkrkr]

This is the bug:

• [Fleet] 'conflict' creating index-pattern SO when installing package during 8.0.0 to 8.1.0 upgrade (docker only?) kibana#126611

Fixed in 8.1.x

• [Fleet] Retry Saved Object import on conflict error kibana#126900

We already hit it and disabled the tests:

• Disable Agent upgrade test due to Kibana/Fleet issue #5424
• Disable Fleet tests until predefined policy feature is fixed #5432

Then we re-enabled them:

• in Elastic Agent: Set status.ObservedGeneration from metadata.Generation #5383

  • TestAgentVersionUpgradeToLatest8x: ok because latest 8x was already 8.1.1

• in Upgrade stack version to 8.1.0 in samples and tests #5497

  • TestFleetMode: ok, 8.0.x is excluded
  • TestFleetKubernetesIntegrationRecipe: ok because run only for the latest stack version
  • TestFleetCustomLogsIntegrationRecipe: ok because run only for the latest stack version
  • TestFleetAPMIntegrationRecipe: ok because run only for the latest stack version

Edit: I forgot that we run tests with recipes only for the latest stack version.

What about TestFleetAgentWithoutTLS ? It has been added after, so it is run in 8.0.1 💥:

• Allow Fleet Server to be run without TLS. #6020