[Fleet] Add support for dynamic_namespace and dynamic_dataset

[joshdover]

Summary

Closes #134971

This adds support for data_streams in packages to specify elasticsearch.dynamic_dataset and elasticsearch.dynamic_namespace in order to get wildcard privileges on the API key that is granted by Fleet Server to the agent running this integration. This is necessary for integrations that want to support centralized document routing via ingest pipelines.

I opted to keep this simple and not de-dupe the overlap between the wildcard privileges granted in one data stream with the other privileges for other data streams. Elasticsearch will already handle this correctly when defining the API key role definition (I verified manually).

A new integration test has been added with a test package that defines these options. This is necessary because there are many moving parts to the code that needed to change to support this.

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ruflin]

Is the following assumption correct:

• Set elasticsearch.dynamic_dataset: true for a logs and a metrics dataset
• Set elasticsearch.dynamic_namespace: true for a logs and a metrics dataset
• The resulting permissions are logs-*-*,metrics-*-* basically all logs and metrics (I know, more fine grained permissions will coexist which is fine)

[joshdover]

@ruflin Yes, that should be right. I can update the integration test to include a metrics data stream as well if you'd like.

[kpollich]

Code LGTM, but let's resolve the open conversation around moving this option to a higher level in the package manifest if needed.

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 75d0b58

Failed CI Steps

• FTR Configs #28

Test Failures

• [job] [logs] FTR Configs #28 / logstash pipelines delete "before all" hook: load pipelines archive for "should delete the specified pipelines"

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	994	996	+2

Unknown metric groups

API count

id	before	after	diff
fleet	1099	1101	+2

ESLint disabled line counts

id	before	after	diff
securitySolution	432	435	+3

Total ESLint disabled count

id	before	after	diff
securitySolution	512	515	+3

History

• 💚 Build #120107 succeeded 95f5fb5
• 💔 Build #120019 failed f3d0c9a
• 💔 Build #120011 failed 750deb6
• 💔 Build #119064 failed ad6c4e87a54564d5ba89dbf06df06ba14c390cc8
• 💔 Build #119054 failed 649f4c71588fd056acf3cb52861f91ce40408cf1

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream