[Dataset Quality] Change logic to identify integrations

[achyutjhunjhunwala]

Summary

• 🏁 closes: [Dataset Quality] APM is not being marked as an integration #197445
• 🏁 closes: Failing test: Serverless Observability Functional Tests.x-pack/test_serverless/functional/test_suites/observability/dataset_quality/degraded_field_flyout·ts - serverless observability UI Dataset Quality Degraded fields flyout testing root cause for ignored fields current field limit issues should let user increase the field limit for integrations #200287

What has changed ?

Previously

1. The state machine would load Dataset Settings. This is done by calling the Settings Endpoint which would get the Integration Name from the _meta tag present in ES DataStream Endpoint.
2. Then with this Integration Name, parallel call to load both Integration and Integration Dashboards will be done.

This process of getting Integration Name from _meta tag was not reliable hence required change

Currently

1. Introduced a new top level state which would call the integration/check endpoint. What this endpoint does is

• Call the datastream endpoint to check for 3 things and all 3 must be true - GET _data_stream/logs-apm.app.adservice-default
  • _meta.managed:true Check if the DS is managed or not.
  • extract the template value and check if to have type and dataset set as part of DSNS naming convention
  • check the template value to starts with the DS name
• If the above evaluates to true, use the index template to query the Index Template endpoint - GET _index_template/logs-apm.app@template
• if the above checks result in true, get all integrations and filter the integration by dataset name derived from the datastream name originally passed and return the that particular integration.

2. The above API still checks for meta field for Integration name, if its finds that, it simply returns the Integrations. Else the api would check based on the logic above and then load integration. If its integration, load it and then call the function to load integration dashboards.

3. APM App is not an integration. Its a special package with all assets

What's pending ?

• A new API was added, hence a new Deployment Agnostic Test
• FTR Stateful test
• FTR Serverless test

Any Screenshots ?

The logs-apm.app.*-* data stream now shows Fix It Flow with option to increase field limit for Users with proper rights

Want to run this locally?

Synthtrace CLI is a bit problematic to generate data for this scenario - #199280

1. Start the FTR Server, (stateful or serverless)
2. Go to degraded_field_flyout.ts file,
3. comment out the after block at the end
4. replace any of the it with it.only in the last describe block of this page.
5. Once the test execution has completed, you will have an environment running at port 5620

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[weltenwort]

@elasticmachine merge upstream

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7429

[✅] x-pack/test_serverless/functional/test_suites/observability/config.ts: 25/25 tests passed.

see run history

[tonyghiani]

LGTM, thanks for applying the suggestions I left, in particular the work to optimize the integration retrieval 👏

[weltenwort]

@elasticmachine merge upstream

[weltenwort]

@elasticmachine merge upstream

[cauemarcondes]

LGTM

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: cd42ce6
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-198692-cd42ce665bd7

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1306	1309	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
dataQuality	28.4KB	28.4KB	+50.0B
datasetQuality	250.7KB	252.0KB	+1.3KB
total			+1.4KB

Unknown metric groups

API count

id	before	after	diff
fleet	1432	1435	+3

History

• 💛 Build #253487 was flaky 3e515f5
• 💚 Build #252534 succeeded 4467d7f
• 💔 Build #252467 failed 3ea6f9a
• 💔 Build #252442 failed 38e3655
• 💚 Build #250858 succeeded cc93662

cc @weltenwort @achyutjhunjhunwala

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12125764819

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation