[Dataset Quality] APM is not being marked as an integration

[yngrdyn]

Before #190330 this issue was pure visual problem, but now we have mitigation flows based on whether the data stream belongs to an integration or not.

The principal problem now is that we are able to offer an automagic solution, in the fix it flows, for dataset belonging to integrations while we offer "manual" support for the rest (see example bellow)

Screen.Recording.2024-10-23.at.13.00.50.mov

The problem is generated by the introduction of x-pack-apm plugin which basically installs index templates, component templates, and ingest pipelines for Elastic APM instead of relying in fleet packages.

We rely on dataStream metadata to infer the integration, e.g.

{
  "package": {
    "name": "endpoint"
  },
  ...
}

but with the introduction of the aforementioned plugin this metadata is not present for APM dataStreams.

Being APM part of elastic we should offer the best possible experience for users.

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[achyutjhunjhunwala]

Picking up from the discussions which happened on Slack, @felixbarny @flash1293

Is the below understanding correct -

Definition of Integration for Dataset Quality purposes -

Instead of - checking metadata available on the data stream API

We will - Call the index template API extracting the index template name from the data stream and see if its present or not. If its present, we will check if any of the composes_of component templates with @custom exist or not. If that happens, we will consider that DataStream to be part of an Integration.

This will solve the problem with APM Integration. I just hope we are not missing anything here as we will do change for all integration check in Dataset Quality and will not this specific for APM

[achyutjhunjhunwala]

Latest update post team discussion

Identifying if a Dataset belongs to an Integration or not

1. Call the datastream endpoint to check for 3 things and all 3 must be true - GET _data_stream/logs-apm.app.adservice-default
   • _meta.managed:true Check if the DS is managed or not
   • extract the template value and check if to have type and dataset set as part of DSNS naming convention
   • check the template value to starts with the DS name
2. If the above evaluates to true, use the index template to query the Index Template endpoint - GET _index_template/logs-apm.app@template

• The composed_of section must have a @custom component with the same name as index template without the @ part

If the above evaluates to true, we will consider the DS part of an integration or equivalent to it as this can be used for fix it flow.

[achyutjhunjhunwala]

Update due to Editor User having limited access on Serverless

• We will keep the existing integrations check in place, that is call the _data_stream API and see the meta information for _meta.package.name.

• If the above does not exists (like for APM), we will do the enhanced check documented here.

Since the enhanced check won’t work for Editor users in Serverless, due to the 1st check, Editors in serverless will see all integrations normally as integrations. Only APM integration will appear to them as Non Integration.

Regarding the Fix IT Flow, Editor users cannot click on the button as they have limited permissions. This means for all integrations, they will see the button, its just it will error out (that’s handled gracefully in current main). For APM integration, they won’t see the button, as it will not load as an Integration for them

[achyutjhunjhunwala]

This solution will only fix APM integration issue on the Dataset Quality Details page.

The Dataset Quality main page which still loads the table and merges data, displays the apm.app.service still as non integration in the table.

That would require a different fix, i believe the solution can be reused from this fix

cc: @yngrdyn

[yngrdyn]

The Dataset Quality main page which still loads the table and merges data, displays the apm.app.service still as non integration in the table

yes, this is what I was thinking when talking about APM being a special integration.
We could reuse this solution, somehow, whenever package name is empty. But, would that be expensive? What are your thoughts in that?

Also, could you create an issue so we don't forget about that one?

[achyutjhunjhunwala]

@yngrdyn I have already created and linked the issue for the main page.

The statement - "APM is a Special Integration" is wrong. It's simply not an integration unless explicitly installed via Fleet.

APM is a special package which has Component Templates and Index Templates, which are required for fix it flow, hence only in details page the logic automatically check for those things and display the Fix It Flow accordingly.

Key Points

1. What is an integration ? (Rough definition from my side)
   A) Something which has these -

• is installed using the UI/Fleet.
• When you do getAllIntegration API call from fleet, you should get it.
• It must have icons

The default APM package does not passes any of the above unless explicitly installed via the UI

Can we reuse the logic on Details Page ?

Yes the logic can be easily reused. In the state machine, we need another parallel state which does the API call and then you merge the data like its already done. You need to remove the existing Integration merge logic as its obsolete