[fleet] Integration ILM Policies have insufficient permissions to perform some actions

[graphaelli]

Kibana version: 8.2.0-SNAPSHOT

Elasticsearch version: 8.2.0-SNAPSHOT

Steps to reproduce:

1. Configure installation of integrations via kibana.yml, eg for the APM Integration
2. Wait for ILM phase changes
3. Note that some cannot be executed due to permissions issues

Expected behavior:
ILM policices are executed as specified by the integration.

Provide logs and/or server output (if relevant):
Elasticsearch logs:

security_exception: action [indices:admin/delete] is unauthorized for user [kibana_system_user] with roles [kibana_system] on indices [.ds-traces-apm-default-2022.03.17-000001], this action is granted by the index privileges [delete_index,manage,all]

Any additional context:
Opened here as I expect the impact is likely beyond APM and Fleet has a centralized view of all packages available in the package repository. The corresponding APM issue is elastic/apm-server#7568.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[axw]

I think we'll need to extend elastic/elasticsearch#81811 to cover the rest of the data streams.

@joshdover do we have longer term plans for this, so we don't need to keep updating kibana_system when we add data streams?

[joshdover]

We haven't agreed on a plan yet, but I did file an issue here for discussion: elastic/package-spec#293.

Ideally, we consider starting to explore the long-term solution to package management, which would involve Elasticsearch installing these assets directly from packages, and requiring a manage_package privilege for end users or similar.

[joshdover]

I think we'll need to extend elastic/elasticsearch#81811 to cover the rest of the data streams.

@axw would you mind handling this PR? I'm happy to review it or help as needed

[axw]

@joshdover yup, I created elastic/elasticsearch#85085

[joshdover]

How to test this manually:

• Setup 8.2 cluster, do APM managed migration in Kibana -> APM -> Settings -> Schema -> Switch to Elastic Agent
• Ingest some traces manually:
  • Download https://github.com/elastic/apm-server/blob/main/testdata/intake-v2/events.ndjson
  • Get APM server URL and secret token from Cloud UI
  • Run curl -i -H "Content-type: application/x-ndjson" -H "Authorization: Bearer <secret_token>" --data-binary @events.ndjson <APM server URL>/intake/v2/events
• Rollover the data stream manually and confirm no errors, run this from Dev Tools:

POST traces-apm-default/_rollover 
{
  "conditions": {
    "max_docs":   "0"
  }
}

[amolnater-qasource]

Hi @joshdover
We have validated this on 8.2 Snapshot Kibana cloud environment and found it fixed now.

• No errors observed on running below api from Dev tools:

POST traces-apm-default/_rollover 
{
  "conditions": {
    "max_docs":   "0"
  }
}

Build details:
BUILD: 51431
COMMIT: a743498

Screenshot:

Thanks!