[Fleet] /api/fleet/setup fails with HTTP 500

[mtojek]

Hi,

while executing Integration Tests in package-storage (elastic/package-storage#824 (comment)), we spotted a problem with Kibana and Elasticsearch. Kibana failed with HTTP 500 due to unavailable_shards_exception for .security shard.

The goal of this issue is to research if there is a possibility to find a mitigation for this.

Logs available at: elastic/package-storage#824 (comment)

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[mtojek]

While discussing this issue with @ph (thanks!) , he suggested to add another setting for the Elasticsearch docker image:

xpack.security.authc.realms.file.file1.order=0"

I think I will transfer this issue to the elastic-package, so we can fix it in all places.

[ph]

Just to make sure the loop is complete.

By default ES will uses indices to store the credentials, the shard needs to go green before it's available. This means that if the host machine is over capacity it can take a while for it to go green. When I was hit by this issue, I was not able to reproduce it easily. Adding timeout or weirds check never really fixed the problem.

If you look at Elasticsearch integration testing they are all using the file ream as the backend for credentials and by using them it won't need to replicate the shard and the test will be stable and fast.

This is the strategy we have used in https://github.com/elastic/beats/blob/master/x-pack/libbeat/docker-compose.yml

[jsoriano]

Please reopen if still an issue.