[Discuss] Disabling Core blocking checks during testing

[rudolf]

Since Core blocks Kibana startup based on checks against ES, integration tests which don't have a running ES server need a way to configure Core not to block on these tests. However, disabling these tests shouldn't be done in production since Kibana could break in unexpected ways, but for performance and reproducibility we want our integration test servers to run in production mode.

We currently have two such checks:

• Elasticsearch Version check Perform successful Elasticsearch version check before migrations #51311 This can be disabled in dev mode with elasticsearch.ignoreVersionMismatch=true
• Saved Object Migrations This can be disabled with migrations.skip=true in both production and dev

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[rudolf]

I can think of two options:

1. Make these configuration options available in production but document that they should only be used for testing as was done for migrations.skip
2. Create a "testing" environment which matches production but which allows for these configuration options to be set.

[pgayvallet]

Create a "testing" environment which matches production but which allows for these configuration options to be set.

We wanted to avoid this as much as possible to have tests (at least FTR) run in the 'exact' production mode, however this need start to appear quite frequently (we got the same need to expose the 'rerunMigration' API in #55860 for instance). We need to make a decision about that.

[mshustov]

We wanted to avoid this as much as possible to have tests (at least FTR) run in the 'exact' production mode, however this need start to appear quite frequently

Yeah, there is another case not related to ES #52768 (comment)
I still want to avoid divergence between prod and dev. Would this problem with saved objects migration exist in SO v2? Can we restructure our code/test env not to fail on checks? For example: why we cannot run FTR without migrations.skip? Because it might affect local indexes? Can we run elasticsearch in an isolated env instead?

[rudolf]

Would this problem with saved objects migration exist in SO v2?

I initially thought we won't block, because the migrations themselves will no longer be a blocking operation in 8.0.0 but there is still a certain amount of "setup" before we even start running plugin migrations like creating the index, checking that the mappings are correct etc. In legacy this used to be non-blocking, so plugins could initialize before the kibana indexes were ready, we would just block any saved object operations until everything was ready. There has been some bad bugs in the past where we didn't properly block these operations which led to broken deployments. So completely blocking Core until it's done was a nice guarantee that such bugs won't be possible. But maybe not blocking on any service inside Core is more inline with the async/reactive/decoupled principles we're trying to follow.

[rudolf]

The Platform team discussed this and our conclusion was that we should preferably remove the blocking behaviour. Where this isn't possible we prefer an undocumented configuration option that works in both production and development, but that prints a big warning banner to discourage users from using these options.

[pgayvallet]

For integration tests, using elasticsearch.ignoreVersionMismatch=true and migrations.skip=true when necessary has been the recommended solution.

For FTR tests, today we're still running them via "production builds", and AFAIK the need of disabling the ES version check or migration did not come up again in discussions (aside than from serverless tests and we're allowing ignoreVersionMismatch in this environment anyway).

I think we don't need this issue anymore, so I will close it. Please feel free to reopen otherwise.