Failing test: Plugin Functional Tests.test/plugin_functional/test_suites/saved_objects_hidden_type/delete·ts - Saved objects with hidden type delete "before all" hook for "should return generic 404 when trying to delete a doc with importableAndExportable types"

[kibanamachine]

A test failed on a tracked branch

Error: Forcing mocha to abort
    at Object.next (node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/run_tests.js:42:25)
    at SafeSubscriber.__tryOrUnsub (node_modules/rxjs/src/internal/Subscriber.ts:265:10)
    at SafeSubscriber.next (node_modules/rxjs/src/internal/Subscriber.ts:207:14)
    at Subscriber._next (node_modules/rxjs/src/internal/Subscriber.ts:139:22)
    at Subscriber.next (node_modules/rxjs/src/internal/Subscriber.ts:99:12)
    at RaceSubscriber.notifyNext (node_modules/rxjs/src/internal/observable/race.ts:137:22)
    at InnerSubscriber._next (node_modules/rxjs/src/internal/InnerSubscriber.ts:17:17)
    at InnerSubscriber.Subscriber.next (node_modules/rxjs/src/internal/Subscriber.ts:99:12)
    at TakeSubscriber._next (node_modules/rxjs/src/internal/operators/take.ts:92:24)
    at TakeSubscriber.Subscriber.next (node_modules/rxjs/src/internal/Subscriber.ts:99:12)
    at AbortSignal.handler (node_modules/rxjs/src/internal/observable/fromEvent.ts:201:20)
    at AbortSignal.[nodejs.internal.kHybridDispatch] (node:internal/event_target:647:20)
    at AbortSignal.dispatchEvent (node:internal/event_target:589:26)
    at abortSignal (node:internal/abort_controller:283:10)
    at AbortController.abort (node:internal/abort_controller:314:5)
    at onEarlyExit (node_modules/@kbn/test/target_node/functional_tests/tasks.js:112:21) {
  uncaught: true
}

First failure: CI Build - 7.17

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[jsanz]

@elastic/kibana-core this seems to be related with your area, please reassign if I'm wrong.

[kibanamachine]

New failure: CI Build - 7.17

[TinaHeiligers]

@jsanz you are correct with the team assignment. The tests were added in #90178

[kibanamachine]

New failure: CI Build - 7.17

[kibanamachine]

New failure: CI Build - 7.17

[mistic]

Skipped. These tests began to be flaky after an update to Chromium.

7.17: 035bffc

[gsoldevila]

Same underlying cause as #131830

These tests load documents in the .kibana index through ES archiver, in a before block.

There seems to be a race condition, probably competing against the Kibana startup migration logic.

1. ES archiver’s load operation performs a migration of the .kibana index after loading the documents into it.
2. This migration calls the /internal/saved_objects/_migrate endpoint, which runs the .kibana migrator.

This operation fails, and it seems that since the Promise rejection is not handled, brings the Kibana server down.

ERROR [migrate saved objects] request failed (attempt=1/5): read ECONNRESET
ERROR [kibana] exitted early: [kibana] exited with code 1

Internally, during this migration, one of the calls to ES fails due to an unexisting index alias:

proc [kibana] Unhandled Promise rejection detected:
[00:09:13]             │ proc [kibana] 
[00:09:13]             │ proc [kibana] ResponseError: Saved object index alias [.kibana_7.17.6] not found: index_not_found_exception: [index_not_found_exception] Reason: no such index [.kibana_7.17.6] and [require_alias] request flag is [true] and [.kibana_7.17.6] is not an alias
[00:09:13]             │ proc [kibana]     at onBody (/opt/local-ssd/buildkite/builds/kb-n2-4-29dc84655aa33b78/elastic/kibana-on-merge/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:367:23)
[00:09:13]             │ proc [kibana]     at IncomingMessage.onEnd (/opt/local-ssd/buildkite/builds/kb-n2-4-29dc84655aa33b78/elastic/kibana-on-merge/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:291:11)
[00:09:13]             │ proc [kibana]     at IncomingMessage.emit (node:events:539:35)
[00:09:13]             │ proc [kibana]     at endReadableNT (node:internal/streams/readable:1345:12)
[00:09:13]             │ proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
[00:09:13]             │ proc [kibana]   meta: {
[00:09:13]             │ proc [kibana]     body: { error: [Object], status: 404 },
[00:09:13]             │ proc [kibana]     statusCode: 404,
[00:09:13]             │ proc [kibana]     headers: {
[00:09:13]             │ proc [kibana]       'x-opaque-id': 'unknownId',
[00:09:13]             │ proc [kibana]       'x-elastic-product': 'Elasticsearch',
[00:09:13]             │ proc [kibana]       'content-type': 'application/json; charset=UTF-8',
[00:09:13]             │ proc [kibana]       'content-length': '445'
[00:09:13]             │ proc [kibana]     },
[00:09:13]             │ proc [kibana]     meta: {
[00:09:13]             │ proc [kibana]       context: null,
[00:09:13]             │ proc [kibana]       request: [Object],
[00:09:13]             │ proc [kibana]       name: 'elasticsearch-js',
[00:09:13]             │ proc [kibana]       connection: [Object],
[00:09:13]             │ proc [kibana]       attempts: 0,
[00:09:13]             │ proc [kibana]       aborted: false
[00:09:13]             │ proc [kibana]     }
[00:09:13]             │ proc [kibana]   },
[00:09:13]             │ proc [kibana]   isBoom: true,
[00:09:13]             │ proc [kibana]   isServer: true,
[00:09:13]             │ proc [kibana]   data: null,
[00:09:13]             │ proc [kibana]   output: {
[00:09:13]             │ proc [kibana]     statusCode: 500,
[00:09:13]             │ proc [kibana]     payload: {
[00:09:13]             │ proc [kibana]       message: 'An internal server error occurred',
[00:09:13]             │ proc [kibana]       statusCode: 500,
[00:09:13]             │ proc [kibana]       error: 'Internal Server Error'
[00:09:13]             │ proc [kibana]     },
[00:09:13]             │ proc [kibana]     headers: {}
[00:09:13]             │ proc [kibana]   },
[00:09:13]             │ proc [kibana]   [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/generalError'
[00:09:13]             │ proc [kibana] }
[00:09:13]             │ proc [kibana] 
[00:09:13]             │ proc [kibana] Terminating process...

I imagine that this alias is missing occasionally (flakily) due to other asynchronous actors deleting it.

• My best guess is Kibana’s startup migration logic, which runs a migrator for every index. It would seem that this logic must run BEFORE starting the tests, but I haven’t confirmed that this is executed sequentially with the proper awaits.
• Could be that ES fails for some other reason and takes Kibana down with it too.
• Could other FT be running in parallel and impacting each other?