esArchiver fails with no_shard_available_action_exception after deleting indexes #158918
Assignees
Labels
Epic:ScaleMigrations
Scale upgrade migrations to millions of saved objects
Feature:Migrations
Feature:Saved Objects
Team:Core
Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Comments
dmlemeshko
added
the
Team:Core
|
Pinging @elastic/kibana-core (Team:Core) |
|
One possible strategy could be to update This way, any asynchronous calls made by Kibana would either not see the index and return 404, or have the index on a "ready" state. |
9 tasks
jeramysoucy
added a commit
to jeramysoucy/kibana
that referenced
this issue
Jun 13, 2023
…ble_action (elastic#159039) ## Summary Skips several tests failing with the `no_shard_available_action_exception`. Also adds comments noting the related test failure issues and the [master issue being tracked](elastic#158918). See issue elastic#158918 and PR elastic#159002. (cherry picked from commit 63bd0ba) # Conflicts: # x-pack/test/saved_object_api_integration/spaces_only/apis/resolve_import_errors.ts # x-pack/test/spaces_api_integration/spaces_only/apis/get.ts # x-pack/test/spaces_api_integration/spaces_only/apis/update.ts
jeramysoucy
added a commit
that referenced
this issue
Jun 13, 2023
…available_action (#159039) (#159546) # Backport This will backport the following commits from `main` to `8.8`: - [[Failing Tests] Skips and comments tests failing with no_shard_available_action (#159039)](#159039) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Jeramy Soucy","email":"jeramy.soucy@elastic.co"},"sourceCommit":{"committedDate":"2023-06-06T07:11:22Z","message":"[Failing Tests] Skips and comments tests failing with no_shard_available_action (#159039)\n\n## Summary\r\n\r\nSkips several tests failing with the\r\n`no_shard_available_action_exception`. Also adds comments noting the\r\nrelated test failure issues and the [master issue being\r\ntracked](https://github.com/elastic/kibana/issues/158918).\r\n\r\nSee issue #158918 and PR #159002.","sha":"63bd0ba9f9cec244cac38affc2bc572ad5f6a6aa","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Feature:Saved Objects","Feature:Security/Spaces","skipped-test","release_note:skip","backport:prev-minor","v8.9.0"],"number":159039,"url":"https://github.com/elastic/kibana/pull/159039","mergeCommit":{"message":"[Failing Tests] Skips and comments tests failing with no_shard_available_action (#159039)\n\n## Summary\r\n\r\nSkips several tests failing with the\r\n`no_shard_available_action_exception`. Also adds comments noting the\r\nrelated test failure issues and the [master issue being\r\ntracked](https://github.com/elastic/kibana/issues/158918).\r\n\r\nSee issue #158918 and PR #159002.","sha":"63bd0ba9f9cec244cac38affc2bc572ad5f6a6aa"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/159039","number":159039,"mergeCommit":{"message":"[Failing Tests] Skips and comments tests failing with no_shard_available_action (#159039)\n\n## Summary\r\n\r\nSkips several tests failing with the\r\n`no_shard_available_action_exception`. Also adds comments noting the\r\nrelated test failure issues and the [master issue being\r\ntracked](https://github.com/elastic/kibana/issues/158918).\r\n\r\nSee issue #158918 and PR #159002.","sha":"63bd0ba9f9cec244cac38affc2bc572ad5f6a6aa"}}]}] BACKPORT-->
gsoldevila
added a commit
that referenced
this issue
Jun 19, 2023
…ound in data.json (#159582) The ultimate goal of this PR is to lay the groundwork to be able to remove the "dynamic" `mappings.json`, which probably should have never existed. With the PR, detecting SO documents in the `data.json` will automatically trigger a cleanup of the SO indices. This, in turn, will allow not having to define "dynamic" saved objects indices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the `mappings.json`). IIUC the idea behind the dynamic indices was to have SO indices that are aligned with the current stack version, avoiding the extra overhead of having to migrate the inserted documents, and reducing overall test times. Nonetheless, what is happening today is: 1. FTR starts ES and Kibana. 2. Kibana creates current version SO indices at startup (empty ones). 3. `esArchiver.load()` processes the `mappings.json`. 3.1. It detects that we are defining SO indices and **deletes** existing saved object indices. 3.2 It then re-creates these indices according to the definitions on `mappings.json`. 4. `esArchiver.load()` processes the `data.json`. Specifically, it inserts SO documents present in `data.json`. 5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the inserted documents are up-to-date, hoping they are already aligned with current stack version (which is not always the case, not even with "dynamic" mappings). Two interesting things to note: - Steps 3 to 5 happen whilst Kibana is already started and running. If Kibana queries SO indices during `esArchiver.load()`, and a request to ES is made **right after** 3.2, the result might be #158918. - Having dynamic SO indices' definitions, deleting the "official" indices created by Kibana (3.1), and recreating them hoping to be aligned with current stack version (3.2) is non-sense. We could use the existing SO indices instead, and simply clean them up whenever we are about to insert SO documents. Performing that cleanup is precisely the goal of this PR. Then, in subsequent PRs like https://github.com/elastic/kibana/pull/159397/files, tackling the flaky tests, we'll be able to simply remove the "dynamic" `mappings.json` definitions, causing `esArchiver` to rely on SO indices created by Kibana. Thanks to this PR, the FTR tests won't need to explicitly cleanup saved object indices in the `before` hooks.
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Jun 19, 2023
…ound in data.json (elastic#159582) The ultimate goal of this PR is to lay the groundwork to be able to remove the "dynamic" `mappings.json`, which probably should have never existed. With the PR, detecting SO documents in the `data.json` will automatically trigger a cleanup of the SO indices. This, in turn, will allow not having to define "dynamic" saved objects indices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the `mappings.json`). IIUC the idea behind the dynamic indices was to have SO indices that are aligned with the current stack version, avoiding the extra overhead of having to migrate the inserted documents, and reducing overall test times. Nonetheless, what is happening today is: 1. FTR starts ES and Kibana. 2. Kibana creates current version SO indices at startup (empty ones). 3. `esArchiver.load()` processes the `mappings.json`. 3.1. It detects that we are defining SO indices and **deletes** existing saved object indices. 3.2 It then re-creates these indices according to the definitions on `mappings.json`. 4. `esArchiver.load()` processes the `data.json`. Specifically, it inserts SO documents present in `data.json`. 5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the inserted documents are up-to-date, hoping they are already aligned with current stack version (which is not always the case, not even with "dynamic" mappings). Two interesting things to note: - Steps 3 to 5 happen whilst Kibana is already started and running. If Kibana queries SO indices during `esArchiver.load()`, and a request to ES is made **right after** 3.2, the result might be elastic#158918. - Having dynamic SO indices' definitions, deleting the "official" indices created by Kibana (3.1), and recreating them hoping to be aligned with current stack version (3.2) is non-sense. We could use the existing SO indices instead, and simply clean them up whenever we are about to insert SO documents. Performing that cleanup is precisely the goal of this PR. Then, in subsequent PRs like https://github.com/elastic/kibana/pull/159397/files, tackling the flaky tests, we'll be able to simply remove the "dynamic" `mappings.json` definitions, causing `esArchiver` to rely on SO indices created by Kibana. Thanks to this PR, the FTR tests won't need to explicitly cleanup saved object indices in the `before` hooks. (cherry picked from commit bbb5fc4)
kibanamachine
added a commit
that referenced
this issue
Jun 19, 2023
… are found in data.json (#159582) (#159910) # Backport This will backport the following commits from `main` to `8.8`: - [[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)](#159582) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gerard Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2023-06-19T11:08:03Z","message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Team:Operations","technical debt","release_note:skip","backport:prev-minor","v8.9.0","FTR","v8.8.2"],"number":159582,"url":"https://github.com/elastic/kibana/pull/159582","mergeCommit":{"message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa"}},"sourceBranch":"main","suggestedTargetBranches":["8.8"],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/159582","number":159582,"mergeCommit":{"message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa"}},{"branch":"8.8","label":"v8.8.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
gsoldevila
added a commit
that referenced
this issue
Jun 29, 2023
Tackles #158918 Updates `esArchiver` so that SO indices are created in two separate steps: `indices.create()` and `indices.updateAliases()` This way, any Kibana requests that target SO indices (through their aliases) will either find that the indices exist, or that they do not. This is a less invasive approach than #159397, as it does not modify the `esArchiver.load` flow (we don't delete the `mappings.json` files here).
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Jun 29, 2023
Tackles elastic#158918 Updates `esArchiver` so that SO indices are created in two separate steps: `indices.create()` and `indices.updateAliases()` This way, any Kibana requests that target SO indices (through their aliases) will either find that the indices exist, or that they do not. This is a less invasive approach than elastic#159397, as it does not modify the `esArchiver.load` flow (we don't delete the `mappings.json` files here). (cherry picked from commit 295b4d4)
kibanamachine
referenced
this issue
Jun 29, 2023
#160853) # Backport This will backport the following commits from `main` to `8.9`: - [[esArchiver] Update aliases after creating the indices (#160584)](#160584) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gerard Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2023-06-29T09:30:15Z","message":"[esArchiver] Update aliases after creating the indices (#160584)\n\nTackles https://github.com/elastic/kibana/issues/158918\r\n\r\nUpdates `esArchiver` so that SO indices are created in two separate\r\nsteps:\r\n\r\n`indices.create()` and `indices.updateAliases()`\r\n\r\nThis way, any Kibana requests that target SO indices (through their\r\naliases) will either find that the indices exist, or that they do not.\r\n\r\nThis is a less invasive approach than\r\nhttps://github.com//pull/159397, as it does not modify the\r\n`esArchiver.load` flow (we don't delete the `mappings.json` files here).","sha":"295b4d4a34bf147ba2d270833355d7c9e6839cc8","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Team:Operations","release_note:skip","test-failure-flaky","backport:prev-minor","v8.9.0","v8.10.0"],"number":160584,"url":"https://github.com/elastic/kibana/pull/160584","mergeCommit":{"message":"[esArchiver] Update aliases after creating the indices (#160584)\n\nTackles https://github.com/elastic/kibana/issues/158918\r\n\r\nUpdates `esArchiver` so that SO indices are created in two separate\r\nsteps:\r\n\r\n`indices.create()` and `indices.updateAliases()`\r\n\r\nThis way, any Kibana requests that target SO indices (through their\r\naliases) will either find that the indices exist, or that they do not.\r\n\r\nThis is a less invasive approach than\r\nhttps://github.com//pull/159397, as it does not modify the\r\n`esArchiver.load` flow (we don't delete the `mappings.json` files here).","sha":"295b4d4a34bf147ba2d270833355d7c9e6839cc8"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/160584","number":160584,"mergeCommit":{"message":"[esArchiver] Update aliases after creating the indices (#160584)\n\nTackles https://github.com/elastic/kibana/issues/158918\r\n\r\nUpdates `esArchiver` so that SO indices are created in two separate\r\nsteps:\r\n\r\n`indices.create()` and `indices.updateAliases()`\r\n\r\nThis way, any Kibana requests that target SO indices (through their\r\naliases) will either find that the indices exist, or that they do not.\r\n\r\nThis is a less invasive approach than\r\nhttps://github.com//pull/159397, as it does not modify the\r\n`esArchiver.load` flow (we don't delete the `mappings.json` files here).","sha":"295b4d4a34bf147ba2d270833355d7c9e6839cc8"}}]}] BACKPORT--> Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co> Co-authored-by: Alex Szabo <alex.szabo@elastic.co>
gsoldevila
added a commit
that referenced
this issue
Jul 7, 2023
## Summary Addresses root cause of #158918 Underlying cause is that _esArchiver_ is messing up with the SO indices whilst Kibana is already running. This can cause some asynchronous calls made by Kibana (e.g. `GET /.kibana_8.8.0/telemetry:telemetry`) to hit ES at the exact time where the underlying SO indices are **just** recreated, causing the error described in the related issue. The idea of the fix is to delete `mappings.json`, used by _esArchiver_ to create the SO indices. This way, _esArchiver_ will use existing SO indices instead (aka the "official" ones, created by Kibana at startup), thus avoiding the problem altogether. As a side effect: - Documents in `data.json` must be updated so that they are correctly inserted. - The different FTR tests must make sure the SO indices are empty before inserting those documents (done in the `before(), beforeEach()` statements).
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Jul 7, 2023
## Summary Addresses root cause of elastic#158918 Underlying cause is that _esArchiver_ is messing up with the SO indices whilst Kibana is already running. This can cause some asynchronous calls made by Kibana (e.g. `GET /.kibana_8.8.0/telemetry:telemetry`) to hit ES at the exact time where the underlying SO indices are **just** recreated, causing the error described in the related issue. The idea of the fix is to delete `mappings.json`, used by _esArchiver_ to create the SO indices. This way, _esArchiver_ will use existing SO indices instead (aka the "official" ones, created by Kibana at startup), thus avoiding the problem altogether. As a side effect: - Documents in `data.json` must be updated so that they are correctly inserted. - The different FTR tests must make sure the SO indices are empty before inserting those documents (done in the `before(), beforeEach()` statements). (cherry picked from commit 79f7bb4)
jeramysoucy
added a commit
that referenced
this issue
Jul 7, 2023
Closes #155723,#155846,#156045,#156041,#156074,#156130,#156151,#156739,#157673,#156422,#156998,#156922,#156921,#157452,#158366,#158586,#158711 ## Summary Issue was resolved by PR #160584. This PR unskips saved object tests documented as skipped in #158918. ### Testing Flaky test runner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2543
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Jul 7, 2023
Closes elastic#155723,elastic#155846,elastic#156045,elastic#156041,elastic#156074,elastic#156130,elastic#156151,elastic#156739,elastic#157673,elastic#156422,elastic#156998,elastic#156922,elastic#156921,elastic#157452,elastic#158366,elastic#158586,elastic#158711 ## Summary Issue was resolved by PR elastic#160584. This PR unskips saved object tests documented as skipped in elastic#158918. ### Testing Flaky test runner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2543 (cherry picked from commit 4fc4ded)
kibanamachine
referenced
this issue
Jul 7, 2023
…1468) # Backport This will backport the following commits from `main` to `8.9`: - [Attempt to fix flaky tests after split migration (#159397)](#159397) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gerard Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2023-07-07T15:05:42Z","message":"Attempt to fix flaky tests after split migration (#159397)\n\n## Summary\r\nAddresses root cause of https://github.com/elastic/kibana/issues/158918\r\n\r\nUnderlying cause is that _esArchiver_ is messing up with the SO indices\r\nwhilst Kibana is already running.\r\n\r\nThis can cause some asynchronous calls made by Kibana (e.g. `GET\r\n/.kibana_8.8.0/telemetry:telemetry`) to hit ES at the exact time where\r\nthe underlying SO indices are **just** recreated, causing the error\r\ndescribed in the related issue.\r\n\r\nThe idea of the fix is to delete `mappings.json`, used by _esArchiver_\r\nto create the SO indices. This way, _esArchiver_ will use existing SO\r\nindices instead (aka the \"official\" ones, created by Kibana at startup),\r\nthus avoiding the problem altogether.\r\n\r\nAs a side effect:\r\n\r\n- Documents in `data.json` must be updated so that they are correctly\r\ninserted.\r\n- The different FTR tests must make sure the SO indices are empty before\r\ninserting those documents (done in the `before(), beforeEach()`\r\nstatements).","sha":"79f7bb45fd44e5c25302dcc23e73c0ff0cda52d0","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","Team:Core","technical debt","release_note:skip","test-failure-flaky","Team:Security Solution Platform","backport:prev-minor","backport:prev-MAJOR","backport:all-open","v8.9.0","FTR","v8.10.0","v7.17.12"],"number":159397,"url":"https://github.com/elastic/kibana/pull/159397","mergeCommit":{"message":"Attempt to fix flaky tests after split migration (#159397)\n\n## Summary\r\nAddresses root cause of https://github.com/elastic/kibana/issues/158918\r\n\r\nUnderlying cause is that _esArchiver_ is messing up with the SO indices\r\nwhilst Kibana is already running.\r\n\r\nThis can cause some asynchronous calls made by Kibana (e.g. `GET\r\n/.kibana_8.8.0/telemetry:telemetry`) to hit ES at the exact time where\r\nthe underlying SO indices are **just** recreated, causing the error\r\ndescribed in the related issue.\r\n\r\nThe idea of the fix is to delete `mappings.json`, used by _esArchiver_\r\nto create the SO indices. This way, _esArchiver_ will use existing SO\r\nindices instead (aka the \"official\" ones, created by Kibana at startup),\r\nthus avoiding the problem altogether.\r\n\r\nAs a side effect:\r\n\r\n- Documents in `data.json` must be updated so that they are correctly\r\ninserted.\r\n- The different FTR tests must make sure the SO indices are empty before\r\ninserting those documents (done in the `before(), beforeEach()`\r\nstatements).","sha":"79f7bb45fd44e5c25302dcc23e73c0ff0cda52d0"}},"sourceBranch":"main","suggestedTargetBranches":["8.9","7.17"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/159397","number":159397,"mergeCommit":{"message":"Attempt to fix flaky tests after split migration (#159397)\n\n## Summary\r\nAddresses root cause of https://github.com/elastic/kibana/issues/158918\r\n\r\nUnderlying cause is that _esArchiver_ is messing up with the SO indices\r\nwhilst Kibana is already running.\r\n\r\nThis can cause some asynchronous calls made by Kibana (e.g. `GET\r\n/.kibana_8.8.0/telemetry:telemetry`) to hit ES at the exact time where\r\nthe underlying SO indices are **just** recreated, causing the error\r\ndescribed in the related issue.\r\n\r\nThe idea of the fix is to delete `mappings.json`, used by _esArchiver_\r\nto create the SO indices. This way, _esArchiver_ will use existing SO\r\nindices instead (aka the \"official\" ones, created by Kibana at startup),\r\nthus avoiding the problem altogether.\r\n\r\nAs a side effect:\r\n\r\n- Documents in `data.json` must be updated so that they are correctly\r\ninserted.\r\n- The different FTR tests must make sure the SO indices are empty before\r\ninserting those documents (done in the `before(), beforeEach()`\r\nstatements).","sha":"79f7bb45fd44e5c25302dcc23e73c0ff0cda52d0"}},{"branch":"7.17","label":"v7.17.12","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
kibanamachine
added a commit
that referenced
this issue
Jul 7, 2023
…161472) # Backport This will backport the following commits from `main` to `8.9`: - [Unskips tests related to no_shard_available failures (#161109)](#161109) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Jeramy Soucy","email":"jeramy.soucy@elastic.co"},"sourceCommit":{"committedDate":"2023-07-07T15:34:47Z","message":"Unskips tests related to no_shard_available failures (#161109)\n\nCloses\r\n#155723,#155846,#156045,#156041,#156074,#156130,#156151,#156739,#157673,#156422,#156998,#156922,#156921,#157452,#158366,#158586,#158711\r\n\r\n## Summary\r\n\r\nIssue was resolved by PR #160584. This PR unskips saved object tests\r\ndocumented as skipped in #158918.\r\n\r\n### Testing\r\nFlaky test runner:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2543","sha":"4fc4ded06bc2289c4506941d07942952cef29657","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Security","release_note:skip","test-failure-flaky","backport:prev-minor","v8.9.0","v8.10.0"],"number":161109,"url":"https://github.com/elastic/kibana/pull/161109","mergeCommit":{"message":"Unskips tests related to no_shard_available failures (#161109)\n\nCloses\r\n#155723,#155846,#156045,#156041,#156074,#156130,#156151,#156739,#157673,#156422,#156998,#156922,#156921,#157452,#158366,#158586,#158711\r\n\r\n## Summary\r\n\r\nIssue was resolved by PR #160584. This PR unskips saved object tests\r\ndocumented as skipped in #158918.\r\n\r\n### Testing\r\nFlaky test runner:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2543","sha":"4fc4ded06bc2289c4506941d07942952cef29657"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/161109","number":161109,"mergeCommit":{"message":"Unskips tests related to no_shard_available failures (#161109)\n\nCloses\r\n#155723,#155846,#156045,#156041,#156074,#156130,#156151,#156739,#157673,#156422,#156998,#156922,#156921,#157452,#158366,#158586,#158711\r\n\r\n## Summary\r\n\r\nIssue was resolved by PR #160584. This PR unskips saved object tests\r\ndocumented as skipped in #158918.\r\n\r\n### Testing\r\nFlaky test runner:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2543","sha":"4fc4ded06bc2289c4506941d07942952cef29657"}}]}] BACKPORT--> Co-authored-by: Jeramy Soucy <jeramy.soucy@elastic.co>
|
I merged two fixes that should tackle this flakiness:
@jeramysoucy unskipped many of the tests with #161109. |
|
There are 7 archives that are still defining "dynamic" SO indices, i.e. indices that aim at being at the latest version, by using the Also ATM, there are a total of 49
|
This was referenced Jul 13, 2023
Merged
gsoldevila
added a commit
to gsoldevila/kibana
that referenced
this issue
Aug 4, 2023
…ound in data.json (elastic#159582) The ultimate goal of this PR is to lay the groundwork to be able to remove the "dynamic" `mappings.json`, which probably should have never existed. With the PR, detecting SO documents in the `data.json` will automatically trigger a cleanup of the SO indices. This, in turn, will allow not having to define "dynamic" saved objects indices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the `mappings.json`). IIUC the idea behind the dynamic indices was to have SO indices that are aligned with the current stack version, avoiding the extra overhead of having to migrate the inserted documents, and reducing overall test times. Nonetheless, what is happening today is: 1. FTR starts ES and Kibana. 2. Kibana creates current version SO indices at startup (empty ones). 3. `esArchiver.load()` processes the `mappings.json`. 3.1. It detects that we are defining SO indices and **deletes** existing saved object indices. 3.2 It then re-creates these indices according to the definitions on `mappings.json`. 4. `esArchiver.load()` processes the `data.json`. Specifically, it inserts SO documents present in `data.json`. 5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the inserted documents are up-to-date, hoping they are already aligned with current stack version (which is not always the case, not even with "dynamic" mappings). Two interesting things to note: - Steps 3 to 5 happen whilst Kibana is already started and running. If Kibana queries SO indices during `esArchiver.load()`, and a request to ES is made **right after** 3.2, the result might be elastic#158918. - Having dynamic SO indices' definitions, deleting the "official" indices created by Kibana (3.1), and recreating them hoping to be aligned with current stack version (3.2) is non-sense. We could use the existing SO indices instead, and simply clean them up whenever we are about to insert SO documents. Performing that cleanup is precisely the goal of this PR. Then, in subsequent PRs like https://github.com/elastic/kibana/pull/159397/files, tackling the flaky tests, we'll be able to simply remove the "dynamic" `mappings.json` definitions, causing `esArchiver` to rely on SO indices created by Kibana. Thanks to this PR, the FTR tests won't need to explicitly cleanup saved object indices in the `before` hooks. (cherry picked from commit bbb5fc4) # Conflicts: # packages/kbn-es-archiver/src/lib/indices/create_index_stream.test.mock.ts # packages/kbn-es-archiver/src/lib/indices/create_index_stream.test.ts # packages/kbn-es-archiver/src/lib/indices/create_index_stream.ts # packages/kbn-es-archiver/src/lib/indices/delete_index_stream.ts # packages/kbn-es-archiver/src/lib/indices/kibana_index.ts
gsoldevila
added a commit
that referenced
this issue
Aug 7, 2023
…s are found in data.json (#159582) (#163178) # Backport This will backport the following commits from `main` to `7.17`: - [[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)](#159582) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gerard Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2023-06-19T11:08:03Z","message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Team:Operations","technical debt","release_note:skip","backport:prev-minor","backport:prev-MAJOR","v8.9.0","FTR","v8.8.2","v8.10.0","v7.17.13"],"number":159582,"url":"https://github.com/elastic/kibana/pull/159582","mergeCommit":{"message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa"}},"sourceBranch":"main","suggestedTargetBranches":["7.17"],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/159582","number":159582,"mergeCommit":{"message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa"}},{"branch":"8.8","label":"v8.8.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/159910","number":159910,"state":"MERGED","mergeCommit":{"sha":"d2683cba7bfaff545a90c5a7daccb0b731fa1ba0","message":"[8.8] [esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582) (#159910)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.8`:\n- [[esArchiver] Automatically cleanup SO indices when SO documents are\nfound in data.json\n(#159582)](https://github.com/elastic/kibana/pull/159582)\n\n<!--- Backport version: 8.9.7 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Gerard\nSoldevila\",\"email\":\"gerard.soldevila@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2023-06-19T11:08:03Z\",\"message\":\"[esArchiver]\nAutomatically cleanup SO indices when SO documents are found in\ndata.json (#159582)\\n\\nThe ultimate goal of this PR is to lay the\ngroundwork to be able to\\r\\nremove the \\\"dynamic\\\" `mappings.json`,\nwhich probably should have never\\r\\nexisted.\\r\\n\\r\\nWith the PR,\ndetecting SO documents in the `data.json` will\\r\\nautomatically trigger\na cleanup of the SO indices.\\r\\nThis, in turn, will allow not having to\ndefine \\\"dynamic\\\" saved objects\\r\\nindices (i.e. those with the\n`$KIBANA_PACKAGE_VERSION` variable in\nthe\\r\\n`mappings.json`).\\r\\n\\r\\nIIUC the idea behind the dynamic indices\nwas to have SO indices that are\\r\\naligned with the current stack\nversion, avoiding the extra overhead of\\r\\nhaving to migrate the\ninserted documents, and reducing overall\ntest\\r\\ntimes.\\r\\n\\r\\nNonetheless, what is happening today is:\\r\\n1. FTR\nstarts ES and Kibana.\\r\\n2. Kibana creates current version SO indices at\nstartup (empty ones).\\r\\n3. `esArchiver.load()` processes the\n`mappings.json`.\\r\\n3.1. It detects that we are defining SO indices and\n**deletes** existing\\r\\nsaved object indices.\\r\\n3.2 It then re-creates\nthese indices according to the definitions on\\r\\n`mappings.json`.\\r\\n4.\n`esArchiver.load()` processes the `data.json`. Specifically,\nit\\r\\ninserts SO documents present in `data.json`.\\r\\n5.\n`esArchiver.load()` calls the _KibanaMigrator_ to make sure that\nthe\\r\\ninserted documents are up-to-date, hoping they are already\naligned with\\r\\ncurrent stack version (which is not always the case, not\neven with\\r\\n\\\"dynamic\\\" mappings).\\r\\n\\r\\nTwo interesting things to\nnote:\\r\\n- Steps 3 to 5 happen whilst Kibana is already started and\nrunning. If\\r\\nKibana queries SO indices during `esArchiver.load()`, and\na request to\\r\\nES is made **right after** 3.2, the result might\nbe\\r\\nhttps://github.com//issues/158918.\\r\\n- Having\ndynamic SO indices' definitions, deleting the \\\"official\\\"\\r\\nindices\ncreated by Kibana (3.1), and recreating them hoping to be\\r\\naligned\nwith current stack version (3.2) is non-sense. We could use\nthe\\r\\nexisting SO indices instead, and simply clean them up whenever we\nare\\r\\nabout to insert SO documents.\\r\\n\\r\\nPerforming that cleanup is\nprecisely the goal of this PR.\\r\\nThen, in subsequent PRs\nlike\\r\\nhttps://github.com//pull/159397/files, tackling\nthe flaky\\r\\ntests, we'll be able to simply remove the \\\"dynamic\\\"\n`mappings.json`\\r\\ndefinitions, causing `esArchiver` to rely on SO\nindices created by\\r\\nKibana.\\r\\n\\r\\nThanks to this PR, the FTR tests\nwon't need to explicitly cleanup saved\\r\\nobject indices in the `before`\nhooks.\",\"sha\":\"bbb5fc4abe7dd530d8248a09a9638cd3438202aa\",\"branchLabelMapping\":{\"^v8.9.0$\":\"main\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"Team:Core\",\"Team:Operations\",\"technical\ndebt\",\"release_note:skip\",\"backport:prev-minor\",\"v8.9.0\",\"FTR\",\"v8.8.2\"],\"number\":159582,\"url\":\"https://github.com/elastic/kibana/pull/159582\",\"mergeCommit\":{\"message\":\"[esArchiver]\nAutomatically cleanup SO indices when SO documents are found in\ndata.json (#159582)\\n\\nThe ultimate goal of this PR is to lay the\ngroundwork to be able to\\r\\nremove the \\\"dynamic\\\" `mappings.json`,\nwhich probably should have never\\r\\nexisted.\\r\\n\\r\\nWith the PR,\ndetecting SO documents in the `data.json` will\\r\\nautomatically trigger\na cleanup of the SO indices.\\r\\nThis, in turn, will allow not having to\ndefine \\\"dynamic\\\" saved objects\\r\\nindices (i.e. those with the\n`$KIBANA_PACKAGE_VERSION` variable in\nthe\\r\\n`mappings.json`).\\r\\n\\r\\nIIUC the idea behind the dynamic indices\nwas to have SO indices that are\\r\\naligned with the current stack\nversion, avoiding the extra overhead of\\r\\nhaving to migrate the\ninserted documents, and reducing overall\ntest\\r\\ntimes.\\r\\n\\r\\nNonetheless, what is happening today is:\\r\\n1. FTR\nstarts ES and Kibana.\\r\\n2. Kibana creates current version SO indices at\nstartup (empty ones).\\r\\n3. `esArchiver.load()` processes the\n`mappings.json`.\\r\\n3.1. It detects that we are defining SO indices and\n**deletes** existing\\r\\nsaved object indices.\\r\\n3.2 It then re-creates\nthese indices according to the definitions on\\r\\n`mappings.json`.\\r\\n4.\n`esArchiver.load()` processes the `data.json`. Specifically,\nit\\r\\ninserts SO documents present in `data.json`.\\r\\n5.\n`esArchiver.load()` calls the _KibanaMigrator_ to make sure that\nthe\\r\\ninserted documents are up-to-date, hoping they are already\naligned with\\r\\ncurrent stack version (which is not always the case, not\neven with\\r\\n\\\"dynamic\\\" mappings).\\r\\n\\r\\nTwo interesting things to\nnote:\\r\\n- Steps 3 to 5 happen whilst Kibana is already started and\nrunning. If\\r\\nKibana queries SO indices during `esArchiver.load()`, and\na request to\\r\\nES is made **right after** 3.2, the result might\nbe\\r\\nhttps://github.com//issues/158918.\\r\\n- Having\ndynamic SO indices' definitions, deleting the \\\"official\\\"\\r\\nindices\ncreated by Kibana (3.1), and recreating them hoping to be\\r\\naligned\nwith current stack version (3.2) is non-sense. We could use\nthe\\r\\nexisting SO indices instead, and simply clean them up whenever we\nare\\r\\nabout to insert SO documents.\\r\\n\\r\\nPerforming that cleanup is\nprecisely the goal of this PR.\\r\\nThen, in subsequent PRs\nlike\\r\\nhttps://github.com//pull/159397/files, tackling\nthe flaky\\r\\ntests, we'll be able to simply remove the \\\"dynamic\\\"\n`mappings.json`\\r\\ndefinitions, causing `esArchiver` to rely on SO\nindices created by\\r\\nKibana.\\r\\n\\r\\nThanks to this PR, the FTR tests\nwon't need to explicitly cleanup saved\\r\\nobject indices in the `before`\nhooks.\",\"sha\":\"bbb5fc4abe7dd530d8248a09a9638cd3438202aa\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"8.8\"],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v8.9.0\",\"labelRegex\":\"^v8.9.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/159582\",\"number\":159582,\"mergeCommit\":{\"message\":\"[esArchiver]\nAutomatically cleanup SO indices when SO documents are found in\ndata.json (#159582)\\n\\nThe ultimate goal of this PR is to lay the\ngroundwork to be able to\\r\\nremove the \\\"dynamic\\\" `mappings.json`,\nwhich probably should have never\\r\\nexisted.\\r\\n\\r\\nWith the PR,\ndetecting SO documents in the `data.json` will\\r\\nautomatically trigger\na cleanup of the SO indices.\\r\\nThis, in turn, will allow not having to\ndefine \\\"dynamic\\\" saved objects\\r\\nindices (i.e. those with the\n`$KIBANA_PACKAGE_VERSION` variable in\nthe\\r\\n`mappings.json`).\\r\\n\\r\\nIIUC the idea behind the dynamic indices\nwas to have SO indices that are\\r\\naligned with the current stack\nversion, avoiding the extra overhead of\\r\\nhaving to migrate the\ninserted documents, and reducing overall\ntest\\r\\ntimes.\\r\\n\\r\\nNonetheless, what is happening today is:\\r\\n1. FTR\nstarts ES and Kibana.\\r\\n2. Kibana creates current version SO indices at\nstartup (empty ones).\\r\\n3. `esArchiver.load()` processes the\n`mappings.json`.\\r\\n3.1. It detects that we are defining SO indices and\n**deletes** existing\\r\\nsaved object indices.\\r\\n3.2 It then re-creates\nthese indices according to the definitions on\\r\\n`mappings.json`.\\r\\n4.\n`esArchiver.load()` processes the `data.json`. Specifically,\nit\\r\\ninserts SO documents present in `data.json`.\\r\\n5.\n`esArchiver.load()` calls the _KibanaMigrator_ to make sure that\nthe\\r\\ninserted documents are up-to-date, hoping they are already\naligned with\\r\\ncurrent stack version (which is not always the case, not\neven with\\r\\n\\\"dynamic\\\" mappings).\\r\\n\\r\\nTwo interesting things to\nnote:\\r\\n- Steps 3 to 5 happen whilst Kibana is already started and\nrunning. If\\r\\nKibana queries SO indices during `esArchiver.load()`, and\na request to\\r\\nES is made **right after** 3.2, the result might\nbe\\r\\nhttps://github.com//issues/158918.\\r\\n- Having\ndynamic SO indices' definitions, deleting the \\\"official\\\"\\r\\nindices\ncreated by Kibana (3.1), and recreating them hoping to be\\r\\naligned\nwith current stack version (3.2) is non-sense. We could use\nthe\\r\\nexisting SO indices instead, and simply clean them up whenever we\nare\\r\\nabout to insert SO documents.\\r\\n\\r\\nPerforming that cleanup is\nprecisely the goal of this PR.\\r\\nThen, in subsequent PRs\nlike\\r\\nhttps://github.com//pull/159397/files, tackling\nthe flaky\\r\\ntests, we'll be able to simply remove the \\\"dynamic\\\"\n`mappings.json`\\r\\ndefinitions, causing `esArchiver` to rely on SO\nindices created by\\r\\nKibana.\\r\\n\\r\\nThanks to this PR, the FTR tests\nwon't need to explicitly cleanup saved\\r\\nobject indices in the `before`\nhooks.\",\"sha\":\"bbb5fc4abe7dd530d8248a09a9638cd3438202aa\"}},{\"branch\":\"8.8\",\"label\":\"v8.8.2\",\"labelRegex\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>"}},{"branch":"8.10","label":"v8.10.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"7.17","label":"v7.17.13","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
|
All of the tests' issues that are actually related have been unskipped + closed. |
rudolf
added
the
Epic:ScaleMigrations
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Related to #154888
Skipped tests:
#156998 (#156922, #156921)
#156739 (#157673)
#156151
#156130 (#156074)
#155846 (#156045, #156041)
#155723 (#156422)
#131830#131871#157452
#158711 (#158366)
#158586
Probably more issues to be discovered, usually has
Error: Forcing mocha to abortin description and more details in buildkite log will point tono_shard_available_action_exceptionAfter the PR was merged, multiple API tests became flaky on CI, e.g. x-pack/test/spaces_api_integration/spaces_only/apis/update_objects_spaces.ts
I was not able to spot a single failure on the last commit before PR merge 4e4f408 . Afaik the easiest way to spot the problem is to unskip both https://github.com/elastic/kibana/blob/main/x-pack/test/saved_object_api_integration/spaces_only/apis/resolve_import_errors.ts#L134 and https://github.com/elastic/kibana/blob/main/x-pack/test/saved_object_api_integration/spaces_only/apis/update.ts#L56 and run flaky-test-runner, usually 100 runs is enough
After split we re-create ~6 indexes (it was 2 before) via esArchiver and it might be the breaking point.
Original discussion is here
The text was updated successfully, but these errors were encountered: