Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.8] [esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582) #159910

Merged
merged 1 commit into from
Jun 19, 2023

Conversation

kibanamachine
Copy link
Contributor

Backport

This will backport the following commits from main to 8.8:

Questions ?

Please refer to the Backport tool documentation

…ound in data.json (elastic#159582)

The ultimate goal of this PR is to lay the groundwork to be able to
remove the "dynamic" `mappings.json`, which probably should have never
existed.

With the PR, detecting SO documents in the `data.json` will
automatically trigger a cleanup of the SO indices.
This, in turn, will allow not having to define "dynamic" saved objects
indices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the
`mappings.json`).

IIUC the idea behind the dynamic indices was to have SO indices that are
aligned with the current stack version, avoiding the extra overhead of
having to migrate the inserted documents, and reducing overall test
times.

Nonetheless, what is happening today is:
1. FTR starts ES and Kibana.
2. Kibana creates current version SO indices at startup (empty ones).
3. `esArchiver.load()` processes the `mappings.json`.
3.1. It detects that we are defining SO indices and **deletes** existing
saved object indices.
3.2 It then re-creates these indices according to the definitions on
`mappings.json`.
4. `esArchiver.load()` processes the `data.json`. Specifically, it
inserts SO documents present in `data.json`.
5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the
inserted documents are up-to-date, hoping they are already aligned with
current stack version (which is not always the case, not even with
"dynamic" mappings).

Two interesting things to note:
- Steps 3 to 5 happen whilst Kibana is already started and running. If
Kibana queries SO indices during `esArchiver.load()`, and a request to
ES is made **right after** 3.2, the result might be
elastic#158918.
- Having dynamic SO indices' definitions, deleting the "official"
indices created by Kibana (3.1), and recreating them hoping to be
aligned with current stack version (3.2) is non-sense. We could use the
existing SO indices instead, and simply clean them up whenever we are
about to insert SO documents.

Performing that cleanup is precisely the goal of this PR.
Then, in subsequent PRs like
https://github.com/elastic/kibana/pull/159397/files, tackling the flaky
tests, we'll be able to simply remove the "dynamic" `mappings.json`
definitions, causing `esArchiver` to rely on SO indices created by
Kibana.

Thanks to this PR, the FTR tests won't need to explicitly cleanup saved
object indices in the `before` hooks.

(cherry picked from commit bbb5fc4)
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #53 / dashboard app - group 1 dashboard embeddable rendering adding saved searches
  • [job] [logs] FTR Configs #53 / dashboard app - group 1 dashboard embeddable rendering adding visualizations
  • [job] [logs] FTR Configs #53 / dashboard app - group 1 dashboard embeddable rendering data rendered correctly when dashboard is hard refreshed
  • [job] [logs] FTR Configs #53 / dashboard app - group 1 dashboard embeddable rendering data rendered correctly when dashboard is opened from listing page
  • [job] [logs] FTR Configs #53 / dashboard app - group 1 dashboard embeddable rendering initial render test
  • [job] [logs] FTR Configs #53 / dashboard app - group 1 dashboard embeddable rendering panels are updated when time changes inside of data
  • [job] [logs] FTR Configs #53 / dashboard app - group 1 dashboard embeddable rendering panels are updated when time changes outside of data
  • [job] [logs] FTR Configs #60 / dashboard app - group 2 dashboard filtering disabling a filter unfilters the data on "before all" hook for "pie charts"
  • [job] [logs] FTR Configs #60 / dashboard app - group 2 dashboard filtering using a pinned filter that excludes all data "after all" hook for "vega is filtered"
  • [job] [logs] FTR Configs #60 / dashboard app - group 2 dashboard filtering using a pinned filter that excludes all data "before all" hook for "filters on pie charts"

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id before after diff
enterpriseSearch 17 19 +2
securitySolution 401 405 +4
total +6

Total ESLint disabled count

id before after diff
enterpriseSearch 18 20 +2
securitySolution 481 485 +4
total +6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @gsoldevila

@kibanamachine kibanamachine merged commit d2683cb into elastic:8.8 Jun 19, 2023
gsoldevila added a commit that referenced this pull request Aug 7, 2023
…s are found in data.json (#159582) (#163178)

# Backport

This will backport the following commits from `main` to `7.17`:
- [[esArchiver] Automatically cleanup SO indices when SO documents are
found in data.json
(#159582)](#159582)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2023-06-19T11:08:03Z","message":"[esArchiver]
Automatically cleanup SO indices when SO documents are found in
data.json (#159582)\n\nThe ultimate goal of this PR is to lay the
groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`,
which probably should have never\r\nexisted.\r\n\r\nWith the PR,
detecting SO documents in the `data.json` will\r\nautomatically trigger
a cleanup of the SO indices.\r\nThis, in turn, will allow not having to
define \"dynamic\" saved objects\r\nindices (i.e. those with the
`$KIBANA_PACKAGE_VERSION` variable in
the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices
was to have SO indices that are\r\naligned with the current stack
version, avoiding the extra overhead of\r\nhaving to migrate the
inserted documents, and reducing overall
test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR
starts ES and Kibana.\r\n2. Kibana creates current version SO indices at
startup (empty ones).\r\n3. `esArchiver.load()` processes the
`mappings.json`.\r\n3.1. It detects that we are defining SO indices and
**deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates
these indices according to the definitions on\r\n`mappings.json`.\r\n4.
`esArchiver.load()` processes the `data.json`. Specifically,
it\r\ninserts SO documents present in `data.json`.\r\n5.
`esArchiver.load()` calls the _KibanaMigrator_ to make sure that
the\r\ninserted documents are up-to-date, hoping they are already
aligned with\r\ncurrent stack version (which is not always the case, not
even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to
note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and
running. If\r\nKibana queries SO indices during `esArchiver.load()`, and
a request to\r\nES is made **right after** 3.2, the result might
be\r\nhttps://github.com//issues/158918.\r\n- Having
dynamic SO indices' definitions, deleting the \"official\"\r\nindices
created by Kibana (3.1), and recreating them hoping to be\r\naligned
with current stack version (3.2) is non-sense. We could use
the\r\nexisting SO indices instead, and simply clean them up whenever we
are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is
precisely the goal of this PR.\r\nThen, in subsequent PRs
like\r\nhttps://github.com//pull/159397/files, tackling
the flaky\r\ntests, we'll be able to simply remove the \"dynamic\"
`mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO
indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests
won't need to explicitly cleanup saved\r\nobject indices in the `before`
hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Team:Operations","technical
debt","release_note:skip","backport:prev-minor","backport:prev-MAJOR","v8.9.0","FTR","v8.8.2","v8.10.0","v7.17.13"],"number":159582,"url":"https://github.com/elastic/kibana/pull/159582","mergeCommit":{"message":"[esArchiver]
Automatically cleanup SO indices when SO documents are found in
data.json (#159582)\n\nThe ultimate goal of this PR is to lay the
groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`,
which probably should have never\r\nexisted.\r\n\r\nWith the PR,
detecting SO documents in the `data.json` will\r\nautomatically trigger
a cleanup of the SO indices.\r\nThis, in turn, will allow not having to
define \"dynamic\" saved objects\r\nindices (i.e. those with the
`$KIBANA_PACKAGE_VERSION` variable in
the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices
was to have SO indices that are\r\naligned with the current stack
version, avoiding the extra overhead of\r\nhaving to migrate the
inserted documents, and reducing overall
test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR
starts ES and Kibana.\r\n2. Kibana creates current version SO indices at
startup (empty ones).\r\n3. `esArchiver.load()` processes the
`mappings.json`.\r\n3.1. It detects that we are defining SO indices and
**deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates
these indices according to the definitions on\r\n`mappings.json`.\r\n4.
`esArchiver.load()` processes the `data.json`. Specifically,
it\r\ninserts SO documents present in `data.json`.\r\n5.
`esArchiver.load()` calls the _KibanaMigrator_ to make sure that
the\r\ninserted documents are up-to-date, hoping they are already
aligned with\r\ncurrent stack version (which is not always the case, not
even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to
note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and
running. If\r\nKibana queries SO indices during `esArchiver.load()`, and
a request to\r\nES is made **right after** 3.2, the result might
be\r\nhttps://github.com//issues/158918.\r\n- Having
dynamic SO indices' definitions, deleting the \"official\"\r\nindices
created by Kibana (3.1), and recreating them hoping to be\r\naligned
with current stack version (3.2) is non-sense. We could use
the\r\nexisting SO indices instead, and simply clean them up whenever we
are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is
precisely the goal of this PR.\r\nThen, in subsequent PRs
like\r\nhttps://github.com//pull/159397/files, tackling
the flaky\r\ntests, we'll be able to simply remove the \"dynamic\"
`mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO
indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests
won't need to explicitly cleanup saved\r\nobject indices in the `before`
hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa"}},"sourceBranch":"main","suggestedTargetBranches":["7.17"],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/159582","number":159582,"mergeCommit":{"message":"[esArchiver]
Automatically cleanup SO indices when SO documents are found in
data.json (#159582)\n\nThe ultimate goal of this PR is to lay the
groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`,
which probably should have never\r\nexisted.\r\n\r\nWith the PR,
detecting SO documents in the `data.json` will\r\nautomatically trigger
a cleanup of the SO indices.\r\nThis, in turn, will allow not having to
define \"dynamic\" saved objects\r\nindices (i.e. those with the
`$KIBANA_PACKAGE_VERSION` variable in
the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices
was to have SO indices that are\r\naligned with the current stack
version, avoiding the extra overhead of\r\nhaving to migrate the
inserted documents, and reducing overall
test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR
starts ES and Kibana.\r\n2. Kibana creates current version SO indices at
startup (empty ones).\r\n3. `esArchiver.load()` processes the
`mappings.json`.\r\n3.1. It detects that we are defining SO indices and
**deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates
these indices according to the definitions on\r\n`mappings.json`.\r\n4.
`esArchiver.load()` processes the `data.json`. Specifically,
it\r\ninserts SO documents present in `data.json`.\r\n5.
`esArchiver.load()` calls the _KibanaMigrator_ to make sure that
the\r\ninserted documents are up-to-date, hoping they are already
aligned with\r\ncurrent stack version (which is not always the case, not
even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to
note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and
running. If\r\nKibana queries SO indices during `esArchiver.load()`, and
a request to\r\nES is made **right after** 3.2, the result might
be\r\nhttps://github.com//issues/158918.\r\n- Having
dynamic SO indices' definitions, deleting the \"official\"\r\nindices
created by Kibana (3.1), and recreating them hoping to be\r\naligned
with current stack version (3.2) is non-sense. We could use
the\r\nexisting SO indices instead, and simply clean them up whenever we
are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is
precisely the goal of this PR.\r\nThen, in subsequent PRs
like\r\nhttps://github.com//pull/159397/files, tackling
the flaky\r\ntests, we'll be able to simply remove the \"dynamic\"
`mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO
indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests
won't need to explicitly cleanup saved\r\nobject indices in the `before`
hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa"}},{"branch":"8.8","label":"v8.8.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/159910","number":159910,"state":"MERGED","mergeCommit":{"sha":"d2683cba7bfaff545a90c5a7daccb0b731fa1ba0","message":"[8.8]
[esArchiver] Automatically cleanup SO indices when SO documents are
found in data.json (#159582) (#159910)\n\n# Backport\n\nThis will
backport the following commits from `main` to `8.8`:\n- [[esArchiver]
Automatically cleanup SO indices when SO documents are\nfound in
data.json\n(#159582)](https://github.com/elastic/kibana/pull/159582)\n\n<!---
Backport version: 8.9.7 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Gerard\nSoldevila\",\"email\":\"gerard.soldevila@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2023-06-19T11:08:03Z\",\"message\":\"[esArchiver]\nAutomatically
cleanup SO indices when SO documents are found in\ndata.json
(#159582)\\n\\nThe ultimate goal of this PR is to lay the\ngroundwork to
be able to\\r\\nremove the \\\"dynamic\\\" `mappings.json`,\nwhich
probably should have never\\r\\nexisted.\\r\\n\\r\\nWith the
PR,\ndetecting SO documents in the `data.json` will\\r\\nautomatically
trigger\na cleanup of the SO indices.\\r\\nThis, in turn, will allow not
having to\ndefine \\\"dynamic\\\" saved objects\\r\\nindices (i.e. those
with the\n`$KIBANA_PACKAGE_VERSION` variable
in\nthe\\r\\n`mappings.json`).\\r\\n\\r\\nIIUC the idea behind the
dynamic indices\nwas to have SO indices that are\\r\\naligned with the
current stack\nversion, avoiding the extra overhead of\\r\\nhaving to
migrate the\ninserted documents, and reducing
overall\ntest\\r\\ntimes.\\r\\n\\r\\nNonetheless, what is happening
today is:\\r\\n1. FTR\nstarts ES and Kibana.\\r\\n2. Kibana creates
current version SO indices at\nstartup (empty ones).\\r\\n3.
`esArchiver.load()` processes the\n`mappings.json`.\\r\\n3.1. It detects
that we are defining SO indices and\n**deletes** existing\\r\\nsaved
object indices.\\r\\n3.2 It then re-creates\nthese indices according to
the definitions on\\r\\n`mappings.json`.\\r\\n4.\n`esArchiver.load()`
processes the `data.json`. Specifically,\nit\\r\\ninserts SO documents
present in `data.json`.\\r\\n5.\n`esArchiver.load()` calls the
_KibanaMigrator_ to make sure that\nthe\\r\\ninserted documents are
up-to-date, hoping they are already\naligned with\\r\\ncurrent stack
version (which is not always the case, not\neven
with\\r\\n\\\"dynamic\\\" mappings).\\r\\n\\r\\nTwo interesting things
to\nnote:\\r\\n- Steps 3 to 5 happen whilst Kibana is already started
and\nrunning. If\\r\\nKibana queries SO indices during
`esArchiver.load()`, and\na request to\\r\\nES is made **right after**
3.2, the result
might\nbe\\r\\nhttps://github.com//issues/158918.\\r\\n-
Having\ndynamic SO indices' definitions, deleting the
\\\"official\\\"\\r\\nindices\ncreated by Kibana (3.1), and recreating
them hoping to be\\r\\naligned\nwith current stack version (3.2) is
non-sense. We could use\nthe\\r\\nexisting SO indices instead, and
simply clean them up whenever we\nare\\r\\nabout to insert SO
documents.\\r\\n\\r\\nPerforming that cleanup is\nprecisely the goal of
this PR.\\r\\nThen, in subsequent
PRs\nlike\\r\\nhttps://github.com//pull/159397/files,
tackling\nthe flaky\\r\\ntests, we'll be able to simply remove the
\\\"dynamic\\\"\n`mappings.json`\\r\\ndefinitions, causing `esArchiver`
to rely on SO\nindices created by\\r\\nKibana.\\r\\n\\r\\nThanks to this
PR, the FTR tests\nwon't need to explicitly cleanup saved\\r\\nobject
indices in the
`before`\nhooks.\",\"sha\":\"bbb5fc4abe7dd530d8248a09a9638cd3438202aa\",\"branchLabelMapping\":{\"^v8.9.0$\":\"main\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"Team:Core\",\"Team:Operations\",\"technical\ndebt\",\"release_note:skip\",\"backport:prev-minor\",\"v8.9.0\",\"FTR\",\"v8.8.2\"],\"number\":159582,\"url\":\"https://github.com/elastic/kibana/pull/159582\",\"mergeCommit\":{\"message\":\"[esArchiver]\nAutomatically
cleanup SO indices when SO documents are found in\ndata.json
(#159582)\\n\\nThe ultimate goal of this PR is to lay the\ngroundwork to
be able to\\r\\nremove the \\\"dynamic\\\" `mappings.json`,\nwhich
probably should have never\\r\\nexisted.\\r\\n\\r\\nWith the
PR,\ndetecting SO documents in the `data.json` will\\r\\nautomatically
trigger\na cleanup of the SO indices.\\r\\nThis, in turn, will allow not
having to\ndefine \\\"dynamic\\\" saved objects\\r\\nindices (i.e. those
with the\n`$KIBANA_PACKAGE_VERSION` variable
in\nthe\\r\\n`mappings.json`).\\r\\n\\r\\nIIUC the idea behind the
dynamic indices\nwas to have SO indices that are\\r\\naligned with the
current stack\nversion, avoiding the extra overhead of\\r\\nhaving to
migrate the\ninserted documents, and reducing
overall\ntest\\r\\ntimes.\\r\\n\\r\\nNonetheless, what is happening
today is:\\r\\n1. FTR\nstarts ES and Kibana.\\r\\n2. Kibana creates
current version SO indices at\nstartup (empty ones).\\r\\n3.
`esArchiver.load()` processes the\n`mappings.json`.\\r\\n3.1. It detects
that we are defining SO indices and\n**deletes** existing\\r\\nsaved
object indices.\\r\\n3.2 It then re-creates\nthese indices according to
the definitions on\\r\\n`mappings.json`.\\r\\n4.\n`esArchiver.load()`
processes the `data.json`. Specifically,\nit\\r\\ninserts SO documents
present in `data.json`.\\r\\n5.\n`esArchiver.load()` calls the
_KibanaMigrator_ to make sure that\nthe\\r\\ninserted documents are
up-to-date, hoping they are already\naligned with\\r\\ncurrent stack
version (which is not always the case, not\neven
with\\r\\n\\\"dynamic\\\" mappings).\\r\\n\\r\\nTwo interesting things
to\nnote:\\r\\n- Steps 3 to 5 happen whilst Kibana is already started
and\nrunning. If\\r\\nKibana queries SO indices during
`esArchiver.load()`, and\na request to\\r\\nES is made **right after**
3.2, the result
might\nbe\\r\\nhttps://github.com//issues/158918.\\r\\n-
Having\ndynamic SO indices' definitions, deleting the
\\\"official\\\"\\r\\nindices\ncreated by Kibana (3.1), and recreating
them hoping to be\\r\\naligned\nwith current stack version (3.2) is
non-sense. We could use\nthe\\r\\nexisting SO indices instead, and
simply clean them up whenever we\nare\\r\\nabout to insert SO
documents.\\r\\n\\r\\nPerforming that cleanup is\nprecisely the goal of
this PR.\\r\\nThen, in subsequent
PRs\nlike\\r\\nhttps://github.com//pull/159397/files,
tackling\nthe flaky\\r\\ntests, we'll be able to simply remove the
\\\"dynamic\\\"\n`mappings.json`\\r\\ndefinitions, causing `esArchiver`
to rely on SO\nindices created by\\r\\nKibana.\\r\\n\\r\\nThanks to this
PR, the FTR tests\nwon't need to explicitly cleanup saved\\r\\nobject
indices in the
`before`\nhooks.\",\"sha\":\"bbb5fc4abe7dd530d8248a09a9638cd3438202aa\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"8.8\"],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v8.9.0\",\"labelRegex\":\"^v8.9.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/159582\",\"number\":159582,\"mergeCommit\":{\"message\":\"[esArchiver]\nAutomatically
cleanup SO indices when SO documents are found in\ndata.json
(#159582)\\n\\nThe ultimate goal of this PR is to lay the\ngroundwork to
be able to\\r\\nremove the \\\"dynamic\\\" `mappings.json`,\nwhich
probably should have never\\r\\nexisted.\\r\\n\\r\\nWith the
PR,\ndetecting SO documents in the `data.json` will\\r\\nautomatically
trigger\na cleanup of the SO indices.\\r\\nThis, in turn, will allow not
having to\ndefine \\\"dynamic\\\" saved objects\\r\\nindices (i.e. those
with the\n`$KIBANA_PACKAGE_VERSION` variable
in\nthe\\r\\n`mappings.json`).\\r\\n\\r\\nIIUC the idea behind the
dynamic indices\nwas to have SO indices that are\\r\\naligned with the
current stack\nversion, avoiding the extra overhead of\\r\\nhaving to
migrate the\ninserted documents, and reducing
overall\ntest\\r\\ntimes.\\r\\n\\r\\nNonetheless, what is happening
today is:\\r\\n1. FTR\nstarts ES and Kibana.\\r\\n2. Kibana creates
current version SO indices at\nstartup (empty ones).\\r\\n3.
`esArchiver.load()` processes the\n`mappings.json`.\\r\\n3.1. It detects
that we are defining SO indices and\n**deletes** existing\\r\\nsaved
object indices.\\r\\n3.2 It then re-creates\nthese indices according to
the definitions on\\r\\n`mappings.json`.\\r\\n4.\n`esArchiver.load()`
processes the `data.json`. Specifically,\nit\\r\\ninserts SO documents
present in `data.json`.\\r\\n5.\n`esArchiver.load()` calls the
_KibanaMigrator_ to make sure that\nthe\\r\\ninserted documents are
up-to-date, hoping they are already\naligned with\\r\\ncurrent stack
version (which is not always the case, not\neven
with\\r\\n\\\"dynamic\\\" mappings).\\r\\n\\r\\nTwo interesting things
to\nnote:\\r\\n- Steps 3 to 5 happen whilst Kibana is already started
and\nrunning. If\\r\\nKibana queries SO indices during
`esArchiver.load()`, and\na request to\\r\\nES is made **right after**
3.2, the result
might\nbe\\r\\nhttps://github.com//issues/158918.\\r\\n-
Having\ndynamic SO indices' definitions, deleting the
\\\"official\\\"\\r\\nindices\ncreated by Kibana (3.1), and recreating
them hoping to be\\r\\naligned\nwith current stack version (3.2) is
non-sense. We could use\nthe\\r\\nexisting SO indices instead, and
simply clean them up whenever we\nare\\r\\nabout to insert SO
documents.\\r\\n\\r\\nPerforming that cleanup is\nprecisely the goal of
this PR.\\r\\nThen, in subsequent
PRs\nlike\\r\\nhttps://github.com//pull/159397/files,
tackling\nthe flaky\\r\\ntests, we'll be able to simply remove the
\\\"dynamic\\\"\n`mappings.json`\\r\\ndefinitions, causing `esArchiver`
to rely on SO\nindices created by\\r\\nKibana.\\r\\n\\r\\nThanks to this
PR, the FTR tests\nwon't need to explicitly cleanup saved\\r\\nobject
indices in the
`before`\nhooks.\",\"sha\":\"bbb5fc4abe7dd530d8248a09a9638cd3438202aa\"}},{\"branch\":\"8.8\",\"label\":\"v8.8.2\",\"labelRegex\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Gerard Soldevila
<gerard.soldevila@elastic.co>"}},{"branch":"8.10","label":"v8.10.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"7.17","label":"v7.17.13","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants