Proactively improve robustness of FTR tests that use esArchiver #161882
Assignees
Labels
Team:QA
Team label for QA Team
Comments
|
I created a script to locally update the |
gsoldevila
added
the
Team:Core
|
Pinging @elastic/kibana-core (Team:Core) |
This was referenced Jul 14, 2023
gsoldevila
added a commit
that referenced
this issue
Jul 17, 2023
…OT delete SO indices (#161852) ## Summary Part of #161882 Updates the `x-pack/test/saved_object_api_integration/common/fixtures/es_archiver/saved_objects/spaces` archive: * deleting `mappings.json` * updating documents in `data.json` This way, _esArchiver_ will not delete SO indices and recreate them, eliminating the odds of the related tests being flaky. CC @jeramysoucy @dmlemeshko
gsoldevila
added a commit
that referenced
this issue
Jul 17, 2023
…e SO indices (#161936) ## Summary Part of #161882 Updates the `test/api_integration/fixtures/es_archiver/saved_objects/delete_unknown_types` archive: * deleting `mappings.json` * updating documents in `data.json` This way, _esArchiver_ will not delete SO indices and recreate them, eliminating the odds of the related tests being flaky.
gsoldevila
added a commit
that referenced
this issue
Jul 17, 2023
…e SO indices (#161937) ## Summary Part of #161882 Updates the `test/functional/fixtures/es_archiver/deprecations_service` archive: * deleting `mappings.json` * updating documents in `data.json` This way, _esArchiver_ will not delete SO indices and recreate them, eliminating the odds of the related tests being flaky.
This was referenced Jul 17, 2023
Ruhshan
pushed a commit
to Ruhshan/kibana
that referenced
this issue
Jul 19, 2023
…mappings.json (elastic#162029) ## Summary Part of elastic#161882 - Updates the remaining "dynamic" es archives (aka those using the `$KIBANA_PACKAGE_VERSION` variable), deleting mappings.json. - It also removes the piece of logic in `esArchiver` that replaces that variable.
ThomThomson
pushed a commit
to ThomThomson/kibana
that referenced
this issue
Aug 1, 2023
…mappings.json (elastic#162029) ## Summary Part of elastic#161882 - Updates the remaining "dynamic" es archives (aka those using the `$KIBANA_PACKAGE_VERSION` variable), deleting mappings.json. - It also removes the piece of logic in `esArchiver` that replaces that variable.
ThomThomson
pushed a commit
to ThomThomson/kibana
that referenced
this issue
Aug 1, 2023
… reduce flakiness (elastic#162259) ## Summary Part of elastic#161882 Updates all archives of the `test/functional/fixtures/es_archiver/saved_objects_management` test suite: * Removing SO index definitions. * Updating documents defined in `data.json`.
ThomThomson
pushed a commit
to ThomThomson/kibana
that referenced
this issue
Aug 1, 2023
…est archives to reduce flakiness (elastic#162321) ## Summary Part of elastic#161882 Updates all archives of the `encrypted_saved_objects_api_integration` test suite: * Removing SO index definitions. * Updating documents defined in `data.json`.
gsoldevila
added a commit
to gsoldevila/kibana
that referenced
this issue
Aug 9, 2023
… reduce flakiness (elastic#162259) ## Summary Part of elastic#161882 Updates all archives of the `test/functional/fixtures/es_archiver/saved_objects_management` test suite: * Removing SO index definitions. * Updating documents defined in `data.json`. (cherry picked from commit 59450f0) # Conflicts: # test/functional/fixtures/es_archiver/saved_objects_management/hidden_from_http_apis/data.json # test/functional/fixtures/es_archiver/saved_objects_management/hidden_saved_objects/data.json # test/functional/fixtures/es_archiver/saved_objects_management/hidden_types/data.json # test/functional/fixtures/es_archiver/saved_objects_management/hidden_types/mappings.json # x-pack/test/functional/es_archives/visualize/default/mappings.json
gsoldevila
referenced
this issue
Aug 11, 2023
…ives to reduce flakiness (#162259) (#163479) # Backport This will backport the following commits from `main` to `7.17`: - [[FTR - esArchiver] Update `saved_objects_management` test archives to reduce flakiness (#162259)](#162259) <!--- Backport version: 8.9.8 --> ### Related issues: - #132855 - #130305 - #147635 - #132566 - #131871 - #131830 - #131834 ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gerard Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2023-07-24T08:11:39Z","message":"[FTR - esArchiver] Update `saved_objects_management` test archives to reduce flakiness (#162259)\n\n## Summary\r\n\r\nPart of https://github.com/elastic/kibana/issues/161882\r\n\r\nUpdates all archives of the\r\n`test/functional/fixtures/es_archiver/saved_objects_management` test\r\nsuite:\r\n* Removing SO index definitions.\r\n* Updating documents defined in `data.json`.","sha":"59450f0a22bb8c34598d7156dcbf144d41752259","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","technical debt","release_note:skip","backport:skip","FTR","v8.10.0"],"number":162259,"url":"https://github.com/elastic/kibana/pull/162259","mergeCommit":{"message":"[FTR - esArchiver] Update `saved_objects_management` test archives to reduce flakiness (#162259)\n\n## Summary\r\n\r\nPart of https://github.com/elastic/kibana/issues/161882\r\n\r\nUpdates all archives of the\r\n`test/functional/fixtures/es_archiver/saved_objects_management` test\r\nsuite:\r\n* Removing SO index definitions.\r\n* Updating documents defined in `data.json`.","sha":"59450f0a22bb8c34598d7156dcbf144d41752259"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/162259","number":162259,"mergeCommit":{"message":"[FTR - esArchiver] Update `saved_objects_management` test archives to reduce flakiness (#162259)\n\n## Summary\r\n\r\nPart of https://github.com/elastic/kibana/issues/161882\r\n\r\nUpdates all archives of the\r\n`test/functional/fixtures/es_archiver/saved_objects_management` test\r\nsuite:\r\n* Removing SO index definitions.\r\n* Updating documents defined in `data.json`.","sha":"59450f0a22bb8c34598d7156dcbf144d41752259"}}]}] BACKPORT-->
dmlemeshko
added a commit
that referenced
this issue
Oct 30, 2023
) (#170113) # Backport This will backport the following commits from `main` to `8.10`: - [update canvas a11y test to import SO not re-create index (#168944)](#168944) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Dzmitry Lemechko","email":"dzmitry.lemechko@elastic.co"},"sourceCommit":{"committedDate":"2023-10-16T16:28:43Z","message":"update canvas a11y test to import SO not re-create index (#168944)\n\n## Summary\r\n\r\nRelated to https://github.com/elastic/kibana/issues/161882\r\n\r\nThere is no need to re-create `.kibana` index, changing to imported\r\nsaved objects.","sha":"a1f360eac5d538cebd13af795dfdc6e14e944f2a","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v8.11.0","v8.12.0","v8.11.1"],"number":168944,"url":"https://github.com/elastic/kibana/pull/168944","mergeCommit":{"message":"update canvas a11y test to import SO not re-create index (#168944)\n\n## Summary\r\n\r\nRelated to https://github.com/elastic/kibana/issues/161882\r\n\r\nThere is no need to re-create `.kibana` index, changing to imported\r\nsaved objects.","sha":"a1f360eac5d538cebd13af795dfdc6e14e944f2a"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/169003","number":169003,"state":"MERGED","mergeCommit":{"sha":"bfbf388bb8e21ab718383e445fcf44edab9e0b37","message":"[8.11] update canvas a11y test to import SO not re-create index (#168944) (#169003)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.11`:\n- [update canvas a11y test to import SO not re-create index\n(#168944)](https://github.com/elastic/kibana/pull/168944)\n\n<!--- Backport version: 8.9.7 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Dzmitry\nLemechko\",\"email\":\"dzmitry.lemechko@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2023-10-16T16:28:43Z\",\"message\":\"update\ncanvas a11y test to import SO not re-create index (#168944)\\n\\n##\nSummary\\r\\n\\r\\nRelated to\nhttps://github.com//issues/161882\\r\\n\\r\\nThere is no need\nto re-create `.kibana` index, changing to imported\\r\\nsaved\nobjects.\",\"sha\":\"a1f360eac5d538cebd13af795dfdc6e14e944f2a\",\"branchLabelMapping\":{\"^v8.12.0$\":\"main\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"v8.12.0\",\"v8.11.1\"],\"number\":168944,\"url\":\"https://github.com/elastic/kibana/pull/168944\",\"mergeCommit\":{\"message\":\"update\ncanvas a11y test to import SO not re-create index (#168944)\\n\\n##\nSummary\\r\\n\\r\\nRelated to\nhttps://github.com//issues/161882\\r\\n\\r\\nThere is no need\nto re-create `.kibana` index, changing to imported\\r\\nsaved\nobjects.\",\"sha\":\"a1f360eac5d538cebd13af795dfdc6e14e944f2a\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"8.11\"],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v8.12.0\",\"labelRegex\":\"^v8.12.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/168944\",\"number\":168944,\"mergeCommit\":{\"message\":\"update\ncanvas a11y test to import SO not re-create index (#168944)\\n\\n##\nSummary\\r\\n\\r\\nRelated to\nhttps://github.com//issues/161882\\r\\n\\r\\nThere is no need\nto re-create `.kibana` index, changing to imported\\r\\nsaved\nobjects.\",\"sha\":\"a1f360eac5d538cebd13af795dfdc6e14e944f2a\"}},{\"branch\":\"8.11\",\"label\":\"v8.11.1\",\"labelRegex\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>"}},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/168944","number":168944,"mergeCommit":{"message":"update canvas a11y test to import SO not re-create index (#168944)\n\n## Summary\r\n\r\nRelated to https://github.com/elastic/kibana/issues/161882\r\n\r\nThere is no need to re-create `.kibana` index, changing to imported\r\nsaved objects.","sha":"a1f360eac5d538cebd13af795dfdc6e14e944f2a"}}]}] BACKPORT-->
kibanamachine
added a commit
that referenced
this issue
Oct 30, 2023
…69852) (#170064) # Backport This will backport the following commits from `main` to `8.10`: - [[esArchiver] restrict from modifying saved objects indexes (#169852)](#169852) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Dzmitry Lemechko","email":"dzmitry.lemechko@elastic.co"},"sourceCommit":{"committedDate":"2023-10-27T18:27:10Z","message":"[esArchiver] restrict from modifying saved objects indexes (#169852)\n\n## Summary\r\n\r\nRelated to #161882\r\n\r\nThe goal is to prevent FTR tests from new esArchives that overrides SO\r\nindexes.\r\n\r\nThis PR adds the existing archives that re-create Saved Objects indexes\r\ninto temporary exception list, located in\r\n`packages/kbn-es-archiver/src/fixtures/override_saved_objects_index/exception_list.json`.\r\nWhenever tests/archives are updated to not modify SO indexes, archive is\r\nexpected to be removed from the list (progress can be tracked in\r\n#169075, #168973, #168969, #168926 )\r\n\r\nLoad action has a check if index is SO index and if the archive is in\r\nthe exception list. This will throw error for the new archives, but\r\nstill work as usual for the existing ones while teams updating the\r\ntests.\r\n\r\nWhenever test is loading archive listed in the exception list, the\r\nfollowing warning message is logged:\r\n```\r\nwarn x-pack/test/functional/es_archives/data/search_sessions overrides Saved Objects index(es) and placed temporary in the exception list.\r\nPlease fix the archive and remove it from /Users/dmle/github/kibana/packages/kbn-es-archiver/src/fixtures/override_saved_objects_index/exception_list.json.\r\nFor more details see: https://github.com/elastic/kibana/issues/161882\r\n```\r\n\r\nIf the test loads a newly added archive that modifies a Saved Object\r\nindex (e.g. `.kibana`), esArchiver will throw the error:\r\n```\r\n Error: esArchiver doesn't support modifying the existing Saved Objects index: '.kibana_1',\r\n please update its definition in mappings.json\r\n```","sha":"669881095881b599262d8ba19bdf68207c86bfa0","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","ES-ARCHIVER","v8.12.0","v8.11.1","v8.10.5"],"number":169852,"url":"https://github.com/elastic/kibana/pull/169852","mergeCommit":{"message":"[esArchiver] restrict from modifying saved objects indexes (#169852)\n\n## Summary\r\n\r\nRelated to #161882\r\n\r\nThe goal is to prevent FTR tests from new esArchives that overrides SO\r\nindexes.\r\n\r\nThis PR adds the existing archives that re-create Saved Objects indexes\r\ninto temporary exception list, located in\r\n`packages/kbn-es-archiver/src/fixtures/override_saved_objects_index/exception_list.json`.\r\nWhenever tests/archives are updated to not modify SO indexes, archive is\r\nexpected to be removed from the list (progress can be tracked in\r\n#169075, #168973, #168969, #168926 )\r\n\r\nLoad action has a check if index is SO index and if the archive is in\r\nthe exception list. This will throw error for the new archives, but\r\nstill work as usual for the existing ones while teams updating the\r\ntests.\r\n\r\nWhenever test is loading archive listed in the exception list, the\r\nfollowing warning message is logged:\r\n```\r\nwarn x-pack/test/functional/es_archives/data/search_sessions overrides Saved Objects index(es) and placed temporary in the exception list.\r\nPlease fix the archive and remove it from /Users/dmle/github/kibana/packages/kbn-es-archiver/src/fixtures/override_saved_objects_index/exception_list.json.\r\nFor more details see: https://github.com/elastic/kibana/issues/161882\r\n```\r\n\r\nIf the test loads a newly added archive that modifies a Saved Object\r\nindex (e.g. `.kibana`), esArchiver will throw the error:\r\n```\r\n Error: esArchiver doesn't support modifying the existing Saved Objects index: '.kibana_1',\r\n please update its definition in mappings.json\r\n```","sha":"669881095881b599262d8ba19bdf68207c86bfa0"}},"sourceBranch":"main","suggestedTargetBranches":["8.11","8.10"],"targetPullRequestStates":[{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/169852","number":169852,"mergeCommit":{"message":"[esArchiver] restrict from modifying saved objects indexes (#169852)\n\n## Summary\r\n\r\nRelated to #161882\r\n\r\nThe goal is to prevent FTR tests from new esArchives that overrides SO\r\nindexes.\r\n\r\nThis PR adds the existing archives that re-create Saved Objects indexes\r\ninto temporary exception list, located in\r\n`packages/kbn-es-archiver/src/fixtures/override_saved_objects_index/exception_list.json`.\r\nWhenever tests/archives are updated to not modify SO indexes, archive is\r\nexpected to be removed from the list (progress can be tracked in\r\n#169075, #168973, #168969, #168926 )\r\n\r\nLoad action has a check if index is SO index and if the archive is in\r\nthe exception list. This will throw error for the new archives, but\r\nstill work as usual for the existing ones while teams updating the\r\ntests.\r\n\r\nWhenever test is loading archive listed in the exception list, the\r\nfollowing warning message is logged:\r\n```\r\nwarn x-pack/test/functional/es_archives/data/search_sessions overrides Saved Objects index(es) and placed temporary in the exception list.\r\nPlease fix the archive and remove it from /Users/dmle/github/kibana/packages/kbn-es-archiver/src/fixtures/override_saved_objects_index/exception_list.json.\r\nFor more details see: https://github.com/elastic/kibana/issues/161882\r\n```\r\n\r\nIf the test loads a newly added archive that modifies a Saved Object\r\nindex (e.g. `.kibana`), esArchiver will throw the error:\r\n```\r\n Error: esArchiver doesn't support modifying the existing Saved Objects index: '.kibana_1',\r\n please update its definition in mappings.json\r\n```","sha":"669881095881b599262d8ba19bdf68207c86bfa0"}},{"branch":"8.11","label":"v8.11.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.10","label":"v8.10.5","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
|
Pinging @elastic/appex-qa (Team:QA) |
dmlemeshko
removed
the
Team:Core
doakalexi
added a commit
that referenced
this issue
May 2, 2024
…/spaces_only/tests/actions/migrations·ts (#182279) Resolves #169159 ## Summary I could not reproduce the failure in this test, but it's using `esArchiver` which is known to be flaky. I followed the instructions in this issue, [Proactively improve robustness of FTR tests that use esArchiver #161882](#161882), to help reduce flakiness in the future. Flaky test runner: [run 1](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5829#_) x 50 [run 2](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5830) x 200 [run 3](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5831) x 200
doakalexi
added a commit
to doakalexi/kibana
that referenced
this issue
May 2, 2024
…/spaces_only/tests/actions/migrations·ts (elastic#182279) Resolves elastic#169159 ## Summary I could not reproduce the failure in this test, but it's using `esArchiver` which is known to be flaky. I followed the instructions in this issue, [Proactively improve robustness of FTR tests that use esArchiver elastic#161882](elastic#161882), to help reduce flakiness in the future. Flaky test runner: [run 1](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5829#_) x 50 [run 2](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5830) x 200 [run 3](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5831) x 200 (cherry picked from commit e062a9f) # Conflicts: # x-pack/test/alerting_api_integration/spaces_only/tests/actions/migrations.ts
doakalexi
added a commit
that referenced
this issue
May 2, 2024
…gration/spaces_only/tests/actions/migrations·ts (#182279) (#182506) # Backport This will backport the following commits from `main` to `8.14`: - [[ResponseOps] Fix for flaky test x-pack/test/alerting_api_integration/spaces_only/tests/actions/migrations·ts (#182279)](#182279) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Alexi Doak","email":"109488926+doakalexi@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-05-02T18:33:58Z","message":"[ResponseOps] Fix for flaky test x-pack/test/alerting_api_integration/spaces_only/tests/actions/migrations·ts (#182279)\n\nResolves https://github.com/elastic/kibana/issues/169159\r\n\r\n## Summary\r\n\r\nI could not reproduce the failure in this test, but it's using\r\n`esArchiver` which is known to be flaky. I followed the instructions in\r\nthis issue, [Proactively improve robustness of FTR tests that use\r\nesArchiver #161882](#161882), to\r\nhelp reduce flakiness in the future.\r\n\r\nFlaky test runner:\r\n[run\r\n1](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5829#_)\r\nx 50\r\n[run\r\n2](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5830)\r\nx 200\r\n[run\r\n3](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5831)\r\nx 200","sha":"e062a9fbc9696ca872da98de63fb598b65ea6467","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","v8.14.0","v8.15.0"],"number":182279,"url":"https://github.com/elastic/kibana/pull/182279","mergeCommit":{"message":"[ResponseOps] Fix for flaky test x-pack/test/alerting_api_integration/spaces_only/tests/actions/migrations·ts (#182279)\n\nResolves https://github.com/elastic/kibana/issues/169159\r\n\r\n## Summary\r\n\r\nI could not reproduce the failure in this test, but it's using\r\n`esArchiver` which is known to be flaky. I followed the instructions in\r\nthis issue, [Proactively improve robustness of FTR tests that use\r\nesArchiver #161882](#161882), to\r\nhelp reduce flakiness in the future.\r\n\r\nFlaky test runner:\r\n[run\r\n1](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5829#_)\r\nx 50\r\n[run\r\n2](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5830)\r\nx 200\r\n[run\r\n3](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5831)\r\nx 200","sha":"e062a9fbc9696ca872da98de63fb598b65ea6467"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","labelRegex":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/182279","number":182279,"mergeCommit":{"message":"[ResponseOps] Fix for flaky test x-pack/test/alerting_api_integration/spaces_only/tests/actions/migrations·ts (#182279)\n\nResolves https://github.com/elastic/kibana/issues/169159\r\n\r\n## Summary\r\n\r\nI could not reproduce the failure in this test, but it's using\r\n`esArchiver` which is known to be flaky. I followed the instructions in\r\nthis issue, [Proactively improve robustness of FTR tests that use\r\nesArchiver #161882](#161882), to\r\nhelp reduce flakiness in the future.\r\n\r\nFlaky test runner:\r\n[run\r\n1](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5829#_)\r\nx 50\r\n[run\r\n2](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5830)\r\nx 200\r\n[run\r\n3](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5831)\r\nx 200","sha":"e062a9fbc9696ca872da98de63fb598b65ea6467"}}]}] BACKPORT-->
doakalexi
added a commit
that referenced
this issue
May 31, 2024
…only/tests/action_task_params/migrations·ts (#184473) Resolves #154358 ## Summary I couldn't get this to fail, but it's using `esArchiver` which is known to be flaky. I used the info in this issue, [Proactively improve robustness of FTR tests that use esArchiver #161882](#161882), to maybe help reduce the flakiness. Flaky test runner: [run 1](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6163) x 50 [run 2](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6164) x 200 [run 3](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6165) x 200
This was referenced Nov 26, 2024
JiaweiWu
added a commit
that referenced
this issue
Dec 2, 2024
## Summary Issue: #200154 Fixes flaky task manager remove types test by following the guidance here: #161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
hop-dev
pushed a commit
to hop-dev/kibana
that referenced
this issue
Dec 5, 2024
## Summary Issue: elastic#200154 Fixes flaky task manager remove types test by following the guidance here: elastic#161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
JiaweiWu
added a commit
that referenced
this issue
Dec 6, 2024
## Summary Issue: #198664 Fixes task manager schedule flaky test according to guidance posted here: #161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
markov00
pushed a commit
to markov00/kibana
that referenced
this issue
Dec 7, 2024
## Summary Issue: elastic#198664 Fixes task manager schedule flaky test according to guidance posted here: elastic#161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
SoniaSanzV
pushed a commit
to SoniaSanzV/kibana
that referenced
this issue
Dec 9, 2024
## Summary Issue: elastic#198664 Fixes task manager schedule flaky test according to guidance posted here: elastic#161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
SoniaSanzV
pushed a commit
to SoniaSanzV/kibana
that referenced
this issue
Dec 9, 2024
## Summary Issue: elastic#198664 Fixes task manager schedule flaky test according to guidance posted here: elastic#161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Dec 9, 2024
## Summary Issue: elastic#200154 Fixes flaky task manager remove types test by following the guidance here: elastic#161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Dec 9, 2024
## Summary Issue: elastic#198664 Fixes task manager schedule flaky test according to guidance posted here: elastic#161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Samiul-TheSoccerFan
pushed a commit
to Samiul-TheSoccerFan/kibana
that referenced
this issue
Dec 10, 2024
## Summary Issue: elastic#198664 Fixes task manager schedule flaky test according to guidance posted here: elastic#161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
mykolaharmash
pushed a commit
to mykolaharmash/kibana
that referenced
this issue
Dec 11, 2024
## Summary Issue: elastic#198664 Fixes task manager schedule flaky test according to guidance posted here: elastic#161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Dec 12, 2024
## Summary Issue: elastic#200154 Fixes flaky task manager remove types test by following the guidance here: elastic#161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Dec 12, 2024
## Summary Issue: elastic#198664 Fixes task manager schedule flaky test according to guidance posted here: elastic#161882 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There are multiple FTR tests that still rely on
esArchiverto import test data into ES indices.Some of the archives used by these tests are actually re-defining SO indices and documents.
When
esArchiver.load(...)is called in the beginning of a test (usually in a before hook), multiple things happen:esArchiverdetects that SO indices are being definedmappings.json.data.json.KibanaMigrator, hoping to update the documents present in the indices.All of that happens whilst Kibana is already running, and potentially issuing requests against these indices.
Sometimes these requests happen right when
esArchiveris re-creating the SO indices, causing different sorts of flakiness (as seen on #158918).The aforementioned issue has been tackled in 2 stages:
esArchiverso that it cleans up the SO indices whenever it detects that SO are defined in thedata.json(without actually deleting the indices). ([esArchiver] Automatically cleanup SO indices when SO documents are found in data.json #159582)mappings.json. This way,esArchiverwill NOT delete + recreate the SO indices.data.jsonto make sure they are aligned with the current version indices.In parallel, the
esArchiver.load(...)was updated so that when it actually creates SO indices (for those tests that still define them inmappings.json), it does so in two separate steps: first create indices, and then create aliases. This helps circumvent some of the no_shards_errors that happen when the index is still not ready to be queried. (#160584)What's next
ATM there are 7 other archives that are still defining "dynamic" SO indices, i.e. indices that aim at being at the latest version, by using the $KIBANA_PACKAGE_VERSION in the mappings.json. These should be easy to update, removing the corresponding definitions from the mappings.json, and ensuring they will not cause flakiness.
Also ATM, there are a total of 49 mappings.json files that presumably define SO indices (they contain the .kibana keyword). Even though they are not dynamic, they could also be updated to not cause SO indices to be deleted.
But in order to do so, we must:
The goal of this issue is to proactively tackle the tests that are still defining SO indices, in order to:
mappings.jsondon't closely mirror the structure of the official ones:dynamic: falseinstead of'strict').Dynamic
esArchiverarchives:x-pack/test/saved_object_api_integration/common/fixtures/es_archiver/saved_objects/spacessaved_object_api_integrationarchive to NOT delete SO indices #161852test/api_integration/fixtures/es_archiver/saved_objects/delete_unknown_typesdelete_unknown_typesarchive to NOT delete SO indices #161936test/functional/fixtures/es_archiver/deprecations_servicedeprecations_servicearchive to NOT delete SO indices #161937test/functional/fixtures/es_archiver/saved_objects_management/export_transformTODOtest/functional/fixtures/es_archiver/saved_objects_management/hidden_saved_objectsTODOtest/functional/fixtures/es_archiver/saved_objects_management/nested_export_transformTODOtest/functional/fixtures/es_archiver/saved_objects_management/visible_in_managementTODOTasks
The text was updated successfully, but these errors were encountered: