Migrations v2: don't auto-create indices + FTR/esArchiver support #85778
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
17 tasks
rudolf
force-pushed
the
so-migrations-on-by-default
branch
from
14372d3 to
a974a4a
Compare
rudolf
added
Feature:Saved Objects
project:ResilientSavedObjectMigrations
rudolf
changed the title
rudolf
force-pushed
the
so-migrations-on-by-default
branch
from
a974a4a to
c82a595
Compare
rudolf
changed the title
9 tasks
|
merge conflict between base and head |
pgayvallet
reviewed
Feb 1, 2021
pgayvallet
approved these changes
Feb 1, 2021
💛 Build succeeded, but was flaky
Test FailuresKibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/ml/permissions/full_ml_access·ts.machine learning permissions for user with full ML access with data loaded (ft_ml_poweruser) should display elements on Data Frame Analytics results view page correctlyStandard OutStack TraceMetrics [docs]
History
To update your PR or re-run it, just comment with: |
This was referenced Feb 2, 2021
rudolf
added a commit
to rudolf/kibana
that referenced
this pull request
Feb 2, 2021
…iver support (elastic#85778)"" This reverts commit f979580.
9 tasks
gmmorris
added a commit
to lizozom/kibana
that referenced
this pull request
Feb 2, 2021
…om/kibana into pr/89570 * 'sessions/save-all-sessions' of https://github.com/lizozom/kibana: (44 commits) [ML] Functional tests - skip DFA clone tests [Uptime] Fix synthetics detail step count (elastic#89940) Fixes the permissions to require cluster.manage in order to create an index and in order to update an index (elastic#89947) [Security Solution] [Detections] adds log info level for logging in cloud (elastic#89941) [Time to Visualize] Dashboard By Value Testing Lens (elastic#89581) [Uptime] Expand synthetic journey step thumbnail on hover (elastic#89179) TS project refs: Migrates snapshot_restore to a TS Project (elastic#89653) docs: APM 7.11 updates (elastic#89789) move skip to higher level (elastic#86952) Revert "Migrations v2: don't auto-create indices + FTR/esArchiver support (elastic#85778)" Revert "Revert "Enable v2 so migrations, disable in FTR tests (elastic#89297)"" Revert "Enable v2 so migrations, disable in FTR tests (elastic#89297)" [data.search] Allow search response to follow new hits format (elastic#88115) [Maps] Change 'create multi-layer map' title to be use-case focused (elastic#89520) skip flaky suite (elastic#86952) [Security Solution] Remove focustrap (elastic#89905) [Workplace Search] Add remaining i18n support for the Content Sources tree (elastic#89910) [esArchiver] log when migrations complete and we're done loading data (elastic#89938) Add --ssl flag to make resolver generator use ssl with kbn and elasticsearch clients (elastic#89873) TS project refs: Migrates grokdebugger (elastic#89652) ...
kibanamachine
added
the
backport missing
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
kibanamachine
removed
the
backport missing
rudolf
added a commit
that referenced
this pull request
Feb 6, 2021
…upport (#85778)" (#89992) * Revert "Revert "Migrations v2: don't auto-create indices + FTR/esArchiver support (#85778)"" This reverts commit f979580. * Fix flaky saved objects management test #89953 * If a clone target exists, wait for yellow, not green, index status * Fix test after master merge * Fix types Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Feb 8, 2021
* master: (55 commits) [APM-UI][E2E] use githubNotify step (elastic#90514) [APM] Export ProcessorEvent type (elastic#90540) [Lens] Retain column config (elastic#90048) [Data Table] Add unit tests (elastic#90173) Migrate most plugins to synchronous lifecycle (elastic#89562) skip flaky suite (elastic#90555) skip flaky suite (elastic#64473) [actions] improve email action doc (elastic#90020) [Fleet] Support Fleet server system indices (elastic#89372) skip flaky suite (elastic#90552) Bump immer dependencies (elastic#90267) Unrevert "Migrations v2: don't auto-create indices + FTR/esArchiver support (elastic#85778)" (elastic#89992) [Search Sessions] Use sync config (elastic#90138) chore(NA): add safe guard to remove bazelisk from yarn global at bootstrap (elastic#90538) [test] Await retry.waitFor (elastic#90456) chore(NA): integrate build buddy with our bazel setup and remote cache for ci (elastic#90116) Skip failing suite (elastic#90526) [Fleet] Fix incorrect conversion of string to numeric values in agent YAML (elastic#90371) [Docs] Update reporting troubleshooting for arm rhel/centos (elastic#90385) chore(NA): build bazel projects all at once in the distributable build process (elastic#90328) ...
rudolf
added a commit
to rudolf/kibana
that referenced
this pull request
Feb 9, 2021
…upport (elastic#85778)" (elastic#89992) * Revert "Revert "Migrations v2: don't auto-create indices + FTR/esArchiver support (elastic#85778)"" This reverts commit f979580. * Fix flaky saved objects management test elastic#89953 * If a clone target exists, wait for yellow, not green, index status * Fix test after master merge * Fix types Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
spalger
added a commit
to spalger/kibana
that referenced
this pull request
Feb 13, 2021
…chiver support (elastic#85778)" (elastic#89992)" This reverts commit fd1d965.
rudolf
added a commit
that referenced
this pull request
Feb 16, 2021
…iver support (#85778)" (#89992) (#90839) * Unrevert "Migrations v2: don't auto-create indices + FTR/esArchiver support (#85778)" (#89992) * Revert "Revert "Migrations v2: don't auto-create indices + FTR/esArchiver support (#85778)"" This reverts commit f979580. * Fix flaky saved objects management test #89953 * If a clone target exists, wait for yellow, not green, index status * Fix test after master merge * Fix types Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> * Fix visualize fixture after backport * Disable monitoring plugin email notifications in functional tests * Disabling monitoring emails doesnt disable it, configure instead * Fix monitoring test * Remove debug logging from FTR config Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Saved objects no longer auto-creates indices
The new migrations aren't compatible with an auto-created saved objects index, so when tests delete
.kibanaand then use the saved objects client, the next test will fail because migrations fail. We could hack the v2 migrations to still work on the auto-created indices, but this is never the behaviour a user wants. If the saved objects index was deleted while Kibana was running we should "stop the world" and let users investigate. Our current behaviour means users sometimes end up with some changes in a new index and cannot "merge" the two indices and causes all sorts of runtime failures because of incompatible mappings or upgrade failures.So this PR starts implementing the decision in #81790 to never auto-create system indices. I'm using the
require_aliasquery parameter which means we could theoretically miss a route, but something is better than nothing until we move to a system index.It reverses the decision in #14141 where we formalized the auto-create index behaviour in 6.1. However, when this decision was made Kibana used an index template so an auto-created index was acceptable since it would always have the correct mappings. With this PR, if the saved objects index and aliases doesn't exist, indexing actions will return a 500. I think we should also return a 500 for read operations when the index doesn't exist since Kibana really is broken, it's not the same as not being able to find a particular saved object, but I felt like this could be postponed into a follow-up PR (created #89642 to track this).
Enables v2 migrations in all tests
Because of removing the auto-create index behaviour I had to adapt tests to always initialize an empty kibana with
esArchiver.emptyKibanaIndex()or move somebeforestatements around so that we don't set UiSettings before the index gets initialized.Closes #85911
Checklist
Delete any items that are not applicable to this PR.
For maintainers