From 78a030d5c61b3747a607a87817462f76217de25e Mon Sep 17 00:00:00 2001 From: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Date: Thu, 24 Jun 2021 18:56:32 -0400 Subject: [PATCH] [docs][migrations v2] Update SO migration docs to include removal of index write block when handling corrupt SOs. (#103014) (#103372) Co-authored-by: Luke Elmers --- .../setup/upgrade/upgrade-migrations.asciidoc | 49 ++++++++++++++++--- 1 file changed, 41 insertions(+), 8 deletions(-) diff --git a/docs/setup/upgrade/upgrade-migrations.asciidoc b/docs/setup/upgrade/upgrade-migrations.asciidoc index fdcd71791ad3a..947043b21ef50 100644 --- a/docs/setup/upgrade/upgrade-migrations.asciidoc +++ b/docs/setup/upgrade/upgrade-migrations.asciidoc @@ -55,22 +55,55 @@ This section highlights common causes of {kib} upgrade failures and how to preve There is a known issue in v7.12.0 for users who tried the fleet beta. Upgrade migrations fail because of a large number of documents in the `.kibana` index. This can cause Kibana to log errors like: -> Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms] -> Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54] + +[source,sh] +-------------------------------------------- +Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms] + +Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54] +-------------------------------------------- See https://github.com/elastic/kibana/issues/95321 for instructions to work around this issue. [float] ===== Corrupt saved objects -We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment. Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed. +We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment. + +Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed. For example, given the following error message: -> Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index. -The following steps must be followed to allow the upgrade migration to succeed. -Please be aware the Dashboard having ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` belonging to the space `marketing_space` will no more be available: -1. Delete the corrupt document with `DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275` -2. Restart {kib} +[source,sh] +-------------------------------------------- +Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index. +-------------------------------------------- + +The following steps must be followed to delete the document that is causing the migration to fail: + +. Remove the write block which the migration system has placed on the previous index: ++ +[source,sh] +-------------------------------------------- +PUT .kibana_7.12.1_001/_settings +{ + "index": { + "blocks.write": false + } +} +-------------------------------------------- + +. Delete the corrupt document: ++ +[source,sh] +-------------------------------------------- +DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275 +-------------------------------------------- + +. Restart {kib}. + +In this example, the Dashboard with ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` that belongs to the space `marketing_space` **will no longer be available**. + +Be sure you have a snapshot before you delete the corrupt document. If restoring from a snapshot is not an option, it is recommended to also delete the `temp` and `target` indices the migration created before restarting {kib} and retrying. [float] ===== User defined index templates that causes new `.kibana*` indices to have incompatible settings or mappings