Resilient saved object migration algorithm

docs/development/core/server/kibana-plugin-core-server.savedobjectsrawdoc.md

@@ -20,5 +20,4 @@ export interface SavedObjectsRawDoc
 |  [\_primary\_term](./kibana-plugin-core-server.savedobjectsrawdoc._primary_term.md) | <code>number</code> |  |
 |  [\_seq\_no](./kibana-plugin-core-server.savedobjectsrawdoc._seq_no.md) | <code>number</code> |  |
 |  [\_source](./kibana-plugin-core-server.savedobjectsrawdoc._source.md) | <code>SavedObjectsRawDocSource</code> |  |
-|  [\_type](./kibana-plugin-core-server.savedobjectsrawdoc._type.md) | <code>string</code> |  |
 

rfcs/text/0013_saved_object_migrations.md

@@ -214,31 +214,43 @@ Note:
 2. If the source is a < v6.5 `.kibana` index or < 7.4 `.kibana_task_manager`
    index prepare the legacy index for a migration:
    1. Mark the legacy index as read-only and wait for all in-flight operations to drain (requires https://github.com/elastic/elasticsearch/pull/58094). This prevents any further writes from outdated nodes. Assuming this API is similar to the existing `/<index>/_close` API, we expect to receive `"acknowledged" : true` and `"shards_acknowledged" : true`. If all shards don’t acknowledge within the timeout, retry the operation until it succeeds.
-   2. Clone the legacy index into a new index which has writes enabled. Use a fixed index name i.e `.kibana_pre6.5.0_001` or `.kibana_task_manager_pre7.4.0_001`. `POST /.kibana/_clone/.kibana_pre6.5.0_001?wait_for_active_shards=all {"settings": {"index.blocks.write": false}}`. Ignore errors if the clone already exists. Ignore errors if the legacy source doesn't exist.
-   3. Wait for the cloning to complete `GET /_cluster/health/.kibana_pre6.5.0_001?wait_for_status=green&timeout=60s` If cloning doesn’t complete within the 60s timeout, log a warning for visibility and poll again.
-   4. Apply the `convertToAlias` script if defined `POST /.kibana_pre6.5.0_001/_update_by_query?conflicts=proceed {"script": {...}}`. The `convertToAlias` script will have to be idempotent, preferably setting `ctx.op="noop"` on subsequent runs to avoid unecessary writes.
+   2. Create a new index which will become the source index after the legacy
+      pre-migration is complete. This index should have the same mappings as
+      the legacy index. Use a fixed index name i.e `.kibana_pre6.5.0_001` or
+      `.kibana_task_manager_pre7.4.0_001`. Ignore index already exists errors.
+   3. Reindex the legacy index into the new source index with the
+      `convertToAlias` script if specified. Use `wait_for_completion: false`
+      to run this as a task. Ignore errors if the legacy source doesn't exist. 
+   4. Wait for the reindex task to complete. If the task doesn’t complete
+      within the 60s timeout, log a warning for visibility and poll again.
+      Ignore errors if the legacy source doesn't exist.
    5. Delete the legacy index and replace it with an alias of the same name
       ```
       POST /_aliases
       {
         "actions" : [
-            { "add":  { "index": ".kibana_pre6.5.0_001", "alias": ".kibana" } },
             { "remove_index": { "index": ".kibana" } }
+            { "add":  { "index": ".kibana_pre6.5.0_001", "alias": ".kibana" } },
         ]
       }
       ```.
       Unlike the delete index API, the `remove_index` action will fail if
-      provided with an _alias_. Ignore "The provided expression [.kibana]
-      matches an alias, specify the corresponding concrete indices instead."
-      or "index_not_found_exception" errors. These actions are applied
-      atomically so that other Kibana instances will always see either a
-      `.kibana` index or an alias, but never neither.
-   6. Use the cloned `.kibana_pre6.5.0_001` as the source for the rest of the migration algorithm.
+      provided with an _alias_. Therefore, if another instance completed this
+      step, the `.kibana` alias won't be added to `.kibana_pre6.5.0_001` a
+      second time. This avoids a situation where `.kibana` could point to both
+      `.kibana_pre6.5.0_001` and `.kibana_7.10.0_001`. These actions are
+      applied atomically so that other Kibana instances will always see either
+      a `.kibana` index or an alias, but never neither. 
+
+      Ignore "The provided expression [.kibana] matches an alias, specify the
+      corresponding concrete indices instead." or "index_not_found_exception"
+      errors as this means another instance has already completed this step.
+   6. Use the reindexed legacy `.kibana_pre6.5.0_001` as the source for the rest of the migration algorithm.
 3. If `.kibana` and `.kibana_7.10.0` both exists and are pointing to the same index this version's migration has already been completed.
    1. Because the same version can have plugins enabled at any point in time,
-      perform the mappings update in step (6) and migrate outdated documents
-      with step (7).
-   2. Skip to step (9) to start serving traffic.
+      perform the mappings update in step (7) and migrate outdated documents
+      with step (8).
+   2. Skip to step (10) to start serving traffic.
 4. Fail the migration if:
    1. `.kibana` is pointing to an index that belongs to a later version of Kibana .e.g. `.kibana_7.12.0_001`
    2. (Only in 8.x) The source index contains documents that belong to an unknown Saved Object type (from a disabled plugin). Log an error explaining that the plugin that created these documents needs to be enabled again or that these objects should be deleted. See section (4.2.1.4).

src/core/public/public.api.md

@@ -7,6 +7,7 @@
 import { Action } from 'history';
 import { ApiResponse } from '@elastic/elasticsearch/lib/Transport';
 import Boom from '@hapi/boom';
+import { ConfigDeprecationProvider } from '@kbn/config';
 import { ConfigPath } from '@kbn/config';
 import { EnvironmentMode } from '@kbn/config';
 import { EuiBreadcrumb } from '@elastic/eui';
@@ -18,7 +19,6 @@ import { History } from 'history';
 import { Href } from 'history';
 import { IconType } from '@elastic/eui';
 import { KibanaClient } from '@elastic/elasticsearch/api/kibana';
-import { KibanaConfigType } from 'src/core/server/kibana_config';
 import { Location } from 'history';
 import { LocationDescriptorObject } from 'history';
 import { Logger } from '@kbn/logging';

src/core/server/elasticsearch/client/mocks.ts

@@ -22,7 +22,9 @@ import type { DeeplyMockedKeys } from '@kbn/utility-types/jest';
 import { ElasticsearchClient } from './types';
 import { ICustomClusterClient } from './cluster_client';
 
-const createInternalClientMock = (): DeeplyMockedKeys<Client> => {
+const createInternalClientMock = (
+  res?: MockedTransportRequestPromise<unknown>
+): DeeplyMockedKeys<Client> => {
   // we mimic 'reflection' on a concrete instance of the client to generate the mocked functions.
   const client = new Client({
     node: 'http://localhost',
@@ -59,7 +61,7 @@ const createInternalClientMock = (): DeeplyMockedKeys<Client> => {
       .filter(([key]) => !omitted.includes(key))
       .forEach(([key, descriptor]) => {
         if (typeof descriptor.value === 'function') {
-          obj[key] = jest.fn(() => createSuccessTransportRequestPromise({}));
+          obj[key] = jest.fn(() => res ?? createSuccessTransportRequestPromise({}));
         } else if (typeof obj[key] === 'object' && obj[key] != null) {
           mockify(obj[key], omitted);
         }
@@ -95,8 +97,8 @@ const createInternalClientMock = (): DeeplyMockedKeys<Client> => {
 
 export type ElasticsearchClientMock = DeeplyMockedKeys<ElasticsearchClient>;
 
-const createClientMock = (): ElasticsearchClientMock =>
-  (createInternalClientMock() as unknown) as ElasticsearchClientMock;
+const createClientMock = (res?: MockedTransportRequestPromise<unknown>): ElasticsearchClientMock =>
+  (createInternalClientMock(res) as unknown) as ElasticsearchClientMock;
 
 export interface ScopedClusterClientMock {
   asInternalUser: ElasticsearchClientMock;

src/core/server/saved_objects/migrations/core/document_migrator.ts

@@ -312,7 +312,7 @@ function wrapWithTry(
       const failedTransform = `${type}:${version}`;
       const failedDoc = JSON.stringify(doc);
       log.warn(
-        `Failed to transform document ${doc}. Transform: ${failedTransform}\nDoc: ${failedDoc}`
+        `Failed to transform document ${doc?.id}. Transform: ${failedTransform}\nDoc: ${failedDoc}`
       );
       throw error;
     }

src/core/server/saved_objects/migrations/core/migration_context.ts

@@ -24,7 +24,7 @@
  * serves as a central blueprint for what migrations will end up doing.
  */
 
-import { Logger } from 'src/core/server/logging';
+import { Logger } from '../../../logging';
 import { MigrationEsClient } from './migration_es_client';
 import { SavedObjectsSerializer } from '../../serialization';
 import {

src/core/server/saved_objects/migrations/kibana/kibana_migrator.mock.ts

@@ -16,9 +16,7 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-import type { PublicMethodsOf } from '@kbn/utility-types';
-
-import { KibanaMigrator, KibanaMigratorStatus } from './kibana_migrator';
+import { IKibanaMigrator, KibanaMigratorStatus } from './kibana_migrator';
 import { buildActiveMappings } from '../core';
 const { mergeTypes } = jest.requireActual('./kibana_migrator');
 import { SavedObjectsType } from '../../types';
@@ -45,7 +43,16 @@ const createMigrator = (
     types: SavedObjectsType[];
   } = { types: defaultSavedObjectTypes }
 ) => {
-  const mockMigrator: jest.Mocked<PublicMethodsOf<KibanaMigrator>> = {
+  const mockMigrator: jest.Mocked<IKibanaMigrator> = {
+    kibanaVersion: '8.0.0-testing',
+    savedObjectsConfig: {
+      batchSize: 100,
+      scrollDuration: '15m',
+      pollInterval: 1500,
+      skip: false,
+      // TODO migrationsV2: remove/deprecate once we release migrations v2
+      enableV2: false,
+    },
     runMigrations: jest.fn(),
     getActiveMappings: jest.fn(),
     migrateDocument: jest.fn(),