Merge branch '7.x' into backport/7.x/pr-111663

dev_docs/key_concepts/persistable_state.mdx

@@ -0,0 +1,83 @@
+---
+id: kibDevDocsPersistableStateIntro
+slug: /kibana-dev-docs/persistable-state-intro
+title: Persistable State
+summary: Persitable state is a key concept to understand when building a Kibana plugin.
+date: 2021-02-02
+tags: ['kibana','dev', 'contributor', 'api docs']
+---
+
+ “Persistable state” is developer-defined state that supports being persisted by a plugin other than the one defining it. Persistable State needs to be serializable and the owner can/should provide utilities to migrate it, extract and inject any <DocLink id="kibDevDocsSavedObjectsIntro" section="references" text="references to Saved Objects"/> it may contain, as well as telemetry collection utilities.
+
+## Exposing state that can be persisted
+
+Any plugin that exposes state that another plugin might persist should implement <DocLink id="kibKibanaUtilsPluginApi " section="def-common.PersistableStateService" text="`PersistableStateService`"/> interface on their `setup` contract. This will allow plugins persisting the state to easily access migrations and other utilities.
+
+Example: Data plugin allows you to generate filters. Those filters can be persisted by applications in their saved 
+objects or in the URL. In order to allow apps to migrate the filters in case the structure changes in the future, the Data plugin implements `PersistableStateService` on <DocLink id="kibDataQueryPluginApi " section="def-public.FilterManager" text="`data.query.filterManager`"/>.
+
+note: There is currently no obvious way for a plugin to know which state is safe to persist. The developer must manually look for a matching `PersistableStateService`, or ad-hoc provided migration utilities (as is the case with Rule Type Parameters).
+In the future, we hope to make it easier for producers of state to understand when they need to write a migration with changes, and also make it easier for consumers of such state, to understand whether it is safe to persist.
+
+## Exposing state that can be persisted but is not owned by plugin exposing it (registry)
+
+Any plugin that owns collection of items (registry) whose state/configuration can be persisted should implement `PersistableStateService`
+interface on their `setup` contract and each item in the collection should implement <DocLink id="kibKibanaUtilsPluginApi" section="def-common.PersistableStateDefinition" text="`PersistableStateDefinition`"/> interface.
+
+Example: Embeddable plugin owns the registry of embeddable factories to which other plugins can register new embeddable factories. Dashboard plugin
+stores a bunch of embeddable panels input in its saved object and URL. Embeddable plugin setup contract implements `PersistableStateService`
+interface and each `EmbeddableFactory`  needs to implement `PersistableStateDefinition` interface.
+
+Embeddable plugin exposes this interfaces:
+```
+// EmbeddableInput implements Serializable
+
+export interface EmbeddableRegistryDefinition extends PersistableStateDefinition<EmbeddableInput> {
+  id: string;
+  ...
+}
+
+export interface EmbeddableSetup extends PersistableStateService<EmbeddableInput>;
+```
+
+Note: if your plugin doesn't expose the state (it is the only one storing state), the plugin doesn't need to implement the `PersistableStateService` interface.
+If the state your plugin is storing can be provided by other plugins (your plugin owns a registry) items in that registry still need to implement `PersistableStateDefinition` interface.
+
+## Storing persistable state as part of saved object
+
+Any plugin that stores any persistable state as part of their saved object should make sure that its saved object migration 
+and reference extraction and injection methods correctly use the matching `PersistableStateService` implementation for the state they are storing.
+
+Take a look at [example saved object](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/server/searchable_list_saved_object.ts#L32) which stores an embeddable state. Note how the `migrations`, `extractReferences` and `injectReferences` are defined.
+
+## Storing persistable state as part of URL
+
+When storing persistable state as part of URL you must make sure your URL is versioned. When loading the state `migrateToLatest` method
+of `PersistableStateService` should be called, which will migrate the state from its original version to latest.
+
+note: Currently there is no recommended way on how to store version in url and its up to every application to decide on how to implement that.
+
+## Available state operations
+
+### Extraction/Injection of References
+
+In order to support import and export, and space-sharing capabilities, Saved Objects need to explicitly list any references they contain to other Saved Objects.
+To support persisting your state in saved objects owned by another plugin, the <DocLink id="kibKibanaUtilsPluginApi" section="def-common.PersistableState.extract" text="`extract`"/> and <DocLink id="kibKibanaUtilsPluginApi" section="def-common.PersistableState.inject" text="`inject`"/> methods of Persistable State interface should be implemented.
+
+<DocLink id="kibDevTutorialSavedObject" section="references" text="Learn how to define Saved Object references"/>
+
+[See example embeddable providing extract/inject functions](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/public/migrations/migrations_embeddable_factory.ts)
+
+### Migrations and Backward compatibility
+
+As your plugin evolves, you may need to change your state in a breaking way. If that happens, you should write a migration to upgrade the state that existed prior to the change.
+
+<DocLink id="kibDevTutorialSavedObject" section="migrations" text="How to write a migration"/>.
+
+[See an example saved object storing embeddable state implementing saved object migration function](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/server/searchable_list_saved_object.ts)
+
+[See example embeddable providing migration functions](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/public/migrations/migrations_embeddable_factory.ts)
+
+## Telemetry
+
+You might want to collect statistics about how your state is used. If that is the case you should implement the telemetry method of Persistable State interface. 

examples/embeddable_examples/kibana.json

@@ -9,7 +9,7 @@
     "githubTeam": "kibana-app-services"
   },
   "description": "Example app that shows how to register custom embeddables",
-  "requiredPlugins": ["embeddable", "uiActions", "savedObjects", "dashboard"],
+  "requiredPlugins": ["embeddable", "uiActions", "savedObjects", "dashboard", "kibanaUtils"],
   "optionalPlugins": [],
   "extraPublicDirs": ["public/todo", "public/hello_world", "public/todo/todo_ref_embeddable"],
   "requiredBundles": ["kibanaReact"]

examples/embeddable_examples/public/migrations/migrations_embeddable_factory.ts

@@ -7,6 +7,7 @@
  */
 
 import { i18n } from '@kbn/i18n';
+import { EmbeddableStateWithType } from '../../../../src/plugins/embeddable/common';
 import {
   IContainer,
   EmbeddableInput,
@@ -35,6 +36,16 @@ export class SimpleEmbeddableFactoryDefinition
     '7.3.0': migration730,
   };
 
+  public extract(state: EmbeddableStateWithType) {
+    // this embeddable does not store references to other saved objects
+    return { state, references: [] };
+  }
+
+  public inject(state: EmbeddableStateWithType) {
+    // this embeddable does not store references to other saved objects
+    return state;
+  }
+
   /**
    * In our simple example, we let everyone have permissions to edit this. Most
    * embeddables should check the UI Capabilities service to be sure of

examples/embeddable_examples/server/merge_migration_function_maps.ts

@@ -0,0 +1,25 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { mergeWith } from 'lodash';
+import type { SerializableRecord } from '@kbn/utility-types';
+import { MigrateFunctionsObject, MigrateFunction } from '../../../src/plugins/kibana_utils/common';
+
+export const mergeMigrationFunctionMaps = (
+  obj1: MigrateFunctionsObject,
+  obj2: MigrateFunctionsObject
+) => {
+  const customizer = (objValue: MigrateFunction, srcValue: MigrateFunction) => {
+    if (!srcValue || !objValue) {
+      return srcValue || objValue;
+    }
+    return (state: SerializableRecord) => objValue(srcValue(state));
+  };
+
+  return mergeWith({ ...obj1 }, obj2, customizer);
+};

examples/embeddable_examples/server/searchable_list_saved_object.ts

@@ -9,9 +9,12 @@
 import { mapValues } from 'lodash';
 import { SavedObjectsType, SavedObjectUnsanitizedDoc } from 'kibana/server';
 import { EmbeddableSetup } from '../../../src/plugins/embeddable/server';
+// NOTE: this should rather be imported from 'plugins/kibana_utils/server' but examples at the moment don't
+// allow static imports from plugins so this code was duplicated
+import { mergeMigrationFunctionMaps } from './merge_migration_function_maps';
 
 export const searchableListSavedObject = (embeddable: EmbeddableSetup) => {
-  return {
+  const searchableListSO: SavedObjectsType = {
     name: 'searchableList',
     hidden: false,
     namespaceType: 'single',
@@ -30,14 +33,22 @@ export const searchableListSavedObject = (embeddable: EmbeddableSetup) => {
       },
     },
     migrations: () => {
-      // we assume all the migration will be done by embeddables service and that saved object holds no extra state besides that of searchable list embeddable input\
-      // if saved object would hold additional information we would need to merge the response from embeddables.getAllMigrations with our custom migrations.
-      return mapValues(embeddable.getAllMigrations(), (migrate) => {
+      // there are no migrations defined for the saved object at the moment, possibly they would be added in the future
+      const searchableListSavedObjectMigrations = {};
+
+      // we don't know if embeddables have any migrations defined so we need to fetch them and map the received functions so we pass
+      // them the correct input and that we correctly map the response
+      const embeddableMigrations = mapValues(embeddable.getAllMigrations(), (migrate) => {
         return (state: SavedObjectUnsanitizedDoc) => ({
           ...state,
           attributes: migrate(state.attributes),
         });
       });
+
+      // we merge our and embeddable migrations and return
+      return mergeMigrationFunctionMaps(searchableListSavedObjectMigrations, embeddableMigrations);
     },
-  } as SavedObjectsType;
+  };
+
+  return searchableListSO;
 };

src/plugins/kibana_utils/server/index.ts

@@ -15,6 +15,7 @@ export {
   Get,
   Set,
   url,
+  mergeMigrationFunctionMaps,
 } from '../common';
 
 export { KbnServerError, reportServerError, getKbnServerError } from './report_server_error';

x-pack/plugins/apm/public/components/app/RumDashboard/VisitorBreakdownMap/useLayerList.ts

@@ -15,6 +15,7 @@ import {
   COLOR_MAP_TYPE,
   FIELD_ORIGIN,
   LABEL_BORDER_SIZES,
+  LAYER_TYPE,
   SOURCE_TYPES,
   STYLE_TYPE,
   SYMBOLIZE_AS_TYPES,
@@ -154,7 +155,7 @@ export function useLayerList() {
     maxZoom: 24,
     alpha: 0.75,
     visible: true,
-    type: 'VECTOR',
+    type: LAYER_TYPE.VECTOR,
   };
 
   ES_TERM_SOURCE_REGION.whereQuery = getWhereQuery(serviceName!);
@@ -178,7 +179,7 @@ export function useLayerList() {
     maxZoom: 24,
     alpha: 0.75,
     visible: true,
-    type: 'VECTOR',
+    type: LAYER_TYPE.VECTOR,
   };
 
   return [

x-pack/plugins/data_visualizer/public/application/common/components/stats_table/components/field_data_expanded_row/choropleth_map.tsx

@@ -11,6 +11,7 @@ import { i18n } from '@kbn/i18n';
 import { FormattedMessage } from '@kbn/i18n/react';
 import {
   FIELD_ORIGIN,
+  LAYER_TYPE,
   SOURCE_TYPES,
   STYLE_TYPE,
   COLOR_MAP_TYPE,
@@ -85,7 +86,7 @@ export const getChoroplethTopValuesLayer = (
       },
       isTimeAware: true,
     },
-    type: 'VECTOR',
+    type: LAYER_TYPE.VECTOR,
   };
 };
 

x-pack/plugins/maps/common/descriptor_types/layer_descriptor_types.ts

@@ -23,6 +23,7 @@ import {
   KBN_IS_TILE_COMPLETE,
   KBN_METADATA_FEATURE,
   KBN_VECTOR_SHAPE_TYPE_COUNTS,
+  LAYER_TYPE,
 } from '../constants';
 
 export type Attribution = {
@@ -56,7 +57,6 @@ export type LayerDescriptor = {
   alpha?: number;
   attribution?: Attribution;
   id: string;
-  joins?: JoinDescriptor[];
   label?: string | null;
   areLabelsOnTop?: boolean;
   minZoom?: number;
@@ -70,9 +70,12 @@ export type LayerDescriptor = {
 };
 
 export type VectorLayerDescriptor = LayerDescriptor & {
+  type: LAYER_TYPE.VECTOR | LAYER_TYPE.TILED_VECTOR | LAYER_TYPE.BLENDED_VECTOR;
+  joins?: JoinDescriptor[];
   style: VectorStyleDescriptor;
 };
 
 export type HeatmapLayerDescriptor = LayerDescriptor & {
+  type: LAYER_TYPE.HEATMAP;
   style: HeatmapStyleDescriptor;
 };

x-pack/plugins/maps/common/index.ts

@@ -12,6 +12,7 @@ export {
   FIELD_ORIGIN,
   INITIAL_LOCATION,
   LABEL_BORDER_SIZES,
+  LAYER_TYPE,
   MAP_SAVED_OBJECT_TYPE,
   SOURCE_TYPES,
   STYLE_TYPE,

x-pack/plugins/maps/common/migrations/add_type_to_termjoin.ts

@@ -6,8 +6,8 @@
  */
 
 import { MapSavedObjectAttributes } from '../map_saved_object_type';
-import { JoinDescriptor, LayerDescriptor } from '../descriptor_types';
-import { LAYER_TYPE, SOURCE_TYPES } from '../constants';
+import { JoinDescriptor, LayerDescriptor, VectorLayerDescriptor } from '../descriptor_types';
+import { SOURCE_TYPES } from '../constants';
 
 // enforce type property on joins. It's possible older saved-objects do not have this correctly filled in
 // e.g. sample-data was missing the right.type field.
@@ -24,14 +24,15 @@ export function addTypeToTermJoin({
   const layerList: LayerDescriptor[] = JSON.parse(attributes.layerListJSON);
 
   layerList.forEach((layer: LayerDescriptor) => {
-    if (layer.type !== LAYER_TYPE.VECTOR) {
+    if (!('joins' in layer)) {
       return;
     }
 
-    if (!layer.joins) {
+    const vectorLayer = layer as VectorLayerDescriptor;
+    if (!vectorLayer.joins) {
       return;
     }
-    layer.joins.forEach((join: JoinDescriptor) => {
+    vectorLayer.joins.forEach((join: JoinDescriptor) => {
       if (!join.right) {
         return;
       }

x-pack/plugins/maps/common/migrations/references.ts

@@ -9,7 +9,7 @@
 
 import { SavedObjectReference } from '../../../../../src/core/types';
 import { MapSavedObjectAttributes } from '../map_saved_object_type';
-import { LayerDescriptor } from '../descriptor_types';
+import { LayerDescriptor, VectorLayerDescriptor } from '../descriptor_types';
 
 interface IndexPatternReferenceDescriptor {
   indexPatternId?: string;
@@ -44,21 +44,24 @@ export function extractReferences({
       sourceDescriptor.indexPatternRefName = refName;
     }
 
-    // Extract index-pattern references from join
-    const joins = layer.joins ? layer.joins : [];
-    joins.forEach((join, joinIndex) => {
-      if ('indexPatternId' in join.right) {
-        const sourceDescriptor = join.right as IndexPatternReferenceDescriptor;
-        const refName = `layer_${layerIndex}_join_${joinIndex}_index_pattern`;
-        extractedReferences.push({
-          name: refName,
-          type: 'index-pattern',
-          id: sourceDescriptor.indexPatternId!,
-        });
-        delete sourceDescriptor.indexPatternId;
-        sourceDescriptor.indexPatternRefName = refName;
-      }
-    });
+    if ('joins' in layer) {
+      // Extract index-pattern references from join
+      const vectorLayer = layer as VectorLayerDescriptor;
+      const joins = vectorLayer.joins ? vectorLayer.joins : [];
+      joins.forEach((join, joinIndex) => {
+        if ('indexPatternId' in join.right) {
+          const sourceDescriptor = join.right as IndexPatternReferenceDescriptor;
+          const refName = `layer_${layerIndex}_join_${joinIndex}_index_pattern`;
+          extractedReferences.push({
+            name: refName,
+            type: 'index-pattern',
+            id: sourceDescriptor.indexPatternId!,
+          });
+          delete sourceDescriptor.indexPatternId;
+          sourceDescriptor.indexPatternRefName = refName;
+        }
+      });
+    }
   });
 
   return {
@@ -99,16 +102,19 @@ export function injectReferences({
       delete sourceDescriptor.indexPatternRefName;
     }
 
-    // Inject index-pattern references into join
-    const joins = layer.joins ? layer.joins : [];
-    joins.forEach((join) => {
-      if ('indexPatternRefName' in join.right) {
-        const sourceDescriptor = join.right as IndexPatternReferenceDescriptor;
-        const reference = findReference(sourceDescriptor.indexPatternRefName!, references);
-        sourceDescriptor.indexPatternId = reference.id;
-        delete sourceDescriptor.indexPatternRefName;
-      }
-    });
+    if ('joins' in layer) {
+      // Inject index-pattern references into join
+      const vectorLayer = layer as VectorLayerDescriptor;
+      const joins = vectorLayer.joins ? vectorLayer.joins : [];
+      joins.forEach((join) => {
+        if ('indexPatternRefName' in join.right) {
+          const sourceDescriptor = join.right as IndexPatternReferenceDescriptor;
+          const reference = findReference(sourceDescriptor.indexPatternRefName!, references);
+          sourceDescriptor.indexPatternId = reference.id;
+          delete sourceDescriptor.indexPatternRefName;
+        }
+      });
+    }
   });
 
   return {