Basic server side import API for saved objects (elastic#32158) (elast…

…ic#32790)

* Initial work

* Add overwrite and skip support

* Cleanup and add tests

* Move code into separate files

* Remove reduce

* New API parameters

* Add support to replace references

* Add better error handling

* Add spaces tests

* Fix return type in collectSavedObjects

* Apply PR feedback

* Update jest tests due to jest version upgrade

* Add docs

* WIP

* Split import routes pt1

* Add tests

* Fix broken tests

* Update docs and fix broken test

* Add successCount to _import endpoint

* Make skip by default in resolution API

* Update tests for removal of skips

* Add back support for skips

* Add success count

* Add back resolve import conflicts x-pack tests

* Remove writev from filter stream

* Delete _mock_server.d.ts file

* Rename lib/import_saved_objects to lib/import

* Filter records at stream level for conflict resolution

* Update docs

* Add tests to validate documentation

* Return 200 instead of other code for errors, include errors array

* Change [] to {}

* Apply PR feedback

* Fix import object limit to not return 500

* Change some wording in the docs

* Fix status code

* Apply PR feedback pt2

* Lower maxImportPayloadBytes to 10MB

* Add unknown type tests for import

* Add unknown type tests for resolve_import_conflicts

* Fix tslint issues

docs/api/saved-objects.asciidoc

@@ -19,6 +19,8 @@ NOTE: You cannot access these endpoints via the Console in Kibana.
 * <<saved-objects-api-update>>
 * <<saved-objects-api-delete>>
 * <<saved-objects-api-export>>
+* <<saved-objects-api-import>>
+* <<saved-objects-api-resolve-import-conflicts>>
 
 include::saved-objects/get.asciidoc[]
 include::saved-objects/bulk_get.asciidoc[]
@@ -28,3 +30,5 @@ include::saved-objects/bulk_create.asciidoc[]
 include::saved-objects/update.asciidoc[]
 include::saved-objects/delete.asciidoc[]
 include::saved-objects/export.asciidoc[]
+include::saved-objects/import.asciidoc[]
+include::saved-objects/resolve_import_conflicts.asciidoc[]

docs/api/saved-objects/import.asciidoc

@@ -0,0 +1,96 @@
+[[saved-objects-api-import]]
+=== Import Objects
+
+experimental[This functionality is *experimental* and may be changed or removed completely in a future release.]
+
+The import saved objects API enables you to create a set of Kibana saved objects from a file created by the export API.
+
+Note: You cannot access this endpoint via the Console in Kibana.
+
+==== Request
+
+`POST /api/saved_objects/_import`
+
+==== Query Parameters
+
+`overwrite` (optional)::
+  (boolean) Overwrite saved objects if they exist already
+
+==== Request body
+
+The request body must be of type multipart/form-data.
+
+`file`::
+  A file exported using the export API.
+
+==== Response body
+
+The response body will have a top level `success` property that indicates
+if the import was successful or not as well as a `successCount` indicating how many records are successfully imported.
+In the scenario the import wasn't successful a top level `errors` array will contain the objects that failed to import.
+
+==== Examples
+
+The following example imports an index pattern and dashboard.
+
+[source,js]
+--------------------------------------------------
+POST api/saved_objects/_import
+Content-Type: multipart/form-data; boundary=EXAMPLE
+--EXAMPLE
+Content-Disposition: form-data; name="file"; filename="export.ndjson"
+Content-Type: application/ndjson
+
+{"type":"index-pattern","id":"my-pattern","attributes":{"title":"my-pattern-*"}}
+{"type":"dashboard","id":"my-dashboard","attributes":{"title":"Look at my dashboard"}}
+--EXAMPLE--
+--------------------------------------------------
+// KIBANA
+
+A successful call returns a response code of `200` and a response body
+containing a JSON structure similar to the following example:
+
+[source,js]
+--------------------------------------------------
+{
+  "success": true,
+  "successCount": 2
+}
+--------------------------------------------------
+
+The following example imports an index pattern and dashboard but has a conflict on the index pattern.
+
+[source,js]
+--------------------------------------------------
+POST api/saved_objects/_import
+Content-Type: multipart/form-data; boundary=EXAMPLE
+--EXAMPLE
+Content-Disposition: form-data; name="file"; filename="export.ndjson"
+Content-Type: application/ndjson
+
+{"type":"index-pattern","id":"my-pattern","attributes":{"title":"my-pattern-*"}}
+{"type":"dashboard","id":"my-dashboard","attributes":{"title":"Look at my dashboard"}}
+--EXAMPLE--
+--------------------------------------------------
+// KIBANA
+
+The call returns a response code of `200` and a response body
+containing a JSON structure similar to the following example:
+
+[source,js]
+--------------------------------------------------
+{
+  "success": false,
+  "successCount": 1,
+  "errors": [
+    {
+      "id": "my-pattern",
+      "type": "index-pattern",
+      "error": {
+        "statusCode": 409,
+        "message": "version conflict, document already exists",
+      },
+    },
+  ],
+}
+--------------------------------------------------

docs/api/saved-objects/resolve_import_conflicts.asciidoc

@@ -0,0 +1,104 @@
+[[saved-objects-api-resolve-import-conflicts]]
+=== Resolve Import Conflicts
+
+experimental[This functionality is *experimental* and may be changed or removed completely in a future release.]
+
+The resolve import conflicts API enables you to resolve conflicts given by the import API by either overwriting specific saved objects or changing references to a newly created object.
+
+Note: You cannot access this endpoint via the Console in Kibana.
+
+==== Request
+
+`POST /api/saved_objects/_resolve_import_conflicts`
+
+==== Request body
+
+The request body must be of type multipart/form-data.
+
+`file`::
+  (ndjson) The same new line delimited JSON objects given to the import API.
+
+`overwrites` (optional)::
+  (array) A list of `type` and `id` objects allowed to be overwritten on import.
+
+`replaceReferences` (optional)::
+  (array) A list of `type`, `from` and `to` used to change imported saved object references to.
+
+`skips` (optional)::
+  (array) A list of `type` and `id` objects to skip importing.
+
+==== Response body
+
+The response body will have a top level `success` property that indicates
+if the import was successful or not as well as a `successCount` indicating how many records are successfully resolved.
+In the scenario the import wasn't successful a top level `errors` array will contain the objects that failed to import.
+
+==== Examples
+
+The following example resolves conflicts for an index pattern and dashboard but indicates to skip the index pattern.
+This will cause the index pattern to not be in the system and the dashboard to overwrite the existing saved object.
+
+[source,js]
+--------------------------------------------------
+POST api/saved_objects/_resolve_import_conflicts
+Content-Type: multipart/form-data; boundary=EXAMPLE
+--EXAMPLE
+Content-Disposition: form-data; name="file"; filename="export.ndjson"
+Content-Type: application/ndjson
+
+{"type":"index-pattern","id":"my-pattern","attributes":{"title":"my-pattern-*"}}
+{"type":"dashboard","id":"my-dashboard","attributes":{"title":"Look at my dashboard"}}
+--EXAMPLE
+Content-Disposition: form-data; name="skips"
+
+[{"type":"index-pattern","id":"my-pattern"}]
+--EXAMPLE
+Content-Disposition: form-data; name="overwrites"
+
+[{"type":"dashboard","id":"my-dashboard"}]
+--EXAMPLE--
+--------------------------------------------------
+// KIBANA
+
+A successful call returns a response code of `200` and a response body
+containing a JSON structure similar to the following example:
+
+[source,js]
+--------------------------------------------------
+{
+  "success": true,
+  "successCount": 1
+}
+--------------------------------------------------
+
+The following example resolves conflicts for a visualization and dashboard but indicates
+to replace the dashboard references to another visualization.
+
+[source,js]
+--------------------------------------------------
+POST api/saved_objects/_resolve_import_conflicts
+Content-Type: multipart/form-data; boundary=EXAMPLE
+--EXAMPLE
+Content-Disposition: form-data; name="file"; filename="export.ndjson"
+Content-Type: application/ndjson
+
+{"type":"visualization","id":"my-vis","attributes":{"title":"Look at my visualization"}}
+{"type":"dashboard","id":"my-dashboard","attributes":{"title":"Look at my dashboard"},"references":[{"name":"panel_0","type":"visualization","id":"my-vis"}]}
+--EXAMPLE
+Content-Disposition: form-data; name="replaceReferences"
+
+[{"type":"visualization","from":"my-vis","to":"my-vis-2"}]
+--EXAMPLE--
+--------------------------------------------------
+// KIBANA
+
+A successful call returns a response code of `200` and a response body
+containing a JSON structure similar to the following example:
+
+[source,js]
+--------------------------------------------------
+{
+  "success": true,
+  "successCount": 1
+}
+--------------------------------------------------

src/legacy/server/config/schema.js

@@ -252,6 +252,7 @@ export default () => Joi.object({
   }).default(),
 
   savedObjects: Joi.object({
+    maxImportPayloadBytes: Joi.number().default(10485760),
     maxImportExportSize: Joi.number().default(10000),
   }).default(),