[EPM][Security Solution] Implementing dataset component templates #70517

jonathan-buttner · 2020-07-01T21:43:49Z

This PR adds support for installing two component templates for each dataset that specifies the elasticsearch field.

A settings and mappings component template will be created depending on whether mappings and/or settings is included in the dataset of the package in the registry.

The fields being added to the Dataset interface correspond to here: https://github.com/elastic/package-registry/blob/master/util/dataset.go#L42 and this PR: https://github.com/elastic/package-registry/pull/552/files

Endpoint package installed component templates

Endpoint package mapping now including dynamic: false

elasticmachine · 2020-07-01T21:43:51Z

Pinging @elastic/endpoint-data-visibility-team (Team:Endpoint Data Visibility)

elasticmachine · 2020-07-01T21:43:53Z

Pinging @elastic/endpoint-app-team (Feature:Endpoint)

elasticmachine · 2020-07-01T21:43:54Z

Pinging @elastic/ingest-management (Team:Ingest Management)

jfsiii

Thanks for the link to the Registry type and updating Ingest's TS types. LGTM

jfsiii · 2020-07-02T07:13:48Z

x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/install.ts

+    componentPromises.push(settings.clusterPromise);
+  }
+
+  // TODO: Check return values for errors


Add/link a ticket?

Good call. I created a new ticket here: #70586

I'll update the second link once this PR is merged.

jfsiii · 2020-07-02T07:18:20Z

x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.ts

 }): IndexTemplate {
-  const template = getBaseTemplate(type, templateName, mappings, packageName);
+  const template = getBaseTemplate(type, templateName, mappings, packageName, composedOfTemplates);


5 params is hefty, but seemingly not introduced here and can be reduced later

ruflin · 2020-07-02T14:46:57Z

I wonder how #64760 plays into this? I assume these are additional components? Mentioning here so we can plan on the naming side for it.

…t-mappings-settings

jonathan-buttner · 2020-07-02T20:03:42Z

I wonder how #64760 plays into this? I assume these are additional components? Mentioning here so we can plan on the naming side for it.

Yeah I guess I kind of stole the names that we'd probably use 😆 . We can always refactor it so that the Elasticsearch overrides from the manifest.yml would have names like <template name>-override-<mappings|settings> something like that maybe?

jonathan-buttner · 2020-07-02T22:44:42Z

@elasticmachine merge upstream

jonathan-buttner · 2020-07-03T00:29:59Z

x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/install.ts

+          ...registryElasticsearch['index_template.mappings'],
+          // temporary change until https://github.com/elastic/elasticsearch/issues/58956 is resolved
+          properties: {
+            '@timestamp': {


I think I'll be able to remove this once this: elastic/elasticsearch#58642 is merged.

ruflin · 2020-07-03T09:33:13Z

++ on refactoring and making sure the "override" are in its own place and cannot conflict. Perhaps we should use _override to not conflict with a potential namespace. But not important at the moment.

…t-mappings-settings

…gest-support-mappings-settings

x-pack/test/ingest_manager_api_integration/apis/file.ts

x-pack/test/ingest_manager_api_integration/apis/fixtures/package_registry_config.yml

x-pack/test/ingest_manager_api_integration/config.ts

jonathan-buttner · 2020-07-06T16:37:12Z

x-pack/test/ingest_manager_api_integration/apis/list.ts

@@ -29,7 +29,7 @@ export default function ({ getService }: FtrProviderContext) {
          return response.body;
        };
        const listResponse = await fetchPackageList();
-        expect(listResponse.response.length).to.be(11);
+        expect(listResponse.response.length).to.be(12);


Incrementing because I add the overrides package.

…t-mappings-settings

kibanamachine · 2020-07-06T19:02:59Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: fd973f1

Build metrics

✅ unchanged

History

💚 Build #59061 succeeded d6d9ed2
💚 Build #58554 succeeded e5be224
💚 Build #58440 succeeded 79785e0
💔 Build #58235 failed bda26bb
💔 Build #58064 failed efaea2b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…astic#70517) * Implementing dataset component templates * Fixing test * Temporary fix to include timestamp with any component template created * Update package registry docker image for CI. * Adapt to new registry filesystem layout. * Adjust tests to changed registry behavior. * Adding a test for mappings and settings overrides * Wrap all the tests in the docker check Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Sonja Krause-Harder <sonja.krause-harder@elastic.co>

…0517) (#70862) * Implementing dataset component templates * Fixing test * Temporary fix to include timestamp with any component template created * Update package registry docker image for CI. * Adapt to new registry filesystem layout. * Adjust tests to changed registry behavior. * Adding a test for mappings and settings overrides * Wrap all the tests in the docker check Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Sonja Krause-Harder <sonja.krause-harder@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Sonja Krause-Harder <sonja.krause-harder@elastic.co>

Implementing dataset component templates

efaea2b

jonathan-buttner requested review from jfsiii, skh, ruflin, michaelolo24 and a team July 1, 2020 21:43

This was referenced Jul 1, 2020

[EPM] [Endpoint] Disable dynamic mapping on a template #68380

Closed

Disable dynamic mapping for all datasets elastic/endpoint-package#31

Merged

jfsiii approved these changes Jul 2, 2020

View reviewed changes

Fixing test

bda26bb

jonathan-buttner added 2 commits July 2, 2020 15:08

Merge branch 'master' of github.com:elastic/kibana into ingest-suppor…

cdbb7e3

…t-mappings-settings

Temporary fix to include timestamp with any component template created

79785e0

ruflin added the Ingest Management:beta1 label Jul 2, 2020

ruflin assigned jonathan-buttner Jul 2, 2020

Merge branch 'master' into ingest-support-mappings-settings

e5be224

jonathan-buttner commented Jul 3, 2020

View reviewed changes

skh and others added 7 commits July 3, 2020 15:43

Update package registry docker image for CI.

1dee045

Adapt to new registry filesystem layout.

df2346e

Adjust tests to changed registry behavior.

31e15b1

Merge branch 'master' of github.com:elastic/kibana into ingest-suppor…

715e218

…t-mappings-settings

Merge branch 'master' into update-registry-in-ci

74c1446

Merge branch 'update-registry-in-ci' of github.com:skh/kibana into in…

710dc53

…gest-support-mappings-settings

Adding a test for mappings and settings overrides

7189679

jonathan-buttner commented Jul 6, 2020

View reviewed changes

x-pack/test/ingest_manager_api_integration/apis/file.ts Outdated Show resolved Hide resolved

jonathan-buttner commented Jul 6, 2020

View reviewed changes

x-pack/test/ingest_manager_api_integration/apis/fixtures/package_registry_config.yml Outdated Show resolved Hide resolved

jonathan-buttner commented Jul 6, 2020

View reviewed changes

x-pack/test/ingest_manager_api_integration/config.ts Outdated Show resolved Hide resolved

jonathan-buttner commented Jul 6, 2020

View reviewed changes

jonathan-buttner added 2 commits July 6, 2020 12:51

Wrap all the tests in the docker check

d6d9ed2

Merge branch 'master' of github.com:elastic/kibana into ingest-suppor…

fd973f1

…t-mappings-settings

jonathan-buttner merged commit 321fb87 into elastic:master Jul 6, 2020

jonathan-buttner deleted the ingest-support-mappings-settings branch July 6, 2020 19:09

jonathan-buttner mentioned this pull request Jul 6, 2020

[7.x] [EPM][Security Solution] Implementing dataset component templates (#70517) #70862

Merged

neptunian mentioned this pull request Jul 22, 2020

[Fleet] Add installed template components to installation object and remove them #72959

Closed

ph mentioned this pull request Oct 16, 2020

[EPM] Use component templates for generated mappings #64760

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EPM][Security Solution] Implementing dataset component templates #70517

[EPM][Security Solution] Implementing dataset component templates #70517

jonathan-buttner commented Jul 1, 2020

elasticmachine commented Jul 1, 2020

elasticmachine commented Jul 1, 2020

elasticmachine commented Jul 1, 2020

jfsiii left a comment

jfsiii Jul 2, 2020

jonathan-buttner Jul 2, 2020

jfsiii Jul 2, 2020

ruflin commented Jul 2, 2020

jonathan-buttner commented Jul 2, 2020

jonathan-buttner commented Jul 2, 2020

jonathan-buttner Jul 3, 2020

ruflin commented Jul 3, 2020

jonathan-buttner Jul 6, 2020

kibanamachine commented Jul 6, 2020

[EPM][Security Solution] Implementing dataset component templates #70517

[EPM][Security Solution] Implementing dataset component templates #70517

Conversation

jonathan-buttner commented Jul 1, 2020

elasticmachine commented Jul 1, 2020

elasticmachine commented Jul 1, 2020

elasticmachine commented Jul 1, 2020

jfsiii left a comment

Choose a reason for hiding this comment

jfsiii Jul 2, 2020

Choose a reason for hiding this comment

jonathan-buttner Jul 2, 2020

Choose a reason for hiding this comment

jfsiii Jul 2, 2020

Choose a reason for hiding this comment

ruflin commented Jul 2, 2020

jonathan-buttner commented Jul 2, 2020

jonathan-buttner commented Jul 2, 2020

jonathan-buttner Jul 3, 2020

Choose a reason for hiding this comment

ruflin commented Jul 3, 2020

jonathan-buttner Jul 6, 2020

Choose a reason for hiding this comment

kibanamachine commented Jul 6, 2020

💚 Build Succeeded

Build metrics

History