Add datastream fields to all datasets #213

ruflin · 2020-08-03T13:25:19Z

It is planned to move from dataset.* to datastream.. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream. fields. Because of this, these should also be added to the mappings of the packages.

elasticmachine · 2020-08-03T13:28:32Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #213 updated]
Start Time: 2020-08-04T11:31:19.078+0000
Duration: 3 min 1 sec

ruflin · 2020-08-03T14:54:06Z

Change for the agent can be found here: elastic/beats#20402

ruflin · 2020-08-03T15:12:41Z

@mtojek I will probably need your help to get this over the line. After we get this in, what is the best way to release a new version of the packages to snapshot and then promote it through?

mtojek · 2020-08-04T07:35:32Z

what is the best way to release a new version of the packages to snapshot and then promote it through?

currently it's the command mage UpdatePackageStorage and later on juggling packages through branches of the package-storage

mtojek

Do we have any plans for Kibana dashboards?

If so, there are few places in https://github.com/elastic/integrations/blob/master/dev/import-beats/kibana.go to replace dataset.name.

BTW Does it mean we don't have anymore datasets and just datastreams? Should we update all places in integrations? code, READMEs. Code changes can be postponed, but what about READMEs?

packages/apache/dataset/access/fields/base-fields.yml

mtojek · 2020-08-04T07:42:58Z

I had plan to promote packages today. I can push changes in this PR too.

ruflin · 2020-08-04T08:19:09Z

@mtojek At the moment the Agent ships both fields so we need both mappings. But the plan is to follow up on removing dataset.* completely. The first step is getting rid of it in queries (dashboards), second step adjust the package spec but we plan to do this later on in small steps.

How tricky do you think it's going to be to adjust all dashboards and README's?

ruflin · 2020-08-04T08:23:33Z

@mtojek I'll do a follow up PR to adjust all the scripts if that is ok for you.

ruflin · 2020-08-04T08:35:00Z

@mtojek Here is the script change PR: #216

Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213

mtojek · 2020-08-04T08:56:55Z

dev/import-beats/fields_base_fields.go

+			Type:        "constant_keyword",
+			Description: "Datastream namespace.",
+		},
+		// TODO: This should be removed as soon as it is not a requirement anymore by the validation


Hmm.. shouldn't be validation in the package-registry already adjusted?

If we adjust it now, all old packages are invalid.

PR for validation: elastic/package-registry#618

once this one is pushed, you can update the reference to the commit

You can update the reference in docs.

mtojek · 2020-08-04T08:58:55Z

You need to bump up versions in all manifest.yml files.

ruflin · 2020-08-04T09:39:21Z

I cherry-picked the changes from #216 into this one. I initially expected the script changes to be larger and the version bumps went into the wrong branch. Please have a look again.

mtojek · 2020-08-04T10:55:07Z

packages/mongodb/dataset/replstatus/fields/base-fields.yml

+  description: Datastream type.
+- name: datastream.dataset
+  type: constant_keyword
+  description: Datastream dataset name.


nit: Datastream dataset

adjusted in all packages.

mtojek · 2020-08-04T10:58:51Z

Side note:

we have some packages in the package storage: barracuda and endpoint, and there is bunch of new waiting for merge: elastic/package-storage#181

any plan for these ones?

ruflin · 2020-08-04T11:01:03Z

My thinking for the pending packages in snapshot was:

Update registry first for production, so validation passes
Later updated it for snapshot and with it also all the packages

mtojek · 2020-08-04T11:06:52Z

OK, I'll push first all integrations in the package-storage to the production and leave the ones that we (integrations) do not own.

go.mod

It is planned to move from dataset.* to datastream.*. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream.* fields. Because of this, these should also be added to the mappings of the packages.

The agent will ship datastream.* fields. Because of this, the import scripts can be adjusted to only use these fields. This PR doese not rename any Golang variables. This should be done in a follow up and in sync with potential changes to the registry.

mtojek

LGTM

Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213

…#74342) * [Ingest Manager] Adjust dataset aggs to use datastream fields instead Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213 * Update datastream to data_stream * Update data stream name generation * Fix typo * Temporarily use datastream instead of data_stream * updating to use `data_stream` instead of `datastream` Co-authored-by: ruflin <spam@ruflin.com> Co-authored-by: Jen Huang <its.jenetic@gmail.com>

…elastic#74342) * [Ingest Manager] Adjust dataset aggs to use datastream fields instead Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213 * Update datastream to data_stream * Update data stream name generation * Fix typo * Temporarily use datastream instead of data_stream * updating to use `data_stream` instead of `datastream` Co-authored-by: ruflin <spam@ruflin.com> Co-authored-by: Jen Huang <its.jenetic@gmail.com>

…#74342) (#74355) * [Ingest Manager] Adjust dataset aggs to use datastream fields instead Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213 * Update datastream to data_stream * Update data stream name generation * Fix typo * Temporarily use datastream instead of data_stream * updating to use `data_stream` instead of `datastream` Co-authored-by: ruflin <spam@ruflin.com> Co-authored-by: Jen Huang <its.jenetic@gmail.com> Co-authored-by: ruflin <spam@ruflin.com> Co-authored-by: Jen Huang <its.jenetic@gmail.com>

…#74342) (#74356) Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213

It is planned to move from dataset.* to datastream.*. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream.* fields. Because of this, these should also be added to the mappings of the packages. The agent will ship datastream.* fields. Because of this, the import scripts can be adjusted to only use these fields. This PR doese not rename any Golang variables. This should be done in a follow up and in sync with potential changes to the registry.

ruflin self-assigned this Aug 3, 2020

ruflin force-pushed the add-datastream-fields branch from 919ca47 to 9107ee6 Compare August 3, 2020 13:59

michalpristas mentioned this pull request Aug 3, 2020

[Ingest Manager] Send datastreams fields elastic/beats#20402

Merged

6 tasks

ruflin requested a review from mtojek August 3, 2020 14:52

ruflin marked this pull request as ready for review August 3, 2020 14:57

michalpristas mentioned this pull request Aug 4, 2020

[Ingest Manager] Send datastreams fields elastic/beats#20416

Merged

6 tasks

mtojek reviewed Aug 4, 2020

View reviewed changes

packages/apache/dataset/access/fields/base-fields.yml Show resolved Hide resolved

ruflin mentioned this pull request Aug 4, 2020

[Ingest Manager] Adjust dataset aggs to use datastream fields instead elastic/kibana#74226

Closed

mtojek reviewed Aug 4, 2020

View reviewed changes

ruflin mentioned this pull request Aug 4, 2020

Update scripts to used datastream.dataset for dashboards #216

Closed

michalpristas mentioned this pull request Aug 4, 2020

[Ingest Manager] Move to datastream instead of dataset elastic/beats#20420

Merged

6 tasks

mtojek reviewed Aug 4, 2020

View reviewed changes

ruflin commented Aug 4, 2020

View reviewed changes

go.mod Outdated Show resolved Hide resolved

ruflin added 3 commits August 4, 2020 13:17

update readme files

aac1001

adjust field generation

67fc2de

ruflin added 5 commits August 4, 2020 13:17

bump version of all packages

5449221

update description

5142ac4

update many more packages

8c3d420

update package import beats script

9f6e769

ruflin force-pushed the add-datastream-fields branch from d5c82fc to 9f6e769 Compare August 4, 2020 11:20

update readme files

aafc5ab

mtojek self-requested a review August 4, 2020 11:41

mtojek approved these changes Aug 4, 2020

View reviewed changes

mtojek mentioned this pull request Aug 4, 2020

Add 16 new experimental packages elastic/package-storage#181

Merged

2 tasks

ruflin merged commit da4873a into elastic:master Aug 4, 2020

ruflin deleted the add-datastream-fields branch August 4, 2020 11:57

michalpristas mentioned this pull request Aug 5, 2020

Cherry-pick #20402 to 7.x: Send datastreams fields elastic/beats#20442

Merged

6 tasks

skh mentioned this pull request Aug 5, 2020

[Ingest Manager] Adjust dataset aggs to use datastream fields instead elastic/kibana#74342

Merged

This was referenced Aug 6, 2020

Cherry-pick #20420 to 7.x: Move to datastream instead of dataset elastic/beats#20481

Merged

Cherry-pick #20420 to 7.9: Move to datastream instead of dataset elastic/beats#20482

Merged

jsoriano mentioned this pull request May 3, 2023

Test integrations with latest stack version on PRs #6072

Open

ali786XI mentioned this pull request Aug 24, 2023

Add metrics datastream for Azure Functions #7130

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add datastream fields to all datasets #213

Add datastream fields to all datasets #213

ruflin commented Aug 3, 2020

elasticmachine commented Aug 3, 2020 •

edited

Loading

Build stats

ruflin commented Aug 3, 2020 •

edited

Loading

ruflin commented Aug 3, 2020

mtojek commented Aug 4, 2020

mtojek left a comment

mtojek commented Aug 4, 2020

ruflin commented Aug 4, 2020

ruflin commented Aug 4, 2020

ruflin commented Aug 4, 2020

mtojek Aug 4, 2020

ruflin Aug 4, 2020

mtojek Aug 4, 2020

mtojek Aug 4, 2020

ruflin Aug 4, 2020

mtojek commented Aug 4, 2020

ruflin commented Aug 4, 2020

mtojek Aug 4, 2020

ruflin Aug 4, 2020

mtojek commented Aug 4, 2020

ruflin commented Aug 4, 2020

mtojek commented Aug 4, 2020 •

edited

Loading

mtojek left a comment

Add datastream fields to all datasets #213

Add datastream fields to all datasets #213

Conversation

ruflin commented Aug 3, 2020

elasticmachine commented Aug 3, 2020 • edited Loading

💚 Build Succeeded

Build stats

ruflin commented Aug 3, 2020 • edited Loading

ruflin commented Aug 3, 2020

mtojek commented Aug 4, 2020

mtojek left a comment

Choose a reason for hiding this comment

mtojek commented Aug 4, 2020

ruflin commented Aug 4, 2020

ruflin commented Aug 4, 2020

ruflin commented Aug 4, 2020

mtojek Aug 4, 2020

Choose a reason for hiding this comment

ruflin Aug 4, 2020

Choose a reason for hiding this comment

mtojek Aug 4, 2020

Choose a reason for hiding this comment

mtojek Aug 4, 2020

Choose a reason for hiding this comment

ruflin Aug 4, 2020

Choose a reason for hiding this comment

mtojek commented Aug 4, 2020

ruflin commented Aug 4, 2020

mtojek Aug 4, 2020

Choose a reason for hiding this comment

ruflin Aug 4, 2020

Choose a reason for hiding this comment

mtojek commented Aug 4, 2020

ruflin commented Aug 4, 2020

mtojek commented Aug 4, 2020 • edited Loading

mtojek left a comment

Choose a reason for hiding this comment

elasticmachine commented Aug 3, 2020 •

edited

Loading

ruflin commented Aug 3, 2020 •

edited

Loading

mtojek commented Aug 4, 2020 •

edited

Loading