Debug log the latest saved object migrations per type #89722

rudolf · 2021-01-29T14:27:06Z

Summary

Adds the following debug logging to make it easier to see which saved object types will be migrated and what the expected migrationVersion should be after the migration was applied:

server    log   [15:10:10.466] [info][savedobjects-service] Starting saved objects migrations
server    log   [15:10:10.479] [debug][savedobjects-service] applying registered migrations for the following saved object types:
server    log   [15:10:10.480] [debug][savedobjects-service] migrationVersion: 6.6.0 saved object type: space
server    log   [15:10:10.481] [debug][savedobjects-service] migrationVersion: 7.0.0 saved object type: canvas-workpad
server    log   [15:10:10.481] [debug][savedobjects-service] migrationVersion: 7.6.0 saved object type: task
server    log   [15:10:10.481] [debug][savedobjects-service] migrationVersion: 7.9.0 saved object type: config
server    log   [15:10:10.481] [debug][savedobjects-service] migrationVersion: 7.9.0 saved object type: infrastructure-ui-source
server    log   [15:10:10.481] [debug][savedobjects-service] migrationVersion: 7.9.3 saved object type: search
server    log   [15:10:10.481] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: map
server    log   [15:10:10.482] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: exception-list-agnostic
server    log   [15:10:10.482] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: exception-list
server    log   [15:10:10.482] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: ingest_manager_settings
server    log   [15:10:10.482] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: fleet-agent-actions
server    log   [15:10:10.482] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: fleet-agent-events
server    log   [15:10:10.482] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: ingest-agent-policies
server    log   [15:10:10.482] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: fleet-enrollment-api-keys
server    log   [15:10:10.482] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: ml-job
server    log   [15:10:10.482] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: cases-configure
server    log   [15:10:10.483] [debug][savedobjects-service] migrationVersion: 7.10.0 saved object type: cases-user-actions
server    log   [15:10:10.483] [debug][savedobjects-service] migrationVersion: 7.11.0 saved object type: index-pattern
server    log   [15:10:10.483] [debug][savedobjects-service] migrationVersion: 7.11.0 saved object type: graph-workspace
server    log   [15:10:10.483] [debug][savedobjects-service] migrationVersion: 7.11.0 saved object type: dashboard
server    log   [15:10:10.483] [debug][savedobjects-service] migrationVersion: 7.11.0 saved object type: lens
server    log   [15:10:10.483] [debug][savedobjects-service] migrationVersion: 7.11.0 saved object type: action
server    log   [15:10:10.483] [debug][savedobjects-service] migrationVersion: 7.11.0 saved object type: alert
server    log   [15:10:10.483] [debug][savedobjects-service] migrationVersion: 7.11.0 saved object type: cases-comments
server    log   [15:10:10.483] [debug][savedobjects-service] migrationVersion: 7.11.0 saved object type: cases
server    log   [15:10:10.484] [debug][savedobjects-service] migrationVersion: 7.12.0 saved object type: visualization
server    log   [15:10:10.484] [debug][savedobjects-service] migrationVersion: 7.12.0 saved object type: fleet-agents
server    log   [15:10:10.484] [debug][savedobjects-service] migrationVersion: 7.12.0 saved object type: ingest-package-policies

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2021-01-29T14:27:08Z

Pinging @elastic/kibana-core (Team:Core)

afharo

LGTM! Just one question/NIT I'd like to be clear about.

afharo · 2021-01-29T17:55:51Z

src/core/server/saved_objects/migrations/kibana/kibana_migrator.ts

+    this.log.debug('Applying registered migrations for the following saved object types:');
+    Object.entries(this.documentMigrator.migrationVersion)
+      .sort(([t1, v1], [t2, v2]) => {
+        return Semver.compare(v1, v2);


NIT: are v1 and v2 ensured to always exist? If any don't, can this logic break the rest of the migration logic?

The code is hard to follow, but from what I understand, I think that this.documentMigrator.migrationVersion will always have a version, but that types without any migration registered will not be present in the map. Which sounds acceptable @rudolf?

kibana/src/core/server/saved_objects/migrations/core/document_migrator.ts

Lines 388 to 390 in 8534faf

if (!transforms.length) {

return migrations;

}

kibana/src/core/server/saved_objects/migrations/core/document_migrator.ts

Lines 178 to 189 in 8534faf

public get migrationVersion(): SavedObjectsMigrationVersion {

if (!this.migrations) {

throw new Error('Migrations are not ready. Make sure prepareMigrations is called first.');

}

return Object.entries(this.migrations).reduce((acc, [prop, { latestMigrationVersion }]) => {

// some migration objects won't have a latestMigrationVersion (they only contain reference transforms that are applied from other types)

if (latestMigrationVersion) {

return { ...acc, [prop]: latestMigrationVersion };

}

return acc;

}, {});

yeah, these versions come from plugins registering migrations, so it will always be present (and I think we validate that it's a valid semver). If a type doesn't register a migration it won't be in this map.

kibanamachine · 2021-02-01T19:40:11Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d5274e1

Metrics [docs]

✅ unchanged

History

💔 Build #102140 failed eeb461e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2021-02-04T09:44:37Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 89722 or prevent reminders by adding the backport:skip label.

* Debug log the latest saved object migrations per type * Fix broken test

Debug log the latest saved object migrations per type

eeb461e

rudolf requested a review from a team as a code owner January 29, 2021 14:27

afharo approved these changes Jan 29, 2021

View reviewed changes

rudolf added 2 commits February 1, 2021 18:47

Merge branch 'master' into debug-log-registered-migrations

94e06ce

Fix broken test

d5274e1

rudolf merged commit 7fbec26 into elastic:master Feb 2, 2021

rudolf deleted the debug-log-registered-migrations branch February 2, 2021 09:42

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 4, 2021

rudolf mentioned this pull request Feb 4, 2021

[7.x] Debug log the latest saved object migrations per type (#89722) #90266

Merged

rudolf added a commit to rudolf/kibana that referenced this pull request Feb 4, 2021

Debug log the latest saved object migrations per type (elastic#89722)

e1d6a54

* Debug log the latest saved object migrations per type * Fix broken test

rudolf added a commit that referenced this pull request Feb 4, 2021

Debug log the latest saved object migrations per type (#89722) (#90266)

f5bd166

* Debug log the latest saved object migrations per type * Fix broken test

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Debug log the latest saved object migrations per type #89722

Debug log the latest saved object migrations per type #89722

rudolf commented Jan 29, 2021 •

edited

Loading

elasticmachine commented Jan 29, 2021

afharo left a comment

afharo Jan 29, 2021

pgayvallet Feb 1, 2021

rudolf Feb 1, 2021

kibanamachine commented Feb 1, 2021

kibanamachine commented Feb 4, 2021

	public get migrationVersion(): SavedObjectsMigrationVersion {
	if (!this.migrations) {
	throw new Error('Migrations are not ready. Make sure prepareMigrations is called first.');
	}

	return Object.entries(this.migrations).reduce((acc, [prop, { latestMigrationVersion }]) => {
	// some migration objects won't have a latestMigrationVersion (they only contain reference transforms that are applied from other types)
	if (latestMigrationVersion) {
	return { ...acc, [prop]: latestMigrationVersion };
	}
	return acc;
	}, {});

Debug log the latest saved object migrations per type #89722

Debug log the latest saved object migrations per type #89722

Conversation

rudolf commented Jan 29, 2021 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Jan 29, 2021

afharo left a comment

Choose a reason for hiding this comment

afharo Jan 29, 2021

Choose a reason for hiding this comment

pgayvallet Feb 1, 2021

Choose a reason for hiding this comment

rudolf Feb 1, 2021

Choose a reason for hiding this comment

kibanamachine commented Feb 1, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Feb 4, 2021

rudolf commented Jan 29, 2021 •

edited

Loading