Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Workspace]Import sample data to current workspace #6105

Merged
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
4321cb4
Import sample data to workspace
wanglam Mar 11, 2024
d29f691
Enable workspace ui plugin
wanglam Mar 11, 2024
54adb59
Add changelog for import sample data to current workspace
wanglam Mar 11, 2024
5240160
Merge remote-tracking branch 'origin/main' into feat-import-sample-da…
wanglam Mar 22, 2024
ee39682
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam Apr 10, 2024
b1da07e
Merge branch 'main' into feat-import-sample-data-to-workspace
SuZhou-Joe Apr 18, 2024
b4c0c27
feat: register sample data as standalone app (#8)
SuZhou-Joe May 6, 2024
0d44b0f
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 6, 2024
e4a6c74
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 7, 2024
9b3be8f
Retrieve workspace id from request
wanglam May 7, 2024
5098c09
Remove workspace id in query
wanglam May 7, 2024
58adba2
Move changelog to fragments
wanglam May 8, 2024
0e7c5e1
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 8, 2024
f11844f
Fix sample data list unit tests
wanglam May 8, 2024
efc5b1c
Remove no need workspaces deps
wanglam May 8, 2024
8b67b94
Remove manual created changelogs
wanglam May 8, 2024
e87cb72
Changeset file for PR #6105 created/updated
opensearch-changeset-bot[bot] May 8, 2024
799f475
Enable sample data in workspace overview page (#9)
Hailong-am May 9, 2024
fb4c3a2
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 9, 2024
5767ffc
Add unit tests for getFinalSavedObjects in data sets util file
wanglam May 9, 2024
eda80ff
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 13, 2024
dd998f3
Add unit tests for renderImportSampleDataApp destroy
wanglam May 13, 2024
7d7a831
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 14, 2024
a7e8c72
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 15, 2024
9769622
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 16, 2024
28d1f74
Address PR comments
wanglam May 17, 2024
6162c2d
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 17, 2024
b82fcec
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 18, 2024
5f8f9f4
Remove history listen in renderImportSampleDataApp
wanglam May 21, 2024
3f27d71
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 21, 2024
4d560c6
Remove Route for workspace import sample data entry point
wanglam May 22, 2024
9a2bd3e
Merge branch 'main' into feat-import-sample-data-to-workspace
wanglam May 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- [Workspace] Consume workspace id in saved object client ([#6014](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6014))

- [Workspace] Add delete saved objects by workspace functionality([#6013](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6013))
- [Workspace] Import sample data to current workspace([#6105](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6105))

### 🐛 Bug Fixes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,10 @@ export class SampleDataSetCards extends React.Component {
loadSampleDataSets = async (dataSourceId) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems we are adding workspaceId into the parameters of listSampleDataSets/install/uninstall but not in loadSampleDataSets, any reason for such difference?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The loadSampleDataSets function call listSampleDataSets below. It will read the current workspace id from the workspace service directly. So we don't add the parameter for workspace id.

let sampleDataSets;
try {
sampleDataSets = await listSampleDataSets(dataSourceId);
sampleDataSets = await listSampleDataSets(
dataSourceId,
getServices().workspaces.currentWorkspaceId$.getValue()
);
} catch (fetchError) {
this.toastNotifications.addDanger({
title: i18n.translate('home.sampleDataSet.unableToLoadListErrorMessage', {
Expand Down Expand Up @@ -114,7 +117,12 @@ export class SampleDataSetCards extends React.Component {
}));

try {
await installSampleDataSet(id, targetSampleDataSet.defaultIndex, dataSourceId);
await installSampleDataSet(
id,
targetSampleDataSet.defaultIndex,
dataSourceId,
getServices().workspaces.currentWorkspaceId$.getValue()
);
} catch (fetchError) {
if (this._isMounted) {
this.setState((prevState) => ({
Expand Down Expand Up @@ -162,7 +170,12 @@ export class SampleDataSetCards extends React.Component {
}));

try {
await uninstallSampleDataSet(id, targetSampleDataSet.defaultIndex, dataSourceId);
await uninstallSampleDataSet(
id,
targetSampleDataSet.defaultIndex,
dataSourceId,
getServices().workspaces.currentWorkspaceId$.getValue()
);
} catch (fetchError) {
if (this._isMounted) {
this.setState((prevState) => ({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ import {
SavedObjectsClientContract,
IUiSettingsClient,
ApplicationStart,
WorkspaceStart,
} from 'opensearch-dashboards/public';
import { UiStatsMetricType } from '@osd/analytics';
import { TelemetryPluginStart } from '../../../telemetry/public';
Expand Down Expand Up @@ -73,6 +74,7 @@ export interface HomeOpenSearchDashboardsServices {
getBranding: () => HomePluginBranding;
};
dataSource?: DataSourcePluginStart;
workspaces: WorkspaceStart;
}

let services: HomeOpenSearchDashboardsServices | null = null;
Expand Down
23 changes: 16 additions & 7 deletions src/plugins/home/public/application/sample_data_client.js
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,13 @@ function clearIndexPatternsCache() {
getServices().indexPatternService.clearCache();
}

export async function listSampleDataSets(dataSourceId) {
const query = buildQuery(dataSourceId);
export async function listSampleDataSets(dataSourceId, workspaceId) {
const query = buildQuery(dataSourceId, workspaceId);
return await getServices().http.get(sampleDataUrl, { query });
}

export async function installSampleDataSet(id, sampleDataDefaultIndex, dataSourceId) {
const query = buildQuery(dataSourceId);
export async function installSampleDataSet(id, sampleDataDefaultIndex, dataSourceId, workspaceId) {
const query = buildQuery(dataSourceId, workspaceId);
await getServices().http.post(`${sampleDataUrl}/${id}`, { query });

if (getServices().uiSettings.isDefault('defaultIndex')) {
Expand All @@ -52,8 +52,13 @@ export async function installSampleDataSet(id, sampleDataDefaultIndex, dataSourc
clearIndexPatternsCache();
}

export async function uninstallSampleDataSet(id, sampleDataDefaultIndex, dataSourceId) {
const query = buildQuery(dataSourceId);
export async function uninstallSampleDataSet(
id,
sampleDataDefaultIndex,
dataSourceId,
workspaceId
) {
const query = buildQuery(dataSourceId, workspaceId);
await getServices().http.delete(`${sampleDataUrl}/${id}`, { query });

const uiSettings = getServices().uiSettings;
Expand All @@ -68,12 +73,16 @@ export async function uninstallSampleDataSet(id, sampleDataDefaultIndex, dataSou
clearIndexPatternsCache();
}

function buildQuery(dataSourceId) {
function buildQuery(dataSourceId, workspaceId) {
const query = {};

if (dataSourceId) {
query.data_source_id = dataSourceId;
}

if (workspaceId) {
query.workspace_id = workspaceId;
}

return query;
}
1 change: 1 addition & 0 deletions src/plugins/home/public/plugin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ export class HomePublicPlugin
featureCatalogue: this.featuresCatalogueRegistry,
injectedMetadata: coreStart.injectedMetadata,
dataSource,
workspaces: coreStart.workspaces,
});
coreStart.chrome.docTitle.change(
i18n.translate('home.pageTitle', { defaultMessage: 'Home' })
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ import { i18n } from '@osd/i18n';
import { getSavedObjects } from './saved_objects';
import { fieldMappings } from './field_mappings';
import { SampleDatasetSchema, AppLinkSchema } from '../../lib/sample_dataset_registry_types';
import { getSavedObjectsWithDataSource, appendDataSourceId } from '../util';
import { addPrefixTo } from '../util';

const ecommerceName = i18n.translate('home.sampleData.ecommerceSpecTitle', {
defaultMessage: 'Sample eCommerce orders',
Expand All @@ -55,13 +55,11 @@ export const ecommerceSpecProvider = function (): SampleDatasetSchema {
darkPreviewImagePath: '/plugins/home/assets/sample_data_resources/ecommerce/dashboard_dark.png',
hasNewThemeImages: true,
overviewDashboard: DASHBOARD_ID,
getDataSourceIntegratedDashboard: appendDataSourceId(DASHBOARD_ID),
getDashboardWithPrefix: addPrefixTo(DASHBOARD_ID),
appLinks: initialAppLinks,
defaultIndex: DEFAULT_INDEX,
getDataSourceIntegratedDefaultIndex: appendDataSourceId(DEFAULT_INDEX),
getDataSourceIntegratedDefaultIndex: addPrefixTo(DEFAULT_INDEX),
savedObjects: getSavedObjects(),
getDataSourceIntegratedSavedObjects: (dataSourceId?: string, dataSourceTitle?: string) =>
getSavedObjectsWithDataSource(getSavedObjects(), dataSourceId, dataSourceTitle),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it save to delete this field?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method has been moved to src/plugins/home/server/services/sample_data/data_sets/util.ts, searching all OSD, no other places use it.

dataIndices: [
{
id: 'ecommerce',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ import { i18n } from '@osd/i18n';
import { getSavedObjects } from './saved_objects';
import { fieldMappings } from './field_mappings';
import { SampleDatasetSchema, AppLinkSchema } from '../../lib/sample_dataset_registry_types';
import { getSavedObjectsWithDataSource, appendDataSourceId } from '../util';
import { addPrefixTo } from '../util';

const flightsName = i18n.translate('home.sampleData.flightsSpecTitle', {
defaultMessage: 'Sample flight data',
Expand All @@ -55,13 +55,11 @@ export const flightsSpecProvider = function (): SampleDatasetSchema {
darkPreviewImagePath: '/plugins/home/assets/sample_data_resources/flights/dashboard_dark.png',
hasNewThemeImages: true,
overviewDashboard: DASHBOARD_ID,
getDataSourceIntegratedDashboard: appendDataSourceId(DASHBOARD_ID),
getDashboardWithPrefix: addPrefixTo(DASHBOARD_ID),
appLinks: initialAppLinks,
defaultIndex: DEFAULT_INDEX,
getDataSourceIntegratedDefaultIndex: appendDataSourceId(DEFAULT_INDEX),
getDataSourceIntegratedDefaultIndex: addPrefixTo(DEFAULT_INDEX),
savedObjects: getSavedObjects(),
getDataSourceIntegratedSavedObjects: (dataSourceId?: string, dataSourceTitle?: string) =>
SuZhou-Joe marked this conversation as resolved.
Show resolved Hide resolved
getSavedObjectsWithDataSource(getSavedObjects(), dataSourceId, dataSourceTitle),
dataIndices: [
{
id: 'flights',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ import { i18n } from '@osd/i18n';
import { getSavedObjects } from './saved_objects';
import { fieldMappings } from './field_mappings';
import { SampleDatasetSchema, AppLinkSchema } from '../../lib/sample_dataset_registry_types';
import { appendDataSourceId, getSavedObjectsWithDataSource } from '../util';
import { addPrefixTo } from '../util';

const logsName = i18n.translate('home.sampleData.logsSpecTitle', {
defaultMessage: 'Sample web logs',
Expand All @@ -55,13 +55,11 @@ export const logsSpecProvider = function (): SampleDatasetSchema {
darkPreviewImagePath: '/plugins/home/assets/sample_data_resources/logs/dashboard_dark.png',
hasNewThemeImages: true,
overviewDashboard: DASHBOARD_ID,
getDataSourceIntegratedDashboard: appendDataSourceId(DASHBOARD_ID),
getDashboardWithPrefix: addPrefixTo(DASHBOARD_ID),
appLinks: initialAppLinks,
defaultIndex: DEFAULT_INDEX,
getDataSourceIntegratedDefaultIndex: appendDataSourceId(DEFAULT_INDEX),
getDataSourceIntegratedDefaultIndex: addPrefixTo(DEFAULT_INDEX),
savedObjects: getSavedObjects(),
getDataSourceIntegratedSavedObjects: (dataSourceId?: string, dataSourceTitle?: string) =>
SuZhou-Joe marked this conversation as resolved.
Show resolved Hide resolved
getSavedObjectsWithDataSource(getSavedObjects(), dataSourceId, dataSourceTitle),
dataIndices: [
{
id: 'logs',
Expand Down
125 changes: 76 additions & 49 deletions src/plugins/home/server/services/sample_data/data_sets/util.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,59 +5,74 @@

import { SavedObject } from 'opensearch-dashboards/server';

export const appendDataSourceId = (id: string) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems this logic has gone?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. This function will be refactor to addPrefixTo in below.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any updates on this comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The appendDataSourceId has been added back. To support workspace, the workspaceId has been added to the appendDataSourceId result function. It will add workspace id prefix at the return value of getDataSourceIntegratedDashboard and getDataSourceIntegratedDefaultIndex functions.

return (dataSourceId?: string) => (dataSourceId ? `${dataSourceId}_` + id : id);
const cloneDeep = <T extends unknown>(payload: T): T => JSON.parse(JSON.stringify(payload));

const withPrefix = (...args: Array<string | undefined>) => (id: string) => {
const prefix = args.filter(Boolean).join('_');
if (prefix) {
return `${prefix}_${id}`;
}
return id;
};

export const getSavedObjectsWithDataSource = (
saveObjectList: SavedObject[],
dataSourceId?: string,
dataSourceTitle?: string
): SavedObject[] => {
if (dataSourceId) {
return saveObjectList.map((saveObject) => {
saveObject.id = `${dataSourceId}_` + saveObject.id;
// update reference
if (saveObject.type === 'dashboard') {
saveObject.references.map((reference) => {
if (reference.id) {
reference.id = `${dataSourceId}_` + reference.id;
}
});
export const addPrefixTo = (id: string) => (...args: Array<string | undefined>) => {
return withPrefix(...args)(id);
};

const overrideSavedObjectId = (savedObject: SavedObject, idGenerator: (id: string) => string) => {
savedObject.id = idGenerator(savedObject.id);
// update reference
if (savedObject.type === 'dashboard') {
savedObject.references.map((reference) => {
if (reference.id) {
reference.id = idGenerator(reference.id);
}
});
}

// update reference
if (saveObject.type === 'visualization' || saveObject.type === 'search') {
const searchSourceString = saveObject.attributes?.kibanaSavedObjectMeta?.searchSourceJSON;
const visStateString = saveObject.attributes?.visState;
// update reference
if (savedObject.type === 'visualization' || savedObject.type === 'search') {
const searchSourceString = savedObject.attributes?.kibanaSavedObjectMeta?.searchSourceJSON;
const visStateString = savedObject.attributes?.visState;

if (searchSourceString) {
const searchSource = JSON.parse(searchSourceString);
if (searchSource.index) {
searchSource.index = `${dataSourceId}_` + searchSource.index;
saveObject.attributes.kibanaSavedObjectMeta.searchSourceJSON = JSON.stringify(
searchSource
);
}
}
if (searchSourceString) {
const searchSource = JSON.parse(searchSourceString);
if (searchSource.index) {
searchSource.index = idGenerator(searchSource.index);
savedObject.attributes.kibanaSavedObjectMeta.searchSourceJSON = JSON.stringify(
searchSource
);
}
}

if (visStateString) {
const visState = JSON.parse(visStateString);
const controlList = visState.params?.controls;
if (controlList) {
controlList.map((control) => {
if (control.indexPattern) {
control.indexPattern = `${dataSourceId}_` + control.indexPattern;
}
});
if (visStateString) {
const visState = JSON.parse(visStateString);
const controlList = visState.params?.controls;
if (controlList) {
controlList.map((control) => {
if (control.indexPattern) {
control.indexPattern = idGenerator(control.indexPattern);
}
saveObject.attributes.visState = JSON.stringify(visState);
}
});
}
savedObject.attributes.visState = JSON.stringify(visState);
}
}
};

export const getDataSourceIntegratedSavedObjects = (
savedObjectList: SavedObject[],
dataSourceId?: string,
dataSourceTitle?: string
): SavedObject[] => {
savedObjectList = cloneDeep(savedObjectList);
if (dataSourceId) {
return savedObjectList.map((savedObject) => {
overrideSavedObjectId(savedObject, withPrefix(dataSourceId));

// update reference
if (saveObject.type === 'index-pattern') {
Copy link
Collaborator

@BionIT BionIT May 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why removing the data source reference for index pattern?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wanglam Is it expected? If not, can we fix it? #6850

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's removed by accident. Will add them back and add some unit tests.

saveObject.references = [
if (savedObject.type === 'index-pattern') {
savedObject.references = [
{
id: `${dataSourceId}`,
type: 'data-source',
Expand All @@ -68,17 +83,29 @@

if (dataSourceTitle) {
if (
saveObject.type === 'dashboard' ||
saveObject.type === 'visualization' ||
saveObject.type === 'search'
savedObject.type === 'dashboard' ||
savedObject.type === 'visualization' ||
savedObject.type === 'search'
) {
saveObject.attributes.title = saveObject.attributes.title + `_${dataSourceTitle}`;
savedObject.attributes.title = savedObject.attributes.title + `_${dataSourceTitle}`;

Check warning on line 90 in src/plugins/home/server/services/sample_data/data_sets/util.ts

View check run for this annotation

Codecov / codecov/patch

src/plugins/home/server/services/sample_data/data_sets/util.ts#L90

Added line #L90 was not covered by tests
}
}

return saveObject;
return savedObject;
});
}

return saveObjectList;
return savedObjectList;

Check warning on line 98 in src/plugins/home/server/services/sample_data/data_sets/util.ts

View check run for this annotation

Codecov / codecov/patch

src/plugins/home/server/services/sample_data/data_sets/util.ts#L98

Added line #L98 was not covered by tests
};

export const getWorkspaceIntegratedSavedObjects = (
savedObjectList: SavedObject[],
workspaceId?: string
) => {
const savedObjectListCopy = cloneDeep(savedObjectList);

savedObjectListCopy.forEach((savedObject) => {
overrideSavedObjectId(savedObject, withPrefix(workspaceId));
});
return savedObjectListCopy;
};
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ export interface SampleDatasetSchema<T = unknown> {

// saved object id of main dashboard for sample data set
overviewDashboard: string;
getDataSourceIntegratedDashboard: (dataSourceId?: string) => string;
getDashboardWithPrefix: (...args: Array<string | undefined>) => string;
appLinks: AppLinkSchema[];

// saved object id of default index-pattern for sample data set
Expand All @@ -99,10 +99,6 @@ export interface SampleDatasetSchema<T = unknown> {
// OpenSearch Dashboards saved objects (index patter, visualizations, dashboard, ...)
// Should provide a nice demo of OpenSearch Dashboards's functionality with the sample data set
savedObjects: Array<SavedObject<T>>;
getDataSourceIntegratedSavedObjects: (
dataSourceId?: string,
dataSourceTitle?: string
) => Array<SavedObject<T>>;
dataIndices: DataIndexSchema[];
status?: string | undefined;
statusMsg?: unknown;
Expand Down
Loading
Loading