Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: data streams #605

Merged
merged 43 commits into from
Feb 22, 2023
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
80d7fb6
feat: temp commit
SuZhou-Joe Feb 2, 2023
3b31eae
feat: update
SuZhou-Joe Feb 3, 2023
9723a78
feat: update
SuZhou-Joe Feb 6, 2023
23cc264
refractor: move index mapping to common components
SuZhou-Joe Feb 7, 2023
8e6b052
feat: enable create data stream
SuZhou-Joe Feb 7, 2023
55fb11c
feat: enable data stream detail
SuZhou-Joe Feb 7, 2023
57ecdbe
feat: some typo fix
SuZhou-Joe Feb 7, 2023
4c0ee82
feat: enable data stream management
SuZhou-Joe Feb 8, 2023
4a17353
feat: update
SuZhou-Joe Feb 10, 2023
52abd71
feat: update
SuZhou-Joe Feb 10, 2023
57b93ae
feat: enable unit test
SuZhou-Joe Feb 11, 2023
de0edd3
feat: add data streams cypress test
SuZhou-Joe Feb 11, 2023
1c525df
feat: update version
SuZhou-Joe Feb 11, 2023
ffd6a00
feat: update
SuZhou-Joe Feb 12, 2023
92f4b7b
feat: update
SuZhou-Joe Feb 12, 2023
968d1f8
feat: update
SuZhou-Joe Feb 12, 2023
7b9b70f
feat: update
SuZhou-Joe Feb 12, 2023
e3cdfe1
feat: update
SuZhou-Joe Feb 12, 2023
6e88099
feat: update
SuZhou-Joe Feb 12, 2023
c6e08cd
feat: update
SuZhou-Joe Feb 12, 2023
83c5264
feat: make cypress run
SuZhou-Joe Feb 13, 2023
ba3511e
feat: update snapshot
SuZhou-Joe Feb 13, 2023
091158c
feat: update
SuZhou-Joe Feb 14, 2023
68d53de
feat: update
SuZhou-Joe Feb 14, 2023
9480c74
feat: update
SuZhou-Joe Feb 14, 2023
db1b3f7
feat: optimize search interaction
SuZhou-Joe Feb 15, 2023
ecf7af0
feat: update
SuZhou-Joe Feb 15, 2023
14fa1e2
feat: add explanation
SuZhou-Joe Feb 15, 2023
8ff4b8d
feat: add time field
SuZhou-Joe Feb 15, 2023
56f67f9
feat: update
SuZhou-Joe Feb 16, 2023
d5172c5
feat: copy the properties
SuZhou-Joe Feb 16, 2023
829f09c
feat: update
SuZhou-Joe Feb 18, 2023
7c4aaf3
feat: update
SuZhou-Joe Feb 18, 2023
8878ba4
feat: update
SuZhou-Joe Feb 19, 2023
180068c
feat: update
SuZhou-Joe Feb 19, 2023
7b1c495
feat: update
SuZhou-Joe Feb 19, 2023
24b45d0
feat: update
SuZhou-Joe Feb 20, 2023
5813c68
feat: update
SuZhou-Joe Feb 20, 2023
17dce6c
feat: make unit test run
SuZhou-Joe Feb 20, 2023
f7c8cde
feat: make unit test run
SuZhou-Joe Feb 20, 2023
8c82c6e
feat: update
SuZhou-Joe Feb 20, 2023
8e289a7
feat: merge
SuZhou-Joe Feb 22, 2023
1be89f7
feat: merge
SuZhou-Joe Feb 22, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/cypress-workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ jobs:
with:
path: index-management
repository: opensearch-project/index-management
ref: 'main'
ref: '2.5'
- name: Run opensearch with plugin
run: |
cd index-management
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/unit-tests-workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ on:
branches:
- "*"
env:
OPENSEARCH_DASHBOARDS_VERSION: '2.x'
OPENSEARCH_DASHBOARDS_VERSION: '2.5'
jobs:
tests:
name: Run unit tests
Expand Down
91 changes: 91 additions & 0 deletions cypress/integration/data_streams.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
import { PLUGIN_NAME } from "../support/constants";

describe("Data stream", () => {
before(() => {
// Set welcome screen tracking to false
localStorage.setItem("home:welcome:show", "false");
cy.deleteTemplate("index-common-template");
cy.createIndexTemplate("index-common-template", {
index_patterns: ["ds-*"],
data_stream: {},
template: {
aliases: {
alias_for_common_1: {},
alias_for_common_2: {},
},
settings: {
number_of_shards: 2,
number_of_replicas: 1,
},
},
});
cy.request({
url: `${Cypress.env("opensearch")}/_data_stream/*`,
method: "DELETE",
failOnStatusCode: false,
});
});

beforeEach(() => {
// Visit ISM OSD
cy.visit(`${Cypress.env("opensearch_dashboards")}/app/${PLUGIN_NAME}#/data-streams`);

// Common text to wait for to confirm page loaded, give up to 60 seconds for initial load
cy.contains("Data streams", { timeout: 60000 });
});

describe("can create a data stream", () => {
it("successfully", () => {
cy.get('[data-test-subj="Create data streamButton"]').click();
cy.get('[data-test-subj="form-row-name"] [data-test-subj="comboBoxSearchInput"]').type(`ds-{enter}`);
cy.get("body").click();
cy.get('[data-test-subj="CreateDataStreamCreateButton"]').click();
cy.contains("ds- has been successfully created.");
});
});

describe("can be searched / sorted / paginated", () => {
it("successfully", () => {
cy.contains("ds-");
cy.contains("index-common-template");
});
});

describe("can delete a data stream", () => {
it("successfully", () => {
cy.get('[data-test-subj="moreAction"] button')
.click()
.get('[data-test-subj="deleteAction"]')
.should("be.disabled")
.get(`#_selection_column_ds--checkbox`)
.click()
.get('[data-test-subj="moreAction"] button')
.click()
.get('[data-test-subj="deleteAction"]')
.click();
// The confirm button should be disabled
cy.get('[data-test-subj="deleteConfirmButton"]').should("be.disabled");
// type delete
cy.wait(500).get('[data-test-subj="deleteInput"]').type("delete");
cy.get('[data-test-subj="deleteConfirmButton"]').should("not.be.disabled");
// click to delete
cy.get('[data-test-subj="deleteConfirmButton"]').click();
// the alias should not exist
cy.wait(500);
cy.get(`#_selection_column_ds--checkbox`).should("not.exist");
});
});

after(() => {
cy.request({
url: `${Cypress.env("opensearch")}/_data_stream`,
method: "DELETE",
failOnStatusCode: false,
});
cy.deleteTemplate("index-common-template");
});
});
2 changes: 1 addition & 1 deletion opensearch_dashboards.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "indexManagementDashboards",
"version": "2.5.0.0",
"opensearchDashboardsVersion": "2.5.0",
"opensearchDashboardsVersion": "2.5.1",
"configPath": ["opensearch_index_management"],
"requiredPlugins": ["navigation", "opensearchDashboardsReact"],
"server": true,
Expand Down
10 changes: 5 additions & 5 deletions public/components/DescriptionListHoz/DescriptionListHoz.tsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { EuiDescriptionList, EuiDescriptionListProps, EuiFlexGroup, EuiFlexItem } from "@elastic/eui";
import { EuiDescriptionList, EuiDescriptionListProps, EuiFlexGrid, EuiFlexGridProps, EuiFlexItem } from "@elastic/eui";
import React from "react";

const DisplayItem = (
Expand All @@ -20,15 +20,15 @@ const DisplayItem = (
);
};

export default function DescriptionListHoz(props: EuiDescriptionListProps) {
const { listItems, ...others } = props;
export default function DescriptionListHoz(props: EuiDescriptionListProps & Pick<EuiFlexGridProps, "columns">) {
const { listItems, columns = 4, ...others } = props;
return (
<EuiFlexGroup>
<EuiFlexGrid columns={columns}>
{listItems?.map((item) => (
<EuiFlexItem key={item.title as string}>
<DisplayItem listItem={item} {...others} />
</EuiFlexItem>
))}
</EuiFlexGroup>
</EuiFlexGrid>
);
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
exports[`<DescriptionListHoz /> spec renders the component 1`] = `
<div>
<div
class="euiFlexGroup euiFlexGroup--gutterLarge euiFlexGroup--directionRow euiFlexGroup--responsive"
class="euiFlexGrid euiFlexGrid--gutterLarge euiFlexGrid--fourths euiFlexGrid--responsive"
>
<div
class="euiFlexItem"
Expand Down
4 changes: 2 additions & 2 deletions public/components/FormGenerator/built_in_components/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import EuiComboBox from "../../ComboBoxWithoutWarning";
export type ComponentMapEnum = "Input" | "Number" | "Switch" | "Select" | "Text" | "ComboBoxSingle";

export interface IFieldComponentProps extends IEuiToolTipWrapperProps {
onChange: (val: IFieldComponentProps["value"]) => void;
onChange: (val: IFieldComponentProps["value"], ...args: any) => void;
value?: any;
[key: string]: any;
}
Expand Down Expand Up @@ -55,7 +55,7 @@ const componentMap: Record<ComponentMapEnum, React.ComponentType<IFieldComponent
ref={ref}
onChange={(selectedOptions) => {
if (selectedOptions && selectedOptions[0]) {
onChange(selectedOptions[0].label);
onChange(selectedOptions[0].label, selectedOptions[0]);
} else {
onChange(undefined);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import React, { forwardRef, Ref, useRef, useState } from "react";
import { render, fireEvent, waitFor, act } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import IndexMapping, { IIndexMappingsRef, IndexMappingProps, transformObjectToArray } from "./IndexMapping";
import { MappingsProperties } from "../../../../../models/interfaces";
import { MappingsProperties } from "../../../models/interfaces";
import { renderHook } from "@testing-library/react-hooks";

const IndexMappingOnChangeWrapper = forwardRef((props: Partial<IndexMappingProps>, ref: Ref<IIndexMappingsRef>) => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@
import React, { forwardRef, useCallback, useState, Ref, useRef, useMemo, useImperativeHandle } from "react";
import { EuiTreeView, EuiIcon, EuiTreeViewProps, EuiButton, EuiSpacer, EuiButtonGroup, EuiLink } from "@elastic/eui";
import { set, get, isEmpty } from "lodash";
import JSONEditor, { IJSONEditorRef } from "../../../../components/JSONEditor";
import { Modal } from "../../../../components/Modal";
import { MappingsProperties } from "../../../../../models/interfaces";
import CustomFormRow from "../../../../components/CustomFormRow";
import JSONEditor, { IJSONEditorRef } from "../JSONEditor";
import { Modal } from "../Modal";
import { MappingsProperties } from "../../../models/interfaces";
import CustomFormRow from "../CustomFormRow";
import MappingLabel, { IMappingLabelRef } from "../MappingLabel";
import { transformObjectToArray, transformArrayToObject, countNodesInTree } from "./helper";
import { IndexMappingsObjectAll, IndexMappingProps, EDITOR_MODE, IIndexMappingsRef } from "./interfaces";
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { MappingsProperties, MappingsPropertiesObject } from "../../../../../models/interfaces";
import { MappingsProperties, MappingsPropertiesObject } from "../../../models/interfaces";

export const transformObjectToArray = (obj: MappingsPropertiesObject): MappingsProperties => {
return Object.entries(obj).map(([fieldName, fieldSettings]) => {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { MappingsProperties, MappingsPropertiesObject } from "../../../../../models/interfaces";
import { MappingsProperties, MappingsPropertiesObject } from "../../../models/interfaces";

export type IndexMappingsAll = {
properties?: MappingsProperties;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@ import {
EuiFormRowProps,
} from "@elastic/eui";
import { set, pick } from "lodash";
import { MappingsProperties } from "../../../../../models/interfaces";
import { AllBuiltInComponents } from "../../../../components/FormGenerator";
import useField, { transformNameToString } from "../../../../lib/field";
import { INDEX_MAPPING_TYPES, INDEX_MAPPING_TYPES_WITH_CHILDREN } from "../../../../utils/constants";
import SimplePopover from "../../../../components/SimplePopover";
import { MappingsProperties } from "../../../models/interfaces";
import { AllBuiltInComponents } from "../FormGenerator";
import useField, { transformNameToString } from "../../lib/field";
import { INDEX_MAPPING_TYPES, INDEX_MAPPING_TYPES_WITH_CHILDREN } from "../../utils/constants";
import SimplePopover from "../SimplePopover";

const OLD_VALUE_DISABLED_REASON = "Old mappings cannot be modified";

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
import React from "react";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copyright missing

import { EuiLink, EuiSpacer, EuiTitle } from "@elastic/eui";
import flat from "flat";
import CustomFormRow from "../../../../components/CustomFormRow";
import { AllBuiltInComponents } from "../../../../components/FormGenerator";
import AdvancedSettings from "../../../../components/AdvancedSettings";
import DescriptionListHoz from "../../../../components/DescriptionListHoz";
import { INDEX_SETTINGS_URL } from "../../../../utils/constants";
import { SubDetailProps } from "../../interface";
import { getCommonFormRowProps } from "../../hooks";

export default function IndexSettings(props: SubDetailProps) {
const { readonly, field } = props;
const values = field.getValues();
return (
<>
<EuiTitle size="s">
<span>Index settings</span>
</EuiTitle>
<EuiSpacer size="s" />
{readonly ? (
<DescriptionListHoz
listItems={[
{
title: "Number of primary shards",
description: values.template?.settings?.["index.number_of_shards"] || "-",
},
{
title: "Number of replicas",
description: values.template?.settings?.["index.number_of_replicas"] || "-",
},
{
title: "Refresh interval",
description: values.template?.settings?.["index.refresh_interval"] || "-",
},
]}
/>
) : (
<>
<CustomFormRow
label="Number of primary shards"
helpText="Specify the number of primary shards in the index. Default is 1."
{...getCommonFormRowProps(["template", "settings", "index.number_of_shards"], field)}
>
<AllBuiltInComponents.Text
{...field.registerField({
name: ["template", "settings", "index.number_of_shards"],
})}
/>
</CustomFormRow>
<EuiSpacer />
<CustomFormRow
fullWidth
label="Number of replicas"
helpText="Specify the number of replicas each primary shard should have. Default is 1."
{...getCommonFormRowProps(["template", "settings", "index.number_of_replicas"], field)}
>
<AllBuiltInComponents.Text
{...field.registerField({
name: ["template", "settings", "index.number_of_replicas"],
})}
/>
</CustomFormRow>
<EuiSpacer />
<CustomFormRow
label="Refresh interval"
helpText="Specify how often the index should refresh, which publishes its most recent changes and makes them available for search. Default is 1s."
{...getCommonFormRowProps(["template", "settings", "index.refresh_interval"], field)}
>
<AllBuiltInComponents.Text
{...field.registerField({
name: ["template", "settings", "index.refresh_interval"],
})}
/>
</CustomFormRow>
</>
)}
<EuiSpacer />
<AdvancedSettings
value={field.getValues().template.settings || {}}
onChange={(totalValue) => {
field.setValue(["template", "settings"], totalValue);
field.validatePromise();
}}
accordionProps={{
initialIsOpen: false,
id: "accordionForCreateDataStreamSettings",
buttonContent: <h4>Advanced settings</h4>,
}}
editorProps={{
disabled: true,
width: "100%",
formatValue: flat,
}}
rowProps={{
fullWidth: true,
label: "Specify advanced index settings",
helpText: (
<>
<p>
Specify a comma-delimited list of settings.{" "}
<EuiLink href={INDEX_SETTINGS_URL} target="_blank" external>
View index settings
</EuiLink>
</p>
<p>
All the settings will be handled in flat structure.{" "}
<EuiLink
href="https://opensearch.org/docs/latest/api-reference/index-apis/get-index/#url-parameters"
external
target="_blank"
>
Learn more.
</EuiLink>
</p>
</>
),
}}
/>
</>
);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import IndexSettings from "./IndexSettings";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copyright missing


export default IndexSettings;
Loading