Skip to content

Commit

Permalink
[8.x] [Mappings Editor] Add support for synthetic _source (#199854) (#…
Browse files Browse the repository at this point in the history
…200776)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Mappings Editor] Add support for synthetic _source
(#199854)](#199854)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Elena
Stoeva","email":"59341489+ElenaStoeva@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-11-19T16:49:50Z","message":"[Mappings
Editor] Add support for synthetic _source (#199854)\n\nCloses
https://github.com/elastic/kibana/issues/198621\r\n\r\n##
Summary\r\n\r\nThis PR adds support for the synthetic _source field in
the mappings\r\nAdvanced options.\r\n\r\nStored option:\r\n<img
width=\"1184\" alt=\"Screenshot 2024-11-14 at 19 19
19\"\r\nsrc=\"https://github.com/user-attachments/assets/086f7a3e-9ca1-42de-9f9c-d3599d839ccf\">\r\n\r\nSynthetic
option selected:\r\n<img width=\"1184\" alt=\"Screenshot 2024-11-14 at
19 19
27\"\r\nsrc=\"https://github.com/user-attachments/assets/3700bced-212a-4378-b51a-ab7a0f4f7b99\">\r\n\r\nDisabled
option selected:\r\n<img width=\"1184\" alt=\"Screenshot 2024-11-14 at
19 19
36\"\r\nsrc=\"https://github.com/user-attachments/assets/c7ddcbae-7c78-4477-824e-99b144a1f750\">\r\n\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/399d0f95-a5dd-4874-bb8c-e95d6ed38465\r\n\r\n\r\n\r\n\r\nHow
to test:\r\n1. Start Es with `yarn es snapshot --license` (we need
Enterprise\r\nlicense to see the Synthetic source option) and Kibana
with `yarn start`\r\n2. Go to Index templates/Component templates and
start creating a\r\ntemplate\r\n3. At the Mappings step, go to Advanced
options.\r\n4. Verify that selecting a _source field option translates
to the\r\ncorrect Es request.\r\n5. In Index templates form, verify that
the default _source option\r\ndepends on the index mode selected in the
Logistics step. For LogsDB and\r\nTime series index mode, the default
should be synthetic mode; otherwise,\r\nthe stored option.\r\n6. Verify
that in Basic license, the synthetic option is not
displayed.\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added
follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] Any UI
touched in this PR is usable by keyboard only (learn more\r\nabout
[keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n-
[x] Any UI touched in this PR does not create any new axe
failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[x] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[x] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"8be679ae20fb7f5c41ccb3554b6db3dc2cad0678","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Kibana
Management","release_note:skip","Feature:Mappings
Editor","v9.0.0","backport:prev-minor","v8.17.0"],"title":"[Mappings
Editor] Add support for synthetic
_source","number":199854,"url":"https://github.com/elastic/kibana/pull/199854","mergeCommit":{"message":"[Mappings
Editor] Add support for synthetic _source (#199854)\n\nCloses
https://github.com/elastic/kibana/issues/198621\r\n\r\n##
Summary\r\n\r\nThis PR adds support for the synthetic _source field in
the mappings\r\nAdvanced options.\r\n\r\nStored option:\r\n<img
width=\"1184\" alt=\"Screenshot 2024-11-14 at 19 19
19\"\r\nsrc=\"https://github.com/user-attachments/assets/086f7a3e-9ca1-42de-9f9c-d3599d839ccf\">\r\n\r\nSynthetic
option selected:\r\n<img width=\"1184\" alt=\"Screenshot 2024-11-14 at
19 19
27\"\r\nsrc=\"https://github.com/user-attachments/assets/3700bced-212a-4378-b51a-ab7a0f4f7b99\">\r\n\r\nDisabled
option selected:\r\n<img width=\"1184\" alt=\"Screenshot 2024-11-14 at
19 19
36\"\r\nsrc=\"https://github.com/user-attachments/assets/c7ddcbae-7c78-4477-824e-99b144a1f750\">\r\n\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/399d0f95-a5dd-4874-bb8c-e95d6ed38465\r\n\r\n\r\n\r\n\r\nHow
to test:\r\n1. Start Es with `yarn es snapshot --license` (we need
Enterprise\r\nlicense to see the Synthetic source option) and Kibana
with `yarn start`\r\n2. Go to Index templates/Component templates and
start creating a\r\ntemplate\r\n3. At the Mappings step, go to Advanced
options.\r\n4. Verify that selecting a _source field option translates
to the\r\ncorrect Es request.\r\n5. In Index templates form, verify that
the default _source option\r\ndepends on the index mode selected in the
Logistics step. For LogsDB and\r\nTime series index mode, the default
should be synthetic mode; otherwise,\r\nthe stored option.\r\n6. Verify
that in Basic license, the synthetic option is not
displayed.\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added
follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] Any UI
touched in this PR is usable by keyboard only (learn more\r\nabout
[keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n-
[x] Any UI touched in this PR does not create any new axe
failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[x] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[x] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"8be679ae20fb7f5c41ccb3554b6db3dc2cad0678"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199854","number":199854,"mergeCommit":{"message":"[Mappings
Editor] Add support for synthetic _source (#199854)\n\nCloses
https://github.com/elastic/kibana/issues/198621\r\n\r\n##
Summary\r\n\r\nThis PR adds support for the synthetic _source field in
the mappings\r\nAdvanced options.\r\n\r\nStored option:\r\n<img
width=\"1184\" alt=\"Screenshot 2024-11-14 at 19 19
19\"\r\nsrc=\"https://github.com/user-attachments/assets/086f7a3e-9ca1-42de-9f9c-d3599d839ccf\">\r\n\r\nSynthetic
option selected:\r\n<img width=\"1184\" alt=\"Screenshot 2024-11-14 at
19 19
27\"\r\nsrc=\"https://github.com/user-attachments/assets/3700bced-212a-4378-b51a-ab7a0f4f7b99\">\r\n\r\nDisabled
option selected:\r\n<img width=\"1184\" alt=\"Screenshot 2024-11-14 at
19 19
36\"\r\nsrc=\"https://github.com/user-attachments/assets/c7ddcbae-7c78-4477-824e-99b144a1f750\">\r\n\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/399d0f95-a5dd-4874-bb8c-e95d6ed38465\r\n\r\n\r\n\r\n\r\nHow
to test:\r\n1. Start Es with `yarn es snapshot --license` (we need
Enterprise\r\nlicense to see the Synthetic source option) and Kibana
with `yarn start`\r\n2. Go to Index templates/Component templates and
start creating a\r\ntemplate\r\n3. At the Mappings step, go to Advanced
options.\r\n4. Verify that selecting a _source field option translates
to the\r\ncorrect Es request.\r\n5. In Index templates form, verify that
the default _source option\r\ndepends on the index mode selected in the
Logistics step. For LogsDB and\r\nTime series index mode, the default
should be synthetic mode; otherwise,\r\nthe stored option.\r\n6. Verify
that in Basic license, the synthetic option is not
displayed.\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added
follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] Any UI
touched in this PR is usable by keyboard only (learn more\r\nabout
[keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n-
[x] Any UI touched in this PR does not create any new axe
failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[x] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[x] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"8be679ae20fb7f5c41ccb3554b6db3dc2cad0678"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Elena Stoeva <59341489+ElenaStoeva@users.noreply.github.com>
  • Loading branch information
kibanamachine and ElenaStoeva authored Nov 19, 2024
1 parent cbe6d4a commit 12031fa
Show file tree
Hide file tree
Showing 22 changed files with 562 additions and 51 deletions.
1 change: 1 addition & 0 deletions packages/kbn-doc-links/src/get_doc_links.ts
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,7 @@ export const getDocLinks = ({ kibanaBranch, buildFlavor }: GetDocLinkOptions): D
mappingSimilarity: `${ELASTICSEARCH_DOCS}similarity.html`,
mappingSourceFields: `${ELASTICSEARCH_DOCS}mapping-source-field.html`,
mappingSourceFieldsDisable: `${ELASTICSEARCH_DOCS}mapping-source-field.html#disable-source-field`,
mappingSyntheticSourceFields: `${ELASTICSEARCH_DOCS}mapping-source-field.html#synthetic-source`,
mappingStore: `${ELASTICSEARCH_DOCS}mapping-store.html`,
mappingSubobjects: `${ELASTICSEARCH_DOCS}subobjects.html`,
mappingTermVector: `${ELASTICSEARCH_DOCS}term-vector.html`,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -459,6 +459,7 @@ export type TestSubjects =
| 'advancedConfiguration.dynamicMappingsToggle.input'
| 'advancedConfiguration.metaField'
| 'advancedConfiguration.routingRequiredToggle.input'
| 'sourceValueField'
| 'sourceField.includesField'
| 'sourceField.excludesField'
| 'dynamicTemplatesEditor'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,24 @@ describe('Mappings editor: core', () => {
let onChangeHandler: jest.Mock = jest.fn();
let getMappingsEditorData = getMappingsEditorDataFactory(onChangeHandler);
let testBed: MappingsEditorTestBed;
const appDependencies = { plugins: { ml: { mlApi: {} } } };
let hasEnterpriseLicense = true;
const mockLicenseCheck = jest.fn((type: any) => hasEnterpriseLicense);
const appDependencies = {
plugins: {
ml: { mlApi: {} },
licensing: {
license$: {
subscribe: jest.fn((callback: any) => {
callback({
isActive: true,
hasAtLeast: mockLicenseCheck,
});
return { unsubscribe: jest.fn() };
}),
},
},
},
};

beforeAll(() => {
jest.useFakeTimers({ legacyFakeTimers: true });
Expand Down Expand Up @@ -456,13 +473,96 @@ describe('Mappings editor: core', () => {
updatedMappings = {
...updatedMappings,
dynamic: false,
// The "enabled": true is removed as this is the default in Es
_source: {
includes: defaultMappings._source.includes,
excludes: defaultMappings._source.excludes,
},
};
delete updatedMappings.date_detection;
delete updatedMappings.dynamic_date_formats;
delete updatedMappings.numeric_detection;

expect(data).toEqual(updatedMappings);
});

describe('props.indexMode sets the correct default value of _source field', () => {
it("defaults to 'stored' with 'standard' index mode prop", async () => {
await act(async () => {
testBed = setup(
{
value: { ...defaultMappings, _source: undefined },
onChange: onChangeHandler,
indexMode: 'standard',
},
ctx
);
});
testBed.component.update();

const {
actions: { selectTab },
find,
} = testBed;

await selectTab('advanced');

// Check that the stored option is selected
expect(find('sourceValueField').prop('value')).toBe('stored');
});

['logsdb', 'time_series'].forEach((indexMode) => {
it(`defaults to 'synthetic' with ${indexMode} index mode prop on enterprise license`, async () => {
hasEnterpriseLicense = true;
await act(async () => {
testBed = setup(
{
value: { ...defaultMappings, _source: undefined },
onChange: onChangeHandler,
indexMode,
},
ctx
);
});
testBed.component.update();

const {
actions: { selectTab },
find,
} = testBed;

await selectTab('advanced');

// Check that the synthetic option is selected
expect(find('sourceValueField').prop('value')).toBe('synthetic');
});

it(`defaults to 'standard' with ${indexMode} index mode prop on basic license`, async () => {
hasEnterpriseLicense = false;
await act(async () => {
testBed = setup(
{
value: { ...defaultMappings, _source: undefined },
onChange: onChangeHandler,
indexMode,
},
ctx
);
});
testBed.component.update();

const {
actions: { selectTab },
find,
} = testBed;

await selectTab('advanced');

// Check that the stored option is selected
expect(find('sourceValueField').prop('value')).toBe('stored');
});
});
});
});

describe('multi-fields support', () => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,21 @@
* 2.0.
*/

import React, { useEffect, useRef, useCallback } from 'react';
import React, { useEffect, useRef } from 'react';
import { EuiSpacer } from '@elastic/eui';

import { FormData } from '@kbn/es-ui-shared-plugin/static/forms/hook_form_lib';
import { useAppContext } from '../../../../app_context';
import { useForm, Form } from '../../shared_imports';
import { GenericObject, MappingsConfiguration } from '../../types';
import { MapperSizePluginId } from '../../constants';
import { useDispatch } from '../../mappings_state_context';
import { DynamicMappingSection } from './dynamic_mapping_section';
import { SourceFieldSection } from './source_field_section';
import {
SourceFieldSection,
STORED_SOURCE_OPTION,
SYNTHETIC_SOURCE_OPTION,
DISABLED_SOURCE_OPTION,
} from './source_field_section';
import { MetaFieldSection } from './meta_field_section';
import { RoutingSection } from './routing_section';
import { MapperSizePluginSection } from './mapper_size_plugin_section';
Expand All @@ -28,7 +32,14 @@ interface Props {
esNodesPlugins: string[];
}

const formSerializer = (formData: GenericObject, sourceFieldMode?: string) => {
interface SerializedSourceField {
enabled?: boolean;
mode?: string;
includes?: string[];
excludes?: string[];
}

export const formSerializer = (formData: GenericObject) => {
const { dynamicMapping, sourceField, metaField, _routing, _size, subobjects } = formData;

const dynamic = dynamicMapping?.enabled
Expand All @@ -37,12 +48,30 @@ const formSerializer = (formData: GenericObject, sourceFieldMode?: string) => {
? 'strict'
: dynamicMapping?.enabled;

const _source =
sourceField?.option === SYNTHETIC_SOURCE_OPTION
? { mode: SYNTHETIC_SOURCE_OPTION }
: sourceField?.option === DISABLED_SOURCE_OPTION
? { enabled: false }
: sourceField?.option === STORED_SOURCE_OPTION
? {
mode: 'stored',
includes: sourceField?.includes,
excludes: sourceField?.excludes,
}
: sourceField?.includes || sourceField?.excludes
? {
includes: sourceField?.includes,
excludes: sourceField?.excludes,
}
: undefined;

const serialized = {
dynamic,
numeric_detection: dynamicMapping?.numeric_detection,
date_detection: dynamicMapping?.date_detection,
dynamic_date_formats: dynamicMapping?.dynamic_date_formats,
_source: sourceFieldMode ? { mode: sourceFieldMode } : sourceField,
_source: _source as SerializedSourceField,
_meta: metaField,
_routing,
_size,
Expand All @@ -52,19 +81,15 @@ const formSerializer = (formData: GenericObject, sourceFieldMode?: string) => {
return serialized;
};

const formDeserializer = (formData: GenericObject) => {
export const formDeserializer = (formData: GenericObject) => {
const {
dynamic,
/* eslint-disable @typescript-eslint/naming-convention */
numeric_detection,
date_detection,
dynamic_date_formats,
/* eslint-enable @typescript-eslint/naming-convention */
_source: { enabled, includes, excludes } = {} as {
enabled?: boolean;
includes?: string[];
excludes?: string[];
},
_source: { enabled, mode, includes, excludes } = {} as SerializedSourceField,
_meta,
_routing,
// For the Mapper Size plugin
Expand All @@ -81,7 +106,14 @@ const formDeserializer = (formData: GenericObject) => {
dynamic_date_formats,
},
sourceField: {
enabled,
option:
mode === 'stored'
? STORED_SOURCE_OPTION
: mode === 'synthetic'
? SYNTHETIC_SOURCE_OPTION
: enabled === false
? DISABLED_SOURCE_OPTION
: undefined,
includes,
excludes,
},
Expand All @@ -99,14 +131,9 @@ export const ConfigurationForm = React.memo(({ value, esNodesPlugins }: Props) =

const isMounted = useRef(false);

const serializerCallback = useCallback(
(formData: FormData) => formSerializer(formData, value?._source?.mode),
[value?._source?.mode]
);

const { form } = useForm({
schema: configurationFormSchema,
serializer: serializerCallback,
serializer: formSerializer,
deserializer: formDeserializer,
defaultValue: value,
id: 'configurationForm',
Expand Down Expand Up @@ -165,7 +192,7 @@ export const ConfigurationForm = React.memo(({ value, esNodesPlugins }: Props) =
<EuiSpacer size="xl" />
<MetaFieldSection />
<EuiSpacer size="xl" />
{enableMappingsSourceFieldSection && !value?._source?.mode && (
{enableMappingsSourceFieldSection && (
<>
<SourceFieldSection /> <EuiSpacer size="xl" />
</>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,9 @@ export const configurationFormSchema: FormSchema = {
},
},
sourceField: {
enabled: {
label: i18n.translate('xpack.idxMgmt.mappingsEditor.configuration.sourceFieldLabel', {
defaultMessage: 'Enable _source field',
}),
type: FIELD_TYPES.TOGGLE,
defaultValue: true,
option: {
type: FIELD_TYPES.SUPER_SELECT,
defaultValue: 'stored',
},
includes: {
label: i18n.translate('xpack.idxMgmt.mappingsEditor.configuration.includeSourceFieldsLabel', {
Expand Down
Loading

0 comments on commit 12031fa

Please sign in to comment.