Skip to content

Commit

Permalink
[Ingest Node Pipelines] Pipeline Processors Editor (#66021)
Browse files Browse the repository at this point in the history
* initial plugin setup

* add smoke test

* fix license check

* refactor plugin setup

* Server-side create/update ingest pipelines (#62744)

* List pipelines (#62785)

* First iteration of ingest table

* Add action placeholders

* Refactor list and address PR feedback

Refactored the list into smaller pieces and assemble in main.tsx

Also addressed feedback on copy, removed unused notifications dep

* WiP on flyout

Showing name in title

* Add reload button

* Finish first version of flyout

* Slight update to copy

* `delete` -> `edit`

* Address PR feedback

Copy and a11y updates

* Add on failure JSON to flyout if it is available

* Add details json block file and remove ununsed import

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* [Ingest pipelines] Create pipeline UI (#63017)

* First vertical slice of pipeline editor component

* Made a space for common parameters

* [Ingest pipelines] Edit pipeline page (#63522)

* First iteration of CRUD functionality working

* WiP on moving the pipeline editor to pipeline processor editor

* Finish refactor to work with passing state out

* Refactor and fix tests

* [Ingest pipelines] Polish details panel and empty list (#63926)

* Address some early feedback and use FormDataProvider

FormDataProvider gives a more declarative way to listen to form
state updates.

Also refactored the state reader mechanism.

* [Ingest pipelines] Delete pipeline (#63635)

* [Ingest Node Pipelines] Clone Pipeline (#64049)

* First iteration of clone functionality

Wired up for both the list table and the details flyout in the
list section.

* satisfy eslint

* Turn on sorting for the list table

* Clean up const declarations

* Address PR feedback

Sentence-casify and update some other copy.

* Mark edit and delete as primary actions in list table

* Handle URI encoded chars in pipeline name when cloning

* Update to using the more flexible controlled component pattern

To make this component and mappings editor more consistent, both
will expose a more traditional controlled component interface to
consumers.

In the current implementation this requires that consumers use
`useCallback` for `onUpdate` handlers to avoid an infinite
rendering cycle and to avoid staleness bugs in their `onUpdate`
handlers they need to think about what might make the handler
stale.

This approach comes with the benefits and flexibility of
controlled components at the cost of slightly more complex
consumption of the components.

In future, we can explore adding the uncontrolled component
interface too so that consumers have the option to more simply
render and pull data out only when needed

* Handle sub-form validity

The pipelines processor editor not emits overall validity to
consumers

* Fix Jest test

* Refactor some names

prepareDataOut -> serialize
prepareDataIn -> deserialize
EditorProcessor -> ProcessorInternal

* Mark as private

* Major WiP

Started working on the drag-and-drop-tree and updated some
typings stuff

* [Ingest node pipelines] Privileges (#63850)

* Create privileges check for ingest pipelines app

Also moved the public side logic for checking and rendering
privilege related messages to es_ui_shared/public following the
new __packages_do_not_import__ convention.

* Add ,

* Fix import paths

* Address PR feedback

Fix i18n strings (remove reference to snapshot and restore) and
fix copy referencing snapshot and restore - all copy-pasta errors.

Also remove unused field from missing privileges object.

* Fix issue from resolving merge conflicts

* Add missing app privilege

* Use non-deprecated privilige name

* First iteration of drag and drop tree on feature parity

* First steps toward add on failure handler

Updated reducer logic to create the next copy value using immer.

* First iteration of nested tree with add on failure working

* Refactor and some UI layout updates

- Remove the "id" field on processors for now
- Implement the nested remove and update functionality again

* Remove immer (not call stack safe)

Refac to remove immer and reimplemented the immutable set and
get functions.

* [Ingest Node Pipelines] More lenient treatment of on-failure value (#64411)

* Move file to components folder

* [Ingest pipelines] Simulate pipeline (#64223)

* Updated tree rendering

- turn off dropzones for children
- fixed up some padding and margins
- fixed integration with pipeline_form.tsx

The current implementation still has a lot of jank stemming from
the UX with DnD. Unfortunately the nesting has opened a number
of issues here.

* [Ingest Node Pipelines] Show flyout after editing or creating a pipeline (#64409)

* Show flyout after editing or creating a pipeline

* JSX comment

* Show not found flyout

Copied from CCR

* update not found flyout and fix behavior when viewing details from table

* Reset pipeline name in URI when closing flyout

* Remove encodeURI

Already using encodingURIComponent for unsafe string.

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* Clarification of terms

- addProcessor -> addTopLevelProcessor (same in editor modes)
- Expanded comment on ProcessorSelector

* Implement move between lists functionality

Added tests to the reducer for moving in and out of a recusrsive
structure.

* fix TS

* Prevent nesting a parent inside of its own child

* Add comment

* [Ingest pipelines] Cleanup (#64794)

* address review feedback

* remove unused import

* Big refactor to tree rendering structure

DnD tree now converts the nested tree to a flat one and only
consists of _1_ droppable area with a flat array of draggables
that can be combined.

Using the existing logic in the reducer combined with translating
the flat structure changes to a format the nested reducer can
understand looks like a really promising avenue.

There still seems to be a bug with a longer list where items do
not interact properly.

* Remove unused component

* A number of NB changes

- Fixed a subtle serialisation bug with on_failure in options
- Fixed a performance bug by using useRef in pipeline form
- Memoized the drag and drop tree
- Removed use of "isValid" from form lib... I think this should
be removed entirely.

* fix bad conflict resolution

* Implemented a slightly better destination resolution algo.

Also added tests.

* Fix subtle staleness bug, whitelist keys for setValue

* NB styling fix!!

Due to a parent's setting of overflow: hidden the drag and drop
tree had a dead zone that would equal the page overflow limit,
unsetting on the component itself (overflow: auto) relaxes this
limitation.

* Fix stale delete bug too

* Update naming of editor modes and update comments

* Use field types combo box

* Add delete confirmation modal

* Refactor delete modal component file name

* Better visual integration with existing form

* Update layout and styling of form

- added some padding around the new processors editor field
- Updated use of flex box in the pipeline form fields

* Move pipeline processor copy into pipeline processor component

The test button is also now inside of the pipeline processor
editor. Eventually all of this functionality should be moved
into the pipeline processor editor.

* First step of refactor to moving between trees

* First iteration of x-tree drag and drop

* Remove unused import

* Fix jest test types

* Fix up minor i18n issues and fix up layout of on failure

* Remove unnecessary prop

* Update spacing above add processor button to make it more center

* Fix destination resolution algo

* Update dragging resolver unit tests and add a lot more comments

* Use one sorting algo (removed use of euiReorder for now)

* Add placeholder tests and update comments

* Quite a big refactor

- Remove DraggableLocation entirely, only use ProcessorSelector
this simplifies mapping back to reducer instructions quite a lot
- Add tests for special case dragging behaviour

* Fix off by one bug in tests and implementation 🤦🏼‍♂

- Also move processor reducer to it's own folder

* Update behaviour for dragging up across trees and add tests

* Fix combine instruction

* Fix test and i18n issues

* Remove background color

* Fix selector after selector refactor

* A major performance

- Do not re-render the entire tree when editing settings

* Fix component smoke test

* Fix reading value from processor state using processor selector

* [Ingest pipelines] Custom processor form (#66022)

* Re add background color and refactor name of processor item

* Fix file naming and refactor 🚜 dnd tree rendering

- deserialze and serialize were backwards, fixed
- the dnd tree was rendering a flat tree which needed to be
mapped back to the nested tree for updates. This is still the
case but we do use recursive rendering. This enables tree nodes
to better hold local state like whether they are collapsed or not.

* Fix getting of initial on failure processors value

* Update padding styles for containers

* A lot of styling updates to get closer to look of mockup

* A WiP version of th click-tree an alternative to dnd

As a response to really long pipelines we may pivot away from
dnd.

* Remove dnd tree

* clean up reamining dnd tree references

* Clean up and refactor of tree component

To simplify the logic of the tree component processor id's
were added. Also had to update the jest spec to support this

* Added the ability to duplicate a processor

* Fix types in test

* Added duplicate functionality to ui

* Memoize tree components

* address es lint issues

* remove unused import

* Fix editing of custom json

* Address form performance issues

* Add all known missing processors

* Add ability to cancel move

* Fix staleness in test and view request flyouts

* fix type issue

* Remove unused translations and skip funcitonal test for now

* add todo comment

* Fix type issues

* remove poc styles

* disable other move buttons if we have a selected processor

* Refactor drop zone pin to button and add some styling

* Refactor processor editor item

* Update styling and use icon for cancel move action too

* fix nasty integration bug

* some minor optimizations

* prevent parent from being placed in own on failure handler

* Re-add cancel button

* Re-introduce failure processors toggle

* Fix typo

* Add Handler types for processor editor item

* Fix staleness bug for type, refactor classname and fix duplicate
bug not immutably copying values

* Experimenting with padels (revert this to undo if no further
changes have been made)

* Add description and unique ids

* Share links via component-wide context rather than props

* Virtualized list and back to outline dropzones

* Refactor id getter to a service and make it an incrementing number

* Temporary fix for double rendering issue.

This means the pipeline processors component is not controllable
but fixes the double rendering even when processors have not
changed. This is a problem coming from the ingest pipelines
plugin form system outside of this component

* add todo comment

* remove euicode element

* properly handle duplicate flow

* attempt to fix i18n

* split private_tree into it's own component and add comments

* refactor 🚜. rename Tree to ProcessorsTree and move things around

* do not delete the top level arrays for processors and onfailures

* fix typescript error

* Move duplicate, addOnFailure and delete actions into ctx menu

* remove unused import

* add support for pressing esc key to cancel move

* Add outside click listener

* always prompt before deleting a processor

* refactor remove distinction between adding top level and on fail

* add processor button to tree

* Hide the add on failure context menu item for processors with
failure handlers

* Reinstated x-tree moving and highlight and disable for buttons
on move and on edit

* removing ids step 1: remove idGenerator

* remove ids step 2: added inline text input

Also refactored a lot of the tree actions away. Now using
context and the processors editor tree item component for
dispatching actions. The tree item has access to the dispatch
and to the selector which makes it a well positioned component.

Also saves on some props drilling.

* Slight improvement to styling of text input (border)

* Re-implement missing failure toggle test

* address type todo

* Address many type issues and fix yarn.lock

* re-enable create pipeline functional test

* prevent multiple flyouts from opening

* change flyout title when editing an on-failure processor

* absolutely position the failure handlers label

when we render a dropzone on the label, then we move the label
up without affecting overall component height

* fix description behaviour not removing tag if empty

* some minor clean up

* add onflyoutopen cb to tests

* refactor processors editor item to multiple files

also refactored i18n into it's own file. would be good to come
up with a pttaern for doing this more broadly.

* fix add on-failure handler in context menu after refactor

* tag -> new description field

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alison.goryachev@elastic.co>
  • Loading branch information
4 people committed Jun 16, 2020
1 parent 5c9734d commit b71edab
Show file tree
Hide file tree
Showing 59 changed files with 3,438 additions and 197 deletions.
1 change: 1 addition & 0 deletions x-pack/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,7 @@
"react-syntax-highlighter": "^5.7.0",
"react-tiny-virtual-list": "^2.2.0",
"react-use": "^13.27.0",
"react-virtualized": "^9.21.2",
"react-vis": "^1.8.1",
"react-visibility-sensor": "^5.1.1",
"recompose": "^0.26.0",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -68,21 +68,6 @@ describe('<PipelinesCreate />', () => {
expect(exists('versionField')).toBe(true);
});

test('should toggle the on-failure processors editor', async () => {
const { actions, component, exists } = testBed;

// On-failure editor should be hidden by default
expect(exists('onFailureEditor')).toBe(false);

await act(async () => {
actions.toggleOnFailureSwitch();
await nextTick();
component.update();
});

expect(exists('onFailureEditor')).toBe(true);
});

test('should show the request flyout', async () => {
const { actions, component, find, exists } = testBed;

Expand Down
14 changes: 10 additions & 4 deletions x-pack/plugins/ingest_pipelines/common/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,16 @@
* you may not use this file except in compliance with the Elastic License.
*/

interface Processor {
[key: string]: {
[key: string]: unknown;
};
export interface ESProcessorConfig {
on_failure?: Processor[];
ignore_failure?: boolean;
if?: string;
tag?: string;
[key: string]: any;
}

export interface Processor {
[typeName: string]: ESProcessorConfig;
}

export interface Pipeline {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
* you may not use this file except in compliance with the Elastic License.
*/

import React, { useState } from 'react';
import React, { useState, useCallback, useRef } from 'react';
import { FormattedMessage } from '@kbn/i18n/react';
import { EuiButton, EuiButtonEmpty, EuiFlexGroup, EuiFlexItem, EuiSpacer } from '@elastic/eui';

Expand All @@ -16,6 +16,11 @@ import { PipelineTestFlyout } from './pipeline_test_flyout';
import { PipelineFormFields } from './pipeline_form_fields';
import { PipelineFormError } from './pipeline_form_error';
import { pipelineFormSchema } from './schema';
import {
OnUpdateHandlerArg,
OnUpdateHandler,
SerializeResult,
} from '../pipeline_processors_editor';

export interface PipelineFormProps {
onSave: (pipeline: Pipeline) => void;
Expand All @@ -30,8 +35,8 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({
defaultValue = {
name: '',
description: '',
processors: '',
on_failure: '',
processors: [],
on_failure: [],
version: '',
},
onSave,
Expand All @@ -44,10 +49,25 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({

const [isTestingPipeline, setIsTestingPipeline] = useState<boolean>(false);

const processorStateRef = useRef<OnUpdateHandlerArg>();

const handleSave: FormConfig['onSubmit'] = async (formData, isValid) => {
if (isValid) {
onSave(formData as Pipeline);
let override: SerializeResult | undefined;

if (!isValid) {
return;
}

if (processorStateRef.current) {
const processorsState = processorStateRef.current;
if (await processorsState.validate()) {
override = processorsState.getData();
} else {
return;
}
}

onSave({ ...formData, ...(override || {}) } as Pipeline);
};

const handleTestPipelineClick = () => {
Expand All @@ -60,6 +80,10 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({
onSubmit: handleSave,
});

const onEditorFlyoutOpen = useCallback(() => {
setIsRequestVisible(false);
}, [setIsRequestVisible]);

const saveButtonLabel = isSaving ? (
<FormattedMessage
id="xpack.ingestPipelines.form.savingButtonLabel"
Expand All @@ -77,6 +101,11 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({
/>
);

const onProcessorsChangeHandler = useCallback<OnUpdateHandler>(
(arg) => (processorStateRef.current = arg),
[]
);

return (
<>
<Form
Expand All @@ -90,10 +119,13 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({

{/* All form fields */}
<PipelineFormFields
onEditorFlyoutOpen={onEditorFlyoutOpen}
initialProcessors={defaultValue.processors}
initialOnFailureProcessors={defaultValue.on_failure}
onProcessorsUpdate={onProcessorsChangeHandler}
hasVersion={Boolean(defaultValue.version)}
isTestButtonDisabled={isTestingPipeline || form.isValid === false}
onTestPipelineClick={handleTestPipelineClick}
hasOnFailure={Boolean(defaultValue.on_failure)}
isEditing={isEditing}
/>

Expand Down Expand Up @@ -147,13 +179,19 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({
{/* ES request flyout */}
{isRequestVisible ? (
<PipelineRequestFlyout
readProcessors={() =>
processorStateRef.current?.getData() || { processors: [], on_failure: [] }
}
closeFlyout={() => setIsRequestVisible((prevIsRequestVisible) => !prevIsRequestVisible)}
/>
) : null}

{/* Test pipeline flyout */}
{isTestingPipeline ? (
<PipelineTestFlyout
readProcessors={() =>
processorStateRef.current?.getData() || { processors: [], on_failure: [] }
}
closeFlyout={() => {
setIsTestingPipeline((prevIsTestingPipeline) => !prevIsTestingPipeline);
}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,39 +6,41 @@

import React, { useState } from 'react';
import { FormattedMessage } from '@kbn/i18n/react';
import { i18n } from '@kbn/i18n';
import { EuiButton, EuiSpacer, EuiSwitch, EuiLink } from '@elastic/eui';
import { EuiSpacer, EuiSwitch } from '@elastic/eui';

import {
getUseField,
getFormRow,
Field,
JsonEditorField,
useKibana,
} from '../../../shared_imports';
import { Processor } from '../../../../common/types';
import { FormDataProvider } from '../../../shared_imports';
import { PipelineProcessorsEditor, OnUpdateHandler } from '../pipeline_processors_editor';

import { getUseField, getFormRow, Field, useKibana } from '../../../shared_imports';

interface Props {
initialProcessors: Processor[];
initialOnFailureProcessors?: Processor[];
onProcessorsUpdate: OnUpdateHandler;
hasVersion: boolean;
hasOnFailure: boolean;
isTestButtonDisabled: boolean;
onTestPipelineClick: () => void;
onEditorFlyoutOpen: () => void;
isEditing?: boolean;
}

const UseField = getUseField({ component: Field });
const FormRow = getFormRow({ titleTag: 'h3' });

export const PipelineFormFields: React.FunctionComponent<Props> = ({
initialProcessors,
initialOnFailureProcessors,
onProcessorsUpdate,
isEditing,
hasVersion,
hasOnFailure,
isTestButtonDisabled,
onTestPipelineClick,
onEditorFlyoutOpen,
}) => {
const { services } = useKibana();

const [isVersionVisible, setIsVersionVisible] = useState<boolean>(hasVersion);
const [isOnFailureEditorVisible, setIsOnFailureEditorVisible] = useState<boolean>(hasOnFailure);

return (
<>
Expand Down Expand Up @@ -110,127 +112,32 @@ export const PipelineFormFields: React.FunctionComponent<Props> = ({
/>
</FormRow>

{/* Processors field */}
<FormRow
title={
<FormattedMessage
id="xpack.ingestPipelines.form.processorsFieldTitle"
defaultMessage="Processors"
/>
}
description={
<>
<FormattedMessage
id="xpack.ingestPipelines.form.processorsFieldDescription"
defaultMessage="The processors to use to transform the documents before indexing. {learnMoreLink}"
values={{
learnMoreLink: (
<EuiLink
href={services.documentation.getProcessorsUrl()}
target="_blank"
external
>
{i18n.translate('xpack.ingestPipelines.form.processorsDocumentionLink', {
defaultMessage: 'Learn more',
})}
</EuiLink>
),
}}
/>

<EuiSpacer />
{/* Pipeline Processors Editor */}
<FormDataProvider pathsToWatch={['processors', 'on_failure']}>
{({ processors, on_failure: onFailure }) => {
const processorProp =
typeof processors === 'string' && processors
? JSON.parse(processors)
: initialProcessors ?? [];

<EuiButton
size="s"
onClick={onTestPipelineClick}
disabled={isTestButtonDisabled}
data-test-subj="testPipelineButton"
>
<FormattedMessage
id="xpack.ingestPipelines.form.testPipelineButtonLabel"
defaultMessage="Test pipeline"
/>
</EuiButton>
</>
}
>
<UseField
path="processors"
component={JsonEditorField}
componentProps={{
euiCodeEditorProps: {
'data-test-subj': 'processorsEditor',
height: '300px',
'aria-label': i18n.translate('xpack.ingestPipelines.form.processorsFieldAriaLabel', {
defaultMessage: 'Processors JSON editor',
}),
},
}}
/>
</FormRow>
const onFailureProp =
typeof onFailure === 'string' && onFailure
? JSON.parse(onFailure)
: initialOnFailureProcessors ?? [];

{/* On-failure field */}
<FormRow
title={
<FormattedMessage
id="xpack.ingestPipelines.form.onFailureTitle"
defaultMessage="Failure processors"
/>
}
description={
<>
<FormattedMessage
id="xpack.ingestPipelines.form.onFailureDescription"
defaultMessage="The alternate processors to execute after a processor fails. {learnMoreLink}"
values={{
learnMoreLink: (
<EuiLink
href={services.documentation.getHandlingFailureUrl()}
target="_blank"
external
>
{i18n.translate('xpack.ingestPipelines.form.onFailureDocumentionLink', {
defaultMessage: 'Learn more',
})}
</EuiLink>
),
}}
/>
<EuiSpacer size="m" />
<EuiSwitch
label={
<FormattedMessage
id="xpack.ingestPipelines.form.onFailureToggleDescription"
defaultMessage="Add failure processors"
/>
}
checked={isOnFailureEditorVisible}
onChange={(e) => setIsOnFailureEditorVisible(e.target.checked)}
data-test-subj="onFailureToggle"
return (
<PipelineProcessorsEditor
onFlyoutOpen={onEditorFlyoutOpen}
learnMoreAboutProcessorsUrl={services.documentation.getProcessorsUrl()}
learnMoreAboutOnFailureProcessorsUrl={services.documentation.getHandlingFailureUrl()}
isTestButtonDisabled={isTestButtonDisabled}
onTestPipelineClick={onTestPipelineClick}
onUpdate={onProcessorsUpdate}
value={{ processors: processorProp, onFailure: onFailureProp }}
/>
</>
}
>
{isOnFailureEditorVisible ? (
<UseField
path="on_failure"
component={JsonEditorField}
componentProps={{
euiCodeEditorProps: {
'data-test-subj': 'onFailureEditor',
height: '300px',
'aria-label': i18n.translate('xpack.ingestPipelines.form.onFailureFieldAriaLabel', {
defaultMessage: 'Failure processors JSON editor',
}),
},
}}
/>
) : (
// <FormRow/> requires children or a field
// For now, we return an empty <div> if the editor is not visible
<div />
)}
</FormRow>
);
}}
</FormDataProvider>
</>
);
};
Loading

0 comments on commit b71edab

Please sign in to comment.