Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.x] [Ingest Node Pipelines] Pipeline Processors Editor (#66021) #69273

Merged
merged 1 commit into from
Jun 16, 2020

Conversation

jloleysens
Copy link
Contributor

Backports the following commits to 7.x:

* initial plugin setup

* add smoke test

* fix license check

* refactor plugin setup

* Server-side create/update ingest pipelines (elastic#62744)

* List pipelines (elastic#62785)

* First iteration of ingest table

* Add action placeholders

* Refactor list and address PR feedback

Refactored the list into smaller pieces and assemble in main.tsx

Also addressed feedback on copy, removed unused notifications dep

* WiP on flyout

Showing name in title

* Add reload button

* Finish first version of flyout

* Slight update to copy

* `delete` -> `edit`

* Address PR feedback

Copy and a11y updates

* Add on failure JSON to flyout if it is available

* Add details json block file and remove ununsed import

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* [Ingest pipelines] Create pipeline UI (elastic#63017)

* First vertical slice of pipeline editor component

* Made a space for common parameters

* [Ingest pipelines] Edit pipeline page (elastic#63522)

* First iteration of CRUD functionality working

* WiP on moving the pipeline editor to pipeline processor editor

* Finish refactor to work with passing state out

* Refactor and fix tests

* [Ingest pipelines] Polish details panel and empty list (elastic#63926)

* Address some early feedback and use FormDataProvider

FormDataProvider gives a more declarative way to listen to form
state updates.

Also refactored the state reader mechanism.

* [Ingest pipelines] Delete pipeline (elastic#63635)

* [Ingest Node Pipelines] Clone Pipeline (elastic#64049)

* First iteration of clone functionality

Wired up for both the list table and the details flyout in the
list section.

* satisfy eslint

* Turn on sorting for the list table

* Clean up const declarations

* Address PR feedback

Sentence-casify and update some other copy.

* Mark edit and delete as primary actions in list table

* Handle URI encoded chars in pipeline name when cloning

* Update to using the more flexible controlled component pattern

To make this component and mappings editor more consistent, both
will expose a more traditional controlled component interface to
consumers.

In the current implementation this requires that consumers use
`useCallback` for `onUpdate` handlers to avoid an infinite
rendering cycle and to avoid staleness bugs in their `onUpdate`
handlers they need to think about what might make the handler
stale.

This approach comes with the benefits and flexibility of
controlled components at the cost of slightly more complex
consumption of the components.

In future, we can explore adding the uncontrolled component
interface too so that consumers have the option to more simply
render and pull data out only when needed

* Handle sub-form validity

The pipelines processor editor not emits overall validity to
consumers

* Fix Jest test

* Refactor some names

prepareDataOut -> serialize
prepareDataIn -> deserialize
EditorProcessor -> ProcessorInternal

* Mark as private

* Major WiP

Started working on the drag-and-drop-tree and updated some
typings stuff

* [Ingest node pipelines] Privileges (elastic#63850)

* Create privileges check for ingest pipelines app

Also moved the public side logic for checking and rendering
privilege related messages to es_ui_shared/public following the
new __packages_do_not_import__ convention.

* Add ,

* Fix import paths

* Address PR feedback

Fix i18n strings (remove reference to snapshot and restore) and
fix copy referencing snapshot and restore - all copy-pasta errors.

Also remove unused field from missing privileges object.

* Fix issue from resolving merge conflicts

* Add missing app privilege

* Use non-deprecated privilige name

* First iteration of drag and drop tree on feature parity

* First steps toward add on failure handler

Updated reducer logic to create the next copy value using immer.

* First iteration of nested tree with add on failure working

* Refactor and some UI layout updates

- Remove the "id" field on processors for now
- Implement the nested remove and update functionality again

* Remove immer (not call stack safe)

Refac to remove immer and reimplemented the immutable set and
get functions.

* [Ingest Node Pipelines] More lenient treatment of on-failure value (elastic#64411)

* Move file to components folder

* [Ingest pipelines] Simulate pipeline (elastic#64223)

* Updated tree rendering

- turn off dropzones for children
- fixed up some padding and margins
- fixed integration with pipeline_form.tsx

The current implementation still has a lot of jank stemming from
the UX with DnD. Unfortunately the nesting has opened a number
of issues here.

* [Ingest Node Pipelines] Show flyout after editing or creating a pipeline (elastic#64409)

* Show flyout after editing or creating a pipeline

* JSX comment

* Show not found flyout

Copied from CCR

* update not found flyout and fix behavior when viewing details from table

* Reset pipeline name in URI when closing flyout

* Remove encodeURI

Already using encodingURIComponent for unsafe string.

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* Clarification of terms

- addProcessor -> addTopLevelProcessor (same in editor modes)
- Expanded comment on ProcessorSelector

* Implement move between lists functionality

Added tests to the reducer for moving in and out of a recusrsive
structure.

* fix TS

* Prevent nesting a parent inside of its own child

* Add comment

* [Ingest pipelines] Cleanup (elastic#64794)

* address review feedback

* remove unused import

* Big refactor to tree rendering structure

DnD tree now converts the nested tree to a flat one and only
consists of _1_ droppable area with a flat array of draggables
that can be combined.

Using the existing logic in the reducer combined with translating
the flat structure changes to a format the nested reducer can
understand looks like a really promising avenue.

There still seems to be a bug with a longer list where items do
not interact properly.

* Remove unused component

* A number of NB changes

- Fixed a subtle serialisation bug with on_failure in options
- Fixed a performance bug by using useRef in pipeline form
- Memoized the drag and drop tree
- Removed use of "isValid" from form lib... I think this should
be removed entirely.

* fix bad conflict resolution

* Implemented a slightly better destination resolution algo.

Also added tests.

* Fix subtle staleness bug, whitelist keys for setValue

* NB styling fix!!

Due to a parent's setting of overflow: hidden the drag and drop
tree had a dead zone that would equal the page overflow limit,
unsetting on the component itself (overflow: auto) relaxes this
limitation.

* Fix stale delete bug too

* Update naming of editor modes and update comments

* Use field types combo box

* Add delete confirmation modal

* Refactor delete modal component file name

* Better visual integration with existing form

* Update layout and styling of form

- added some padding around the new processors editor field
- Updated use of flex box in the pipeline form fields

* Move pipeline processor copy into pipeline processor component

The test button is also now inside of the pipeline processor
editor. Eventually all of this functionality should be moved
into the pipeline processor editor.

* First step of refactor to moving between trees

* First iteration of x-tree drag and drop

* Remove unused import

* Fix jest test types

* Fix up minor i18n issues and fix up layout of on failure

* Remove unnecessary prop

* Update spacing above add processor button to make it more center

* Fix destination resolution algo

* Update dragging resolver unit tests and add a lot more comments

* Use one sorting algo (removed use of euiReorder for now)

* Add placeholder tests and update comments

* Quite a big refactor

- Remove DraggableLocation entirely, only use ProcessorSelector
this simplifies mapping back to reducer instructions quite a lot
- Add tests for special case dragging behaviour

* Fix off by one bug in tests and implementation 🤦🏼‍♂

- Also move processor reducer to it's own folder

* Update behaviour for dragging up across trees and add tests

* Fix combine instruction

* Fix test and i18n issues

* Remove background color

* Fix selector after selector refactor

* A major performance

- Do not re-render the entire tree when editing settings

* Fix component smoke test

* Fix reading value from processor state using processor selector

* [Ingest pipelines] Custom processor form (elastic#66022)

* Re add background color and refactor name of processor item

* Fix file naming and refactor 🚜 dnd tree rendering

- deserialze and serialize were backwards, fixed
- the dnd tree was rendering a flat tree which needed to be
mapped back to the nested tree for updates. This is still the
case but we do use recursive rendering. This enables tree nodes
to better hold local state like whether they are collapsed or not.

* Fix getting of initial on failure processors value

* Update padding styles for containers

* A lot of styling updates to get closer to look of mockup

* A WiP version of th click-tree an alternative to dnd

As a response to really long pipelines we may pivot away from
dnd.

* Remove dnd tree

* clean up reamining dnd tree references

* Clean up and refactor of tree component

To simplify the logic of the tree component processor id's
were added. Also had to update the jest spec to support this

* Added the ability to duplicate a processor

* Fix types in test

* Added duplicate functionality to ui

* Memoize tree components

* address es lint issues

* remove unused import

* Fix editing of custom json

* Address form performance issues

* Add all known missing processors

* Add ability to cancel move

* Fix staleness in test and view request flyouts

* fix type issue

* Remove unused translations and skip funcitonal test for now

* add todo comment

* Fix type issues

* remove poc styles

* disable other move buttons if we have a selected processor

* Refactor drop zone pin to button and add some styling

* Refactor processor editor item

* Update styling and use icon for cancel move action too

* fix nasty integration bug

* some minor optimizations

* prevent parent from being placed in own on failure handler

* Re-add cancel button

* Re-introduce failure processors toggle

* Fix typo

* Add Handler types for processor editor item

* Fix staleness bug for type, refactor classname and fix duplicate
bug not immutably copying values

* Experimenting with padels (revert this to undo if no further
changes have been made)

* Add description and unique ids

* Share links via component-wide context rather than props

* Virtualized list and back to outline dropzones

* Refactor id getter to a service and make it an incrementing number

* Temporary fix for double rendering issue.

This means the pipeline processors component is not controllable
but fixes the double rendering even when processors have not
changed. This is a problem coming from the ingest pipelines
plugin form system outside of this component

* add todo comment

* remove euicode element

* properly handle duplicate flow

* attempt to fix i18n

* split private_tree into it's own component and add comments

* refactor 🚜. rename Tree to ProcessorsTree and move things around

* do not delete the top level arrays for processors and onfailures

* fix typescript error

* Move duplicate, addOnFailure and delete actions into ctx menu

* remove unused import

* add support for pressing esc key to cancel move

* Add outside click listener

* always prompt before deleting a processor

* refactor remove distinction between adding top level and on fail

* add processor button to tree

* Hide the add on failure context menu item for processors with
failure handlers

* Reinstated x-tree moving and highlight and disable for buttons
on move and on edit

* removing ids step 1: remove idGenerator

* remove ids step 2: added inline text input

Also refactored a lot of the tree actions away. Now using
context and the processors editor tree item component for
dispatching actions. The tree item has access to the dispatch
and to the selector which makes it a well positioned component.

Also saves on some props drilling.

* Slight improvement to styling of text input (border)

* Re-implement missing failure toggle test

* address type todo

* Address many type issues and fix yarn.lock

* re-enable create pipeline functional test

* prevent multiple flyouts from opening

* change flyout title when editing an on-failure processor

* absolutely position the failure handlers label

when we render a dropzone on the label, then we move the label
up without affecting overall component height

* fix description behaviour not removing tag if empty

* some minor clean up

* add onflyoutopen cb to tests

* refactor processors editor item to multiple files

also refactored i18n into it's own file. would be good to come
up with a pttaern for doing this more broadly.

* fix add on-failure handler in context menu after refactor

* tag -> new description field

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alison.goryachev@elastic.co>
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

id value diff baseline
ingestPipelines 469 +222 247

page load asset size

beta
id value diff baseline
/bundles/plugin/ingestPipelines/ingestPipelines.plugin.js 300.0KB +698.0B 299.4KB

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jloleysens jloleysens merged commit 927bf48 into elastic:7.x Jun 16, 2020
@jloleysens jloleysens deleted the backport/7.x/pr-66021 branch June 16, 2020 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants