Add support for runtime field types to mappings editor. #77420

cjcenizal · 2020-09-14T22:49:24Z

To test, add a runtime field to an index template's mappings via the UI or via Console like so:

PUT /my-index/_mappings
{
    "properties" : {
        "day_of_week" : {
            "type" : "runtime",
            "runtime_type" : "keyword",
            "script" : {
                "source" : "emitValue(doc['timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
            }
        }
    }
}

Release note

Index templates can now be configured with runtime fields in their mappings.

- Add tests for getTypeLabelFromField util.

elasticmachine · 2020-09-14T22:49:26Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

cjcenizal · 2020-09-14T22:49:53Z

...management/public/application/components/mappings_editor/constants/data_types_definition.tsx

+      defaultMessage: 'Runtime',
+    }),
+    documentation: {
+      main: '/runtime_field.html',


Note that these docs don't exist yet.

Should we hold off on adding a value then? Or are we confident the doc url will eventually be /runtime_field.html?

alisonelizabeth

Thanks for working on this @cjcenizal!

I left a few comments in the code.

I'm also wondering about the general workflow. For other field types that have required fields (such as alias and token_count), we require the required fields before adding it.

For example:

I think we probably want to do the same for runtime field types. As it is, if I add a runtime field type via the UI first, I see this:

I'm also able to proceed to the review step, but end up getting an error back from ES since runtime_type and script are both required.

For the runtime_type parameter, I think we could possibly do something similar for numeric and range types, and have a subtype dropdown.

For example:

Let me know what you think!

alisonelizabeth · 2020-09-15T14:30:25Z

...on/components/mappings_editor/components/document_fields/fields/field_types/runtime_type.tsx

+
+export const RuntimeType = () => {
+  return (
+    <>


nit: unnecessary fragment

alisonelizabeth · 2020-09-15T14:50:34Z

...nents/mappings_editor/components/document_fields/field_parameters/runtime_type_parameter.tsx

+                )}
+                singleSelection={{ asPlainText: true }}
+                options={RUNTIME_FIELD_OPTIONS}
+                selectedOptions={value as ComboBoxOption[]}


It's probably more appropriate to use the EuiComboBoxOptionOption type now (and at some point remove ComboBoxOption).

alisonelizabeth · 2020-09-15T14:54:08Z

...management/public/application/components/mappings_editor/constants/data_types_definition.tsx

+      defaultMessage: 'Runtime',
+    }),
+    documentation: {
+      main: '/runtime_field.html',


Should we hold off on adding a value then? Or are we confident the doc url will eventually be /runtime_field.html?

alisonelizabeth · 2020-09-15T14:59:46Z

x-pack/plugins/index_management/public/application/components/mappings_editor/lib/utils.ts

+
+export const getTypeLabelFromField = (field: Field) => {
+  // eslint-disable-next-line @typescript-eslint/naming-convention
+  const { type, runtime_type } = field;


Can we just camelcase here so we don't need the eslint-disable?

const { type, runtime_type: runtimeType } = field;

alisonelizabeth · 2020-09-15T15:04:00Z

x-pack/plugins/index_management/public/application/index.tsx

-          </MappingsEditorProvider>
-        </AppContextProvider>
-      </Provider>
+      <KibanaReactContextProvider>


I might be missing something here, but is this necessary? I don't see it being used anywhere.

The CodeEditor component used for editing the painless script requires it to provide uiSettings, which is really unclear from the code. I'll add a comment.

Ah. Yeah, it isn't very clear, but I see that now after looking at the code_editor code. Thanks for adding the comment!

cjcenizal · 2020-09-15T23:20:59Z

Thanks for the great review @alisonelizabeth! I incorporated your suggestions, and added a screenshot to the PR description to show how the UI now requires you to define the type before adding a runtime field. Could you please review again?

alisonelizabeth

Thanks for making the changes @cjcenizal! This is looking great.

script also appears to be a required field. I understand the editor takes up some real estate, but I think it's important to make the user provide a value here before being allowed to submit the form. WDYT?

Also, I had a question about the copy for keyword and ip runtime types.

Is "...use the text data type" and "...use the IP range data type" still relevant in the context of runtime fields?

alisonelizabeth · 2020-09-16T14:15:53Z

x-pack/plugins/index_management/public/application/index.tsx

-          </MappingsEditorProvider>
-        </AppContextProvider>
-      </Provider>
+      <KibanaReactContextProvider>


Ah. Yeah, it isn't very clear, but I see that now after looking at the code_editor code. Thanks for adding the comment!

…sn't defined.

cjcenizal · 2020-09-21T22:48:24Z

script also appears to be a required field. I understand the editor takes up some real estate, but I think it's important to make the user provide a value here before being allowed to submit the form. WDYT?

Whoops, good catch! Thanks for calling this out. I added validation. I had to surface the error message as a callout in addition to an error text, since the code editor is tall enough that it pushes the error text out of view on shorter screens. I think we need to solve this in the code editor component, e.g. by making it resize vertically to accommodate its content.

I also update a couple other places where we were using callouts similarly, to follow the patterns shown in the EUI examples.

Is "...use the text data type" and "...use the IP range data type" still relevant in the context of runtime fields?

This is a great question. It's difficult for me to answer because I have a hard time understanding the user's intention when defining runtime fields in the index template. Ostensibly, if the user is defining or editing an index template, they have some level of control over the ingestion system, which includes the source of the documents being ingested and the ingest node pipeline used to process them. If this is the case, then they might be making decisions about how best to store, say, an IP. Maybe they're still trying to figure out the right design for their ingestion system, and maybe this text helps them realize they actually do want an IP range instead of an IP? Or maybe they started with the intention to define an IP range, but they didn't realize it's not yet supported by runtime fields? 🤷 I guess I'm not entirely sure if it's helpful, but I don't see how it hurts. Do you find it confusing?

cjcenizal · 2020-09-21T22:50:48Z

@elasticmachine merge upstream

alisonelizabeth

Thanks for making the changes @cjcenizal. Great work!

I still wonder if the script code editor should be present when initially adding the field (since it's required), but I'll leave that up to you.

(in this view:)

This is a great question. It's difficult for me to answer because I have a hard time understanding the user's intention when defining runtime fields in the index template.

👍 I think it is fine to leave as-is. I was also wondering what the use case would look like.

cjcenizal · 2020-09-22T16:10:05Z

@alisonelizabeth Good point. I was originally on the fence about adding the code editor to the "add field" component because I thought its height would disrupt the UX by causing content below it to jump around. But I think it's more important to be consistent with required parameters, so I added it in. I made the editor shorter, which I think helps (and also allowed me to remove the callout I had put in). I doubt these scripts will be so involved that they require as much height as I originally provided.

…tor. Remove error callout.

sebelga · 2020-09-24T11:18:44Z

@cjcenizal Can you explain why you are not using 2 columns for the required fields? I am surprised to see a huge select occupying the whole width. I understand in smaller screens those 2 will stack on top of another but on desktop?

Or maybe have a described group with the "Emited type" on the left and the select on the right. Then the same for the script.

cjcenizal · 2020-09-25T05:57:09Z

Great idea, @sebelga! Done.

sebelga · 2020-09-25T06:32:11Z

Awesome thanks @cjcenizal !

I haven't played with it yet but maybe in the future for those scenarios, it would be nice to make use of the Resize observer and CSS (or maybe set the stack var in your case) to change the layout depending of the width of the parent.

This would allow us to create components with a layout that adapts to wherever they are thrown in.

kibanamachine · 2020-09-25T07:29:02Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 1b01e7e

Metrics [docs]

@kbn/optimizer bundle module count

id	value	diff	baseline
indexManagement	508	+5	503

async chunks size

id	value	diff	baseline
indexManagement	1.6MB	+8.6KB	1.6MB

page load bundle size

id	value	diff	baseline
indexManagement	122.8KB	+158.0B	122.6KB

History

💔 Build #76239 failed 3ac63a6
💔 Build #76232 failed 00573c153579d9cbe65509e1ab297ee9df7a8f79
💔 Build #76039 failed 8a53f68
💔 Build #76036 failed 9af93bc
💚 Build #74653 succeeded ef9e325

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

) * Add support for runtime field types to mappings editor. * Add tests for getTypeLabelFromField util. * Refine callout appearance in term vector and alias parameters.

@timestamp

* master: (226 commits) [Enterprise Search] Added Logic for the Credentials View (elastic#77626) [CSM] Js errors (elastic#77919) Add the @kbn/apm-config-loader package (elastic#77855) [Security Solution] Refactor useSelector (elastic#75297) Implement tagcloud renderer (elastic#77910) [APM] Alerting: Add global option to create all alert types (elastic#78151) [Ingest pipelines] Upload indexed document to test a pipeline (elastic#77939) TypeScript cleanup in visualizations plugin (elastic#78428) Lazy load metric & mardown visualizations (elastic#78391) [Detections][EQL] EQL rule execution in detection engine (elastic#77419) Update tutorial-full-experience.asciidoc (elastic#75836) Update tutorial-define-index.asciidoc (elastic#75754) Add support for runtime field types to mappings editor. (elastic#77420) [Monitoring] Usage collection (elastic#75878) [Docs][Actions] Add docs for Jira and IBM Resilient (elastic#78316) [Security Solution][Resolver] Update @timestamp formatting (elastic#78166) [Security Solution] Fix app layout (elastic#76668) [Security Solution][Resolver] 2 new functions to DAL (elastic#78477) Adds new elasticsearch client to telemetry plugin (elastic#78046) skip flaky suite (elastic#78512) (elastic#78511) (elastic#78510) (elastic#78509) (elastic#78508) (elastic#78507) (elastic#78506) (elastic#78505) (elastic#78504) (elastic#78503) (elastic#78502) (elastic#78501) (elastic#78500) ...

…tic#77420)" This reverts commit 0dc89cb.

…)" (#79611) This reverts commit 0dc89cb.

…) (#78539)" (#79612) This reverts commit 52b1ad9.

…r. (elastic#77420)" (elastic#79611)" This reverts commit e01d538.

…into feature/task_manager_429 * 'feature/task_manager_429' of github.com:elastic/kibana: (158 commits) Add license check to direct package upload handler. (#79653) [Ingest Manager] Rename API /api/ingest_manager => /api/fleet (#79193) [Security Solution][Resolver] Simplify CopyableField styling and add comments (#79594) Fine-tunes ML related text on Metrics UI (#79425) [ML] DF Analytics creation wizard: ensure job creation possible when model memory lower than estimate (#79229) Add new "Add Data" tutorials (#77237) Update APM telemetry docs (#79583) Revert "Add support for runtime field types to mappings editor. (#77420)" (#79611) Kibana request headers (#79218) ensure missing indexPattern error is bubbled up to error callout (#79378) Missing space fix (#79585) remove duplicate tab states (#79501) [data.ui] Lazy load UI components in data plugin. (#78889) Add generic type params to search dependency. (#79608) [Ingest Manager] Internal action for policy reassign (#78493) [ILM] Add index_codec to forcemerge action in hot and warm phases (#78175) [Ingest Manager] Update open API spec and add condition to agent upgrade endpoint (#79579) [ML] Hide Data Grid column options when histogram charts are enabled. (#79459) [Telemetry] Synchronous `setup` and `start` methods (#79457) [Observability] Persist time range across apps (#79258) ...

…r. (#77420)" (#79611)" (#79940) This reverts commit e01d538. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…r. (#77420) (#78539)" (#79612)" (#79942) This reverts commit e3270da. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Add support for runtime field types to mappings editor.

02e43dd

- Add tests for getTypeLabelFromField util.

cjcenizal added release_note:enhancement Feature:Index Management Index and index templates UI v8.0.0 Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more Feature:Mappings Editor Index mappings editor UI v7.10.0 labels Sep 14, 2020

cjcenizal requested a review from alisonelizabeth September 14, 2020 22:49

cjcenizal requested a review from a team as a code owner September 14, 2020 22:49

cjcenizal commented Sep 14, 2020

View reviewed changes

Ignore TS error.

1e7b0fc

alisonelizabeth reviewed Sep 15, 2020

View reviewed changes

Require runtime type to be defined before adding the field to the tree.

ef9e325

alisonelizabeth reviewed Sep 16, 2020

View reviewed changes

cjcenizal added 4 commits September 17, 2020 11:19

Merge branch 'master' into feature/index-templates-runtime-fields

a45b918

Merge branch 'master' into feature/index-templates-runtime-fields

3f24754

Make script a required parameter and surface validation error if it i…

a5214ac

…sn't defined.

Refine callout appearance in term vector and alias parameters.

90e1445

cjcenizal force-pushed the feature/index-templates-runtime-fields branch from 7ef7643 to 90e1445 Compare September 21, 2020 22:37

Merge branch 'master' into feature/index-templates-runtime-fields

9af93bc

Merge branch 'master' into feature/index-templates-runtime-fields

8a53f68

alisonelizabeth approved these changes Sep 22, 2020

View reviewed changes

Require script when first adding a field. Shorten painless script edi…

3ac63a6

…tor. Remove error callout.

cjcenizal force-pushed the feature/index-templates-runtime-fields branch from 00573c1 to 3ac63a6 Compare September 22, 2020 16:24

cjcenizal added 3 commits September 24, 2020 14:52

Merge branch 'master' into feature/index-templates-runtime-fields

4449800

Improve layout of create and edit forms.

ca02afb

Add beta badge.

1b01e7e

cjcenizal merged commit 0dc89cb into elastic:master Sep 25, 2020

cjcenizal deleted the feature/index-templates-runtime-fields branch September 25, 2020 15:25

cjcenizal mentioned this pull request Sep 25, 2020

[7.x] Add support for runtime field types to mappings editor. (#77420) #78539

Merged

lockewritesdocs mentioned this pull request Sep 28, 2020

Update runtime fields URL #78609

Closed

cjcenizal added a commit to cjcenizal/kibana that referenced this pull request Oct 6, 2020

Revert "Add support for runtime field types to mappings editor. (elas…

a05cee7

…tic#77420)" This reverts commit 0dc89cb.

cjcenizal mentioned this pull request Oct 6, 2020

Revert "Add support for runtime field types to mappings editor" #79611

Merged

cjcenizal added reverted and removed release_note:enhancement labels Oct 6, 2020

cjcenizal added a commit that referenced this pull request Oct 6, 2020

Revert "Add support for runtime field types to mappings editor. (#77420…

e01d538

…)" (#79611) This reverts commit 0dc89cb.

cjcenizal added a commit that referenced this pull request Oct 6, 2020

Revert "Add support for runtime field types to mappings editor. (#77420…

e3270da

…) (#78539)" (#79612) This reverts commit 52b1ad9.

cjcenizal added a commit to cjcenizal/kibana that referenced this pull request Oct 7, 2020

Revert "Revert "Add support for runtime field types to mappings edito…

02b7432

…r. (elastic#77420)" (elastic#79611)" This reverts commit e01d538.

cjcenizal mentioned this pull request Oct 7, 2020

Reintroduce "Add support for runtime field types to mappings editor" #79940

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for runtime field types to mappings editor. #77420

Add support for runtime field types to mappings editor. #77420

cjcenizal commented Sep 14, 2020 •

edited

Loading

elasticmachine commented Sep 14, 2020

cjcenizal Sep 14, 2020

alisonelizabeth Sep 15, 2020

alisonelizabeth left a comment •

edited

Loading

alisonelizabeth Sep 15, 2020

alisonelizabeth Sep 15, 2020

alisonelizabeth Sep 15, 2020

alisonelizabeth Sep 15, 2020

alisonelizabeth Sep 15, 2020

cjcenizal Sep 15, 2020

alisonelizabeth Sep 16, 2020

cjcenizal commented Sep 15, 2020

alisonelizabeth left a comment

alisonelizabeth Sep 16, 2020

cjcenizal commented Sep 21, 2020

cjcenizal commented Sep 21, 2020

alisonelizabeth left a comment

cjcenizal commented Sep 22, 2020

sebelga commented Sep 24, 2020

cjcenizal commented Sep 25, 2020

sebelga commented Sep 25, 2020

kibanamachine commented Sep 25, 2020

Add support for runtime field types to mappings editor. #77420

Add support for runtime field types to mappings editor. #77420

Conversation

cjcenizal commented Sep 14, 2020 • edited Loading

Release note

elasticmachine commented Sep 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alisonelizabeth left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjcenizal commented Sep 15, 2020

alisonelizabeth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjcenizal commented Sep 21, 2020

cjcenizal commented Sep 21, 2020

alisonelizabeth left a comment

Choose a reason for hiding this comment

cjcenizal commented Sep 22, 2020

sebelga commented Sep 24, 2020

cjcenizal commented Sep 25, 2020

sebelga commented Sep 25, 2020

kibanamachine commented Sep 25, 2020

💚 Build Succeeded

Metrics [docs]

@kbn/optimizer bundle module count

async chunks size

page load bundle size

History

cjcenizal commented Sep 14, 2020 •

edited

Loading

alisonelizabeth left a comment •

edited

Loading