[TSVB] Support custom field format #101245

DianaDerevyankina · 2021-06-03T07:37:11Z

Closes: #97213, Closes: #32318

Summary

This PR implements support of custom field formatting for all TSVB panel types.

Field format was added to Data formatters select as default option.

Cases to test for those who will review this PR:

Labels formatting

How to test label formatting:

In the index pattern management section set field formatting for any field type that can be used for terms grouping.
Create visualization with series grouped by terms with this field.

Expect:

Labels shouldn't be formatted until the series is grouped by terms (i.e. in other cases simply series labels should be displayed);
Labels shouldn't be formatted in string index pattern mode;
Label formatting should be applied to all panel types;
Label formatting should be applied to all field types that terms grouping accepts;
Label formatting should work for runtime and scripted fields too;
Include and Exclude terms functionality should work with raw (not formatted) field names;
Markdown variables values should be formatted, while their keys (labels) shouldn't;
Fields used in annotation template should be formatted too.

Values formatting

How to test values formatting:

In the index pattern management section set field formatting for any aggregatable field type.
Create visualization with series based on aggregation with this field.

Expect:

Value formatting should be applied to all panel types;
Value formatting should be applied to all aggregatable field types;
Value formatting should work for runtime and scripted fields too;
Formatting should not be applied if aggregation is not based on any field;
For string index pattern Data formatters shouldn't include Default option. In case it was selected before and index pattern was switched to a string, Number option should be selected for numeric value, Custom for other value type (this could be checked in Metric with Top Hit aggregation);
Previous case should also be valid for series overridden index pattern;
For kibana index pattern mode each series should have field formatting selected by default, template should be disabled;
If series selected field doesn't have field formatting or string index pattern mode enabled it should have simple number formatting;
Other formatters should work correctly and have template enabled;
If aggregation returns not numeric value (e.g. string, date), Date formatters should have only Default and Custom options for kibana index pattern mode and only Custom for string index pattern. (Currently only Top Hit returns not numeric result, this could be checked in Metric);
For string displayed value and Custom formatter selected, Numeral.js format pattern field shouldn't be displayed, but template could be applied;
If series previously had some numeric formatting and was changed to the metric with a string result, it should have Default option formatting selected for kibana index pattern mode and Custom for string index pattern;
For the series with multiple metrics formatting should work for the visible (last) one;
For multiple series with different formatters Y-Axis should show simple numbers;
For Table, TopN and Metrics values could have Color or URL field formatters;
In TopN and Table Data tab Item url should have bigger priority than URL formatting;
In Table, TopN and Metrics color rules should have bigger priority than color formatting;
In case all series have the same TSVB type and template formatters or the same formatting from index pattern managements section, Y-Axis should format its values accordingly to that formatter.

Checklist

Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how
they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

Add format_label response processor for series vis data and bucket key formatting to process_bucket for table vis data

…r all panel types except markdown

…teCustomFieldFormatter

…on script to set ignore_field_formatting to true for the series

…espect only active metrics field name, refactor vis files and fix label formatting only for grouped by terms series

… and get_table_data, replace getCustomFieldFormatter with createCustomFieldFormatter

DianaDerevyankina · 2021-06-14T09:23:50Z

@elasticmachine merge upstream

DianaDerevyankina · 2021-09-09T08:55:55Z

@elasticmachine merge upstream

stratoula

Code LGTM, I tested it and works fine. I think that it is a great addition for TSVB. Thank you @dziyanadzeraviankina for your effort and for your patience with it :D

I would only like @mbondyra to also do a sanity check in case I missed anything ❤️

VladLasitsa · 2021-09-10T09:29:38Z

LGTM, tested locally in chrome!

dej611

Code review only 👍

Left some nitpick comments.

src/plugins/vis_type_timeseries/public/application/components/data_format_picker.tsx

src/plugins/vis_type_timeseries/public/application/components/aggs/agg.tsx

dej611 · 2021-09-10T09:41:17Z

src/plugins/vis_type_timeseries/public/application/components/vis_types/table/vis.js

+        const isFieldFormatter = column.formatter === DATA_FORMATTERS.DEFAULT;
+        const field = getMetricsField(column.metrics);
+        const formatter = isFieldFormatter
+          ? createFieldFormatter(field, fieldFormatMap, 'html')
+          : createTickFormatter(column.formatter, column.value_template, getConfig);


nitpick: this formula gets repeated over and over in the PR. Maybe it can be isolated in a separated function that returns {isFieldFormatter, formatter}?

updated createFieldFormatter function, could you review that part once again, please?

src/plugins/vis_type_timeseries/public/timeseries_vis_renderer.tsx

src/plugins/vis_type_timeseries/server/plugin.ts

stratoula · 2021-09-10T10:04:35Z

I don't know if this bug is caused due to the new changes or is something that I missed but something goes really wrong here

Same with a number field

…d_formatter

stratoula · 2021-09-10T14:08:32Z

@dziyanadzeraviankina thanx, bug is fixed 🎉 Gauge changes also look great :)

mbondyra · 2021-09-10T15:55:25Z

@elasticmachine merge upstream

cchaos

Thanks for the Sass comments. I didn't do any testing of this, just code-owner review of the Sass file.

mbondyra · 2021-09-13T06:11:54Z

@elasticmachine merge upstream

mbondyra

Works as described, couldn't find anything suspicious. Approved! 👍🏼

kibanamachine · 2021-09-13T08:27:40Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`visTypeTimeseries`	539	544	+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`visTypeTimeseries`	1002.5KB	1017.5KB	+15.0KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`visTypeTimeseries`	26.0KB	26.4KB	+393.0B

Unknown metric groups

References to deprecated APIs

id	before	after	diff
`visTypeTimeseries`	218	224	+6

History

💚 Build #152613 succeeded 16c9938
💚 Build #152271 succeeded aba4a19
💔 Build #152177 failed 77c3b0f
💔 Build #152134 failed 09394d3
💔 Build #152113 failed f6e09b9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @dziyanadzeraviankina

* [TSVB] Support custom field format Add format_label response processor for series vis data and bucket key formatting to process_bucket for table vis data * Add ignore_field_formatting for series to support value formatting for all panel types except markdown * Fix type issue for visData and rename getCustomFieldFormatter to createCustomFieldFormatter * Update vis.test to cover custom field formats logic and add a migration script to set ignore_field_formatting to true for the series * Move createCustomFieldFormatter to a separate file, make formatting respect only active metrics field name, refactor vis files and fix label formatting only for grouped by terms series * Remove services, add getFieldFormatsService to use it in format_label and get_table_data, replace getCustomFieldFormatter with createCustomFieldFormatter * Update plugin.ts * Update start for plugin.ts * Add formatting for annotations and markdown values * Refactor some code * Update some naming and conditions * Fix formatting of data type labels * In process_bucket fix case for no getFieldFormatByName * Add field formatting functional tests for all panel types * Update tests to make them run correctly for firefox * Update _tsvb_markdown test setup * Move series ignoreFieldFormatting check to a separate file, change convertSeriesToVars signature, update migration script and refactor a bit functional tests * Fix type check for timeseries_visualization.tsx * Update migrations.js test expected version to 7.15 * Fix tests in _tsvb_chart.ts * Fix merge conflict remove process_bucket.js * Update process_bucket.test.ts * Fix markdown labels formatting * Add ignore_field_formatting for annotations, enhanced migration script to set that flag to true, refactor data_format_picker * Fix migration script and add disabling for ignore component when string index pattern is used * Add supporting URL and color formatters in tsvb table * Fix eslint * Remove ignore formatting component, add field formatting option to TSVB data format picker and make it default, remove migration script, update tests and refactor some files * Fix failing tests, refactor create_field_formatter and add test to it, update some other files * Fix series formatting for top hit when it has not numeric result * Handle no fieldFormatMap case for table/vis.js * Remove "Default" option form DataFormatPicker when index pattern is string, fix markdown variables issue and refactor some code * Chore(TSVB): Replace aggregations lookup with map * Fix types, update test expected data and remove unused translations * Fix i18 check and useEffect in agg.tsx * Handle aggregations field formatting case * Fix agg_utils, vis and check_if_numeric_metric tests * Correct typo and refactor condition in std_metric * Fix type check * Get rid of IFieldType * Add URL and color formatting for topN and metric tabs, fix setting initial custom formatter and switching formatter in agg.tsx * Update tsvb.asciidoc * Remove link icon from Date format field help text, update click logic for top N in case of custom field format and fix CI * Remove unused import * Revert top N bar extra logic for click * Refactor some code in agg.tsx * Add URL and color formatting to Gauge * Fix bug with terms formatting, refactor some code, update create_field_formatter * Add comments to _gauge.scss * Remove unnecessary await Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Uladzislau Lasitsa <Uladzislau_Lasitsa@epam.com>

…-link-to-kibana-app * 'master' of github.com:elastic/kibana: (120 commits) [TSVB] Support custom field format (elastic#101245) [VisEditors] Add code ownership to the functional tests (elastic#111680) [Lens] Make Lens saved object share-capable (elastic#111403) [Graph] Make Graph saved object share-capable (elastic#111404) [Stack Monitoring] Add breadcrumb support (elastic#111850) Update Jira Cloud to use OAuth2.0 (elastic#111493) Show warning message when attempting to create an APM alert in stack management (elastic#111781) Skip suite blocking ES snapshot promotion (elastic#111907) Respect `auth_provider_hint` if session is not authenticated. (elastic#111521) Added in 'Responses' field in alert telemetry & updated test (elastic#111892) [Usage collection] refactor cloud detector collector (elastic#110439) Make classnames a shared dep (elastic#111636) Fix link to e2e tests in APM testing.md (elastic#111869) [Security Solution] Add host.os.name.caseless mapping and runtime field (elastic#111455) [APM] Removes the beta label from APM tutorial (elastic#111499) (elastic#111828) [RAC] [Observability] Expand Observability alerts page functional tests (elastic#111297) Fix extra white space on the alert table whe page size is 50 or 100 (elastic#111568) [Metrics UI] Add Inventory Timeline open/close state to context and URL state (elastic#111034) [Graph] Switch to SavedObjectClient.resolve (elastic#109617) [APM] Adding lambda icon (elastic#111834) ... # Conflicts: # x-pack/plugins/reporting/public/management/__snapshots__/report_listing.test.tsx.snap

* [TSVB] Support custom field format Add format_label response processor for series vis data and bucket key formatting to process_bucket for table vis data * Add ignore_field_formatting for series to support value formatting for all panel types except markdown * Fix type issue for visData and rename getCustomFieldFormatter to createCustomFieldFormatter * Update vis.test to cover custom field formats logic and add a migration script to set ignore_field_formatting to true for the series * Move createCustomFieldFormatter to a separate file, make formatting respect only active metrics field name, refactor vis files and fix label formatting only for grouped by terms series * Remove services, add getFieldFormatsService to use it in format_label and get_table_data, replace getCustomFieldFormatter with createCustomFieldFormatter * Update plugin.ts * Update start for plugin.ts * Add formatting for annotations and markdown values * Refactor some code * Update some naming and conditions * Fix formatting of data type labels * In process_bucket fix case for no getFieldFormatByName * Add field formatting functional tests for all panel types * Update tests to make them run correctly for firefox * Update _tsvb_markdown test setup * Move series ignoreFieldFormatting check to a separate file, change convertSeriesToVars signature, update migration script and refactor a bit functional tests * Fix type check for timeseries_visualization.tsx * Update migrations.js test expected version to 7.15 * Fix tests in _tsvb_chart.ts * Fix merge conflict remove process_bucket.js * Update process_bucket.test.ts * Fix markdown labels formatting * Add ignore_field_formatting for annotations, enhanced migration script to set that flag to true, refactor data_format_picker * Fix migration script and add disabling for ignore component when string index pattern is used * Add supporting URL and color formatters in tsvb table * Fix eslint * Remove ignore formatting component, add field formatting option to TSVB data format picker and make it default, remove migration script, update tests and refactor some files * Fix failing tests, refactor create_field_formatter and add test to it, update some other files * Fix series formatting for top hit when it has not numeric result * Handle no fieldFormatMap case for table/vis.js * Remove "Default" option form DataFormatPicker when index pattern is string, fix markdown variables issue and refactor some code * Chore(TSVB): Replace aggregations lookup with map * Fix types, update test expected data and remove unused translations * Fix i18 check and useEffect in agg.tsx * Handle aggregations field formatting case * Fix agg_utils, vis and check_if_numeric_metric tests * Correct typo and refactor condition in std_metric * Fix type check * Get rid of IFieldType * Add URL and color formatting for topN and metric tabs, fix setting initial custom formatter and switching formatter in agg.tsx * Update tsvb.asciidoc * Remove link icon from Date format field help text, update click logic for top N in case of custom field format and fix CI * Remove unused import * Revert top N bar extra logic for click * Refactor some code in agg.tsx * Add URL and color formatting to Gauge * Fix bug with terms formatting, refactor some code, update create_field_formatter * Add comments to _gauge.scss * Remove unnecessary await Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Uladzislau Lasitsa <Uladzislau_Lasitsa@epam.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Uladzislau Lasitsa <Uladzislau_Lasitsa@epam.com>

DianaDerevyankina added 3 commits May 27, 2021 14:15

[TSVB] Support custom field format

5a3025c

Add format_label response processor for series vis data and bucket key formatting to process_bucket for table vis data

Merge branch 'master' into issues/97213

f8e27db

Add ignore_field_formatting for series to support value formatting fo…

3eee90a

…r all panel types except markdown

DianaDerevyankina requested a review from alexwizp June 3, 2021 07:37

DianaDerevyankina self-assigned this Jun 3, 2021

DianaDerevyankina added 12 commits June 3, 2021 11:34

Fix type issue for visData and rename getCustomFieldFormatter to crea…

2c7e8e2

…teCustomFieldFormatter

Update vis.test to cover custom field formats logic and add a migrati…

cc1999e

…on script to set ignore_field_formatting to true for the series

Merge branch 'master' into issues/97213

71494d1

Move createCustomFieldFormatter to a separate file, make formatting r…

84fddd2

…espect only active metrics field name, refactor vis files and fix label formatting only for grouped by terms series

Merge branch 'master' into issues/97213

9ec68ae

Remove services, add getFieldFormatsService to use it in format_label…

85d3a92

… and get_table_data, replace getCustomFieldFormatter with createCustomFieldFormatter

Update plugin.ts

e7107be

Update start for plugin.ts

23da97b

Add formatting for annotations and markdown values

9d42fef

Refactor some code

e800a9e

Merge branch 'master' into issues/97213

8edfb3c

Merge branch 'master' into issues/97213

5317e01

DianaDerevyankina requested a review from VladLasitsa June 11, 2021 11:14

DianaDerevyankina added 4 commits June 11, 2021 18:55

Update some naming and conditions

5046d27

Fix formatting of data type labels

17918cf

Merge branch 'master' into issues/97213

60b6f2e

In process_bucket fix case for no getFieldFormatByName

b984653

kibanamachine and others added 5 commits June 14, 2021 05:23

Merge branch 'master' into issues/97213

56fdcb1

Merge branch 'master' into issues/97213

995adca

Add field formatting functional tests for all panel types

4ab0509

Merge branch 'master' into issues/97213

0ab3ef6

Update tests to make them run correctly for firefox

75ceb1a

DianaDerevyankina added Feature:TSVB TSVB (Time Series Visual Builder) release_note:enhancement labels Jun 15, 2021

DianaDerevyankina added 2 commits September 9, 2021 09:59

Merge branch 'master' into issues/97213

f6e09b9

Revert top N bar extra logic for click

114ea94

Merge branch 'master' into issues/97213

09394d3

stratoula approved these changes Sep 9, 2021

View reviewed changes

DianaDerevyankina added 2 commits September 9, 2021 14:38

Refactor some code in agg.tsx

77c3b0f

Add URL and color formatting to Gauge

0c1d7bf

DianaDerevyankina requested a review from a team as a code owner September 9, 2021 15:09

Merge branch 'master' into issues/97213

aba4a19

dej611 approved these changes Sep 10, 2021

View reviewed changes

Fix bug with terms formatting, refactor some code, update create_fiel…

287ec81

…d_formatter

Add comments to _gauge.scss

38b7bc4

Merge branch 'master' into issues/97213

870a54b

cchaos approved these changes Sep 10, 2021

View reviewed changes

Remove unnecessary await

16c9938

Merge branch 'master' into issues/97213

d3483e1

mbondyra approved these changes Sep 13, 2021

View reviewed changes

DianaDerevyankina merged commit 33cfc41 into elastic:master Sep 13, 2021

DianaDerevyankina mentioned this pull request Sep 13, 2021

[7.x] [TSVB] Support custom field format (#101245) #111928

Merged

KOTungseth mentioned this pull request Nov 8, 2021

[DOCS] Adds the 7.16 TSVB docs #117931

Merged

iget-master mentioned this pull request Dec 12, 2021

[TSVB] Support custom field format opensearch-project/OpenSearch-Dashboards#1017

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TSVB] Support custom field format #101245

[TSVB] Support custom field format #101245

DianaDerevyankina commented Jun 3, 2021 •

edited

Loading

DianaDerevyankina commented Jun 14, 2021

DianaDerevyankina commented Sep 9, 2021

stratoula left a comment

VladLasitsa commented Sep 10, 2021

dej611 left a comment

dej611 Sep 10, 2021

DianaDerevyankina Sep 10, 2021

stratoula commented Sep 10, 2021 •

edited

Loading

stratoula commented Sep 10, 2021

mbondyra commented Sep 10, 2021

cchaos left a comment

mbondyra commented Sep 13, 2021

mbondyra left a comment

kibanamachine commented Sep 13, 2021

References to deprecated APIs

[TSVB] Support custom field format #101245

[TSVB] Support custom field format #101245

Conversation

DianaDerevyankina commented Jun 3, 2021 • edited Loading

Summary

This PR implements support of custom field formatting for all TSVB panel types.

Cases to test for those who will review this PR:

Labels formatting

Values formatting

Checklist

Risk Matrix

For maintainers

DianaDerevyankina commented Jun 14, 2021

DianaDerevyankina commented Sep 9, 2021

stratoula left a comment

Choose a reason for hiding this comment

VladLasitsa commented Sep 10, 2021

dej611 left a comment

Choose a reason for hiding this comment

dej611 Sep 10, 2021

Choose a reason for hiding this comment

DianaDerevyankina Sep 10, 2021

Choose a reason for hiding this comment

stratoula commented Sep 10, 2021 • edited Loading

stratoula commented Sep 10, 2021

mbondyra commented Sep 10, 2021

cchaos left a comment

Choose a reason for hiding this comment

mbondyra commented Sep 13, 2021

mbondyra left a comment

Choose a reason for hiding this comment

kibanamachine commented Sep 13, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

References to deprecated APIs

History

DianaDerevyankina commented Jun 3, 2021 •

edited

Loading

stratoula commented Sep 10, 2021 •

edited

Loading