Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ESQL] Add chart switch #177790

Merged
merged 13 commits into from
Mar 8, 2024
Merged

[ESQL] Add chart switch #177790

merged 13 commits into from
Mar 8, 2024

Conversation

mbondyra
Copy link
Contributor

@mbondyra mbondyra commented Feb 26, 2024

Summary

Solves the first part of #173222

Adds chart switcher to ESQL inline Lens editor only (only on Dashbaord):
Screenshot 2024-02-26 at 13 19 41
Screenshot 2024-02-26 at 13 19 46

Very slight design (paddings) changes

Screenshot 2024-02-26 at 13 46 39

Added padding to not cut the icon buttons when hovered:
Screenshot 2024-02-26 at 13 56 02

@mbondyra mbondyra added release_note:skip Skip the PR/issue when compiling release notes Feature:ES|QL ES|QL related features in Kibana v8.14.0 Team:Visualizations Visualization editors, elastic-charts and infrastructure labels Feb 26, 2024
@mbondyra mbondyra force-pushed the lens/switch_chart_esql branch 6 times, most recently from f723bd7 to c053c05 Compare February 27, 2024 12:29
@elastic elastic deleted a comment from kibana-ci Feb 27, 2024
@mbondyra mbondyra force-pushed the lens/switch_chart_esql branch from c053c05 to dd373d3 Compare March 4, 2024 09:47
@mbondyra mbondyra marked this pull request as ready for review March 4, 2024 09:47
@mbondyra mbondyra requested review from a team as code owners March 4, 2024 09:47
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-visualizations (Team:Visualizations)

@mbondyra mbondyra added release_note:enhancement and removed release_note:skip Skip the PR/issue when compiling release notes labels Mar 4, 2024
@mbondyra mbondyra changed the title [ESQL] add chart switch to esql [ESQL] Add chart switch to esql Mar 4, 2024
@mbondyra mbondyra changed the title [ESQL] Add chart switch to esql [ESQL] Add chart switch Mar 4, 2024
@elastic elastic deleted a comment from kibana-ci Mar 4, 2024
Copy link
Contributor

@dej611 dej611 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review only. Will submit the final local check tomorrow 🚀

Copy link
Contributor

@stratoula stratoula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mbondyra this is fantastic 😍

I just wonder if we can hide the legacy metric from the suggestions but keep it on the switch. It works slightly different from the new metric, It will display all rows (while the new metric displays only the first) so in the case you don't run an aggregation but just render the numeric column the display is a bit weird.

image

I want to be hidden by the suggestions because I want to advertise in this section the charts that make sense. This is why I had hid it in the first place.

So ideally let's hide it from suggestions but allow the selection on the chart switcher. If this is difficult ping me to discuss async!

Copy link
Contributor

@dej611 dej611 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far I've found only one bug here related to this PR:

  • if I define a chart with 2 layers with different metrics, changing the chart type of the second layer will migrate the first layer configuration to the new chart type rather than the second one.

As for a general problem, but it is not specific to this PR necessarily, I see that unsaved changed are then persisted when clicking on the Edit in Lens editor link with no easy way to get back to the original state.

Copy link
Contributor

@MichaelMarcialis MichaelMarcialis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together, @mbondyra! I'm excited to see this first step towards getting rid of the dueling visualization selectors in the Lens app. Leaving you a few comments below for your first review. Let me know if you have any questions or need any design/CSS support. Thanks!


  • Currently the resizable border between the ES|QL editor and chart configuration visually appears to be 2px in its idle/default state. In reality, there are just two 1px borders stacked on top of each other (one from the editor footer and one from the resizable button). Could you remove the bottom border on the editor footer to prevent this from looking like 2px?

    CleanShot 2024-03-06 at 11 21 25

  • The configuration toolbar currently scrolls with any overflowing layer panel(s). Would it be possible to restructure the markup or CSS in a way that would limit the scrolling container to the layer panel(s) only, thus allowing the toolbar to always be visible to the user?

    CleanShot 2024-03-06 at 11 23 48

  • It appears that the odd jumpy animation when closing the configuration accordion has returned. Not sure if it is directly due to this PR, but would be nice to fix if possible.

    CleanShot 2024-03-06 at 11 28 15

  • The padding reductions made to the accordion contents feels odd. Now the accordion contents are no longer properly aligned with other content in the flyout. If this was done to address the fact that excessive padding was previously appearing to the right of the scrollbar, I think we should try an alternative approach. The reason this padding was appearing to the right was because a parent element of the scrolling container was receiving that padding. Instead, applying the padding to the scrolling container itself (or an element within it) should correctly address the issue without adversely affecting the alignment with other flyout content. Thoughts?

    CleanShot 2024-03-06 at 11 44 33

  • In reading the Google doc comments, perhaps changing the current warning icon to the dot icon (similar to how we do for incompatible quick functions in Lens) would make it appear less scary and distracting. Similarly, it might also be nice to reduce the "Technical preview" badge to a simple beaker icon with accompanying tooltip. And perhaps both can be moved to be adjacent to text (thus avoiding the jumping of icons on focus). Thoughts? I've placed a quick example mockup below.

    CleanShot 2024-03-06 at 12 45 48

  • Also per the Google doc comments, I agree that it's a bit problematic that deleting a required dimensions doesn't properly inform the user. Would it be possible to show the standard Lens "Requires field" validation message in red as we typically do for users that outright deleted required dimensions?

    CleanShot 2024-03-06 at 12 31 58

@@ -0,0 +1,78 @@
/*
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MichaelMarcialis thank you for your comments! I'll keep the responses here so we can take advantage of @markov00 proposal to keep the context in one place. First issue posted by you:

The configuration toolbar currently scrolls with any overflowing layer panel(s). Would it be possible to restructure the markup or CSS in a way that would limit the scrolling container to the layer panel(s) only, thus allowing the toolbar to always be visible to the user?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure it's a way to go. Here's how it looks right now in my Macbook 16'. The scrolling area is already extremely small for ESQL charts and making it even smaller makes it not even fit one layer configuration. I think it makes it worse for the user and I'd prefer to keep scroll as it is. Whether we decide to change it or not, I can add an issue and keep it separate from this PR.

Mar-07-2024 13-40-51

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a fair point. Yeah, let's split it off as a separate issue to tackle, since it will likely require more design thought and possible layout shuffling (beyond the scope of this PR). Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done here: #178234

@@ -0,0 +1,78 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MichaelMarcialis

  • In reading the Google doc comments, perhaps changing the current warning icon to the dot icon (similar to how we do for incompatible quick functions in Lens) would make it appear less scary and distracting. Similarly, it might also be nice to reduce the "Technical preview" badge to a simple beaker icon with accompanying tooltip. And perhaps both can be moved to be adjacent to text (thus avoiding the jumping of icons on focus). Thoughts? I've placed a quick example mockup below.

    CleanShot 2024-03-06 at 12 45 48

Copy link
Contributor Author

@mbondyra mbondyra Mar 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love the beaker idea! Regarding the other one, but shouldn't be just a little scary though? We don't offer any undo functionality and user can loose a big chunk of their configuration with clicking this button. I am happy to implement it though, it's just my thoughts.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I submitted an issue here: #178232

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for splitting off as a separate issue. And I do share your concern about making a potentially destructive action less scary while we continue to operate without undo functionality. On the other hand, the warning icon redundancy was so overwhelming in some scenarios, and something that has been bugging me for a while. Perhaps a good interim solution (until we have an undo) is to hit the user with a confirmation modal that they can optionally choose to "Never show again", similar to how we do when deleting layers? That way we ensure the user knows they are taking a destructive action without making the visualization selector overly noisy. Thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it, let's do it! I think we can also make the warning more specific. For example.

  1. We can know which columns will be persisted
  2. Which layers will be persisted

Maybe we could use this information while thinking about this new design? I will create a small POC (outside of this PR) and link it to the issue

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this proposal 😍

@@ -0,0 +1,78 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
Copy link
Contributor Author

@mbondyra mbondyra Mar 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MichaelMarcialis

  • Also per the Google doc comments, I agree that it's a bit problematic that deleting a required dimensions doesn't properly inform the user. Would it be possible to show the standard Lens "Requires field" validation message in red as we typically do for users that outright deleted required dimensions?

    CleanShot 2024-03-06 at 12 31 58

We still do it, we just don't do it when the visualization is completely cleared out. Do you think we should change it? It's possible, but I'd do it separately 🙏🏼

Screenshot 2024-03-07 at 15 30 34 Screenshot 2024-03-07 at 15 30 42

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think it'd be a better user experience to show that sort of "required" error on the appropriate dimensions even when the entire configuration is cleared out. I'm fine with splitting it out as a separate issue as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: #178237

@mbondyra mbondyra force-pushed the lens/switch_chart_esql branch from 430a808 to 800a58b Compare March 8, 2024 11:33
Copy link
Contributor

@stratoula stratoula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!! Thanx Marta, this PR has made my day! It is the beginning of many great enhancements and a great feature for ES|QL inline editing 😍

I tested everything and I can't find any regression

Copy link
Contributor

@MichaelMarcialis MichaelMarcialis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making those changes and creating separate issues for things that are beyond the scope of this PR, @mbondyra. It looks great. I have two final comments below, but nothing worth holding you up over. Approving now, assuming you're able to address (or split off as a separate issue).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • It looks like the jumpy accordion animation wasn't resolved, as I'm still seeing it in local testing. Possible to give it another look?

    CleanShot 2024-03-08 at 10 21 23

  • There are currently some issues present in the flyout when the user has greatly increased the height of the ES|QL editor and/or they have reduced the height of their browser's viewport:

    • There are overflow issues present with the accordion arrows (see screenshot below).
    • Additionally, open accordions may also be rendered so short in height that they would be unusable in some extreme scenarios. Should we attempt to prevent open accordions from getting that small (via a min-height or some other method)? Also, for the sake of responsiveness, at very small viewport heights, should we remove the accordions' existing inner content scrolling styles (allowing them to grow as tall as the height of their contents) and move the scrolling styles to a parent element of the accordions? This one may be a can of worms, so feel free to split off as a separate PR if you wish.

    CleanShot 2024-03-08 at 10 25 29

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Michael! As you say, it is a can of worms – I fixed the jump before but I broke another thing and then I thought I fixed it again, but it's not fixed for all the cases and broke the second thing you mention... I will submit another issue to make sure it's tested in separation 🙏🏼

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
eventAnnotationListing 469 470 +1
lens 1345 1346 +1
total +2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/visualization-ui-components 149 151 +2
lens 562 564 +2
total +4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
lens 1.4MB 1.4MB +1.3KB
textBasedLanguages 140.7KB 140.7KB -21.0B
total +1.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
expressionLegacyMetricVis 10.3KB 10.1KB -231.0B
Unknown metric groups

API count

id before after diff
@kbn/visualization-ui-components 152 154 +2
lens 663 665 +2
total +4

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@mbondyra mbondyra merged commit 57f659f into elastic:main Mar 8, 2024
16 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Mar 8, 2024
@mbondyra mbondyra deleted the lens/switch_chart_esql branch March 8, 2024 19:14
jughosta added a commit that referenced this pull request Apr 8, 2024
)

- Resolves #167887

## Summary

On Discover page user can see a visualization for data view and ES|QL
modes. For ES|QL mode it's also allowed to customize the visualization.
This PR allows to save such customization together with a saved search.

In more details, various types of Lens visualization can be shown on
Discover page:
- If in the default (data view) mode, Unified Histogram shows a
"formBased" histogram (`type:
UnifiedHistogramSuggestionType.histogramForDataView` in this PR)
- If in the ES|QL mode, 2 scenarios are possible (so far only these are
customizable):
- If Lens has suggestions for the current query, Unified Histogram shows
one of them (`type: UnifiedHistogramSuggestionType.lensSuggestion` in
this PR) Example query: `from kibana_sample_data_logs | stats avg(bytes)
by message.keyword`
- If Lens suggestion list is empty, Unified Histogram shows a
"textBased" histogram (`type:
UnifiedHistogramSuggestionType.histogramForESQL` in this PR). Example
query: `from kibana_sample_data_logs | limit 10`
   
The main flow is that Unified Histogram first picks a suggestion (one of
`UnifiedHistogramSuggestionType` type), then calculates lens attributes
which are necessary to build Lens embeddable. With a saved search we are
saving those calculated lens attributes under `savedSearch.visContext`.
For handling this logic, I refactored `useLensSuggestion`,
`getLensAttributes` into `LensVisService`.

Restoring a saved customization adds complexity to the flow as it should
pick now not just any available suggestion but the suggestion which
matches to the previously saved lens attributes.

Changes to the current query, time range, time field etc can make the
current vis context incompatible and we have to drop the vis
customization. This PR already includes this logic of invalidating the
stored lens attributes if they are not compatible any more. New vis
context will override the previous one when user presses Save for the
current search. Until then, we try to restore the customization from the
previously saved vis context (for example when the query changes back to
the compatible one).


What can invalidate the saved vis context and drop the user's
customization:
- data view id
- data view time field name
- query/filters
- time range if it has a different time interval
- text based columns affect what lens suggestions are available

Flow of creating a new search:

![1](https://github.com/elastic/kibana/assets/1415710/9274d895-cedb-454a-9a9d-3b0cf600d801)

Flow of editing a saved search:

![2](https://github.com/elastic/kibana/assets/1415710/086ce4a0-f679-4d96-892b-631bcfee7ee3)





<details>
<summary>Previous details</summary>

- Previous approach #174373
(saving current suggestion instead of lens attributes)
- Previous approach #174783
(saving lens attributes but it's based on existing hooks)

But I was stuck with how to make "Unsaved changes" badge work well when
user tries to revert changes.

For testing in ES|QL mode I use `from kibana_sample_data_logs | limit
10` as query, customize color of a lens histogram, and save it with a
saved search. Next I check 2 cases:
1. edit query limit `from kibana_sample_data_logs | limit 100`, see that
vis customization gets reset which is expected, press "Revert changes"
in the "Unsaved changes" badge => notice that reset did not work
2. edit only histogram color, press "Revert changes" in the "Unsaved
changes" badge => notice that reset did not work

Here are some nuances with the state management I am seeing which
together do not allow to successfully revert unsaved changes:
- For ES|QL histogram lens attributes include a modified query `from
kibana_sample_data_logs | limit 10 | EVAL timestamp=DATE_TRUNC(30
second, @timestamp) | stats results = count(*) by timestamp | rename
timestamp as "@timestamp every 30 second"` which means that not only
changes to the original query but also a different time interval
invalidates the saved lens attributes.
- In ES|QL mode, `query` prop update is delayed for
`UnifiedHistogramContainer` component until Discover finishes the
documents fetch
https://github.com/elastic/kibana/blob/fc2ec957fe78900967da26c80817aea8a0bd2c65/src/plugins/discover/public/application/main/components/layout/use_discover_histogram.ts#L346
which means that Discover should make a request on revert changes. And
It's not happening for (2) as it does not make sense for Discover to
trigger refetch if only `visContext` changes so we should find another
way. With (1) there is another problem that Discover `visContext` state
gets hijacked by lens attributes invalidation logic (as query is not
sync yet to UnifiedHistogram) before fetch is completed or get [a chance
to be
fired](https://github.com/elastic/kibana/blob/6038f92b1fcaeedf635a0eab68fd9cdadd1103d3/src/plugins/discover/public/application/main/hooks/utils/build_state_subscribe.ts#L51-L54).
I tried delaying `externalVisContext` prop update too (to keep in sync
with `query` update) but it does not help
https://github.com/elastic/kibana/blob/fc2ec957fe78900967da26c80817aea8a0bd2c65/src/plugins/discover/public/application/main/components/layout/use_discover_histogram.ts#L437
- Unified Histogram should signal to Discover to start a refetch when
current suggestion changes
https://github.com/elastic/kibana/blob/fc2ec957fe78900967da26c80817aea8a0bd2c65/src/plugins/discover/public/application/main/components/layout/use_discover_histogram.ts#L289
- for some reason this logic is required for "Revert changes" to work as
it triggers the refetch. I would expect Discover on its own to notice
the change in query and refetch data but it does not seem to be the
case.

</details>

<details>
<summary>Other challenges</summary>

- [ ] Since we are starting to save lens attributes inside a saved
search object (similar to how Dashboard saves lens vis by value), we
should integrate Lens migrations into saved search migrations logic. I
made a quick PoC how it could look like here
jughosta@4529711
This showed that adding Lens plugin as a dependency to saved search
plugin causes lots of circular deps in Kibana. To resolve that I am
suggesting to spit saved search plugin into 2 plugins
#174939 - not the best solution
but it seems impossible to split lens plugins instead.
    Updates here: 
- [x] revert the code regarding migrations and saved search plugin split
- [x] create a github issue to handle client side migrations once their
API is available #179151
- [x] Discover syncs app state with URL which means that the new
`visContext` (large lens attributes object) ends up in the URL too. We
should exclude `visContext` from URL sync as it can make the URL too
long.
    Updates here:  we are not using appState for this any more
- [x] Changes from #171081 would
need to be refactored and integrated into the new `LensVisService`.
- [x]  Refactor after #177790
- [x] Handle a case when no chart is available for current ES|QL query
- [ ] For ES|QL histogram lens attributes include a modified query `from
kibana_sample_data_logs | limit 10 | EVAL timestamp=DATE_TRUNC(30
second, @timestamp) | stats results = count(*) by timestamp | rename
timestamp as "@timestamp every 30 second"` which means that not only
changes to the original query but also a different time range can reset
the customization of lens vis as it gets a different time interval based
on current time range
- New update from Stratoula: 
- [ ] would it help to persist response of `onApplyCb` instead of lens
attributes? <= the shape does not seem to be different and it works as
it is so I'm keeping lens attributes
- [x] use new `getLensAttributes` from
#174677

</details>

10x flaky test
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5578

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:ES|QL ES|QL related features in Kibana release_note:enhancement Team:Visualizations Visualization editors, elastic-charts and infrastructure v8.14.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants