Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Lens] Turn off Group remaining values as "Other" setting when visualization has more than 1000 terms aggregation #162456

Closed
mbondyra opened this issue Jul 25, 2023 · 5 comments · Fixed by #167141
Closed

[Lens] Turn off Group remaining values as "Other" setting when visualization has more than 1000 terms aggregation #162456

mbondyra opened this issue Jul 25, 2023 · 5 comments · Fixed by #167141
Assignees
@stratoula
Labels
enhancement New value added to drive a business result Feature:Lens impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. loe:small Small Level of Effort performance Team:Visualizations Visualization editors, elastic-charts and infrastructure

Comments

@mbondyra
Copy link
Contributor

mbondyra commented Jul 25, 2023 •
edited
Loading

Recently we got a fair number of performance problems coming from the customers who want to do visualizations with >1000 of top values (usually to get a datatable and export it to csv and use their own tools). The problem is that by default we have a setting Group remaining values as "Other" on and that makes the performance way worse.
Screenshot 2023-07-25 at 10 10 24

(Tech explanation: The reason for the performance problems is how ES works under the hood. Kibana has to send another, more expensive call to ES to get other. ES gathers all the other terms from all the documents that exist except from the ones from the x top values and that can be very heavy when there's a lot of terms and documents)

For the users, the most common usecase for having >1000 of top values is what I mentioned above - get a datatable of data and export it to CSV, so the Group as other is not really needed in this case.

So we could solve the customers problem in two ways:

  1. Turn off and disable the setting Group remaining values as "Other" if there are more than 1000 of top values (or we could go with 100 or whatever arbitrary number we find useful)
  2. Turn it off by default when there's more than 1000 top values but not disable it - this way the user would still configure them and be conscious of the performance impact it has.

Let me know your thoughts!
@botelastic botelastic bot added the needs-team Issues missing a team label label Jul 25, 2023
@mbondyra mbondyra added enhancement New value added to drive a business result Team:Visualizations Visualization editors, elastic-charts and infrastructure Feature:Lens and removed needs-team Issues missing a team label labels Jul 25, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

@mbondyra mbondyra changed the title [Lens] turn off the group other setting when visualization is made for more than 1000 top values [Lens] Turn off Group remaining values as "Other" setting when visualization has more than 1000 terms aggregation Jul 25, 2023
@stratoula stratoula added the impact:needs-assessment Product and/or Engineering needs to evaluate the impact of the change. label Jul 26, 2023
@timductive timductive added loe:small Small Level of Effort impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. and removed impact:needs-assessment Product and/or Engineering needs to evaluate the impact of the change. labels Jul 26, 2023
@timductive
Copy link
Member

Agree with option 2, let's just turn it off at 1000 terms but let the user re-enable it.

@markov00
Copy link
Member

markov00 commented Aug 3, 2023

I also agree with option 2.
I also believe that Group remaining values as "Other" should be enabled by default only when using partition charts (because of the nature of the partition) and 100% stacked bar charts (same reason), for all the other configurations I believe we can disable by default that option.

@stratoula stratoula mentioned this issue Aug 9, 2023
Closed
@mbudge
Copy link

mbudge commented Aug 16, 2023

"Agree with option 2, let's just turn it off at 1000 terms but let the user re-enable it."

100% agree. Training operators reviewing security log data with these issues is too much overhead. Will be a lot easier to disable the Grouping feature.

@yomduf
Copy link

yomduf commented Aug 16, 2023

Agree with option 2

@stratoula stratoula self-assigned this Sep 26, 2023
@stratoula stratoula mentioned this issue Sep 26, 2023
Merged
1 task
stratoula added a commit that referenced this issue Oct 2, 2023
@stratoula @kibanamachine
[Lens] Other bucket defaults to false for top values greater than equ…
93f9213 
…al 1000 (#167141)

## Summary

Closes #162456

It switches off the other bucket switch if the user updates the terms to
a number greater than equal 1000.

In the beginning I had made it simpler so the switch would change to
false if size >= 1000 and to true if size < 1000. But I am not sure I
like this experience so I decided to do something else:

- The other bucket defaults to true if size < 1000
- If the user changes to a value >=1000 then it changes to false (The
user can always enable it)
- If the user changes this to a value < 1000 it doesn't change back to
true. The user must change it manually.

Let me know if you agree that this experience makes more sense.

<img width="2346" alt="image"
src="https://github.com/elastic/kibana/assets/17003240/29d64ab7-7e55-4b2a-b769-b5f9c6b6d385">


### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@drewdaemon drewdaemon mentioned this issue Mar 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stratoula stratoula

Labels
enhancement New value added to drive a business result Feature:Lens impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. loe:small Small Level of Effort performance Team:Visualizations Visualization editors, elastic-charts and infrastructure
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Lens] Other bucket defaults to false for top values greater than equal 1000 stratoula/kibana
7 participants
@timductive @markov00 @mbondyra @mbudge @elasticmachine @stratoula @yomduf