[SIP-34] Proposal to establish a new design direction, system, and process for Superset

[rusackas]

[SIP-34] Proposal to establish a new design direction, system, and process for Superset

Motivation

In August, 2019, a project was kicked off to re-imagine the design of Superset, and create a “North Star” for the interface, flows, and visual aesthetic of Superset. The primary goal of the project—and this SIP—is to create a new visual design framework for Superset's user interface.

Since its inception, Superset has rapidly grown in scope, size, audience, and complexity thanks to the work of this community. This growth has led to an accumulation of “design debt” in forms including problematic user flows, inconsistencies in interface elements, or general complexities that may be simplified/clarified by revisiting the design of Superset holistically.

The intent of this undertaking was to smooth out the user experience of Superset, targeting all of the following:

• Supporting most use cases and functionality currently in use by known Superset users in the design process, while reducing dead-ends and friction in these flows. Features and flows removed or deprecated are listed under "Removals and deprecations" below
• Focusing on the “objects” of superset (namely Datasets, Queries, Charts, and Dashboards) and how those are structured/accessed throughout the flows of Explore and SQL Lab
• Smoothing out and simplifying the general experience, making Superset easier to use for new users, and faster for more experienced ones
• Creating a more consistent design and experience throughout the product, with a well defined style guide, and coherent design system

Participants in this project included:

• Preset, Inc. - a startup founded by @mistercrunch to offer Superset as a hosted service, acting as primary sponsor of this design effort
• Cartel - a digital product design firm focusing on research, strategy, and UX design for the next generation of enterprise software companies
• Various members of this community, including Airbnb, Lyft, and others

We hope that this new design direction will carry Superset forward for the foreseeable future, and create a unified design system upon which to build future features. Community feedback is welcomed, and when this SIP is voted in, further efforts will be undertaken to break down the design into smaller units of work.

Process used in this effort:

Efforts began with the announcement of a call for volunteers in a design interest group, via the Superset dev mailing list, and on the Superset Slack workspace. The search for members of this small group targeted individuals who were able to commit 2+ hours per week, have a history of involvement/merit with the project, have a sense of design, and understand Superset users and use cases.

A "Superset Design Group" email was set up, as well as a private channel on the Superset Slack workspace, to maintain an open line of communication throughout the duration of the project

A kickoff meeting of the resulting group was held on September 4th, 2019. This initiated the creation and iteration of core user stories. This was quickly followed by an effort to assess the Analytics / BI marketplace, resulting in a competitive analysis report (this research is proprietary, and cannot be shared publicly).

Weekly design sync meetings took place each Thursday, allowing Cartel to get feedback on their latest design proposals from the team/community, and gather insight/direction on "big question" issues about the product's direction. After each session, prototypes were circulated, and feedback gathered via the slack and email channels. Each 1-week design sprint was comprised of ongoing work on larger concepts (e.g. the relationship between SQL Lab and Explore, or controls used to build data visualizations) as well as smaller, more specific tasks.

Summary of key findings

A product audit of Superset, user personas, and general product recommendations are available in an Experience Brief.

Special thanks

This process involved significant time and involvement from the following members of the Superset community for their participation:

• Grace Guo (Airbnb / @graceguo-supercat)
• Vyl Chang (Airbnb)
• John Bodley (Airbnb / @john-bodley)
• Sylvia Tomiyama (Airbnb)
• Eli Brumbaugh (Airbnb / @elibrumbaugh)
• Michelle Thomas (Airbnb / @michellethomas)
• Maxime Beauchemin (Preset / @mistercrunch)
• Rob DiCiuccio (Preset / @robdiciuccio)
• Aaron Suddjian (Preset / @suddjian)
• Tai Dupree (Preset / @nytai)
• Beto Dealmeida (Lyft / @betodealmeida)
• Micah Delfino (Cartel)
• Jef Bekes (Cartel)

Proposed Changes

Notable changes to design and functionality

Visual Design

• All UI fonts are changed to Inter UI for increased legibility, and fixed-width numeral options for tabular data. Code fonts have been changed to Fira Code. Both of these fonts are open source (license information below).

• New default color palettes for dataviz. These have been checked for accessibility issues. Old palettes will be supported for backward compatibility.

• A full style guide is available in the prototype, with specs for a variety of UI element interaction states, layout spacings (all based on an 8 pixel grid)

  Style Guide (see abridged InVision)

Simplified global navigation.

• Links to Home (see next item), Dashboards, Charts, Explore, SQL Lab, and Data

  Current	New

Dashboards

• New list/card view of all dashboards with ability to favorite dashboards, filter them by owner/status/dataset, search by name, sort by various metrics, or perform bulk actions.

• New card view displays cards with rendered previews of the dashboard

• Individual dashboards include new tools for each chart indicating data recency and filters applied

• Edit a chart's display options directly from the dashboard

• New tab UI with dropdown overflow menu when there are too many tabs to display

• New UI modal to define/edit global filters, with bulk action support

  Current	New

New "Home" page

• Offers the ability to view Dashboards, Charts, and Queries, sorted by popularity, recency, favorites, and ownership.

• Activity steam, showing who did what in a Superset instance

  Current	New

Explore

• Visualization type selector, with viz plugins broken out into categories (e.g. timeseries, distribution, composition, etc)

• Left panel for dataset selection and resulting display of fields

• Updated query panel, with visualization controls utilizing new "pills" containing field metadata and controls (e.g. aggregation, sorting)

• New Save/Save As controls, highlighting potential implications from overwriting an existing chart

• Dataset preview area, including data profiling in the column headers (i.e. histograms, null value count, statistical rollups, etc)

• New Chart Options drawer in the top right, focused on visual aspects of the resulting chart

• New UI to edit a dataset's source, fields, metrics, and more

• History timeline with icons denoting query "run" and "save" events

• New Time Picker input modal, with quick selection of various time ranges and granularities

  Current	New

SQL Lab

• Left panel has tabs for Data (a picker for databases/tables/fields) and Saved (a selector for saved Queries)

• Data preview table with headers containing data profiling information (histograms, composition, etc)

• Chart area to build quick/simple data viz to preview query results and/or add viz to a dashboard. Viz options here are limited in scope/complexity. If your needs expand beyond these simplified viz options, there's an option to move over to Explore.

• Explore button, allowing users to take their SQL queries into Explore for more advanced viz options with expanded controls

• History panel with icons and data-rich tooltips indicating when queries were run or saved, and the details thereof

• Full Query history with timeline, query preview, and option to group by session

  Current	New

Data section

• Contains sortable/searchable/filterable card or list views of Datasets (i.e. "physical" static/tablular datasets or "virtual" dynamic query results), Databases, Saved Queries, and Query History. Each of those tabs displays various metadata (e.g. access statistics) and actions (edit, delete, etc) for the resulting items

• Viewing a Physical or Virtual Database displays its Fields, Metrics, and Usage tabs displaying various statistical and historical/analytic metadata details

  Current	New (Datasets)	New (Fields)

Charts

• New list/card view of all charts with data source display, ability to favorite items, sorting (e.g. by time last updated), and filtering by ownership and status.

  Current	New

Global Search

• Improved search and discovery of dashboards, queries, dataset titles, and possibly more

• Results can be filtered by type

• Default (prior to typing) list of recent or commonly accessed items

  Current	New (default, no input)	New (search results)
  N/A

Design proposals, in the form of InVision prototypes, are encapsulated in the deliverables available here:

• Select screens (key features and concepts)
• Unabridged screens (contains some details yet to be entirely fleshed out)

Removals and deprecations

• Tabs in SQL Lab have been removed (See related SIP).

Maintenance and improvement of design assets

For maintaining the new design work (and assets stemming from it), the current proposal is to maintain the creation of a Superset team on Abstract. The latest Sketch files will be made public and open source, and managed via Abstract's version control system. PMC members from the Superset community may reach out to the PMC mailing list to submit designers or design teams to be added to the Abstract team to contribute to the design. The latest master version of the files will be maintained in the public InVision prototype, maintained by Preset (though interested parties may request comment or write access, granted on a case-by-case basis).

User testing and validation plan

Design and feature proposals submitted herein were conducted using the Superset Design Group as a focus group, making sure that functional and visual changes to the UI address the issues and use case criteria of those involved in the discussion. It is our intent, and belief, that these designs push the usability of Superset forward greatly. We anticipate that individual features and UI elements will require further input and discussion, as this "north star" for design begins to play out. Our hope is that this round of design will provide a foundation to test further iterations/variations, and those results will bear out in additional SIPs and PRs on a case-by-case basis.

Updates to Superset documentation

Documentation and Screenshots for Superset exists in a number of places, most notably on the Apache website and on the Github repo. A best effort should be made to update these resources with updated screenshots and/or instructions as relevant as PRs are written to move the project toward this new UI. If any changes are encountered that require large scale changes to documentation, those changed should be discussed in an additional SIP.

New or Changed Public Interfaces

TBD — Impact will be minimized on existing interfaces, but implementation of workflows and features may require changes to how APIs and metadata are structured and/or accessed.

New dependencies

The scope of third party dependencies is not entirely known at this time, but will be defined and assessed in future PRs as the work is broken down into smaller tasks. An effort will be made to make sure all licenses are compatible with Apache License v2.0.

Some known licenses thus far include:

• Inter UI (font) — SIL OpenFont license (needs to be labeled in ASF releases)
• Fira Code (font) — SIL OpenFont license (needs to be labeled in ASF releases)

Migration Plan and Compatibility

Due to the broad scope of design and functional changes, several core sections of the codebase — both frontend and backend — will need to be modified in functionality and/or refactored. This will likely lead to a significant number of required database migrations. While breaking changes in backward compatibility will be made as minimal as possible, they may occur in subsequent PRs as this work is further scoped and ticketed, and should be tracked with semantic versioning in future releases.

For frontend implementation, it should be noted that an older version of React-Bootstrap is used in existing Superset UI components. This older version is built upon Bootstrap 3, which is built with LESS (as is much of Superset, allowing shared styles/variables). Upgrading components to a newer React-Bootstrap (and thus Bootstrap 4) would require migrating the bulk of the CSS codebase over to SASS, which is not ideal. The implementation of this SIP should iterate toward deprecating the React-Bootstrap dependency, in favor of bespoke components with custom LESS styles built around this new design system.

All changes, in the design and implementation of new components, should be "atomic" in nature, steering toward a more unified design/component system, targeting reusability and consistency.

Rejected Alternatives

No alternatives at this time. We've considered previous design-related SIPs in this process, and have reached this proposal with input from several active PMC members. We're open to feedback, however, so please feel free to leave constructive feedback below. This work is to be considered foundational, and will undoubtedly continue to evolve with the ongoing support of the Superset community.

[issue-label-bot]

Issue-Label Bot is automatically applying the label #enhancement to this issue, with a confidence of 0.81. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[john-bodley]

Thanks @rusackas et al. for putting this great synopsis together.

[williaster]

The implementation of this SIP should iterate toward deprecating the React-Bootstrap dependency, in favor of bespoke components with custom LESS styles built around this new design system.

I'm curious if it would make sense to move to a more modern CSS in JS approach for styling components. This supports a shared theme object for fonts, colors, sizing, etc., while encapsulating and co-locating specific styles to individual components. This tends to mitigate the risk of bloated css over time which occurs because it's very difficult to isolate and remove class-name-based styles. Lunar (demo) is an example of such a library, we could consider it.

in favor of bespoke components with custom LESS styles

By this phrasing do you mean not using any component library (like react bootstrap)? I would note that it is a significant amount of work to write a component library from scratch, even if components are replaced individually one at a time.

All changes, in the design and implementation of new components, should be "atomic" in nature, steering toward a more unified design/component system, targeting reusability and consistency.

I generally agree with this, but it seems worthwhile to align on a more specific component library or approach for new / implemented components in this SIP. If there is ambiguity, there might be a variety of libraries / frameworks introduced, or if it is unclear what to use Bootstrap 3 might continue to be used.

[rusackas]

@williaster I agree with your concerns, and share your curiosities. I probably shouldn't have gone as deep in component implementation details in this SIP, as the intent here is to align on the design direction. I figured it was warranted to at least address the inevitable "how do we implement this" question, particularly as it relates to ongoing use of React-Bootstrap.

I'm open to CSS-in-JS approaches, but want to make sure that vars/styles/mixins are shareable between custom components, and new or existing component libraries.

I also share the concern about not opening the floodgates to disparate libraries/frameworks. That's why I'm personally hesitant to even add a second one while React-Bootstrap is still in the picture. I would be inclined to at least attempt a few components from scratch, thus the suggestion. Indeed, it's more work, but if a system evolves from it, that may lead to a longer lifecycle, rather than growing attached to another library which may potentially cause this same situation a year or two down the road.

All that said, the component implementation plan is worthy of deeper discussion, and probably being augmented by a related implementation-oriented SIP. The focus of this SIP was intended to be around the design work itself, and getting this new design direction adopted. I'm open to revision or removal those implementation suggestions if they're of major concern to this proposal.

[mistercrunch]

Clearly Superset is very deep into react-bootstrap now (103 references in the codebase) and we'll need to tweak/bend it and in cases just deviate from it to get closer to this north star, as needed. I think that generally using the bootstrap-defined classes conventions seems reasonable to me (panel, table, form, btn, btn-sm, ...). In many cases react-bootstrap is simply wrapping/exposing classes as components and/or props. If we commit loosely to the bootstrap CSS conventions, we may not have to touch many of the react-bootstrap components for a moment, and iterate slowly towards the look and feel proposed here.

Maybe this is shortsighted, but a decent percentage of the theme-related-work to make Superset look like the design suggested in this SIP can be done directly in the existing files under superset/assets/stylesheets/less/.

One big question is whether we want to make the theme somewhat environment or user-tunable. We'd like to enable admins and/or users to define their brand colors / palettes, navbar logo, and some other aspects of how things look.

[gbrian]

Amazing @rusackas ,
Question: is it worth to merge SQLAB and Chart Explore?
Sorry for the quick mockup 🙈

[villebro]

Can't agree more with @gbrian , really impressive work and obviously a lot of thought has gone into this, hence I'm confident most things I say have already come up in discussions before. Amazing on so many levels. However, I feel inclined to extend a few thoughts:

• The general look and feel is really fresh and much more intuitive and modern than the current appearance.
• Two of the things that really stand out to me are as follows:
  1. showing more context around objects (datasets, charts, columns): if the backend can handle the additional load of showing column-wise distributions, usage stats on the chart pages etc then it will make data ever more approachable. It might be a good idea to make this behavior configurable, too, especially for for environments with dbs that aren't up to the challenge of serving all this additional info. Parts of this retrieval should probably be highly async to ensure the UX stays responsive.
  2. The metrics/columns in the Explore view really make charting much more approachable for newbies. I've noticed that some BI tools have a very steep learning curve; Using terminology/labels like "X-axis", "Y-axis" make it much more intuitive for novices than "metric", "temporal column". Having said that, there is the risk of limiting functionality at the expense of lower learning curve (especially thinking Tableau vs PowerBI here), so one might need to proceed with caution on making big changes here.
• Regarding widespread customizability of appearance, I personally usually prefer to have products look like themselves rather than the company in which they're used. However, being able to configure Dashboard/Viz appearances (fonts, colors etc) is really important, and can be vital in ensuring that the visualizations can be used as widely as possible in the target organization, both for internal and external presentations/materials.
• I think this was contained in another SIP, but feel worthwile mentioning here comments on the removal of tabs in Sql Lab. @cgivre mentioned hesitation at removal of tabs, and that was my first reaction, too. However, having thought more closely about it I believe the user experience could benefit from a the new approach, especially if old queries are easy to find. I've noticed that tabs tend to pile up after a while, and generally only leads to lots of tabs with useless stale queries. But that might just be me.

[CoryChaplin]

Hi,

First of all, huge thanks for this work, it's totally impressive and promises to make Superset change category. Awesome level up.

I have a few questions:

• What's the reflexion behind the Explore view overhaul? The proposal looks very similar to several other BI tools but my personal feeling is that the way Superset currently does it is superior. The datasource being totally displayed probably helps data discovery but also makes the display less light. Also, the new way to build metrics, filters, group by's, etc, feels less intuitive to me. Last point, don't you think pushing the results table to the default display instead of in a modal makes the process more directed towards experts who want to review the underlying data and less towards self service for everyone?
• Though I love the idea of having previews of the graphs everywhere, I'm a little concerned about the way that would be fulfilled. If my memory is correct, Max did a PR in the past to go that way but it relied on Selenium doing screenshots of everything. That dependency feels a bit fat to me, what do you think? And rendering real graphs from queries seems to considerably increase the load on the databases.
• Last comment, I like the tabs in the SQLlab 😺

Sorry if I'm mistaken somewhere and thanks again for the awesome contribution!

[mistercrunch]

• metadata browser is collapsible, both the drag-n-drop and dropdown approaches will work
• agreed: the data browser provides more context but steals the show a bit, it is collapsible though so you can get it out of the way if you don't want it
• the thumbnail / card views is very likely to be put behind a feature flag as it does involve more infrastructure and background processing. The new list views offer both card and table, and cards will depend on that feature flag
• Tabs seem to be contentious, there's another SIP specifically for that. We do like tabs too, but we'd like to encourage people to use their browser's tabs instead of ones inside the app

The SIP is really about direction, and individual issues will be brought back to light either at implementation time or in a SIP if they are controversial and/or high importance.

[cgivre]

@villebro
After seeing the new design, I really like it. I may miss the tabs but overall, I'm really impressed and think this is a major improvement.

[rplescia]

I really like the designs you guys have produced here, clearly allot of thought has gone into this. I think even though the interface is still very good is due for a refresh, there are some bits which are starting to show their age from a UX perspective. Good job folks, this SIP has my full support

[mistercrunch]

FYI - I kicked off a vote on the dev@ mailing list

[ktmud]

🎉 🎉 Congratulation on passing the SIP! Will there be another SIP soon to discuss the frontend solution to implement this? It seems there are a lot of unclosed items in this thread.

[rusackas]

@ktmud There will probably be several SIPs as this gets parceled out, but there are a few worth noting right now:
SIP-37 deals with using CSS-in-JS to migrate away from LESS and restructure how and where styles are applied
SIP-30 tackles the removal of SQL Lab tabs
SIP-38 deals with visualization plugins, which is at least part of the puzzle toward this SIP.

Some of it will probably just start trickling in via PRs, but anything large/contentious may trigger a SIP, whether it's a specific feature proposal, or some new dev process/pattern.

[ktmud]

Thanks for the links, @rusackas !

I think one of the most important questions still not clear to me is, are we going to completely migrate away from react-bootstrap and FAB? And

• Should we use another full-fledged UI library like Lunar or Ant Design and build a custom theme?
• Or, if we are to build all UI components by ourselves, will we consider using a primitive UI component library such as Rebass, React components and Theme UI?

This seems to warrant its own SIP.

[rusackas]

@ktmud SIP-37 takes on a little of this discussion, so perhaps that's a better place for it, but one key idea is that we should remove the need for Bootswatch/Cosmo, which at least then provides the opportunity to at least upgrade react-bootstrap. After that, I think the situation is worth reassessment, and possibly another SIP. My thought would be to start whittling away react-bootstrap using custom components - at least where it's easy/pragmatic to do so, just to reduce reliance on that dependency. I'm hesitant to embrace another full library, but with CSS-in-JS, it should be possible to make a transition if necessary, or even mix and match more easily, if needed.

[richlysakowski]

I am very concerned that the proposed major re-designs proposed (and approved) appear to add significant complexity for SuperSet administrators and end users. Also, a big concern about a push to "custom components", rather than more standardized open source components provided with the open source kit now. Statements by @rusackas "whittling away react-bootstrap using custom components" and by @ktmud "if we are to build all UI components by ourselves" suggest that this is what is underway. @rusackas "The implementation of this SIP should iterate toward deprecating the React-Bootstrap dependency, in favor of bespoke components with custom LESS styles built around this new design system." The comment by @gbrian "merge SQLAB and Chart Explore" seems to decomponentize the design; it does not retain separation of GUI concerns for the user.

Deprecating major technologies like React-Bootstrap in the name of "simplification" for developers will introduce yet another reactive framework, and delay the user community getting a version 1.0 product. It adds a lot of complexity for end users who just want to stand up something quickly that scales in performance and functioality, and is highly usable.

This complete redesign completely torpedoes a lot of standard functionality provided "out of the box" that is required to almost effortlessly stand up a running system with minimal effort. We want a completely working system that includes all servers, subsystems, data connectors, and a decent looking GUI that requires NO customization to get running, and provides tool to customize from a working system.

This product has been "incubating" for several years already. The proposed redesign will require Apache "re-incubation" for another year or two. If it ain't broke, don't fix it.

What is really going on here? It appears like commercial professional software consultancies wanting to add complexity to a product to guarantee job security for staff, or perhaps guaranteeing that SuperSet v1.0 never finishes incubating, and always requires experts to set up. This is reminiscent of the ways that Oracle, IBM, and Microsoft wage in IEEE, ASTM, and ISO fight standards wars by putting technical staff on IEEE, ASTM, and ISO standards committees to guarantee that a standard specification is never implemented. The likely culprits in this advanced analytics and business intelligence open source war for software are Tableau, Tibco, Microsoft, and other BI advanced analytics vendors who don't want more open source competition. They would love to stall Superset more or less permanently.

Users DO NOT WANT a complex expert-driven procedure to stand up a Superset system or tailor it.

We need a SIMPLER process to stand up a Superset instance, NOT MORE COMPLEX. We need a simple one-step installation to get up and running and configuring "sensible default" dashboards. Something as simple as "docker pull" followed by "docker run", point your browser here, and open your first visualization.

Remember to target end users (analysts and dashboard assemblers, not programmers) in 95% of all work that you do, because that's who going to be using it the most.

Finish incubating and shipping v1.x of SuperSet before a doing total redesign.

[rplescia]

Is this currently being worked on? Does anyone know when we are likely to see these changes being implemented?

[benceorlai]

Hi @rplescia, yes, the team at Preset is working on it. We'll be releasing in small increments, the first pieces of refactored UI should be out (behind a feature flag) in the next 4-8 weeks

[rusackas]

Sorry for any Github notification noise, but I made a small attempt at finding recent PRs that were in service of this SIP. As we go forward, hopefully we can keep adding the mentions, to keep people informed here about progress being made.

[ktmud]

@rusackas should we just add a SIP-34 label to those PRs or create a SIP-34 project?

[richlysakowski]

Is there a Docker container available that just runs successfully on Windows 10, i.e., is Windows 10 supported yet?
If Can someone provide a pointer to a video tutorial to set up Superset on Windows 10?

[rusackas]

@rusackas should we just add a SIP-34 label to those PRs or create a SIP-34 project?

A label is a good idea for sure, but my main intent was to show forward motion on this issue's thread. There seemed to be a pattern around issue mentions already, particularly the "Has associated issue:" checkbox, which is what got me thinking the mentions might be a good idea. A project might also be helpful, but I didn't want to add more overhead work.

[rusackas]

Is there a Docker container available that just runs successfully on Windows 10, i.e., is Windows 10 supported yet?
If Can someone provide a pointer to a video tutorial to set up Superset on Windows 10?

Not to my knowledge, but there are a number of issues touching on the topic, or you can open a new issue specifically about that.