chore(explore): Migrate BigNumber to v1 api [ID-28][ID-55] #17587

kgabryje · 2021-11-30T08:22:03Z

SUMMARY

This PR's goal is to make the legacy Big Number chart plugin not-so-legacy-anymore.

First improvement is migrating the chart endpoint from legacy to V1 API. This change will also allow proper handling of timestamp format metrics. The legacy API assumed that metric result is always a number - if you used a metric that should return a datetime (e.g. MAX(temporal_column)) with a db that returns dates as strings (like sqlite or druid), you'd get nulls in response (see screenshots).

Second thing - to render the trendline, we used @data-ui/xy-chart library, which hasn't been maintained for 2 years. I replaced it with Echarts, which is our go-to viz library.

Lastly, the legacy plugin wasn't able to handle rolling aggregates properly if values were missing. The new version ensures that rolling windows are resampled with null values when a time grain is used to ensure that the calculation window always has the correct number of lagged values applied.

DB migration benchmark results:

Current: 0.42 s
10+: 0.33 s
100+: 0.30 s
1000+: 0.30 s
10000+: 0.42 s

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before:

After:

TESTING INSTRUCTIONS

Big Number and Big Number with Trendline core functionalities should work like before.
Timestamp metrics should be formatted automatically.

Steps to reproduce bug shown on screenshots:

Create a BigNumber or Big Number with Trendline chart using SQLite or Druid datasource
Create a metric that should return a date (like MAX(temporal_column))
Verify that Big Number and Trendline are rendered properly, and that big number and trendline's tooltip are formatted as a date

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

CC @junlincc @jinghua-qa

codecov · 2021-11-30T12:30:28Z

Codecov Report

Merging #17587 (e674657) into master (07bbe84) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head e674657 differs from pull request most recent head 944ae17. Consider uploading reports for the commit 944ae17 to get more accurate results

@@           Coverage Diff           @@
##           master   #17587   +/-   ##
=======================================
  Coverage   67.72%   67.72%           
=======================================
  Files        1602     1602           
  Lines       64146    64154    +8     
  Branches     6773     6773           
=======================================
+ Hits        43442    43450    +8     
  Misses      18851    18851           
  Partials     1853     1853

Flag	Coverage Δ
hive	`81.78% <100.00%> (+<0.01%)`	⬆️
mysql	`82.15% <100.00%> (+<0.01%)`	⬆️
postgres	`82.21% <100.00%> (+<0.01%)`	⬆️
python	`82.55% <100.00%> (+<0.01%)`	⬆️
sqlite	`81.89% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...-frontend/src/visualizations/presets/MainPreset.js	`100.00% <ø> (ø)`
superset/charts/schemas.py	`99.34% <100.00%> (+<0.01%)`	⬆️
superset/common/query_context_processor.py	`91.38% <100.00%> (+0.08%)`	⬆️
superset/models/helpers.py	`90.66% <100.00%> (+0.07%)`	⬆️
...hart-echarts/src/BigNumber/BigNumberTotal/index.ts
...lugin-chart-echarts/src/BigNumber/BigNumberViz.tsx
.../BigNumber/BigNumberWithTrendline/controlPanel.tsx
...arts/src/BigNumber/BigNumberWithTrendline/index.ts
...ugin-chart-echarts/src/BigNumber/sharedControls.ts

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07bbe84...944ae17. Read the comment docs.

kgabryje · 2021-12-01T19:30:44Z

/testenv up

github-actions · 2021-12-01T19:32:48Z

@kgabryje Ephemeral environment spinning up at http://54.218.60.103:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

villebro

One last nit, other than that LGTM and works and looks really great!

villebro · 2021-12-11T15:04:34Z

superset-frontend/packages/superset-ui-core/src/query/types/QueryResponse.ts

+  from_dttm: number | null;
+  to_dttm: number | null;


It appears the ChartDataResponseResult schema is pretty badly out of sync with this. For example, the following appear to be missing or have an incorrect name:

here we have cache_dttm, on ChartDataResponseResult we have cached_dttm

colnames and coltypes are missing on ChartDataResponseResult.

from_dttm and to_dttm should also be added to ChartDataResponseResult.
Let's try to make sure these are in sync.

ktmud

LGTM but I think we may need a db migration script

ktmud · 2021-12-11T00:10:40Z

superset-frontend/plugins/plugin-chart-echarts/src/BigNumber/BigNumberTotal/controlPanel.ts

            },
          },
        ],
        [
          {
-            name: 'header_timestamp_format',
+            name: 'force_timestamp_formatting',


Are we renaming existing config props in use? Might need a db migration script if that's the case.

Good point, done

villebro

LGTM - tested to be compatible with the old metadata in all cases that I was able to come up with.

superset/migrations/versions/fe23025b9441_rename_big_viz_total_form_data_fields.py

villebro · 2021-12-13T13:49:02Z

superset/migrations/versions/fe23025b9441_rename_big_viz_total_form_data_fields.py

+            header_format_selector = params.pop("header_format_selector", None)
+            header_timestamp_format = params.pop("header_timestamp_format", None)
+            if header_format_selector:
+                params["time_format"] = header_format_selector
+            if header_timestamp_format:
+                params["force_timestamp_formatting"] = header_timestamp_format


I think these are inverted - shouldn't header_format_selector be migrated to force_timestamp_formatting and header_timestamp_format to time_format?

Thanks for spotting that!

villebro

LGTM - Retested with migrations and before and after look the same now with forced timestamp on BigNumberTotal.

betodealmeida · 2021-12-13T16:31:48Z

superset/migrations/versions/fe23025b9441_rename_big_viz_total_form_data_fields.py

+        except Exception as e:
+            print(e)
+            print(f"Parsing params for slice {slc.id} failed.")


I've seen a couple migrations that succeeded even when exceptions were raised, and they always had to be fixed later to address inconsistent states.

Here, it would be better to raise the exception, so that the migration fails and the admin has the chance to fix the offending charts before trying again. Otherwise, if the migration succeeds they might not even see the message, leaving their system inconsistent, and even if they see the message they need to downgrade and upgrade, which is more work.

Also, can you update your summary confirming that migrations are atomic, have been tested, and share the benchmark numbers? You can use scripts/benchmark_migration.py to get the numbers.

Thanks for the comment! I updated the description and and re-raised the exception

betodealmeida

LGTM, thanks for fixing the migration!

superset/migrations/versions/fe23025b9441_rename_big_viz_total_form_data_fields.py

github-actions · 2021-12-13T21:05:42Z

⚠️ @kgabryje Your base branch master has just also updated superset/migrations.

❗ Please consider rebasing your branch to avoid db migration conflicts.

github-actions · 2021-12-13T22:29:12Z

⚠️ @kgabryje Your base branch master has just also updated superset/migrations.

❗ Please consider rebasing your branch to avoid db migration conflicts.

github-actions · 2021-12-15T09:30:27Z

Ephemeral environment shutdown and build artifacts deleted.

AAfghahi · 2021-12-15T23:16:37Z

Hello! I am pretty sure this is currently breaking Charts and Dashboards for me. When I try to open a dashboard I get:

It fails here: https://github.com/apache/superset/blob/master/superset-frontend/src/visualizations/presets/MainPreset.js#L98

When I comment this out it continues working, but obviously then there are no Big Numbers.

) * chore(explore): Migrate BigNumber to v1 api * Move to echarts * Use Echarts trendline * Fix imports * Fix parsing dates as strings * Add from_dttm and to_dttm to v1 chart response * Fix post processing * Fix timeRangeFixed * Fix tests * Remove from and to dttm from cache * Cleanup date formatting * Fix storybook * Fix missing types * Fix timestamp with timezone * Add types to demo's tsconfig * bug fix * fix import * Fix cypress tests * add sort * add resample to handle missing values properly * Sync ChartDataResponseResult schema with ts interface * Lint fix * Add migration * Fix migration * Remove pass * Re-raise the exception in migration * Typo fix * Update revision Co-authored-by: Ville Brofeldt <ville.v.brofeldt@gmail.com>

kgabryje requested a review from villebro November 30, 2021 08:22

pull-request-size bot added size/L size/XL size/XXL and removed size/L size/XL labels Nov 30, 2021

kgabryje requested review from zhaoyongjie and rusackas November 30, 2021 11:02

kgabryje changed the title ~~[WIP] chore(explore): Migrate BigNumber to v1 api~~ chore(explore): Migrate BigNumber to v1 api Nov 30, 2021

kgabryje changed the title ~~chore(explore): Migrate BigNumber to v1 api~~ chore(explore): Migrate BigNumber to v1 api [ID-28][ID-55] Nov 30, 2021

villebro mentioned this pull request Dec 1, 2021

Time-series line chart errors with data containing "Nan" value #17224

Closed

3 tasks

kgabryje force-pushed the fix/bignumber-timestamp branch from 14f2ffb to 36bdfbc Compare December 10, 2021 12:05

rusackas requested a review from ktmud December 10, 2021 23:57

villebro reviewed Dec 11, 2021

View reviewed changes

ktmud reviewed Dec 11, 2021

View reviewed changes

villebro approved these changes Dec 13, 2021

View reviewed changes

kgabryje force-pushed the fix/bignumber-timestamp branch from c994d96 to 9267afc Compare December 13, 2021 12:29

kgabryje requested a review from a team as a code owner December 13, 2021 13:26

villebro reviewed Dec 13, 2021

View reviewed changes

villebro approved these changes Dec 13, 2021

View reviewed changes

betodealmeida requested changes Dec 13, 2021

View reviewed changes

betodealmeida approved these changes Dec 13, 2021

View reviewed changes

superset/migrations/versions/fe23025b9441_rename_big_viz_total_form_data_fields.py Show resolved Hide resolved

kgabryje added 3 commits December 14, 2021 11:32

chore(explore): Migrate BigNumber to v1 api

4a935cc

Move to echarts

fc3e98f

Use Echarts trendline

00ae74c

kgabryje and others added 22 commits December 14, 2021 11:32

Fix post processing

0916313

Fix timeRangeFixed

a70ab9d

Fix tests

00d76ec

Remove from and to dttm from cache

2bde10f

Cleanup date formatting

dc7bf33

Fix storybook

caf6c7e

Fix missing types

6e7dbfc

Fix timestamp with timezone

2c3cc66

Add types to demo's tsconfig

2402221

bug fix

0545cca

fix import

29bb524

Fix cypress tests

41f30bc

add sort

0196cda

add resample to handle missing values properly

3b7ed7b

Sync ChartDataResponseResult schema with ts interface

28381df

Lint fix

add6d28

Add migration

129df69

Fix migration

a75e398

Remove pass

f04f2a6

Re-raise the exception in migration

ad16153

Typo fix

26ed2c0

Update revision

944ae17

kgabryje force-pushed the fix/bignumber-timestamp branch from 8a10fd0 to 944ae17 Compare December 14, 2021 10:36

kgabryje merged commit 124af4c into apache:master Dec 15, 2021

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.5.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(explore): Migrate BigNumber to v1 api [ID-28][ID-55] #17587

chore(explore): Migrate BigNumber to v1 api [ID-28][ID-55] #17587

kgabryje commented Nov 30, 2021 •

edited

Loading

codecov bot commented Nov 30, 2021 •

edited

Loading

kgabryje commented Dec 1, 2021

github-actions bot commented Dec 1, 2021

villebro left a comment

villebro Dec 11, 2021

ktmud left a comment

ktmud Dec 11, 2021

kgabryje Dec 13, 2021

villebro left a comment

villebro Dec 13, 2021

kgabryje Dec 13, 2021

villebro left a comment

betodealmeida Dec 13, 2021

kgabryje Dec 13, 2021

betodealmeida left a comment

github-actions bot commented Dec 13, 2021

github-actions bot commented Dec 13, 2021

github-actions bot commented Dec 15, 2021

AAfghahi commented Dec 15, 2021

chore(explore): Migrate BigNumber to v1 api [ID-28][ID-55] #17587

chore(explore): Migrate BigNumber to v1 api [ID-28][ID-55] #17587

Conversation

kgabryje commented Nov 30, 2021 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

codecov bot commented Nov 30, 2021 • edited Loading

Codecov Report

kgabryje commented Dec 1, 2021

github-actions bot commented Dec 1, 2021

villebro left a comment

Choose a reason for hiding this comment

villebro Dec 11, 2021

Choose a reason for hiding this comment

ktmud left a comment

Choose a reason for hiding this comment

ktmud Dec 11, 2021

Choose a reason for hiding this comment

kgabryje Dec 13, 2021

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

villebro Dec 13, 2021

Choose a reason for hiding this comment

kgabryje Dec 13, 2021

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

betodealmeida Dec 13, 2021

Choose a reason for hiding this comment

kgabryje Dec 13, 2021

Choose a reason for hiding this comment

betodealmeida left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 13, 2021

github-actions bot commented Dec 13, 2021

github-actions bot commented Dec 15, 2021

AAfghahi commented Dec 15, 2021

kgabryje commented Nov 30, 2021 •

edited

Loading

codecov bot commented Nov 30, 2021 •

edited

Loading