feat: add optional prophet forecasting functionality to chart data api #10324

villebro · 2020-07-15T16:59:42Z

SUMMARY

This PR adds Facebook Prophet as an optional dependency to enable time series forecasting in viz plugins. For each series (=column), it will add three new columns with the following suffix:

__yhat: forecast
__yhat_lower: lower confidence level
__yhat_upper: uper confidence level
In addition, it extrapolates as many time grain steps into the future as have been specified in the periods parameter. For instance, 52 periods with weekly time grains will create a forecast of 52 weekly observations = 1 year into the future. The forecast along with confidence intervals will be made available for the whole time period, while the observations (without a suffix) will be null for the forecasted dates. For more details please refer to the excellent tutorial by the Prophet team.

This treats each series as an individual time series model, hence will be computationally very expensive in the case of multiple columns. However, since the new chart data endpoint caches the end result, only the first calculation will be slow.

I've tried to add as many different tests as possible (schema validation, post processing, e2e API test) to ensure functionality. The end-to-end tests require fbprophet to be installed, however I decided against adding it to requirements-dev.txt, as it will make CI much more sluggish. However, anyone interested in developing this feature further can install the dependency locally and activate the e2e tests that are otherwise skipped.

TEST PLAN

CI + new tests

ADDITIONAL INFORMATION

willbarrett

A couple code structure suggestions, but other than that looks good.

willbarrett · 2020-07-15T21:19:02Z

superset/utils/pandas_postprocessing.py

@@ -544,3 +563,105 @@ def contribution(
    if temporal_series is not None:
        contribution_df.insert(0, DTTM_ALIAS, temporal_series)
    return contribution_df
+
+
+def prophet(  # pylint: disable=too-many-arguments,too-many-locals


Could this function be broken up? Generally comments in a large function like this are a good place to draw divisions

These unfortunately need to be one single function with too-many-arguments, as they are called based on instructions provided in the chart data request. But you're right, too-many-locals doesn't need to be part of the design here, so will try to break out some internal logic.

willbarrett · 2020-07-15T21:54:40Z

tests/pandas_postprocessing_tests.py

@@ -508,3 +509,119 @@ def test_contribution(self):
        self.assertListEqual(df.columns.tolist(), ["a", "b"])
        self.assertListEqual(series_to_list(column_df["a"]), [0.25, 0.75])
        self.assertListEqual(series_to_list(column_df["b"]), [0.1, 0.9])
+
+    def test_prophet_incorrect_values(self):


Breaking this up into a test for each incorrect value type may make debugging test failures easier down the road.

Good idea, will break this up.

villebro · 2020-07-20T09:42:11Z

@willbarrett this is ready for re-review. I split out the training part into a private method to make the main method simpler (was able to drop the disable too-many-locals), and also made each unit test a separate function. However, the too-many-arguments can't really be changed due to the nature of how post processing operations are called from the chart data API.

willbarrett

LGTM, one suggestion for avoiding another pylint disable, but it's non-blocking.

enjineerMan · 2020-11-06T22:55:57Z

Hi, this might be a dumb question but where can we find this forecasting function in Superset?

villebro · 2020-11-09T08:05:30Z

Hi @enjineerMan this feature is available on master branch and on the forthcoming 0.38 release of Apache Superset. on the new ECharts based timeseries viz type. To enable, you need to make sure the fbprophet package is installed, after which the Predictive Analytics feature can be used in the "Time-series chart" visualization type. Please keep in mind that this functionality is still experimental and has known performance issues that are gradually being worked on.

apache#10324) * feat: add prophet post processing operation * add tests * lint * whitespace * remove whitespace * address comments * add note to UPDATING.md

wakilkhan96 · 2021-03-12T17:28:44Z

UPDATING.md

@@ -23,6 +23,8 @@ assists people when migrating to a new version.

 ## Next

+* [10324](https://github.com/apache/incubator-superset/pull/10324): Facebook Prophet has been introduced as an optional dependency to add support for timeseries forecasting in the chart data API. To enable this feature, install Superset with the optional dependency `prophet` or directly `pip install fbprophet`.


how to install Superset with the optional dependency prophet? i have already installed superset. Now how to install prophet to my existing superset?

Hi @wakilkhan96

If you haven't managed to solve installing prophet, access the terminal for your superset_app and run the following commands:

docker exec -it superset_app /bin/bash

pip uninstall fbprophet pystan

pip --no-cache-dir install pystan==2.19.1.1

pip install prophet

That should solve that issue.
pystan>=3.0 is currently not supported for prophet that why you should specify that version.

superset-github-bot bot added the preset-io label Jul 15, 2020

pull-request-size bot added the size/L label Jul 15, 2020

villebro force-pushed the villebro/prophet branch 4 times, most recently from c57a8e1 to aeeb919 Compare July 15, 2020 20:15

willbarrett reviewed Jul 15, 2020

View reviewed changes

villebro added 6 commits July 20, 2020 08:20

feat: add prophet post processing operation

8c2f919

add tests

4ff25f5

lint

ad6e4fe

whitespace

5c961ab

remove whitespace

6a61ffd

address comments

f24b785

villebro force-pushed the villebro/prophet branch from 43d24ec to f24b785 Compare July 20, 2020 07:21

add note to UPDATING.md

2bfad7e

villebro requested a review from willbarrett July 20, 2020 09:40

willbarrett approved these changes Jul 20, 2020

View reviewed changes

villebro merged commit 7af8b2b into apache:master Jul 20, 2020

villebro deleted the villebro/prophet branch July 27, 2020 18:20

villebro mentioned this pull request Sep 1, 2020

feat(viz): add ECharts Timeseries chart #10752

Merged

6 tasks

kkalyan mentioned this pull request Sep 29, 2020

Feature Request: Prediction #5300

Closed

iercan mentioned this pull request Dec 1, 2020

adding fbprophet to requirements amancevice/docker-superset#198

Merged

wakilkhan96 reviewed Mar 12, 2021

View reviewed changes

This was referenced Jun 15, 2023

Tests fail for optional prophet forecasting functionality #24406

Closed

Predictive analytics not working out of the box when using docker. #24357

Closed

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.38.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add optional prophet forecasting functionality to chart data api #10324

feat: add optional prophet forecasting functionality to chart data api #10324

villebro commented Jul 15, 2020

willbarrett left a comment

willbarrett Jul 15, 2020

villebro Jul 16, 2020

willbarrett Jul 15, 2020

villebro Jul 16, 2020

villebro commented Jul 20, 2020

willbarrett left a comment

enjineerMan commented Nov 6, 2020

villebro commented Nov 9, 2020

wakilkhan96 Mar 12, 2021

muchemwal Jul 9, 2021 •

edited

Loading

		@@ -23,6 +23,8 @@ assists people when migrating to a new version.

		## Next

		* [10324](https://github.com/apache/incubator-superset/pull/10324): Facebook Prophet has been introduced as an optional dependency to add support for timeseries forecasting in the chart data API. To enable this feature, install Superset with the optional dependency `prophet` or directly `pip install fbprophet`.

feat: add optional prophet forecasting functionality to chart data api #10324

feat: add optional prophet forecasting functionality to chart data api #10324

Conversation

villebro commented Jul 15, 2020

SUMMARY

TEST PLAN

ADDITIONAL INFORMATION

willbarrett left a comment

Choose a reason for hiding this comment

willbarrett Jul 15, 2020

Choose a reason for hiding this comment

villebro Jul 16, 2020

Choose a reason for hiding this comment

willbarrett Jul 15, 2020

Choose a reason for hiding this comment

villebro Jul 16, 2020

Choose a reason for hiding this comment

villebro commented Jul 20, 2020

willbarrett left a comment

Choose a reason for hiding this comment

enjineerMan commented Nov 6, 2020

villebro commented Nov 9, 2020

wakilkhan96 Mar 12, 2021

Choose a reason for hiding this comment

muchemwal Jul 9, 2021 • edited Loading

Choose a reason for hiding this comment

muchemwal Jul 9, 2021 •

edited

Loading