Optimisation of historical forecast for regression models

[madtoinou]

Fixes #1233

Summary

Reduce historical_forecasts() runtime for all the RegressionModels using some tricks (enforce retrain=False, use boundaries of the "forecastable indexes" instead of a range, predictions are vectorized with stride).

Other Information

forecast_horizon > model.output_chunk_length is not supported at the moment because it would require auto-regression and num_samples > 1 needs a bit of work before being fully supported.

[EDIT] : Some speed up statistics, also included some comparison between the refactored and "legacy" implementation.

# parameters
multi_models = [True, False]
multivariate = [True, False]
forecast_horizon = [1,7]
stride = 1
start = [700, 800, 900, 990]
length_ts = 1000

ts_univariate = tg.linear_timeseries(start_value=1, end_value=length_ts, length=length_ts)
ts_multivariate = ts_univariate.stack(tg.sine_timeseries(length=length_ts))

# two models, to test forecast_horizon different or equal to output_chunk_length
model1 = LinearRegressionModel(lags=3, output_chunk_length=forecast_horizon
model2 = LinearRegressionModel(lags=3, output_chunk_length=7)

# loop iterating on all the parameters
t0 = time.time()
for i in range(replicates):
    hist_fct = model.historical_forecasts(
       series=ts,
        start=start,
        stride=stride,
        last_points_only=last_points_only,
        forecast_horizon=forecast_horizon,
        num_samples=num_samples,
        retrain=False,
        enable_optimisation=False
        )
t1 = time.time()
for i in range(replicates):
    opti_hist_fct =  model.historical_forecasts(
        series=ts,
        start=start,
        stride=stride,
        last_points_only=last_points_only,
        forecast_horizon=forecast_horizon,
        num_samples=num_samples,
        retrain=False,
        enable_optimisation=True
t2 = time.time()

Refactorization of the original method did not affect the performance, but it should be easier to read.

The ratio on the y axis is (t1-t0)/(t2-t1)

Speed up when last_points_only=True

When the conditions are met, the gain scales with the length of the forecasted period.

Speed up when last_points_only=False

The bottleneck is the creation of the returned TimeSeries, the gain are less significant.

[codecov-commenter]

Codecov Report

Patch coverage: 92.30% and project coverage change: -0.11% ⚠️

Comparison is base (7f64c92) 93.82% compared to head (ce508ea) 93.72%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1885      +/-   ##
==========================================
- Coverage   93.82%   93.72%   -0.11%     
==========================================
  Files         128      131       +3     
  Lines       12475    12646     +171     
==========================================
+ Hits        11705    11852     +147     
- Misses        770      794      +24     

Files Changed	Coverage Δ
darts/utils/__init__.py	100.00% <ø> (ø)
darts/utils/timeseries_generation.py	96.15% <ø> (ø)
darts/utils/utils.py	80.58% <ø> (-11.09%)	⬇️
darts/models/forecasting/forecasting_model.py	95.21% <85.00%> (+0.20%)	⬆️
...orical_forecasts/optimized_historical_forecasts.py	88.46% <88.46%> (ø)
darts/models/forecasting/regression_model.py	95.23% <93.75%> (-0.13%)	⬇️
darts/utils/historical_forecasts/utils.py	94.73% <94.73%> (ø)
darts/dataprocessing/encoders/encoder_base.py	94.69% <100.00%> (+0.02%)	⬆️
darts/utils/historical_forecasts/__init__.py	100.00% <100.00%> (ø)

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dennisbader]

Great PR, thanks a lot and congratulations @madtoinou! This will give such a boost to historical forecasting and backtesting for RegressionModels! 🚀

I left some comments and suggestions regarding what we discussed offline, to add some more information on specific parts, etc.

[dennisbader]

can you explain why this is not required for ocl < forecast_horizon?

[madtoinou]

it's the other way around, forecast_horizon < ocl, and it's because these timestamps will be covered by the last prediction of length output_chunk_length. By leaving them, the prediction will go too far.

[dennisbader]

this part is tricky to understand, could you add some more comments that describe how the output of _predict_and_sample is different for multi_model True/False, and share a bit more info for the slicing/reshaping.

We will be thankful in the future :)

[madtoinou]

Updated the comments, let me know if it's enough

[dennisbader]

Very, very cool stuff @madtoinou 🚀

I really only had some minor suggestions :)

• add some tests
• add support for optimization with encoders (or at least make it non-optimizable if they use encoders)

We can

[dennisbader]

in the current implemenetation models using encoders are not yet optimizable.
Do you think we could add support for this?

Edit: I'm thinking about adding something like a generate_fit_predict_encodings which would make this a bit easier. Maybe we could drop optimization support for encoders until then.

Edit 2: I added the generate_fit_predict_encodings in #1925. Would be cool to merge that one and then add the support for optimization with encodings here as well :)

[madtoinou]

Good catch, I tend to forget about the encoders... Thank you for implementing this, I will adjust this PR as soon as the other one is merged.

[dennisbader]

🚀 Super nice and great job @madtoinou 👏
We're ready to merge now once unit tests have passed (i adapted slightly to reduce testing time).