Skip to content

Add refit parameter into backtest #1154

Closed
Mr-Geekman opened this issue Mar 6, 2023 · 1 comment
Closed

Add refit parameter into backtest #1154

Mr-Geekman opened this issue Mar 6, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@Mr-Geekman
Copy link
Contributor

Mr-Geekman commented Mar 6, 2023

🚀 Feature Request

After adding possibility to forecast on future with gap, we can modify our backtest to work with this feature.

Proposal

Add parameter refit into BasePipeline.backtest.

Possible values:

  • True (default)
    • fit on every fold
    • all folds should be processed in parallel (like now)
  • False
    • fit only on the earliest fold
    • first fold is processed separately and pipeline is fit
    • consecutive folds should be processed in parallel
  • value: int
    • refit pipeline every value folds starting from the earliest
    • in every group of folds that uses the same pipeline we should work like with refit=False

How to make this change:

  • Extract logic of parallel run of folds from BasePipeline.backtest into a separate method
  • Modify BasePipeline._run_fold to return fitted pipeline and accept fitted pipeline
  • Implement logic with fitting folds and making predictions
    • We can interpret refit=True as refit: int = 1, because we consider each fold to be in the separate group
    • We can interpret refit=False as refit: int = n_folds, because we consider all folds to be in the same group

To discuss: what to do with FoldMask?

Test cases

  • Fix tests for changed methods
  • Check that n_jobs doesn't influence the results with different refit
  • Check that pipeline is working with models and transforms that supports inference mode
  • Check that pipeline doesn't work with models and transforms that doesn't support inference mode

Additional context

No response

@Mr-Geekman Mr-Geekman added the enhancement New feature or request label Mar 6, 2023
@Mr-Geekman Mr-Geekman added this to the Inference 2.1 milestone Mar 6, 2023
@Mr-Geekman Mr-Geekman self-assigned this Mar 7, 2023
@Mr-Geekman
Copy link
Contributor Author

Closed by #1159.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
Status: Done
Development

No branches or pull requests

1 participant