Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Questions about the information leak #28

Open
Dutch-voyage opened this issue Dec 19, 2024 · 0 comments
Open

Questions about the information leak #28

Dutch-voyage opened this issue Dec 19, 2024 · 0 comments

Comments

@Dutch-voyage
Copy link

Hi, dear authors, I appreciate your thorough work on exploring design choices of time series foundation models. However, noticing the significant improvement in comparison with baseline models, I doubt that the setting of TS prompt might influence the validity of the experiment results.
Quite often, time series forecasting models use the statistics of train set (means and standard deviations) to normalize inputs. Using statistics of the validation set and test set is considered to bring about information leaks. A recent work on NIPS 2024 has made a similar mistake, mentioned in ForestsKing/GLAFF#5 and other posts in their github repo issues.
In this work, TS prompts (manually extracted features) are computed on the whole validation set and test set, which provide models with information of whole val/test set in evaluation.
(see promt_generation() in ltsm/prompt_bank/stat-prompt
/prompt_generate_split.py)
image
It would be very appreciated if the authors could provide experiment results to address this concern. E.g. using TS prompts over a past period instead of the whole val/test set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant