This small project was performed as an assignment for the course of Artificial Neural Networks & Deep Learning and it's part of the exam.
The goal of the project is to predict future samples of a multivariate time series.
The TimeSeries Forecasting problem is characterized by analysing a sequence of data showing past interactions and behaviour, and eventually exploiting recurrencies and seasonalities the model can make a reliable prediction of how will be the future for a fixed number of timesteps
In the end, what proved to make a significant difference was the implementation of an Attention layer. The model is as follows:
-
Bidirectional LSTM, 512 units and return sequences + Dropout(p=0.3)
-
Attention Layer
-
RepeatVector
-
TimeDistributed Dense, 512 units + Dropout(p=0.3)
-
Bidirectional LSTM, 512 units and return sequences + Dropout(p=0.3)
-
Attention Layer
-
Dense + Reshape as seen in class to correctly format output tensor
Hyperparameters: Window: 800 | Stride: 5 | Telescope: 144 | Batch Size = 128
We tried to combine this Attention Layer with various models, using different approaches
and different architectures but in the end this model listed here was the best performing
one on the leaderboard, with a score of 3.6204 RMSE.
Group Name: Gamma