Menu
About Myself Introduction Data Sources Data Visualization Exploratory Data Analysis ARMA/ARIMA/SARIMA Models Financial Time Series Models (ARCH/GARCH) Deep Learning for TS Conclusions

Deep Learning for TS

Inspect Data

Solar Power	Gas
(Click to see a larger view.)	(Click to see a larger view.)

Both daily solar power production of solar panels and daily used gas show strong seasonality, with strong cyclic behavior with a period of about one year. However, the peaks and nadirs of daily used gas are opposite to solar power production. It seems there were more solar power production in summers and less in winters, which means that warm days are likely to produce more solar power. On the contrary, more gas were used in winters, mainly because people used more gas to produce heat and keep warm in winters.

Simple RNN Models

	Solar Power	Gas
Training and validation loss	(Click to see a larger view.)	(Click to see a larger view.)
Predictions	(Click to see a larger view.)	(Click to see a larger view.)

The above plots show the training and validation mean absolute error or loss of the model. For solar power data, the validation loss is larger than training loss, which means this model may be overfitting. For gas data, the training loss is very small where most of them are aroung 1.3.

LSTM Models

	Solar Power	Gas
Training and validation loss	(Click to see a larger view.)	(Click to see a larger view.)
Predictions	(Click to see a larger view.)	(Click to see a larger view.)

The LSTM models have worse predictions than simple RNN models. In addition, for solar power data, the LSTM model may also be overfitting. For gas data, the validation error and training error get close even overlap in the end.

GRU Models

	Solar Power	Gas
Training and validation loss	(Click to see a larger view.)	(Click to see a larger view.)
Predictions	(Click to see a larger view.)	(Click to see a larger view.)

The performances of GRU models are similar to LSTM models, with test error around 3.5 for solar power data and 1.5 for gas data. The prediction made for test set shows great consistency with the true value, though with a bit lag. What is beyond expectation is that the model seems to have better performance on the test set in the end for gas data.

RNN with L1 Regularization

	Solar Power	Gas
Training and validation loss	(Click to see a larger view.)	(Click to see a larger view.)
Predictions	(Click to see a larger view.)	(Click to see a larger view.)

The simple RNN models with L1 regularization have a little better performaces than the original models. However, after regularizing the model, the training result looks less stable for solar power data.

RNN with L2 Regularization

	Solar Power	Gas
Training and validation loss	(Click to see a larger view.)	(Click to see a larger view.)
Predictions	(Click to see a larger view.)	(Click to see a larger view.)

The simple RNN models with L2 regularization have similar performaces to RNN with L1 regularization. For gas data, the model does not get overfitted since test error is very close to the training performance, while for solar power data, the model has big time lag when making predictions for the future.

LSTM with L1 Regularization

	Solar Power	Gas
Training and validation loss	(Click to see a larger view.)	(Click to see a larger view.)
Predictions	(Click to see a larger view.)	(Click to see a larger view.)

LSTM has more complicated model structure and it took more time to train on each epoch. The LSTM models with L1 regularization have a little better performaces than the original models. For gas data, the model converges at around 50 during the progress.

LSTM with L2 Regularization

	Solar Power	Gas
Training and validation loss	(Click to see a larger view.)	(Click to see a larger view.)
Predictions	(Click to see a larger view.)	(Click to see a larger view.)

The LSTM models with L2 regularization have similar performaces to LSTM with L1 regularization. The training error is always slightly lower than the validation error for solar power data, while for gas data, the prediction for test set is close to the true value.

GRU with L1 Regularization

	Solar Power	Gas
Training and validation loss	(Click to see a larger view.)	(Click to see a larger view.)
Predictions	(Click to see a larger view.)	(Click to see a larger view.)

Compared to the other 2 models, GRU converges relatively slow. The GRU models with L1 regularization have a little worse performaces than the original models. The variances of training and validation performances get higher as epoch increases.

GRU with L2 Regularization

	Solar Power	Gas
Training and validation loss	(Click to see a larger view.)	(Click to see a larger view.)
Predictions	(Click to see a larger view.)	(Click to see a larger view.)

The GRU models with L2 regularization have similar performaces to GRU with L1 regularization. After L2 regularization, the model validation progresses become more unstable. For solar power data, the test performance is bad since great lag can be observed between the test prediction and the true value.

Compared to ARIMA/SARIMA models

	Solar Power	Gas
ARIMA models	(Click to see a larger view.)	(Click to see a larger view.)
SARIMA models	(Click to see a larger view.)	(Click to see a larger view.)

The RNN models have slightly smaller training mean absolute error compared to the ARIMA/SARIMA models. The RNN models have closer prediction to the true value and more robust predicative power. However, it takes much more time to train a neural network and tune the hyperparams to avoid overfitting. Besides, the neural networks lack good interpretability.