docs/ep_deep.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ep_deep.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AQxZCzK8070V",
        "colab_type": "text"
      },
      "source": [
        "Supplement to Part 3 of **time series forecasting with energy**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pa4tolGd086-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# LOAD THE REPOSITORY\n",
        "# if you are working from outside the repository\n",
        "# this happens if you use colab like I do\n",
        "!git clone https://github.com/sandeshbhatjr/energy-prediction.git\n",
        "!pip install -U --quiet pandas tables numpy"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uxjSPgsSUmsD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "df = pd.read_hdf('energy-prediction/data/clean_german_df', key='df_with_load_and_gen')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wi9OOkLI1EFe",
        "colab_type": "text"
      },
      "source": [
        "# Basic Deep Learning Models\n",
        "\n",
        "Deep learning has shown promising results in many areas of machine learning problems, and it is natural to wonder if it can have any significant impact in the time-series forecasting arena. In the image and NLP domains, convNets and LSTMs respectively reign supreme, and the question is- what is the relevant architecture for a SOTA time-series prediction?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NdIY8COe1FLZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%tensorflow_version 2.x\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras.layers import SimpleRNN\n",
        "from tensorflow.keras.layers import GRU\n",
        "from tensorflow.keras.layers import LSTM\n",
        "from tensorflow.keras.layers import BatchNormalization"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JJhLRwGg29v5",
        "colab_type": "text"
      },
      "source": [
        "**Some preprocessing for NNs**\n",
        "\n",
        "The preprocessing is pretty standard: we scale all continuous features by variance since NNs are extremely sensitive to scale, and one-hot encode the categoric variables."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zzvmocP23G2w",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# NNs are sensitive to range, so rescale\n",
        "mean = german_df_with_load_and_gen['Day Ahead Price'].mean()\n",
        "var = german_df_with_load_and_gen['Day Ahead Price'].var()\n",
        "german_df_with_load_and_gen['Day Ahead Price (rescaled)'] = (german_df_with_load_and_gen['Day Ahead Price'] - mean) / var\n",
        "\n",
        "mean = german_df_with_load_and_gen['Total Load'].mean()\n",
        "var = german_df_with_load_and_gen['Total Load'].var()\n",
        "german_df_with_load_and_gen['Total Load (rescaled)'] = (german_df_with_load_and_gen['Total Load'] - mean) / var\n",
        "\n",
        "mean = german_df_with_load_and_gen['Total Generation'].mean()\n",
        "var = german_df_with_load_and_gen['Total Generation'].var()\n",
        "german_df_with_load_and_gen['Total Generation (rescaled)'] = (german_df_with_load_and_gen['Total Generation'] - mean) / var"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jQd57_CK3M9X",
        "colab_type": "text"
      },
      "source": [
        "We have a few categorical variables in our data: deep models will need them to be one-hot encoded for use."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SlCQPRSM3Nr0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ohe_holiday_df = pd.get_dummies(german_df_with_load_and_gen[['DE', 'LU']])\n",
        "ohe_bidding_zones_df = pd.get_dummies(german_df_with_load_and_gen['Bidding Zone'])\n",
        "\n",
        "proc_german_df = pd.concat([german_df_with_load_and_gen, ohe_bidding_zones_df], axis=1)\n",
        "proc_german_df2 = pd.concat([proc_german_df, ohe_holiday_df], axis=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3sUzABfP3Qbe",
        "colab_type": "text"
      },
      "source": [
        "Finally, we split the dataset into a a training and validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZI2hocD_3S81",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_df, val_df = test_train_timesplit(proc_german_df2, train_size=0.9, test_size=0.1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C5hAX-0c3WET",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# convert each hour to multi-sequential form\n",
        "price_train_X_data = extractHourlyData(train_df, ['Day Ahead Price (rescaled)', 'Day of Week'] + sorted(ohe_bidding_zones_df) + sorted(ohe_holiday_df))\n",
        "price_train_y_data = extractHourlyData(train_df, ['Day Ahead Price (rescaled)'])\n",
        "price_val_X_data = extractHourlyData(val_df, ['Day Ahead Price (rescaled)', 'Day of Week'] + sorted(ohe_bidding_zones_df) + sorted(ohe_holiday_df))\n",
        "price_val_y_data = extractHourlyData(val_df, ['Day Ahead Price (rescaled)'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sg2lQqJK3z_U",
        "colab_type": "text"
      },
      "source": [
        "## Feedforward NN\n",
        "The simplest deep model that we can come up with is a feed-forward neural network that takes into account the time and date of the price, and the bidding zone. This does not take into account the recent history of the prices, hence is not capable of noticing local trends in price based on previous day prices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hK-4G_bC5TXx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X_train = train_df[[\n",
        "                  'Day', \n",
        "                  'Month', \n",
        "                  'Year', \n",
        "                  'Hour', \n",
        "                  'Day of Week', \n",
        "                  'Daylight Savings Time'] \\\n",
        "                   + sorted(ohe_bidding_zones_df) + sorted(ohe_holiday_df)].to_numpy()\n",
        "y_train = train_df['Day Ahead Price (rescaled)'].to_numpy()\n",
        "\n",
        "# define model\n",
        "ffmodelA = Sequential()\n",
        "ffmodelA.add(Dense(80))\n",
        "ffmodelA.add(Dense(30))\n",
        "ffmodelA.add(Dense(1))\n",
        "\n",
        "# train the model\n",
        "ffmodelA.compile(loss='mae', optimizer='Adam')\n",
        "ffmodelA.fit(X_train, y_train, epochs=50, validation_split=0.01, verbose=0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX1hlRRc34CJ",
        "colab_type": "text"
      },
      "source": [
        "## RNN\n",
        "\n",
        "First up, we will tackle standard architectures for sequential data: RNN, GRU and LSTM. `Keras` makes it a relatively easy task to set them up, and see how they perform.\n",
        "\n",
        "RNN is the most basic one amongst them, though one shortcoming may be that it only takes into account the previous few terms in the sequence; so weekly, monthly and annual patterns are probably not accounted for by this model. But enough of speculations- it's time to see how it actually performs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aRiIyU8i5V5v",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# generate windowed datasets\n",
        "window_size = 30\n",
        "\n",
        "generator = TimeseriesGenerator(\n",
        "    price_train_X_data, \n",
        "    price_train_y_data, \n",
        "    length = window_size, \n",
        "    sampling_rate = 1,\n",
        "    batch_size = 32,\n",
        "    shuffle = True)\n",
        "\n",
        "val_generator = TimeseriesGenerator(\n",
        "    price_val_X_data, \n",
        "    price_val_y_data, \n",
        "    length = window_size, \n",
        "    sampling_rate = 1,\n",
        "    batch_size = 32)\n",
        "\n",
        "rmodelA = Sequential()\n",
        "rmodelA.add(SimpleRNN(20))\n",
        "rmodelA.add(Dense(24, activation='linear'))\n",
        "\n",
        "rmodelA.compile(loss='mae', optimizer='Adam')\n",
        "rmodelA_history = rmodelA.fit(generator, validation_data=val_generator, epochs=5000, verbose=0)\n",
        "\n",
        "plt.figure(figsize=(15,8))\n",
        "plt.plot(rmodelA_history.history['loss'])\n",
        "plt.plot(rmodelA_history.history['val_loss'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J_GnR_f_5Z-z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "rmodelA_error = error_analysis(lambda X: rmodelA.predict(X))\n",
        "rmodelA_error.total_smape(val_generator)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xc1cVbqB4frU",
        "colab_type": "text"
      },
      "source": [
        "## Some attention: LSTM and GRU"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3n80hxk4526C",
        "colab_type": "text"
      },
      "source": [
        "### GRU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JR7O5yW-5xmq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "window_size = 7\n",
        "\n",
        "generator = TimeseriesGenerator(\n",
        "    price_train_X_data, \n",
        "    price_train_y_data, \n",
        "    length = window_size, \n",
        "    sampling_rate = 1,\n",
        "    batch_size = 16)\n",
        "\n",
        "val_generator = TimeseriesGenerator(\n",
        "    price_val_X_data, \n",
        "    price_val_y_data, \n",
        "    length = window_size, \n",
        "    sampling_rate = 1,\n",
        "    batch_size = 16)\n",
        "\n",
        "rmodelB = Sequential()\n",
        "rmodelB.add(GRU(100, dropout=0.3, return_sequences=True))\n",
        "rmodelB.add(BatchNormalization())\n",
        "rmodelB.add(GRU(100, dropout=0.3, return_sequences=True))\n",
        "rmodelB.add(BatchNormalization())\n",
        "rmodelB.add(GRU(50, dropout=0.2, return_sequences=True))\n",
        "rmodelB.add(BatchNormalization())\n",
        "rmodelB.add(GRU(24))\n",
        "\n",
        "rmodelB.compile(loss='mae', optimizer='Adam')\n",
        "\n",
        "rmodelB_history = rmodelB.fit(generator, validation_data=val_generator, epochs=100, verbose=0)\n",
        "\n",
        "plt.figure(figsize=(15,8))\n",
        "plt.plot(rmodelB_history.history['loss'])\n",
        "plt.plot(rmodelB_history.history['val_loss'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OfjH0f2o5yVk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "rmodelB_error = error_analysis(lambda X: rmodelB.predict(X))\n",
        "rmodelB_error.total_smape(val_generator)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XrZ_Px7g6PkU",
        "colab_type": "text"
      },
      "source": [
        "### LSTM"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cgF7GeVE51Qj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# generate windowed datasets\n",
        "window_size = 3\n",
        "\n",
        "generator = TimeseriesGenerator(\n",
        "    price_train_X_data, \n",
        "    price_train_y_data, \n",
        "    length = window_size, \n",
        "    sampling_rate = 1,\n",
        "    batch_size = 4)\n",
        "\n",
        "val_generator = TimeseriesGenerator(\n",
        "    price_val_X_data, \n",
        "    price_val_y_data, \n",
        "    length = window_size, \n",
        "    sampling_rate = 1,\n",
        "    batch_size = 4)\n",
        "\n",
        "rmodelC = Sequential()\n",
        "# rmodelC.add(LSTM(200, dropout=0.1, return_sequences=True))\n",
        "# rmodelC.add(BatchNormalization())\n",
        "rmodelC.add(LSTM(50, dropout=0.1, return_sequences=True))\n",
        "rmodelC.add(BatchNormalization())\n",
        "rmodelC.add(LSTM(50, dropout=0.1, return_sequences=True))\n",
        "rmodelC.add(BatchNormalization())\n",
        "rmodelC.add(LSTM(24))\n",
        "\n",
        "rmodelC.compile(loss='mae', optimizer='Adam')\n",
        "\n",
        "rmodelC_history = rmodelC.fit(generator, validation_data=val_generator, epochs=100, verbose=0)\n",
        "\n",
        "plt.figure(figsize=(15,8))\n",
        "plt.plot(rmodelC_history.history['loss'])\n",
        "plt.plot(rmodelC_history.history['val_loss'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SrxCn-SU6ZYN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "rmodelC_error = error_analysis(lambda X: rmodelC.predict(X))\n",
        "rmodelC_error.total_smape(val_generator)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jnABWDmr5ks8",
        "colab_type": "text"
      },
      "source": [
        "## Convolutional LSTM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mcPfu4_X3pGp",
        "colab_type": "text"
      },
      "source": [
        "# Inception Time\n",
        "\n",
        "InceptionNet is one of the SOTA models in image classification, which uses a very wide CNN with the help of depthwise 1D-convolution to improve the accuracy of the model. The ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bESX3Ih343Aq",
        "colab_type": "text"
      },
      "source": [
        "# LSTM-MS Net\n",
        "\n",
        "The SOTA approach for multi-seasonal sequences is supposed to be the LSTM-MSNet model. The idea here is quite simple; deseasonalise the time-series, then feed into a deep neural network, and then seasonalise it again."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wFpds6Ud_VaE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q8ZWg2kb1esr",
        "colab_type": "text"
      },
      "source": [
        "# Bibliography\n",
        "\n",
        "**[CHN11]** Sven F. Crone, Michèle Hibon, Konstantinos Nikolopoulos, *Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction*, International Journal of Forecasting\n",
        "Volume 27, Issue 3, July–September 2011, Pages 635-660\n",
        "\n",
        "**[JFB17]** J. Lago, F. D. Ridder, B. D. Schutter, *Forecasting spot electricity prices: Deep learning approaches and empirical comparison of traditional algorithms*, doi:https://doi.org/10.1016/j.apenergy.2018.02.069 (2017).\n",
        "\n",
        "**[BBH19]** K. Bandara, C. Bergmeir, H. Hewamalage, *LSTM-MSNet: Leveraging Forecasts on Sets of\n",
        "Related Time Series with Multiple Seasonal Patterns* arXiv:1909.04293, Sep. (2019). [https://arxiv.org/pdf/1909.04293.pdf]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wu5eebbd12F2",
        "colab_type": "text"
      },
      "source": [
        "Supplements:  \n",
        "[The Uber approach and M4 winner: ES-RNN]()  \n",
        "\n",
        "Next part: Analysis and discussion `WIP`"
      ]
    }
  ]
}