Neural_Nets_Gesture_Recognition_Draft.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gesture Recognition\n",
    "In this group project, you are going to build a 3D Conv model that will be able to predict the 5 gestures correctly. Please import the following libraries to get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "from scipy.misc import imread, imresize\n",
    "import datetime\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set the random seed so that the results don't vary drastically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(30)\n",
    "import random as rn\n",
    "rn.seed(30)\n",
    "from keras import backend as K\n",
    "import tensorflow as tf\n",
    "tf.set_random_seed(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this block, you read the folder names for training and validation. You also set the `batch_size` here. Note that you set the batch size in such a way that you are able to use the GPU in full capacity. You keep increasing the batch size until the machine throws an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_doc = np.random.permutation(open('./Project_data/train.csv').readlines())\n",
    "val_doc = np.random.permutation(open('./Project_data/val.csv').readlines())\n",
    "batch_size = 10 #experiment with the batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_doc)\n",
    "len(val_doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper Functions and our current preprogessing pipeline\n",
    "# pre process ===> conbert_to_grayscale --> normalize_data\n",
    "\n",
    "# Convert to grayscale\n",
    "def convert_to_grayscale(data):\n",
    "    #return np.sum(data/3, axis=3, keepdims=True)\n",
    "    return data.mean(axis=-1,keepdims=1) \n",
    "\n",
    "def normalize_data(data):\n",
    "    return data/127.5-1\n",
    "\n",
    "def preprocess_pipeline(data):\n",
    "    return normalize_data(convert_to_grayscale(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator\n",
    "This is one of the most important part of the code. The overall structure of the generator has been given. In the generator, you are going to preprocess the images as you have images of 2 different dimensions as well as create a batch of video frames. You have to experiment with `img_idx`, `y`,`z` and normalization such that you get high accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "c = 3\n",
    "    \n",
    "def generator(source_path, folder_list, batch_size):\n",
    "    \n",
    "    x = 30 # number of frames\n",
    "    y = 120 # image width\n",
    "    z = 120 # image height\n",
    "    \n",
    "    print( 'Source path = ', source_path, '; batch size =', batch_size)\n",
    "    img_idx = [x for x in range(0,x)] #create a list of image numbers you want to use for a particular video\n",
    "    while True:\n",
    "        t = np.random.permutation(folder_list)\n",
    "        num_batches = len(folder_list)//batch_size # calculate the number of batches\n",
    "        for batch in range(num_batches): # we iterate over the number of batches\n",
    "            batch_data = np.zeros((batch_size,x,y,z,c)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
    "                    \n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
    "                    temp = imresize(image,(120,120))\n",
    "                    temp = normalize_data(temp)\n",
    "                    \n",
    "                    batch_data[folder,idx,:,:,0] = (temp[:,:,0])\n",
    "                    batch_data[folder,idx,:,:,1] = (temp[:,:,1])\n",
    "                    batch_data[folder,idx,:,:,2] = (temp[:,:,2])\n",
    "       \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "            yield batch_data, batch_labels #you yield the batch_data and the batch_labels, remember what does yield do\n",
    "\n",
    "        \n",
    "        # write the code for the remaining data points which are left after full batches\n",
    "        if (len(folder_list) != batch_size*num_batches):\n",
    "            print(\"Batch: \",num_batches+1,\"Index:\", batch_size)\n",
    "            batch_size = len(folder_list) - (batch_size*num_batches)\n",
    "            batch_data = np.zeros((batch_size,x,y,z,c)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
    "                    \n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
    "                    temp = imresize(image,(120,120))\n",
    "                    temp = normalize_data(temp)\n",
    "                    \n",
    "                    batch_data[folder,idx,:,:,0] = (temp[:,:,0])\n",
    "                    batch_data[folder,idx,:,:,1] = (temp[:,:,1])\n",
    "                    batch_data[folder,idx,:,:,2] = (temp[:,:,2])\n",
    "                   \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "            yield batch_data, batch_labels\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note here that a video is represented above in the generator as (number of images, height, width, number of channels). Take this into consideration while creating the model architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# training sequences = 663\n",
      "# validation sequences = 100\n",
      "# epochs = 10\n"
     ]
    }
   ],
   "source": [
    "curr_dt_time = datetime.datetime.now()\n",
    "train_path = './Project_data/train'\n",
    "val_path = './Project_data/val'\n",
    "num_train_sequences = len(train_doc)\n",
    "print('# training sequences =', num_train_sequences)\n",
    "num_val_sequences = len(val_doc)\n",
    "print('# validation sequences =', num_val_sequences)\n",
    "num_epochs = 10# choose the number of epochs\n",
    "print ('# epochs =', num_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "Here you make the model using different functionalities that Keras provides. Remember to use `Conv3D` and `MaxPooling3D` and not `Conv2D` and `Maxpooling2D` for a 3D convolution model. You would want to use `TimeDistributed` while building a Conv2D + RNN model. Also remember that the last layer is the softmax. Design the network in such a way that the model is able to give good accuracy on the least number of parameters so that it can fit in the memory of the webcam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_filters = [8,16,32,64]\n",
    "nb_dense = [1000, 500, 5]\n",
    "input_shape = (30, 120, 120, c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, GRU, Flatten, TimeDistributed, Flatten, BatchNormalization, Activation, Dropout\n",
    "from keras.layers.convolutional import Conv3D, MaxPooling3D\n",
    "from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\n",
    "from keras import optimizers\n",
    "\n",
    "#write your model here\n",
    "\n",
    "model_a = Sequential()\n",
    "\n",
    "model_a.add(Conv3D(nb_filters[0], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 input_shape=input_shape,\n",
    "                 padding='same'))\n",
    "model_a.add(BatchNormalization())\n",
    "model_a.add(Activation('relu'))\n",
    "\n",
    "model_a.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model_a.add(Conv3D(nb_filters[1], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 padding='same'))\n",
    "model_a.add(BatchNormalization())\n",
    "model_a.add(Activation('relu'))\n",
    "\n",
    "model_a.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model_a.add(Conv3D(nb_filters[2], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model_a.add(BatchNormalization())\n",
    "model_a.add(Activation('relu'))\n",
    "\n",
    "model_a.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model_a.add(Conv3D(nb_filters[3], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model_a.add(BatchNormalization())\n",
    "model_a.add(Activation('relu'))\n",
    "\n",
    "model_a.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "#Flatten Layers\n",
    "model_a.add(Flatten())\n",
    "\n",
    "model_a.add(Dense(nb_dense[0], activation='relu'))\n",
    "model_a.add(Dropout(0.5))\n",
    "\n",
    "model_a.add(Dense(nb_dense[1], activation='relu'))\n",
    "model_a.add(Dropout(0.5))\n",
    "\n",
    "#softmax layer\n",
    "model_a.add(Dense(nb_dense[2], activation='softmax'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have written the model, the next step is to `compile` the model. When you print the `summary` of the model, you'll see the total number of parameters you have to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_1 (Conv3D)            (None, 30, 120, 120, 8)   656       \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 30, 120, 120, 8)   32        \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 30, 120, 120, 8)   0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_1 (MaxPooling3 (None, 15, 60, 60, 8)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_2 (Conv3D)            (None, 15, 60, 60, 16)    3472      \n",
      "_________________________________________________________________\n",
      "batch_normalization_2 (Batch (None, 15, 60, 60, 16)    64        \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 15, 60, 60, 16)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_2 (MaxPooling3 (None, 7, 30, 30, 16)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_3 (Conv3D)            (None, 7, 30, 30, 32)     4640      \n",
      "_________________________________________________________________\n",
      "batch_normalization_3 (Batch (None, 7, 30, 30, 32)     128       \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 7, 30, 30, 32)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_3 (MaxPooling3 (None, 3, 15, 15, 32)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_4 (Conv3D)            (None, 3, 15, 15, 64)     18496     \n",
      "_________________________________________________________________\n",
      "batch_normalization_4 (Batch (None, 3, 15, 15, 64)     256       \n",
      "_________________________________________________________________\n",
      "activation_4 (Activation)    (None, 3, 15, 15, 64)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_4 (MaxPooling3 (None, 1, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 3136)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1000)              3137000   \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 500)               500500    \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 5)                 2505      \n",
      "=================================================================\n",
      "Total params: 3,667,749\n",
      "Trainable params: 3,667,509\n",
      "Non-trainable params: 240\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "#rate = 0.00097\n",
    "rate = 0.001\n",
    "optimiser = optimizers.Adam(lr=rate) #write your optimizer\n",
    "model_a.compile(optimizer=optimiser, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "print (model_a.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us create the `train_generator` and the `val_generator` which will be used in `.fit_generator`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = generator(train_path, train_doc, batch_size)\n",
    "val_generator = generator(val_path, val_doc, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'model_init' + '_' + str(curr_dt_time).replace(' ','').replace(':','_') + '/'\n",
    "    \n",
    "if not os.path.exists(model_name):\n",
    "    os.mkdir(model_name)\n",
    "        \n",
    "filepath = model_name + 'model-{epoch:05d}-{loss:.5f}-{categorical_accuracy:.5f}-{val_loss:.5f}-{val_categorical_accuracy:.5f}.h5'\n",
    "\n",
    "checkpoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=False, save_weights_only=False, mode='auto', period=1)\n",
    "\n",
    "LR = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=2, cooldown=1, verbose=1)# write the Reducelronplateau code here\n",
    "callbacks_list = [checkpoint, LR]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `steps_per_epoch` and `validation_steps` are used by `fit_generator` to decide the number of next() calls it need to make."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now fit the model. This will start training the model and with the help of the checkpoints, you'll be able to save the model at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "Source path =  ./Project_data/train ; batch size = 10\n",
      "Source path =  ./Project_data/val ; batch size = 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:20: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:24: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65/67 [============================>.] - ETA: 1s - loss: 6.3312 - categorical_accuracy: 0.2831Batch:  67 Index: 10\n",
      "66/67 [============================>.] - ETA: 0s - loss: 6.2914 - categorical_accuracy: 0.2833"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:44: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:48: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "67/67 [==============================] - 46s 687ms/step - loss: 6.2396 - categorical_accuracy: 0.2791 - val_loss: 4.1348 - val_categorical_accuracy: 0.3700\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2007_29_22.855379/model-00001-6.27555-0.28205-4.13475-0.37000.h5\n",
      "Epoch 2/10\n",
      "67/67 [==============================] - 15s 217ms/step - loss: 3.6477 - categorical_accuracy: 0.2090 - val_loss: 2.2293 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2007_29_22.855379/model-00002-3.64774-0.20896-2.22925-0.22000.h5\n",
      "Epoch 3/10\n",
      "67/67 [==============================] - 16s 243ms/step - loss: 1.9255 - categorical_accuracy: 0.3483 - val_loss: 1.2189 - val_categorical_accuracy: 0.4300\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2007_29_22.855379/model-00003-1.92554-0.34826-1.21889-0.43000.h5\n",
      "Epoch 4/10\n",
      "67/67 [==============================] - 14s 208ms/step - loss: 1.6871 - categorical_accuracy: 0.3383 - val_loss: 1.2734 - val_categorical_accuracy: 0.4400\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2007_29_22.855379/model-00004-1.68707-0.33831-1.27342-0.44000.h5\n",
      "Epoch 5/10\n",
      "67/67 [==============================] - 16s 231ms/step - loss: 1.5593 - categorical_accuracy: 0.3433 - val_loss: 1.3547 - val_categorical_accuracy: 0.3700\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2007_29_22.855379/model-00005-1.55930-0.34328-1.35466-0.37000.h5\n",
      "\n",
      "Epoch 00005: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.\n",
      "Epoch 6/10\n",
      "67/67 [==============================] - 15s 228ms/step - loss: 1.4464 - categorical_accuracy: 0.3582 - val_loss: 1.2704 - val_categorical_accuracy: 0.5000\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2007_29_22.855379/model-00006-1.44639-0.35821-1.27043-0.50000.h5\n",
      "Epoch 7/10\n",
      "67/67 [==============================] - 15s 225ms/step - loss: 1.3834 - categorical_accuracy: 0.3781 - val_loss: 1.1600 - val_categorical_accuracy: 0.5600\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2007_29_22.855379/model-00007-1.38342-0.37811-1.15999-0.56000.h5\n",
      "Epoch 8/10\n",
      "67/67 [==============================] - 15s 219ms/step - loss: 1.3903 - categorical_accuracy: 0.3831 - val_loss: 1.2167 - val_categorical_accuracy: 0.4500\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2007_29_22.855379/model-00008-1.39033-0.38308-1.21668-0.45000.h5\n",
      "Epoch 9/10\n",
      "67/67 [==============================] - 16s 237ms/step - loss: 1.3560 - categorical_accuracy: 0.3682 - val_loss: 1.1717 - val_categorical_accuracy: 0.6100\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2007_29_22.855379/model-00009-1.35600-0.36816-1.17169-0.61000.h5\n",
      "\n",
      "Epoch 00009: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628.\n",
      "Epoch 10/10\n",
      "67/67 [==============================] - 15s 224ms/step - loss: 1.3142 - categorical_accuracy: 0.4478 - val_loss: 1.1629 - val_categorical_accuracy: 0.5700\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2007_29_22.855379/model-00010-1.31420-0.44776-1.16292-0.57000.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f50e4b92208>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_a.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_frames = 30 # number of frames\n",
    "nb_rows = 120 # image width\n",
    "nb_cols = 120 # image height \n",
    "\n",
    "nb_classes = 5\n",
    "nb_channel = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "def new_generator(source_path, folder_list, batch_size):\n",
    "      \n",
    "    print( 'Source path = ', source_path, '; batch size =', batch_size)\n",
    "    img_idx = [x for x in range(0,nb_frames)] #create a list of image numbers you want to use for a particular video\n",
    "    while True:\n",
    "        t = np.random.permutation(folder_list)\n",
    "        num_batches = len(folder_list)//batch_size # calculate the number of batches\n",
    "        for batch in range(num_batches): # we iterate over the number of batches\n",
    "            batch_data = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
    "                    \n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
    "                    temp = imresize(image,(nb_rows,nb_cols))\n",
    "                    #temp = normalize_data(temp)\n",
    "                    temp = preprocess_pipeline(temp)\n",
    "                    \n",
    "                    batch_data[folder,idx] = temp\n",
    "\n",
    "       \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "            yield batch_data, batch_labels #you yield the batch_data and the batch_labels, remember what does yield do\n",
    "\n",
    "        \n",
    "        # write the code for the remaining data points which are left after full batches\n",
    "        if (len(folder_list) != batch_size*num_batches):\n",
    "            print(\"Batch: \",num_batches+1,\"Index:\", batch_size)\n",
    "            batch_size = len(folder_list) - (batch_size*num_batches)\n",
    "            batch_data = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
    "                    \n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
    "                    temp = imresize(image,(nb_rows,nb_cols))\n",
    "                    #temp = normalize_data(temp)\n",
    "                    temp = preprocess_pipeline(temp)\n",
    "                    \n",
    "                    batch_data[folder,idx] = temp\n",
    "                   \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "            yield batch_data, batch_labels\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_61 (Conv3D)           (None, 30, 120, 120, 32)  2624      \n",
      "_________________________________________________________________\n",
      "activation_61 (Activation)   (None, 30, 120, 120, 32)  0         \n",
      "_________________________________________________________________\n",
      "conv3d_62 (Conv3D)           (None, 30, 120, 120, 32)  27680     \n",
      "_________________________________________________________________\n",
      "activation_62 (Activation)   (None, 30, 120, 120, 32)  0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_33 (MaxPooling (None, 10, 40, 40, 32)    0         \n",
      "_________________________________________________________________\n",
      "dropout_45 (Dropout)         (None, 10, 40, 40, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_63 (Conv3D)           (None, 10, 40, 40, 64)    55360     \n",
      "_________________________________________________________________\n",
      "activation_63 (Activation)   (None, 10, 40, 40, 64)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_64 (Conv3D)           (None, 10, 40, 40, 64)    110656    \n",
      "_________________________________________________________________\n",
      "activation_64 (Activation)   (None, 10, 40, 40, 64)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_34 (MaxPooling (None, 4, 14, 14, 64)     0         \n",
      "_________________________________________________________________\n",
      "dropout_46 (Dropout)         (None, 4, 14, 14, 64)     0         \n",
      "_________________________________________________________________\n",
      "flatten_16 (Flatten)         (None, 50176)             0         \n",
      "_________________________________________________________________\n",
      "dense_32 (Dense)             (None, 512)               25690624  \n",
      "_________________________________________________________________\n",
      "dropout_47 (Dropout)         (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_33 (Dense)             (None, 5)                 2565      \n",
      "=================================================================\n",
      "Total params: 25,889,509\n",
      "Trainable params: 25,889,509\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.losses import categorical_crossentropy\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "# Define model\n",
    "model = Sequential()\n",
    "model.add(Conv3D(32, kernel_size=(3, 3, 3), input_shape=(nb_frames,nb_rows,nb_cols,nb_channel), padding='same'))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Conv3D(32, kernel_size=(3, 3, 3), padding='same'))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling3D(pool_size=(3, 3, 3), padding='same'))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Conv3D(64, kernel_size=(3, 3, 3), padding='same'))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Conv3D(64, kernel_size=(3, 3, 3), padding='same'))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling3D(pool_size=(3, 3, 3), padding='same'))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(512, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model.summary()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = new_generator(train_path, train_doc, batch_size)\n",
    "val_generator = new_generator(val_path, val_doc, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "Source path =  ./Project_data/val ; batch size = 10\n",
      "Source path =  ./Project_data/train ; batch size = 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:21: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:25: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65/67 [============================>.] - ETA: 1s - loss: 1.7075 - categorical_accuracy: 0.1938Batch:  67 Index: 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:45: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:49: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "67/67 [==============================] - 49s 725ms/step - loss: 1.7046 - categorical_accuracy: 0.1911 - val_loss: 1.6084 - val_categorical_accuracy: 0.2300\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2007_29_22.855379/model-00001-1.70556-0.19306-1.60844-0.23000.h5\n",
      "Epoch 2/10\n",
      "67/67 [==============================] - 15s 228ms/step - loss: 1.6091 - categorical_accuracy: 0.2040 - val_loss: 1.6064 - val_categorical_accuracy: 0.2300\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2007_29_22.855379/model-00002-1.60910-0.20398-1.60639-0.23000.h5\n",
      "Epoch 3/10\n",
      "67/67 [==============================] - 16s 243ms/step - loss: 1.6104 - categorical_accuracy: 0.1940 - val_loss: 1.4960 - val_categorical_accuracy: 0.2300\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2007_29_22.855379/model-00003-1.61044-0.19403-1.49597-0.23000.h5\n",
      "Epoch 4/10\n",
      "67/67 [==============================] - 16s 239ms/step - loss: 1.5694 - categorical_accuracy: 0.2289 - val_loss: 1.4670 - val_categorical_accuracy: 0.3000\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2007_29_22.855379/model-00004-1.56938-0.22886-1.46697-0.30000.h5\n",
      "Epoch 5/10\n",
      "67/67 [==============================] - 16s 244ms/step - loss: 1.4796 - categorical_accuracy: 0.3433 - val_loss: 1.4515 - val_categorical_accuracy: 0.2800\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2007_29_22.855379/model-00005-1.47957-0.34328-1.45151-0.28000.h5\n",
      "Epoch 6/10\n",
      "67/67 [==============================] - 16s 245ms/step - loss: 1.3945 - categorical_accuracy: 0.3781 - val_loss: 1.3062 - val_categorical_accuracy: 0.4200\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2007_29_22.855379/model-00006-1.39453-0.37811-1.30618-0.42000.h5\n",
      "Epoch 7/10\n",
      "67/67 [==============================] - 16s 246ms/step - loss: 1.2933 - categorical_accuracy: 0.4478 - val_loss: 1.2610 - val_categorical_accuracy: 0.4500\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2007_29_22.855379/model-00007-1.29331-0.44776-1.26100-0.45000.h5\n",
      "Epoch 8/10\n",
      "67/67 [==============================] - 16s 235ms/step - loss: 1.3530 - categorical_accuracy: 0.4129 - val_loss: 1.2117 - val_categorical_accuracy: 0.5100\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2007_29_22.855379/model-00008-1.35299-0.41294-1.21172-0.51000.h5\n",
      "Epoch 9/10\n",
      "67/67 [==============================] - 17s 246ms/step - loss: 1.1890 - categorical_accuracy: 0.4726 - val_loss: 1.2187 - val_categorical_accuracy: 0.5300\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2007_29_22.855379/model-00009-1.18902-0.47264-1.21867-0.53000.h5\n",
      "Epoch 10/10\n",
      "67/67 [==============================] - 16s 234ms/step - loss: 1.1739 - categorical_accuracy: 0.5124 - val_loss: 1.2052 - val_categorical_accuracy: 0.4600\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2007_29_22.855379/model-00010-1.17388-0.51244-1.20524-0.46000.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f50a60dd518>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_65 (Conv3D)           (None, 30, 120, 120, 32)  2624      \n",
      "_________________________________________________________________\n",
      "activation_65 (Activation)   (None, 30, 120, 120, 32)  0         \n",
      "_________________________________________________________________\n",
      "conv3d_66 (Conv3D)           (None, 30, 120, 120, 32)  27680     \n",
      "_________________________________________________________________\n",
      "activation_66 (Activation)   (None, 30, 120, 120, 32)  0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_35 (MaxPooling (None, 10, 40, 40, 32)    0         \n",
      "_________________________________________________________________\n",
      "dropout_48 (Dropout)         (None, 10, 40, 40, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_67 (Conv3D)           (None, 10, 40, 40, 64)    55360     \n",
      "_________________________________________________________________\n",
      "activation_67 (Activation)   (None, 10, 40, 40, 64)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_68 (Conv3D)           (None, 10, 40, 40, 64)    110656    \n",
      "_________________________________________________________________\n",
      "activation_68 (Activation)   (None, 10, 40, 40, 64)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_36 (MaxPooling (None, 4, 14, 14, 64)     0         \n",
      "_________________________________________________________________\n",
      "dropout_49 (Dropout)         (None, 4, 14, 14, 64)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_69 (Conv3D)           (None, 4, 14, 14, 64)     110656    \n",
      "_________________________________________________________________\n",
      "activation_69 (Activation)   (None, 4, 14, 14, 64)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_70 (Conv3D)           (None, 4, 14, 14, 64)     110656    \n",
      "_________________________________________________________________\n",
      "activation_70 (Activation)   (None, 4, 14, 14, 64)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_37 (MaxPooling (None, 2, 5, 5, 64)       0         \n",
      "_________________________________________________________________\n",
      "dropout_50 (Dropout)         (None, 2, 5, 5, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_17 (Flatten)         (None, 3200)              0         \n",
      "_________________________________________________________________\n",
      "dense_34 (Dense)             (None, 512)               1638912   \n",
      "_________________________________________________________________\n",
      "batch_normalization_5 (Batch (None, 512)               2048      \n",
      "_________________________________________________________________\n",
      "dropout_51 (Dropout)         (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_35 (Dense)             (None, 5)                 2565      \n",
      "=================================================================\n",
      "Total params: 2,061,157\n",
      "Trainable params: 2,060,133\n",
      "Non-trainable params: 1,024\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Define model\n",
    "model = Sequential()\n",
    "model.add(Conv3D(32, kernel_size=(3, 3, 3), input_shape=(nb_frames,nb_rows,nb_cols,nb_channel), padding=\"same\"))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Conv3D(32, padding=\"same\", kernel_size=(3, 3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling3D(pool_size=(3, 3, 3), padding=\"same\"))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Conv3D(64, padding=\"same\", kernel_size=(3, 3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Conv3D(64, padding=\"same\", kernel_size=(3, 3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling3D(pool_size=(3, 3, 3), padding=\"same\"))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Conv3D(64, padding=\"same\", kernel_size=(3, 3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Conv3D(64, padding=\"same\", kernel_size=(3, 3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling3D(pool_size=(3, 3, 3), padding=\"same\"))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(512, activation='relu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes, activation='softmax'))\n",
    "model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = new_generator(train_path, train_doc, batch_size)\n",
    "val_generator = new_generator(val_path, val_doc, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "Source path =  ./Project_data/train ; batch size = 10\n",
      "Source path =  ./Project_data/val ; batch size = 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:21: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:25: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65/67 [============================>.] - ETA: 1s - loss: 2.0563 - categorical_accuracy: 0.2723Batch:  67 Index: 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:45: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:49: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "67/67 [==============================] - 48s 723ms/step - loss: 2.0599 - categorical_accuracy: 0.2687 - val_loss: 5.0915 - val_categorical_accuracy: 0.2300\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2007_29_22.855379/model-00001-2.04827-0.27149-5.09154-0.23000.h5\n",
      "Epoch 2/10\n",
      "67/67 [==============================] - 15s 231ms/step - loss: 1.8005 - categorical_accuracy: 0.3383 - val_loss: 4.6516 - val_categorical_accuracy: 0.2300\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2007_29_22.855379/model-00002-1.80049-0.33831-4.65156-0.23000.h5\n",
      "Epoch 3/10\n",
      "67/67 [==============================] - 17s 249ms/step - loss: 1.9274 - categorical_accuracy: 0.2537 - val_loss: 1.4311 - val_categorical_accuracy: 0.4200\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2007_29_22.855379/model-00003-1.92736-0.25373-1.43114-0.42000.h5\n",
      "Epoch 4/10\n",
      "67/67 [==============================] - 15s 225ms/step - loss: 1.6868 - categorical_accuracy: 0.3383 - val_loss: 1.8842 - val_categorical_accuracy: 0.2400\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2007_29_22.855379/model-00004-1.68680-0.33831-1.88423-0.24000.h5\n",
      "Epoch 5/10\n",
      "67/67 [==============================] - 17s 251ms/step - loss: 1.7094 - categorical_accuracy: 0.2935 - val_loss: 2.7662 - val_categorical_accuracy: 0.2100\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2007_29_22.855379/model-00005-1.70939-0.29353-2.76621-0.21000.h5\n",
      "\n",
      "Epoch 00005: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.\n",
      "Epoch 6/10\n",
      "67/67 [==============================] - 15s 231ms/step - loss: 1.6358 - categorical_accuracy: 0.2786 - val_loss: 1.5532 - val_categorical_accuracy: 0.3400\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2007_29_22.855379/model-00006-1.63580-0.27861-1.55325-0.34000.h5\n",
      "Epoch 7/10\n",
      "67/67 [==============================] - 17s 251ms/step - loss: 1.6310 - categorical_accuracy: 0.3383 - val_loss: 1.3692 - val_categorical_accuracy: 0.3700\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2007_29_22.855379/model-00007-1.63097-0.33831-1.36916-0.37000.h5\n",
      "Epoch 8/10\n",
      "67/67 [==============================] - 17s 260ms/step - loss: 1.5225 - categorical_accuracy: 0.3383 - val_loss: 1.4198 - val_categorical_accuracy: 0.3800\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2007_29_22.855379/model-00008-1.52253-0.33831-1.41976-0.38000.h5\n",
      "Epoch 9/10\n",
      "67/67 [==============================] - 16s 239ms/step - loss: 1.6347 - categorical_accuracy: 0.2935 - val_loss: 1.3355 - val_categorical_accuracy: 0.4000\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2007_29_22.855379/model-00009-1.63471-0.29353-1.33554-0.40000.h5\n",
      "Epoch 10/10\n",
      "67/67 [==============================] - 17s 246ms/step - loss: 1.5839 - categorical_accuracy: 0.3134 - val_loss: 1.3241 - val_categorical_accuracy: 0.3600\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2007_29_22.855379/model-00010-1.58392-0.31343-1.32410-0.36000.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f50a564ec88>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_epochs = 20\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_71 (Conv3D)           (None, 30, 120, 120, 8)   656       \n",
      "_________________________________________________________________\n",
      "batch_normalization_6 (Batch (None, 30, 120, 120, 8)   32        \n",
      "_________________________________________________________________\n",
      "activation_71 (Activation)   (None, 30, 120, 120, 8)   0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_38 (MaxPooling (None, 15, 60, 60, 8)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_72 (Conv3D)           (None, 15, 60, 60, 16)    3472      \n",
      "_________________________________________________________________\n",
      "batch_normalization_7 (Batch (None, 15, 60, 60, 16)    64        \n",
      "_________________________________________________________________\n",
      "activation_72 (Activation)   (None, 15, 60, 60, 16)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_39 (MaxPooling (None, 7, 30, 30, 16)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_73 (Conv3D)           (None, 7, 30, 30, 32)     4640      \n",
      "_________________________________________________________________\n",
      "batch_normalization_8 (Batch (None, 7, 30, 30, 32)     128       \n",
      "_________________________________________________________________\n",
      "activation_73 (Activation)   (None, 7, 30, 30, 32)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_40 (MaxPooling (None, 3, 15, 15, 32)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_74 (Conv3D)           (None, 3, 15, 15, 64)     18496     \n",
      "_________________________________________________________________\n",
      "batch_normalization_9 (Batch (None, 3, 15, 15, 64)     256       \n",
      "_________________________________________________________________\n",
      "activation_74 (Activation)   (None, 3, 15, 15, 64)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_41 (MaxPooling (None, 1, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_18 (Flatten)         (None, 3136)              0         \n",
      "_________________________________________________________________\n",
      "dense_36 (Dense)             (None, 1000)              3137000   \n",
      "_________________________________________________________________\n",
      "dropout_52 (Dropout)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "dense_37 (Dense)             (None, 500)               500500    \n",
      "_________________________________________________________________\n",
      "dropout_53 (Dropout)         (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_38 (Dense)             (None, 5)                 2505      \n",
      "=================================================================\n",
      "Total params: 3,667,749\n",
      "Trainable params: 3,667,509\n",
      "Non-trainable params: 240\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "input_shape=(nb_frames,nb_rows,nb_cols,nb_channel)\n",
    "\n",
    "# Define model\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv3D(nb_filters[0], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 input_shape=input_shape,\n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[1], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[2], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[3], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "#Flatten Layers\n",
    "model.add(Flatten())\n",
    "\n",
    "model.add(Dense(nb_dense[0], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "model.add(Dense(nb_dense[1], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "#softmax layer\n",
    "model.add(Dense(nb_dense[2], activation='softmax'))\n",
    "model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "Source path =  ./Project_data/val ; batch size = 10\n",
      "Source path =  ./Project_data/train ; batch size = 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:21: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:25: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65/67 [============================>.] - ETA: 1s - loss: 5.1395 - categorical_accuracy: 0.2708Batch:  67 Index: 10\n",
      "66/67 [============================>.] - ETA: 0s - loss: 5.1125 - categorical_accuracy: 0.2682"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:45: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:49: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "67/67 [==============================] - 47s 695ms/step - loss: 5.0848 - categorical_accuracy: 0.2691 - val_loss: 5.6818 - val_categorical_accuracy: 0.3100\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2007_29_22.855379/model-00001-5.10398-0.26848-5.68178-0.31000.h5\n",
      "Epoch 2/20\n",
      "67/67 [==============================] - 14s 210ms/step - loss: 2.3025 - categorical_accuracy: 0.3134 - val_loss: 1.6048 - val_categorical_accuracy: 0.3500\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2007_29_22.855379/model-00002-2.30247-0.31343-1.60477-0.35000.h5\n",
      "Epoch 3/20\n",
      "67/67 [==============================] - 16s 240ms/step - loss: 1.6830 - categorical_accuracy: 0.3582 - val_loss: 1.1917 - val_categorical_accuracy: 0.5200\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2007_29_22.855379/model-00003-1.68301-0.35821-1.19174-0.52000.h5\n",
      "Epoch 4/20\n",
      "67/67 [==============================] - 16s 242ms/step - loss: 1.4793 - categorical_accuracy: 0.3682 - val_loss: 1.4588 - val_categorical_accuracy: 0.2700\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2007_29_22.855379/model-00004-1.47932-0.36816-1.45882-0.27000.h5\n",
      "Epoch 5/20\n",
      "67/67 [==============================] - 16s 240ms/step - loss: 1.4716 - categorical_accuracy: 0.3930 - val_loss: 1.1414 - val_categorical_accuracy: 0.5300\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2007_29_22.855379/model-00005-1.47156-0.39303-1.14136-0.53000.h5\n",
      "Epoch 6/20\n",
      "67/67 [==============================] - 16s 242ms/step - loss: 1.4172 - categorical_accuracy: 0.3881 - val_loss: 1.1863 - val_categorical_accuracy: 0.4500\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2007_29_22.855379/model-00006-1.41717-0.38806-1.18631-0.45000.h5\n",
      "Epoch 7/20\n",
      "67/67 [==============================] - 16s 234ms/step - loss: 1.3899 - categorical_accuracy: 0.3582 - val_loss: 1.2232 - val_categorical_accuracy: 0.4600\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2007_29_22.855379/model-00007-1.38988-0.35821-1.22315-0.46000.h5\n",
      "\n",
      "Epoch 00007: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.\n",
      "Epoch 8/20\n",
      "67/67 [==============================] - 16s 240ms/step - loss: 1.4063 - categorical_accuracy: 0.4229 - val_loss: 1.1582 - val_categorical_accuracy: 0.5300\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2007_29_22.855379/model-00008-1.40632-0.42289-1.15819-0.53000.h5\n",
      "Epoch 9/20\n",
      "67/67 [==============================] - 16s 243ms/step - loss: 1.2461 - categorical_accuracy: 0.4179 - val_loss: 1.0156 - val_categorical_accuracy: 0.5000\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2007_29_22.855379/model-00009-1.24608-0.41791-1.01561-0.50000.h5\n",
      "Epoch 10/20\n",
      "67/67 [==============================] - 16s 240ms/step - loss: 1.2321 - categorical_accuracy: 0.5124 - val_loss: 1.0143 - val_categorical_accuracy: 0.5700\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2007_29_22.855379/model-00010-1.23207-0.51244-1.01429-0.57000.h5\n",
      "Epoch 11/20\n",
      "67/67 [==============================] - 18s 265ms/step - loss: 1.2131 - categorical_accuracy: 0.4279 - val_loss: 1.1200 - val_categorical_accuracy: 0.5200\n",
      "\n",
      "Epoch 00011: saving model to model_init_2018-10-2007_29_22.855379/model-00011-1.21308-0.42786-1.12000-0.52000.h5\n",
      "Epoch 12/20\n",
      "67/67 [==============================] - 17s 247ms/step - loss: 1.0503 - categorical_accuracy: 0.5373 - val_loss: 0.9656 - val_categorical_accuracy: 0.6200\n",
      "\n",
      "Epoch 00012: saving model to model_init_2018-10-2007_29_22.855379/model-00012-1.05032-0.53731-0.96556-0.62000.h5\n",
      "Epoch 13/20\n",
      "67/67 [==============================] - 16s 244ms/step - loss: 1.1566 - categorical_accuracy: 0.4826 - val_loss: 1.1755 - val_categorical_accuracy: 0.5000\n",
      "\n",
      "Epoch 00013: saving model to model_init_2018-10-2007_29_22.855379/model-00013-1.15661-0.48259-1.17550-0.50000.h5\n",
      "Epoch 14/20\n",
      "67/67 [==============================] - 16s 235ms/step - loss: 1.1906 - categorical_accuracy: 0.5224 - val_loss: 0.8626 - val_categorical_accuracy: 0.6500\n",
      "\n",
      "Epoch 00014: saving model to model_init_2018-10-2007_29_22.855379/model-00014-1.19060-0.52239-0.86257-0.65000.h5\n",
      "Epoch 15/20\n",
      "67/67 [==============================] - 16s 240ms/step - loss: 1.0602 - categorical_accuracy: 0.5622 - val_loss: 0.8362 - val_categorical_accuracy: 0.6600\n",
      "\n",
      "Epoch 00015: saving model to model_init_2018-10-2007_29_22.855379/model-00015-1.06025-0.56219-0.83618-0.66000.h5\n",
      "Epoch 16/20\n",
      "67/67 [==============================] - 16s 241ms/step - loss: 1.0578 - categorical_accuracy: 0.5572 - val_loss: 0.8148 - val_categorical_accuracy: 0.7300\n",
      "\n",
      "Epoch 00016: saving model to model_init_2018-10-2007_29_22.855379/model-00016-1.05780-0.55721-0.81483-0.73000.h5\n",
      "Epoch 17/20\n",
      "67/67 [==============================] - 16s 236ms/step - loss: 1.0654 - categorical_accuracy: 0.5473 - val_loss: 0.8857 - val_categorical_accuracy: 0.6800\n",
      "\n",
      "Epoch 00017: saving model to model_init_2018-10-2007_29_22.855379/model-00017-1.06537-0.54726-0.88573-0.68000.h5\n",
      "Epoch 18/20\n",
      "67/67 [==============================] - 17s 250ms/step - loss: 0.9668 - categorical_accuracy: 0.6318 - val_loss: 0.8337 - val_categorical_accuracy: 0.6800\n",
      "\n",
      "Epoch 00018: saving model to model_init_2018-10-2007_29_22.855379/model-00018-0.96676-0.63184-0.83368-0.68000.h5\n",
      "\n",
      "Epoch 00018: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628.\n",
      "Epoch 19/20\n",
      "67/67 [==============================] - 17s 248ms/step - loss: 0.9899 - categorical_accuracy: 0.5025 - val_loss: 0.7201 - val_categorical_accuracy: 0.7200\n",
      "\n",
      "Epoch 00019: saving model to model_init_2018-10-2007_29_22.855379/model-00019-0.98989-0.50249-0.72010-0.72000.h5\n",
      "Epoch 20/20\n",
      "67/67 [==============================] - 17s 249ms/step - loss: 0.8203 - categorical_accuracy: 0.6617 - val_loss: 0.7544 - val_categorical_accuracy: 0.7300\n",
      "\n",
      "Epoch 00020: saving model to model_init_2018-10-2007_29_22.855379/model-00020-0.82028-0.66169-0.75436-0.73000.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f50a53bee48>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_generator = new_generator(train_path, train_doc, batch_size)\n",
    "val_generator = new_generator(val_path, val_doc, batch_size)\n",
    "num_epochs = 20\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_frames = 30 # number of frames\n",
    "nb_rows = 64 # image width\n",
    "nb_cols = 64 # image height \n",
    "\n",
    "nb_classes = 5\n",
    "nb_channel = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_75 (Conv3D)           (None, 30, 64, 64, 32)    896       \n",
      "_________________________________________________________________\n",
      "activation_75 (Activation)   (None, 30, 64, 64, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_76 (Conv3D)           (None, 30, 64, 64, 32)    27680     \n",
      "_________________________________________________________________\n",
      "activation_76 (Activation)   (None, 30, 64, 64, 32)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_42 (MaxPooling (None, 10, 22, 22, 32)    0         \n",
      "_________________________________________________________________\n",
      "dropout_54 (Dropout)         (None, 10, 22, 22, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_77 (Conv3D)           (None, 10, 22, 22, 64)    55360     \n",
      "_________________________________________________________________\n",
      "activation_77 (Activation)   (None, 10, 22, 22, 64)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_78 (Conv3D)           (None, 10, 22, 22, 64)    110656    \n",
      "_________________________________________________________________\n",
      "activation_78 (Activation)   (None, 10, 22, 22, 64)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_43 (MaxPooling (None, 4, 8, 8, 64)       0         \n",
      "_________________________________________________________________\n",
      "dropout_55 (Dropout)         (None, 4, 8, 8, 64)       0         \n",
      "_________________________________________________________________\n",
      "conv3d_79 (Conv3D)           (None, 4, 8, 8, 64)       110656    \n",
      "_________________________________________________________________\n",
      "activation_79 (Activation)   (None, 4, 8, 8, 64)       0         \n",
      "_________________________________________________________________\n",
      "conv3d_80 (Conv3D)           (None, 4, 8, 8, 64)       110656    \n",
      "_________________________________________________________________\n",
      "activation_80 (Activation)   (None, 4, 8, 8, 64)       0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_44 (MaxPooling (None, 2, 3, 3, 64)       0         \n",
      "_________________________________________________________________\n",
      "dropout_56 (Dropout)         (None, 2, 3, 3, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_19 (Flatten)         (None, 1152)              0         \n",
      "_________________________________________________________________\n",
      "dense_39 (Dense)             (None, 512)               590336    \n",
      "_________________________________________________________________\n",
      "batch_normalization_10 (Batc (None, 512)               2048      \n",
      "_________________________________________________________________\n",
      "dropout_57 (Dropout)         (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_40 (Dense)             (None, 5)                 2565      \n",
      "=================================================================\n",
      "Total params: 1,010,853\n",
      "Trainable params: 1,009,829\n",
      "Non-trainable params: 1,024\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Define model\n",
    "model = Sequential()\n",
    "model.add(Conv3D(32, kernel_size=(3, 3, 3), input_shape=(nb_frames,nb_rows,nb_cols,nb_channel), padding=\"same\"))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Conv3D(32, padding=\"same\", kernel_size=(3, 3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling3D(pool_size=(3, 3, 3), padding=\"same\"))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Conv3D(64, padding=\"same\", kernel_size=(3, 3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Conv3D(64, padding=\"same\", kernel_size=(3, 3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling3D(pool_size=(3, 3, 3), padding=\"same\"))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Conv3D(64, padding=\"same\", kernel_size=(3, 3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Conv3D(64, padding=\"same\", kernel_size=(3, 3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling3D(pool_size=(3, 3, 3), padding=\"same\"))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(512, activation='relu'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes, activation='softmax'))\n",
    "model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "Source path =  ./Project_data/train ; batch size = 5\n",
      "Source path =  ./Project_data/val ; batch size = 5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:14: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "  \n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:18: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "67/67 [==============================] - 19s 290ms/step - loss: 1.6940 - categorical_accuracy: 0.2687 - val_loss: 2.0147 - val_categorical_accuracy: 0.2000\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2007_29_22.855379/model-00001-1.69395-0.26866-2.01474-0.20000.h5\n",
      "Epoch 2/20\n",
      "64/67 [===========================>..] - ETA: 0s - loss: 1.6278 - categorical_accuracy: 0.3281Batch:  133 Index: 5\n",
      "66/67 [============================>.] - ETA: 0s - loss: 1.6394 - categorical_accuracy: 0.3212"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:38: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:42: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "67/67 [==============================] - 20s 296ms/step - loss: 1.6489 - categorical_accuracy: 0.3165 - val_loss: 1.5015 - val_categorical_accuracy: 0.3200\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2007_29_22.855379/model-00002-1.64084-0.32024-1.50153-0.32000.h5\n",
      "Epoch 3/20\n",
      "67/67 [==============================] - 12s 181ms/step - loss: 1.6278 - categorical_accuracy: 0.2886 - val_loss: 1.5605 - val_categorical_accuracy: 0.3000\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2007_29_22.855379/model-00003-1.62783-0.28856-1.56051-0.30000.h5\n",
      "Epoch 4/20\n",
      "67/67 [==============================] - 13s 188ms/step - loss: 1.6503 - categorical_accuracy: 0.2637 - val_loss: 1.4302 - val_categorical_accuracy: 0.3800\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2007_29_22.855379/model-00004-1.65028-0.26368-1.43023-0.38000.h5\n",
      "Epoch 5/20\n",
      "67/67 [==============================] - 13s 191ms/step - loss: 1.7246 - categorical_accuracy: 0.2587 - val_loss: 1.5076 - val_categorical_accuracy: 0.3600\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2007_29_22.855379/model-00005-1.72462-0.25871-1.50756-0.36000.h5\n",
      "Epoch 6/20\n",
      "67/67 [==============================] - 13s 198ms/step - loss: 1.6528 - categorical_accuracy: 0.2786 - val_loss: 1.4727 - val_categorical_accuracy: 0.3400\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2007_29_22.855379/model-00006-1.65277-0.27861-1.47266-0.34000.h5\n",
      "\n",
      "Epoch 00006: ReduceLROnPlateau reducing learning rate to 6.103515914901436e-08.\n",
      "Epoch 7/20\n",
      "67/67 [==============================] - 13s 196ms/step - loss: 1.6948 - categorical_accuracy: 0.2587 - val_loss: 1.4919 - val_categorical_accuracy: 0.3600\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2007_29_22.855379/model-00007-1.69483-0.25871-1.49186-0.36000.h5\n",
      "Epoch 8/20\n",
      "67/67 [==============================] - 14s 205ms/step - loss: 1.7275 - categorical_accuracy: 0.2935 - val_loss: 1.4778 - val_categorical_accuracy: 0.3200\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2007_29_22.855379/model-00008-1.72747-0.29353-1.47775-0.32000.h5\n",
      "\n",
      "Epoch 00008: ReduceLROnPlateau reducing learning rate to 3.051757957450718e-08.\n",
      "Epoch 9/20\n",
      "67/67 [==============================] - 12s 179ms/step - loss: 1.7710 - categorical_accuracy: 0.2488 - val_loss: 1.4294 - val_categorical_accuracy: 0.4000\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2007_29_22.855379/model-00009-1.77100-0.24876-1.42941-0.40000.h5\n",
      "Epoch 10/20\n",
      "67/67 [==============================] - 12s 185ms/step - loss: 1.7617 - categorical_accuracy: 0.2289 - val_loss: 1.5355 - val_categorical_accuracy: 0.2600\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2007_29_22.855379/model-00010-1.76167-0.22886-1.53546-0.26000.h5\n",
      "Epoch 11/20\n",
      "67/67 [==============================] - 13s 197ms/step - loss: 1.7104 - categorical_accuracy: 0.2786 - val_loss: 1.5409 - val_categorical_accuracy: 0.3000\n",
      "\n",
      "Epoch 00011: saving model to model_init_2018-10-2007_29_22.855379/model-00011-1.71041-0.27861-1.54095-0.30000.h5\n",
      "\n",
      "Epoch 00011: ReduceLROnPlateau reducing learning rate to 1.525878978725359e-08.\n",
      "Epoch 12/20\n",
      "67/67 [==============================] - 13s 199ms/step - loss: 1.7791 - categorical_accuracy: 0.2239 - val_loss: 1.4325 - val_categorical_accuracy: 0.3200\n",
      "\n",
      "Epoch 00012: saving model to model_init_2018-10-2007_29_22.855379/model-00012-1.77912-0.22388-1.43253-0.32000.h5\n",
      "Epoch 13/20\n",
      "67/67 [==============================] - 13s 191ms/step - loss: 1.6520 - categorical_accuracy: 0.2935 - val_loss: 1.4999 - val_categorical_accuracy: 0.2600\n",
      "\n",
      "Epoch 00013: saving model to model_init_2018-10-2007_29_22.855379/model-00013-1.65203-0.29353-1.49995-0.26000.h5\n",
      "\n",
      "Epoch 00013: ReduceLROnPlateau reducing learning rate to 7.629394893626795e-09.\n",
      "Epoch 14/20\n",
      "67/67 [==============================] - 13s 192ms/step - loss: 1.6277 - categorical_accuracy: 0.2836 - val_loss: 1.4781 - val_categorical_accuracy: 0.4200\n",
      "\n",
      "Epoch 00014: saving model to model_init_2018-10-2007_29_22.855379/model-00014-1.62773-0.28358-1.47815-0.42000.h5\n",
      "Epoch 15/20\n",
      "67/67 [==============================] - 12s 181ms/step - loss: 1.6737 - categorical_accuracy: 0.2935 - val_loss: 1.4966 - val_categorical_accuracy: 0.3600\n",
      "\n",
      "Epoch 00015: saving model to model_init_2018-10-2007_29_22.855379/model-00015-1.67371-0.29353-1.49664-0.36000.h5\n",
      "\n",
      "Epoch 00015: ReduceLROnPlateau reducing learning rate to 3.814697446813398e-09.\n",
      "Epoch 16/20\n",
      "67/67 [==============================] - 13s 195ms/step - loss: 1.6733 - categorical_accuracy: 0.2935 - val_loss: 1.4808 - val_categorical_accuracy: 0.3800\n",
      "\n",
      "Epoch 00016: saving model to model_init_2018-10-2007_29_22.855379/model-00016-1.67328-0.29353-1.48076-0.38000.h5\n",
      "Epoch 17/20\n",
      "67/67 [==============================] - 13s 191ms/step - loss: 1.7023 - categorical_accuracy: 0.2836 - val_loss: 1.4916 - val_categorical_accuracy: 0.3600\n",
      "\n",
      "Epoch 00017: saving model to model_init_2018-10-2007_29_22.855379/model-00017-1.70234-0.28358-1.49163-0.36000.h5\n",
      "\n",
      "Epoch 00017: ReduceLROnPlateau reducing learning rate to 1.907348723406699e-09.\n",
      "Epoch 18/20\n",
      "67/67 [==============================] - 13s 198ms/step - loss: 1.7021 - categorical_accuracy: 0.2786 - val_loss: 1.4924 - val_categorical_accuracy: 0.3200\n",
      "\n",
      "Epoch 00018: saving model to model_init_2018-10-2007_29_22.855379/model-00018-1.70211-0.27861-1.49245-0.32000.h5\n",
      "Epoch 19/20\n",
      "67/67 [==============================] - 12s 175ms/step - loss: 1.6264 - categorical_accuracy: 0.2687 - val_loss: 1.4680 - val_categorical_accuracy: 0.4200\n",
      "\n",
      "Epoch 00019: saving model to model_init_2018-10-2007_29_22.855379/model-00019-1.62638-0.26866-1.46798-0.42000.h5\n",
      "\n",
      "Epoch 00019: ReduceLROnPlateau reducing learning rate to 9.536743617033494e-10.\n",
      "Epoch 20/20\n",
      "67/67 [==============================] - 13s 195ms/step - loss: 1.6896 - categorical_accuracy: 0.2687 - val_loss: 1.5152 - val_categorical_accuracy: 0.3200\n",
      "\n",
      "Epoch 00020: saving model to model_init_2018-10-2007_29_22.855379/model-00020-1.68960-0.26866-1.51518-0.32000.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f50a02feac8>"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 5\n",
    "train_generator = new_generator(train_path, train_doc, batch_size)\n",
    "val_generator = new_generator(val_path, val_doc, batch_size)\n",
    "num_epochs = 20\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_85 (Conv3D)           (None, 30, 120, 120, 8)   224       \n",
      "_________________________________________________________________\n",
      "batch_normalization_15 (Batc (None, 30, 120, 120, 8)   32        \n",
      "_________________________________________________________________\n",
      "activation_85 (Activation)   (None, 30, 120, 120, 8)   0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_49 (MaxPooling (None, 15, 60, 60, 8)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_86 (Conv3D)           (None, 15, 60, 60, 16)    3472      \n",
      "_________________________________________________________________\n",
      "batch_normalization_16 (Batc (None, 15, 60, 60, 16)    64        \n",
      "_________________________________________________________________\n",
      "activation_86 (Activation)   (None, 15, 60, 60, 16)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_50 (MaxPooling (None, 7, 30, 30, 16)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_87 (Conv3D)           (None, 7, 30, 30, 32)     4640      \n",
      "_________________________________________________________________\n",
      "batch_normalization_17 (Batc (None, 7, 30, 30, 32)     128       \n",
      "_________________________________________________________________\n",
      "activation_87 (Activation)   (None, 7, 30, 30, 32)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_51 (MaxPooling (None, 3, 15, 15, 32)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_88 (Conv3D)           (None, 3, 15, 15, 64)     18496     \n",
      "_________________________________________________________________\n",
      "batch_normalization_18 (Batc (None, 3, 15, 15, 64)     256       \n",
      "_________________________________________________________________\n",
      "activation_88 (Activation)   (None, 3, 15, 15, 64)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_52 (MaxPooling (None, 1, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_21 (Flatten)         (None, 3136)              0         \n",
      "_________________________________________________________________\n",
      "dense_44 (Dense)             (None, 1000)              3137000   \n",
      "_________________________________________________________________\n",
      "dropout_60 (Dropout)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "dense_45 (Dense)             (None, 500)               500500    \n",
      "_________________________________________________________________\n",
      "dropout_61 (Dropout)         (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_46 (Dense)             (None, 5)                 2505      \n",
      "=================================================================\n",
      "Total params: 3,667,317\n",
      "Trainable params: 3,667,077\n",
      "Non-trainable params: 240\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "nb_frames = 30 # number of frames\n",
    "nb_rows = 120 # image width\n",
    "nb_cols = 120 # image height \n",
    "\n",
    "nb_classes = 5\n",
    "nb_channel = 1\n",
    "\n",
    "input_shape=(nb_frames,nb_rows,nb_cols,nb_channel)\n",
    "\n",
    "# Define model\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv3D(nb_filters[0], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 input_shape=input_shape,\n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[1], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[2], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[3], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "#Flatten Layers\n",
    "model.add(Flatten())\n",
    "\n",
    "model.add(Dense(nb_dense[0], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "model.add(Dense(nb_dense[1], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "#softmax layer\n",
    "model.add(Dense(nb_dense[2], activation='softmax'))\n",
    "model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "Source path =  ./Project_data/train ; batch size = 10\n",
      "Source path =  ./Project_data/val ; batch size = 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:14: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "  \n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:18: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65/67 [============================>.] - ETA: 1s - loss: 0.9864 - categorical_accuracy: 0.5877Batch:  67 Index: 10\n",
      "66/67 [============================>.] - ETA: 0s - loss: 0.9877 - categorical_accuracy: 0.5864"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:38: DeprecationWarning: `imread` is deprecated!\n",
      "`imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``imageio.imread`` instead.\n",
      "/home/santhosh/anaconda3/envs/tensor36/lib/python3.6/site-packages/ipykernel_launcher.py:42: DeprecationWarning: `imresize` is deprecated!\n",
      "`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.\n",
      "Use ``skimage.transform.resize`` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "67/67 [==============================] - 45s 664ms/step - loss: 0.9844 - categorical_accuracy: 0.5925 - val_loss: 0.8864 - val_categorical_accuracy: 0.6900\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2007_29_22.855379/model-00001-0.98671-0.58824-0.88639-0.69000.h5\n",
      "Epoch 2/30\n",
      "67/67 [==============================] - 16s 246ms/step - loss: 1.0896 - categorical_accuracy: 0.5124 - val_loss: 0.8828 - val_categorical_accuracy: 0.6900\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2007_29_22.855379/model-00002-1.08958-0.51244-0.88280-0.69000.h5\n",
      "Epoch 3/30\n",
      "67/67 [==============================] - 16s 232ms/step - loss: 1.0954 - categorical_accuracy: 0.5572 - val_loss: 0.8796 - val_categorical_accuracy: 0.6900\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2007_29_22.855379/model-00003-1.09540-0.55721-0.87963-0.69000.h5\n",
      "Epoch 4/30\n",
      "67/67 [==============================] - 14s 215ms/step - loss: 0.9611 - categorical_accuracy: 0.6269 - val_loss: 0.8779 - val_categorical_accuracy: 0.6900\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2007_29_22.855379/model-00004-0.96114-0.62687-0.87786-0.69000.h5\n",
      "Epoch 5/30\n",
      "67/67 [==============================] - 17s 256ms/step - loss: 1.0649 - categorical_accuracy: 0.5423 - val_loss: 0.8788 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2007_29_22.855379/model-00005-1.06491-0.54229-0.87876-0.71000.h5\n",
      "Epoch 6/30\n",
      "67/67 [==============================] - 16s 240ms/step - loss: 0.9377 - categorical_accuracy: 0.6269 - val_loss: 0.8746 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2007_29_22.855379/model-00006-0.93773-0.62687-0.87460-0.71000.h5\n",
      "Epoch 7/30\n",
      "67/67 [==============================] - 15s 229ms/step - loss: 1.0654 - categorical_accuracy: 0.5323 - val_loss: 0.8745 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2007_29_22.855379/model-00007-1.06536-0.53234-0.87454-0.71000.h5\n",
      "Epoch 8/30\n",
      "67/67 [==============================] - 15s 225ms/step - loss: 1.0513 - categorical_accuracy: 0.5721 - val_loss: 0.8773 - val_categorical_accuracy: 0.7000\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2007_29_22.855379/model-00008-1.05129-0.57214-0.87734-0.70000.h5\n",
      "\n",
      "Epoch 00008: ReduceLROnPlateau reducing learning rate to 3.906250185536919e-06.\n",
      "Epoch 9/30\n",
      "67/67 [==============================] - 15s 218ms/step - loss: 1.0080 - categorical_accuracy: 0.5871 - val_loss: 0.8772 - val_categorical_accuracy: 0.7000\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2007_29_22.855379/model-00009-1.00796-0.58706-0.87722-0.70000.h5\n",
      "Epoch 10/30\n",
      "67/67 [==============================] - 16s 232ms/step - loss: 1.0542 - categorical_accuracy: 0.6169 - val_loss: 0.8750 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2007_29_22.855379/model-00010-1.05420-0.61692-0.87504-0.71000.h5\n",
      "\n",
      "Epoch 00010: ReduceLROnPlateau reducing learning rate to 1.9531250927684596e-06.\n",
      "Epoch 11/30\n",
      "67/67 [==============================] - 15s 227ms/step - loss: 1.1104 - categorical_accuracy: 0.5323 - val_loss: 0.8718 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00011: saving model to model_init_2018-10-2007_29_22.855379/model-00011-1.11036-0.53234-0.87182-0.71000.h5\n",
      "Epoch 12/30\n",
      "67/67 [==============================] - 16s 246ms/step - loss: 1.0097 - categorical_accuracy: 0.5970 - val_loss: 0.8736 - val_categorical_accuracy: 0.7000\n",
      "\n",
      "Epoch 00012: saving model to model_init_2018-10-2007_29_22.855379/model-00012-1.00968-0.59701-0.87356-0.70000.h5\n",
      "Epoch 13/30\n",
      "67/67 [==============================] - 14s 216ms/step - loss: 1.0587 - categorical_accuracy: 0.6119 - val_loss: 0.8740 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00013: saving model to model_init_2018-10-2007_29_22.855379/model-00013-1.05875-0.61194-0.87400-0.71000.h5\n",
      "\n",
      "Epoch 00013: ReduceLROnPlateau reducing learning rate to 9.765625463842298e-07.\n",
      "Epoch 14/30\n",
      "67/67 [==============================] - 15s 217ms/step - loss: 0.9855 - categorical_accuracy: 0.6318 - val_loss: 0.8722 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00014: saving model to model_init_2018-10-2007_29_22.855379/model-00014-0.98550-0.63184-0.87220-0.71000.h5\n",
      "Epoch 15/30\n",
      "67/67 [==============================] - 15s 225ms/step - loss: 1.1200 - categorical_accuracy: 0.5174 - val_loss: 0.8716 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00015: saving model to model_init_2018-10-2007_29_22.855379/model-00015-1.12001-0.51741-0.87156-0.71000.h5\n",
      "Epoch 16/30\n",
      "67/67 [==============================] - 15s 228ms/step - loss: 1.0656 - categorical_accuracy: 0.5522 - val_loss: 0.8707 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00016: saving model to model_init_2018-10-2007_29_22.855379/model-00016-1.06557-0.55224-0.87075-0.71000.h5\n",
      "Epoch 17/30\n",
      "67/67 [==============================] - 15s 221ms/step - loss: 1.0315 - categorical_accuracy: 0.5920 - val_loss: 0.8736 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00017: saving model to model_init_2018-10-2007_29_22.855379/model-00017-1.03151-0.59204-0.87360-0.71000.h5\n",
      "Epoch 18/30\n",
      "67/67 [==============================] - 15s 226ms/step - loss: 1.0434 - categorical_accuracy: 0.5721 - val_loss: 0.8716 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00018: saving model to model_init_2018-10-2007_29_22.855379/model-00018-1.04339-0.57214-0.87157-0.71000.h5\n",
      "\n",
      "Epoch 00018: ReduceLROnPlateau reducing learning rate to 4.882812731921149e-07.\n",
      "Epoch 19/30\n",
      "67/67 [==============================] - 15s 220ms/step - loss: 1.0401 - categorical_accuracy: 0.5821 - val_loss: 0.8728 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00019: saving model to model_init_2018-10-2007_29_22.855379/model-00019-1.04012-0.58209-0.87279-0.71000.h5\n",
      "Epoch 20/30\n",
      "67/67 [==============================] - 16s 233ms/step - loss: 1.1060 - categorical_accuracy: 0.5473 - val_loss: 0.8726 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00020: saving model to model_init_2018-10-2007_29_22.855379/model-00020-1.10602-0.54726-0.87256-0.71000.h5\n",
      "\n",
      "Epoch 00020: ReduceLROnPlateau reducing learning rate to 2.4414063659605745e-07.\n",
      "Epoch 21/30\n",
      "67/67 [==============================] - 16s 239ms/step - loss: 0.9981 - categorical_accuracy: 0.5920 - val_loss: 0.8725 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00021: saving model to model_init_2018-10-2007_29_22.855379/model-00021-0.99811-0.59204-0.87255-0.71000.h5\n",
      "Epoch 22/30\n",
      "67/67 [==============================] - 16s 235ms/step - loss: 1.0289 - categorical_accuracy: 0.6119 - val_loss: 0.8707 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00022: saving model to model_init_2018-10-2007_29_22.855379/model-00022-1.02894-0.61194-0.87073-0.71000.h5\n",
      "\n",
      "Epoch 00022: ReduceLROnPlateau reducing learning rate to 1.2207031829802872e-07.\n",
      "Epoch 23/30\n",
      "67/67 [==============================] - 16s 235ms/step - loss: 1.0525 - categorical_accuracy: 0.5672 - val_loss: 0.8711 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00023: saving model to model_init_2018-10-2007_29_22.855379/model-00023-1.05248-0.56716-0.87115-0.71000.h5\n",
      "Epoch 24/30\n",
      "67/67 [==============================] - 15s 223ms/step - loss: 1.0133 - categorical_accuracy: 0.5274 - val_loss: 0.8726 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00024: saving model to model_init_2018-10-2007_29_22.855379/model-00024-1.01326-0.52736-0.87265-0.71000.h5\n",
      "\n",
      "Epoch 00024: ReduceLROnPlateau reducing learning rate to 6.103515914901436e-08.\n",
      "Epoch 25/30\n",
      "67/67 [==============================] - 15s 221ms/step - loss: 0.9891 - categorical_accuracy: 0.6219 - val_loss: 0.8730 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00025: saving model to model_init_2018-10-2007_29_22.855379/model-00025-0.98910-0.62189-0.87298-0.71000.h5\n",
      "Epoch 26/30\n",
      "67/67 [==============================] - 16s 239ms/step - loss: 1.0619 - categorical_accuracy: 0.5423 - val_loss: 0.8727 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00026: saving model to model_init_2018-10-2007_29_22.855379/model-00026-1.06188-0.54229-0.87274-0.71000.h5\n",
      "\n",
      "Epoch 00026: ReduceLROnPlateau reducing learning rate to 3.051757957450718e-08.\n",
      "Epoch 27/30\n",
      "67/67 [==============================] - 16s 231ms/step - loss: 1.0131 - categorical_accuracy: 0.5920 - val_loss: 0.8721 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00027: saving model to model_init_2018-10-2007_29_22.855379/model-00027-1.01310-0.59204-0.87210-0.71000.h5\n",
      "Epoch 28/30\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "67/67 [==============================] - 16s 236ms/step - loss: 1.0715 - categorical_accuracy: 0.5771 - val_loss: 0.8734 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00028: saving model to model_init_2018-10-2007_29_22.855379/model-00028-1.07152-0.57711-0.87341-0.71000.h5\n",
      "\n",
      "Epoch 00028: ReduceLROnPlateau reducing learning rate to 1.525878978725359e-08.\n",
      "Epoch 29/30\n",
      "67/67 [==============================] - 16s 237ms/step - loss: 0.9801 - categorical_accuracy: 0.6020 - val_loss: 0.8736 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00029: saving model to model_init_2018-10-2007_29_22.855379/model-00029-0.98013-0.60199-0.87356-0.71000.h5\n",
      "Epoch 30/30\n",
      "67/67 [==============================] - 15s 220ms/step - loss: 0.9854 - categorical_accuracy: 0.5871 - val_loss: 0.8700 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00030: saving model to model_init_2018-10-2007_29_22.855379/model-00030-0.98538-0.58706-0.87003-0.71000.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f507c0b2b38>"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 10\n",
    "train_generator = new_generator(train_path, train_doc, batch_size)\n",
    "val_generator = new_generator(val_path, val_doc, batch_size)\n",
    "num_epochs = 30\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_random_affine():\n",
    "    dx, dy = np.random.randint(-1.7, 1.8, 2)\n",
    "    M = np.float32([[1, 0, dx], [0, 1, dy]])\n",
    "    return M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "def aug_generator(source_path, folder_list, batch_size):\n",
    "      \n",
    "    print( 'Source path = ', source_path, '; batch size =', batch_size)\n",
    "    img_idx = [x for x in range(0,nb_frames)] #create a list of image numbers you want to use for a particular video\n",
    "    while True:\n",
    "        t = np.random.permutation(folder_list)\n",
    "        num_batches = len(folder_list)//batch_size # calculate the number of batches\n",
    "        for batch in range(num_batches): # we iterate over the number of batches\n",
    "            batch_data = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "            \n",
    "            batch_data_aug = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels_aug = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "  \n",
    "            batch_data_aug2 = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels_aug2 = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    " \n",
    "            batch_data_aug3 = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels_aug3 = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "      \n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                M = get_random_affine()\n",
    "                M2 = get_random_affine()\n",
    "                M3 = get_random_affine()\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = cv2.imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item], cv2.IMREAD_COLOR)\n",
    "                    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes   \n",
    "                    resized = cv2.resize(image, (nb_rows,nb_cols), interpolation = cv2.INTER_AREA)\n",
    "                    batch_data[folder,idx] = resized\n",
    "                    batch_data_aug[folder,idx] = cv2.warpAffine(resized, M, (resized.shape[0], resized.shape[1]))\n",
    "                    batch_data_aug2[folder,idx] = cv2.warpAffine(resized, M2, (resized.shape[0], resized.shape[1]))\n",
    "                    batch_data_aug3[folder,idx] = cv2.warpAffine(resized, M3, (resized.shape[0], resized.shape[1]))\n",
    "                    \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "                batch_labels_aug[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "                batch_labels_aug2[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "                batch_labels_aug3[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "            \n",
    "            batch_data = np.append(batch_data, batch_data_aug, axis = 0) \n",
    "            batch_data = np.append(batch_data, batch_data_aug2, axis = 0) \n",
    "            batch_data = np.append(batch_data, batch_data_aug3, axis = 0)\n",
    "            batch_labels = np.append(batch_labels, batch_labels_aug, axis = 0) \n",
    "            batch_labels = np.append(batch_labels, batch_labels_aug2, axis = 0) \n",
    "            batch_labels = np.append(batch_labels, batch_labels_aug3, axis = 0)\n",
    "            yield batch_data, batch_labels #you yield the batch_data and the batch_labels, remember what does yield do\n",
    "\n",
    "        \n",
    "        # write the code for the remaining data points which are left after full batches\n",
    "        if (len(folder_list) != batch_size*num_batches):\n",
    "            print(\"Batch: \",num_batches+1,\"Index:\", batch_size)\n",
    "            batch_size = len(folder_list) - (batch_size*num_batches)\n",
    "            \n",
    "            batch_data = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "            \n",
    "            batch_data_aug = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels_aug = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "  \n",
    "            batch_data_aug2 = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels_aug2 = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "   \n",
    "            batch_data_aug3 = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels_aug3 = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "      \n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                M = get_random_affine()\n",
    "                M2 = get_random_affine()\n",
    "                M3 = get_random_affine()\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = cv2.imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item], cv2.IMREAD_COLOR)\n",
    "                    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
    "                    resized = cv2.resize(image, (nb_rows,nb_cols), interpolation = cv2.INTER_AREA)\n",
    "                    batch_data[folder,idx] = resized\n",
    "                    batch_data_aug[folder,idx] = cv2.warpAffine(resized, M, (resized.shape[0], resized.shape[1]))\n",
    "                    batch_data_aug2[folder,idx] = cv2.warpAffine(resized, M2, (resized.shape[0], resized.shape[1]))\n",
    "                    batch_data_aug3[folder,idx] = cv2.warpAffine(resized, M3, (resized.shape[0], resized.shape[1]))\n",
    "                    \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "                batch_labels_aug[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "                batch_labels_aug2[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "                batch_labels_aug3[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "            \n",
    "            batch_data = np.append(batch_data, batch_data_aug, axis = 0) \n",
    "            batch_data = np.append(batch_data, batch_data_aug2, axis = 0) \n",
    "            batch_data = np.append(batch_data, batch_data_aug3, axis = 0)\n",
    "            batch_labels = np.append(batch_labels, batch_labels_aug, axis = 0) \n",
    "            batch_labels = np.append(batch_labels, batch_labels_aug2, axis = 0) \n",
    "            batch_labels = np.append(batch_labels, batch_labels_aug3, axis = 0) \n",
    "            yield batch_data, batch_labels\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_129 (Conv3D)          (None, 30, 120, 120, 8)   656       \n",
      "_________________________________________________________________\n",
      "batch_normalization_59 (Batc (None, 30, 120, 120, 8)   32        \n",
      "_________________________________________________________________\n",
      "activation_129 (Activation)  (None, 30, 120, 120, 8)   0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_93 (MaxPooling (None, 15, 60, 60, 8)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_130 (Conv3D)          (None, 15, 60, 60, 16)    3472      \n",
      "_________________________________________________________________\n",
      "batch_normalization_60 (Batc (None, 15, 60, 60, 16)    64        \n",
      "_________________________________________________________________\n",
      "activation_130 (Activation)  (None, 15, 60, 60, 16)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_94 (MaxPooling (None, 7, 30, 30, 16)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_131 (Conv3D)          (None, 7, 30, 30, 32)     4640      \n",
      "_________________________________________________________________\n",
      "batch_normalization_61 (Batc (None, 7, 30, 30, 32)     128       \n",
      "_________________________________________________________________\n",
      "activation_131 (Activation)  (None, 7, 30, 30, 32)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_95 (MaxPooling (None, 3, 15, 15, 32)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_132 (Conv3D)          (None, 3, 15, 15, 64)     18496     \n",
      "_________________________________________________________________\n",
      "batch_normalization_62 (Batc (None, 3, 15, 15, 64)     256       \n",
      "_________________________________________________________________\n",
      "activation_132 (Activation)  (None, 3, 15, 15, 64)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_96 (MaxPooling (None, 1, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_32 (Flatten)         (None, 3136)              0         \n",
      "_________________________________________________________________\n",
      "dense_77 (Dense)             (None, 1000)              3137000   \n",
      "_________________________________________________________________\n",
      "dropout_82 (Dropout)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "dense_78 (Dense)             (None, 500)               500500    \n",
      "_________________________________________________________________\n",
      "dropout_83 (Dropout)         (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_79 (Dense)             (None, 5)                 2505      \n",
      "=================================================================\n",
      "Total params: 3,667,749\n",
      "Trainable params: 3,667,509\n",
      "Non-trainable params: 240\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "nb_frames = 30 # number of frames\n",
    "nb_rows = 120 # image width\n",
    "nb_cols = 120 # image height \n",
    "\n",
    "nb_classes = 5\n",
    "nb_channel = 3\n",
    "\n",
    "input_shape=(nb_frames,nb_rows,nb_cols,nb_channel)\n",
    "\n",
    "# Define model\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv3D(nb_filters[0], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 input_shape=input_shape,\n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[1], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[2], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[3], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "#Flatten Layers\n",
    "model.add(Flatten())\n",
    "\n",
    "model.add(Dense(nb_dense[0], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "model.add(Dense(nb_dense[1], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "#softmax layer\n",
    "model.add(Dense(nb_dense[2], activation='softmax'))\n",
    "model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "Source path =  ./Project_data/val ; batch size = 5\n",
      "Source path =  ./Project_data/train ; batch size = 5\n",
      "67/67 [==============================] - 26s 382ms/step - loss: 0.5547 - categorical_accuracy: 0.7769 - val_loss: 0.6688 - val_categorical_accuracy: 0.6550\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2007_29_22.855379/model-00001-0.55475-0.77687-0.66884-0.65500.h5\n",
      "Epoch 2/20\n",
      "64/67 [===========================>..] - ETA: 0s - loss: 0.5608 - categorical_accuracy: 0.7906Batch:  133 Index: 5\n",
      "67/67 [==============================] - 23s 347ms/step - loss: 0.5549 - categorical_accuracy: 0.7933 - val_loss: 0.8516 - val_categorical_accuracy: 0.7250\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2007_29_22.855379/model-00002-0.55736-0.79230-0.85163-0.72500.h5\n",
      "Epoch 3/20\n",
      "67/67 [==============================] - 14s 208ms/step - loss: 0.5180 - categorical_accuracy: 0.8085 - val_loss: 0.4881 - val_categorical_accuracy: 0.7900\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2007_29_22.855379/model-00003-0.51799-0.80846-0.48806-0.79000.h5\n",
      "Epoch 4/20\n",
      "67/67 [==============================] - 15s 226ms/step - loss: 0.5945 - categorical_accuracy: 0.7724 - val_loss: 1.0219 - val_categorical_accuracy: 0.6150\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2007_29_22.855379/model-00004-0.59445-0.77239-1.02186-0.61500.h5\n",
      "Epoch 5/20\n",
      "67/67 [==============================] - 14s 207ms/step - loss: 0.5470 - categorical_accuracy: 0.7823 - val_loss: 0.7027 - val_categorical_accuracy: 0.7250\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2007_29_22.855379/model-00005-0.54695-0.78234-0.70270-0.72500.h5\n",
      "\n",
      "Epoch 00005: ReduceLROnPlateau reducing learning rate to 3.125000148429535e-05.\n",
      "Epoch 6/20\n",
      "67/67 [==============================] - 15s 217ms/step - loss: 0.5925 - categorical_accuracy: 0.7649 - val_loss: 0.8062 - val_categorical_accuracy: 0.7400\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2007_29_22.855379/model-00006-0.59252-0.76493-0.80621-0.74000.h5\n",
      "Epoch 7/20\n",
      "67/67 [==============================] - 14s 208ms/step - loss: 0.5644 - categorical_accuracy: 0.7836 - val_loss: 0.9718 - val_categorical_accuracy: 0.6200\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2007_29_22.855379/model-00007-0.56444-0.78358-0.97176-0.62000.h5\n",
      "\n",
      "Epoch 00007: ReduceLROnPlateau reducing learning rate to 1.5625000742147677e-05.\n",
      "Epoch 8/20\n",
      "67/67 [==============================] - 14s 211ms/step - loss: 0.5251 - categorical_accuracy: 0.7886 - val_loss: 0.4813 - val_categorical_accuracy: 0.8300\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2007_29_22.855379/model-00008-0.52515-0.78856-0.48127-0.83000.h5\n",
      "Epoch 9/20\n",
      "67/67 [==============================] - 14s 206ms/step - loss: 0.5025 - categorical_accuracy: 0.8060 - val_loss: 0.8412 - val_categorical_accuracy: 0.6350\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2007_29_22.855379/model-00009-0.50249-0.80597-0.84124-0.63500.h5\n",
      "Epoch 10/20\n",
      "67/67 [==============================] - 14s 205ms/step - loss: 0.4668 - categorical_accuracy: 0.8134 - val_loss: 0.6001 - val_categorical_accuracy: 0.7850\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2007_29_22.855379/model-00010-0.46677-0.81343-0.60014-0.78500.h5\n",
      "\n",
      "Epoch 00010: ReduceLROnPlateau reducing learning rate to 7.812500371073838e-06.\n",
      "Epoch 11/20\n",
      "67/67 [==============================] - 14s 213ms/step - loss: 0.6311 - categorical_accuracy: 0.7649 - val_loss: 0.6811 - val_categorical_accuracy: 0.7400\n",
      "\n",
      "Epoch 00011: saving model to model_init_2018-10-2007_29_22.855379/model-00011-0.63108-0.76493-0.68113-0.74000.h5\n",
      "Epoch 12/20\n",
      "67/67 [==============================] - 14s 203ms/step - loss: 0.5182 - categorical_accuracy: 0.8109 - val_loss: 0.7561 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00012: saving model to model_init_2018-10-2007_29_22.855379/model-00012-0.51820-0.81095-0.75611-0.71000.h5\n",
      "\n",
      "Epoch 00012: ReduceLROnPlateau reducing learning rate to 3.906250185536919e-06.\n",
      "Epoch 13/20\n",
      "67/67 [==============================] - 14s 215ms/step - loss: 0.5551 - categorical_accuracy: 0.7948 - val_loss: 0.6891 - val_categorical_accuracy: 0.6800\n",
      "\n",
      "Epoch 00013: saving model to model_init_2018-10-2007_29_22.855379/model-00013-0.55508-0.79478-0.68906-0.68000.h5\n",
      "Epoch 14/20\n",
      "67/67 [==============================] - 14s 214ms/step - loss: 0.5557 - categorical_accuracy: 0.7711 - val_loss: 0.7511 - val_categorical_accuracy: 0.7750\n",
      "\n",
      "Epoch 00014: saving model to model_init_2018-10-2007_29_22.855379/model-00014-0.55572-0.77114-0.75114-0.77500.h5\n",
      "\n",
      "Epoch 00014: ReduceLROnPlateau reducing learning rate to 1.9531250927684596e-06.\n",
      "Epoch 15/20\n",
      "67/67 [==============================] - 14s 214ms/step - loss: 0.4544 - categorical_accuracy: 0.8085 - val_loss: 0.7409 - val_categorical_accuracy: 0.7650\n",
      "\n",
      "Epoch 00015: saving model to model_init_2018-10-2007_29_22.855379/model-00015-0.45439-0.80846-0.74089-0.76500.h5\n",
      "Epoch 16/20\n",
      "67/67 [==============================] - 14s 203ms/step - loss: 0.4594 - categorical_accuracy: 0.8333 - val_loss: 0.6880 - val_categorical_accuracy: 0.6500\n",
      "\n",
      "Epoch 00016: saving model to model_init_2018-10-2007_29_22.855379/model-00016-0.45936-0.83333-0.68796-0.65000.h5\n",
      "\n",
      "Epoch 00016: ReduceLROnPlateau reducing learning rate to 9.765625463842298e-07.\n",
      "Epoch 17/20\n",
      "67/67 [==============================] - 14s 216ms/step - loss: 0.5507 - categorical_accuracy: 0.7848 - val_loss: 0.6228 - val_categorical_accuracy: 0.7500\n",
      "\n",
      "Epoch 00017: saving model to model_init_2018-10-2007_29_22.855379/model-00017-0.55075-0.78483-0.62279-0.75000.h5\n",
      "Epoch 18/20\n",
      "67/67 [==============================] - 14s 203ms/step - loss: 0.5119 - categorical_accuracy: 0.8172 - val_loss: 0.8037 - val_categorical_accuracy: 0.7050\n",
      "\n",
      "Epoch 00018: saving model to model_init_2018-10-2007_29_22.855379/model-00018-0.51187-0.81716-0.80370-0.70500.h5\n",
      "\n",
      "Epoch 00018: ReduceLROnPlateau reducing learning rate to 4.882812731921149e-07.\n",
      "Epoch 19/20\n",
      "67/67 [==============================] - 15s 218ms/step - loss: 0.5225 - categorical_accuracy: 0.7998 - val_loss: 0.7891 - val_categorical_accuracy: 0.7450\n",
      "\n",
      "Epoch 00019: saving model to model_init_2018-10-2007_29_22.855379/model-00019-0.52253-0.79975-0.78914-0.74500.h5\n",
      "Epoch 20/20\n",
      "67/67 [==============================] - 15s 219ms/step - loss: 0.5478 - categorical_accuracy: 0.8072 - val_loss: 0.6490 - val_categorical_accuracy: 0.7000\n",
      "\n",
      "Epoch 00020: saving model to model_init_2018-10-2007_29_22.855379/model-00020-0.54776-0.80721-0.64898-0.70000.h5\n",
      "\n",
      "Epoch 00020: ReduceLROnPlateau reducing learning rate to 2.4414063659605745e-07.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f50046ad668>"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 5\n",
    "train_generator = aug_generator(train_path, train_doc, batch_size)\n",
    "val_generator = aug_generator(val_path, val_doc, batch_size)\n",
    "num_epochs = 20\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "def aug_generator(source_path, folder_list, batch_size):\n",
    "      \n",
    "    print( 'Source path = ', source_path, '; batch size =', batch_size)\n",
    "    img_idx = [x for x in range(0,nb_frames)] #create a list of image numbers you want to use for a particular video\n",
    "    while True:\n",
    "        t = np.random.permutation(folder_list)\n",
    "        num_batches = len(folder_list)//batch_size # calculate the number of batches\n",
    "        for batch in range(num_batches): # we iterate over the number of batches\n",
    "            batch_data = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "            \n",
    "            batch_data_aug = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels_aug = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "  \n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                M = get_random_affine()\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = cv2.imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item], cv2.IMREAD_COLOR)\n",
    "                    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes   \n",
    "                    resized = cv2.resize(image, (nb_rows,nb_cols), interpolation = cv2.INTER_AREA)\n",
    "                    batch_data[folder,idx] = resized\n",
    "                    batch_data_aug[folder,idx] = cv2.warpAffine(resized, M, (resized.shape[0], resized.shape[1]))\n",
    "                    \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "                batch_labels_aug[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "            \n",
    "            batch_data = np.append(batch_data, batch_data_aug, axis = 0) \n",
    "            batch_labels = np.append(batch_labels, batch_labels_aug, axis = 0) \n",
    "            yield batch_data, batch_labels #you yield the batch_data and the batch_labels, remember what does yield do\n",
    "\n",
    "        \n",
    "        # write the code for the remaining data points which are left after full batches\n",
    "        if (len(folder_list) != batch_size*num_batches):\n",
    "            print(\"Batch: \",num_batches+1,\"Index:\", batch_size)\n",
    "            batch_size = len(folder_list) - (batch_size*num_batches)\n",
    "            \n",
    "            batch_data = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "            \n",
    "            batch_data_aug = np.zeros((batch_size,nb_frames,nb_rows,nb_cols,nb_channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels_aug = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "\n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                M = get_random_affine()\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = cv2.imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item], cv2.IMREAD_COLOR)\n",
    "                    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
    "                    resized = cv2.resize(image, (nb_rows,nb_cols), interpolation = cv2.INTER_AREA)\n",
    "                    batch_data[folder,idx] = resized\n",
    "                    batch_data_aug[folder,idx] = cv2.warpAffine(resized, M, (resized.shape[0], resized.shape[1]))\n",
    "                    \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "                batch_labels_aug[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "            \n",
    "            batch_data = np.append(batch_data, batch_data_aug, axis = 0) \n",
    "            batch_labels = np.append(batch_labels, batch_labels_aug, axis = 0) \n",
    "\n",
    "            yield batch_data, batch_labels\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_133 (Conv3D)          (None, 30, 120, 120, 8)   656       \n",
      "_________________________________________________________________\n",
      "batch_normalization_63 (Batc (None, 30, 120, 120, 8)   32        \n",
      "_________________________________________________________________\n",
      "activation_133 (Activation)  (None, 30, 120, 120, 8)   0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_97 (MaxPooling (None, 15, 60, 60, 8)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_134 (Conv3D)          (None, 15, 60, 60, 16)    3472      \n",
      "_________________________________________________________________\n",
      "batch_normalization_64 (Batc (None, 15, 60, 60, 16)    64        \n",
      "_________________________________________________________________\n",
      "activation_134 (Activation)  (None, 15, 60, 60, 16)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_98 (MaxPooling (None, 7, 30, 30, 16)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_135 (Conv3D)          (None, 7, 30, 30, 32)     4640      \n",
      "_________________________________________________________________\n",
      "batch_normalization_65 (Batc (None, 7, 30, 30, 32)     128       \n",
      "_________________________________________________________________\n",
      "activation_135 (Activation)  (None, 7, 30, 30, 32)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_99 (MaxPooling (None, 3, 15, 15, 32)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_136 (Conv3D)          (None, 3, 15, 15, 64)     18496     \n",
      "_________________________________________________________________\n",
      "batch_normalization_66 (Batc (None, 3, 15, 15, 64)     256       \n",
      "_________________________________________________________________\n",
      "activation_136 (Activation)  (None, 3, 15, 15, 64)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_100 (MaxPoolin (None, 1, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_33 (Flatten)         (None, 3136)              0         \n",
      "_________________________________________________________________\n",
      "dense_80 (Dense)             (None, 1000)              3137000   \n",
      "_________________________________________________________________\n",
      "dropout_84 (Dropout)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "dense_81 (Dense)             (None, 500)               500500    \n",
      "_________________________________________________________________\n",
      "dropout_85 (Dropout)         (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_82 (Dense)             (None, 5)                 2505      \n",
      "=================================================================\n",
      "Total params: 3,667,749\n",
      "Trainable params: 3,667,509\n",
      "Non-trainable params: 240\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "nb_frames = 30 # number of frames\n",
    "nb_rows = 120 # image width\n",
    "nb_cols = 120 # image height \n",
    "\n",
    "nb_classes = 5\n",
    "nb_channel = 3\n",
    "\n",
    "input_shape=(nb_frames,nb_rows,nb_cols,nb_channel)\n",
    "\n",
    "# Define model\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv3D(nb_filters[0], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 input_shape=input_shape,\n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[1], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[2], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[3], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "#Flatten Layers\n",
    "model.add(Flatten())\n",
    "\n",
    "model.add(Dense(nb_dense[0], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "model.add(Dense(nb_dense[1], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "#softmax layer\n",
    "model.add(Dense(nb_dense[2], activation='softmax'))\n",
    "model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "Source path =  ./Project_data/train ; batch size = 10\n",
      "Source path =  ./Project_data/val ; batch size = 10\n",
      "65/67 [============================>.] - ETA: 0s - loss: 0.3042 - categorical_accuracy: 0.8969Batch:  67 Index: 10\n",
      "67/67 [==============================] - 34s 503ms/step - loss: 0.3006 - categorical_accuracy: 0.8992 - val_loss: 0.5441 - val_categorical_accuracy: 0.8100\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2007_29_22.855379/model-00001-0.30186-0.89819-0.54408-0.81000.h5\n",
      "Epoch 2/20\n",
      "67/67 [==============================] - 10s 145ms/step - loss: 0.3312 - categorical_accuracy: 0.8781 - val_loss: 0.5516 - val_categorical_accuracy: 0.8050\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2007_29_22.855379/model-00002-0.33119-0.87811-0.55165-0.80500.h5\n",
      "Epoch 3/20\n",
      "67/67 [==============================] - 11s 158ms/step - loss: 0.4095 - categorical_accuracy: 0.8483 - val_loss: 0.5543 - val_categorical_accuracy: 0.8150\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2007_29_22.855379/model-00003-0.40951-0.84826-0.55427-0.81500.h5\n",
      "\n",
      "Epoch 00003: ReduceLROnPlateau reducing learning rate to 3.906250185536919e-06.\n",
      "Epoch 4/20\n",
      "67/67 [==============================] - 11s 159ms/step - loss: 0.3713 - categorical_accuracy: 0.8657 - val_loss: 0.5583 - val_categorical_accuracy: 0.8100\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2007_29_22.855379/model-00004-0.37131-0.86567-0.55831-0.81000.h5\n",
      "Epoch 5/20\n",
      "67/67 [==============================] - 11s 164ms/step - loss: 0.3719 - categorical_accuracy: 0.8607 - val_loss: 0.5475 - val_categorical_accuracy: 0.8050\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2007_29_22.855379/model-00005-0.37189-0.86070-0.54747-0.80500.h5\n",
      "\n",
      "Epoch 00005: ReduceLROnPlateau reducing learning rate to 1.9531250927684596e-06.\n",
      "Epoch 6/20\n",
      "67/67 [==============================] - 11s 157ms/step - loss: 0.3256 - categorical_accuracy: 0.8781 - val_loss: 0.5519 - val_categorical_accuracy: 0.8050\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2007_29_22.855379/model-00006-0.32558-0.87811-0.55187-0.80500.h5\n",
      "Epoch 7/20\n",
      "67/67 [==============================] - 10s 151ms/step - loss: 0.3095 - categorical_accuracy: 0.8881 - val_loss: 0.5528 - val_categorical_accuracy: 0.8050\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2007_29_22.855379/model-00007-0.30946-0.88806-0.55279-0.80500.h5\n",
      "\n",
      "Epoch 00007: ReduceLROnPlateau reducing learning rate to 9.765625463842298e-07.\n",
      "Epoch 8/20\n",
      "67/67 [==============================] - 11s 160ms/step - loss: 0.3318 - categorical_accuracy: 0.8905 - val_loss: 0.5535 - val_categorical_accuracy: 0.8050\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2007_29_22.855379/model-00008-0.33184-0.89055-0.55348-0.80500.h5\n",
      "Epoch 9/20\n",
      "67/67 [==============================] - 10s 154ms/step - loss: 0.3472 - categorical_accuracy: 0.8532 - val_loss: 0.5552 - val_categorical_accuracy: 0.7950\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2007_29_22.855379/model-00009-0.34723-0.85323-0.55521-0.79500.h5\n",
      "\n",
      "Epoch 00009: ReduceLROnPlateau reducing learning rate to 4.882812731921149e-07.\n",
      "Epoch 10/20\n",
      "67/67 [==============================] - 11s 158ms/step - loss: 0.3215 - categorical_accuracy: 0.8806 - val_loss: 0.5551 - val_categorical_accuracy: 0.8100\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2007_29_22.855379/model-00010-0.32152-0.88060-0.55507-0.81000.h5\n",
      "Epoch 11/20\n",
      "67/67 [==============================] - 11s 160ms/step - loss: 0.3314 - categorical_accuracy: 0.8756 - val_loss: 0.5554 - val_categorical_accuracy: 0.7950\n",
      "\n",
      "Epoch 00011: saving model to model_init_2018-10-2007_29_22.855379/model-00011-0.33142-0.87562-0.55540-0.79500.h5\n",
      "\n",
      "Epoch 00011: ReduceLROnPlateau reducing learning rate to 2.4414063659605745e-07.\n",
      "Epoch 12/20\n",
      "67/67 [==============================] - 11s 158ms/step - loss: 0.3530 - categorical_accuracy: 0.8582 - val_loss: 0.5521 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00012: saving model to model_init_2018-10-2007_29_22.855379/model-00012-0.35304-0.85821-0.55207-0.80000.h5\n",
      "Epoch 13/20\n",
      "67/67 [==============================] - 10s 157ms/step - loss: 0.3745 - categorical_accuracy: 0.8657 - val_loss: 0.5508 - val_categorical_accuracy: 0.7950\n",
      "\n",
      "Epoch 00013: saving model to model_init_2018-10-2007_29_22.855379/model-00013-0.37454-0.86567-0.55077-0.79500.h5\n",
      "\n",
      "Epoch 00013: ReduceLROnPlateau reducing learning rate to 1.2207031829802872e-07.\n",
      "Epoch 14/20\n",
      "67/67 [==============================] - 11s 161ms/step - loss: 0.3758 - categorical_accuracy: 0.8632 - val_loss: 0.5544 - val_categorical_accuracy: 0.7950\n",
      "\n",
      "Epoch 00014: saving model to model_init_2018-10-2007_29_22.855379/model-00014-0.37578-0.86318-0.55440-0.79500.h5\n",
      "Epoch 15/20\n",
      "67/67 [==============================] - 10s 152ms/step - loss: 0.3495 - categorical_accuracy: 0.8607 - val_loss: 0.5486 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00015: saving model to model_init_2018-10-2007_29_22.855379/model-00015-0.34946-0.86070-0.54857-0.80000.h5\n",
      "\n",
      "Epoch 00015: ReduceLROnPlateau reducing learning rate to 6.103515914901436e-08.\n",
      "Epoch 16/20\n",
      "67/67 [==============================] - 10s 155ms/step - loss: 0.3610 - categorical_accuracy: 0.8731 - val_loss: 0.5447 - val_categorical_accuracy: 0.7950\n",
      "\n",
      "Epoch 00016: saving model to model_init_2018-10-2007_29_22.855379/model-00016-0.36098-0.87313-0.54469-0.79500.h5\n",
      "Epoch 17/20\n",
      "67/67 [==============================] - 11s 162ms/step - loss: 0.3295 - categorical_accuracy: 0.8881 - val_loss: 0.5494 - val_categorical_accuracy: 0.8100\n",
      "\n",
      "Epoch 00017: saving model to model_init_2018-10-2007_29_22.855379/model-00017-0.32947-0.88806-0.54938-0.81000.h5\n",
      "\n",
      "Epoch 00017: ReduceLROnPlateau reducing learning rate to 3.051757957450718e-08.\n",
      "Epoch 18/20\n",
      "67/67 [==============================] - 11s 170ms/step - loss: 0.3347 - categorical_accuracy: 0.8806 - val_loss: 0.5541 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00018: saving model to model_init_2018-10-2007_29_22.855379/model-00018-0.33468-0.88060-0.55413-0.80000.h5\n",
      "Epoch 19/20\n",
      "67/67 [==============================] - 11s 159ms/step - loss: 0.3072 - categorical_accuracy: 0.8930 - val_loss: 0.5622 - val_categorical_accuracy: 0.8050\n",
      "\n",
      "Epoch 00019: saving model to model_init_2018-10-2007_29_22.855379/model-00019-0.30717-0.89303-0.56216-0.80500.h5\n",
      "\n",
      "Epoch 00019: ReduceLROnPlateau reducing learning rate to 1.525878978725359e-08.\n",
      "Epoch 20/20\n",
      "67/67 [==============================] - 10s 147ms/step - loss: 0.3612 - categorical_accuracy: 0.8682 - val_loss: 0.5530 - val_categorical_accuracy: 0.7900\n",
      "\n",
      "Epoch 00020: saving model to model_init_2018-10-2007_29_22.855379/model-00020-0.36120-0.86816-0.55299-0.79000.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f4ffab18518>"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 10\n",
    "train_generator = aug_generator(train_path, train_doc, batch_size)\n",
    "val_generator = aug_generator(val_path, val_doc, batch_size)\n",
    "num_epochs = 20\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_141 (Conv3D)          (None, 30, 120, 120, 8)   656       \n",
      "_________________________________________________________________\n",
      "batch_normalization_70 (Batc (None, 30, 120, 120, 8)   32        \n",
      "_________________________________________________________________\n",
      "activation_141 (Activation)  (None, 30, 120, 120, 8)   0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_105 (MaxPoolin (None, 15, 60, 60, 8)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_142 (Conv3D)          (None, 15, 60, 60, 16)    3472      \n",
      "_________________________________________________________________\n",
      "batch_normalization_71 (Batc (None, 15, 60, 60, 16)    64        \n",
      "_________________________________________________________________\n",
      "activation_142 (Activation)  (None, 15, 60, 60, 16)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_106 (MaxPoolin (None, 7, 30, 30, 16)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_143 (Conv3D)          (None, 7, 30, 30, 32)     4640      \n",
      "_________________________________________________________________\n",
      "batch_normalization_72 (Batc (None, 7, 30, 30, 32)     128       \n",
      "_________________________________________________________________\n",
      "activation_143 (Activation)  (None, 7, 30, 30, 32)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_107 (MaxPoolin (None, 3, 15, 15, 32)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_144 (Conv3D)          (None, 3, 15, 15, 64)     18496     \n",
      "_________________________________________________________________\n",
      "dropout_89 (Dropout)         (None, 3, 15, 15, 64)     0         \n",
      "_________________________________________________________________\n",
      "activation_144 (Activation)  (None, 3, 15, 15, 64)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_108 (MaxPoolin (None, 1, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_35 (Flatten)         (None, 3136)              0         \n",
      "_________________________________________________________________\n",
      "dense_86 (Dense)             (None, 1000)              3137000   \n",
      "_________________________________________________________________\n",
      "dropout_90 (Dropout)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "dense_87 (Dense)             (None, 500)               500500    \n",
      "_________________________________________________________________\n",
      "dropout_91 (Dropout)         (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_88 (Dense)             (None, 5)                 2505      \n",
      "=================================================================\n",
      "Total params: 3,667,493\n",
      "Trainable params: 3,667,381\n",
      "Non-trainable params: 112\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "nb_frames = 30 # number of frames\n",
    "nb_rows = 120 # image width\n",
    "nb_cols = 120 # image height \n",
    "\n",
    "nb_classes = 5\n",
    "nb_channel = 3\n",
    "\n",
    "input_shape=(nb_frames,nb_rows,nb_cols,nb_channel)\n",
    "\n",
    "# Define model\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv3D(nb_filters[0], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 input_shape=input_shape,\n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[1], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[2], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[3], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "#Flatten Layers\n",
    "model.add(Flatten())\n",
    "\n",
    "model.add(Dense(nb_dense[0], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "model.add(Dense(nb_dense[1], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "#softmax layer\n",
    "model.add(Dense(nb_dense[2], activation='softmax'))\n",
    "model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "Source path =  ./Project_data/val ; batch size = 10\n",
      "Source path =  ./Project_data/train ; batch size = 10\n",
      "65/67 [============================>.] - ETA: 0s - loss: 1.6681 - categorical_accuracy: 0.3785Batch:  67 Index: 10\n",
      "67/67 [==============================] - 34s 512ms/step - loss: 1.6505 - categorical_accuracy: 0.3855 - val_loss: 1.2632 - val_categorical_accuracy: 0.3550\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2007_29_22.855379/model-00001-1.65914-0.38084-1.26319-0.35500.h5\n",
      "Epoch 2/20\n",
      "67/67 [==============================] - 9s 140ms/step - loss: 1.4908 - categorical_accuracy: 0.3856 - val_loss: 4.0222 - val_categorical_accuracy: 0.2300\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2007_29_22.855379/model-00002-1.49079-0.38557-4.02221-0.23000.h5\n",
      "Epoch 3/20\n",
      "67/67 [==============================] - 11s 162ms/step - loss: 1.5770 - categorical_accuracy: 0.3035 - val_loss: 1.4175 - val_categorical_accuracy: 0.3050\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2007_29_22.855379/model-00003-1.57701-0.30348-1.41755-0.30500.h5\n",
      "\n",
      "Epoch 00003: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.\n",
      "Epoch 4/20\n",
      "67/67 [==============================] - 10s 155ms/step - loss: 1.4503 - categorical_accuracy: 0.3930 - val_loss: 1.2222 - val_categorical_accuracy: 0.5100\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2007_29_22.855379/model-00004-1.45027-0.39303-1.22220-0.51000.h5\n",
      "Epoch 5/20\n",
      "67/67 [==============================] - 11s 164ms/step - loss: 1.2962 - categorical_accuracy: 0.4876 - val_loss: 1.0393 - val_categorical_accuracy: 0.6100\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2007_29_22.855379/model-00005-1.29621-0.48756-1.03926-0.61000.h5\n",
      "Epoch 6/20\n",
      "67/67 [==============================] - 10s 149ms/step - loss: 1.1905 - categorical_accuracy: 0.5299 - val_loss: 1.1671 - val_categorical_accuracy: 0.5550\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2007_29_22.855379/model-00006-1.19051-0.52985-1.16707-0.55500.h5\n",
      "Epoch 7/20\n",
      "67/67 [==============================] - 11s 167ms/step - loss: 1.2226 - categorical_accuracy: 0.4900 - val_loss: 1.0200 - val_categorical_accuracy: 0.5800\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2007_29_22.855379/model-00007-1.22263-0.49005-1.01996-0.58000.h5\n",
      "Epoch 8/20\n",
      "67/67 [==============================] - 11s 169ms/step - loss: 1.0861 - categorical_accuracy: 0.5348 - val_loss: 0.9479 - val_categorical_accuracy: 0.6200\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2007_29_22.855379/model-00008-1.08610-0.53483-0.94789-0.62000.h5\n",
      "Epoch 9/20\n",
      "67/67 [==============================] - 11s 164ms/step - loss: 0.9508 - categorical_accuracy: 0.6517 - val_loss: 0.9316 - val_categorical_accuracy: 0.6300\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2007_29_22.855379/model-00009-0.95081-0.65174-0.93164-0.63000.h5\n",
      "Epoch 10/20\n",
      "67/67 [==============================] - 10s 156ms/step - loss: 0.9169 - categorical_accuracy: 0.6493 - val_loss: 1.5615 - val_categorical_accuracy: 0.5650\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2007_29_22.855379/model-00010-0.91690-0.64925-1.56153-0.56500.h5\n",
      "Epoch 11/20\n",
      "67/67 [==============================] - 11s 163ms/step - loss: 0.8757 - categorical_accuracy: 0.6418 - val_loss: 0.8443 - val_categorical_accuracy: 0.6350\n",
      "\n",
      "Epoch 00011: saving model to model_init_2018-10-2007_29_22.855379/model-00011-0.87569-0.64179-0.84432-0.63500.h5\n",
      "Epoch 12/20\n",
      "67/67 [==============================] - 11s 161ms/step - loss: 0.7239 - categorical_accuracy: 0.7289 - val_loss: 1.0057 - val_categorical_accuracy: 0.6450\n",
      "\n",
      "Epoch 00012: saving model to model_init_2018-10-2007_29_22.855379/model-00012-0.72393-0.72886-1.00573-0.64500.h5\n",
      "Epoch 13/20\n",
      "67/67 [==============================] - 11s 159ms/step - loss: 0.6749 - categorical_accuracy: 0.7587 - val_loss: 0.8690 - val_categorical_accuracy: 0.6750\n",
      "\n",
      "Epoch 00013: saving model to model_init_2018-10-2007_29_22.855379/model-00013-0.67495-0.75871-0.86897-0.67500.h5\n",
      "\n",
      "Epoch 00013: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628.\n",
      "Epoch 14/20\n",
      "67/67 [==============================] - 11s 169ms/step - loss: 0.6748 - categorical_accuracy: 0.7786 - val_loss: 0.7630 - val_categorical_accuracy: 0.7500\n",
      "\n",
      "Epoch 00014: saving model to model_init_2018-10-2007_29_22.855379/model-00014-0.67481-0.77861-0.76297-0.75000.h5\n",
      "Epoch 15/20\n",
      "67/67 [==============================] - 12s 174ms/step - loss: 0.4946 - categorical_accuracy: 0.8383 - val_loss: 0.8113 - val_categorical_accuracy: 0.7300\n",
      "\n",
      "Epoch 00015: saving model to model_init_2018-10-2007_29_22.855379/model-00015-0.49464-0.83831-0.81135-0.73000.h5\n",
      "Epoch 16/20\n",
      "67/67 [==============================] - 10s 155ms/step - loss: 0.3906 - categorical_accuracy: 0.8582 - val_loss: 0.7424 - val_categorical_accuracy: 0.7450\n",
      "\n",
      "Epoch 00016: saving model to model_init_2018-10-2007_29_22.855379/model-00016-0.39060-0.85821-0.74237-0.74500.h5\n",
      "Epoch 17/20\n",
      "67/67 [==============================] - 11s 159ms/step - loss: 0.5151 - categorical_accuracy: 0.8010 - val_loss: 0.7137 - val_categorical_accuracy: 0.7450\n",
      "\n",
      "Epoch 00017: saving model to model_init_2018-10-2007_29_22.855379/model-00017-0.51510-0.80100-0.71367-0.74500.h5\n",
      "Epoch 18/20\n",
      "67/67 [==============================] - 11s 167ms/step - loss: 0.4002 - categorical_accuracy: 0.8483 - val_loss: 0.6782 - val_categorical_accuracy: 0.7850\n",
      "\n",
      "Epoch 00018: saving model to model_init_2018-10-2007_29_22.855379/model-00018-0.40016-0.84826-0.67818-0.78500.h5\n",
      "Epoch 19/20\n",
      "67/67 [==============================] - 11s 168ms/step - loss: 0.2647 - categorical_accuracy: 0.9030 - val_loss: 0.6986 - val_categorical_accuracy: 0.7550\n",
      "\n",
      "Epoch 00019: saving model to model_init_2018-10-2007_29_22.855379/model-00019-0.26469-0.90299-0.69856-0.75500.h5\n",
      "Epoch 20/20\n",
      "67/67 [==============================] - 11s 166ms/step - loss: 0.3272 - categorical_accuracy: 0.8682 - val_loss: 0.6973 - val_categorical_accuracy: 0.7700\n",
      "\n",
      "Epoch 00020: saving model to model_init_2018-10-2007_29_22.855379/model-00020-0.32724-0.86816-0.69726-0.77000.h5\n",
      "\n",
      "Epoch 00020: ReduceLROnPlateau reducing learning rate to 0.0001250000059371814.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f4ff8ffc860>"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 10\n",
    "train_generator = aug_generator(train_path, train_doc, batch_size)\n",
    "val_generator = aug_generator(val_path, val_doc, batch_size)\n",
    "num_epochs = 20\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}