Reading_multiple_files_in_Tensorflow_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading multiple files in Tensorflow 2\n",
    "Author: [Biswajit Sahoo](https://biswajitsahoo1111.github.io/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/biswajitsahoo1111/cbm_codes_open/blob/master/notebooks/Reading_multiple_files_in_Tensorflow_2.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
    "    Run in Google Colab</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/biswajitsahoo1111/cbm_codes_open/blob/master/notebooks/Reading_multiple_files_in_Tensorflow_2.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
    "    View source on GitHub</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://www.dropbox.com/s/o4aevvuqr39kq20/Reading_multiple_files_in_Tensorflow_2.ipynb?dl=1\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this post, we will read multiple `.csv` files into Tensorflow using generators. But the method we will discuss is general enough to work for other file formats as well. We will demonstrate the procedure using 500 `.csv` files. These files have been created using random numbers. Each file contains only 1024 numbers in one column. This method can easily be extended to huge datasets involving thousands of `.csv` files. As the number of files becomes large, we can't load the whole data into memory. So we have to work with chunks of it. Generators help us do just that conveniently. In this post, we will read multiple files using a custom generator.\n",
    " \n",
    "This post is self-sufficient in the sense that readers don't have to download any data from anywhere. Just run the following codes sequentially. First, a folder named \"random_data\" will be created in current working directory and `.csv` files will be saved in it. Subsequently files will be read from that folder and processed. Just make sure that your current working directory doesn't have an old folder named \"random_data\". Then run the following codes. Jupyter notebook of this post can be found [here](https://github.com/biswajitsahoo1111/cbm_codes_open/blob/master/notebooks/Reading_multiple_files_in_Tensorflow_2.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use `Tensorflow 2` to run our deep learning model. `Tensorflow` is very flexible. A given task can be done in different ways in it. The method we will use is not the only one. Readers are encouraged to explore other ways of doing the same. Below is an outline of three different tasks considered in this post."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Outline:\n",
    "1. Create 500 `\".csv\"` files and save it in the folder \"random_data\" in current directory.\n",
    "2. Write a generator that reads data from the folder in chunks and preprocesses it.\n",
    "3. Feed the chunks of data to a CNN model and train it for several epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Create 500 `.csv` files of random data\n",
    "\n",
    "As we intend to train a CNN model for classification using our data, we will generate data for 5 different classes. Following is the process that we will follow.\n",
    "* Each `.csv` file will have one column of data with 1024 entries.\n",
    "* Each file will be saved using one of the following names (Fault_1, Fault_2, Fault_3, Fault_4, Fault_5). The dataset is balanced, meaning, for each category, we have approximately same number of observations. Data files in \"Fault_1\" \n",
    "category will have names as \"Fault_1_001.csv\", \"Fault_1_002.csv\", \"Fault_1_003.csv\", ..., \"Fault_1_100.csv\". Similarly for other classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import glob\n",
    "np.random.seed(1111)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First create a function that will generate random files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_random_csv_files(fault_classes, number_of_files_in_each_class):\n",
    "    os.mkdir(\"./random_data/\")  # Make a directory to save created files.\n",
    "    for fault_class in fault_classes:\n",
    "        for i in range(number_of_files_in_each_class):\n",
    "            data = np.random.rand(1024,)\n",
    "            file_name = \"./random_data/\" + eval(\"fault_class\") + \"_\" + \"{0:03}\".format(i+1) + \".csv\" # This creates file_name\n",
    "            np.savetxt(eval(\"file_name\"), data, delimiter = \",\", header = \"V1\", comments = \"\")\n",
    "        print(str(eval(\"number_of_files_in_each_class\")) + \" \" + eval(\"fault_class\") + \" files\"  + \" created.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now use the function to create 100 files each for five fault types. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 Fault_1 files created.\n",
      "100 Fault_2 files created.\n",
      "100 Fault_3 files created.\n",
      "100 Fault_4 files created.\n",
      "100 Fault_5 files created.\n"
     ]
    }
   ],
   "source": [
    "create_random_csv_files([\"Fault_1\", \"Fault_2\", \"Fault_3\", \"Fault_4\", \"Fault_5\"], number_of_files_in_each_class = 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of files:  500\n",
      "Showing first 10 files...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['./random_data/Fault_1_001.csv',\n",
       " './random_data/Fault_1_002.csv',\n",
       " './random_data/Fault_1_003.csv',\n",
       " './random_data/Fault_1_004.csv',\n",
       " './random_data/Fault_1_005.csv',\n",
       " './random_data/Fault_1_006.csv',\n",
       " './random_data/Fault_1_007.csv',\n",
       " './random_data/Fault_1_008.csv',\n",
       " './random_data/Fault_1_009.csv',\n",
       " './random_data/Fault_1_010.csv']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files = glob.glob(\"./random_data/*\")\n",
    "print(\"Total number of files: \", len(files))\n",
    "print(\"Showing first 10 files...\")\n",
    "files[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To extract labels from file name, extract the part of the file name that corresponds to fault type. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./random_data/Fault_1_001.csv\n"
     ]
    }
   ],
   "source": [
    "print(files[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fault_1\n"
     ]
    }
   ],
   "source": [
    "print(files[0][14:21])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that data have been created, we will go to the next step. That is, define a generator, preprocess the time series like data into a matrix like shape such that a 2-D CNN can ingest it. \n",
    "\n",
    "## 2. Write a generator that reads data in chunks and preprocesses it\n",
    "Generator are similar to functions with one important difference. While functions produce all their outputs at once, generators produce their outputs one by one and that too when asked. `yield` keyword converts a function into a generator. Generators can run for a fixed number of times or indefinitely depending on the loop structure used inside it. For our application, we will use a generator that runs indefinitely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following generator takes a list of file names as first argument. The second argument is `batch_size`. `batch_size` determines how many files we will process at one go. This is determined by how much memory do we have. If all data can be loaded into memory, there is no need for generators. In case our data size is huge, we can process chunks of it.  \n",
    "\n",
    "As we will be solving a classification problem, we have to assign labels to each raw data. We will use following labels for convenience.\n",
    "\n",
    "|Class| Label|\n",
    "|-----|------|\n",
    "|Fault_1| 0|\n",
    "|Fault_2| 1|\n",
    "|Fault_3| 2|\n",
    "|Fault_4| 3|\n",
    "|Fault_5| 4|\n",
    "\n",
    "The generator will `yield` both data and labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import re            # To match regular expression for extracting labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_generator(file_list, batch_size = 20):\n",
    "    i = 0\n",
    "    while True:\n",
    "        if i*batch_size >= len(file_list):  # This loop is used to run the generator indefinitely.\n",
    "            i = 0\n",
    "            np.random.shuffle(file_list)\n",
    "        else:\n",
    "            file_chunk = file_list[i*batch_size:(i+1)*batch_size] \n",
    "            data = []\n",
    "            labels = []\n",
    "            label_classes = [\"Fault_1\", \"Fault_2\", \"Fault_3\", \"Fault_4\", \"Fault_5\"]\n",
    "            for file in file_chunk:\n",
    "                temp = pd.read_csv(open(file,'r')) # Change this line to read any other type of file\n",
    "                data.append(temp.values.reshape(32,32,1)) # Convert column data to matrix like data with one channel\n",
    "                pattern = \"^\" + eval(\"file[14:21]\")      # Pattern extracted from file_name\n",
    "                for j in range(len(label_classes)):\n",
    "                    if re.match(pattern, label_classes[j]): # Pattern is matched against different label_classes\n",
    "                        labels.append(j)  \n",
    "            data = np.asarray(data).reshape(-1,32,32,1)\n",
    "            labels = np.asarray(labels)\n",
    "            yield data, labels\n",
    "            i = i + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read any other file format, inside the generator change the line that reads files. This will enable us to read different file formats, be it `.txt` or `.npz` or any other. Preprocessing of data, different from what we have done in this blog, can be done within the generator loop.\n",
    "\n",
    "Now we will check whether the generator works as intended or not. We will set `batch_size` to 10. This means that files in chunks of 10 will be read and processed. The list of files from which 10 are chosen can be an ordered file list or shuffled list. In case, the files are not shuffled, use `np.random.shuffle(file_list)` to shuffle files. \n",
    "\n",
    "In the demonstration, we will read files from an ordered list. This will help us check any errors in the code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "generated_data = data_generator(files, batch_size = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n",
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n",
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n",
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n",
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n",
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n"
     ]
    }
   ],
   "source": [
    "num = 0\n",
    "for data, labels in generated_data:\n",
    "    print(data.shape, labels.shape)\n",
    "    print(labels, \"<--Labels\")  # Just to see the lables\n",
    "    print()\n",
    "    num = num + 1\n",
    "    if num > 5: break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the above cell multiple times to observe different labels. Label 1 appears only when all the files corresponding to \"Fault_1\" have been read. There are 100 files for \"Fault_1\" and we have set batch_size to 10. In the above cell we are iterating over the generator only 6 times. When number of iterations become greater than 10, we see label 1 and subsequently other labels. This will happen only if our initial file list is not shuffled. If the original list is shuffled, we will get random labels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will create a `tensorflow dataset` using the generator. `Tensorflow` datasets can conveniently be used to train `tensorflow` models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `tensorflow dataset` can be created form numpy arrays or from generators.Here, we will create it using a generator. Use of the previously created generator as it is in `tensorflow datasets` doesn't work (Readers can verify this). This happens because of the inability of regular expression to compare a \"string\" with a \"byte string\". \"byte strings\" are generated by default in tensorflow. As a way around, we will make modifications to the earlier generator and use it with tensorflow datasets. Note that we will only modified three lines. Modified lines are accompanied by commented texts beside it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tf_data_generator(file_list, batch_size = 20):\n",
    "    i = 0\n",
    "    while True:\n",
    "        if i*batch_size >= len(file_list):  \n",
    "            i = 0\n",
    "            np.random.shuffle(file_list)\n",
    "        else:\n",
    "            file_chunk = file_list[i*batch_size:(i+1)*batch_size] \n",
    "            data = []\n",
    "            labels = []\n",
    "            label_classes = tf.constant([\"Fault_1\", \"Fault_2\", \"Fault_3\", \"Fault_4\", \"Fault_5\"]) # This line has changed.\n",
    "            for file in file_chunk:\n",
    "                temp = pd.read_csv(open(file,'r'))\n",
    "                data.append(temp.values.reshape(32,32,1)) \n",
    "                pattern = tf.constant(eval(\"file[14:21]\"))  # This line has changed\n",
    "                for j in range(len(label_classes)):\n",
    "                    if re.match(pattern.numpy(), label_classes[j].numpy()):  # This line has changed.\n",
    "                        labels.append(j)\n",
    "            data = np.asarray(data).reshape(-1,32,32,1)\n",
    "            labels = np.asarray(labels)\n",
    "            yield data, labels\n",
    "            i = i + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test whether modified generator works or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_data = tf_data_generator(files, batch_size = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n",
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n",
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n",
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n",
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n",
      "(10, 32, 32, 1) (10,)\n",
      "[0 0 0 0 0 0 0 0 0 0] <--Labels\n",
      "\n"
     ]
    }
   ],
   "source": [
    "num = 0\n",
    "for data, labels in check_data:\n",
    "    print(data.shape, labels.shape)\n",
    "    print(labels, \"<--Labels\")\n",
    "    print()\n",
    "    num = num + 1\n",
    "    if num > 5: break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the new generator created by using a few `tensorflow` commands works just fine as our previous generator. This new generator can now be integrated with a `tensorflow dataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 15\n",
    "dataset = tf.data.Dataset.from_generator(tf_data_generator,args= [files, batch_size],output_types = (tf.float32, tf.float32),\n",
    "                                                output_shapes = ((None,32,32,1),(None,)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check whether `dataset` works or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(15, 32, 32, 1) (15,)\n",
      "tf.Tensor([0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.], shape=(15,), dtype=float32)\n",
      "\n",
      "(15, 32, 32, 1) (15,)\n",
      "tf.Tensor([0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.], shape=(15,), dtype=float32)\n",
      "\n",
      "(15, 32, 32, 1) (15,)\n",
      "tf.Tensor([0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.], shape=(15,), dtype=float32)\n",
      "\n",
      "(15, 32, 32, 1) (15,)\n",
      "tf.Tensor([0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.], shape=(15,), dtype=float32)\n",
      "\n",
      "(15, 32, 32, 1) (15,)\n",
      "tf.Tensor([0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.], shape=(15,), dtype=float32)\n",
      "\n",
      "(15, 32, 32, 1) (15,)\n",
      "tf.Tensor([0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.], shape=(15,), dtype=float32)\n",
      "\n",
      "(15, 32, 32, 1) (15,)\n",
      "tf.Tensor([0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 1. 1. 1.], shape=(15,), dtype=float32)\n",
      "\n",
      "(15, 32, 32, 1) (15,)\n",
      "tf.Tensor([1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.], shape=(15,), dtype=float32)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "num = 0\n",
    "for data, labels in dataset:\n",
    "    print(data.shape, labels.shape)\n",
    "    print(labels)\n",
    "    print()\n",
    "    num = num + 1\n",
    "    if num > 7: break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This also works fine. Now, we will train a full CNN model using the generator. As is done in every model, we will first shuffle data files. Split the files into train, validation, and test set. Using the `tf_data_generator` create three tensorflow datasets corresponding to train, validation, and test data respectively. Finally, we will create a simple CNN model. Train it using train dataset, see its performance on validation dataset, and obtain prediction using test dataset. Keep in mind that our aim is not to improve performance of the model. As the data are random, don't expect to see good performance. The aim is only to create a pipeline. \n",
    "\n",
    "## 3. Building data pipeline and training CNN model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before building the data pipeline, we will first move files corresponding to each fault class into different folders. This will make it convenient to split data into training, validation, and test set, keeping the balanced nature of the dataset intact."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create five different folders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "fault_folders = [\"Fault_1\", \"Fault_2\", \"Fault_3\", \"Fault_4\", \"Fault_5\"]\n",
    "for folder_name in fault_folders:\n",
    "    os.mkdir(os.path.join(\"./random_data\", folder_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Move files into those folders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "for file in files:\n",
    "    pattern = \"^\" + eval(\"file[14:21]\")\n",
    "    for j in range(len(fault_folders)):\n",
    "        if re.match(pattern, fault_folders[j]):\n",
    "            dest = os.path.join(\"./random_data/\",eval(\"fault_folders[j]\"))\n",
    "            shutil.move(file, dest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./random_data/Fault_1',\n",
       " './random_data/Fault_2',\n",
       " './random_data/Fault_3',\n",
       " './random_data/Fault_4',\n",
       " './random_data/Fault_5']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glob.glob(\"./random_data/*\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./random_data/Fault_1/Fault_1_001.csv',\n",
       " './random_data/Fault_1/Fault_1_002.csv',\n",
       " './random_data/Fault_1/Fault_1_003.csv',\n",
       " './random_data/Fault_1/Fault_1_004.csv',\n",
       " './random_data/Fault_1/Fault_1_005.csv',\n",
       " './random_data/Fault_1/Fault_1_006.csv',\n",
       " './random_data/Fault_1/Fault_1_007.csv',\n",
       " './random_data/Fault_1/Fault_1_008.csv',\n",
       " './random_data/Fault_1/Fault_1_009.csv',\n",
       " './random_data/Fault_1/Fault_1_010.csv']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glob.glob(\"./random_data/Fault_1/*\")[:10] # Showing first 10 files of Fault_1 folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./random_data/Fault_3/Fault_3_001.csv',\n",
       " './random_data/Fault_3/Fault_3_002.csv',\n",
       " './random_data/Fault_3/Fault_3_003.csv',\n",
       " './random_data/Fault_3/Fault_3_004.csv',\n",
       " './random_data/Fault_3/Fault_3_005.csv',\n",
       " './random_data/Fault_3/Fault_3_006.csv',\n",
       " './random_data/Fault_3/Fault_3_007.csv',\n",
       " './random_data/Fault_3/Fault_3_008.csv',\n",
       " './random_data/Fault_3/Fault_3_009.csv',\n",
       " './random_data/Fault_3/Fault_3_010.csv']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glob.glob(\"./random_data/Fault_3/*\")[:10] # Showing first 10 files of Falut_3 folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare that data for training set, validation set, and test_set. For each fault type, we will keep 70 files for training, 10 files for validation and 20 files for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "fault_1_files = glob.glob(\"./random_data/Fault_1/*\")\n",
    "fault_2_files = glob.glob(\"./random_data/Fault_2/*\")\n",
    "fault_3_files = glob.glob(\"./random_data/Fault_3/*\")\n",
    "fault_4_files = glob.glob(\"./random_data/Fault_4/*\")\n",
    "fault_5_files = glob.glob(\"./random_data/Fault_5/*\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "fault_1_train, fault_1_test = train_test_split(fault_1_files, test_size = 20, random_state = 5)\n",
    "fault_2_train, fault_2_test = train_test_split(fault_2_files, test_size = 20, random_state = 54)\n",
    "fault_3_train, fault_3_test = train_test_split(fault_3_files, test_size = 20, random_state = 543)\n",
    "fault_4_train, fault_4_test = train_test_split(fault_4_files, test_size = 20, random_state = 5432)\n",
    "fault_5_train, fault_5_test = train_test_split(fault_5_files, test_size = 20, random_state = 54321)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "fault_1_train, fault_1_val = train_test_split(fault_1_train, test_size = 10, random_state = 1)\n",
    "fault_2_train, fault_2_val = train_test_split(fault_2_train, test_size = 10, random_state = 12)\n",
    "fault_3_train, fault_3_val = train_test_split(fault_3_train, test_size = 10, random_state = 123)\n",
    "fault_4_train, fault_4_val = train_test_split(fault_4_train, test_size = 10, random_state = 1234)\n",
    "fault_5_train, fault_5_val = train_test_split(fault_5_train, test_size = 10, random_state = 12345)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_file_names = fault_1_train + fault_2_train + fault_3_train + fault_4_train + fault_5_train\n",
    "validation_file_names = fault_1_val + fault_2_val + fault_3_val + fault_4_val + fault_5_val\n",
    "test_file_names = fault_1_test + fault_2_test + fault_3_test + fault_4_test + fault_5_test\n",
    "\n",
    "# Shuffle data (We don't need to shuffle validation and test data)\n",
    "np.random.shuffle(train_file_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of train_files: 350\n",
      "Number of validation_files: 50\n",
      "Number of test_files: 100\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of train_files:\" ,len(train_file_names))\n",
    "print(\"Number of validation_files:\" ,len(validation_file_names))\n",
    "print(\"Number of test_files:\" ,len(test_file_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 10\n",
    "train_dataset = tf.data.Dataset.from_generator(tf_data_generator, args = [train_file_names, batch_size], \n",
    "                                              output_shapes = ((None,32,32,1),(None,)),\n",
    "                                              output_types = (tf.float32, tf.float32))\n",
    "\n",
    "validation_dataset = tf.data.Dataset.from_generator(tf_data_generator, args = [validation_file_names, batch_size],\n",
    "                                                   output_shapes = ((None,32,32,1),(None,)),\n",
    "                                                   output_types = (tf.float32, tf.float32))\n",
    "\n",
    "test_dataset = tf.data.Dataset.from_generator(tf_data_generator, args = [test_file_names, batch_size],\n",
    "                                             output_shapes = ((None,32,32,1),(None,)),\n",
    "                                             output_types = (tf.float32, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 30, 30, 16)        160       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 15, 15, 16)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 13, 13, 32)        4640      \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 6, 6, 32)          0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 1152)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 16)                18448     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 5)                 85        \n",
      "=================================================================\n",
      "Total params: 23,333\n",
      "Trainable params: 23,333\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential([\n",
    "    layers.Conv2D(16, 3, activation = \"relu\", input_shape = (32,32,1)),\n",
    "    layers.MaxPool2D(2),\n",
    "    layers.Conv2D(32, 3, activation = \"relu\"),\n",
    "    layers.MaxPool2D(2),\n",
    "    layers.Flatten(),\n",
    "    layers.Dense(16, activation = \"relu\"),\n",
    "    layers.Dense(5, activation = \"softmax\")\n",
    "])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compile the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss = \"sparse_categorical_crossentropy\", optimizer = \"adam\", metrics = [\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we fit the model, we have to do one important calculation. Remember that our generators are infinite loops. So if no stopping criteria is given, it will run indefinitely. But we want our model to run for, say, 10 epochs. So our generator should loop over the data files just 10 times and no more. This is achieved by setting the arguments `steps_per_epoch` and `validation_steps` to desired numbers in `model.fit()`. Similarly while evaluating model, we need to set the argument `steps` to a desired number in `model.evaluate()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 350 files in training set. Batch_size is 10. So if the generator runs 35 times, it will correspond to one epoch. Therefor, we should set `steps_per_epoch` to 35. Similarly, `validation_steps = 5` and in `model.evaluate()`, `steps = 10`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "steps_per_epoch =  35\n",
      "validation_steps =  5\n",
      "steps =  10\n"
     ]
    }
   ],
   "source": [
    "steps_per_epoch = np.int(np.ceil(len(train_file_names)/batch_size))\n",
    "validation_steps = np.int(np.ceil(len(validation_file_names)/batch_size))\n",
    "steps = np.int(np.ceil(len(test_file_names)/batch_size))\n",
    "print(\"steps_per_epoch = \", steps_per_epoch)\n",
    "print(\"validation_steps = \", validation_steps)\n",
    "print(\"steps = \", steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "35/35 [==============================] - 1s 40ms/step - loss: 1.6268 - accuracy: 0.2029 - val_loss: 1.6111 - val_accuracy: 0.2000\n",
      "Epoch 2/10\n",
      "35/35 [==============================] - 1s 36ms/step - loss: 1.6101 - accuracy: 0.2114 - val_loss: 1.6079 - val_accuracy: 0.2600\n",
      "Epoch 3/10\n",
      "35/35 [==============================] - 1s 35ms/step - loss: 1.6066 - accuracy: 0.2343 - val_loss: 1.6076 - val_accuracy: 0.2000\n",
      "Epoch 4/10\n",
      "35/35 [==============================] - 1s 34ms/step - loss: 1.5993 - accuracy: 0.2143 - val_loss: 1.6085 - val_accuracy: 0.2400\n",
      "Epoch 5/10\n",
      "35/35 [==============================] - 1s 34ms/step - loss: 1.5861 - accuracy: 0.2657 - val_loss: 1.6243 - val_accuracy: 0.2000\n",
      "Epoch 6/10\n",
      "35/35 [==============================] - 1s 35ms/step - loss: 1.5620 - accuracy: 0.3514 - val_loss: 1.6363 - val_accuracy: 0.2000\n",
      "Epoch 7/10\n",
      "35/35 [==============================] - 1s 36ms/step - loss: 1.5370 - accuracy: 0.2857 - val_loss: 1.6171 - val_accuracy: 0.2600\n",
      "Epoch 8/10\n",
      "35/35 [==============================] - 1s 35ms/step - loss: 1.5015 - accuracy: 0.4057 - val_loss: 1.6577 - val_accuracy: 0.2000\n",
      "Epoch 9/10\n",
      "35/35 [==============================] - 1s 35ms/step - loss: 1.4415 - accuracy: 0.5086 - val_loss: 1.6484 - val_accuracy: 0.1400\n",
      "Epoch 10/10\n",
      "35/35 [==============================] - 1s 36ms/step - loss: 1.3363 - accuracy: 0.6143 - val_loss: 1.6672 - val_accuracy: 0.2200\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fcab40f6150>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_dataset, validation_data = validation_dataset, steps_per_epoch = steps_per_epoch,\n",
    "         validation_steps = validation_steps, epochs = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10/10 [==============================] - 0s 25ms/step - loss: 1.6974 - accuracy: 0.1500\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_accuracy = model.evaluate(test_dataset, steps = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss:  1.6973648071289062\n",
      "Test accuracy: 0.15000000596046448\n"
     ]
    }
   ],
   "source": [
    "print(\"Test loss: \", test_loss)\n",
    "print(\"Test accuracy:\", test_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, model performs terribly. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to make predictions?\n",
    "Until now, we have evaluated our model on a kept out test set. For our test set, both data and labels were known. So we evaluated its performance. But often times, for test set, we don't have access to true labels. Rather we have to make predictions on the data available. This is the case in online competitions where we have to submit our predictions on a test set for which don't know the labels. We will call this set (without any labels) the prediction set. This naming convention is arbitray but we will stick with it.\n",
    "\n",
    "If the whole of our prediction set fits into memory, we can just call `model.predict()` on this data and then use `np.argmax()` to obtain predicted class labels. Otherwise, we can read files in prediction set in chunks, make predictions on the chunks and finally append our result.\n",
    "\n",
    "Yet another pedantic way of doing this is to write a generator to read files from the prediciton set in chunks and make predictions on it. We will show how this approach works. As we don't have a prediction set yet, we will first create some files and save it to the prediction set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_prediction_set(num_files = 20):\n",
    "    os.mkdir(\"./random_data/prediction_set\")\n",
    "    for i in range(num_files):\n",
    "        data = np.random.randn(1024,)\n",
    "        file_name = \"./random_data/prediction_set/\"  + \"file_\" + \"{0:03}\".format(i+1) + \".csv\" # This creates file_name\n",
    "        np.savetxt(eval(\"file_name\"), data, delimiter = \",\", header = \"V1\", comments = \"\")\n",
    "    print(str(eval(\"num_files\")) + \" \"+ \" files created in prediction set.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create some files for prediction set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "55  files created in prediction set.\n"
     ]
    }
   ],
   "source": [
    "create_prediction_set(num_files = 55)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of files:  55\n",
      "Showing first 10 files...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['./random_data/prediction_set/file_001.csv',\n",
       " './random_data/prediction_set/file_002.csv',\n",
       " './random_data/prediction_set/file_003.csv',\n",
       " './random_data/prediction_set/file_004.csv',\n",
       " './random_data/prediction_set/file_005.csv',\n",
       " './random_data/prediction_set/file_006.csv',\n",
       " './random_data/prediction_set/file_007.csv',\n",
       " './random_data/prediction_set/file_008.csv',\n",
       " './random_data/prediction_set/file_009.csv',\n",
       " './random_data/prediction_set/file_010.csv']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction_files = glob.glob(\"./random_data/prediction_set/*\")\n",
    "print(\"Total number of files: \", len(prediction_files))\n",
    "print(\"Showing first 10 files...\")\n",
    "prediction_files[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will create a generator to read these files in chunks. This generator will be slightly different from our previous generator. Firstly, we don't want the generator to run indefinitely. Secondly, we don't have any labels. So this generator should only `yield` data. This is how we achieve that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generator_for_prediction(file_list, batch_size = 20):\n",
    "    i = 0\n",
    "    while i <= (len(file_list)/batch_size):\n",
    "        if i == np.floor(len(file_list)/batch_size):\n",
    "            file_chunk = file_list[i*batch_size:len(file_list)]\n",
    "            if len(file_chunk)==0:\n",
    "                break\n",
    "        else:\n",
    "            file_chunk = file_list[i*batch_size:(i+1)*batch_size] \n",
    "        data = []\n",
    "        for file in file_chunk:\n",
    "            temp = pd.read_csv(open(file,'r'))\n",
    "            data.append(temp.values.reshape(32,32,1)) \n",
    "        data = np.asarray(data).reshape(-1,32,32,1)\n",
    "        yield data\n",
    "        i = i + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check whether the generator works or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 32, 32, 1)\n",
      "(10, 32, 32, 1)\n",
      "(10, 32, 32, 1)\n",
      "(10, 32, 32, 1)\n",
      "(10, 32, 32, 1)\n",
      "(5, 32, 32, 1)\n"
     ]
    }
   ],
   "source": [
    "pred_gen = generator_for_prediction(prediction_files,  batch_size = 10)\n",
    "for data in pred_gen:\n",
    "    print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `tensorflow dataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 10\n",
    "prediction_dataset = tf.data.Dataset.from_generator(generator_for_prediction,args=[prediction_files, batch_size],\n",
    "                                                 output_shapes=(None,32,32,1), output_types=(tf.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "steps = np.int(np.ceil(len(prediction_files)/batch_size))\n",
    "predictions = model.predict(prediction_dataset,steps = steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of prediction array:  (55, 5)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.28138927, 0.3383776 , 0.17806269, 0.18918239, 0.01298801],\n",
       "       [0.16730548, 0.20139892, 0.32996896, 0.16305783, 0.13826886],\n",
       "       [0.08079846, 0.35669118, 0.4091237 , 0.13286887, 0.02051783],\n",
       "       [0.01697877, 0.79075295, 0.17063092, 0.01676028, 0.00487713],\n",
       "       [0.19006915, 0.02615157, 0.39364284, 0.09650648, 0.29362988],\n",
       "       [0.05416911, 0.682985  , 0.19086388, 0.0668761 , 0.00510592],\n",
       "       [0.21325852, 0.27782622, 0.10314588, 0.39539766, 0.01037181],\n",
       "       [0.23633875, 0.3308002 , 0.30727112, 0.09573858, 0.02985144],\n",
       "       [0.06442448, 0.34153524, 0.47356713, 0.08497778, 0.03549532],\n",
       "       [0.37901744, 0.32311487, 0.12875995, 0.16359715, 0.00551067],\n",
       "       [0.12227482, 0.49774405, 0.26021793, 0.1060346 , 0.01372868],\n",
       "       [0.07139122, 0.17324339, 0.5490784 , 0.10136751, 0.10491937],\n",
       "       [0.18757634, 0.2833261 , 0.3367256 , 0.14390293, 0.04846917],\n",
       "       [0.23564269, 0.2800771 , 0.19150141, 0.2686058 , 0.02417296],\n",
       "       [0.4835618 , 0.03908279, 0.09785527, 0.31918615, 0.06031401],\n",
       "       [0.03285189, 0.5866938 , 0.3362034 , 0.0313101 , 0.01294078],\n",
       "       [0.31367007, 0.05583594, 0.24806198, 0.2707511 , 0.1116809 ],\n",
       "       [0.11204866, 0.05982558, 0.44611645, 0.16678827, 0.21522103],\n",
       "       [0.04504926, 0.7100154 , 0.16532828, 0.0747861 , 0.00482096],\n",
       "       [0.22441828, 0.01738338, 0.36729604, 0.0961706 , 0.29473177],\n",
       "       [0.22392808, 0.23958267, 0.11669649, 0.41423568, 0.00555711],\n",
       "       [0.11768451, 0.16422512, 0.49695587, 0.13158153, 0.08955302],\n",
       "       [0.04941175, 0.31670955, 0.46190843, 0.12606393, 0.04590632],\n",
       "       [0.19507076, 0.03239974, 0.3885634 , 0.14447391, 0.23949222],\n",
       "       [0.3530666 , 0.08613478, 0.11636773, 0.4088019 , 0.03562902],\n",
       "       [0.12874755, 0.3140329 , 0.3858064 , 0.1278494 , 0.0435637 ],\n",
       "       [0.3001929 , 0.02791574, 0.11502622, 0.5044482 , 0.05241694],\n",
       "       [0.0929171 , 0.1467541 , 0.6005069 , 0.06660035, 0.09322156],\n",
       "       [0.10712272, 0.5518521 , 0.2632791 , 0.06340495, 0.01434106],\n",
       "       [0.27723876, 0.25847596, 0.18952209, 0.25228631, 0.02247689],\n",
       "       [0.12578863, 0.44461673, 0.25048074, 0.14304985, 0.03606399],\n",
       "       [0.09593316, 0.06914104, 0.49921316, 0.1389045 , 0.19680816],\n",
       "       [0.22185169, 0.0878747 , 0.33703303, 0.23808932, 0.11515129],\n",
       "       [0.0850782 , 0.06328611, 0.57307494, 0.08615369, 0.19240707],\n",
       "       [0.41479778, 0.07033634, 0.22154689, 0.2007963 , 0.09252268],\n",
       "       [0.22052608, 0.10761442, 0.33570328, 0.25846007, 0.07769614],\n",
       "       [0.03679338, 0.4369671 , 0.42453632, 0.07080499, 0.03089818],\n",
       "       [0.17414902, 0.3666445 , 0.26953018, 0.16861232, 0.02106389],\n",
       "       [0.04334973, 0.04427214, 0.5819794 , 0.02825493, 0.30214384],\n",
       "       [0.23099631, 0.31964707, 0.31392127, 0.11803907, 0.01739628],\n",
       "       [0.03072637, 0.6739159 , 0.25826213, 0.0309101 , 0.00618558],\n",
       "       [0.20030826, 0.05058228, 0.42536664, 0.14415787, 0.17958501],\n",
       "       [0.25894472, 0.0410106 , 0.25135538, 0.15487678, 0.29381245],\n",
       "       [0.31544876, 0.05200702, 0.20838396, 0.31984535, 0.10431487],\n",
       "       [0.10788545, 0.31769663, 0.44471365, 0.08522549, 0.04447879],\n",
       "       [0.01864015, 0.35556656, 0.551683  , 0.02805553, 0.04605483],\n",
       "       [0.20043266, 0.1211144 , 0.26670808, 0.33885604, 0.07288874],\n",
       "       [0.29432756, 0.19128233, 0.19503927, 0.2826192 , 0.03673161],\n",
       "       [0.2151616 , 0.05391361, 0.34218988, 0.11304423, 0.27569073],\n",
       "       [0.241943  , 0.05663572, 0.23858468, 0.36390153, 0.09893499],\n",
       "       [0.24665013, 0.22702417, 0.33673155, 0.11996701, 0.06962712],\n",
       "       [0.05448309, 0.33466634, 0.49283266, 0.07876839, 0.03924957],\n",
       "       [0.3060696 , 0.03565398, 0.33453086, 0.12989788, 0.19384763],\n",
       "       [0.1417291 , 0.40642622, 0.20021752, 0.22896914, 0.02265806],\n",
       "       [0.10395318, 0.20624556, 0.46823606, 0.12000521, 0.10156006]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Shape of prediction array: \", predictions.shape)\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outputs of prediction are 5 dimensional vector. This is so because we have used 5 neurons in the output layer and our activation function is softmax. The 5 dimensional output vector for an input add to 1. So it can be interpreted as probability. Thus we should classify the input to a class, for which prediction probability is maximum. To get the class corresponding to maximum probability, we can use `np.argmax()` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 2, 1, 2, 1, 3, 1, 2, 0, 1, 2, 2, 1, 0, 1, 0, 2, 1, 2, 3, 2,\n",
       "       2, 2, 3, 2, 3, 2, 1, 0, 1, 2, 2, 2, 0, 2, 1, 1, 2, 1, 1, 2, 4, 3,\n",
       "       2, 2, 3, 0, 2, 3, 2, 2, 2, 1, 2])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(predictions, axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data are randomly generated. So we should not be surprised by this result. Also note that the for each new data, softmax outputs are close to each other. This means that the network is not very sure about the classification result. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This brings us to the end of the blog. As we had planned in the beginning, we have created random data files, a generator, and trained a model using that generator. The above code can be tweaked slightly to read any type of files other than `.csv`. And now we can train our model without worrying about the data size. Whether the data size is 10GB or 750GB, our approach will work for both.\n",
    "\n",
    "As a final note, I want to stress that, this is not the only approach to do the task. As I have mentioned previously, in `Tensorflow`, you can do the same thing in several different ways. The approach I have chosen seemed natural to me. I have neither strived for efficiency nor elegance. If readers have any better idea, I would be happy to know of it.\n",
    "\n",
    "I hope, this blog will be of help to readers. Please bring any errors or omissions to my notice.\n",
    "\n",
    "**Update 1**: While generators are convenient for handling chunks of data from a large dataset, they have limited portability and scalability. Therefore, in Tensorflow Sequences are preferred instead of generators. [See this blog](https://biswajitsahoo1111.github.io/post/reading-multiple-files-in-tensorflow-2-using-sequence/) for a complete workflow for reading multiple files using Sequence.\n",
    "\n",
    "\n",
    "**Update 2**: If along with reading, one has to perform complex transformations on extracted data (say, doing spectrogram on each segment of data, etc.), the naive approach presented in this blog may turn out to be slow. But there are ways to make these computations faster. One such speedup technique can be found at [this blog](https://biswajitsahoo1111.github.io/post/efficiently-reading-multiple-files-in-tensorflow-2/). Readers should also try using [Sequence](https://biswajitsahoo1111.github.io/post/reading-multiple-files-in-tensorflow-2-using-sequence/) and see if it improves input pipeline efficiency.\n",
    "\n",
    "Last modified: 27th April, 2020."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf_24_env_cpu",
   "language": "python",
   "name": "tf_24_env_cpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}