t81_558_class_05_2_cnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O8hkxXtRCzJj"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_05_2_cnn.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YDTXd8-Lmp8Q"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 6: Convolutional Neural Networks (CNN) for Computer Vision**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ncNrAEpzmp8S"
   },
   "source": [
    "# Module 5 Material\n",
    "\n",
    "- Part 5.1: Image Processing in Python [[Video]](https://www.youtube.com/watch?v=Sob7VDb4xh8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_1_python_images.ipynb)\n",
    "- **Part 5.2: Using Convolutional Neural Networks** [[Video]](https://www.youtube.com/watch?v=jL0_lOpEwSk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_2_cnn.ipynb)\n",
    "- Part 5.3: Using Pretrained Neural Networks[[Video]](https://www.youtube.com/watch?v=W2T-dfiHYSo&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_3_vision_transfer.ipynb)\n",
    "- Part 5.4: Looking at Generators and Image Augmentation [[Video]](https://www.youtube.com/watch?v=20JoEmQb810&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_4_generators.ipynb)\n",
    "- Part 5.5: Recognizing Multiple Images with YOLOv5 [[Video]](https://www.youtube.com/watch?v=7Uu1n9Tp0Mk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_5_yolo.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6hj6qzYNCzJl"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fU9UhAxTmp8S",
    "outputId": "b06312b1-829a-4621-f1f3-d5d0ed2aa0c0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n",
      "Using device: mps\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    import google.colab\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return f\"{h}:{m:>02}:{s:>05.2f}\"\n",
    "\n",
    "# Early stopping (see module 3.4)\n",
    "import copy\n",
    "\n",
    "class EarlyStopping:\n",
    "    def __init__(self, patience=5, min_delta=0, restore_best_weights=True):\n",
    "        self.patience = patience\n",
    "        self.min_delta = min_delta\n",
    "        self.restore_best_weights = restore_best_weights\n",
    "        self.best_model = None\n",
    "        self.best_loss = None\n",
    "        self.counter = 0\n",
    "        self.status = \"\"\n",
    "\n",
    "    def __call__(self, model, val_loss):\n",
    "        if self.best_loss is None:\n",
    "            self.best_loss = val_loss\n",
    "            self.best_model = copy.deepcopy(model.state_dict())\n",
    "        elif self.best_loss - val_loss >= self.min_delta:\n",
    "            self.best_model = copy.deepcopy(model.state_dict())\n",
    "            self.best_loss = val_loss\n",
    "            self.counter = 0\n",
    "            self.status = f\"Improvement found, counter reset to {self.counter}\"\n",
    "        else:\n",
    "            self.counter += 1\n",
    "            self.status = f\"No improvement in the last {self.counter} epochs\"\n",
    "            if self.counter >= self.patience:\n",
    "                self.status = f\"Early stopping triggered after {self.counter} epochs.\"\n",
    "                if self.restore_best_weights:\n",
    "                    model.load_state_dict(self.best_model)\n",
    "                return True\n",
    "        return False\n",
    "\n",
    "\n",
    "# Make use of a GPU or MPS (Apple) if one is available.  (see module 3.2)\n",
    "import torch\n",
    "\n",
    "device = (\n",
    "    \"mps\"\n",
    "    if getattr(torch, \"has_mps\", False)\n",
    "    else \"cuda\"\n",
    "    if torch.cuda.is_available()\n",
    "    else \"cpu\"\n",
    ")\n",
    "print(f\"Using device: {device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jf_otSJdmp8k"
   },
   "source": [
    "# Part 5.2: Keras Neural Networks for Digits and Fashion MNIST\n",
    "\n",
    "This module will focus on computer vision. There are some important differences and similarities with previous neural networks.\n",
    "\n",
    "* We will usually use classification, though regression is still an option.\n",
    "* The input to the neural network is now 3D (height, width, color)\n",
    "* Data are not transformed; no z-scores or dummy variables.\n",
    "* Processing time is much longer.\n",
    "* We now have different layer times: dense layers (just like before), convolution layers, and max-pooling layers.\n",
    "* Data will no longer arrive as CSV files. TensorFlow provides some utilities for going directly from the image to the input for a neural network.\n",
    "\n",
    "## Convolutional Neural Networks (CNNs)\n",
    "\n",
    "The convolutional neural network (CNN) is a neural network technology that has profoundly impacted the area of computer vision (CV). Fukushima  (1980) [[Cite:fukushima1980neocognitron]](https://www.rctn.org/bruno/public/papers/Fukushima1980.pdf) introduced the original concept of a convolutional neural network, and   LeCun, Bottou, Bengio & Haffner (1998) [[Cite:lecun1995convolutional]](http://yann.lecun.com/exdb/publis/pdf/lecun-bengio-95a.pdf) greatly improved this work. From this research, Yan LeCun introduced the famous LeNet-5 neural network architecture. This chapter follows the LeNet-5 style of convolutional neural network.  \n",
    "Although computer vision primarily uses CNNs, this technology has some applications outside of the field. You need to realize that if you want to utilize CNNs on non-visual data, you must find a way to encode your data to mimic the properties of visual data.  \n",
    "\n",
    "The order of the input array elements is crucial to the training. In contrast, most neural networks that are not CNNs treat their input data as a long vector of values, and the order in which you arrange the incoming features in this vector is irrelevant. You cannot change the order for these types of neural networks after you have trained the network. \n",
    "\n",
    "The CNN network arranges the inputs into a grid. This arrangement worked well with images because the pixels in closer proximity to each other are important to each other. The order of pixels in an image is significant. The human body is a relevant example of this type of order. For the design of the face, we are accustomed to eyes being near to each other. \n",
    "\n",
    "This advance in CNNs is due to years of research on biological eyes. In other words, CNNs utilize overlapping fields of input to simulate features of biological eyes. Until this breakthrough, AI had been unable to reproduce the capabilities of biological vision.\n",
    "Scale, rotation, and noise have presented challenges for AI computer vision research. You can observe the complexity of biological eyes in the example that follows. A friend raises a sheet of paper with a large number written on it. As your friend moves nearer to you, the number is still identifiable. In the same way, you can still identify the number when your friend rotates the paper. Lastly, your friend creates noise by drawing lines on the page, but you can still identify the number. As you can see, these examples demonstrate the high function of the biological eye and allow you to understand better the research breakthrough of CNNs. That is, this neural network can process scale, rotation, and noise in the field of computer vision. You can see this network structure in Figure 6.LENET.\n",
    "\n",
    "**Figure 5.LENET: A LeNET-5 Network (LeCun, 1998)**\n",
    "![A LeNET-5 Network](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_lenet5.png \"A LeNET-5 Network\")\n",
    "\n",
    "So far, we have only seen one layer type (dense layers). By the end of this book we will have seen:\n",
    "\n",
    "* **Dense Layers** - Fully connected layers.  \n",
    "* **Convolution Layers** - Used to scan across images. \n",
    "* **Max Pooling Layers** - Used to downsample images. \n",
    "* **Dropout Layers** - Used to add regularization. \n",
    "* **LSTM and Transformer Layers** - Used for time series data.\n",
    "\n",
    "## Convolution Layers\n",
    "\n",
    "The first layer that we will examine is the convolutional layer. We will begin by looking at the hyper-parameters that you must specify for a convolutional layer in most neural network frameworks that support the CNN:\n",
    "\n",
    "* Number of filters\n",
    "* Filter Size\n",
    "* Stride\n",
    "* Padding\n",
    "* Activation Function/Non-Linearity\n",
    "\n",
    "The primary purpose of a convolutional layer is to detect features such as edges, lines, blobs of color, and other visual elements. The filters can detect these features. The more filters we give to a convolutional layer, the more features it can see.\n",
    "\n",
    "A filter is a square-shaped object that scans over the image. A grid can represent the individual pixels of a grid. You can think of the convolutional layer as a smaller grid that sweeps left to right over each image row. There is also a hyperparameter that specifies both the width and height of the square-shaped filter. The following figure shows this configuration in which you see the six convolutional filters sweeping over the image grid:\n",
    "\n",
    "A convolutional layer has weights between it and the previous layer or image grid. Each pixel on each convolutional layer is a weight. Therefore, the number of weights between a convolutional layer and its predecessor layer or image field is the following:\n",
    "\n",
    "```\n",
    "[FilterSize] * [FilterSize] * [# of Filters]\n",
    "```\n",
    "\n",
    "For example, if the filter size were 5 (5x5) for 10 filters, there would be 250 weights.\n",
    "\n",
    "You need to understand how the convolutional filters sweep across the previous layer's output or image grid. Figure 5.CNN illustrates the sweep:\n",
    "\n",
    "**Figure 5.CNN: Convolutional Neural Network**\n",
    "![Convolutional Neural Network](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_cnn_grid.png \"Convolutional Neural Network\")\n",
    "\n",
    "The above figure shows a convolutional filter with 4 and a padding size of 1. The padding size is responsible for the border of zeros in the area that the filter sweeps. Even though the image is 8x7, the extra padding provides a virtual image size of 9x8 for the filter to sweep across. The stride specifies the number of positions the convolutional filters will stop. The convolutional filters move to the right, advancing by the number of cells specified in the stride. Once you reach the far right, the convolutional filter moves back to the far left; then, it moves down by the stride amount and\n",
    "continues to the right again.\n",
    "\n",
    "Some constraints exist concerning the size of the stride. The stride cannot be 0. The convolutional filter would never move if you set the stride. Furthermore, neither the stride nor the convolutional filter size can be larger than the previous grid. There are additional constraints on the stride (*s*), padding (*p*), and the filter width (*f*) for an image of width (*w*). Specifically, the convolutional filter must be able to start at the far left or top border, move a certain number of strides, and land on the far right or bottom border. The following equation shows the number of steps a convolutional operator\n",
    "must take to cross the image:\n",
    "\n",
    "$$ steps = \\frac{w - f + 2p}{s}+1 $$\n",
    "\n",
    "The number of steps must be an integer. In other words, it cannot have decimal places. The purpose of the padding (*p*) is to be adjusted to make this equation become an integer value.\n",
    "\n",
    "## Max Pooling Layers\n",
    "\n",
    "Max-pool layers downsample a 3D box to a new one with smaller dimensions. Typically, you can always place a max-pool layer immediately following the convolutional layer. The LENET shows the max-pool layer immediately after layers C1 and C3. These max-pool layers progressively decrease the size of the dimensions of the 3D boxes passing through them. This technique can avoid overfitting (Krizhevsky, Sutskever & Hinton, 2012).\n",
    "\n",
    "A pooling layer has the following hyper-parameters:\n",
    "\n",
    "* Spatial Extent (*f*)\n",
    "* Stride (*s*)\n",
    "\n",
    "Unlike convolutional layers, max-pool layers do not use padding. Additionally, max-pool layers have no weights, so training does not affect them. These layers downsample their 3D box input. The 3D box output by a max-pool layer will have a width equal to this equation:\n",
    "\n",
    "$$ w_2 = \\frac{w_1 - f}{s} + 1 $$\n",
    "\n",
    "The height of the 3D box produced by the max-pool layer is calculated similarly with this equation:\n",
    "\n",
    "$$ h_2 = \\frac{h_1 - f}{s} + 1 $$\n",
    "\n",
    "The depth of the 3D box produced by the max-pool layer is equal to the depth the 3D box received as input. The most common setting for the hyper-parameters of a max-pool layer is f=2 and s=2. The spatial extent (f) specifies that boxes of 2x2 will be scaled down to single pixels. Of these four pixels, the pixel with the maximum value will represent the 2x2 pixel in the new grid. Because squares of size 4 are replaced with size 1, 75% of the pixel information is lost. The following figure shows this transformation as a 6x6 grid becomes a 3x3:\n",
    "\n",
    "**Figure 5.MAXPOOL: Max Pooling Layer**\n",
    "![Max Pooling Layer](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_conv_maxpool.png \"Max Pooling Layer\")\n",
    "\n",
    "Of course, the above diagram shows each pixel as a single number. A grayscale image would have this characteristic. We usually take the average of the three numbers for an RGB image to determine which pixel has the maximum value.\n",
    "\n",
    "## Regression Convolutional Neural Networks\n",
    "\n",
    "We will now look at two examples, one for regression and another for classification. For supervised computer vision, your dataset will need some labels. For classification, this label usually specifies the subject of the image. For regression, this \"label\" is some numeric quantity the image should produce, such as a count. We will look at two different means of providing this label.\n",
    "\n",
    "The first example will show how to handle regression with convolution neural networks. We will provide an image and expect the neural network to count items in that image. We will use a [dataset](https://www.kaggle.com/jeffheaton/count-the-paperclips) that I created that contains a random number of paperclips. Figure 5.CNN shows a sample from this dataset.\n",
    "\n",
    "\n",
    "**Figure 5.CLIPS: Count the Paperclips**\n",
    "\n",
    "![Max Pooling Layer](https://data.heatonresearch.com/images/wustl/class/clips-25009.jpg \"Count the Paperclips\")\n",
    "\n",
    "As you can see, each of the images contains a random number of randomly placed and sized paperclips.\n",
    "\n",
    "The following code will download this dataset for you.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "GpfadrdQcVg8"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "URL = \"https://github.com/jeffheaton/data-mirror/releases/\"\n",
    "DOWNLOAD_SOURCE = URL+\"download/v1/paperclips.zip\"\n",
    "DOWNLOAD_NAME = DOWNLOAD_SOURCE[DOWNLOAD_SOURCE.rfind('/')+1:]\n",
    "\n",
    "if COLAB:\n",
    "  PATH = \"/content\"\n",
    "else:\n",
    "  # I used this locally on my machine, you likely need different\n",
    "  PATH = \"/Users/jeff/temp\"\n",
    "\n",
    "EXTRACT_TARGET = os.path.join(PATH,\"clips\")\n",
    "SOURCE = os.path.join(EXTRACT_TARGET, \"paperclips\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ivCHAirHpNyT"
   },
   "source": [
    "Next, we download the images. This part depends on the origin of your images. The following code downloads images from a URL, where a ZIP file contains the images. The code unzips the ZIP file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CExT2Z6gpAhz",
    "outputId": "44069f67-c041-45da-a4ab-67f5c135c1df"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-08-15 21:38:06--  https://github.com/jeffheaton/data-mirror/releases/download/v1/paperclips.zip\n",
      "Resolving github.com (github.com)... 140.82.113.3\n",
      "Connecting to github.com (github.com)|140.82.113.3|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/408419764/25830812-b9e6-4ddf-93b6-7932d9ef5982?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230816%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230816T023806Z&X-Amz-Expires=300&X-Amz-Signature=6272ff8ca84fe9ab8bd8d52e4522a4377cef43a6ddf35a6dd088eef8320048d4&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=408419764&response-content-disposition=attachment%3B%20filename%3Dpaperclips.zip&response-content-type=application%2Foctet-stream [following]\n",
      "--2023-08-15 21:38:06--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/408419764/25830812-b9e6-4ddf-93b6-7932d9ef5982?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230816%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230816T023806Z&X-Amz-Expires=300&X-Amz-Signature=6272ff8ca84fe9ab8bd8d52e4522a4377cef43a6ddf35a6dd088eef8320048d4&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=408419764&response-content-disposition=attachment%3B%20filename%3Dpaperclips.zip&response-content-type=application%2Foctet-stream\n",
      "Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ...\n",
      "Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 163590691 (156M) [application/octet-stream]\n",
      "Saving to: ‘/Users/jeff/temp/paperclips.zip’\n",
      "\n",
      "/Users/jeff/temp/pa 100%[===================>] 156.01M  10.3MB/s    in 18s     \n",
      "\n",
      "2023-08-15 21:38:25 (8.54 MB/s) - ‘/Users/jeff/temp/paperclips.zip’ saved [163590691/163590691]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# HIDE OUTPUT\n",
    "!wget -O {os.path.join(PATH,DOWNLOAD_NAME)} {DOWNLOAD_SOURCE}\n",
    "!mkdir -p {SOURCE}\n",
    "!mkdir -p {TARGET}\n",
    "!mkdir -p {EXTRACT_TARGET}\n",
    "!unzip -o -j -d {SOURCE} {os.path.join(PATH, DOWNLOAD_NAME)} >/dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "p4LWl8TzzI8a"
   },
   "source": [
    "The labels are contained in a CSV file named **train.csv**for regression. This file has just two labels, **id** and **clip_count**. The ID specifies the filename; for example, row id 1 corresponds to the file **clips-1.jpg**. The following code loads the labels for the training set and creates a new column, named **filename**, that contains the filename of each image, based on the **id** column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "w4W4LeGSqYya"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    os.path.join(SOURCE,\"train.csv\"), \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "df['filename']=\"clips-\"+df[\"id\"].astype(str)+\".jpg\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AGxjITNtz0sC"
   },
   "source": [
    "This results in the following dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "id": "rJpaMpmfswIp",
    "outputId": "c6462003-f016-4f0c-b8a8-81f276b9b3f4"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>clip_count</th>\n",
       "      <th>filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>30001</td>\n",
       "      <td>11</td>\n",
       "      <td>clips-30001.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>30002</td>\n",
       "      <td>2</td>\n",
       "      <td>clips-30002.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>30003</td>\n",
       "      <td>26</td>\n",
       "      <td>clips-30003.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>30004</td>\n",
       "      <td>41</td>\n",
       "      <td>clips-30004.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30005</td>\n",
       "      <td>49</td>\n",
       "      <td>clips-30005.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19995</th>\n",
       "      <td>49996</td>\n",
       "      <td>35</td>\n",
       "      <td>clips-49996.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19996</th>\n",
       "      <td>49997</td>\n",
       "      <td>54</td>\n",
       "      <td>clips-49997.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19997</th>\n",
       "      <td>49998</td>\n",
       "      <td>72</td>\n",
       "      <td>clips-49998.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19998</th>\n",
       "      <td>49999</td>\n",
       "      <td>24</td>\n",
       "      <td>clips-49999.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19999</th>\n",
       "      <td>50000</td>\n",
       "      <td>35</td>\n",
       "      <td>clips-50000.jpg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20000 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          id  clip_count         filename\n",
       "0      30001          11  clips-30001.jpg\n",
       "1      30002           2  clips-30002.jpg\n",
       "2      30003          26  clips-30003.jpg\n",
       "3      30004          41  clips-30004.jpg\n",
       "4      30005          49  clips-30005.jpg\n",
       "...      ...         ...              ...\n",
       "19995  49996          35  clips-49996.jpg\n",
       "19996  49997          54  clips-49997.jpg\n",
       "19997  49998          72  clips-49998.jpg\n",
       "19998  49999          24  clips-49999.jpg\n",
       "19999  50000          35  clips-50000.jpg\n",
       "\n",
       "[20000 rows x 3 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nN1AsQeysmGd"
   },
   "source": [
    "Separate into a training and validation (for early stopping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CI4_qNaSqp31",
    "outputId": "abe450db-ce00-44ac-cbbd-166912687df4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training size: 18000\n",
      "Validate size: 2000\n"
     ]
    }
   ],
   "source": [
    "TRAIN_PCT = 0.9\n",
    "TRAIN_CUT = int(len(df) * TRAIN_PCT)\n",
    "\n",
    "df_train = df[0:TRAIN_CUT]\n",
    "df_validate = df[TRAIN_CUT:]\n",
    "\n",
    "print(f\"Training size: {len(df_train)}\")\n",
    "print(f\"Validate size: {len(df_validate)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KNoqk5uIz6FW"
   },
   "source": [
    "We are now ready to create a custom class named **ClipCountDataset** that extends the PyTorch **Dataset** class. We will use a technique called augmentation to create additional training data by manipulating the source material. This technique can produce considerably stronger neural networks. The generator below flips the images both vertically and horizontally. PyTorch will train the neuron network both on the original images and the flipped images. This augmentation increases the size of the training data considerably. Module 5.4 will go deeper into the transformations you can perform. You can also specify a target size to resize the images automatically.\n",
    "\n",
    "This class will load the labels from a Pandas dataframe connected to our **train.csv** file. When we demonstrate classification, we will use the new **ClipCountDataset** class; which loads the labels from the directory structure rather than a CSV."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YZzeAHdfsy0O",
    "outputId": "c6741a3f-bf89-4e9e-a730-6559134fcc98"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from torchvision import transforms\n",
    "from PIL import Image\n",
    "import pandas as pd\n",
    "import os\n",
    "import tqdm\n",
    "import torch.nn.functional as F\n",
    "from torch.optim.lr_scheduler import ReduceLROnPlateau\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df_train = pd.read_csv(os.path.join(PATH, \"clips/paperclips/train.csv\"))\n",
    "df_test = pd.read_csv(\n",
    "    os.path.join(PATH, \"clips/paperclips/test.csv\"), na_values=[\"NA\", \"?\"]\n",
    ")\n",
    "df_test[\"filename\"] = \"clips-\" + df_test[\"id\"].astype(str) + \".jpg\"\n",
    "df_train[\"clip_count\"] = df_train[\"clip_count\"].astype(\"float32\")\n",
    "\n",
    "\n",
    "class ClipCountDataset(Dataset):\n",
    "    def __init__(self, dataframe, root_dir, transform=None):\n",
    "        self.data = dataframe\n",
    "        self.root_dir = root_dir\n",
    "        self.transform = transform\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        img_name = os.path.join(\n",
    "            self.root_dir, \"clips-\" + str(self.data.iloc[idx, 0]) + \".jpg\"\n",
    "        )\n",
    "        image = Image.open(img_name)\n",
    "        clip_count = self.data.iloc[idx, 1]\n",
    "        sample = {\"image\": image, \"clip_count\": clip_count}\n",
    "        if self.transform:\n",
    "            sample[\"image\"] = self.transform(sample[\"image\"])\n",
    "        return sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we define a transformation chain, this includes changes that we wish to make to the images. As you can see, we standardize the images to a size of 256x256 and normalize the RGB color components into a standard distribution. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_transform = transforms.Compose([\n",
    "        transforms.Resize((256, 256)),\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize(mean=[0.485, 0.456, 0.406],\n",
    "                             std=[0.229, 0.224, 0.225])\n",
    "    ])\n",
    "\n",
    "train_dataset = ClipCountDataset(df_train, SOURCE, transform=data_transform)\n",
    "val_dataset = ClipCountDataset(df_validate, SOURCE, transform=data_transform)\n",
    "test_dataset = ClipCountDataset(df_test, SOURCE, transform=data_transform)\n",
    "\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=32, shuffle=True)\n",
    "val_dataloader = DataLoader(val_dataset, batch_size=32, shuffle=True)\n",
    "test_dataloader = DataLoader(test_dataset, batch_size=1, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_GVeYo6p2sdG"
   },
   "source": [
    "The constants you see in transforms.Normalize are specific to the ImageNet dataset and the model training process. They are used to normalize the image tensor before passing it to a model trained on ImageNet.\n",
    "\n",
    "Here's a breakdown of what they are:\n",
    "\n",
    "* mean=[0.485, 0.456, 0.406]: This is the mean of the RGB channels of the ImageNet dataset. When models are trained on ImageNet, images are typically zero-centered by subtracting the mean value of each channel from the respective channel in the image. This helps the model converge faster during training.\n",
    "\n",
    "* std=[0.229, 0.224, 0.225]: This is the standard deviation of the RGB channels of the ImageNet dataset. After zero-centering, each channel is typically divided by its standard deviation. This normalization process makes the values of each channel have a zero mean and a standard deviation of 1.\n",
    "\n",
    "The normalization values (both mean and standard deviation) for ImageNet were computed from the dataset and are widely used in the deep learning community for models pre-trained on ImageNet.\n",
    "\n",
    "When you apply this normalization to your input image, you are effectively putting it on the same scale as the images the model was trained on. This ensures that the model will process the image in a manner consistent with its training.\n",
    "\n",
    "If you were working with a different model trained on a different dataset, you would use different mean and standard deviation values specific to that dataset.\n",
    "\n",
    "We can now train the neural network. The code to build and train the neural network is not that different than in the previous modules. We will use the PyTorch **Sequential** class to provide layers to the neural network. We now have several new layer types that we did not previously see.\n",
    "\n",
    "* **Conv2d** - The convolution layers.\n",
    "* **MaxPooling2d** - The max-pooling layers.\n",
    "* **Flatten** - Flatten the 2D (and higher) tensors to allow a Dense layer to process.\n",
    "* **Linear** - Dense layers, the same as demonstrated previously. Dense layers often form the final output layers of the neural network.\n",
    "\n",
    "The training code is very similar to previously. This code is for regression, so a final linear activation is used, along with mean_squared_error for the loss function. The generator provides both the *x* and *y* matrixes we previously supplied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DBHlqCIgtWSq",
    "outputId": "89e48c26-3776-41fa-bff0-bb2b6a39d5d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 625/625 [06:28<00:00,  1.61it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/1, loss: 13.497312545776367\n",
      "Finished Training\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Conv2d(3, 64, 3),  # 3 input channels, 64 output channels, 3x3 kernel\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2, 2),  # 2x2 pooling kernel with stride 2\n",
    "    nn.Conv2d(64, 64, 3), # 64 input channels, 64 output channels, 3x3 kernel\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2, 2),  # 2x2 pooling kernel with stride 2\n",
    "    nn.Flatten(),       # Flattening the tensor for the fully connected layers\n",
    "    nn.Linear(64 * 62 * 62, 512), # 64 * 62 * 62 input features, 512 output features\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(512, 1)    # 512 input features, 1 output feature\n",
    ")\n",
    "model = torch.compile(model,backend=\"aot_eager\").to(device)\n",
    "\n",
    "\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = torch.optim.Adam(model.parameters())\n",
    "scheduler = ReduceLROnPlateau(optimizer, 'min')\n",
    "\n",
    "EPOCHS = 1\n",
    "\n",
    "print(\"Training\")\n",
    "for epoch in range(EPOCHS):\n",
    "    running_loss = 0.0\n",
    "    steps = list(enumerate(train_dataloader, 0))\n",
    "    for i, data in tqdm.tqdm(steps):\n",
    "        inputs, labels = data['image'].to(device).float(), data['clip_count'].to(device).float()\n",
    "        optimizer.zero_grad()\n",
    "        outputs = model(inputs)\n",
    "        loss = criterion(outputs.view(-1), labels.view(-1))\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        running_loss += loss.item()\n",
    "\n",
    "    scheduler.step(running_loss)\n",
    "    print(f\"Epoch {epoch}/{EPOCHS}, loss: {loss.item()}\")\n",
    "\n",
    "print('Finished Training')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gNiELOX53PtU"
   },
   "source": [
    "This code will run very slowly if you do not use a GPU. The above code takes approximately 13 minutes with a GPU.\n",
    "\n",
    "## Score Regression Image Data\n",
    "\n",
    "Scoring/predicting from a generator is a bit different than training. We do not want augmented images, and we do not wish to have the dataset shuffled. For scoring, we want a prediction for each input. We use the same batch size to guarantee that we do not run out of GPU memory if our prediction set is large and concatenate all of the results together. You can increase this value for better performance. We can now generate a CSV file to hold the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QFXvtr-NtlQt",
    "outputId": "01c75325-c863-4fdd-b6a2-22f9b397ec69"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/5000 [00:00<?, ?it/s]/var/folders/ys/hps4l__90n3_2gns3lzbt9fw0000gn/T/ipykernel_90206/3270005210.py:8: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  images = torch.tensor(data['image']).to(device)\n",
      "  0%|          | 1/5000 [00:00<35:48,  2.33it/s]/var/folders/ys/hps4l__90n3_2gns3lzbt9fw0000gn/T/ipykernel_90206/3270005210.py:8: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  images = torch.tensor(data['image']).to(device)\n",
      "100%|██████████| 5000/5000 [03:25<00:00, 24.29it/s]\n"
     ]
    }
   ],
   "source": [
    "# Testing\n",
    "predictions = []\n",
    "with torch.no_grad():\n",
    "    for data in tqdm.tqdm(test_dataloader):\n",
    "        images = torch.tensor(data['image']).to(device)\n",
    "        outputs = model(images)\n",
    "        predictions.append(outputs.item())\n",
    "\n",
    "df_submit = pd.DataFrame({'id': df_test['id'], 'clip_count': predictions})\n",
    "df_submit.to_csv(\"submit.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sicFJd8u5c3v"
   },
   "source": [
    "## Classification Neural Networks\n",
    "\n",
    "Just like earlier in this module, we will load data. However, this time we will use a dataset of images of three different types of the iris flower. This zip file contains three different directories that specify each image's label. The directories are named the same as the labels:\n",
    "\n",
    "* iris-setosa\n",
    "* iris-versicolour\n",
    "* iris-virginica\n",
    "\n",
    "This dataset contains images of actual flowers, such as Figure 5.IRIS.\n",
    "\n",
    "**Figure 5.IRIS: Iris Flower**\n",
    "\n",
    "![Iris Flower](https://s3.amazonaws.com/data.heatonresearch.com/images/wustl/class/iris-0c826b6f4648edf507e0cafdab53712bb6fd1f04dab453cee8db774a728dd640.jpg \"Iris Flower\")\n",
    "\n",
    "We will begin by loading a local copy of the dataset, like we did earlier for the paperclips.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "zxeLaa1c5gGA"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "URL = \"https://github.com/jeffheaton/data-mirror/releases\"\n",
    "DOWNLOAD_SOURCE = URL+\"/download/v1/iris-image.zip\"\n",
    "DOWNLOAD_NAME = DOWNLOAD_SOURCE[DOWNLOAD_SOURCE.rfind('/')+1:]\n",
    "\n",
    "if COLAB:\n",
    "  PATH = \"/content\"\n",
    "  EXTRACT_TARGET = os.path.join(PATH,\"iris\")\n",
    "  SOURCE = EXTRACT_TARGET # In this case its the same, no subfolder\n",
    "else:\n",
    "  # I used this locally on my machine, you may need different\n",
    "  PATH = \"/Users/jeff/temp\"\n",
    "  EXTRACT_TARGET = os.path.join(PATH,\"iris\")\n",
    "  SOURCE = EXTRACT_TARGET # In this case its the same, no subfolder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hset2s3P9MFV"
   },
   "source": [
    "Just as before, we unzip the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9sQdvl9F6Xru",
    "outputId": "51172c54-1fd8-4f85-a71e-fd899f7d7aaa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-08-15 21:48:56--  https://github.com/jeffheaton/data-mirror/releases/download/v1/iris-image.zip\n",
      "Resolving github.com (github.com)... 140.82.112.3\n",
      "Connecting to github.com (github.com)|140.82.112.3|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/408419764/d548babd-36c3-414e-add2-a5d9ab941e6e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230816%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230816T024856Z&X-Amz-Expires=300&X-Amz-Signature=1902b47c7c916265b57e6a0d77ff04afb0914f67bb2d2f0431b87b94231a4a9c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=408419764&response-content-disposition=attachment%3B%20filename%3Diris-image.zip&response-content-type=application%2Foctet-stream [following]\n",
      "--2023-08-15 21:48:56--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/408419764/d548babd-36c3-414e-add2-a5d9ab941e6e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230816%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230816T024856Z&X-Amz-Expires=300&X-Amz-Signature=1902b47c7c916265b57e6a0d77ff04afb0914f67bb2d2f0431b87b94231a4a9c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=408419764&response-content-disposition=attachment%3B%20filename%3Diris-image.zip&response-content-type=application%2Foctet-stream\n",
      "Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ...\n",
      "Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 5587253 (5.3M) [application/octet-stream]\n",
      "Saving to: ‘/Users/jeff/temp/iris-image.zip’\n",
      "\n",
      "/Users/jeff/temp/ir 100%[===================>]   5.33M  4.87MB/s    in 1.1s    \n",
      "\n",
      "2023-08-15 21:48:57 (4.87 MB/s) - ‘/Users/jeff/temp/iris-image.zip’ saved [5587253/5587253]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# HIDE OUTPUT\n",
    "!wget -O {os.path.join(PATH,DOWNLOAD_NAME)} {DOWNLOAD_SOURCE}\n",
    "!mkdir -p {SOURCE}\n",
    "!mkdir -p {TARGET}\n",
    "!mkdir -p {EXTRACT_TARGET}\n",
    "!unzip -o -d {EXTRACT_TARGET} {os.path.join(PATH, DOWNLOAD_NAME)} >/dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bS1p_9Mn9EPz"
   },
   "source": [
    "You can see these folders with the following command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UrtRO9O-SBQc",
    "outputId": "15f20490-a321-4348-844a-fcc24a0c7cf7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34miris-setosa\u001b[m\u001b[m      \u001b[34miris-versicolour\u001b[m\u001b[m \u001b[34miris-virginica\u001b[m\u001b[m\n"
     ]
    }
   ],
   "source": [
    "!ls {EXTRACT_TARGET}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cYDFKA8i9KMP"
   },
   "source": [
    "We set up a pipeline, similar to before.  We will transform the images, however, we also flip the images for additional augmented data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "u7EgpVqAdGvI",
    "outputId": "7c9cc17b-5425-4378-e0ca-561a68fbb9ce"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torchvision import transforms\n",
    "from torchvision.datasets import ImageFolder\n",
    "from torch.utils.data import DataLoader\n",
    "from sklearn.metrics import accuracy_score\n",
    "import numpy as np\n",
    "\n",
    "# Data augmentation and normalization for training\n",
    "train_transforms = transforms.Compose([\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.RandomVerticalFlip(),\n",
    "    transforms.RandomRotation(360),\n",
    "    transforms.RandomResizedCrop(256, scale=(0.5, 1.0)),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
    "])\n",
    "\n",
    "# Just normalization for validation\n",
    "validation_transforms = transforms.Compose([\n",
    "    transforms.Resize(256),\n",
    "    transforms.CenterCrop(256),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
    "])\n",
    "\n",
    "train_dataset = ImageFolder(root=SOURCE, transform=train_transforms)\n",
    "train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)\n",
    "\n",
    "validation_dataset = ImageFolder(root=SOURCE, transform=validation_transforms)\n",
    "validation_loader = DataLoader(validation_dataset, batch_size=32, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AFesQtNZBZP0"
   },
   "source": [
    "The neural network is similar for classification. However, the output is equal to the number of classes, or iris types, which is 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MBnwiM-XflQc",
    "outputId": "3279a8b4-3828-4ad3-8c03-6361cb10a975"
   },
   "outputs": [],
   "source": [
    "class_count = len(train_dataset.classes)\n",
    "\n",
    "# Define the CNN architecture\n",
    "model = nn.Sequential(\n",
    "            # Features\n",
    "            nn.Conv2d(3, 16, 3),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(2, 2),\n",
    "            nn.Conv2d(16, 32, 3),\n",
    "            nn.ReLU(),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.MaxPool2d(2, 2),\n",
    "            nn.Conv2d(32, 64, 3),\n",
    "            nn.ReLU(),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.MaxPool2d(2, 2),\n",
    "            nn.Conv2d(64, 64, 3),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(2, 2),\n",
    "            nn.Conv2d(64, 64, 3),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(2, 2),\n",
    "\n",
    "            # Classifier\n",
    "            nn.Flatten(),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(64 * 6 * 6, 512),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Linear(512, class_count)).to(device)\n",
    "\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will create function, named **validate** that will loop over the provided validation data and loss function to calculate a score. We simply take an average of each of the batches to compute the overall score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate(model, loader):\n",
    "    loss_num = loss_denom = 0.0\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        for inputs, labels in loader:\n",
    "            outputs = model(inputs.to(device))\n",
    "            loss_num += loss_fn(outputs, labels.to(device))\n",
    "            loss_denom += 1\n",
    "\n",
    "    return loss_num / loss_denom\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training works similar to regression, we loop over the training batches and update the weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 1, tloss: 0.8976501226425171, vloss: 0.991944, : 100%|██████████| 14/14 [00:04<00:00,  3.34it/s]\n",
      "Epoch: 2, tloss: 0.6614288091659546, vloss: 0.953182, Improvement found, counter reset to 0: 100%|██████████| 14/14 [00:01<00:00,  7.57it/s]\n",
      "Epoch: 3, tloss: 0.6952608823776245, vloss: 0.956685, No improvement in the last 1 epochs: 100%|██████████| 14/14 [00:01<00:00,  7.84it/s]\n",
      "Epoch: 4, tloss: 0.5599308609962463, vloss: 0.914633, Improvement found, counter reset to 0: 100%|██████████| 14/14 [00:01<00:00,  7.79it/s]\n",
      "Epoch: 5, tloss: 1.548996925354004, vloss: 0.960487, No improvement in the last 1 epochs: 100%|██████████| 14/14 [00:01<00:00,  7.69it/s]\n",
      "Epoch: 6, tloss: 0.9194130897521973, vloss: 0.917065, No improvement in the last 2 epochs: 100%|██████████| 14/14 [00:01<00:00,  7.64it/s]\n",
      "Epoch: 7, tloss: 1.0552207231521606, vloss: 0.952847, No improvement in the last 3 epochs: 100%|██████████| 14/14 [00:01<00:00,  7.53it/s]\n",
      "Epoch: 8, tloss: 0.6120624542236328, vloss: 0.950160, No improvement in the last 4 epochs: 100%|██████████| 14/14 [00:01<00:00,  7.32it/s]\n",
      "Epoch: 9, tloss: 1.2777814865112305, vloss: 0.934004, Early stopping triggered after 5 epochs.: 100%|██████████| 14/14 [00:01<00:00,  7.43it/s]\n"
     ]
    }
   ],
   "source": [
    "# Create datasets\n",
    "BATCH_SIZE = 16\n",
    "\n",
    "es = EarlyStopping()\n",
    "\n",
    "epoch = 0\n",
    "done = False\n",
    "while epoch < 1000 and not done:\n",
    "    epoch += 1\n",
    "    steps = list(enumerate(train_loader))\n",
    "    pbar = tqdm.tqdm(steps)\n",
    "    model.train()\n",
    "    for i, (x_batch, y_batch) in pbar:\n",
    "        y_batch_pred = model(x_batch.to(device))\n",
    "        loss = loss_fn(y_batch_pred, y_batch.to(device))\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        loss, current = loss.item(), (i + 1) * len(x_batch)\n",
    "        if i == len(steps) - 1:\n",
    "            model.eval()\n",
    "            vloss = validate(model, validation_loader)\n",
    "            if es(model, vloss):\n",
    "                done = True\n",
    "            pbar.set_description(\n",
    "                f\"Epoch: {epoch}, tloss: {loss}, vloss: {vloss:>7f}, {es.status}\"\n",
    "            )\n",
    "        else:\n",
    "            pbar.set_description(f\"Epoch: {epoch}, tloss {loss:}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fmjjtfuv4R9-"
   },
   "source": [
    "The iris image dataset is not easy to predict; it turns out that a tabular dataset of measurements is more manageable.  However, we can achieve a 63%. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BUvoMBK5uYKs",
    "outputId": "f8091a91-f841-45a9-af9a-b531b0741899"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.6389548693586699\n"
     ]
    }
   ],
   "source": [
    "# Validation and accuracy calculation\n",
    "model.eval()\n",
    "preds = []\n",
    "targets = []\n",
    "with torch.no_grad():\n",
    "    for inputs, labels in validation_loader:\n",
    "        outputs = model(inputs.to(device))\n",
    "        _, predictions = torch.max(outputs, 1)\n",
    "        preds.extend(predictions.cpu().numpy())\n",
    "        targets.extend(labels.cpu().numpy())\n",
    "\n",
    "correct = accuracy_score(targets, preds)\n",
    "print(f\"Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QFC6ukZ9cTbp"
   },
   "source": [
    "\n",
    "# Other Resources\n",
    "\n",
    "* [Imagenet:Large Scale Visual Recognition Challenge 2014](http://image-net.org/challenges/LSVRC/2014/index)\n",
    "* [Andrej Karpathy](http://cs.stanford.edu/people/karpathy/) - PhD student/instructor at Stanford.\n",
    "* [CS231n Convolutional Neural Networks for Visual Recognition](http://cs231n.stanford.edu/) - Stanford course on computer vision/CNN's.\n",
    "* [CS231n - GitHub](http://cs231n.github.io/)\n",
    "* [ConvNetJS](http://cs.stanford.edu/people/karpathy/convnetjs/) - JavaScript library for deep learning."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of Copy of t81_558_class_06_2_cnn.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}