M2NIST_Text_Segmentation_FCN8_encoder-decoder_C3W3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    },
    "colab": {
      "name": "Copy of C3W3_Assignment.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Sujay8070/Advanced-Computer-Vision-myTasks/blob/main/M2NIST_Text_Segmentation_inceptionv2_C3W3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k6x_sGPQocpw"
      },
      "source": [
        "# Week 3 Assignment: Image Segmentation of Handwritten Digits\n",
        "\n",
        "<img src='https://drive.google.com/uc?export=view&id=1-WBX7w_R9abwdGmAUBFWWEcJ0LkMppq2' alt='m2nist digits'>\n",
        "\n",
        "In this week's assignment, you will build a model that predicts the segmentation masks (pixel-wise label map) of handwritten digits. This model will be trained on the [M2NIST dataset](https://www.kaggle.com/farhanhubble/multimnistm2nist), a multi digit MNIST. If you've done the ungraded lab on the CamVid dataset, then many of the steps here will look familiar.\n",
        "\n",
        "You will build a Convolutional Neural Network (CNN) from scratch for the downsampling path and use a Fully Convolutional Network, FCN-8, to upsample and produce the pixel-wise label map. The model will be evaluated using the intersection over union (IOU) and Dice Score. Finally, you will download the model and upload it to the grader in Coursera to get your score for the assignment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cnr0kbM0g2LD"
      },
      "source": [
        "## Exercises\n",
        "\n",
        "We've given you some boilerplate code to work with and these are the 5 exercises you need to fill out before you can successfully get the segmentation masks.\n",
        "\n",
        "* [Exercise 1 - Define the Basic Convolution Block](#exercise-1)\n",
        "* [Exercise 2 - Define the Downsampling Path](#exercise-2)\n",
        "* [Exercise 3 - Define the FCN-8 decoder](#exercise-3)\n",
        "* [Exercise 4 - Compile the Model](#exercise-4)\n",
        "* [Exercise 5 - Model Training](#exercise-5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MZ3g9dJxSxmN"
      },
      "source": [
        "## Imports\n",
        "\n",
        "As usual, let's start by importing the packages you will use in this lab."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aifz2907kxYN"
      },
      "source": [
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "\n",
        "import os\n",
        "import zipfile\n",
        "\n",
        "import PIL.Image, PIL.ImageFont, PIL.ImageDraw\n",
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "print(\"Tensorflow version \" + tf.__version__)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7RYh6cCzXE6R"
      },
      "source": [
        "## Download the dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QUGGF3wfqYni"
      },
      "source": [
        "[M2NIST](https://www.kaggle.com/farhanhubble/multimnistm2nist) is a **multi digit** [MNIST](http://yann.lecun.com/exdb/mnist/). \n",
        "Each image has up to 3 digits from MNIST digits and the corresponding labels file has the segmentation masks.\n",
        "\n",
        "The dataset is available on [Kaggle](https://www.kaggle.com) and you can find it [here](https://www.kaggle.com/farhanhubble/multimnistm2nist)\n",
        "\n",
        "To make it easier for you, we're hosting it on Google Cloud so you can download without Kaggle credentials.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ROok0i9rMcu0"
      },
      "source": [
        "# download zipped dataset\n",
        "!wget --no-check-certificate \\\n",
        "    https://storage.googleapis.com/laurencemoroney-blog.appspot.com/m2nist.zip \\\n",
        "    -O /tmp/m2nist.zip\n",
        "\n",
        "# find and extract to a local folder ('/tmp/training')\n",
        "local_zip = '/tmp/m2nist.zip'\n",
        "zip_ref = zipfile.ZipFile(local_zip, 'r')\n",
        "zip_ref.extractall('/tmp/training')\n",
        "zip_ref.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xy17LYR7XJNa"
      },
      "source": [
        "## Load and Preprocess the Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LXGMrWnkrvpK"
      },
      "source": [
        "This dataset can be easily preprocessed since it is available as **Numpy Array Files (.npy)**\n",
        "\n",
        "1. **combined.npy** has the image files containing the multiple MNIST digits. Each image is of size **64 x 84** (height x width, in pixels).\n",
        "\n",
        "2. **segmented.npy** has the corresponding segmentation masks. Each segmentation mask is also of size **64 x 84**.\n",
        "\n",
        "This dataset has **5000** samples and you can make appropriate training, validation, and test splits as required for the problem.\n",
        "\n",
        "With that, let's define a few utility functions for loading and preprocessing the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jy_pw5I2-xLP"
      },
      "source": [
        "BATCH_SIZE = 32\n",
        "\n",
        "def read_image_and_annotation(image, annotation):\n",
        "  '''\n",
        "  Casts the image and annotation to their expected data type and\n",
        "  normalizes the input image so that each pixel is in the range [-1, 1]\n",
        "\n",
        "  Args:\n",
        "    image (numpy array) -- input image\n",
        "    annotation (numpy array) -- ground truth label map\n",
        "\n",
        "  Returns:\n",
        "    preprocessed image-annotation pair\n",
        "  '''\n",
        "\n",
        "  image = tf.cast(image, dtype=tf.float32)\n",
        "  image = tf.reshape(image, (image.shape[0], image.shape[1], 1,))\n",
        "  annotation = tf.cast(annotation, dtype=tf.int32)\n",
        "  image = image / 127.5\n",
        "  image -= 1\n",
        "\n",
        "  return image, annotation\n",
        "\n",
        "\n",
        "def get_training_dataset(images, annos):\n",
        "  '''\n",
        "  Prepares shuffled batches of the training set.\n",
        "  \n",
        "  Args:\n",
        "    images (list of strings) -- paths to each image file in the train set\n",
        "    annos (list of strings) -- paths to each label map in the train set\n",
        "\n",
        "  Returns:\n",
        "    tf Dataset containing the preprocessed train set\n",
        "  '''\n",
        "  training_dataset = tf.data.Dataset.from_tensor_slices((images, annos))\n",
        "  training_dataset = training_dataset.map(read_image_and_annotation)\n",
        "\n",
        "  training_dataset = training_dataset.shuffle(512, reshuffle_each_iteration=True)\n",
        "  training_dataset = training_dataset.batch(BATCH_SIZE)\n",
        "  training_dataset = training_dataset.repeat()\n",
        "  training_dataset = training_dataset.prefetch(-1)\n",
        "\n",
        "  return training_dataset\n",
        "\n",
        "\n",
        "def get_validation_dataset(images, annos):\n",
        "  '''\n",
        "  Prepares batches of the validation set.\n",
        "  \n",
        "  Args:\n",
        "    images (list of strings) -- paths to each image file in the val set\n",
        "    annos (list of strings) -- paths to each label map in the val set\n",
        "\n",
        "  Returns:\n",
        "    tf Dataset containing the preprocessed validation set\n",
        "  '''\n",
        "  validation_dataset = tf.data.Dataset.from_tensor_slices((images, annos))\n",
        "  validation_dataset = validation_dataset.map(read_image_and_annotation)\n",
        "  validation_dataset = validation_dataset.batch(BATCH_SIZE)\n",
        "  validation_dataset = validation_dataset.repeat()\n",
        "\n",
        "  return validation_dataset\n",
        "\n",
        "\n",
        "def get_test_dataset(images, annos):\n",
        "  '''\n",
        "  Prepares batches of the test set.\n",
        "  \n",
        "  Args:\n",
        "    images (list of strings) -- paths to each image file in the test set\n",
        "    annos (list of strings) -- paths to each label map in the test set\n",
        "\n",
        "  Returns:\n",
        "    tf Dataset containing the preprocessed validation set\n",
        "  '''\n",
        "  test_dataset = tf.data.Dataset.from_tensor_slices((images, annos))\n",
        "  test_dataset = test_dataset.map(read_image_and_annotation)\n",
        "  test_dataset = test_dataset.batch(BATCH_SIZE, drop_remainder=True)\n",
        "\n",
        "  return test_dataset\n",
        "\n",
        "\n",
        "def load_images_and_segments():\n",
        "  '''\n",
        "  Loads the images and segments as numpy arrays from npy files \n",
        "  and makes splits for training, validation and test datasets.\n",
        "\n",
        "  Returns:\n",
        "    3 tuples containing the train, val, and test splits\n",
        "  '''\n",
        "\n",
        "  #Loads images and segmentation masks.\n",
        "  images = np.load('/tmp/training/combined.npy')\n",
        "  segments = np.load('/tmp/training/segmented.npy')\n",
        "\n",
        "  #Makes training, validation, test splits from loaded images and segmentation masks.\n",
        "  train_images, val_images, train_annos, val_annos = train_test_split(images, segments, test_size=0.2, shuffle=True)\n",
        "  val_images, test_images, val_annos, test_annos = train_test_split(val_images, val_annos, test_size=0.2, shuffle=True)\n",
        "\n",
        "  return (train_images, train_annos), (val_images, val_annos), (test_images, test_annos)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iPHO1YbTACcu"
      },
      "source": [
        "You can now load the preprocessed dataset and define the training, validation, and test sets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hIS70_um_Y7n",
        "lines_to_next_cell": 2
      },
      "source": [
        "# Load Dataset\n",
        "train_slices, val_slices, test_slices = load_images_and_segments()\n",
        "\n",
        "# Create training, validation, test datasets.\n",
        "training_dataset = get_training_dataset(train_slices[0], train_slices[1])\n",
        "validation_dataset = get_validation_dataset(val_slices[0], val_slices[1])\n",
        "test_dataset = get_test_dataset(test_slices[0], test_slices[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oKXJYZi7A0dF"
      },
      "source": [
        "## Let's Take a Look at the Dataset\n",
        "\n",
        "You may want to visually inspect the dataset before and after training. Like above, we've included utility functions to help show a few images as well as their annotations (i.e. labels)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "both",
        "id": "d46YCbvPafbp"
      },
      "source": [
        "# Visualization Utilities\n",
        "\n",
        "# there are 11 classes in the dataset: one class for each digit (0 to 9) plus the background class\n",
        "n_classes = 11\n",
        "\n",
        "# assign a random color for each class\n",
        "colors = [tuple(np.random.randint(256, size=3) / 255.0) for i in range(n_classes)]\n",
        "\n",
        "def fuse_with_pil(images):\n",
        "  '''\n",
        "  Creates a blank image and pastes input images\n",
        "\n",
        "  Args:\n",
        "    images (list of numpy arrays) - numpy array representations of the images to paste\n",
        "  \n",
        "  Returns:\n",
        "    PIL Image object containing the images\n",
        "  '''\n",
        "\n",
        "  widths = (image.shape[1] for image in images)\n",
        "  heights = (image.shape[0] for image in images)\n",
        "  total_width = sum(widths)\n",
        "  max_height = max(heights)\n",
        "\n",
        "  new_im = PIL.Image.new('RGB', (total_width, max_height))\n",
        "\n",
        "  x_offset = 0\n",
        "  for im in images:\n",
        "    pil_image = PIL.Image.fromarray(np.uint8(im))\n",
        "    new_im.paste(pil_image, (x_offset,0))\n",
        "    x_offset += im.shape[1]\n",
        "  \n",
        "  return new_im\n",
        "\n",
        "\n",
        "def give_color_to_annotation(annotation):\n",
        "  '''\n",
        "  Converts a 2-D annotation to a numpy array with shape (height, width, 3) where\n",
        "  the third axis represents the color channel. The label values are multiplied by\n",
        "  255 and placed in this axis to give color to the annotation\n",
        "\n",
        "  Args:\n",
        "    annotation (numpy array) - label map array\n",
        "  \n",
        "  Returns:\n",
        "    the annotation array with an additional color channel/axis\n",
        "  '''\n",
        "  seg_img = np.zeros( (annotation.shape[0],annotation.shape[1], 3) ).astype('float')\n",
        "  \n",
        "  for c in range(n_classes):\n",
        "    segc = (annotation == c)\n",
        "    seg_img[:,:,0] += segc*( colors[c][0] * 255.0)\n",
        "    seg_img[:,:,1] += segc*( colors[c][1] * 255.0)\n",
        "    seg_img[:,:,2] += segc*( colors[c][2] * 255.0)\n",
        "  \n",
        "  return seg_img\n",
        "\n",
        "\n",
        "def show_annotation_and_prediction(image, annotation, prediction, iou_list, dice_score_list):\n",
        "  '''\n",
        "  Displays the images with the ground truth and predicted label maps. Also overlays the metrics.\n",
        "\n",
        "  Args:\n",
        "    image (numpy array) -- the input image\n",
        "    annotation (numpy array) -- the ground truth label map\n",
        "    prediction (numpy array) -- the predicted label map\n",
        "    iou_list (list of floats) -- the IOU values for each class\n",
        "    dice_score_list (list of floats) -- the Dice Score for each class\n",
        "  '''\n",
        "\n",
        "  new_ann = np.argmax(annotation, axis=2)\n",
        "  true_img = give_color_to_annotation(new_ann)\n",
        "  pred_img = give_color_to_annotation(prediction)\n",
        "\n",
        "  image = image + 1\n",
        "  image = image * 127.5\n",
        "  image = np.reshape(image, (image.shape[0], image.shape[1],))\n",
        "  image = np.uint8(image)\n",
        "  images = [image, np.uint8(pred_img), np.uint8(true_img)]\n",
        "\n",
        "  metrics_by_id = [(idx, iou, dice_score) for idx, (iou, dice_score) in enumerate(zip(iou_list, dice_score_list)) if iou > 0.0 and idx < 10]\n",
        "  metrics_by_id.sort(key=lambda tup: tup[1], reverse=True)  # sorts in place\n",
        "\n",
        "  display_string_list = [\"{}: IOU: {} Dice Score: {}\".format(idx, iou, dice_score) for idx, iou, dice_score in metrics_by_id]\n",
        "  display_string = \"\\n\".join(display_string_list)\n",
        "\n",
        "  plt.figure(figsize=(15, 4))\n",
        "\n",
        "  for idx, im in enumerate(images):\n",
        "    plt.subplot(1, 3, idx+1)\n",
        "    if idx == 1:\n",
        "      plt.xlabel(display_string)\n",
        "    plt.xticks([])\n",
        "    plt.yticks([])\n",
        "    plt.imshow(im)\n",
        "\n",
        "\n",
        "def show_annotation_and_image(image, annotation):\n",
        "  '''\n",
        "  Displays the image and its annotation side by side\n",
        "\n",
        "  Args:\n",
        "    image (numpy array) -- the input image\n",
        "    annotation (numpy array) -- the label map\n",
        "  '''\n",
        "  new_ann = np.argmax(annotation, axis=2)\n",
        "  seg_img = give_color_to_annotation(new_ann)\n",
        "  \n",
        "  image = image + 1\n",
        "  image = image * 127.5\n",
        "  image = np.reshape(image, (image.shape[0], image.shape[1],))\n",
        "\n",
        "  image = np.uint8(image)\n",
        "  images = [image, seg_img]\n",
        "  \n",
        "  images = [image, seg_img]\n",
        "  fused_img = fuse_with_pil(images)\n",
        "  plt.imshow(fused_img)\n",
        "\n",
        "\n",
        "def list_show_annotation(dataset, num_images):\n",
        "  '''\n",
        "  Displays images and its annotations side by side\n",
        "\n",
        "  Args:\n",
        "    dataset (tf Dataset) -- batch of images and annotations\n",
        "    num_images (int) -- number of images to display\n",
        "  '''\n",
        "  ds = dataset.unbatch()\n",
        "\n",
        "  plt.figure(figsize=(20, 15))\n",
        "  plt.title(\"Images And Annotations\")\n",
        "  plt.subplots_adjust(bottom=0.1, top=0.9, hspace=0.05)\n",
        "\n",
        "  for idx, (image, annotation) in enumerate(ds.take(num_images)):\n",
        "    plt.subplot(5, 5, idx + 1)\n",
        "    plt.yticks([])\n",
        "    plt.xticks([])\n",
        "    show_annotation_and_image(image.numpy(), annotation.numpy())\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hEfyChmKEFKe"
      },
      "source": [
        "You can view a subset of the images from the dataset with the `list_show_annotation()` function defined above. Run the cells below to see the image on the left and its pixel-wise ground truth label map on the right."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AFO_hIhLWYT4"
      },
      "source": [
        "# get 10 images from the training set\n",
        "list_show_annotation(training_dataset, 10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bdgVkp8wZua0"
      },
      "source": [
        "# get 10 images from the validation set\n",
        "list_show_annotation(validation_dataset, 5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KkpgIcE2FeKI"
      },
      "source": [
        "You see from the images above the colors assigned to each class (i.e 0 to 9 plus the background). If you don't like these colors, feel free to rerun the cell where `colors` is defined to get another set of random colors. Alternatively, you can assign the RGB values for each class instead of relying on random values."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QFv2k8xabRb8"
      },
      "source": [
        "## Define the Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q_ylpyqJDQiF"
      },
      "source": [
        "As discussed in the lectures, the image segmentation model will have two paths:\n",
        "\n",
        "1. **Downsampling Path** - This part of the network extracts the features in the image. This is done through a series of convolution and pooling layers. The final output is a reduced image (because of the pooling layers) with the extracted features. You will build a custom CNN from scratch for this path.\n",
        "\n",
        "2. **Upsampling Path** - This takes the output of the downsampling path and generates the predictions while also converting the image back to its original size. You will use an FCN-8 decoder for this path."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BHlBUZvsDybt"
      },
      "source": [
        "### Define the Basic Convolution Block"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T-XO9xkN1OR3"
      },
      "source": [
        "<a name='exercise-1'></a>\n",
        "\n",
        "#### **Exercise 1**\n",
        "\n",
        "Please complete the function below to build the basic convolution block for our CNN. This will have two [Conv2D](https://keras.io/api/layers/convolution_layers/convolution2d/) layers each followed by a [LeakyReLU](https://keras.io/api/layers/activation_layers/leaky_relu/), then [max pooled](https://keras.io/api/layers/pooling_layers/max_pooling2d/) and [batch-normalized](https://keras.io/api/layers/normalization_layers/batch_normalization/). Use the functional syntax to stack these layers.\n",
        "\n",
        "$$Input -> Conv2D -> LeakyReLU -> Conv2D -> LeakyReLU -> MaxPooling2D -> BatchNormalization$$\n",
        "\n",
        "When defining the Conv2D layers, note that our data inputs will have the 'channels' dimension last. You may want to check the `data_format` argument in the [docs](https://keras.io/api/layers/convolution_layers/convolution2d/) regarding this. Take note of the `padding` argument too like you did in the ungraded labs.\n",
        "\n",
        "Lastly, to use the `LeakyReLU` activation, you **do not** need to nest it inside an `Activation` layer (e.g. `x = tf.keras.layers.Activation(tf.keras.layers.LeakyReLU()(x)`). You can simply stack the layer directly instead (e.g. `x = tf.keras.layers.LeakyReLU()(x)`)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "azEEVytHR0Kn"
      },
      "source": [
        "# parameter describing where the channel dimension is found in our dataset\n",
        "IMAGE_ORDERING = 'channels_last'\n",
        "\n",
        "def conv_block(input, filters, kernel_size, pooling_size, pool_strides):\n",
        "  '''\n",
        "  Args:\n",
        "    input (tensor) -- batch of images or features\n",
        "    filters (int) -- number of filters of the Conv2D layers\n",
        "    kernel_size (int) -- kernel_size setting of the Conv2D layers\n",
        "    pooling_size (int) -- pooling size of the MaxPooling2D layers\n",
        "    pool_strides (int) -- strides setting of the MaxPooling2D layers\n",
        "  \n",
        "  Returns:\n",
        "    (tensor) max pooled and batch-normalized features of the input \n",
        "  '''\n",
        "  ### START CODE HERE ###\n",
        "  # use the functional syntax to stack the layers as shown in the diagram above\n",
        "  x = tf.keras.layers.Conv2D(filters, kernel_size, padding='same', data_format=IMAGE_ORDERING)(input)\n",
        "  x = tf.keras.layers.LeakyReLU()(x)\n",
        "  x = tf.keras.layers.Conv2D(filters, kernel_size, padding='same', data_format=IMAGE_ORDERING)(x)\n",
        "  x = tf.keras.layers.LeakyReLU()(x)\n",
        "  x = tf.keras.layers.MaxPooling2D(2)(x)\n",
        "  x = tf.keras.layers.BatchNormalization()(x)\n",
        "  ### END CODE HERE ###\n",
        "\n",
        "  return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uGtlHuM6KCRg"
      },
      "source": [
        "# TEST CODE:\n",
        "\n",
        "test_input = tf.keras.layers.Input(shape=(64,84, 1))\n",
        "test_output = conv_block(test_input, 32, 3, 2, 2)\n",
        "test_model = tf.keras.Model(inputs=test_input, outputs=test_output)\n",
        "\n",
        "print(test_model.summary())\n",
        "\n",
        "# free up test resources\n",
        "del test_input, test_output, test_model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mmf_ZaMP2rV3"
      },
      "source": [
        "**Expected Output**:\n",
        "\n",
        "Please pay attention to the *(type)* and *Output Shape* columns. The *Layer* name beside the type may be different depending on how many times you ran the cell (e.g. `input_7` can be `input_1`)\n",
        "\n",
        "```txt\n",
        "Model: \"functional_1\"\n",
        "_________________________________________________________________\n",
        "Layer (type)                 Output Shape              Param #   \n",
        "=================================================================\n",
        "input_1 (InputLayer)         [(None, 64, 84, 1)]       0         \n",
        "_________________________________________________________________\n",
        "conv2d (Conv2D)              (None, 64, 84, 32)        320       \n",
        "_________________________________________________________________\n",
        "leaky_re_lu (LeakyReLU)      (None, 64, 84, 32)        0         \n",
        "_________________________________________________________________\n",
        "conv2d_1 (Conv2D)            (None, 64, 84, 32)        9248      \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_1 (LeakyReLU)    (None, 64, 84, 32)        0         \n",
        "_________________________________________________________________\n",
        "max_pooling2d (MaxPooling2D) (None, 32, 42, 32)        0         \n",
        "_________________________________________________________________\n",
        "batch_normalization (BatchNo (None, 32, 42, 32)        128       \n",
        "=================================================================\n",
        "Total params: 9,696\n",
        "Trainable params: 9,632\n",
        "Non-trainable params: 64\n",
        "_________________________________________________________________\n",
        "None\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4-jJbC91EXTV"
      },
      "source": [
        "### Define the Downsampling Path"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e2ueOTmc3HxJ"
      },
      "source": [
        "<a name='exercise-2'></a>\n",
        "\n",
        "#### **Exercise 2**\n",
        "\n",
        "Now that we've defined the building block of our encoder, you can now build the downsampling path. Please complete the function below to create the encoder. This should chain together five convolution building blocks to create a feature extraction CNN minus the fully connected layers.\n",
        "\n",
        "*Notes*: \n",
        "1. To optimize processing or to make the output dimensions of each layer easier to work with, it is sometimes advisable to apply some zero-padding to the input image. With the boilerplate code we have provided below, we have padded the input width to 96 pixels using the [ZeroPadding2D layer](https://keras.io/api/layers/reshaping_layers/zero_padding2d/). This works well if you're going to use the first ungraded lab of this week as reference. This is not required however. You can remove it later and see how it will affect your parameters. For instance, you might need to pass in a non-square kernel size to the decoder in Exercise 3 (e.g. `(4,5)`) to match the output dimensions of Exercise 2. \n",
        "\n",
        "2. We recommend keeping the pool size and stride parameters constant at 2."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F2VNB99LRwQr"
      },
      "source": [
        "def FCN8(input_height=64, input_width=84):\n",
        "    '''\n",
        "    Defines the downsampling path of the image segmentation model.\n",
        "\n",
        "    Args:\n",
        "      input_height (int) -- height of the images\n",
        "      width (int) -- width of the images\n",
        "\n",
        "    Returns:\n",
        "    (tuple of tensors, tensor)\n",
        "      tuple of tensors -- features extracted at blocks 3 to 5\n",
        "      tensor -- copy of the input\n",
        "    '''\n",
        "    img_input = tf.keras.layers.Input(shape=(input_height,input_width, 1))\n",
        "\n",
        "    ### START CODE HERE ###\n",
        "    \n",
        "    # pad the input image to have dimensions to the nearest power of two\n",
        "    x = tf.keras.layers.ZeroPadding2D((0,6))(img_input)\n",
        "\n",
        "    # Block 1\n",
        "    x = conv_block(x,32,3,2,2)\n",
        "    \n",
        "    # Block 2\n",
        "    x = conv_block(x,64,3,2,2)\n",
        "\n",
        "    # Block 3\n",
        "    x = conv_block(x,128,3,2,2)\n",
        "\n",
        "    # save the feature map at this stage\n",
        "    f3 = x\n",
        "\n",
        "    # Block 4\n",
        "    x = conv_block(x,256,3,2,2)\n",
        "    # save the feature map at this stage\n",
        "    f4 = x\n",
        "\n",
        "    # Block 5\n",
        "    x = conv_block(x,256,3,2,2)\n",
        "    # save the feature map at this stage\n",
        "    f5 = x\n",
        "\n",
        "\n",
        "    ### END CODE HERE ###\n",
        "  \n",
        "    return (f3, f4, f5), img_input"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HVQm1W0CNICS"
      },
      "source": [
        "# TEST CODE:\n",
        "\n",
        "test_convs, test_img_input = FCN8()\n",
        "test_model = tf.keras.Model(inputs=test_img_input, outputs=[test_convs, test_img_input])\n",
        "\n",
        "print(test_model.summary())\n",
        "\n",
        "del test_convs, test_img_input, test_model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sxklZe_D3jOI"
      },
      "source": [
        "**Expected Output**:\n",
        "\n",
        "You should see the layers of your `conv_block()` being repeated 5 times like the output below.\n",
        "\n",
        "```txt\n",
        "Model: \"functional_3\"\n",
        "_________________________________________________________________\n",
        "Layer (type)                 Output Shape              Param #   \n",
        "=================================================================\n",
        "input_3 (InputLayer)         [(None, 64, 84, 1)]       0         \n",
        "_________________________________________________________________\n",
        "zero_padding2d (ZeroPadding2 (None, 64, 96, 1)         0         \n",
        "_________________________________________________________________\n",
        "conv2d_2 (Conv2D)            (None, 64, 96, 32)        320       \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_2 (LeakyReLU)    (None, 64, 96, 32)        0         \n",
        "_________________________________________________________________\n",
        "conv2d_3 (Conv2D)            (None, 64, 96, 32)        9248      \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_3 (LeakyReLU)    (None, 64, 96, 32)        0         \n",
        "_________________________________________________________________\n",
        "max_pooling2d_1 (MaxPooling2 (None, 32, 48, 32)        0         \n",
        "_________________________________________________________________\n",
        "batch_normalization_1 (Batch (None, 32, 48, 32)        128       \n",
        "_________________________________________________________________\n",
        "conv2d_4 (Conv2D)            (None, 32, 48, 64)        18496     \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_4 (LeakyReLU)    (None, 32, 48, 64)        0         \n",
        "_________________________________________________________________\n",
        "conv2d_5 (Conv2D)            (None, 32, 48, 64)        36928     \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_5 (LeakyReLU)    (None, 32, 48, 64)        0         \n",
        "_________________________________________________________________\n",
        "max_pooling2d_2 (MaxPooling2 (None, 16, 24, 64)        0         \n",
        "_________________________________________________________________\n",
        "batch_normalization_2 (Batch (None, 16, 24, 64)        256       \n",
        "_________________________________________________________________\n",
        "conv2d_6 (Conv2D)            (None, 16, 24, 128)       73856     \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_6 (LeakyReLU)    (None, 16, 24, 128)       0         \n",
        "_________________________________________________________________\n",
        "conv2d_7 (Conv2D)            (None, 16, 24, 128)       147584    \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_7 (LeakyReLU)    (None, 16, 24, 128)       0         \n",
        "_________________________________________________________________\n",
        "max_pooling2d_3 (MaxPooling2 (None, 8, 12, 128)        0         \n",
        "_________________________________________________________________\n",
        "batch_normalization_3 (Batch (None, 8, 12, 128)        512       \n",
        "_________________________________________________________________\n",
        "conv2d_8 (Conv2D)            (None, 8, 12, 256)        295168    \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_8 (LeakyReLU)    (None, 8, 12, 256)        0         \n",
        "_________________________________________________________________\n",
        "conv2d_9 (Conv2D)            (None, 8, 12, 256)        590080    \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_9 (LeakyReLU)    (None, 8, 12, 256)        0         \n",
        "_________________________________________________________________\n",
        "max_pooling2d_4 (MaxPooling2 (None, 4, 6, 256)         0         \n",
        "_________________________________________________________________\n",
        "batch_normalization_4 (Batch (None, 4, 6, 256)         1024      \n",
        "_________________________________________________________________\n",
        "conv2d_10 (Conv2D)           (None, 4, 6, 256)         590080    \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_10 (LeakyReLU)   (None, 4, 6, 256)         0         \n",
        "_________________________________________________________________\n",
        "conv2d_11 (Conv2D)           (None, 4, 6, 256)         590080    \n",
        "_________________________________________________________________\n",
        "leaky_re_lu_11 (LeakyReLU)   (None, 4, 6, 256)         0         \n",
        "_________________________________________________________________\n",
        "max_pooling2d_5 (MaxPooling2 (None, 2, 3, 256)         0         \n",
        "_________________________________________________________________\n",
        "batch_normalization_5 (Batch (None, 2, 3, 256)         1024      \n",
        "=================================================================\n",
        "Total params: 2,354,784\n",
        "Trainable params: 2,353,312\n",
        "Non-trainable params: 1,472\n",
        "_________________________________________________________________\n",
        "None\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LbjYEQU8Eq-T"
      },
      "source": [
        "### Define the FCN-8 decoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ux5nAU155E5O"
      },
      "source": [
        "<a name='exercise-3'></a>\n",
        "\n",
        "#### **Exercise 3**\n",
        "\n",
        "Now you can define the upsampling path taking the outputs of convolutions at each stage as arguments. This will be very similar to what you did in the ungraded lab (VGG16-FCN8-CamVid) so you can refer to it if you need a refresher. \n",
        "* Note: remember to set the `data_format` parameter for the Conv2D layers. \n",
        "\n",
        "Here is also the diagram you saw in class on how it should work:\n",
        "\n",
        "<img src='https://drive.google.com/uc?export=view&id=1lrqB4YegV8jXWNfyYAaeuFlwXIc54aRP' alt='fcn-8'>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "giYEct_Se5Xj"
      },
      "source": [
        "def fcn8_decoder(convs, n_classes):\n",
        "  # features from the encoder stage\n",
        "  f3, f4, f5 = convs\n",
        "\n",
        "  # number of filters\n",
        "  n = 512\n",
        "\n",
        "  # add convolutional layers on top of the CNN extractor.\n",
        "  o = tf.keras.layers.Conv2D(n , (7 , 7) , activation='relu' , padding='same', name=\"conv6\", data_format=IMAGE_ORDERING)(f5)\n",
        "  o = tf.keras.layers.Dropout(0.5)(o)\n",
        "\n",
        "  o = tf.keras.layers.Conv2D(n , (1 , 1) , activation='relu' , padding='same', name=\"conv7\", data_format=IMAGE_ORDERING)(o)\n",
        "  o = tf.keras.layers.Dropout(0.5)(o)\n",
        "\n",
        "  o = tf.keras.layers.Conv2D(n_classes,  (1, 1), activation='relu' , padding='same', data_format=IMAGE_ORDERING)(o)\n",
        "\n",
        "    \n",
        "  ### START CODE HERE ###\n",
        "\n",
        "  # upsample the output of the encoder then crop extra pixels that were introduced\n",
        "  o = tf.keras.layers.Conv2DTranspose(n_classes , kernel_size=(4,4) ,  strides=(2,2) , use_bias=False )(f5)\n",
        "  o = tf.keras.layers.Cropping2D(cropping=(1,1))(o)\n",
        "\n",
        "  # load the pool 4 prediction and do a 1x1 convolution to reshape it to the same shape of `o` above\n",
        "  o2 = f4\n",
        "  o2 = ( tf.keras.layers.Conv2D(n_classes , ( 1 , 1 ) , activation='relu' , padding='same'))(o2)\n",
        "\n",
        "  # add the results of the upsampling and pool 4 prediction\n",
        "  o = tf.keras.layers.Add()([o, o2])\n",
        "\n",
        "  # upsample the resulting tensor of the operation you just did\n",
        "  o = (tf.keras.layers.Conv2DTranspose( n_classes , kernel_size=(4,4) ,  strides=(2,2) , use_bias=False ))(o)\n",
        "  o = tf.keras.layers.Cropping2D(cropping=(1, 1))(o)\n",
        "\n",
        "  # load the pool 3 prediction and do a 1x1 convolution to reshape it to the same shape of `o` above\n",
        "  o2 = f3\n",
        "  o2 = ( tf.keras.layers.Conv2D(n_classes , ( 1 , 1 ) , activation='relu' , padding='same'))(o2)\n",
        "\n",
        "  # add the results of the upsampling and pool 3 prediction\n",
        "  o = tf.keras.layers.Add()([o, o2])\n",
        "\n",
        "  # upsample up to the size of the original image\n",
        "  o = tf.keras.layers.Conv2DTranspose(n_classes , kernel_size=(8,8) ,  strides=(8,8) , use_bias=False )(o)\n",
        "  o = tf.keras.layers.Cropping2D(((0, 0), (0, 96-84)))(o)\n",
        "\n",
        "  # append a sigmoid activation\n",
        "  o = (tf.keras.layers.Activation('softmax'))(o)\n",
        "  ### END CODE HERE ###\n",
        "\n",
        "  return o"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lQHioDlR5K1_"
      },
      "source": [
        "# TEST CODE\n",
        "\n",
        "test_convs, test_img_input = FCN8()\n",
        "test_fcn8_decoder = fcn8_decoder(test_convs, 11)\n",
        "\n",
        "print(test_fcn8_decoder.shape)\n",
        "\n",
        "del test_convs, test_img_input, test_fcn8_decoder"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C1qTwTX-5fwH"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```txt\n",
        "(None, 64, 84, 11)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cJOhQz86Qk6n"
      },
      "source": [
        "### Define the Complete Model\n",
        "\n",
        "The downsampling and upsampling paths can now be combined as shown below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9EJEf484312h",
        "lines_to_next_cell": 2
      },
      "source": [
        "# start the encoder using the default input size 64 x 84\n",
        "convs, img_input = FCN8()\n",
        "\n",
        "# pass the convolutions obtained in the encoder to the decoder\n",
        "dec_op = fcn8_decoder(convs, n_classes)\n",
        "\n",
        "# define the model specifying the input (batch of images) and output (decoder output)\n",
        "model = tf.keras.Model(inputs = img_input, outputs = dec_op)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2GAenp1M4gXx"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wAAXygZtbZmu"
      },
      "source": [
        "## Compile the Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sC-9m1R_5tjV"
      },
      "source": [
        "<a name='exercise-4'></a>\n",
        "\n",
        "### **Exercise 4**\n",
        "\n",
        "Compile the model using an appropriate loss, optimizer, and metric.\n",
        "\n",
        "_**Note:** There is a current issue with the grader accepting certain loss functions. We will be upgrading it but while in progress, please use this syntax:_\n",
        "\n",
        "```\n",
        "loss='<loss string name>'\n",
        "```\n",
        "\n",
        "*instead of:*\n",
        "\n",
        "```\n",
        "loss=tf.keras.losses.<StringCassName>\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZpWpp8h4g_rE"
      },
      "source": [
        "### START CODE HERE ###\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "### END CODE HERE ###"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "510v0aVDXv1f"
      },
      "source": [
        "## Model Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1udK8qps6PcG"
      },
      "source": [
        "<a name='exercise-5'></a>\n",
        "\n",
        "### **Exercise 5**\n",
        "\n",
        "You can now train the model. Set the number of epochs and observe the metrics returned at each iteration. You can also terminate the cell execution if you think your model is performing well already."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8HoZwpGWhMB-"
      },
      "source": [
        "# OTHER THAN SETTING THE EPOCHS NUMBER, DO NOT CHANGE ANY OTHER CODE\n",
        "\n",
        "### START CODE HERE ###\n",
        "EPOCHS = 100\n",
        "### END CODE HERE ###\n",
        "\n",
        "steps_per_epoch = 4000//BATCH_SIZE\n",
        "validation_steps = 800//BATCH_SIZE\n",
        "test_steps = 200//BATCH_SIZE\n",
        "\n",
        "\n",
        "history = model.fit(training_dataset,\n",
        "                    steps_per_epoch=steps_per_epoch, validation_data=validation_dataset, validation_steps=validation_steps, epochs=EPOCHS)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BLNzLB3peeNG"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "The losses should generally be decreasing and the accuracies should generally be increasing. For example, observing the first 4 epochs should output something similar:\n",
        "\n",
        "```txt\n",
        "Epoch 1/70\n",
        "125/125 [==============================] - 6s 50ms/step - loss: 0.5542 - accuracy: 0.8635 - val_loss: 0.5335 - val_accuracy: 0.9427\n",
        "Epoch 2/70\n",
        "125/125 [==============================] - 6s 47ms/step - loss: 0.2315 - accuracy: 0.9425 - val_loss: 0.3362 - val_accuracy: 0.9427\n",
        "Epoch 3/70\n",
        "125/125 [==============================] - 6s 47ms/step - loss: 0.2118 - accuracy: 0.9426 - val_loss: 0.2592 - val_accuracy: 0.9427\n",
        "Epoch 4/70\n",
        "125/125 [==============================] - 6s 47ms/step - loss: 0.1782 - accuracy: 0.9431 - val_loss: 0.1770 - val_accuracy: 0.9432\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eih-Q7GoXzJe"
      },
      "source": [
        "## Model Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2bTkaFM2X1gr"
      },
      "source": [
        "### Make Predictions\n",
        "\n",
        "Let's get the predictions using our test dataset as input and print the shape."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zENjQuK0luH5"
      },
      "source": [
        "results = model.predict(test_dataset, steps=test_steps)\n",
        "\n",
        "print(results.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IReKPT_DZHjA"
      },
      "source": [
        "As you can see, the resulting shape is `(192, 64, 84, 11)`. This means that for each of the 192 images that we have in our test set, there are 11 predictions generated (i.e. one for each class: 0 to 1 plus background)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OBPBqnHyaSaG"
      },
      "source": [
        "Thus, if you want to see the *probability* of the upper leftmost pixel of the 1st image belonging to class 0, then you can print something like `results[0,0,0,0]`. If you want the probability of the same pixel at class 10, then do `results[0,0,0,10]`.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jwFiR9WAf0Av"
      },
      "source": [
        "print(results[0,0,0,0])\n",
        "print(results[0,0,0,10])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wKDsqzKEf28V"
      },
      "source": [
        "What we're interested in is to get the *index* of the highest probability of each of these 11 slices and combine them in a single image. We can do that by getting the [argmax](https://numpy.org/doc/stable/reference/generated/numpy.argmax.html) at this axis."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0_Uj_uuV9TQt"
      },
      "source": [
        "results = np.argmax(results, axis=3)\n",
        "\n",
        "print(results.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ClaGbOLhdKD2"
      },
      "source": [
        "The new array generated per image now only specifies the indices of the class with the highest probability. Let's see the output class of the upper most left pixel. As you might have observed earlier when you inspected the dataset, the upper left corner is usually just part of the background (class 10). The actual digits are written somewhere in the middle parts of the image."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aBeBwvHQd2pZ"
      },
      "source": [
        "print(results[0,0,0])\n",
        "\n",
        "# prediction map for image 0\n",
        "print(results[0,:,:])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V3FxyNgrg7IE"
      },
      "source": [
        "We will use this `results` array when we evaluate our predictions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpKDUuAWX5Pj"
      },
      "source": [
        "### Metrics\n",
        "\n",
        "We showed in the lectures two ways to evaluate your predictions. The *intersection over union (IOU)* and the *dice score*. Recall that:\n",
        "\n",
        "$$IOU = \\frac{area\\_of\\_overlap}{area\\_of\\_union}$$\n",
        "<br>\n",
        "$$Dice Score = 2 * \\frac{area\\_of\\_overlap}{combined\\_area}$$\n",
        "\n",
        "The code below does that for you as you've also seen in the ungraded lab. A small smoothing factor is introduced in the denominators to prevent possible division by zero."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gKTpLmly_RXb"
      },
      "source": [
        "def class_wise_metrics(y_true, y_pred):\n",
        "  '''\n",
        "  Computes the class-wise IOU and Dice Score.\n",
        "\n",
        "  Args:\n",
        "    y_true (tensor) - ground truth label maps\n",
        "    y_pred (tensor) - predicted label maps\n",
        "  '''\n",
        "  class_wise_iou = []\n",
        "  class_wise_dice_score = []\n",
        "\n",
        "  smoothing_factor = 0.00001\n",
        "\n",
        "  for i in range(n_classes):\n",
        "    intersection = np.sum((y_pred == i) * (y_true == i))\n",
        "    y_true_area = np.sum((y_true == i))\n",
        "    y_pred_area = np.sum((y_pred == i))\n",
        "    combined_area = y_true_area + y_pred_area\n",
        "    \n",
        "    iou = (intersection) / (combined_area - intersection + smoothing_factor)\n",
        "    class_wise_iou.append(iou)\n",
        "    \n",
        "    dice_score =  2 * ((intersection) / (combined_area + smoothing_factor))\n",
        "    class_wise_dice_score.append(dice_score)\n",
        "\n",
        "  return class_wise_iou, class_wise_dice_score\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FfWPwM4ZhHjE"
      },
      "source": [
        "### Visualize Predictions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hkbsk_P1fpRM",
        "lines_to_next_cell": 2
      },
      "source": [
        "# place a number here between 0 to 191 to pick an image from the test set\n",
        "integer_slider = 105\n",
        "\n",
        "ds = test_dataset.unbatch()\n",
        "ds = ds.batch(200)\n",
        "images = []\n",
        "\n",
        "y_true_segments = []\n",
        "for image, annotation in ds.take(2):\n",
        "  y_true_segments = annotation\n",
        "  images = image\n",
        "  \n",
        "  \n",
        "iou, dice_score = class_wise_metrics(np.argmax(y_true_segments[integer_slider], axis=2), results[integer_slider])  \n",
        "show_annotation_and_prediction(image[integer_slider], annotation[integer_slider], results[integer_slider], iou, dice_score)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EiG9K4t6X9iZ"
      },
      "source": [
        "### Compute IOU Score and Dice Score of your model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2706boF0CNNS",
        "lines_to_next_cell": 2
      },
      "source": [
        "cls_wise_iou, cls_wise_dice_score = class_wise_metrics(np.argmax(y_true_segments, axis=3), results)\n",
        "\n",
        "average_iou = 0.0\n",
        "for idx, (iou, dice_score) in enumerate(zip(cls_wise_iou[:-1], cls_wise_dice_score[:-1])):\n",
        "  print(\"Digit {}: IOU: {} Dice Score: {}\".format(idx, iou, dice_score)) \n",
        "  average_iou += iou\n",
        "\n",
        "grade = average_iou * 10\n",
        "\n",
        "print(\"\\nGrade is \" + str(grade))\n",
        "\n",
        "PASSING_GRADE = 60\n",
        "if (grade>PASSING_GRADE):\n",
        "  print(\"You passed!\")\n",
        "else:\n",
        "  print(\"You failed. Please check your model and re-train\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vvw0HLY2kV3w"
      },
      "source": [
        "## Save the Model\n",
        "\n",
        "Once you're satisfied with the results, you will need to save your model so you can upload it to the grader in the Coursera classroom. After running the cell below, please look for `student_model.h5` in the File Explorer on the left and download it. Then go back to the Coursera classroom and upload it to the Lab item that points to the autograder of Week 3."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ULCfGHEKkaO0"
      },
      "source": [
        "model.save(\"model.h5\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QbVfJ2aLONqw"
      },
      "source": [
        "# You can also use this cell as a shortcut for downloading your model\n",
        "from google.colab import files\n",
        "files.download(\"model.h5\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmyvoAynkgVw"
      },
      "source": [
        "**Congratulations on completing this assignment on image segmentation!**"
      ]
    }
  ]
}