Fine_Tune_BERT_for_Text_Classification_with_TensorFlow.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Fine-Tune-BERT-for-Text-Classification-with-TensorFlow.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/abyanjan/Fine-Tune-BERT-for-Text-Classification/blob/master/Fine_Tune_BERT_for_Text_Classification_with_TensorFlow.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zGCJYkQj_Uu2"
      },
      "source": [
        "<h2 align=center> Fine-Tune BERT for Text Classification with TensorFlow</h2>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4y2m1S6e12il"
      },
      "source": [
        "<div align=\"center\">\n",
        "    <img width=\"512px\" src='https://drive.google.com/uc?id=1fnJTeJs5HUpz7nix-F9E6EZdgUflqyEu' />\n",
        "    <p style=\"text-align: center;color:gray\">Figure 1: BERT Classification Model</p>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eYYYWqWr_WCC"
      },
      "source": [
        "In this [project](https://www.coursera.org/projects/fine-tune-bert-tensorflow/), you will learn how to fine-tune a BERT model for text classification using TensorFlow and TF-Hub."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yQG5PCO_WFx"
      },
      "source": [
        "The pretrained BERT model used in this project is [available](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2) on [TensorFlow Hub](https://tfhub.dev/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7pKNS21u_WJo"
      },
      "source": [
        "### Learning Objectives"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_3NHSMXv_WMv"
      },
      "source": [
        "By the time you complete this project, you will be able to:\n",
        "\n",
        "- Build TensorFlow Input Pipelines for Text Data with the [`tf.data`](https://www.tensorflow.org/api_docs/python/tf/data) API\n",
        "- Tokenize and Preprocess Text for BERT\n",
        "- Fine-tune BERT for text classification with TensorFlow 2 and [TF Hub](https://tfhub.dev)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o6BEe-3-AVRQ"
      },
      "source": [
        "### Prerequisites"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sc9f-8rLAVUS"
      },
      "source": [
        "In order to be successful with this project, it is assumed you are:\n",
        "\n",
        "- Competent in the Python programming language\n",
        "- Familiar with deep learning for Natural Language Processing (NLP)\n",
        "- Familiar with TensorFlow, and its Keras API"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MYXXV5n3Ab-4"
      },
      "source": [
        "### Contents"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XhK-SYGyAjxe"
      },
      "source": [
        "This project/notebook consists of several Tasks.\n",
        "\n",
        "- **[Task 1]()**: Introduction to the Project.\n",
        "- **[Task 2]()**: Setup your TensorFlow and Colab Runtime\n",
        "- **[Task 3]()**: Download and Import the Quora Insincere Questions Dataset\n",
        "- **[Task 4]()**: Create tf.data.Datasets for Training and Evaluation\n",
        "- **[Task 5]()**: Download a Pre-trained BERT Model from TensorFlow Hub\n",
        "- **[Task 6]()**: Create a TensorFlow Input Pipeline with `tf.data`\n",
        "- **[Task 7]()**: Add a Classification Head to the BERT `hub.KerasLayer`\n",
        "- **[Task 8]()**: Fine-Tune BERT for Text Classification\n",
        "- **[Task 9]()**: Evaluate the BERT Text Classification Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IaArqXjRAcBa"
      },
      "source": [
        "## Task 2: Setup your TensorFlow and Colab Runtime."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GDDhjzZ5A4Q_"
      },
      "source": [
        "You will only be able to use the Colab Notebook after you save it to your Google Drive folder. Click on the File menu and select “Save a copy in Drive…\n",
        "\n",
        "![Copy to Drive](https://drive.google.com/uc?id=1CH3eDmuJL8WR0AP1r3UE6sOPuqq8_Wl7)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mpe6GhLuBJWB"
      },
      "source": [
        "### Check GPU Availability\n",
        "\n",
        "Check if your Colab notebook is configured to use Graphical Processing Units (GPUs). If zero GPUs are available, check if the Colab notebook is configured to use GPUs (Menu > Runtime > Change Runtime Type).\n",
        "\n",
        "![Hardware Accelerator Settings](https://drive.google.com/uc?id=1qrihuuMtvzXJHiRV8M7RngbxFYipXKQx)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8V9c8vzSL3aj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ee76fe5c-8063-44ed-e9d3-b28e9d178bd0"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tue Jul  6 10:26:42 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 465.27       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   66C    P8    11W /  70W |      0MiB / 15109MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Obch3rAuBVf0"
      },
      "source": [
        "### Install TensorFlow and TensorFlow Model Garden"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bUQEY3dFB0jX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5b9da0a5-ed77-4ff8-89d5-70cbc24c8f2a"
      },
      "source": [
        "import tensorflow as tf\n",
        "print(tf.version.VERSION)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2.5.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aU3YLZ1TYKUt"
      },
      "source": [
        "#!pip install -q tensorflow==2.3.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AFRTC-zwUy6D",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "714c6748-b7f5-4fa7-f0fe-657bd2fa6b60"
      },
      "source": [
        "!git clone --depth 1 -b v2.3.0 https://github.com/tensorflow/models.git"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'models'...\n",
            "remote: Enumerating objects: 2650, done.\u001b[K\n",
            "remote: Counting objects: 100% (2650/2650), done.\u001b[K\n",
            "remote: Compressing objects: 100% (2311/2311), done.\u001b[K\n",
            "remote: Total 2650 (delta 506), reused 1386 (delta 306), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (2650/2650), 34.02 MiB | 33.85 MiB/s, done.\n",
            "Resolving deltas: 100% (506/506), done.\n",
            "Note: checking out '400d68abbccda2f0f6609e3a924467718b144233'.\n",
            "\n",
            "You are in 'detached HEAD' state. You can look around, make experimental\n",
            "changes and commit them, and you can discard any commits you make in this\n",
            "state without impacting any branches by performing another checkout.\n",
            "\n",
            "If you want to create a new branch to retain commits you create, you may\n",
            "do so (now or later) by using -b with the checkout command again. Example:\n",
            "\n",
            "  git checkout -b <new-branch-name>\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3H2G0571zLLs"
      },
      "source": [
        "# install requirements to use tensorflow/models repository\n",
        "!pip install -Uqr models/official/requirements.txt\n",
        "# you may have to restart the runtime afterwards"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0xH3jfnZMS5D"
      },
      "source": [
        "!pip install tensorflow-text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GVjksk4yCXur"
      },
      "source": [
        "## Restart the Runtime\n",
        "\n",
        "**Note** \n",
        "After installing the required Python packages, you'll need to restart the Colab Runtime Engine (Menu > Runtime > Restart runtime...)\n",
        "\n",
        "![Restart of the Colab Runtime Engine](https://drive.google.com/uc?id=1xnjAy2sxIymKhydkqb0RKzgVK9rh3teH)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IMsEoT3Fg4Wg"
      },
      "source": [
        "## Task 3: Download and Import the Quora Insincere Questions Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GmqEylyFYTdP"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "import sys\n",
        "sys.path.append('models')\n",
        "from official.nlp.data import classifier_data_lib\n",
        "from official.nlp.bert import tokenization\n",
        "from official.nlp import optimization\n",
        "import tensorflow_text as text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZuX1lB8pPJ-W",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "be9769f2-4385-4560-a92c-787dc767da35"
      },
      "source": [
        "print(\"TF Version: \", tf.__version__)\n",
        "print(\"Eager mode: \", tf.executing_eagerly())\n",
        "print(\"Hub version: \", hub.__version__)\n",
        "print(\"GPU is\", \"available\" if tf.config.experimental.list_physical_devices(\"GPU\") else \"NOT AVAILABLE\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TF Version:  2.5.0\n",
            "Eager mode:  True\n",
            "Hub version:  0.12.0\n",
            "GPU is available\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QtbwpWgyEZg7"
      },
      "source": [
        "A downloadable copy of the [Quora Insincere Questions Classification data](https://www.kaggle.com/c/quora-insincere-questions-classification/data) can be found [https://archive.org/download/fine-tune-bert-tensorflow-train.csv/train.csv.zip](https://archive.org/download/fine-tune-bert-tensorflow-train.csv/train.csv.zip). Decompress and read the data into a pandas DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0nI-9itVwCCQ"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yeHE98KiMvDd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dc4fc64f-8703-43a3-ca6b-6a7ad7bce54f"
      },
      "source": [
        "df = pd.read_csv('https://archive.org/download/fine-tune-bert-tensorflow-train.csv/train.csv.zip', \n",
        "                 compression = 'zip', low_memory = False)\n",
        "df.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1306122, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "leRFRWJMocVa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 669
        },
        "outputId": "d9f1ed00-287b-4bb7-dd18-ad21811cbee0"
      },
      "source": [
        "df.tail(20)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>qid</th>\n",
              "      <th>question_text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1306102</th>\n",
              "      <td>ffff3778790af9baae76</td>\n",
              "      <td>What steps can I take to live a normal life if...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306103</th>\n",
              "      <td>ffff3f0a2449ffe4b9ff</td>\n",
              "      <td>Isn't Trump right after all? Why should the US...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306104</th>\n",
              "      <td>ffff41393389d4206066</td>\n",
              "      <td>Is 33 too late for a career in creative advert...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306105</th>\n",
              "      <td>ffff42493fc203cd9532</td>\n",
              "      <td>What is difference between the filteration wor...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306106</th>\n",
              "      <td>ffff48dd47bee89fff79</td>\n",
              "      <td>If the universe \"popped\" into existence from n...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306107</th>\n",
              "      <td>ffff5fd051a032f32a39</td>\n",
              "      <td>How does a shared service technology team meas...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306108</th>\n",
              "      <td>ffff6d528040d3888b93</td>\n",
              "      <td>How is DSATM civil engineering?</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306109</th>\n",
              "      <td>ffff8776cd30cdc8d7f8</td>\n",
              "      <td>Do you know any problem that depends solely on...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306110</th>\n",
              "      <td>ffff94d427ade3716cd1</td>\n",
              "      <td>What are some comic ideas for you Tube videos ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306111</th>\n",
              "      <td>ffffa382c58368071dc9</td>\n",
              "      <td>If you had $10 million of Bitcoin, could you s...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306112</th>\n",
              "      <td>ffffa5b0fa76431c063f</td>\n",
              "      <td>Are you ashamed of being an Indian?</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306113</th>\n",
              "      <td>ffffae5dbda3dc9e9771</td>\n",
              "      <td>What are the methods to determine fossil ages ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306114</th>\n",
              "      <td>ffffba7c4888798571c1</td>\n",
              "      <td>What is your story today?</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306115</th>\n",
              "      <td>ffffc0c7158658a06fd9</td>\n",
              "      <td>How do I consume 150 gms protein daily both ve...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306116</th>\n",
              "      <td>ffffc404da586ac5a08f</td>\n",
              "      <td>What are the good career options for a msc che...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306117</th>\n",
              "      <td>ffffcc4e2331aaf1e41e</td>\n",
              "      <td>What other technical skills do you need as a c...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306118</th>\n",
              "      <td>ffffd431801e5a2f4861</td>\n",
              "      <td>Does MS in ECE have good job prospects in USA ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306119</th>\n",
              "      <td>ffffd48fb36b63db010c</td>\n",
              "      <td>Is foam insulation toxic?</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306120</th>\n",
              "      <td>ffffec519fa37cf60c78</td>\n",
              "      <td>How can one start a research project based on ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1306121</th>\n",
              "      <td>ffffed09fedb5088744a</td>\n",
              "      <td>Who wins in a battle between a Wolverine and a...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                          qid  ... target\n",
              "1306102  ffff3778790af9baae76  ...      0\n",
              "1306103  ffff3f0a2449ffe4b9ff  ...      1\n",
              "1306104  ffff41393389d4206066  ...      0\n",
              "1306105  ffff42493fc203cd9532  ...      0\n",
              "1306106  ffff48dd47bee89fff79  ...      0\n",
              "1306107  ffff5fd051a032f32a39  ...      0\n",
              "1306108  ffff6d528040d3888b93  ...      0\n",
              "1306109  ffff8776cd30cdc8d7f8  ...      0\n",
              "1306110  ffff94d427ade3716cd1  ...      0\n",
              "1306111  ffffa382c58368071dc9  ...      0\n",
              "1306112  ffffa5b0fa76431c063f  ...      1\n",
              "1306113  ffffae5dbda3dc9e9771  ...      0\n",
              "1306114  ffffba7c4888798571c1  ...      0\n",
              "1306115  ffffc0c7158658a06fd9  ...      0\n",
              "1306116  ffffc404da586ac5a08f  ...      0\n",
              "1306117  ffffcc4e2331aaf1e41e  ...      0\n",
              "1306118  ffffd431801e5a2f4861  ...      0\n",
              "1306119  ffffd48fb36b63db010c  ...      0\n",
              "1306120  ffffec519fa37cf60c78  ...      0\n",
              "1306121  ffffed09fedb5088744a  ...      0\n",
              "\n",
              "[20 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 298
        },
        "id": "kqvN1X8Y-yAN",
        "outputId": "68c44699-9d5d-4ca7-8d72-aa0d9c9e4256"
      },
      "source": [
        "# target label distribution\n",
        "df.target.plot(kind='hist', title=\"Target Distribution\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<AxesSubplot:title={'center':'Target Distribution'}, ylabel='Frequency'>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEICAYAAABS0fM3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWm0lEQVR4nO3dfbRddX3n8fdHHgTkqTVxqgkYrEGJQEe8AtVlwZFaHhah1SklQju4kHRZcVXxAWo7yMJOq7WVGVosRIsUK4/ODBNKkKrF0rFGCUURgmgKCCG0iciTgiD6nT/ODj1zuTf3hHv3Ody736+17sp++J29v7/c5H7u/v322SdVhSSpu54z6gIkSaNlEEhSxxkEktRxBoEkdZxBIEkdZxBIUscZBNIzlOT4JH83g8e7NcmhzfKZSf5mBo/9gSSfnKnjaW4xCDRUSX7Q9/XTJI/1rR8/pBoOTbJ+ijYXJnkiySPN1y1J/jjJbpvbVNVnquqNA5zvwiR/OFW7qnpFVX1poE5s+XxP619V/VFVvW26x9bcZBBoqKpq581fwN3A0X3bPjPIMZJs226VT/mTqtoFmA+8FTgY+HKS583kSYbYH2lCBoGeFZIcmOQrSR5Mcl+Sv0iyfd/+SvKOJN8BvtNse3/TdkOStzVtXtrse26SP01yd5J/S3Jekh2bH+LXAC/quxJ50ZZqq6ofVdUNwFLg+fRCgSQnJvm/zXKSnJ1kY5KHk3wzyb5JlgPHA+9vznVV0/6uJKcluRn4YZJtm22H9Z16hySXNVck/5zkF8b9fby0b/3CJH84Wf/GDzUlWdoMRT2Y5EtJ9unbd1eS9ya5OclDTQ07bM33U7OLQaBni58A7wbmAb8IvAH4nXFtfhU4CFiS5HDgVOAw4KXAoePafhjYG/iPzf4FwBlV9UPgCGBD35XIhkEKrKpHgM8Dr5tg9xuBX2rOuRtwLHB/Va0APkPv6mLnqjq67zXLgKOA3avqyQmOeQxwBfCzwMXAlUm2m6LGKfuXZG/gEuBd9K52VgFX9QdvU//hwF7A/sCJWzqvZrdZGQRJLmh+87plwPbHJlnb/AZ0cdv1aetV1Y1Vtbqqnqyqu4DzgUPGNfvjqvp+VT1G7wfVp6rq1qp6FDhzc6MkAZYD727aPwL8EXDcDJS6gd4P5vF+DOwCvBxIVd1WVfdNcaxzquqepj8TubGqPltVPwY+BuxAb3hqun4DuLqqPt8c+0+BHYHXjKttQ1V9H7iKXqBqjpqVQQBcSO+3lSklWQz8HvDaqnoFvd+C9CyTZO8kf5vkX5M8TO8H97xxze7pW37RuPX+5fnATsCNzdDHg8Dnmu3TtQD4/viNVfX3wF8A5wIbk6xIsusUx7pn0P1V9VNgPb1+T9eLgO+OO/Y99Pq22b/2LT8K7DwD59Wz1KwMgqq6nnH/GZP8fJLPJbkxyT8meXmz62Tg3Kp6oHntxiGXq8H8JfAtYHFV7Qp8AMi4Nv2Pyr0PWNi3vkff8veAx4BXVNXuzdduzQT1+OMMLMnO9Iai/nGi/VV1TlW9ClhCb4jofVOcb6o6nupTkufQ6+/mYZ5H6YXdZj+3FcfdALy479hpznXvFK/THDUrg2ASK4B3Nv8R3wt8vNm+N7B3ki8nWd2MLevZZxfgYeAHTYi/fYr2lwNvTbJPkp2A/7p5R/Mb7ieAs5O8ACDJgiS/0jT5N+D5/beCbkkz8fwq4ErgAeBTE7R5dZKDmjH8HwI/An7ad76XDHKucV6V5E3NXUXvAh4HVjf7vg68Jck2zb/p/mG0qfp3OXBUkjc09b6nOfY/PYMaNQfMiSBoflN7DXBFkq/TG19+YbN7W2AxvcnEZcAnkuw+/Co1hfcCbwEeofdD/LItNa6qa4BzgOuAdfz7D8jHmz9P27y9GWr6AvCy5rXfojdZekczdDTZcMv7kzwC3A9cBNwIvKaZkB1v16buB+gNu9wPfLTZ91f0JrgfTHLllvo1zv+hN57/APCbwJuaMX2A3wWOBh6kd1fSU8edqn9VdTtwAvDn9K6ejqZ3G+8TW1Gb5pDM1g+mSbII+Nuq2rcZi729ql44QbvzgK9W1aea9S8Cpze3A2qOaG5/vAV47iR34EiaxJy4Iqiqh4E7k/w6PHVP9+Z7rq+kubUwyTx6Q0V3jKBMzbAkv9YM2/wM8BHgKkNA2nqzMgiSXAJ8BXhZkvVJTqJ3eXxSkm8At9K7BxvgWuD+JGvpDSO8r6ruH0XdmnG/DWwE/oXe+xCmmleQNIFZOzQkSZoZs/KKQJI0c2bdw67mzZtXixYtGnUZkjSr3Hjjjd+rqgnfVDnrgmDRokWsWbNm1GVI0qyS5LuT7XNoSJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjpu1r2zeDoWnX71yM5914ePGtm5JWlLWrsiSHJBko1Jbplk//FJbk7yzST/1Pf5AZKkIWpzaOhCYEufD3wncEhV7Qd8iN5nDkuShqy1oaGqur75OMnJ9vd/UPZqYGFbtUiSJvdsmSw+Cbhmsp1JlidZk2TNpk2bhliWJM19Iw+CJK+nFwSnTdamqlZU1VhVjc2fP+HjtCVJz9BI7xpKsj/wSeAIP0dYkkZjZFcESfYE/hfwm1X17VHVIUld19oVQZJLgEOBeUnWAx8EtgOoqvOAM4DnAx9PAvBkVY21VY8kaWJt3jW0bIr9bwPe1tb5JUmDGflksSRptAwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp41oLgiQXJNmY5JZJ9ifJOUnWJbk5yQFt1SJJmlybVwQXAodvYf8RwOLmaznwly3WIkmaRGtBUFXXA9/fQpNjgIuqZzWwe5IXtlWPJGlio5wjWADc07e+vtn2NEmWJ1mTZM2mTZuGUpwkdcWsmCyuqhVVNVZVY/Pnzx91OZI0p4wyCO4F9uhbX9hskyQN0SiDYCXwW83dQwcDD1XVfSOsR5I6adu2DpzkEuBQYF6S9cAHge0Aquo8YBVwJLAOeBR4a1u1SJIm11oQVNWyKfYX8I62zi9JGsysmCyWJLXHIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknquFaDIMnhSW5Psi7J6RPs3zPJdUluSnJzkiPbrEeS9HStBUGSbYBzgSOAJcCyJEvGNfsD4PKqeiVwHPDxtuqRJE2szSuCA4F1VXVHVT0BXAocM65NAbs2y7sBG1qsR5I0gW1bPPYC4J6+9fXAQePanAn8XZJ3As8DDmuxHknSBEY9WbwMuLCqFgJHAp9O8rSakixPsibJmk2bNg29SEmay9oMgnuBPfrWFzbb+p0EXA5QVV8BdgDmjT9QVa2oqrGqGps/f35L5UpSN7UZBDcAi5PslWR7epPBK8e1uRt4A0CSfegFgb/yS9IQDRQESfbb2gNX1ZPAKcC1wG307g66NclZSZY2zd4DnJzkG8AlwIlVVVt7LknSMzfoZPHHkzwXuBD4TFU9NMiLqmoVsGrctjP6ltcCrx2wBklSCwa6Iqiq1wHH0xvzvzHJxUl+udXKJElDMfAcQVV9h94bwE4DDgHOSfKtJG9qqzhJUvsGnSPYP8nZ9Mb6/xNwdFXt0yyf3WJ9kqSWDTpH8OfAJ4EPVNVjmzdW1YYkf9BKZZKkoRg0CI4CHquqnwA0b/raoaoerapPt1adJKl1g84RfAHYsW99p2abJGmWGzQIdqiqH2xeaZZ3aqckSdIwDRoEP0xywOaVJK8CHttCe0nSLDHoHMG7gCuSbAAC/BzwG20VJUkanoGCoKpuSPJy4GXNptur6sftlSVJGpat+TyCVwOLmtcckISquqiVqiRJQzNQECT5NPDzwNeBnzSbCzAIJGmWG/SKYAxY4pNBJWnuGfSuoVvoTRBLkuaYQa8I5gFrk3wNeHzzxqpaOvlLJEmzwaBBcGabRUiSRmfQ20f/IcmLgcVV9YUkOwHbtFuaJGkYBn0M9cnAZ4Hzm00LgCtbqkmSNESDTha/g95HSj4MT31IzQvaKkqSNDyDBsHjVfXE5pUk29J7H4EkaZYbNAj+IckHgB2bzyq+AriqvbIkScMyaBCcDmwCvgn8NrCK3ucXS5JmuUHvGvop8InmS5I0hwz6rKE7mWBOoKpeMuMVSZKGamueNbTZDsCvAz878+VIkoZtoDmCqrq/7+veqvrv9D7QfouSHJ7k9iTrkpw+SZtjk6xNcmuSi7eufEnSdA06NHRA3+pz6F0hbPG1SbYBzgV+GVgP3JBkZVWt7WuzGPg94LVV9UAS35sgSUM26NDQn/UtPwncBRw7xWsOBNZV1R0ASS4FjgHW9rU5GTi3qh4AqKqNA9YjSZohg9419PpncOwFwD196+uBg8a12RsgyZfpPbvozKr63PgDJVkOLAfYc889n0EpkqTJDDo0dOqW9lfVx6Zx/sXAocBC4Pok+1XVg+OOvwJYATA2NuY7miVpBm3NXUOvBlY260cDXwO+s4XX3Avs0be+sNnWbz3w1ar6MXBnkm/TC4YbBqxLkjRNgwbBQuCAqnoEIMmZwNVVdcIWXnMDsDjJXvQC4DjgLePaXAksAz6VZB69oaI7Bq5ekjRtgz5i4j8AT/StP9Fsm1RVPQmcAlwL3AZcXlW3JjkryeZPNrsWuD/JWuA64H1Vdf/WdECSND2DXhFcBHwtyf9u1n8V+OupXlRVq+g9l6h/2xl9ywWc2nxJkkZg0LuG/luSa4DXNZveWlU3tVeWJGlYBh0aAtgJeLiq/gewvhn7lyTNcoN+VOUHgdPovQsYYDvgb9oqSpI0PINeEfwasBT4IUBVbQB2aasoSdLwDBoETzQTuwWQ5HntlSRJGqZBg+DyJOcDuyc5GfgCfkiNJM0JU941lCTAZcDLgYeBlwFnVNXnW65NkjQEUwZBVVWSVVW1H+APf0maYwYdGvrnJK9utRJJ0kgM+s7ig4ATktxF786h0LtY2L+twiRJwzHVp4ztWVV3A78ypHokSUM21RXBlfSeOvrdJP+zqt48hJokSUM01RxB+pZf0mYhkqTRmCoIapJlSdIcMdXQ0C8keZjelcGOzTL8+2Txrq1WJ0lq3RaDoKq2GVYhkqTR2JrHUEuS5iCDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknquFaDIMnhSW5Psi7J6Vto9+YklWSszXokSU/XWhAk2QY4FzgCWAIsS7Jkgna7AL8LfLWtWiRJk2vziuBAYF1V3VFVTwCXAsdM0O5DwEeAH7VYiyRpEm0GwQLgnr719c22pyQ5ANijqq7e0oGSLE+yJsmaTZs2zXylktRhI5ssTvIc4GPAe6ZqW1Urqmqsqsbmz5/ffnGS1CFtBsG9wB596wubbZvtAuwLfCnJXcDBwEonjCVpuNoMghuAxUn2SrI9cBywcvPOqnqoquZV1aKqWgSsBpZW1ZoWa5IkjdNaEFTVk8ApwLXAbcDlVXVrkrOSLG3rvJKkrTPVR1VOS1WtAlaN23bGJG0PbbMWSdLEfGexJHWcQSBJHWcQSFLHGQSS1HEGgSR1nEEgSR1nEEhSxxkEktRxBoEkdZxBIEkdZxBIUscZBJLUcQaBJHWcQSBJHWcQSFLHGQSS1HEGgSR1nEEgSR1nEEhSxxkEktRxBoEkdZxBIEkdZxBIUscZBJLUcQaBJHVcq0GQ5PAktydZl+T0CfafmmRtkpuTfDHJi9usR5L0dK0FQZJtgHOBI4AlwLIkS8Y1uwkYq6r9gc8Cf9JWPZKkibV5RXAgsK6q7qiqJ4BLgWP6G1TVdVX1aLO6GljYYj2SpAm0GQQLgHv61tc32yZzEnDNRDuSLE+yJsmaTZs2zWCJkqRnxWRxkhOAMeCjE+2vqhVVNVZVY/Pnzx9ucZI0x23b4rHvBfboW1/YbPv/JDkM+H3gkKp6vMV6JEkTaPOK4AZgcZK9kmwPHAes7G+Q5JXA+cDSqtrYYi2SpEm0FgRV9SRwCnAtcBtweVXdmuSsJEubZh8FdgauSPL1JCsnOZwkqSVtDg1RVauAVeO2ndG3fFib55ckTe1ZMVksSRodg0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOq7Vx1BL0lyz6PSrR3buuz58VCvH9YpAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOazUIkhye5PYk65KcPsH+5ya5rNn/1SSL2qxHkvR0rQVBkm2Ac4EjgCXAsiRLxjU7CXigql4KnA18pK16JEkTa/OK4EBgXVXdUVVPAJcCx4xrcwzw183yZ4E3JEmLNUmSxmnz6aMLgHv61tcDB03WpqqeTPIQ8Hzge/2NkiwHljerP0hy+zOsad74Yw9LRnetM7I+j5B97obO9TkfmVafXzzZjlnxGOqqWgGsmO5xkqypqrEZKGnWsM/dYJ+7oa0+tzk0dC+wR9/6wmbbhG2SbAvsBtzfYk2SpHHaDIIbgMVJ9kqyPXAcsHJcm5XAf2mW/zPw91VVLdYkSRqntaGhZsz/FOBaYBvggqq6NclZwJqqWgn8FfDpJOuA79MLizZNe3hpFrLP3WCfu6GVPsdfwCWp23xnsSR1nEEgSR03J4Ogi4+2GKDPpyZZm+TmJF9MMuk9xbPFVH3ua/fmJJVk1t9qOEifkxzbfK9vTXLxsGucaQP8294zyXVJbmr+fR85ijpnSpILkmxMcssk+5PknObv4+YkB0z7pFU1p77oTUz/C/ASYHvgG8CScW1+BzivWT4OuGzUdQ+hz68HdmqW396FPjftdgGuB1YDY6Ouewjf58XATcDPNOsvGHXdQ+jzCuDtzfIS4K5R1z3NPv8ScABwyyT7jwSuAQIcDHx1uueci1cEXXy0xZR9rqrrqurRZnU1vfd1zGaDfJ8BPkTvGVY/GmZxLRmkzycD51bVAwBVtXHINc60QfpcwK7N8m7AhiHWN+Oq6np6d1FO5hjgoupZDeye5IXTOedcDIKJHm2xYLI2VfUksPnRFrPVIH3udxK93yhmsyn73Fwy71FVVw+zsBYN8n3eG9g7yZeTrE5y+NCqa8cgfT4TOCHJemAV8M7hlDYyW/v/fUqz4hETmjlJTgDGgENGXUubkjwH+Bhw4ohLGbZt6Q0PHUrvqu/6JPtV1YOjLKply4ALq+rPkvwivfcm7VtVPx11YbPFXLwi6OKjLQbpM0kOA34fWFpVjw+ptrZM1eddgH2BLyW5i95Y6spZPmE8yPd5PbCyqn5cVXcC36YXDLPVIH0+CbgcoKq+AuxA74F0c9VA/9+3xlwMgi4+2mLKPid5JXA+vRCY7ePGMEWfq+qhqppXVYuqahG9eZGlVbVmNOXOiEH+bV9J72qAJPPoDRXdMcQaZ9ogfb4beANAkn3oBcGmoVY5XCuB32ruHjoYeKiq7pvOAefc0FA9Ox9t0aoB+/xRYGfgimZe/O6qWjqyoqdpwD7PKQP2+VrgjUnWAj8B3ldVs/Zqd8A+vwf4RJJ305s4PnE2/2KX5BJ6YT6vmff4ILAdQFWdR28e5EhgHfAo8NZpn3MW/31JkmbAXBwakiRtBYNAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI77f8uT0Z8EREO9AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELjswHcFHfp3"
      },
      "source": [
        "## Task 4: Create tf.data.Datasets for Training and Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0eWAyiSCRQ4J"
      },
      "source": [
        "We will be working with only a small portion of data since the data is huge and finetuning the bert model and that big size would considerabely take large amount of time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fScULIGPwuWk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7c53694e-182a-4cb0-c95b-e363af5de352"
      },
      "source": [
        "train_df, remaining = train_test_split(df, random_state = 42, train_size = 0.0075, stratify = df['target'])\n",
        "val_df, _ = train_test_split(remaining, random_state=42, train_size = 0.00075, stratify = remaining['target'])\n",
        "\n",
        "print(train_df.shape, val_df.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(9795, 3) (972, 3)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qQYMGT5_qLPX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ae862048-d860-442e-da42-adb2edcdc99c"
      },
      "source": [
        "with tf.device('/cpu:0'):\n",
        "  train_data = tf.data.Dataset.from_tensor_slices((train_df['question_text'].values, train_df['target'].values))\n",
        "  val_data = tf.data.Dataset.from_tensor_slices((val_df['question_text'].values, val_df['target'].values))\n",
        "\n",
        "  for text, label in train_data.take(1):\n",
        "    print(text)\n",
        "    print(label)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor(b'Why are unhealthy relationships so desirable?', shape=(), dtype=string)\n",
            "tf.Tensor(0, shape=(), dtype=int64)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e2-ReN88Hvy_"
      },
      "source": [
        "## Task 5: Download a Pre-trained BERT Model from TensorFlow Hub"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EMb5M86b4-BU"
      },
      "source": [
        "\"\"\"\n",
        "Each line of the dataset is composed of the review text and its label\n",
        "- Data preprocessing consists of transforming text to BERT input features:\n",
        "input_word_ids, input_mask, segment_ids\n",
        "- In the process, tokenizing the text is done with the provided BERT model tokenizer\n",
        "\"\"\"\n",
        "\n",
        "# Label categories\n",
        "label_list = [0, 1]\n",
        "\n",
        "# maximum length of (token) input sequences\n",
        "max_seq_length = 128  # default max length\n",
        "\n",
        "train_batch_size = 32\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oBv-mBaOI8gs"
      },
      "source": [
        "# Get BERT layer and tokenizer\n",
        "bert_layer = 'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4'\n",
        "tokenizer = 'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e12SGZxZUXTC",
        "outputId": "a58da6ec-086b-470e-89e5-1329da9a7a47"
      },
      "source": [
        "# see an example of tokenization\n",
        "bert_tokenizer = hub.load(tokenizer)\n",
        "\n",
        "bert_tokenizer.tokenize(['hi, how are you doing?'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.RaggedTensor [[[7632], [1010], [2129], [2024], [2017], [2725], [1029]]]>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wEUezMK-zkkI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d49b1af4-d224-4f2f-a7d6-c651c70de7be"
      },
      "source": [
        "# create keras layers to preprocess the text and encode to fit to the bert model\n",
        "\n",
        "text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name = 'inputs')\n",
        "\n",
        "# preprocessing\n",
        "preprocessor = hub.KerasLayer(tokenizer)\n",
        "encoder_inputs = preprocessor(text_input)\n",
        "\n",
        "preprocess_model = tf.keras.Model(text_input, encoder_inputs)\n",
        "\n",
        "preprocess_model(tf.constant(['hi, how are you doing?']))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'input_mask': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
              " array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
              "       dtype=int32)>,\n",
              " 'input_type_ids': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
              " array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
              "       dtype=int32)>,\n",
              " 'input_word_ids': <tf.Tensor: shape=(1, 128), dtype=int32, numpy=\n",
              " array([[ 101, 7632, 1010, 2129, 2024, 2017, 2725, 1029,  102,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "            0,    0,    0,    0,    0,    0,    0]], dtype=int32)>}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qtAPWCqqO_t7"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dhdO6MjTbtn1"
      },
      "source": [
        "## Task 8: Create a TensorFlow Input Pipeline with `tf.data`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LHRdiO3dnPNr"
      },
      "source": [
        "with tf.device('/cpu:0'):\n",
        "  # train\n",
        "  train_data = train_data.shuffle(1000).batch(32, drop_remainder = True).prefetch(tf.data.experimental.AUTOTUNE)\n",
        "\n",
        "  # valid\n",
        "  val_data = val_data.batch(32, drop_remainder = True).prefetch(tf.data.experimental.AUTOTUNE)\n",
        "  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B0Z2cy9GHQ8x",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1c284635-2b30-43ec-f50d-401a21a5c70a"
      },
      "source": [
        "# train data spec\n",
        "train_data.element_spec"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorSpec(shape=(32,), dtype=tf.string, name=None),\n",
              " TensorSpec(shape=(32,), dtype=tf.int64, name=None))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DGAH-ycYOmao",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f8b721fe-133b-4a00-b719-40c157f93640"
      },
      "source": [
        "# valid data spec\n",
        "val_data.element_spec"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorSpec(shape=(32,), dtype=tf.string, name=None),\n",
              " TensorSpec(shape=(32,), dtype=tf.int64, name=None))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZxe-7yhPyQe"
      },
      "source": [
        "## Task 9: Add a Classification Head to the BERT Layer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9THH5V0Dw2HO"
      },
      "source": [
        "<div align=\"center\">\n",
        "    <img width=\"512px\" src='https://drive.google.com/uc?id=1fnJTeJs5HUpz7nix-F9E6EZdgUflqyEu' />\n",
        "    <p style=\"text-align: center;color:gray\">Figure 3: BERT Layer</p>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G9il4gtlADcp"
      },
      "source": [
        "# Building the model\n",
        "def create_model():\n",
        "  \n",
        "  text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name = 'inputs')\n",
        "\n",
        "  # preprocessing\n",
        "  preprocessor = hub.KerasLayer(tokenizer)\n",
        "  encoder_inputs = preprocessor(text_input)\n",
        "\n",
        "  # passing the encoded inputs to bert model\n",
        "  encoder = hub.KerasLayer(bert_layer, trainable=True, name = 'BERT_Encoder')\n",
        "  outputs = encoder(encoder_inputs)\n",
        "\n",
        "  # get the pooled outputs\n",
        "  pooled_output = outputs['pooled_output']\n",
        "\n",
        "  # add output layer\n",
        "  batch_norm = tf.keras.layers.BatchNormalization()(pooled_output)\n",
        "  drop = tf.keras.layers.Dropout(0.4)(batch_norm)\n",
        "  fc = tf.keras.layers.Dense(128, activation='relu')(drop)\n",
        "  final_output = tf.keras.layers.Dense(1, activation='sigmoid', name='classifier')(fc)\n",
        "    \n",
        "  return tf.keras.Model(text_input, final_output)\n",
        "  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S6maM-vr7YaJ"
      },
      "source": [
        "## Task 10: Fine-Tune BERT for Text Classification"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ptCtiiONsBgo"
      },
      "source": [
        "model = create_model()\n",
        "model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate=2e-5), \n",
        "              loss = tf.keras.losses.BinaryCrossentropy(),\n",
        "              metrics = [tf.keras.metrics.BinaryAccuracy()])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6GJaFnkbMtPL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "77b17b1c-e6d2-40e9-c506-bde82fc8a837"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "inputs (InputLayer)             [(None,)]            0                                            \n",
            "__________________________________________________________________________________________________\n",
            "keras_layer_1 (KerasLayer)      {'input_word_ids': ( 0           inputs[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "BERT_Encoder (KerasLayer)       {'pooled_output': (N 109482241   keras_layer_1[0][0]              \n",
            "                                                                 keras_layer_1[0][1]              \n",
            "                                                                 keras_layer_1[0][2]              \n",
            "__________________________________________________________________________________________________\n",
            "batch_normalization (BatchNorma (None, 768)          3072        BERT_Encoder[0][13]              \n",
            "__________________________________________________________________________________________________\n",
            "dropout (Dropout)               (None, 768)          0           batch_normalization[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "dense (Dense)                   (None, 128)          98432       dropout[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "classifier (Dense)              (None, 1)            129         dense[0][0]                      \n",
            "==================================================================================================\n",
            "Total params: 109,583,874\n",
            "Trainable params: 109,582,337\n",
            "Non-trainable params: 1,537\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OcREcgPUHr9O",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff039ce2-984f-4ed9-a50e-b0541063fe0c"
      },
      "source": [
        "# Train model\n",
        "epochs = 3\n",
        "history = model.fit(train_data, \n",
        "                    validation_data=val_data,\n",
        "                    epochs=epochs, \n",
        "                    verbose=1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "306/306 [==============================] - 284s 877ms/step - loss: 0.5032 - binary_accuracy: 0.7270 - val_loss: 0.2437 - val_binary_accuracy: 0.9458\n",
            "Epoch 2/3\n",
            "306/306 [==============================] - 268s 875ms/step - loss: 0.1545 - binary_accuracy: 0.9449 - val_loss: 0.1435 - val_binary_accuracy: 0.9552\n",
            "Epoch 3/3\n",
            "306/306 [==============================] - 267s 873ms/step - loss: 0.1140 - binary_accuracy: 0.9590 - val_loss: 0.1483 - val_binary_accuracy: 0.9615\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kNZl1lx_cA5Y"
      },
      "source": [
        "## Task 11: Evaluate the BERT Text Classification Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dCjgrUYH_IsE"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_graphs(history, metric):\n",
        "  plt.plot(history.history[metric])\n",
        "  plt.plot(history.history['val_'+metric], '')\n",
        "  plt.xlabel(\"Epochs\")\n",
        "  plt.ylabel(metric)\n",
        "  plt.legend([metric, 'val_'+metric])\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v6lrFRra_KmA",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "9f9d1420-050a-4e21-eb14-7f588c370e37"
      },
      "source": [
        "plot_graphs(history=history, metric='loss')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAz+UlEQVR4nO3deXgV1fnA8e+bhWxs2WQLWcAoW2QLuFXUuuEGorWAG6CVKq61tWK11VJtrbbWLqg/VEDrghSX0opaW7VWEUxAIAZklSUBISQQlgCB5P39MRNyCQncQCZzk7yf55mHO2fm3Hkzudw3Z86ZM6KqGGOMMTWF+R2AMcaY0GQJwhhjTK0sQRhjjKmVJQhjjDG1sgRhjDGmVhF+B9BQkpKSND093e8wjDGmSVmwYMFWVU2ubVuzSRDp6enk5ub6HYYxxjQpIrKurm12ickYY0ytLEEYY4yplSUIY4wxtWo2fRDGmJZp//79FBQUsHfvXr9DCWnR0dGkpKQQGRkZdB1LEMaYJq2goIA2bdqQnp6OiPgdTkhSVYqLiykoKCAjIyPoenaJyRjTpO3du5fExERLDkcgIiQmJta7lWUJwhjT5FlyOLpjOUeeJggRGSoiy0VklYhMrGX7WBEpEpFF7vKDgG1jRGSlu4zxKsb9FZX8Zs4yCrfv8eoQxhjTJHmWIEQkHJgMXAz0AkaLSK9adn1dVfu5y/Nu3QTgIeBUYDDwkIjEexFn4bY9vDp/PTdOy2Hn3v1eHMIY08y1bt3a7xA84WULYjCwSlXXqGo5MAMYHmTdi4APVLVEVbcBHwBDvQgyPSmOZ64byOqiXUx4ZSH7Kyq9OIwxxjQ5XiaILsCGgPUCt6ymq0RkiYjMEpGu9akrIuNFJFdEcouKio450O9kJvHIFX3438qtPDQ7H3vKnjHmWKgq9957L3369CErK4vXX38dgE2bNjFkyBD69etHnz59+N///kdFRQVjx449uO8f/vAHn6M/nN/DXP8BvKaq+0Tkh8CLwHeDrayqU4ApANnZ2cf1rT5qcCrrSsp45uPVpCfGMn5I9+N5O2OMD375j3yWbtzRoO/Zq3NbHrq8d1D7vvnmmyxatIjFixezdetWBg0axJAhQ3j11Ve56KKLeOCBB6ioqKCsrIxFixZRWFjIV199BcD27dsbNO6G4GULohDoGrCe4pYdpKrFqrrPXX0eGBhsXS/ce+HJXJrViV/P+Zp38zZ5fThjTDPz6aefMnr0aMLDw+nQoQNnn302OTk5DBo0iGnTpvHwww+Tl5dHmzZt6NatG2vWrOGOO+7gvffeo23btn6HfxgvWxA5QKaIZOB8uY8CrgncQUQ6qWrVN/EwYJn7+n3g1wEd0xcC93sYKwBhYcLvv9+XjaV7uPv1RXRsF03/VE/6xo0xHgj2L/3GNmTIED755BPeeecdxo4dyz333MMNN9zA4sWLef/993n22WeZOXMmU6dO9TvUQ3jWglDVA8DtOF/2y4CZqpovIpNEZJi7250iki8ii4E7gbFu3RLgVzhJJgeY5JZ5LjoynOduyOaEtlHc/FIuG0rKGuOwxphm4KyzzuL111+noqKCoqIiPvnkEwYPHsy6devo0KEDN998Mz/4wQ9YuHAhW7dupbKykquuuopHHnmEhQsX+h3+YTztg1DVOcCcGmW/CHh9P3W0DFR1KuBLOk1qHcW0sYO48um53Dg9h1m3nkG7mODnLzHGtEwjRozg888/p2/fvogIjz/+OB07duTFF1/kiSeeIDIyktatW/PSSy9RWFjIuHHjqKx0Rk7+5je/8Tn6w0lzGbGTnZ2tDf3AoLmrt3LDC19warcEpo8bTGS43XhuTKhZtmwZPXv29DuMJqG2cyUiC1Q1u7b97RvvCM7onsRvrszis1XFPPjWVzb81RjTovg9zDXkXZ3dlfUlZfz5w1WkJcUy4ZwT/Q7JGGMahSWIINxzwUmsKy7j8feWk5oQy2WndPY7JGOM8ZxdYgqCiPD4904hOy2ee2YuZsG6bX6HZIwxnrMEEaToyHCm3JBNp3bR3PxSLuuLbfirMaZ5swRRDwlxrZg2dhAVlcrY6V9QWmazvxpjmi9LEPXULbk1U64fyIaSMn74ci7lB2z2V2NM82QJ4hic2i2Rx793CvPWlHD/m3k2/NUYE7QjPTti7dq19OnTpxGjOTIbxXSMRvRPYV1xGU/9eyXpibHccV6m3yEZY0yDsgRxHO46L5N1xWX8/oMVpCbGMrxfbY+7MMY0mncnwrd5DfueHbPg4sfq3Dxx4kS6du3KbbfdBsDDDz9MREQEH330Edu2bWP//v088sgjDB8e7PPSHHv37uXWW28lNzeXiIgInnzySc4991zy8/MZN24c5eXlVFZW8sYbb9C5c2e+//3vU1BQQEVFBT//+c8ZOXLkcf3YYAniuIgIj12VReH2Pdz7tyV0bh/DoPQEv8MyxjSikSNHcvfddx9MEDNnzuT999/nzjvvpG3btmzdupXTTjuNYcOGISJBv+/kyZMREfLy8vj666+58MILWbFiBc8++yx33XUX1157LeXl5VRUVDBnzhw6d+7MO++8A0BpaWmD/GyWII5TVEQ4U64fyJVPz2X8S7m8NeFM0pPi/A7LmJbpCH/pe6V///5s2bKFjRs3UlRURHx8PB07duRHP/oRn3zyCWFhYRQWFrJ582Y6duwY9Pt++umn3HHHHQD06NGDtLQ0VqxYwemnn86jjz5KQUEBV155JZmZmWRlZfHjH/+Y++67j8suu4yzzjqrQX4266RuAO1jWzF17CAAxk3PYdvucp8jMsY0pquvvppZs2bx+uuvM3LkSF555RWKiopYsGABixYtokOHDuzdu7dBjnXNNdcwe/ZsYmJiuOSSS/jwww856aSTWLhwIVlZWTz44INMmjSpQY5lCaKBpCfF8dwN2RRu28MPX17AvgMVfodkjGkkI0eOZMaMGcyaNYurr76a0tJSTjjhBCIjI/noo49Yt25dvd/zrLPO4pVXXgFgxYoVrF+/npNPPpk1a9bQrVs37rzzToYPH86SJUvYuHEjsbGxXHfdddx7770N9mwJSxANKDs9gSeuPoUvvilh4hs2/NWYlqJ3797s3LmTLl260KlTJ6699lpyc3PJysripZdeokePHvV+zwkTJlBZWUlWVhYjR45k+vTpREVFMXPmTPr06UO/fv346quvuOGGG8jLy2Pw4MH069ePX/7ylzz44IMN8nN5+jwIERkK/BEIB55X1VovEIrIVcAsYJCq5opIOs5T6Ja7u8xT1VuOdCwvngdxrP7y4Up+968V3H1+Jneff5Lf4RjTrNnzIIJX3+dBeNZJLSLhwGTgAqAAyBGR2aq6tMZ+bYC7gPk13mK1qvbzKj4v3Xbuiax175FIS4xlRP8Uv0Myxph683IU02BglaquARCRGcBwYGmN/X4F/Ba418NYGpWI8OsRWRRu28NPZy2hc7sYTu2W6HdYxpgQkZeXx/XXX39IWVRUFPPn1/w72V9e9kF0ATYErBe4ZQeJyACgq6q+U0v9DBH5UkT+KyK1jtkSkfEikisiuUVFRQ0WeENoFRHGs9cNJDUhlvF/XcDqol1+h2RMs9XU+vuysrJYtGjRIYvXyeFYzpFvndQiEgY8Cfy4ls2bgFRV7Q/cA7wqIm1r7qSqU1Q1W1Wzk5OTvQ34GLSLjWTa2MFEhAk3Ts+hxIa/GtPgoqOjKS4ubnJJojGpKsXFxURHR9ernpeXmAqBrgHrKW5ZlTZAH+Bj9+7CjsBsERmmqrnAPgBVXSAiq4GTgNDoha6H1MRYptyQzejn5jH+pVxe/sGpREeG+x2WMc1GSkoKBQUFhNpVhFATHR1NSkr9+kO9TBA5QKaIZOAkhlHANVUbVbUUSKpaF5GPgZ+4o5iSgRJVrRCRbkAmsMbDWD01MC2eJ7/fl9tf/ZJ7Zy3hjyP7ERYW/C33xpi6RUZGkpGR4XcYzZJnCUJVD4jI7cD7OMNcp6pqvohMAnJVdfYRqg8BJonIfqASuEVVS7yKtTFcdkpn1pc4z7VOT4zlxxee7HdIxhhzRJ7OxaSqc4A5Ncp+Uce+5wS8fgN4w8vY/HDr2d1Zt7WMP3+4itSEWK7O7nr0SsYY4xObrK8RiQiPjOhDwfYy7n8zjy7tYzjjxKSjVzTGGB/YVBuNLDI8jKevHUhGUhy3vLyAVVt2+h2SMcbUyhKED9rFRDJ17CBaRYQxbnoOW3ft8zskY4w5jCUIn3RNiOX5MYPYsmMfN7+Uy979NvurMSa0WILwUb+u7XlqZD8WbdjOj2cuprLSbvQxxoQOSxA+uzirE/df3IN38jbxxL+WH72CMcY0EhvFFAJuPqsba4vLeObj1aQlxDJqcKrfIRljjCWIUCAiTBrWm4Jte3jg7a/oEh/DWZmhN7eUMaZlsUtMISIiPIzJ1/Qn84TWTHh5ISs22/BXY4y/LEGEkDbRkbwwdhDRrcIZNy2HLTsb5iHnxhhzLCxBhJgu7WOYOmYQJbvLufnFXPaU2/BXY4w/LEGEoKyUdvxxVD+WFJZy9+tf2vBXY4wvLEGEqAt7d+TBS3vxfv5mHnvva7/DMca0QDaKKYTdeGY664p3M+WTNaQmxHLdaWl+h2SMaUEsQYQwEeEXl/ViQ0kZD83OJyU+hnNOPsHvsIwxLYRdYgpxEeFh/PmaAZzUoQ23v/olX3+7w++QjDEthKcJQkSGishyEVklIhOPsN9VIqIikh1Qdr9bb7mIXORlnKGudVQEU8dmExcVzo3Tctiyw4a/GmO851mCEJFwYDJwMdALGC0ivWrZrw1wFzA/oKwXzjOsewNDgafd92uxOrWL4YUxg9i+Zz83vZhLWfkBv0MyxjRzXrYgBgOrVHWNqpYDM4Dhtez3K+C3QOCfxcOBGaq6T1W/AVa579ei9enSjj+P7k/+xlLumrGIChv+aozxkJcJoguwIWC9wC07SEQGAF1V9Z361m2pzuvZgYcu780HSzfz6znL/A7HGNOM+TaKSUTCgCeBscfxHuOB8QCpqS1nBtQxZ6Sztng3L3z6DWmJsdxwerrfIRljmiEvWxCFQNeA9RS3rEoboA/wsYisBU4DZrsd1UerC4CqTlHVbFXNTk5uWbOfPnhpL87veQIPz87no6+3+B2OMaYZ8jJB5ACZIpIhIq1wOp1nV21U1VJVTVLVdFVNB+YBw1Q1191vlIhEiUgGkAl84WGsTU54mPDHUf3p1bktt7+6kPyNpX6HZIxpZjxLEKp6ALgdeB9YBsxU1XwRmSQiw45SNx+YCSwF3gNuU1Wbta6GuKgIXhgziLYxkdw0PZdvS234qzGm4Yhq8xgJk52drbm5uX6H4Ytlm3bwvWfmkpYYx99uOZ24KLtB3hgTHBFZoKrZtW2zO6mbgZ6d2jL52gEs37yTO1770oa/GmMahCWIZuKck0/g4WG9+fDrLfzqn0v9DscY0wzYtYhm5PrT0lhfvJvn/ucMfx13ZobfIRljmjBLEM3MxIt7sq64jEn/XEpKfCwX9Orgd0jGmCbKLjE1M+FhwlOj+pHVpR13vvYleQU2/NUYc2wsQTRDsa0ieH5MNglxrbjpxRw2bt/jd0jGmCbIEkQzdUKbaKaOHcSe8gpunJ7Dzr37/Q7JGNPEWIJoxk7u2IbJ1w5g5ZZd3P7qlxyoqPQ7JGNME2IJopkbclIyj1zRh/+uKOKh2fk0lxsjjTHes1FMLcDowamsLd7N//13DRlJcfzgrG5+h2SMaQIsQbQQ913Ug/XFZTw6Zxkp8bEM7dPR75CMMSHOLjG1EGFhwh9G9qNvSnvufv1LFm/Y7ndIxpgQZwmiBYmODOe5G7JJah3FTS/mUrCtzO+QjDEhzBJEC5PcJoppYwex74Az/HWHDX81xtTBEkQLlNmhDc9eN5A1Rbu57ZWF7Lfhr8aYWliCaKHOPDGJX4/I4n8rt/Lzt7+y4a/GmMPYKKYW7PuDurKuZDeTP1pNelIct5zd3e+QjDEhxNMWhIgMFZHlIrJKRCbWsv0WEckTkUUi8qmI9HLL00Vkj1u+SESe9TLOluzHF5zMZad04rF3v2ZO3ia/wzHGhBDPWhAiEg5MBi4ACoAcEZmtqoFPs3lVVZ919x8GPAkMdbetVtV+XsVnHGFhwu+u7sum0r386PVFdGwXzYDUeL/DMsaEAC9bEIOBVaq6RlXLgRnA8MAdVHVHwGocYBfCfRAdGc6U6wfSoW00N7+Yy4YSG/5qjPE2QXQBNgSsF7hlhxCR20RkNfA4cGfApgwR+VJE/isiZ9V2ABEZLyK5IpJbVFTUkLG3OImto5g2bhAHKpVx03Mo3WPDX41p6XwfxaSqk1W1O3Af8KBbvAlIVdX+wD3AqyLStpa6U1Q1W1Wzk5OTGy/oZqp7cmuevW4g64p3c+vLCyg/YMNfjWnJvEwQhUDXgPUUt6wuM4ArAFR1n6oWu68XAKuBk7wJ0wQ6vXsij115CnNXF/Pg23k2/NWYFszLBJEDZIpIhoi0AkYBswN3EJHMgNVLgZVuebLbyY2IdAMygTUexmoCXDUwhTvPy2RmbgFPf7za73CMMT7xbBSTqh4QkduB94FwYKqq5ovIJCBXVWcDt4vI+cB+YBswxq0+BJgkIvuBSuAWVS3xKlZzuB+dn8m64t088f5yUhNiubxvZ79DMsY0MmkulxCys7M1NzfX7zCalX0HKrju+fksLijltZtPZWBagt8hGWMamIgsUNXs2rb53kltQldURDhTrs+mS/sYbn5pAeuKd/sdkjGmEVmCMEcUH9eKqWMHUanO8NftZeV+h2SMaSSWIMxRZSTFMeX6bApK9vDDv9rwV2NaCksQJiiDMxJ44upTmP9NCRPfXGLDX41pAWw2VxO04f26sK64jCc/WEF6Yhx3npd59ErGmCbLEoSplzu+eyJri3fz5AcrSE2I5Yr+h82eYoxpJixBmHoRER678hQ2bt/DT2ctoXP7GAZn2PBXY5oj64Mw9dYqIoxnrxtISkIM4/+ayzdbbfirMc2RJQhzTNrHtmLa2EGEiTBu2hds223DX41pboJKECJyl4i0FccLIrJQRC70OjgT2tIS43juhoFsLN3L+L/msu9Ahd8hGWMaULAtiBvdh/tcCMQD1wOPeRaVaTIGpiXwu6v7krN2Gz+dZcNfjWlOgu2kFvffS4C/upPuyZEqmJZjWN/ObCgp44n3l5OWGMc9F9jM7MY0B8EmiAUi8i8gA7hfRNrgzLJqDAATzunO2q27+dN/VpKWEMtVA1P8DskYc5yCTRA3Af2ANapaJiIJwDjPojJNjojw6IgsCrfvYeKbzvDX07sn+h2WMeY4BNsHcTqwXFW3i8h1OI8GLfUuLNMUtYoI45nrBpKWGMcP/5rLqi27/A7JGHMcgk0QzwBlItIX+DHOI0Bf8iwq02S1i4lk2thBRIaHceP0HIp37fM7JGPMMQo2QRxQZ3jKcOAvqjoZaHO0SiIyVESWi8gqEZlYy/ZbRCRPRBaJyKci0itg2/1uveUiclGwP5DxX9eEWJ4bk83mHXu5+aVc9u634a/GNEXBJoidInI/zvDWd0QkDIg8UgX3mdKTgYuBXsDowATgelVVs1S1H/A48KRbtxfOM6x7A0OBp6ueUW2ahgGp8fxhZD8Wrt/OT/62mMpKG/5qTFMTbIIYCezDuR/iWyAFeOIodQYDq1R1jaqWAzNwWiAHufdWVIkDqr5FhgMzVHWfqn4DrHLfzzQhl2R1YuLFPfjnkk38/oPlfodjjKmnoBKEmxReAdqJyGXAXlU9Wh9EF2BDwHqBW3YIEblNRFbjtCDurGfd8SKSKyK5RUVFwfwoppH9cEg3Rg/uyuSPVjMzZ8PRKxhjQkawU218H/gCuBr4PjBfRL7XEAGo6mRV7Q7chzM6qj51p6hqtqpmJycnN0Q4poGJCJOG9+GszCR+9lYen63a6ndIxpggBXuJ6QFgkKqOUdUbcC73/PwodQqBrgHrKW5ZXWYAVxxjXRPCIsPDmHztALolx3HLywtYuXmn3yEZY4IQbIIIU9UtAevFQdTNATJFJENEWuF0Os8O3EFEAh9Jdimw0n09GxglIlEikgFk4rRgTBPVNjqSqWMHERURzrjpORTttOGvxoS6YBPEeyLyvoiMFZGxwDvAnCNVUNUDwO3A+8AyYKY7h9MkERnm7na7iOSLyCLgHmCMWzcfmAksBd4DblNVGyvZxKXEx/LCmGy27tpnw1+NaQIk2Nk3ReQq4Ex39X+q+pZnUR2D7Oxszc3N9TsME4T3vvqWW19ZwMV9OvKX0QMIC7N5H43xi4gsUNXs2rYF/cAgVX1DVe9xl5BKDqZpGdqnIw9c0pM5ed/y+Ps2/NWYUHXEyfpEZCfV9yYcsglQVW3rSVSm2bvpOxmsLd7Ns/9dTVpiLKMHp/odkjGmhiMmCFU96nQaxhwLEeHhy3uzoWQPD779FSnxMZyVaUOVjQkl9kxq45uI8DD+ck1/Mk9ozYSXF7L8Wxv+akwosQRhfNXGHf4a0yqcG6fnsGXnXr9DMsa4LEEY33VuH8PUsYMo2V3OD17MZU+5DX81JhRYgjAhoU+Xdvx5dH/yCku5+/UvqbDZX43xnSUIEzLO79WBX1zWi/fzN/PYu8v8DseYFi/YZ1Ib0yjGnZnBuuIynvvfN6QmxnH9aWl+h2RMi2UJwoScn1/Wiw0lZTz0d2f467knn+B3SMa0SHaJyYSc8DDhT6P707NTW25/ZSFLN+44eiVjTIOzBGFCUlxUBC+MGUSb6EhuejGHzTts+Ksxjc0ShAlZHdtFM3XsIHbs2c+N03PYve+A3yEZ06JYgjAhrVfntvzlmgEs27SDu2bY8FdjGpMlCBPyzu1xAr8c1pt/L9vCI+8s9TscY1oMG8VkmoTrT09nbXEZL3z6DWkJsYw9M8PvkIxp9ixBmCbjZ5f0ZF1xGZP+uZSuCbGc17OD3yEZ06x5eolJRIaKyHIRWSUiE2vZfo+ILBWRJSLyHxFJC9hWISKL3GV2zbqm5XGGv/ajV+e23PHal3xVWOp3SMY0a54lCBEJByYDFwO9gNEi0qvGbl8C2ap6CjALeDxg2x5V7ecuwzAGiG3lDH9tH+MMf91UusfvkIxptrxsQQwGVqnqGlUtB2YAwwN3UNWPVLXMXZ0HpHgYj2kmOrSN5oWxg9i9r4Ibp+eyy4a/GuMJLxNEF2BDwHqBW1aXm4B3A9ajRSRXROaJyBW1VRCR8e4+uUVFRccdsGk6enZqy+RrB7Bi807ueHUhByoq/Q7JmGYnJIa5ish1QDbwREBxmqpmA9cAT4lI95r1VHWKqmaranZysj2usqU5+6RkJg3vzUfLi/jlP5aiavdIGNOQvEwQhUDXgPUUt+wQInI+8AAwTFX3VZWraqH77xrgY6C/J1EeKIfnL4CPfgNbvvbkEMY7156axvgh3fjrvHVM/Wyt3+EY06x4mSBygEwRyRCRVsAo4JDRSCLSH/g/nOSwJaA8XkSi3NdJwJmAN3dI7d4C4a3gv7+Fp0+FyafBx49B0XJPDmca3sShPRjauyOPvLOUf+V/63c4xjQb4mWzXEQuAZ4CwoGpqvqoiEwCclV1toj8G8gCNrlV1qvqMBE5AydxVOIksadU9YUjHSs7O1tzc3OPPdid38Kyf0D+W7BuLqCQ3BN6j4DeV0Dyycf+3sZze8orGDXlc1Zs3sXMH55OVko7v0MypkkQkQXu5fzDtzWX67bHnSAC7fwWls6GpW9XJ4sTejnJotcVkHxSwxzHNKgtO/cyYvJcyisqefu2M+nSPsbvkIwJeZYgjseOTbBsNuS/Des/x0kWvatbFkmZDX9Mc8xWbN7JVU/PpUt8DH+75XTaREf6HZIxIc0SREPZsbG6ZbH+c6esQx8nUfQaAUknent8E5RPV25l7LQvOOPEJKaOySYiPCQG6xkTkixBeKEqWeS/BRvmOWUdspxk0XsEJB42Ktc0ohlfrGfim3lcc2oqj17RBxHxOyRjQpIlCK+VFrqXod6CDfOdso5ZTn+FJQvf/Pa9r3nm49U8cElPbh7Sze9wjAlJliAaU2lBdcui4AunrOMp7mWoKyxZNKLKSuWO175kzlebeObaAQzt08nvkIwJOZYg/FJaAEv/7iaLHKesU1+3ZXEFJNhftV7bu7+C0c/NY9mmHcwYfzr9urb3OyRjQooliFCwfUN1sih04+zUt3robII9AMcrW3ftY8TTn7GnvIK3JpxJ14RYv0MyJmRYggg129cHJIsFTlmnftVDZ+PTfQyueVq1ZSdXPj2XDm2jmXXrGbSLseGvxoAliNC2bZ2TLJa+XZ0sOvevblnEpx2ptqmHuau3csMLX3Bat0SmjRtEpA1/NcYSRJOxba3bsngbNi50yjoPcJPFcEsWDeBvuRu4d9YSRmZ35bGrsmz4q2nxLEE0RdvWOoli6duw8UunrMvA6mTRPtXH4Jq23/9rOX/+cBX3De3BrefYqDLTslmCaOpKvqnus9i0yCnrkl09dLZ91yNUNjWpKnfNWMTsxRuZfM0ALj3Fhr+alssSRHNSsiYgWSx2ylIGOYmi13BLFkHau7+C656fz5LCUl67+TQGpsX7HZIxvrAE0VyVrHEuQ+W/Bd8uccpSBrsti+HQzh7xfSQlu8sZ8fRn7Np7gLcmnElqog1/NS2PJYiWoHi101+R/xZ8m+eUpQyu7rNod6THgbdca4p2MeLpuSS1bsWbt55Ju1gb/mpaFksQLU3xaidR5L8Nm91k0fXU6mTRtrOv4YWa+WuKue6F+WSnJfDijYNpFWHDX03LcaQE4en/BBEZKiLLRWSViEysZfs9IrJURJaIyH9EJC1g2xgRWekuY7yMs9lJ7A5DfgK3fgq3L4DvPgjlZfDeRHiyJ7xwEcx71pmR1nBqt0Qe/94pfL6mmJ+9lUdz+aPJmOPlWQtCRMKBFcAFQAHOM6pHq+rSgH3OBearapmI3Aqco6ojRSQByAWyAQUWAANVdVtdx7MWRBC2rqweOrv5K6cs9XSnZdFzGLRt2aN5nvr3Cp7690p+cuFJ3P5dexCUaRn8akEMBlap6hpVLQdmAMMDd1DVj1S1zF2dB1T1ql4EfKCqJW5S+AAY6mGsLUNSJpx9L9z6GdyWA+c+AHt3wLs/dVoWUy+G+VOcR662QHedl8mI/l343b9W8PdFhX6HY4zvvEwQXYANAesFblldbgLerU9dERkvIrkikltUVHSc4bYwySfB2T+FCXPdZPEz2Lsd3r0Xft8Dpl3S4pKFiPDYVVkMzkjg3r8tIWdtid8hGeOrkOiNE5HrcC4nPVGfeqo6RVWzVTU7OTnZm+BagoPJ4nO47Qs4534oKwlIFpfCF8/Bzs1+R+q5qIhwplw/kJT4GMa/lMvarbv9DskY33iZIAqBwLu2UtyyQ4jI+cADwDBV3VefusYDySfDOffBbfNgwnw4ZyKUbYU5P4HfnwzTL3OSxa4tfkfqmfaxrZg6dhAA46bnsG13uc8RGeMPLzupI3A6qc/D+XLPAa5R1fyAffoDs4ChqroyoDwBp2N6gFu0EKeTus42v3VSe2zLsuqb8rYuBwmDtDOdm/J6DoPWJ/gdYYPLWVvCtc/Np19qe/5602CiIsL9DsmYBufbfRAicgnwFBAOTFXVR0VkEpCrqrNF5N9AFrDJrbJeVYe5dW8EfuaWP6qq0450LEsQjUTVSRZVN+VtXRGQLNzRUK2bz+W+vy8q5K4ZixjRvwtPfr+vzf5qmh27Uc54oypZ5L/lLMUrnWSR/p3qZBGX5HeUx+3P/1nJ7z9Ywd3nZ3L3+Sf5HY4xDcoShPGeKmxZGpAsVrnJ4iw3WVzeZJOFqvKTvy3hjYUF/GFkX0b0tzmuTPNhCcI0LlXYnF+dLEpWg4RDhpsselwOcYl+R1kv5QcquWHqfBas28bLN53Kqd2aVvzG1MUShPGPqnPX9sFkscZNFkOcDu4mlCxKy/Yz4pnPKN5VzlsTzqBbcmu/QzLmuFmCMKFB1ZlpNv8tp5O7Kll0O9t5nkXPyyE2we8oj2h9cRkjnv6MNtERvDnhTBLiWvkdkjHHxRKECT2qzjMsqobObvumOln0HgE9LgvZZLFg3TZGPzePU7q04+UfnEp0pA1/NU2XJQgT2lSdp+NVDZ3dthbCIiCjKllcGnLJ4p9LNnL7q18yrG9n/jiqnw1/NU2WJQjTdKg6z92uallsX+cki27nVCeLmNB4POjTH6/i8feWc+d3T+SeC0/2OxxjjoklCNM0qcLGL6tbFtvXQ1hkQLK4xNdkoapMfCOP13M38Lur+/K9gTb81TQ9liBM01eVLKqelFfqJovu5zrJ4uRLIKZ9o4e1v6KSsdO+4ItvSnjxxsGc0b1p3uthWi5LEKZ5UYWNCwOSxQY3WXzXTRYXN2qyKN2zn+89M5fNO/by5oQzOfEEG/5qmg5LEKb5UoXChZD/Jiz9e3WyOPE8Z+hsj0sgup3nYWwocYa/xrQK5+0JZ5LYOsrzYxrTECxBmJZBFQoXVLcsdhRAeCvofp5zU97JF3uaLBZt2M7I//uc3p3b8urNp9nwV9MkWIIwLU9lpZMslr5dS7JwL0NFt23ww76bt4kJry7kkqxO/HlUf8LCbPirCW2WIEzLVlkJhblOolj6NuwohPAo5zJU7xFw0tAGTRZTPlnNr+d8zYRzuvPToT0a7H2N8cKREkREYwdjTKMLC4Oug53lwkegIKe6ZbF8jpsszndbFkMhqs1xHe7ms7qxtriMpz9eTVpiLCMHpTbIj2FMY7MWhGm5Kiuh4IvqlsXOTU6yyLzAbVlcdMzJ4kBFJTe+mMvcVVuZPm4w38m04a8mNPn5RLmhwB9xnij3vKo+VmP7EJwnzp0CjFLVWQHbKoA8d/Xgk+bqYgnCHJeDycLt4N71LUREV7csThoKUfUbvrpz736ufvZzCrft4dErs8g8oTWpCbHERVnD3YQOXxKEiITjPJP6AqAA55nUo1V1acA+6UBb4CfA7BoJYpeqBv0/0hKEaTCVlbBhvjvr7N+rk0VVyyLzoqCTReH2PVz19Fy+3bH3YFlS61akJsQ6S2IcaQmxpCbGkpYQS3KbKJvXyTQqvxLE6cDDqnqRu34/gKr+ppZ9pwP/tARhQk5lJWyYF5AsNkNEjJssrggqWewpr2DVll2sLyljXclu1heXOa+Ly9hUuofKgP+C0ZFhbvKIIzUhlrTEqkQSS0p8DFERNnTWNCy/Oqm7ABsC1guAU+tRP1pEcoEDwGOq+nYDxmZMcMLCIO0MZxn6GKx3k8Wy2c4SEQMnXejclHfSRdAq7rC3iGkVTlZKO7JSDr8Ho/xAJYXb97CueDcb3KSxvsRZ5q7eSll5xcF9RaBT22hS3aSRlhhH1wSn5ZGWGEu7mEhrfTRVlZVQecBd9kNlRcB61VKzLGA9Mha6DGjwsEL5YmiaqhaKSDfgQxHJU9XVgTuIyHhgPEBqqo0UMR4LC4f0M53l4t/C+s/dlsVsp3UREeMkid5XQOaFtSaLmlpFhJGRFEdG0uH7qipbd5WzvmT3wRbH+pIy1heX8dHyIop2Fhyyf5voiOoWR40WSKd20USEhzXUmfDGIV+SdX0pBrO9ji/Siv1H/6Kttay2L+wg1g8eL4j9Oc4rOV2y4eb/NMivIZCXCaIQ6BqwnuKWBUVVC91/14jIx0B/YHWNfaYAU8C5xHSc8RoTvLBwSP+Os1z8OKyb64yEWvp359/IWCdJ9B7hJovY4N63shLU+XKRygMkhx8gObGCgfHhkBEDlZFQ2RoqD7CnfB+bt+1m8/ZdbN6+i6Id29hauoviDWWsXlbG6soKwqkkggpahVWSHBtOcpyzJMVGkBATRkJMGPHRYbSSYL6ca1uvx5dg4HpFLV+6x/slebwk3Jla/uBSx3p4ZC3bIiAy5gj1a6sTxHp4ZHB1PJohwMsEkQNkikgGTmIYBVwTTEURiQfKVHWfiCQBZwKPexapMccjLBwyznKWix+HdZ85I6GWza5OFm07B/clqpVBHzYGSHeXw9T2P7vcXbbV/Z6VhFEZ8EUZFh6BhEciwX6pRURXv671i/Ro6zWX2rYf7Ys0yDphNerY5bnDeJYgVPWAiNwOvI8zzHWqquaLyCQgV1Vni8gg4C0gHrhcRH6pqr2BnsD/iUglEIbTB7G0jkMZEzrCwiFjiLNUJYtl/4Cy4tq/rIL6Em2gL9Lw6tc79ikbSsvZsL2cddv3sX57OWuL97K2ZO8ROs6rLl3FHOz/6JpgHefNmd0oZ4w5RFXHudPfsfuQjvP1JWW1dpx3PWTElTt0NyGW9rHWcR7qbKoNY0zQDu04Tz5k29E7zvcdsn+b6IiDneXOiKu4ptVx3sJZgjDGBE1ESG4TRXKbKAamJRy2vaz8ABtK9rjJY/fBVsfXm3bywdLN7K+ovmIRESZ0iY859H4Pd/RVamIsre2Oc9/Zb8AY02BiW0Vwcsc2nNzx8DmsKiqVb3fsrfWej38u2cT2sv2H7J8Y1+rgHeZVl66qkskJdsd5o7AEYYxpFOFhQpf2MXRpHwPdD99eumd/9V3mJdVJJGftNmYv3nhYx3nX+MBLV7HuDYRx1nHegCxBGGNCQruYyKPecV5bx/nc1cWHdZx3bBt9cORVWmJA68M6zuvFEoQxJuQF13FexvqS3Yd0nP93RRFbanacR0UcnK7EuYRVfenKOs4PZQnCGNOkHdpxHn/Y9j3lFQdbGwf7P0rKWP7tTv69rO6O88M6z1tgx3nL+mmNMS1OTKvwo3acO30fh166eiev7o7zqstVXd1JE9MSY0luHdXsnkFuCcIY02IFdpyf3j3xsO2le/Yf7CwP7DhfsG4b/6jRcR4VEXDHeeKhHecp8TFERza9jnNLEMYYU4d2MZG069KOPl2O3nEeeOPg52uO3nFe1fpITYglPkQ7zi1BGGPMMfCy4zw14K5zPzvOLUEYY0wDO+aO8807+c+yLZRXVM/qW7PjvLrz3PuOc0sQxhjTyOrTcR546aqujvPTuyfyl2ta1hPljDGmxTmWjvP42FaexGIJwhhjmpAjdZw3NLtl0BhjTK0sQRhjjKmVpwlCRIaKyHIRWSUiE2vZPkREForIARH5Xo1tY0RkpbuM8TJOY4wxh/MsQYhIODAZuBjoBYwWkV41dlsPjAVerVE3AXgIOBUYDDwkIoePFTPGGOMZL1sQg4FVqrpGVcuBGcDwwB1Uda2qLgEqa9S9CPhAVUtUdRvwATDUw1iNMcbU4GWC6AJsCFgvcMsarK6IjBeRXBHJLSoqOuZAjTHGHK5Jd1Kr6hRVzVbV7OTk5KNXMMYYEzQvE0Qh0DVgPcUt87quMcaYBiCqevS9juWNRSKAFcB5OF/uOcA1qppfy77TgX+q6ix3PQFYAFTdO74QGKiqJUc4XhGw7jhCTgK2Hkd9r1hc9WNx1Y/FVT/NMa40Va31EoxnCQJARC4BngLCgamq+qiITAJyVXW2iAwC3gLigb3At6ra2617I/Az960eVdVpngXqHC9XVbO9PMaxsLjqx+KqH4urflpaXJ5OtaGqc4A5Ncp+EfA6B+fyUW11pwJTvYzPGGNM3Zp0J7UxxhjvWIKoNsXvAOpgcdWPxVU/Flf9tKi4PO2DMMYY03RZC8IYY0ytLEEYY4ypVbNPEEHMKBslIq+72+eLSHrAtvvd8uUiclEjx3WPiCwVkSUi8h8RSQvYViEii9xldiPHNVZEigKO/4OAbZ7NwBtEXH8IiGmFiGwP2Obl+ZoqIltE5Ks6touI/MmNe4mIDAjY5uX5Olpc17rx5InIXBHpG7BtrVu+SERyGzmuc0SkNOD39YuAbUf8DHgc170BMX3lfqYS3G1enq+uIvKR+12QLyJ31bKPd58xVW22C879F6uBbkArYDHQq8Y+E4Bn3dejgNfd173c/aOADPd9whsxrnOBWPf1rVVxueu7fDxfY4G/1FI3AVjj/hvvvo5vrLhq7H8Hzn03np4v972H4NzQ+VUd2y8B3gUEOA2Y7/X5CjKuM6qOhzPj8vyAbWuBJJ/O1zk4N80e12egoeOqse/lwIeNdL46AQPc121wbj6u+X/Ss89Yc29BHHVGWXf9Rff1LOA8ERG3fIaq7lPVb4BV7vs1Slyq+pGqlrmr86jjfpEGFsz5qouXM/DWN67RwGsNdOwjUtVPgDrv8MeJ8yV1zAPai0gnPJ6x+Ghxqepc97jQeJ+vYM5XXY7ns9nQcTXm52uTqi50X+8ElnH4xKWefcaae4IIZlbYg/uo6gGgFEgMsq6XcQW6CecvhCrR4sxiO09ErmigmOoT11VuU3aWiFTNmRUS58u9FJcBfBhQ7NX5CkZdsXt5vuqr5udLgX+JyAIRGe9DPKeLyGIReVdEertlIXG+RCQW50v2jYDiRjlf4lz+7g/Mr7HJs8+Yp3dSm+MnItcB2cDZAcVpqlooIt2AD0UkT1VXN1JI/wBeU9V9IvJDnNbXdxvp2MEYBcxS1YqAMj/PV0gTkXNxEsR3Aoq/456vE4APRORr9y/sxrAQ5/e1S5ypet4GMhvp2MG4HPhMD50XzvPzJSKtcZLS3aq6oyHf+0iaewsimFlhD+4jzgSD7YDiIOt6GRcicj7wADBMVfdVlatqofvvGuBjnL8qGiUuVS0OiOV5YGCwdb2MK8AoajT/PTxfwagrdt9nLBaRU3B+h8NVtbiqPOB8bcGZK62hLq0elaruUNVd7us5QKSIJBEC58t1pM+XJ+dLRCJxksMrqvpmLbt49xnzomMlVBacFtIanEsOVR1bvWvscxuHdlLPdF/35tBO6jU0XCd1MHH1x+mUy6xRHg9Eua+TgJU0UGddkHF1Cng9Apin1R1i37jxxbuvExorLne/HjgdhtIY5yvgGOnU3el6KYd2IH7h9fkKMq5UnH61M2qUxwFtAl7PBYY2Ylwdq35/OF+0691zF9RnwKu43O3tcPop4hrrfLk/+0vAU0fYx7PPWIOd3FBdcHr4V+B82T7glk3C+ascIBr4m/uf5QugW0DdB9x6y4GLGzmufwObgUXuMtstPwPIc/+D5AE3NXJcvwHy3eN/BPQIqHujex5XAeMaMy53/WHgsRr1vD5frwGbgP0413hvAm4BbnG3C86z2Ve7x89upPN1tLieB7YFfL5y3fJu7rla7P6eH2jkuG4P+HzNIyCB1fYZaKy43H3G4gxcCazn9fn6Dk4fx5KA39UljfUZs6k2jDHG1Kq590EYY4w5RpYgjDHG1MoShDHGmFpZgjDGGFMrSxDGGGNqZQnCmKOoMRvsooacSVRE0uuaQdQYv9lUG8Yc3R5V7ed3EMY0NmtBGHOM3OcAPO4+C+ALETnRLU8XkQ+l+lkeqW55BxF5y52IbrGInOG+VbiIPOfO9/8vEYlx979Tqp8JMsOnH9O0YJYgjDm6mBqXmEYGbCtV1SzgL8BTbtmfgRdV9RTgFeBPbvmfgP+qal+cZw/ku+WZwGRV7Q1sB65yyycC/d33ucWbH82Yutmd1MYchYjsUtXWtZSvBb6rqmvcCdW+VdVEEdmKM2fVfrd8k6omiUgRkKIBEy+6Uzh/oKqZ7vp9QKSqPiIi7wG7cGY0fVvdSeyMaSzWgjDm+Ggdr+tjX8DrCqr7Bi/FmWNnAJDjzjZsTKOxBGHM8RkZ8O/n7uu5ODMDA1wL/M99/R+cx8ciIuEi0q6uNxWRMKCrqn4E3Iczk+hhrRhjvGR/kRhzdDEisihg/T1VrRrqGi8iS3BaAaPdsjuAaSJyL1AEjHPL7wKmiMhNOC2FW3FmEK1NOPCym0QE+JOqbm+gn8eYoFgfhDHHyO2DyFbVrX7HYowX7BKTMcaYWlkLwhhjTK2sBWGMMaZWliCMMcbUyhKEMcaYWlmCMMYYUytLEMYYY2r1/zG7LLJUIR7CAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "opu9neBA_98R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "5c6ac99a-3cda-49b5-d7ea-ee57cd6d0e83"
      },
      "source": [
        "plot_graphs(history,'binary_accuracy')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEGCAYAAACQO2mwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1LElEQVR4nO3deXxU5dnw8d9FyEJCgEAAEZCgyBoICWERrEuFumClFXldWIQioKL0Rau1xUep2tbnwVrr+ghUWQQUqQvaqnXBl1pQCAFkUShLFKJgIBAgIfv1/jGHMIRMMgMzOZPk+n4++eSsM9ecTO7rnPu+z7lFVTHGGGOq0sjtAIwxxoQvSxLGGGN8siRhjDHGJ0sSxhhjfLIkYYwxxqfGbgcQTImJiZqUlOR2GMYYU6esW7fugKq2rmpdvUoSSUlJZGRkuB2GMcbUKSLyja91Vt1kjDHGJ0sSxhhjfLIkYYwxxidLEsYYY3yyJGGMMcYnSxLGGGN8siRhjDHGp3p1n4QxxtRp5WVQWgglhVB6/PTfpUVQctzZptLvlhdAn1FBD8mShDHGVKW83FP4VlUgV1eQlxT63q/EKeir2r7kOJSXnHm83a+1JGGMaaBU/Sys/SmQayrInW3Kis883kaNoXETiIzx+u38RMZATPNK65pA42iIbOJsU+m3s582juG4RpFXEkFucQSHihtxoFA4WNSIc1o0ZXjwjngFSxLGmMConl74VlcNclphHUhBfmJ94ZnHKxFVFL5ehXbTNlUUzNFVFPKVC/tqCvQI/4rWwpIyDhUUk5tfzKH8EnILijmUX0zuEc+yivn8Yg4VFHMoP5/isvIqX+vH3dswvE+7Mz9OPliSMKYuU/Wc8Z51NUhNVSSFp27HGQ57LI2qLnxPnEXHJp5BwVxDgR4RGdRD7ktpWTmHj5dw6PDxikI9N7+E3PwicvNLTiaDiqRQTH5xmc/XaxEbScvYKBLiouiQEEufDs1pGRdNy7hIEmKjaBnnWXdim2YxoSnOLUkYEwplJVB0FIqPeX4XHYOSgjOsBqmh8fJMC2yk+rPh2JbVVn1UW9j7PMOOBJFgHumQUFWOFJZ6zuILisk95nVWX3F2f7Lgz80vJu+47/aEuKgIT4Hu/HRp3bRi3lPgnyz4W8ZF0bxJJI0jwqPzqSUJY+BkFUrxMSg64inUKwr4SoX9acud+WKv9WdSPVLd2XBMC4hvUkUBfTZn2FF1osAOhuPFZRzMLzq1SsfrrD73lPkSDhcUU1pedfKNimhEQlxkxVl9r3ObeRX2nrP6Vl7zLWIjiYmMqOVPHDyWJEzdpQrF+ScL6uKjlQryyoX9sZMF+SkF/BHPdHmpf+8bGQfR8RDdFKKaeqZbdDw5Hd0Uopzf0fHO8qYQGeu7QG9ABfbZKi4t53CBc4bvVZefe6y4yiqd3IJiCkuqrsdvJJAQe7LapnNiHP06RZ1yVn9i3YnpuKgIpAH9rUKeJETkKuAvQAQwV1Ufr7S+E/AS0BrIBcao6l5nXRmwydn0W1W9LtTxmhArL6t09n2sijP1yoV9VWftzjqt+p//FNLoZKHtXZA3bQPRzbyWN/XMV0zHn17YR8VBo7p7VhhuysuVvOMnz+4P5vuu0jnkJIKjRb6TeXxM44qz+rbNYuh+TjNaNT29SudEwd+8SSSNGjWcAv9MhDRJiEgE8BwwDNgLrBWR5aq61WuzJ4AFqjpfRH4M/BEY66w7rqp9Qxmj8cOJ+nVf1Ss+C/vKZ/PHoCTfv/ds1NirkHYK6pgW0LyjjzP1+FPP2r0L+8hYO0uvBapKfnHZqfX3+aef3Z+s2vFU6/io1SG6cSNPtY1zRt+pVWwVDbaRtIqLJiEukhZNoohqHB71+PVJqK8kBgA7VHUXgIi8CowAvJNET+AeZ3oF8FaIY6r/TvQp9/usvPL6SvuVFfn3vo1jTq9yadoWWnXxUZBXcaZ+4nfjaCvYXeaze+aJn9O6Z5b47J7ZuJGcUrB3Oyf+ZIEfG+V1tn8yATSJsiu2cBDqJNEe2OM1vxcYWGmbjcD1eKqkfg7Ei0grVT0IxIhIBlAKPK6qb1V+AxGZDEwGOO+884L+AWrNKfXrVTWWVlOXXlUCCEr9uq+CvOmpZ/gn9qulroYmcBXdM70K9YqqnSq6Z+bmF1MQYPfMyg22lbtnNqR6/PokHBqufwU8KyLjgZVANnDi29lJVbNF5HzgExHZpKo7vXdW1dnAbID09PQz7Qt4Zirq16trLK2iLr3y8hPL/OrKKFXUo8c79evVVLlENa20X7zVr9dR3t0zT6vD96rLz80v4lBBSUDdMxNiq++emRAXRYsw6p5pQi/USSIb6Og138FZVkFVv8NzJYGINAVGquphZ12283uXiHwKpAKnJImgOH4Ydq2ovi69qjr4kgL/Xv+U+nWngG6S4Dljr67K5ZSeMvFWv15PFRSXVlmlE4zumee2aHJa98yWXj136nr3TBN6oU4Sa4ELRaQznuRwE3CL9wYikgjkqmo58Bs8PZ0QkQSgQFWLnG2GAP8TkiiPZMPr409dVlG/7lXlUl39elVn6la/bvCc+f8tM5vN2XnWPdPUOSFNEqpaKiJ3AR/g6QL7kqpuEZFHgAxVXQ5cBvxRRBRPddNUZ/cewIsiUo5n3IvHK/WKCp6WF8Adq61+3YTEvFVZ/O6drTSNblzRQGvdM01dIaq1W40fSunp6ZqRkeF2GMZUWLk9h/Evr+GKHm15cUw/K/RNWBKRdaqaXtU6a30yJkR2/HCMqYsz6do2nqdu7GsJwtRJliSMCYHDBcXcNn8tURGNmHtrOnHR4dCR0JjA2TfXmCArKStn6uJMsg8fZ/GkQXRIiHU7JGPOmCUJY4LssXe38u8dB5l1Qx/6J7V0OxxjzopVNxkTRK98/g3zV3/DpB91ZlR6x5p3MCbMWZIwJkhW7TzAzOVbuKxbax64uofb4RgTFJYkjAmCbw7mc+eiTJIS43j65lQirCeTqScsSRhzlo4WljBxvuf+nLnj0mkWYzdimvrDkoQxZ6GsXJm2ZD1ZB/J5fnQaSYlxbodkTFBZ7yZjzsLj733Fim05PPazZAZfkOh2OMYEnV1JGHOGlmbsYc6/djPuok6MGdTJ7XCMCQlLEsacgYysXGa8uYmLuyTy0LU93Q7HmJCxJGFMgPYeKmDKwnV0SIjluVvSbAAeU6/Zt9uYAOQXlXLb/AyKy8qZMy6d5rHWk8nUb5YkjPFTebky/bUNbN9/lGdvSaNLm6Zuh2RMyFmSMMZPT364nX9u3c+Dw3tyadfWbodjTK2wJGGMH97ekM2zK3ZwU/+OTBiS5HY4xtQaSxLG1GDDnsPct+xLBiS15JERyTaGtGlQLEkYU419eYVMXpBBm/hoXhiTRlRj+5cxDYt9443x4XhxGZMWZJBfVMpfb+1Pq6bRbodkTK2zx3IYUwVV5VfLNrL5uzzmjE2n2znxbodkjCvsSsKYKjzzyQ7+/uX3/Pqq7gzt2dbtcIxxjSUJYyp5b9P3PPnhdq5Pbc+US853OxxjXGVJwhgvm7PzuGfpRlLPa8Efru9tPZlMg2dJwhjHD0c9PZlaxEby4th+xERGuB2SMa6zhmtjgMKSMqYsXMehghJev/0i2sTHuB2SMWHBkoRp8FSV3765ifXfHub50Wkkt2/udkjGhA2rbjIN3uyVu3gjM5vpQ7tyTe92bodjTFixJGEatI+27ufx979meJ92TLuii9vhGBN2LEmYBmvbvqP88tX1JJ/bnCduSLGeTMZUwZKEaZAOHiti4vy1xEU3Zs64dJpEWU8mY6piDdemwSkuLeeORZn8cLSIpVMu4pzm1pPJGF/sSsI0KKrKQ29vZs3uXGbd0Ie+HVu4HZIxYS3kSUJErhKRbSKyQ0QeqGJ9JxH5WES+FJFPRaSD17pbReQ/zs+toY7V1H/zVmXx6to9TL38Akb0be92OMaEvZAmCRGJAJ4DrgZ6AjeLSM9Kmz0BLFDVPsAjwB+dfVsCDwMDgQHAwyKSEMp4Tf22cnsOj767lWE923LvsG5uh2NMnRDqK4kBwA5V3aWqxcCrwIhK2/QEPnGmV3itvxL4UFVzVfUQ8CFwVYjjNfXUzpxjTF2cSde28Tx1Y18aNbKeTMb4I9RJoj2wx2t+r7PM20bgemf650C8iLTyc19EZLKIZIhIRk5OTtACN/VHXkEJt83PICqiEXNvTScu2vprGOOvcGi4/hVwqYisBy4FsoEyf3dW1dmqmq6q6a1btw5VjKaOKikrZ+riTPYeKuDFsf3okBDrdkjG1CmhPqXKBjp6zXdwllVQ1e9wriREpCkwUlUPi0g2cFmlfT8NZbCm/nns3a18tuMAs27oQ3pSS7fDMabOCfWVxFrgQhHpLCJRwE3Acu8NRCRRRE7E8RvgJWf6A+AnIpLgNFj/xFlmjF9e+fwb5q/+hkk/6syo9I4172CMOU1Ik4SqlgJ34SncvwKWquoWEXlERK5zNrsM2CYi24G2wO+dfXOBR/EkmrXAI84yY2q0aucBZi7fwmXdWvPA1T3cDseYOktU1e0YgiY9PV0zMjLcDsO47JuD+Yx47t8kNo3mjTsH0ywm0u2QjAlrIrJOVdOrWhcODdfGBM3RwhImzvecKMwdl24JwpizZEnC1Btl5cq0JevJOpDP86PTSEqMczskY+o86zBu6o3/fv9rVmzL4bGfJTP4gkS3wzGmXvD7SkJEfurVC8mYsPJ6xh5mr9zFuIs6MWZQJ7fDMabeCKTQvxH4j4j8j4h0D1VAxgQqIyuXGW9u5uIuiTx0beVHgxljzobfSUJVxwCpwE5gnoisdh6JER+y6Iypwd5DBUxZuI72CU147pY0GkfYxa4xwRTQf5SqHgGW4XlQXzs8z1rKFJG7QxCbMdXKLyrltvkZFJeVM2dcOs1jrSeTMcEWSJvEdSLyJp5HY0QCA1T1aiAFuDc04RlTtfJyZfprG9i+/yjP3pJGlzZN3Q7JmHopkN5NI4E/q+pK74WqWiAiE4MbljHVe/LD7fxz634eurYnl3a1BzsaEyqBJImZwPcnZkSkCdBWVbNU9eNgB2aML29vyObZFTu4qX9HJgxJcjscY+q1QNokXgfKvebLnGXG1JoNew5z37IvGZDUkkdGJCNigwcZE0qBJInGzuhyADjTUcEPyZiq7csrZPKCDNrER/PCmDSiGltPJmNCLZD/shyvJ7ciIiOAA8EPyZjTHS8uY/LCDPKLSvnrrf1p1TTa7ZCMaRACaZO4HVgkIs8Cgmdo0XEhicoYL6rKfcs2sik7jzlj0+l2jt2aY0xt8TtJqOpOYJAzehyqeixkURnj5ZlPdvDul9/zwNXdGdqzrdvhGNOgBPSAPxEZDvQCYk40GKrqIyGIyxgA3tv0PU9+uJ3rU9sz5ZLz3Q7HmAYnkJvp/hfP85vuxlPdNAqwJ6mZkNmcncc9SzeSel4L/nB9b+vJZIwLAmm4Hqyq44BDqvo74CKga2jCMg3dD0c9PZlaxEby4th+xERGuB2SMQ1SIEmi0PldICLnAiV4nt9kTFAVlpQxZeE6DhWUMGdcOm3iY9wOyZgGK5A2iXdEpAUwC8gEFJgTiqBMw6Wq/PbNTaz/9jDPj04juX1zt0MypkHzK0k4gw19rKqHgb+JyLtAjKrmhTI40/DMXrmLNzKzmT60K9f0tgtVY9zmV3WTqpYDz3nNF1mCMMH28Vf7efz9rxnepx3TrujidjjGGAJrk/hYREaKdTExIbBt31GmLVlP8rnNeeKGFOvJZEyYCCRJTMHzQL8iETkiIkdF5EiI4jINSG5+MbctWEtcdGPmjEunSZT1ZDImXARyx7U9C8EEXXFpObe/so79R4pYOuUizmluPZmMCSd+JwkRuaSq5ZUHITLGX6rKQ29vZs3uXP5yU1/6dmzhdkjGmEoC6QJ7n9d0DDAAWAf8OKgRmQZj3qosXl27h6mXX8CIvu3dDscYU4VAqpt+6j0vIh2Bp4IdkGkYVm7P4dF3tzKsZ1vuHdbN7XCMMT6czagte4EewQrENBw7c44xdXEmXdvG89SNfWnUyHoyGROuAmmTeAbPXdbgSS598dx5bYzf8gpKuG1+BlERjZh7azpx0QE9iNgYU8sC+Q/N8JouBZao6r+DHI+px0rLypm6OJO9hwpYMmkQHRJi3Q7JGFODQJLEMqBQVcsARCRCRGJVtSA0oZn65rG/f8VnOw4w64Y+pCe1dDscY4wfArrjGmjiNd8E+Ci44Zj6atEX3zBvVRaTftSZUekd3Q7HGOOnQJJEjPeQpc50jfUFInKViGwTkR0i8kAV688TkRUisl5EvhSRa5zlSSJyXEQ2OD//G0CsJoys2nmAh9/ewmXdWvPA1dbXwZi6JJDqpnwRSVPVTAAR6Qccr24HEYnA82DAYXh6Q60VkeWqutVrsweBpar6goj0BP4BJDnrdqpq3wBiNGHmm4P53Lkok6TEOJ6+OZUI68lkTJ0SSJL4v8DrIvIdnuFLz8EznGl1BgA7VHUXgIi8CowAvJOEAs2c6ebAdwHEZMLY0cISJs739HeYOy6dZjGRLkdkjAlUIDfTrRWR7sCJO5+2qWpJDbu1B/Z4ze8FBlbaZibwTxG5G4gDhnqt6ywi64EjwIOq+q/KbyAik4HJAOedd56fn8aEWlm5Mm3JerIO5LNg4gCSEuPcDskYcwb8bpMQkalAnKpuVtXNQFMRuTMIMdwMzFPVDsA1wEJnkKPvgfNUNRW4B1gsIs0q76yqs1U1XVXTW7duHYRwTDD89/tfs2JbDjOv68XgCxLdDscYc4YCabie5IxMB4CqHgIm1bBPNuDdlaWDs8zbRGCp85qr8TwXKtEZ2Oigs3wdsBPoGkC8xiWvZ+xh9spdjLuoE2MGdXI7HGPMWQgkSUR4DzjkNEpH1bDPWuBCEeksIlHATcDyStt8C1zhvGYPPEkiR0RaO++BiJwPXAjsCiBe44KMrFxmvLmZi7sk8tC1Pd0OxxhzlgJpuH4feE1EXnTmpzjLfFLVUhG5C/gAiABeUtUtIvIIkKGqy4F7gTkiMh1PI/Z4VVXn0eSPiEgJUA7crqq5AX06U6v2Hirg9lfW0T6hCc/dkkbjiLN5NJgxJhyIqta8FeC0E0zBOesHPgTmnrgDOxykp6drRkZGzRuaoMsvKmXkC6vIPnycN+8cQpc2Td0OyRjjJxFZp6rpVa0LpHdTOfCC82NMhfJyZfprG9i+/ygvTxhgCcKYeiSQp8BeCPwR6Imn3QAAVT0/BHGZOuTJD7fzz637eejanlza1XqYGVOfBFJp/DKeq4hS4HJgAfBKKIIydcfbG7J5dsUOburfkQlDktwOxxgTZIEkiSaq+jGedoxvVHUmMDw0YZm6YMOew9y37EsGJLXkkRHJeHV+M8bUE4H0bipyGq//4/RYygas8rmB2pdXyOQFGbSJj+aFMWlENbaeTMbUR4H8Z/8Sz1NfpwH9gDHAraEIyoS348VlTF6YQX5RKX+9tT+tmka7HZIxJkQCenaTM3kMmFB5vYg8o6p3ByswE55UlfuWbWRTdh5zxqbT7Zx4t0MyxoRQMOsIhgTxtUyYevaTHbz75ff8+qruDO3Z1u1wjDEhZhXJxm/vbfqeP324netT2zPlEuv5bExDYEnC+GVzdh73LN1I6nkt+MP1va0nkzENRDCThJUa9dQPRz09mVrERvLi2H7EREa4HZIxppYEMp5E7xo2+ctZxmLCUGFJGVMWruNQQQlzxqXTJj6m5p2MMfVGIFcSz4vIGhG5U0SaV16pqvOCF5YJB6rKb9/cxPpvD/On/5NCcvvT/uzGmHrO7yShqj8CRuMZRGidiCwWkWEhi8y4bvbKXbyRmc30oV25pnc7t8MxxrggoDYJVf0P8CDwa+BS4GkR+VpErg9FcMY9H3+1n8ff/5pr+7Rj2hVd3A7HGOOSQNok+ojIn4GvgB8DP1XVHs70n0MUn3HBtn1HmbZkPcnnNmfWDSnWk8mYBiyQZzc9A8wFfquqx08sVNXvROTBoEdmXJGbX8xtC9YSF92YOePSaRJlPZmMacj8ShLOWNPZqrqwqvW+lpu6pbi0nNtfWcf+I0UsnXIR5zS3nkzGNHR+VTc5Q5R2FJGoEMdjXKKqPPT2ZtbszmXWDX3o27GF2yEZY8JAINVNu4F/i8hyIP/EQlV9MuhRmVo3b1UWr67dw9TLL2BE3/Zuh2OMCROBJImdzk8jwB79WY+s3J7Do+9uZVjPttw7rJvb4Rhjwkggjwr/XSgDMe7YmXOMqYsz6do2nqdu7EujRtaTyRhzkt9JQkRaA/cDvYCKFk1V/XEI4jK1IK+ghNvmZxAV0Yi5t6YTFx3IhaUxpiEI5Ga6RcDXQGfgd0AWsLa6HUz4Ki0rZ+riTPYeKuDFsf3okBDrdkjGmDAUSJJopap/BUpU9f+p6i/w3Ehn6qDH/v4Vn+04wB9+3pv0pJZuh2OMCVOB1C+UOL+/F5HhwHeAlS510KIvvmHeqiwm/agzo9I7uh2OMSaMBZIkHnOe/novnruvmwHTQxKVCZnVOw/y8NtbuKxbax64uofb4RhjwlwgvZvedSbzgMtDE44JpW8O5nPHonUkJcbx9M2pRFhPJmNMDQLt3TQJSPLez2mbMGHuaGEJE+dnADB3XDrNYiJdjsgYUxcEUt30NvAv4COgLDThmFAoK1emLVlP1oF8FkwcQFJinNshGWPqiECSRKyq/jpkkZiQ+e/3v2bFthwe+1kygy9IdDscY0wdEkgX2HdF5JqQRWJC4vWMPcxeuYtxF3VizKBObodjjKljAkkSv8STKI6LyBEROSoiR0IVmDl7GVm5zHhzMxd3SeSha3u6HY4xpg4KZIzreFVtpKpNVLWZM9+spv1E5CoR2SYiO0TkgSrWnyciK0RkvYh86X21IiK/cfbbJiJX+v+xzN5DBdz+yjraJzThuVvSaBwR0Ei1xhgD+NEmISLdVfVrEUmrar2qZlazbwTwHDAM2AusFZHlqrrVa7MHgaWq+oKI9AT+ASQ50zfheVbUucBHItLVGdvCVCO/qJRJC9ZRVFrOq+PSaR5rPZmMMWfGn4bre4DJwJ8A9Vouznx1j+YYAOxQ1V0AIvIqMALwThKK58Y8gOZ47uTG2e5VVS0CdovIDuf1VvsRc4NVXq7cs3QD2/Yd4eUJA+jSpqnbIRlj6rAa6yBUdbIzeQ3wdzw30x0GljvLqtMe2OM1v9dZ5m0mMEZE9uK5irg7gH0RkckikiEiGTk5OTV9nHrvyQ+388GW/Tw4vCeXdm3tdjjGmDoukIrq+UAP4Gk8j+XoCSwIQgw3A/NUtQOepLNQRAJpK5mtqumqmt66dcMuFN/ekM2zK3ZwU/+OTBiS5HY4xph6IJD7JJJV1buLzAoR2epza49swPsJch2cZd4mAlcBqOpqEYkBEv3c1zg27DnMfcu+ZEBSSx4ZkYyIPXLDGHP2ArmSyBSRQSdmRGQgkFHDPmuBC0Wks4hE4WmIXl5pm2+BK5zX7IFnQKMcZ7ubRCRaRDoDFwJrAoi3wdiXV8jkBRm0iY/mhTFpRDW2nkzGmODwp3fTJjyNy5HAKhH51pnvhGcQIp9UtVRE7gI+ACKAl1R1i4g8AmSo6nI8T5WdIyLTndcdr6oKbBGRpXgauUuBqdaz6XTHi8uYvDCD/KJSFk4cQqum0W6HZIypR8RTHlezgUi1t+mq6jdBjegspKena0ZGTRc39YeqcveS9fx90/fMGZvO0J5t3Q7JGFMHicg6VU2val2NVxLhlATMqZ79ZAfvfvk9D1zd3RKEMSYkrPK6jnp/8/f86cPtXJ/animXnO92OMaYesqSRB205bs8pr+2kdTzWvCH63tbTyZjTMhYkqhjfjhayKT5GbSIjeTFsf2IiYxwOyRjTD0WyH0SxmWFJWVMWbiOQwUlvH77RbSJj3E7JGNMPWdJoo5QVX775ibWf3uY50enkdy+udshGWMaAKtuqiNmr9zFG5nZTB/alWt6t3M7HGNMA2FJog74+Kv9PP7+11zbpx3TrujidjjGmAbEkkSY27bvKNOWrCf53ObMuiHFejIZY2qVJYkwlptfzG0L1hIX3Zg549JpEmU9mYwxtcsarsNUcWk5d7yyjv1Hilg65SLOaW49mYwxtc+uJMKQqvLw8s18sTuXWTf0oW/HFm6HZIxpoCxJhKF5q7JYsmYPUy+/gBF9TxuMzxhjao0liTCzcnsOj767lWE923LvsG5uh2OMaeAsSYSRnTnHmLo4k65t43nqxr40amQ9mYwx7rIkESbyCkq4bX4GURGNmHtrOnHR1qfAGOM+K4nCQGlZOVMXZ7L3UAFLJg2iQ0Ks2yEZYwxgSSIsPPb3r/hsxwFm3dCH9KSWbodjjDEVrLrJZYu++IZ5q7KY9KPOjErv6HY4xhhzCksSLlq98yAPv72Fy7q15oGre7gdjjHGnMaShEu+PVjAHYvWkZQYx9M3pxJhPZmMMWHIkoQLjhaWMHH+WgDmjkunWUykyxEZY0zVLEnUsrJyZdqS9ew+kM/zo9NISoxzOyRjjPHJejfVsv9+/2tWbMvhsZ8lM/iCRLfDMcaYatmVRC16PWMPs1fu4taLOjFmUCe3wzHGmBpZkqglGVm5zHhzMxd3SeS/ru3pdjjGGOMXq26qBXsPFXD7K+ton9CE525Jo3GE5WYTGiUlJezdu5fCwkK3QzFhKCYmhg4dOhAZ6X9nGUsSIZZfVMqkBesoKi3n1XHpNI+1nkwmdPbu3Ut8fDxJSUk21K05hapy8OBB9u7dS+fOnf3ez05pQ6i8XLln6Qa27TvCs7ek0aVNU7dDMvVcYWEhrVq1sgRhTiMitGrVKuCrTEsSIfTnj7bzwZb9PDi8J5d2be12OKaBsARhfDmT74YliRB5e0M2z3yyg5v6d2TCkCS3wzHGmDNiSSIENuw5zP3LvmRAUkseGZFsZ3bGmDrLkkSQ7csrZPKCDFrHR/PCmDSiGtshNg1HVlYWycnJpy2/7bbb2Lp1qwsRmbMV8t5NInIV8BcgApirqo9XWv9n4HJnNhZoo6otnHVlwCZn3beqel2o4z0bx4vLmLwwg/yiUhZOHEKrptFuh2QasN+9s4Wt3x0J6mv2PLcZD/+0V8D7zZ07NyjvX1paSuPG4dkps6ysjIiICLfDCLqQnuaKSATwHHA10BO4WUROuZNMVaeral9V7Qs8A7zhtfr4iXXhniBUlfuWbWRTdh5/uSmVbufEux2SMa4oLS1l9OjR9OjRgxtuuIGCggIuu+wyMjIyAGjatCkzZswgJSWFQYMGsX//fgDeeecdBg4cSGpqKkOHDq1YPnPmTMaOHcuQIUMYO3Ysl1xyCRs2bKh4v4svvpiNGzdWGcuaNWu46KKLSE1NZfDgwWzbtg3wFOi/+tWvSE5Opk+fPjzzzDMArF27lsGDB5OSksKAAQM4evQo8+bN46677qp4zWuvvZZPP/204rPce++9pKSksHr1ah555BH69+9PcnIykydPRlUB2LFjB0OHDiUlJYW0tDR27tzJuHHjeOuttyped/To0bz99ttn/wcINlUN2Q9wEfCB1/xvgN9Us/0qYJjX/LFA3q9fv37qlqc/2q6dfv2uvvDpDtdiMGbr1q2uvv/u3bsV0M8++0xVVSdMmKCzZs3SSy+9VNeuXauqqoAuX75cVVXvu+8+ffTRR1VVNTc3V8vLy1VVdc6cOXrPPfeoqurDDz+saWlpWlBQoKqq8+bN01/+8peqqrpt2zat7v8+Ly9PS0pKVFX1ww8/1Ouvv15VVZ9//nkdOXJkxbqDBw9qUVGRdu7cWdesWXPKvi+//LJOnTq14jWHDx+uK1asqPgsr732WsW6gwcPVkyPGTOm4nMOGDBA33jjDVVVPX78uObn5+unn36qI0aMUFXVw4cPa1JSUkU8oVTVdwTIUB/laqgrzNsDe7zm9zrLTiMinYDOwCdei2NEJENEPheRn4UsyrP0/ubv+dOH27k+tT1TLjnf7XCMcVXHjh0ZMmQIAGPGjOGzzz47ZX1UVBTXXnstAP369SMrKwvw3Ah45ZVX0rt3b2bNmsWWLVsq9rnuuuto0qQJAKNGjeLdd9+lpKSEl156ifHjx/uMJS8vj1GjRpGcnMz06dMrXvOjjz5iypQpFVVXLVu2ZNu2bbRr147+/fsD0KxZsxqrtiIiIhg5cmTF/IoVKxg4cCC9e/fmk08+YcuWLRw9epTs7Gx+/vOfA567nmNjY7n00kv5z3/+Q05ODkuWLGHkyJFhWZUWTq2qNwHLVLXMa1knVU0HbgGeEpELKu8kIpOdRJKRk5NTW7FW2PJdHtNf20jqeS34w/W9rSeTafAq/w9Uno+MjKxYFhERQWlpKQB33303d911F5s2beLFF1885aavuLiTj9SPjY1l2LBhvP322yxdupTRo0f7jOW//uu/uPzyy9m8eTPvvPPOGT2upHHjxpSXl1fMe79GTExMRTtEYWEhd955J8uWLWPTpk1MmjSpxvcbN24cr7zyCi+//DK/+MUvAo6tNoQ6SWQD3gM3d3CWVeUmYIn3AlXNdn7vAj4FUivvpKqzVTVdVdNbt67dG9ZyjhYxaX4GLWIjeXFsP2Ii61+jlTGB+vbbb1m9ejUAixcv5uKLL/Zrv7y8PNq391Q0zJ8/v9ptb7vtNqZNm0b//v1JSEjw6zXnzZtXsXzYsGG8+OKLFQkqNzeXbt268f3337N2rWdAsKNHj1JaWkpSUhIbNmygvLycPXv2sGbNmirf60RCSExM5NixYyxbtgyA+Ph4OnToUNH+UFRUREFBAQDjx4/nqaeeAqBnz/B88Geok8Ra4EIR6SwiUXgSwfLKG4lIdyABWO21LEFEop3pRGAIEDZ96IpKy5iyMINDBSXMGZdOm/gYt0MyJix069aN5557jh49enDo0CHuuOMOv/abOXMmo0aNol+/fiQmVj/WSr9+/WjWrBkTJkyodrv777+f3/zmN6SmplYkBPAkmfPOO48+ffqQkpLC4sWLiYqK4rXXXuPuu+8mJSWFYcOGUVhYyJAhQ+jcuTM9e/Zk2rRppKWlVfleLVq0YNKkSSQnJ3PllVdWVFsBLFy4kKeffpo+ffowePBg9u3bB0Dbtm3p0aNHjZ/DTaJO63vI3kDkGuApPF1gX1LV34vII3gaSpY728wEYlT1Aa/9BgMvAuV4ktlTqvrX6t4rPT1dT/SgCCVV5d7XN/JGZjbPj07jmt7tQv6exvjjq6++okePHm6HEXLfffcdl112GV9//TWNGoVTrXlgCgoK6N27N5mZmTRv3rxW3rOq74iIrHOq9k8T8lYSVf0H8I9Kyx6qND+ziv1WAb1DGtwZmr1yF29kZjN9aFdLEMbUsgULFjBjxgyefPLJOp0gPvroIyZOnMj06dNrLUGcifBrSg9zH3+1n8ff/5pr+7Rj2hVd3A7HmAZn3LhxjBs37pRlL7/8Mn/5y19OWTZkyBCee+652gwtIEOHDuWbb75xO4waWZIIwLZ9R5m2ZD3J5zZn1g0p1pPJmDAxYcKEsK7Xr8vq7rVaLcvNL+a2BWuJi27MnHHpNImynkzGmPrPriT8UFxazh2vrGP/kSKWTrmIc5pbTyZjTMNgVxI1UFUeXr6ZL3bnMuuGPvTt2MLtkIwxptZYkqjB/FVZLFmzh6mXX8CIvlU+UcQYY+otSxLVWLk9h0fe3cqwnm25d1g3t8Mxpt5p2tT3uO+ffvppxTOeKrvmmms4fPhwiKIy3qxNwoedOceYujiTrm3jeerGvjRqZD2ZTB3z3gOwb1PN2wXinN5w9eM1bxdi//jHP2reyA/hOj5FxRNYw+A+EPcjCEN5BSXcNj+DqIhGzL01nbjo8PsSGROOHnjggVPuTZg5cyaPPfYYV1xxBWlpafTu3TugMROOHDnC8OHD6datG7fffnvFg/aSkpI4cOAAWVlZ9OjRg0mTJtGrVy9+8pOfcPz4cQDmzJlD//79SUlJYeTIkac8L+n2229n4MCB3H///Vx44YWceDhoeXk5Xbp0wdfDQn2NeXHs2DEmTJhA79696dOnD3/7298AeP/990lLSyMlJYUrrrii4pg88cQTFa+ZnJxMVlYWWVlZdOvWjXHjxpGcnMyePXu44447SE9Pp1evXjz88MMV+1Q17kUg42wExNczxOviTzDGkygpLdPRcz7XLr/9u67dfbDmHYwJI26PJ5GZmamXXHJJxXyPHj3022+/1by8PFVVzcnJ0QsuuKBi3Ii4uDifr7VixQqNjo7WnTt3amlpqQ4dOlRff/11VVXt1KmT5uTk6O7duzUiIkLXr1+vqqqjRo3ShQsXqqrqgQMHKl5rxowZ+vTTT6uq6q233qrDhw/X0tJSVVWdOXOm/vnPf1ZV1Q8++KBizImq+Brz4v77768Y4+LEdj/88IN26NBBd+3apaonx5p4+OGHddasWRXb9urVS3fv3q27d+9WEdHVq1dXrDuxT2lpqV566aW6ceNGn+Ne+DvORriNJ1HnPPb3r/hsxwH+8PPepCe1dDscY+qU1NRUfvjhB7777js2btxIQkIC55xzDr/97W/p06cPQ4cOJTs7u+IMvCYDBgzg/PPPJyIigptvvvm0sSkAOnfuTN++fYFTx6fYvHkzP/rRj+jduzeLFi06ZXyKUaNGVTzi+xe/+AULFiwA4KWXXqr2pjxfY1589NFHTJ06tWK7hIQEPv/8cy655BI6d+4MeMasqEmnTp0YNGhQxfzSpUtJS0sjNTWVLVu2sHXrVp/jXgQyzkYgrB7Fy6IvvmHeqiwm/agzo9I71ryDMeY0o0aNYtmyZezbt48bb7yRRYsWkZOTw7p164iMjCQpKcnvcR1qGpsCIDr65FjyERERFdVN48eP56233iIlJYV58+ZVDDkKp45P0bFjR9q2bcsnn3zCmjVrWLRokc947r77bu655x6uu+46Pv30U2bOnOnX5/BW3fgU3nHt3r2bJ554grVr15KQkMD48eOrPW6Vx9lYt25dwLFVxa4kHKt3HuTht7dwWbfWPHB1/X+KpjGhcuONN/Lqq6+ybNkyRo0aRV5eHm3atCEyMpIVK1YE9LyiNWvWsHv3bsrLy3nttdf8HpsCPONBtGvXjpKSkmoLfvA8OnzMmDGnXGFUxdeYF8OGDTulLebQoUMMGjSIlStXsnv3bsAzZgV42lMyMzMByMzMrFhf2ZEjR4iLi6N58+bs37+f9957D8DnuBcnPoc/42wEwpIE8O3BAu5YtI6kxDievjmVCOvJZMwZ69WrF0ePHqV9+/a0a9eO0aNHk5GRQe/evVmwYAHdu3f3+7X69+/PXXfdRY8ePejcuXPFEKD+ePTRRxk4cCBDhgyp8T2vu+66isbn6vga8+LBBx/k0KFDJCcnk5KSwooVK2jdujWzZ8/m+uuvJyUlhRtvvBGAkSNHkpubS69evXj22Wfp2rVrle+VkpJCamoq3bt355ZbbqkYEtbXuBfg/zgbgQj5eBK16UzHk9iXV8h9yzby6IhkkhLjat7BmDDVUMaTCLaMjAymT5/Ov/71L7dDOSv+jLMR6HgSdiUBnNM8hoUTB1qCMKYBevzxxxk5ciR//OMf3Q7lrCxYsICBAwfy+9//Pqj3V9iVhDH1SF28kti0aRNjx449ZVl0dDRffPGFSxHB73//e15//fVTlo0aNYoZM2a4FFHwBHolYUnCmHrkq6++onv37jbWiamSqvL1119bdZMxDVVMTAwHDx6kPp38meBQVQ4ePEhMTGBDHdh9EsbUIx06dGDv3r0+HythGraYmBg6dOgQ0D6WJIypRyIjIyvu8DUmGKy6yRhjjE+WJIwxxvhkScIYY4xP9aoLrIjkAP4/GOZ0icCBIIUTTBZXYCyuwFhcgamPcXVS1dZVrahXSeJsiUiGr77CbrK4AmNxBcbiCkxDi8uqm4wxxvhkScIYY4xPliRONdvtAHywuAJjcQXG4gpMg4rL2iSMMcb4ZFcSxhhjfLIkYYwxxqcGkSRE5CoR2SYiO0TkgSrWR4vIa876L0QkyWvdb5zl20TkylqO6x4R2SoiX4rIxyLSyWtdmYhscH6W13Jc40Ukx+v9b/Nad6uI/Mf5ubWW4/qzV0zbReSw17pQHq+XROQHEdnsY72IyNNO3F+KSJrXulAer5riGu3Es0lEVolIite6LGf5BhEJ6vP3/YjrMhHJ8/p7PeS1rtrvQIjjus8rps3Od6qlsy6Ux6ujiKxwyoItIvLLKrYJ3XdMVev1DxAB7ATOB6KAjUDPStvcCfyvM30T8Joz3dPZPhro7LxORC3GdTkQ60zfcSIuZ/6Yi8drPPBsFfu2BHY5vxOc6YTaiqvS9ncDL4X6eDmvfQmQBmz2sf4a4D1AgEHAF6E+Xn7GNfjE+wFXn4jLmc8CEl06XpcB757tdyDYcVXa9qfAJ7V0vNoBac50PLC9iv/JkH3HGsKVxABgh6ruUtVi4FVgRKVtRgDznellwBUiIs7yV1W1SFV3Azuc16uVuFR1haoWOLOfA4E94zdEcVXjSuBDVc1V1UPAh8BVLsV1M7AkSO9dLVVdCeRWs8kIYIF6fA60EJF2hPZ41RiXqq5y3hdq7/vlz/Hy5Wy+m8GOqza/X9+raqYzfRT4CmhfabOQfccaQpJoD+zxmt/L6Qe4YhtVLQXygFZ+7hvKuLxNxHOmcEKMiGSIyOci8rMgxRRIXCOdy9plItIxwH1DGRdOtVxn4BOvxaE6Xv7wFXsoj1egKn+/FPiniKwTkckuxHORiGwUkfdEpJezLCyOl4jE4ilo/+a1uFaOl3iqwlOBymO7huw7ZuNJ1AEiMgZIBy71WtxJVbNF5HzgExHZpKo7aymkd4AlqlokIlPwXIX9uJbe2x83ActUtcxrmZvHK6yJyOV4ksTFXosvdo5XG+BDEfnaOdOuDZl4/l7HROQa4C3gwlp6b3/8FPi3qnpfdYT8eIlIUzyJ6f+q6pFgvnZ1GsKVRDbQ0Wu+g7Osym1EpDHQHDjo576hjAsRGQrMAK5T1aITy1U12/m9C/gUz9lFrcSlqge9YpkL9PN331DG5eUmKlUFhPB4+cNX7KE8Xn4RkT54/oYjVPXgieVex+sH4E2CV81aI1U9oqrHnOl/AJEikkgYHC9Hdd+vkBwvEYnEkyAWqeobVWwSuu9YKBpawukHz9XSLjzVDycau3pV2mYqpzZcL3Wme3Fqw/Uugtdw7U9cqXga6i6stDwBiHamE4H/EKQGPD/jauc1/XPgcz3ZSLbbiS/BmW5ZW3E523XH04gotXG8vN4jCd8NscM5tVFxTaiPl59xnYennW1wpeVxQLzX9CrgqlqM65wTfz88he23zrHz6zsQqric9c3xtFvE1dbxcj77AuCparYJ2XcsaAc3nH/wtPxvx1PgznCWPYLn7BwgBnjd+YdZA5zvte8MZ79twNW1HNdHwH5gg/Oz3Fk+GNjk/JNsAibWclx/BLY4778C6O617y+c47gDmFCbcTnzM4HHK+0X6uO1BPgeKMFT5zsRuB243VkvwHNO3JuA9Fo6XjXFNRc45PX9ynCWn+8cq43O33lGLcd1l9f363O8klhV34HaisvZZjyezize+4X6eF2Mp83jS6+/1TW19R2zx3IYY4zxqSG0SRhjjDlDliSMMcb4ZEnCGGOMT5YkjDHG+GRJwhhjjE+WJIzxQ6WnyG4I5hNIRSTJ15NHjXGbPZbDGP8cV9W+bgdhTG2zKwljzoIzjsD/OGMJrBGRLs7yJBH5RE6OBXKes7ytiLzpPLxuo4gMdl4qQkTmOOMF/FNEmjjbT5OTY4q86tLHNA2YJQlj/NOkUnXTjV7r8lS1N/As8JSz7Blgvqr2ARYBTzvLnwb+n6qm4Bm7YIuz/ELgOVXtBRwGRjrLHwBSnde5PTQfzRjf7I5rY/wgIsdUtWkVy7OAH6vqLuchbPtUtZWIHMDzjKsSZ/n3qpooIjlAB/V6WKPz+OcPVfVCZ/7XQKSqPiYi7wPH8DwJ9S11HnxnTG2xKwljzp76mA5Ekdd0GSfbC4fjeSZPGrDWeUqxMbXGkoQxZ+9Gr9+rnelVeJ4oDDAa+Jcz/TGeoWgRkQgRae7rRUWkEdBRVVcAv8bzBNLTrmaMCSU7KzHGP01EZIPX/PuqeqIbbIKIfInnauBmZ9ndwMsich+QA0xwlv8SmC0iE/FcMdyB58mjVYkAXnESiQBPq+rhIH0eY/xibRLGnAWnTSJdVQ+4HYsxoWDVTcYYY3yyKwljjDE+2ZWEMcYYnyxJGGOM8cmShDHGGJ8sSRhjjPHJkoQxxhif/j8hwhC7bXfoHQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WquT1v6FbqQ6"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hkhtCCgnUbY6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "31909ce0-dfa2-4713-d837-935397dc5386"
      },
      "source": [
        "# testing on some samples\n",
        "sample_example = [\"Do you have an adopted dog, how would you encourage people to adopt and not shop?\"]\n",
        "model.predict(sample_example)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.00047495]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-wcRtQBHcuxR",
        "outputId": "e949ba9d-f50f-4038-9db9-173d80e45f65"
      },
      "source": [
        "sample_example = [\"HWhy do Europeans say they're the superior race, when in fact it took them over 2,000 years until mid 19th century to surpass China's largest economy?\"]\n",
        "model.predict(sample_example)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.7486439]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aiKuBGgfJUKv"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}