Taxanomy-Prediction-Notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "edits3.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bdYRkEuuu9EF",
        "colab_type": "text"
      },
      "source": [
        "# Taxonomy Prediction\n",
        "## Given some text content, come up with a solution to build the Taxonomy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JNab_S1Dt4DT",
        "colab_type": "text"
      },
      "source": [
        "# Preface\n",
        "</br>\n",
        "\n",
        "## Problem Description\n",
        "In Taxonomy Creation problem, we have a dataset collected from some QnA website which contains Title, Body and associated Tags.</br>\n",
        "\n",
        "Our model will have to categorize text into various domains, based on occurrence of certain words. \n",
        "\n",
        "Any question can be categorized into multiple domains, that is it can have multiple tags.\n",
        "\n",
        "This problem and it’s dataset is taken from [Facebook Recruiting III - Keyword Extraction competition held on Kaggle](https://www.kaggle.com/c/facebook-recruitingiii-keyword-extraction/rules).\n",
        "\n",
        "</br>\n",
        "\n",
        "## Skills required to tackle the problem:\n",
        "▪ Machine Learning </br>\n",
        "▪ Natural Language Processing </br>\n",
        "▪ Data Engineering </br>\n",
        "▪ Working with multiple processors </br>\n",
        "\n",
        "</br>\n",
        "\n",
        "This is a Multi-Class Multi-Label problem, since we have to determine all the classes of tags a question belongs to.\n",
        "\n",
        "Thus, we need to train model to predict a binary output for each tag for any given question. \n",
        "\n",
        "</br>\n",
        "\n",
        "## Training Dataset contains roughly 60M examples in 4 Columns:\n",
        "• Id – Unique identifier for each question</br>\n",
        "• Title – The question's title</br>\n",
        "• Body – The body of the question</br>\n",
        "• Tags – The tags associated with the question</br>\n",
        "\n",
        "</br>\n",
        "\n",
        "## Libraries used:\n",
        "▪ Numpy – To handle efficient arrays in RAM.</br>\n",
        "▪ Pandas – To created manageable Data Frames.</br>\n",
        "▪ Matplotlib – To plot the data in graphs for better understanding.</br>\n",
        "▪ Scikit-Learn – To apply ML models and algorithms on data.</br>\n",
        "▪ Scipy – To store sparse matrices in efficient format.</br>\n",
        "▪ Pickle – To save models and vectorizers used on data.</br>\n",
        "\n",
        "</br>\n",
        "\n",
        "### Since the dataset size is very large for local machines: \n",
        " \n",
        "#### ▪ We only use the most frequent 500 tags from the dataset. \n",
        " \n",
        "> We randomly sample examples from dataset such that their associated tags belong to only the top 500 tags. \n",
        " \n",
        " \n",
        "#### ▪ We only use 500,000 examples all which contain tags belonging to the set of top 500 tags. \n",
        " \n",
        "> This sampling is done to reduce the overhead on the processor and also it can be verified that top 500 tags cover most of the questions and 500,000 quality examples are enough to give highly acceptable output.\n",
        "\n",
        "</br>\n",
        "\n",
        "In our problem, the questions can be classified based on individual words and there is not so much significance of order in which words appear in the text, we can use the simplest model of all: **Bag-Of-Words** model.\n",
        "\n",
        "\n",
        "![Image - Bag of Words](https://ldabook.com/Images/BagOfWords.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q4GswziryJEU",
        "colab_type": "text"
      },
      "source": [
        "# Solution Approach\n",
        "\n",
        "## Data Exploration\n",
        "▪ Check for all NaN values in data. </br>\n",
        "▪ Plot tags frequency.</br>\n",
        "▪ Check how tags are distributed among question.</br>\n",
        "▪ Obtain Conclusion and Strategize next steps.</br> \n",
        "\n",
        "## Data Cleaning and Data Engineering\n",
        "▪ Remove unnecessary features from data [Id]. </br>\n",
        "▪ Drop all rows where Tags column is NaN.</br>\n",
        "▪ Select Most Frequent 500 tags</br>\n",
        "> Create a list of top 500 tags.</br>\n",
        "> Find indices of examples containing all tags as a subset of top 500 tags.</br>\n",
        "\n",
        "▪ Sample 500,000 indices from list of indices obtained from previous step.</br>\n",
        "▪ Sample the training set using those indices and save. </br>\n",
        "\n",
        "## Further Data Engineering\n",
        "▪ Using Regular Expression, clean all the Titles in the Title column.</br>\n",
        "▪ Separate Code part from the Body and put into Code column</br>\n",
        "▪ Clean Body column using Regular Expression</br>\n",
        "▪ Similarly, clean Code Column</br>\n",
        "▪ Create new Data Frame by adding Title, Body and Code columns, separated by space into a single column. </br>\n",
        "\n",
        "\n",
        "## Tokenize + Remove Stop words + Stemming \n",
        "▪ Loop through all the 500,000 examples: </br>\n",
        "> Tokenize the text </br>\n",
        "> Remove Stop words from it </br>\n",
        "> Stem the remaining words </br>\n",
        "> Join the words again to form a string </br>\n",
        "\n",
        "▪ Save the new modified dataset.</br>\n",
        "\n",
        "\n",
        "## Vectorizing\n",
        "▪ Apply binary Count Vectorizer on Tags </br>\n",
        "▪ Apply Count Vectorizer on Text</br>\n",
        "\n",
        "\n",
        "## Training \n",
        "▪ Train Stochastic Gradient Descent model. </br>\n",
        "▪ Train Support Vector Classifier </br>\n",
        "▪ Train Logistic Regression Classifier </br>\n",
        "\n",
        "\n",
        "## Testing \n",
        "▪ Test different models on data. </br>\n",
        "▪ Select the best performance model </br>\n",
        "\n",
        "</br>\n",
        "\n",
        "> ## **Let's Code it in the next section!**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h41Q4k-0skux",
        "colab_type": "text"
      },
      "source": [
        "# Data Exploration\n",
        "> In this part, we'll have a look at how our data is structured, we'll look at the patterns with which labels (Tags) are affected.</br>\n",
        "\n",
        "We'll note our observations and make conclusions about the data. </br>\n",
        "Based on these observations, we'll strategize our next steps to tackle the problem.\n",
        "\n",
        "## Steps:\n",
        "1. We'll look at the shape of data.\n",
        "2. We'll look at few data points.\n",
        "3. We'll look at the no of NaN values in each column.\n",
        "4. We'll plot graphs\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zKg6LQADqyjS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Importing Libraries\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oBPL7EvYqyfv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Import data\n",
        "\n",
        "df = pd.read_csv(\"data/Train.csv\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8ZDXYCt9qydF",
        "colab_type": "code",
        "outputId": "a5701215-f33a-4095-c734-cf28dbde0c48",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        }
      },
      "source": [
        "# Data frame shape and few data points\n",
        "\n",
        "print(df.shape)\n",
        "df.head(5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(6034195, 4)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>Title</th>\n",
              "      <th>Body</th>\n",
              "      <th>Tags</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>How to check if an uploaded file is an image w...</td>\n",
              "      <td>&lt;p&gt;I'd like to check if an uploaded file is an...</td>\n",
              "      <td>php image-processing file-upload upload mime-t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>How can I prevent firefox from closing when I ...</td>\n",
              "      <td>&lt;p&gt;In my favorite editor (vim), I regularly us...</td>\n",
              "      <td>firefox</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>R Error Invalid type (list) for variable</td>\n",
              "      <td>&lt;p&gt;I am import matlab file and construct a dat...</td>\n",
              "      <td>r matlab machine-learning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>How do I replace special characters in a URL?</td>\n",
              "      <td>&lt;p&gt;This is probably very simple, but I simply ...</td>\n",
              "      <td>c# url encoding</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>How to modify whois contact details?</td>\n",
              "      <td>&lt;pre&gt;&lt;code&gt;function modify(.......)\\n{\\n  $mco...</td>\n",
              "      <td>php api file-get-contents</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Id  ...                                               Tags\n",
              "0   1  ...  php image-processing file-upload upload mime-t...\n",
              "1   2  ...                                            firefox\n",
              "2   3  ...                          r matlab machine-learning\n",
              "3   4  ...                                    c# url encoding\n",
              "4   5  ...                          php api file-get-contents\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l68SdfShwlOR",
        "colab_type": "text"
      },
      "source": [
        "**We have over 60 million examples**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4L_mQmtouTVz",
        "colab_type": "code",
        "outputId": "273f35ac-9710-4403-f4e6-e237df4a7409",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "source": [
        "# Checking for NaNs in each column\n",
        "\n",
        "df.isna().sum()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Id       0\n",
              "Title    0\n",
              "Body     0\n",
              "Tags     8\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kfX1ipJAwADg",
        "colab_type": "text"
      },
      "source": [
        "**So, we have 8 such questions which have no tags associated with them.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nc4jwS6TvFBX",
        "colab_type": "code",
        "outputId": "f9ed3954-7204-492b-c73c-4b080eee7831",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        }
      },
      "source": [
        "# Plotting Tag Distribution in questions.\n",
        "\n",
        "# creating a list of all tags\n",
        "all_tags = [tag for slist in df[\"Tags\"].dropna().values for tag in slist.split()]\n",
        "\n",
        "# counting each tag\n",
        "import collections\n",
        "counter=collections.Counter(all_tags)\n",
        "\n",
        "# creating a list of most frequent 100 tags\n",
        "tags_list = [ y for x, y in sorted(counter.items(), key=lambda x: x[1], reverse=True) ]\n",
        "top_100_tags = tags_list[:100]\n",
        "\n",
        "# plotting the frequency\n",
        "plt.plot(top_100_tags)\n",
        "plt.title(\"Tag Distribution in Questions\")\n",
        "plt.grid()\n",
        "plt.xlabel(\"Tag index\")\n",
        "plt.ylabel(\"Frequency in tags\")\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEWCAYAAACwtjr+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcXFWd///Xp6ur93R31iYrSToB\nTIQQCBACYthBNhVQGBT0B+LM4IjLzIiOI6LjMurI4lcRBEZgUERQWQwiW4MIIRB2AoFOQvZ9706n\n18/vj3s6FE0vle6qru6q9/PxqEfXPXc5n1OV9KfPveeea+6OiIhIOuVlOgAREcl+SjYiIpJ2SjYi\nIpJ2SjYiIpJ2SjYiIpJ2SjYiIpJ2SjaSdczsBDN7OYXHe8zMPhne/6OZPZLCY19sZven6ngJx03p\nZ5BJZrbEzI7MdBzSN0o20mdmVpfwajOzhoTlC1Jc1w/NrNnMdobXm2Z2rZmNat/G3R9x9xlJHuum\nnrZz9+Pc/XcpiP0AM2vpcOyb3f2Mvh67o2Q/g66Y2b5mdqeZbQnf4zNmdlIqY+yi3jvN7JuJZe5e\n7e7PpLtuSS8lG+kzdy9rfwErgDMSyu5IQ5W3uvsQYDhwLjAReN7MRqayEjPLM7Oc+z8SEvffgW3A\nAcAo4JfAPWb2kUzGJoNXzv1Hkv5nZkeZ2bNmts3M1pjZ1WaWn7D+NDN7O6y/xszmm9mnejquuze5\n+6vAOUA9cHk43ilmVptw/P80s7VmtsPM3jCzD5nZR4GvABeFv9wXhG3nm9l3zOxZYBcwppN48szs\nhnC8RWZ2TEJd68zs6ITlxN7Tk0Asodc3s+NpOTP7sJm9YGbbQ72HJaybb2ZXhp87zGyemQ3t4jPv\n+BmsM7Mvm9lr4dh3mFlBFx/tvwHr3f0f3X2Du+9y91uBnwD/E473vl5ax8/JzD5vZotD7+jPZjY2\nlMfM7OdmtjHE8rKZ7W9mXwTOBv4zfD6/7/iZmllx2Hetma0ysx+bWTyxzWb2jXDs1Yk9azM7K/SE\nd5rZylCf9BMlG+kPzcAXiHoiHwLOAC4BMLPRwO+ALwMjgTXAoXtzcHdvBu4Px34PM5sBfBY4GKgA\nTgNWufufgJ8S9ZLK3P3whN0+BVwIDAHWdVLlMcDLoT0/BP5kZuVJhHoM0JrQ63uxQ6yjQjt+GI79\nS2CemVUkbPYPwAXAaKCSkGCTdA5wPDAFOCIcqzMnAnd3Un4XcICZje+pIouucX2J6LuuAl4E/i+s\nPh04BKgGhoY4trr7dcA9wHfD53NuJ4e+CjgIOJDo38lc4N8T1u8LGDCG6N/cL82sLKy7Bbgw9IoP\nBv7WUzskdZRsJO3cfYG7P+fure6+BLgJ+HBYfQbwnLs/EJLGT4CtvahmDTCsk/IWoBiYBsTcfam7\nL+vhWDe5+2J3b3b3lk7Wr3T3X4T1twGrgJN7EXNHZwEvuftd7t7i7r8Oxz41YZtfufsSd68nSggH\n78Xxr3b39e6+EZjXzb4jgLWdlLeXJXO68h+B/3L3t8L3ehVwtJlVEf3xUU50ig53f93dNyTZhguA\nK919k7uvB/4L+HTC+l3AD8J380fAiZIrQCsw3cyGuPvmjsle0kvJRtLOzKaZ2YNmtt7MdgDfIvqF\nBtFfoCvbt3X3NmB1L6oZC2zpWOjurwNXAN8DNoTTR1U9HGtlD+tXdVheTtSOvhoTjtXx2GMTlhN7\nWruAMpKX7L6biHpOHY1OWN+TfYl6FdvMbBuwkSjxjwMeBG4GbgDWmdkvEnofXTIzA/bhvZ9Rx89n\nY/g31C6xnWcRnaZbYdEIw8OQfqNkI/3hV8ALQLW7lwPfITrVAdFfy+PaNwwX5Me+7wjdCNd/TqeL\n0yLufqu7zwEmA0VEfw1D9Fdvp7v0UOW4DssTiHpWEF07KklYt89eHHcN0S/pjsfuTfLti0eITrl1\n9AlgKVEyrie6/lSYsD6xrSuBz7h7ZcKr2N0XeuSn7j6T6JTYDN49HdjlZ+TRFPXreO9nlPTn4+7P\nuPvpRKf1/gr8Jpn9JDWUbKQ/DAG2u3udmU0HPpew7j7gCDP7SEgaXyE6j98jM4ub2QeJriUMAa7r\nZJtp4aJ7IdAQXu1/+a4HJoW/mPfG+HBhPz9cEB9P9MsL4CXg/LBuNtFf0+02EP2CntDFce8DZprZ\nOWH/C4l+mT64l/H11Y+B0Wb2SzMbFS7KXwh8DfjP8Et/DVFv5YJwwf+fee8fCb8Evmlm+wOY2VAz\nOzu8n21ms8L3XQ808d7vZHI3sf0WuNLMhodrXP/Bu9eCumRmpWZ2Xri21gzsTKhT+oGSjfSHLwOX\nmFkd8HOiAQEAuPta4HyiRLGJqNfwKtDYzfEuMrOdRENz/0D0l+1hXZz3LyYaQbWJqBdVBvxnWHcn\nUS9ki5k9vRfteRKYSXTa7j+Aj7v79rDuG0QXr7cBXw91tLd1K/AjYGE4vfSeaybhGsSZ4ZibiS5w\nn55w7H4R4jiK6BrYYqCOqHd6sbv/JmzTSjTI40qiz3Y8sDDhGL8F/h/wh3Dq9CWigQcQDWz4NdFn\ntJToVNi1Yd2NwGHh89nz2SX4FrAIeD0c8+9En2ky/r9Q13aiASAXJrmfpIDp4WkykIS/dtcR3auj\nG/kGgDC8+mngdnf/fqbjkcFJPRvJODM71cwqzKyI6C/lXST8lSyZFXpkHwHiluIbZyV3qGcjGWdm\nPwA+D+QDrwH/4u5KNiJZRMlGRETSTqfRREQk7fJ73iQ3jBgxwidOnNirfevr6yktLU1tQINALrY7\nF9sMudlutTk5Cxcu3OTuPV7LU7IJJk6cyPPPP9+rfWtqapg7d25qAxoEcrHdudhmyM12q83JMbOO\ns150SqfRREQk7ZRsREQk7ZRsREQk7ZRsREQk7ZRsREQk7ZRsREQk7ZRsREQk7ZRs+uhPL67msRXN\nmQ5DRGRAU7Lpo3mvruWR5Uo2IiLdUbLpo32Hl7CxwWlr04SmIiJdUbLpownDSmhugw07u3uwpIhI\nblOy6aMJw6NJ61Zs2ZXhSEREBi4lmz6aMKwEgOWb6zMciYjIwKVk00djK4sxYKV6NiIiXVKy6aOC\n/DyGFRnLlWxERLqkZJMCo0pM12xERLqhZJMCI0vydBpNRKQbSjYpMKrE2FTXRF1jS6ZDEREZkJRs\nUmBUcfQxqncjItI5JZsUGFViACzfrGQjItIZJZsUGFmino2ISHeUbFKgNG5UFMdZvkU3doqIdEbJ\nJkUmDCthxZaGTIchIjIgKdmkyIThJazQlDUiIp1SskmRCcNKWLW1gVY9akBE5H2UbFJk32EltLQ5\na7bpVJqISEdKNinSPvuzRqSJiLyfkk2KTBgeHjWgZCMi8j5KNikyuqKYeEwTcoqIdEbJJkVieca4\noSWs0CwCIiLvo2STQuOHlejGThGRTijZpNAHx5TzxtqdbK5rzHQoIiIDipJNCp0xYwytbc6819Zl\nOhQRkQEl7cnGzGJm9qKZPRCWJ5nZs2ZWa2a/M7OCUF4YlmvD+okJx/h6KF9sZicnlJ8SymrN7IqE\n8k7rSLcD9hnCflVl3PfS6v6oTkRk0OiPns3lwBsJy/8NXO3uU4CtwMWh/GJgayi/OmyHmU0DzgOm\nA6cAvwgJLAb8HDgVmAacH7btro60MjPOnDGG597Zymrd3Ckiskdak42ZjQNOA24KywYcB9wdNrkV\n+Gh4f1ZYJqw/Pmx/FnCnuze6+zKgFjg8vGrdfam7NwF3Amf1UEfanTljLAD3v7ymv6oUERnw8tN8\n/GuAfweGhOXhwDZ3b39+8ipgbHg/FlgJ4O4tZrY9bD8WmJ9wzMR9VnYoP6KHOt7DzC4FLgWoqqqi\npqZm71sI1NXVvWffyRV53PHUWxzgK7veKQt0bHcuyMU2Q262W21OrbQlGzM7Hdjg7gvNbG666ukL\nd78RuBFg1qxZPnfu3F4dp6amhsR9l8WXcdX9ixj7gUOZWjWk6x0HuY7tzgW52GbIzXarzamVztNo\nRwFnmtk7RKe4jgOuBSrNrD3JjQPar6avBsYDhPUVwObE8g77dFW+uZs6+sVpB40mz+A+nUoTEQHS\nmGzc/evuPs7dJxJd4H/M3S8AHgfOCZtdBNwb3t8XlgnrH3N3D+XnhdFqk4CpwALgOWBqGHlWEOq4\nL+zTVR39YtSQIuZUj+C+l9cQhSMiktsycZ/N14CvmFkt0fWVm0P5zcDwUP4V4AoAd38duAtYBPwF\nuMzdW8M1mS8ADxGNdrsrbNtdHf3mpOlVLN+8i5V6eqeISNoHCADg7jVATXi/lGgkWcdtdgPndrH/\n94DvdVI+D5jXSXmndfSnOdXDAXh6ySYmDJ+QyVBERDJOMwikSfXIMkYOKeTpJZszHYqISMYp2aSJ\nmTGnejhPL9ms6zYikvOUbNJoTvVwNtU1UruhLtOhiIhklJJNGs2pHgGgU2kikvOUbNJo/LASxg0t\n5uklmzIdiohIRinZpNmc6uHMX7qFtjZdtxGR3KVkk2ZzqkewvaGZRWt3ZDoUEZGMUbJJsyPD/TbP\n6LqNiOQwJZs0qyovonpkqa7biEhOU7LpB3OqR7Bg2RZadd1GRHKUkk0/+ODYcuqbWlm1dVemQxER\nyQglm35QPbIMgKUb6zMciYhIZijZ9IPJIdks2aiZBEQkNynZ9INhpQVUlsRZop6NiOQoJZt+Uj2y\njKXq2YhIjlKy6SeTR5SydJN6NiKSm5Rs+snkkWVs3NnIjt3NmQ5FRKTfKdn0k+qRpYBGpIlIblKy\n6SeT9wx/1nUbEck9Sjb9ZMKwEmJ5puHPIpKTlGz6SUF+HhOGleg0mojkJCWbflQ9slTJRkRy0l4l\nGzMbamYHpSuYbDd5ZBnLNtdrQk4RyTk9JhszqzGzcjMbBrwA/MrMfpr+0LLP5BGlNLW0sXprQ6ZD\nERHpV8n0bCrcfQfwceA2dz8COCG9YWWn6lFhjrRNGiQgIrklmWSTb2ajgU8AD6Q5nqw2eUR0r82S\nDUo2IpJbkkk23wEeAmrd/Tkzmwy8nd6wstOw0gIqiuOatkZEck5+Txu4+++B3ycsLwXOTmdQ2crM\nwog09WxEJLf0mGzM7LpOircDz7v7vakPKbtNHlnGE29tzHQYIiL9KpnTaEXAwUSnzt4GDgLGAReb\n2TVpjC0rVYcJOTfVNWY6FBGRfpNMsjkIONbdf+buPyMaiXYA8DHgpHQGl41O+MAoAO6YvyLDkYiI\n9J9kks1QoCxhuRQY5u6tgP4830tTq4Zw/AGjuPWZd2hoas10OCIi/SKZZPMj4CUz+18z+zXwIvBj\nMysFHklncNnq8x+uZkt9E3cvXJnpUERE+kWPycbdbwbmAH8C/ggc7e43uXu9u/9bugPMRodNHMrM\nCZX86m/LaGlty3Q4IiJpl+zcaLuBtcBWYIqZHZO+kLKfmfH5Y6pZsWUXf3l9XabDERFJu2TmRrsE\neJLoxs6rws9vpzes7HfitComjyjlhieW4q6JOUUkuyXTs7kcOAxY7u7HAjOBbT3tZGZFZrbAzF42\ns9fN7KpQPsnMnjWzWjP7nZkVhPLCsFwb1k9MONbXQ/liMzs5ofyUUFZrZlcklHdax0ASyzMumjOR\nV1dv14wCIpL1kkk2u919N0QJwd3fBPZPYr9G4Dh3n0F0n84pZjYb+G/ganefQnRa7uKw/cXA1lB+\nddgOM5sGnAdMB04BfmFmMTOLAT8HTgWmAeeHbemmjgFlUpgrbUt9U4YjERFJr2SSzSozqyQaIPCw\nmd0LLO9pJ4+0z8sSDy8HjgPuDuW3Ah8N788Ky4T1x5uZhfI73b3R3ZcBtcDh4VXr7kvdvQm4Ezgr\n7NNVHQNKRXEcgB0NzRmOREQkvZKZG+1j4e23zexxoAJ4MJmDh97HQmAKUS9kCbDN3VvCJquAseH9\nWGBlqLPFzLYDw0P5/ITDJu6zskP5EWGfruroGN+lwKUAVVVV1NTUJNOs96mrq+vVvuvro5Fo8194\nhdj6eK/qzqTetnswy8U2Q262W21OrWTmRrvd3T8N4O5PtJcBn+5p33Dj58GhZ/RHopkHBgx3vxG4\nEWDWrFk+d+7cXh2npqaG3uy7pb6Jr/3tYcZMnMLcoyb1qu5M6m27B7NcbDPkZrvV5tRK5jTa9MSF\n0Fs5dG8qcfdtwOPAkUClmbUnuXHA6vB+NTA+1JFP1IPanFjeYZ+uyjd3U8eAUl4UhbijoaWHLUVE\nBrcuk00YAbYTOMjMdoTXTmAD0ONsz2Y2MvRoMLNi4ETgDaKkc07Y7KKEY90XlgnrH/NoTPB9wHlh\ntNokYCqwAHgOmBpGnhUQDSK4L+zTVR0DSn4sj7LCfLbrmo2IZLkuT6O5+w+AH5jZD9z967049mjg\n1tATygPucvcHzGwRcKeZ/RfR1Dc3h+1vBm43s1pgC1HywN1fN7O7gEVAC3BZOD2HmX2B6L6fGHCL\nu78ejvW1LuoYcMqLlGxEJPslM0CgN4kGd3+F6J6cjuVLiUaSdSzfDZzbxbG+B3yvk/J5wLxk6xiI\nyovj7NitZCMi2S3Z6WokTcqL4+rZiEjWU7LJsIriuO6zEZGs1+NpNNgzAq0qcXt319O/UqCiOM7r\nSjYikuWSuc/mX4ArgfVA+3z4TvQET+mj8iKdRhOR7JdMz+ZyYH9335zuYHJRRXGc+qZWWlrbyI/p\nrKaIZKdkfrutBLanO5BcVVEcbuzcrRs7RSR7JdOzWQrUmNmfiWZyBsDdf5q2qHJIeZiMc3tDM8NK\nB9yTEEREUiKZZLMivArCS1JIMz+LSC5I5qbOq/ojkFxVkdCzERHJVl0mGzO7xt2/ZGb3E40+ew93\nPzOtkeWIciUbEckB3fVsbg8/f9IfgeSqPafRNGWNiGSx7ibiXBh+PtF/4eQenUYTkVygGzsyrDA/\nj4JYnp5pIyJZTckmw8xMk3GKSNbrMdmY2YH9EUguqyjO19BnEclqyfRsfmFmC8zsn82sIu0R5SA9\n00ZEsl2PycbdPwRcAIwHFprZb8zsxLRHlkMqdBpNRLJcUtds3P1t4JtEj1v+MHCdmb1pZh9PZ3C5\nQslGRLJdMtdsDjKzq4E3gOOAM9z9A+H91WmOLyeUF+kBaiKS3ZKZG+1nwE3AN9y9ob3Q3deY2TfT\nFlkOqSiOs2N3C+6OmWU6HBGRlEsm2ZwGNLh7K4CZ5QFF7r7L3W/vfldJRnlxPq1tTn1TK2WFST08\nVURkUEnmms0jQHHCckkokxTRLAIiku2SSTZF7l7XvhDel6QvpNyzJ9nsUrIRkeyUTLKpN7ND2hfM\n7FCgoZvtZS+VF2kyThHJbslcIPgS8HszWwMYsA/wybRGlWP0mAERyXbJPDztOTM7ANg/FC12d/1W\nTCFdsxGRbJfs0KfDgIlh+0PMDHe/LW1R5ZhyPRpaRLJcj8nGzG4HqoGXgNZQ7ICSTYoMKczHTMlG\nRLJXMj2bWcA0d3/fo6ElNfLyjPIiTVkjItkrmdForxENCpA0Ki/OZ8duPUBNRLJTMj2bEcAiM1sA\nNLYXuvuZaYsqB2kyThHJZskkm2+nOwgJ86Mp2YhIlkpm6PMTZrYvMNXdHzGzEiCW/tByS3lRnNod\ndT1vKCIyCCXziIHPAXcDN4SiscCf0hlULtJpNBHJZskMELgMOArYAXsepDYqnUHlogo9GlpEslgy\nyabR3ZvaF8wsn+g+m26Z2Xgze9zMFpnZ62Z2eSgfZmYPm9nb4efQUG5mdp2Z1ZrZKx3mY7sobP+2\nmV2UUH6omb0a9rnOwsNguqpjICsvjrO7uY3GltaeNxYRGWSSSTZPmNk3gGIzOxH4PXB/Evu1AF91\n92nAbOAyM5sGXAE86u5TgUfDMsCpwNTwuhS4HqLEAVwJHAEcDlyZkDyuBz6XsN8pobyrOgYszY8m\nItksmWRzBbAReBX4PDAP6PEJne6+1t1fCO93Ej1WeixwFnBr2OxW4KPh/VnAbR6ZD1Sa2WjgZOBh\nd9/i7luBh4FTwrpyd58fbji9rcOxOqtjwKrYM2WN7rURkeyTzGi0NuBX4dUrZjYRmAk8C1S5+9qw\nah1QFd6PBVYm7LYqlHVXvqqTcrqpo2NclxL1oqiqqqKmpmbvGhbU1dX1et92yzdGSeahJ59l1YjB\nMdgvFe0ebHKxzZCb7VabUyuZudGW0ck1GnefnEwFZlYG3AN8yd13hMsq7cdwM0vrNDjd1eHuNwI3\nAsyaNcvnzp3bqzpqamro7b7tDt3dzK9ef4xXGiq4bO6sPh2rv6Si3YNNLrYZcrPdanNqJXMabRbR\nrM+HAR8CrgP+L5mDm1mcKNHc4e5/CMXrwykwws8NoXw1MD5h93GhrLvycZ2Ud1fHgDWkKM5n5kzk\nodfX89b6nZkOR0QkpXpMNu6+OeG12t2vAU7rab8wMuxm4A13/2nCqvuA9hFlFwH3JpRfGEalzQa2\nh1NhDwEnmdnQMDDgJOChsG6Hmc0OdV3Y4Vid1TGgffaoSZQUxPj547WZDkVEJKWSOY12SMJiHlFP\nJ5lpbo4CPg28amYvhbJvAD8E7jKzi4HlwCfCunnAR4BaYBfwWQB332Jm3wWeC9t9x923hPf/DPwa\nKAYeDC+6qWNAG1pawKdm78tNf1vKl0/Yj4kjSjMdkohISiSTNP4n4X0L8A5J/PJ296eIHiPdmeM7\n2d6JbiDt7Fi3ALd0Uv488MFOyjd3VsdgcMnRk/j10+9wfc0S/vucgzIdjohISiQzGu3Y/ghEIqPK\nizjvsPH8dsEKqkeVcuDYSqaPLae8KJ7p0EREei2Z02hf6W59h+sxkgL/NLeav9du4vvz3gQgz+Dy\n4/fji8dPIXE0n4jIYJHskzoPI7roDnAGsAB4O11B5brRFcU8+tW5bKpr5PU1O7h74SqufuQt1u/c\nzXfP+iCxPCUcERlckkk244BDwiwAmNm3gT+7+6fSGZjAiLJCPrzfSI6ZOoJxQ4u5vmYJm+saufa8\nmRTFB8eNnyIikNx9NlVAU8JyE13ckS/pYWZ87ZQDuPKMafx10XrOvv5plm2qz3RYIiJJSybZ3AYs\nMLNvh17Ns7w775j0o88eNYmbL5rF6m0NnH7d37j3pdU97yQiMgAkc1Pn94juedkaXp919++nOzDp\n3HEHVDHvix/iA6PLufzOl/j+vDeIRo2LiAxcyfRsAEqAHe5+LbDKzCalMSbpwZjKYu68dDYXHrkv\nNz65lG/d+zptbUo4IjJwJTP0+UqiEWn7A/8LxInmRjsqvaFJd/JjeVx15nSKC2Lc8MRSGlta+cHH\nD9JINREZkJIZjfYxoscDtD+bZo2ZDUlrVJIUM+OKUw6gKD/GtY++Te2GOv7xw9Wc8IEq8pR0RGQA\nSeY0WlOYSsYBzEwTdg0gZsaXT9yPH519EBt2NnLp7Qs54adP8MySzZkOTURkj2SSzV1mdgPRkzM/\nBzxCHx6kJunxicPGU/Ovc/nZ+TPZ3dzK9+e9kemQRET2SGZutJ+Y2YnADqLrNt9y94fTHpnstfxY\nHmfMGMPSjfVc8+hbbNvVRGVJQabDEhHpvmdjZjEze9zdH3b3f3P3f1WiGfiOmjIcd3QqTUQGjG6T\njbu3Am1mVtFP8UgKzBhfSWlBjL8v2ZTpUEREgORGo9URPQDtYWDPHCnu/sW0RSV9Eo/lccTk4Txd\nq56NiAwMySSbP4SXDCJzqofz2JsbWLOtgTGVxZkOR0RyXJfJxswmuPsKd9c8aIPQUVNGAPD32k2c\nO2t8hqMRkVzX3TWbP7W/MbN7+iEWSaH9q4YwoqyApzVIQEQGgO6STeIt6JPTHYikVl6ecWT1CJ6q\n3aSJOkUk47pLNt7FexkkjqoezsadjdRuqMt0KCKS47obIDDDzHYQ9XCKw3vCsrt7edqjkz5JvG4z\ntUrT2YlI5nSZbNxdzx0e5MYPK2HCsBJuemoZu5pbOWnaPkwZVZbpsEQkByUz9FkGsavOnM7Vj7zF\nj/6ymB/9ZTHjhhZz8PhKDh5fyZzqEUwbow6qiKSfkk2WO/aAURx7wCjWbGvgkTfWM3/pZl5csY0H\nXlkLwDH7jeSLx01h1sRhGY5URLKZkk2OGFNZzIVHTuTCIycCsGHnbu5ZuJqb/raUc375DDPGVzJ7\n8jBmjh/KYROHMrysMLMBi0hWUbLJUaOGFPFPc6v5zJyJ/GbBCu57eQ23PLWM5tallBbEuO9fjqZ6\npK7viEhqJPM8G8lixQUxLj56EvdedhSvfvtk7vr8kcTyjK/f8yptbRrxLiKpoWQjexTFYxw+aRjf\nPG0aC97Zwm8WrMh0SCKSJZRs5H3OnTWOo6YM54cPvsna7Q2ZDkdEsoCSjbyPmfGDjx1Ea5vzzT++\npuluRKTPlGykUxOGl/DlE6fy6JsbeHHltkyHIyKDnJKNdOn8wydQFM/jDy+synQoIjLIKdlIl4YU\nxTl5+j7c//JaGltaMx2OiAxiSjbSrY8fMo7tDc089saGTIciIoOYko106+gpIxg1pJB7Xlid6VBE\nZBBLW7Ixs1vMbIOZvZZQNszMHjazt8PPoaHczOw6M6s1s1fM7JCEfS4K279tZhcllB9qZq+Gfa4z\nM+uuDumdWJ7xsZljqVm8gc11jZkOR0QGqXT2bH4NnNKh7ArgUXefCjwalgFOBaaG16XA9RAlDuBK\n4AjgcODKhORxPfC5hP1O6aEO6aWPHzKOljbn/pfXZDoUERmk0pZs3P1JYEuH4rOAW8P7W4GPJpTf\n5pH5QKWZjQZOBh529y3uvhV4GDglrCt39/ke3QRyW4djdVaH9NL++wxh+phynUoTkV7r74k4q9x9\nbXi/DqgK78cCKxO2WxXKuitf1Ul5d3W8j5ldStSToqqqipqamr1sTqSurq7X+w4WM8qb+c2bTZx7\nzV84Zmw+00fE2FVfn/Xt7igXvuvO5GK71ebUytisz+7uZpbWW9N7qsPdbwRuBJg1a5bPnTu3V/XU\n1NTQ230Hi9nNrRT9ZTF/eHEVz61rZNzQYi4/sITTs7zdHeXCd92ZXGy32pxa/T0abX04BUb42T6e\ndjUwPmG7caGsu/JxnZR3V4f0QVE8xrfOmMaz3zien50/kzXbGnhyVUumwxKRQaK/k819QPuIsouA\nexPKLwyj0mYD28OpsIeAk8x9gJnFAAAQL0lEQVRsaBgYcBLwUFi3w8xmh1FoF3Y4Vmd1SAoU5sc4\nY8YY5lSPYMG6Fs2bJiJJSefQ598CzwD7m9kqM7sY+CFwopm9DZwQlgHmAUuBWuBXwD8DuPsW4LvA\nc+H1nVBG2OamsM8S4MFQ3lUdkkKnHTSa9bucRWt3ZDoUERkE0nbNxt3P72LV8Z1s68BlXRznFuCW\nTsqfBz7YSfnmzuqQ1Dp5+j78xx9f5c+vrGX6mIpMhyMiA5xmEJBeGVZawLRhMR54Za1OpYlIj5Rs\npNcOGx1jxZZdvLZap9JEpHtKNtJrh47KJz/PeOBVzSwgIt1TspFeKyswjp46gj/rVJqI9EDJRvrk\ntANHs2prA1fdv4gHX13Lmm0NmQ5JRAagjM0gINnh1ANHc+9La7jj2eX8+ul3APiX46bw1ZP2z2xg\nIjKgKNlIn5QV5vN/lxzB7uZWFq/bya1Pv8PPHqtleGkBnzlqUqbDE5EBQslGUqIoHmPG+Ep+fO4M\n6hpbuOqBRYwYUsjpB43JdGgiMgDomo2kVCzPuO78mczadyhf+d3L/PSvi3l6ySYamlozHZqIZJB6\nNpJyRfEYN114GP90x0J+9ngt1z1WSzxmzJwwlA/vN5Jjpo5k+phy8vIs06GKSD9RspG0qCiJ85vP\nzWZ7QzMvLN/K/GWbeertTfz4ocX8+KHFjK4o4swZYzjz4DFMG11OeKq3iGQpJRtJq4riOMceMIpj\nDxgFp8LGnY08+dZG5r26lpufWsYNTy5lSFE+I8oKGV5awIiyQvapKGKfiiKqygsZURa9JgwrobRQ\n/1xFBiv975V+NXJIIWcfOo6zDx3HlvomHnxtLW+t28mm+ia21DVRu7GOv9duYmfje5+VM6Qon/85\ndwYnTd8nQ5GLSF8o2UjGDCst4IIj9u10XV1jC+t37GbTzkY21jVywxNLufT2hfzT3Gq+euJ+5Mc0\ntkVkMFGykQGprDCfspFlVI8sA+CED1Rx1f2LuL5mCTWLNzJ1VBmVJXHKi+IUxfMozI9RVpTPIROG\nsl9Vma4BiQwwSjYyKBTFY/zg4wcya9+h3D5/Oa+s2sa2hmZ2NDTT1mFatuGlBRxZPZzPzJnIrInD\nMhOwiLyHko0MKu3Xe9q5Oy1tTmNLG5vrGnl22RbmL9lMzVsbeeCVtcypHs4Xj5/KEZOGqbcjkkFK\nNjKomRnxmBGP5VFWmM++w0v5xKzxNDS1csezy7nhyaWcd+N8po4q4+xDx/GxmWOpKi/KdNgiOUfJ\nRrJScUGMSz40mU/N3pc/vriauxeu4ocPvskPH3yTEWUFe4ZUF8XzMDPyDOKxPIriMQrz8yjIz6Mg\nFv08eHwlxx0wSj0jkT5QspGsVhSPcf7hEzj/8Aks21TPvFfXsmprA5vqGtm4s5Gtu9poc2hrc5pb\n22hsaWN3cytNLW00tUYvdzh4fCX/frJmshbpLSUbyRmTRpRy2bFT9mqfltY27l64imsffZt/uOlZ\nxpQaM1YtZPLIUsYNLWFoSZyhJQUMKYpTkJ+3p1dUlB+jMB4tq0ckomQj0q38WB7nHT6Bj84cy50L\nVnDvs4tZvG4nf120ntaOw+A6MaQon7MPGcenZu/LlFFl/RCxyMCkZCOShKJ4jM8cNYmJzcuZO3cu\nza1tbNjZyNb6JrbtaqausZnGljaaWt49FdfY0sbidTv3PFju4PGVVJbEyQvXiCDq8cRjxuSRpUwb\nXcG0MeWMriiiKB7LaHtFUk3JRqQX4rE8xlYWM7ayuMdtN9VN43fPreSxNzewtb6JNofWNqe9X9TY\n0vq+nlJpQYxhZQWUFuRTFI9RHI8xvKyAMZXFjK4oorIkTlF+jKJ4jIqSOKOGFDJySCGF+UpSMjAp\n2Yik2YiyQi47dkq314t2N7fy9vo63li3g407G9lS38TmukZ2NbWyu6WNhqYWXlu9nb8uWk9TS1uX\nxxlSmM+QonyGhJkVCNeL8gzy84xYnu0ZdVccj1GQn4cRbRbLsz0j8KLReDHyY9HQ8qXLmqmNLY3q\nKMqnojiavaGkMJ+SguhYVeVFFORrGiHpnJKNyABQFI9x4LgKDhxX0e127s7m+ibqdrfQ0NzK7uZW\ntu1qZv2O3WwISWrn7hZ27o5O67Vrc6e1zWlpdXY2t7BxZ+OeUXcOuEOrhxF5zW00trS+b2YGFr/R\nbWx5BuOHlTBxeCmVJXHisTzisTyK4zFKC2OUFuZTEOa0M4M8i5JfLC86rWjhtCJhXV74aRbdT9We\nFI1on6ElcYaXFVBZUkBJQYyi/JiekTSAKdmIDCJmtuceoXRrDcPBm1vbeOqppzj66KNpa4Odjc3s\naGhhe0MzDc0t7GpqZVdjK6u2NbBsUz3LNtWxfHM9za1OU2sbu5taqWtqwXseT9FniT2zovw8xlQW\ns+/wUiaNKGH8sBLGDS1mbGUJJYWxPQlNSap/KNmISKeiXkd0Xag43xhSFAeiB+MxdO+O5e40NLfS\n3OK0X61qbXNaQ48rsRfVFhbae2PtPS/3d983t7axbVczW3Y1sW1XEw1NrexubqNhzz1SrexqamX1\n1gaeqt3IPS80dtvOoSVxhpUWMLaymCmjypgyqoy1G1rwxRvIMyOeZ2Foe4zigjyGFMUZUpRPcTym\noe1JUrIRkbQzM0oK8qEgM/U3NLWyetsuVm5tYPXWBnY3t+45dVi3u4XN9Y1sqmti5ZZd/H3J5nev\ni73wXI/H3nNK0Cxc48ojHrP3nfpr3664IDqlWFYYXesqLsinOB71xuKxaOaKvLxo3zwz8vKiZJcf\njpsf3ufn2Z71e+oJM2GUh2RYUpAfnaK06Hrc0NI4ZYX5GUmQSjYikvWKC2JMGTWEKaOG9Lhta5uz\nemsDjzz1DDNnHkJb6Ek1haHtu5pb2bm7mZ27W9jV2LKnt9XS5rSE047Nbf5ub8zB8T3JraGplfqm\nVuobW9ha30xDcyu7mlpobvU99bhHPbvoldrPoiCWR0VJnIKQvOKxPG6+6DAmDC9JbUUdKNmIiCSI\n5RkThpcwuSLGzAl7eb4wDdpnNm9pdZrb2mhpjZJa+ynI9sTUfk2ssaVtTzKsD9fK2txpamlj664m\nttQ3s72hiaYWp6UtSo6F8fSPIlSyEREZwN6d2RyKGbz3UWlQvIiIpJ2SjYiIpJ2SjYiIpF3WJhsz\nO8XMFptZrZldkel4RERyWVYmGzOLAT8HTgWmAeeb2bTMRiUikruyMtkAhwO17r7U3ZuAO4GzMhyT\niEjOMu+PCYv6mZmdA5zi7peE5U8DR7j7FzpsdylwKUBVVdWhd955Z6/qq6uro6ws9x6MlYvtzsU2\nQ262W21OzrHHHrvQ3Wf1tF1O32fj7jcCNwLMmjXL586d26vj1NTU0Nt9B7NcbHcuthlys91qc2pl\na7JZDYxPWB4Xyrq0cOHCTWa2vJf1jQA29XLfwSwX252LbYbcbLfanJx9k9koW0+j5QNvAccTJZnn\ngH9w99fTVN/zyXQjs00utjsX2wy52W61ObWysmfj7i1m9gXgISAG3JKuRCMiIj3LymQD4O7zgHmZ\njkNERLJ36HN/uzHTAWRILrY7F9sMudlutTmFsvKajYiIDCzq2YiISNop2YiISNop2fRRLkz4aWbj\nzexxM1tkZq+b2eWhfJiZPWxmb4efmX+sYYqZWczMXjSzB8LyJDN7NnzfvzOzgkzHmGpmVmlmd5vZ\nm2b2hpkdme3ftZl9Ofzbfs3MfmtmRdn4XZvZLWa2wcxeSyjr9Lu1yHWh/a+Y2SF9qVvJpg9yaMLP\nFuCr7j4NmA1cFtp5BfCou08FHg3L2eZy4I2E5f8Grnb3KcBW4OKMRJVe1wJ/cfcDgBlE7c/a79rM\nxgJfBGa5+weJbpc4j+z8rn8NnNKhrKvv9lRganhdClzfl4qVbPomJyb8dPe17v5CeL+T6JfPWKK2\n3ho2uxX4aGYiTA8zGwecBtwUlg04Drg7bJKNba4AjgFuBnD3JnffRpZ/10S3gRSHG8JLgLVk4Xft\n7k8CWzoUd/XdngXc5pH5QKWZje5t3Uo2fTMWWJmwvCqUZS0zmwjMBJ4Fqtx9bVi1DqjKUFjpcg3w\n70BbWB4ObHP3lrCcjd/3JGAj8L/h9OFNZlZKFn/X7r4a+AmwgijJbAcWkv3fdbuuvtuU/n5TspGk\nmVkZcA/wJXffkbjOozH0WTOO3sxOBza4+8JMx9LP8oFDgOvdfSZQT4dTZln4XQ8l+it+EjAGKOX9\np5pyQjq/WyWbvtnrCT8HKzOLEyWaO9z9D6F4fXu3OvzckKn40uAo4Ewze4fo9OhxRNcyKsOpFsjO\n73sVsMrdnw3LdxMln2z+rk8Alrn7RndvBv5A9P1n+3fdrqvvNqW/35Rs+uY5YGoYtVJAdFHxvgzH\nlHLhWsXNwBvu/tOEVfcBF4X3FwH39nds6eLuX3f3ce4+keh7fczdLwAeB84Jm2VVmwHcfR2w0sz2\nD0XHA4vI4u+a6PTZbDMrCf/W29uc1d91gq6+2/uAC8OotNnA9oTTbXtNMwj0kZl9hOjcfvuEn9/L\ncEgpZ2ZHA38DXuXd6xffILpucxcwAVgOfMLdO158HPTMbC7wr+5+uplNJurpDANeBD7l7o2ZjC/V\nzOxgokERBcBS4LNEf5hm7XdtZlcBnyQaefkicAnR9Yms+q7N7LfAXKJHCawHrgT+RCffbUi8/4/o\nlOIu4LPu/nyv61ayERGRdNNpNBERSTslGxERSTslGxERSTslGxERSTslGxERSTslG5E+MrPhZvZS\neK0zs9UJy72eKdjMPmZm/7aX+/yfmQ36Obwk++T3vImIdMfdNwMHA5jZt4E6d/9JCo77x74eQ2Sg\nUM9GJI3M7H4zWxielXJJQvnnzeyt8LyUm8zsmk72vaS9PPRYrjWzp81sqZl9LJTnmdkvwrNnHia6\nWa99/8PM7IlQ/4NmVmVm8bB8dNjmx+GGRpG0Us9GJL0uCndjlwDPm9k9QBnR5JaHEE10WQMsSOJY\no4jm7DqQ6I7vPxJNpzKJ6HlKY4imWfmlmRUSzeV2prtvMrMLgO+6+6Vm9lngTjP7EnAscGTKWivS\nBSUbkfT6spmdGd6PA6qBiURzrW0FMLO7iaYK6cmfwqy8r4QHfkH07JnfunsbsMrMakL5B4DpwCPR\nrCPEiCbZxN1fMbM7iebAOiJMPimSVko2ImliZicQJYPZ7t5gZk8BRX04ZOK8XNZT9cAr7v6hLtZ/\nkOi5LaP6EI9I0nTNRiR9KoAtIdFMBw4L5QuAY82sMjy64eN9qONJ4JPh2s1Y4MOhfBEw1swOBzCz\nghADZvZJolN5c4Gfm1l5H+oXSYp6NiLp82fgUjNbBCwmmiUbd19hZj8mekTFlrBuey/ruJvoussi\noqnynwl1NJrZOcB1IZnEgP8xs43Ad4G57r7GzG4ArgYu7mX9IknRrM8iGWBmZe5eF3o29xI9GfP+\nTMclki46jSaSGd81sxeBV4h6Ng9kOB6RtFLPRkRE0k49GxERSTslGxERSTslGxERSTslGxERSTsl\nGxERSbv/H6roYZCIlkhwAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jGosc6aayNfV",
        "colab_type": "text"
      },
      "source": [
        "**So, this was distribution of top 100 tags in all of 60M questions.**\n",
        "\n",
        "> As we can see, most of the tags are used negligibly in the dataset. thus some tags are very highly probable to occur and some are very low probable.\n",
        "\n",
        "This gives our dataset some bais, which hopefully is avoidable."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kMOyaSYjy5UF",
        "colab_type": "code",
        "outputId": "8f501c56-f09e-4c21-c1ca-bcda3223f220",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        }
      },
      "source": [
        "# Plotting no of tags in questions\n",
        "\n",
        "# counting the no of tags in each question\n",
        "df[\"Tag_Counts\"] = df.dropna().Tags.str.split().apply(lambda x: len(x))\n",
        "tag_counts =  df.dropna()[\"Tag_Counts\"].tolist()\n",
        "\n",
        "# dropping unnecessary columns\n",
        "df.drop(columns = [\"Tag_Counts\"], inplace=True)\n",
        "\n",
        "# plotting using seaborn\n",
        "import seaborn as sbn\n",
        "sbn.countplot(tag_counts, palette='gist_rainbow')\n",
        "plt.title(\"Distribution of Number of tags \")\n",
        "plt.xlabel(\"No. of Tags\")\n",
        "plt.ylabel(\"No. of Questions\")\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAEWCAYAAADPZygPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xu8HVV99/HP1wSQcsekFBMgUaKP\nkWrEiPTx8qAoBLSEtsjlpRKVGqlgtaIIagtFUJF6Ka2isUQCKhFBSlQUI6BoLZegkavIAYIkBhIS\nTFAUDXyfP2YdMtmcy05y9p6Q/X2/XvM6s3+zZs3acy6/M2vWniXbRERENOFpTTcgIiJ6V5JQREQ0\nJkkoIiIakyQUERGNSRKKiIjGJAlFRERjkoSiqyR9XtI/j1Bdu0v6raRR5fUPJP39SNRd6vuOpBkj\nVd96HPd0SQ9Kur/bxx6IJEvas6FjP1fSQkkPS/rHJtoQnZUkFCNG0iJJvy9/MH4j6SeSjpX0xM+Z\n7WNtf6TNul4zVBnbv7K9re3HRqDtp0r6ckv9B9mes7F1r2c7dgdOACbb/osBtu9XksLnWuI/lvSW\nLjWzm04Erra9ne2zWzeO9D8e0X1JQjHS/tr2dsAewMeBDwDnjvRBJI0e6To3EbsDK2wvG6LM74A3\nS5rQlRaNkA38nu0B3DrSbYlNR5JQdITtVbbnAUcAMyTtBSDpPEmnl/Uxkr5VrppWSvqRpKdJuoDq\nj/E3S3fbiZImlCuAYyT9CriqFqv/cXu2pOslrZZ0maSdy7H2k7S43sb+qy1J04APAkeU4/28bH/i\nv+zSrg9LulfSMknnS9qhbOtvxwxJvypdaR8a7NxI2qHsv7zU9+FS/2uA+cAzSzvOG6SK3wDnAacM\nUv86V3Wt56m8r9PLlepvJX1T0jMkfaWctxsGSHAHS7q7vLez6le3kt4m6XZJD0m6QtIetW2WdJyk\nO4E7B2nvIZJuLT8HP5D0vBK/CngV8J+lnc9p2e8M4BW17f9Z4v8u6b7yXm6U9IraPltLmlPaenv5\n2Vpc2/4BSUvK1fwdkvYf5HsQI8V2liwjsgCLgNcMEP8V8A9l/Tzg9LL+MeDzwBZleQWggeoCJgAG\nzge2AbauxUaXMj8AlgB7lTKXAF8u2/YDFg/WXuDU/rK17T8A/r6svw3oA54FbAt8A7igpW1fLO16\nIfAo8LxBztP5wGXAdmXfXwLHDNbOln33AxYDfwGsBp5b4j8G3jLQexnkPPUBzwZ2AG4rbXgNMLq0\n70u1/Q1cDexM9c/BL2vnZXqp63ll3w8DP2nZd37Zd+sB3s9zqK7sXlt+Bk4s9W3Z+j0Y5Hw8aTvw\nJuAZpT0nAPcDTy/bPg78ENgJGA/c1H++gecC9wHPrJ23Zzf9e7W5L7kSim74NdUfoVZ/AnYF9rD9\nJ9s/cvntH8Kptn9n+/eDbL/A9i22fwf8M3C4ysCFjfRG4FO277b9W+Bk4MiWq7B/tf172z8Hfk6V\njNZR2nIkcLLth20vAj4JvHl9GmP7fqoEftoGvZsqydxlexXwHeAu29+3vQb4OvCilvJn2l5p+1fA\nZ4CjSvxY4GO2by/7fhSYUr8aKttXDvI9OwL4tu35tv8E/BtVIv+/G/i+sP1l2ytsr7H9SWArqgQD\ncDjwUdsP2V4M1O8zPVbKTpa0he1Ftu/a0HZEe5KEohvGASsHiJ9F9V/v90pXz0lt1HXfemy/l+q/\n6zFttXJozyz11eseDexSi9VHsz1CdcXUakxpU2td4zagTWcCB0p6UrJrwwO19d8P8Lq17a3n9Zll\nfQ/g30tX2m+ovs9i3fcz1PdsnfNq+/FSfkPOBwCS3le62laVNu3A2p+BZ7a054l1233Ae6iuJJdJ\nmivpmURHJQlFR0l6CdUflB+3bitXAifYfhZwCPDeWh/8YFdEw10p7VZb353qautBqi6fP6u1axQw\ndj3q/TXVH9x63WtY9493Ox4sbWqta8l61oPtFVRXJa2jDdd5r1Rddxur9bz+uqzfB7zD9o61ZWvb\nP6k3dYh61zmvklSO1e75WKfucv/nRKornp1s7wisokqMAEupuuEGel/Y/qrtl5c2mSrRRwclCUVH\nSNpe0uuBuVT3J24eoMzrJe1Z/vCsouoOebxsfoDq/sv6epOkyZL+jKqr6mJXQ7h/CTxd0uskbUF1\n72Kr2n4PABPqN9xbXAj8k6SJkral6nb6WumCaltpy0XAGZK2K91W7wW+PPSeg/oUVdfV82qxhcAr\nVX2OageqrsON9X5JO0naDXg38LUS/zxwsqTnwxODLt6wHvVeBLxO0v7l+3IC1f20nwy92xNaf062\no/rnYDkwWtK/ANu3HO/k8l7GAcf3b1D1maRXS9oK+APVFeHjREclCcVI+6akh6n+Q/4Q1R/Jtw5S\ndhLwfeC3wP8Cn7N9ddn2MeDDpZvnfetx/AuoBj/cDzwd+EeoRusB7wT+i+q/7N9R3eDv9/XydYWk\nnw5Q7+xS9zXAPVR/pN61Hu2qe1c5/t1UV4hfLfWvN9urgU9Qu+dmez5VkrgJuBH41ga2s+6yUtdC\n4NuUYfe2L6W6WpgraTVwC3DQerT/DqqBBP9BdZX411TD/P/YZhX/DhxWRrudDVwBfJfqn457qb5P\n9e6306i+7/dQ/exdTJX0oPqn5OOlHfcDf87IJPAYQv9IpIiIniPpH4Ajbf+/ptvSq3IlFBE9Q9Ku\nkl5WPpf1XKruv0ubblcv21w/dR4RMZAtgS8AE6k+9DsX+NyQe0RHpTsuIiIak+64iIhoTLrjhjFm\nzBhPmDCh6WZERDyl3HjjjQ/aHjtcuSShYUyYMIEFCxY03YyIiKcUSfcOXyrdcRER0aAkoYiIaEyS\nUERENKZjSUjSbFWTf91Si31N1XzxC1VNKLawxCeomha6f9vna/u8WNLNkvoknV2eM4aknSXNl3Rn\n+bpTiauU65N0k6S9a3XNKOXvlDSjU+89IiLa08krofOAafWA7SNsT7E9hWrCsW/UNt/Vv832sbX4\nOcDbqZ4zNqlW50nAlbYnAVeW11A9t6q/7MyyP6pm2DwFeCmwD3BKf+KKiIhmdCwJ2b6GgeeQ6X9c\n++FUTyYelKRdge1tX1smOzsfOLRsng7MKetzWuLnu3ItsGOp50Bgfplc6yGq2R7XSZIREdFdTd0T\negXwgO36nPMTJf1M0g9rc8KPY90nHS9m7WRXu9heWtbvZ+3kYuNY96m5/fsMFn8SSTMlLZC0YPny\n5ev51iIiol1NJaGjWPcqaCmwu+0XUc2t8lVJ2w+45wDKVdKIPX/I9izbU21PHTt22M9aRUTEBup6\nEpI0Gvhb1k6Khe1HyyyR2L4RuAt4DtW8L/VZEMezdsbFB0o3W3+33bISX8K6syX27zNYPCIiGtLE\nExNeA/zC9hPdbJLGAittPybpWVSDCu62vVLSakn7AtcBR1NNfgUwD5hBNQnVDKpJt/rjx0uaSzUI\nYZXtpZKuAD5aG4xwAJmwKjpg4qJPNN2EjrhnwolNNyE2Qx1LQpIuBPYDxkhaDJxi+1zgSJ48IOGV\nwGmS/kQ1ne6xtvsHNbyTaqTd1sB3ygJV8rlI0jFUMygeXuKXAwcDfcAjlFk9S0L7CHBDKXda7RgR\nEdGATOUwjKlTpzrPjov1kSuhCJB0o+2pw5XLExMiIqIxSUIREdGYJKGIiGhMklBERDQmSSgiIhqT\nJBQREY1JEoqIiMYkCUVERGOShCIiojFJQhER0ZgkoYiIaEwTT9GOzdDMRRObbkJHzJpwT9NNiNis\n5UooIiIakyQUERGNSRKKiIjGJAlFRERjkoQiIqIxSUIREdGYJKGIiGhMklBERDQmSSgiIhrTsSQk\nabakZZJuqcVOlbRE0sKyHFzbdrKkPkl3SDqwFp9WYn2STqrFJ0q6rsS/JmnLEt+qvO4r2ycMd4yI\niGhGJ6+EzgOmDRD/tO0pZbkcQNJk4Ejg+WWfz0kaJWkU8FngIGAycFQpC3BmqWtP4CHgmBI/Bnio\nxD9dyg16jBF+zxERsR46loRsXwOsbLP4dGCu7Udt3wP0AfuUpc/23bb/CMwFpksS8Grg4rL/HODQ\nWl1zyvrFwP6l/GDHiIiIhjRxT+h4STeV7rqdSmwccF+tzOISGyz+DOA3tte0xNepq2xfVcoPVteT\nSJopaYGkBcuXL9+wdxkREcPqdhI6B3g2MAVYCnyyy8dvi+1Ztqfanjp27NimmxMRsdnqahKy/YDt\nx2w/DnyRtd1hS4DdakXHl9hg8RXAjpJGt8TXqats36GUH6yuiIhoSFeTkKRday//BugfOTcPOLKM\nbJsITAKuB24AJpWRcFtSDSyYZ9vA1cBhZf8ZwGW1umaU9cOAq0r5wY4REREN6dikdpIuBPYDxkha\nDJwC7CdpCmBgEfAOANu3SroIuA1YAxxn+7FSz/HAFcAoYLbtW8shPgDMlXQ68DPg3BI/F7hAUh/V\nwIgjhztGREQ0o2NJyPZRA4TPHSDWX/4M4IwB4pcDlw8Qv5sBRrfZ/gPwhvU5RkRENCNPTIiIiMYk\nCUVERGOShCIiojFJQhER0ZgkoYiIaEySUERENCZJKCIiGpMkFBERjUkSioiIxiQJRUREY5KEIiKi\nMUlCERHRmCShiIhoTJJQREQ0JkkoIiIakyQUERGNSRKKiIjGJAlFRERjkoQiIqIxSUIREdGYJKGI\niGhMx5KQpNmSlkm6pRY7S9IvJN0k6VJJO5b4BEm/l7SwLJ+v7fNiSTdL6pN0tiSV+M6S5ku6s3zd\nqcRVyvWV4+xdq2tGKX+npBmdeu8REdGeTl4JnQdMa4nNB/ay/QLgl8DJtW132Z5SlmNr8XOAtwOT\nytJf50nAlbYnAVeW1wAH1crOLPsjaWfgFOClwD7AKf2JKyIimtGxJGT7GmBlS+x7tteUl9cC44eq\nQ9KuwPa2r7Vt4Hzg0LJ5OjCnrM9piZ/vyrXAjqWeA4H5tlfafogqIbYmyYiI6KIm7wm9DfhO7fVE\nST+T9ENJryixccDiWpnFJQawi+2lZf1+YJfaPvcNsM9g8SeRNFPSAkkLli9fvp5vKyIi2tVIEpL0\nIWAN8JUSWgrsbvtFwHuBr0ravt36ylWSR6p9tmfZnmp76tixY0eq2oiIaDFsEpL0CUnbS9pC0pWS\nlkt604YeUNJbgNcDbyzJA9uP2l5R1m8E7gKeAyxh3S678SUG8EDpZuvvtltW4kuA3QbYZ7B4REQ0\npJ0roQNsr6ZKHIuAPYH3b8jBJE0DTgQOsf1ILT5W0qiy/iyqQQV3l+621ZL2LaPijgYuK7vNA/pH\nuM1oiR9dRsntC6wq9VwBHCBppzIg4YASi4iIhoxejzKvA75ue1UZJT0kSRcC+wFjJC2mGpl2MrAV\nML/UcW0ZCfdK4DRJfwIeB4613T+o4Z1UI+22prqH1H8f6ePARZKOAe4FDi/xy4GDgT7gEeCtALZX\nSvoIcEMpd1rtGBER0YB2ktC3JP0C+D3wD5LGAn8YbifbRw0QPneQspcAlwyybQGw1wDxFcD+A8QN\nHDdIXbOB2YO3OiIiumnY7jjbJwH/F5hq+0/A76iGQUdERGyUdq6EAP4PMEFSvfz5HWhPRET0kGGT\nkKQLgGcDC4HHSrj/g6MREREbrJ0roanA5P7h1BERESOlnSHatwB/0emGRERE72nnSmgMcJuk64FH\n+4O2D+lYqyJiszDx3xY13YQRd8/7JjTdhM1KO0no1E43IiIietOwScj2DyXtArykhK63vWyofSIi\nItrRzrPjDgeuB95A9VSC6yQd1umGRUTE5q+d7rgPAS/pv/opT0z4PnBxJxsWERGbv3ZGxz2tpftt\nRZv7RUREDKmdK6HvSroCuLC8PoLqIaEREREbpZ2BCe+X9HfAy0polu1LO9usiIjoBW09O26op1xH\nRERsqEGTkKQf2365pIdZd+psUc2Y0Pb02xEREQMZNAnZfnn5ul33mhMREb2knc8JXdBOLCIiYn21\nM9T6+fUXZU6hF3emORER0UsGTUKSTi73g14gaXVZHgYeAC7rWgsjImKzNWgSsv2xcj/oLNvbl2U7\n28+wfXIX2xgREZupdrrjviVpGwBJb5L0KUl7dLhdERHRA9pJQucAj0h6IXACcBdtTu0tabakZZJu\nqcV2ljRf0p3l604lLklnS+qTdJOkvWv7zCjl75Q0oxZ/saSbyz5nS9KGHiMiIrqvnSS0pkztPR34\nT9ufBdodtn0eMK0ldhJwpe1JwJXlNcBBwKSyzKRKfkjaGTgFeCmwD3BKf1IpZd5e22/ahhwjIiKa\n0U4SeljSycCbgW9LehqwRTuV274GWNkSng7MKetzgENr8fNduRbYUdKuwIHAfNsrbT8EzAemlW3b\n2762JMnzW+pan2NEREQD2klCR1BN6/022/cD44GzNuKYu9heWtbvB3Yp6+OA+2rlFpfYUPHFA8Q3\n5BjrkDRT0gJJC5YvX74eby0iItbHsEmoJJ5LgK1K6EFgRB5gWq5gPGzBLh/D9izbU21PHTt2bIda\nFhER7Twx4e1UE9h9oYTGAf+9Ecd8oL8LrHztn6toCbBbrdz4EhsqPn6A+IYcIyIiGtBOd9xxVNM4\nrAawfSfw5xtxzHlA/wi3Gaz94Os84Ogygm1fYFXpUrsCOEDSTmVAwgHAFWXbakn7llFxR7fUtT7H\niIiIBrQzlcOjtv9YRj/3P7anre4tSRcC+wFjJC2mGuX2ceAiSccA9wKHl+KXAwcDfcAjwFsBbK+U\n9BHghlLuNNv9gx3eSTUCb2vgO2VhfY8RERHNaCcJ/VDSB4GtJb2W6g//N9up3PZRg2zaf4Cyprrq\nGqie2cDsAeILgL0GiK9Y32NERET3tdMddxKwHLgZeAfV1cSHO9moiIjoDe1M7/048MWyREREjJhh\nk5CkexjgHpDtZ3WkRRER0TPauSc0tbb+dOANwM6daU5ERPSSdj6suqK2LLH9GeB1XWhbRERs5trp\njqs/afppVFdG7VxBRUREDKmdZPLJ2voaYBFrP3cTERGxwdoZHfeqbjQkIiJ6z5D3hCS9SNKXJf20\nLLMk7Vm2pUsuIiI2yqBJSNLfAV8HrgLeUpZrgYsl/RXVM90iIiI22FBXM6cAr7G9qBa7SdJVwC+A\nT3WyYRERsfkbqjtudEsCAqDE7rX9wU41KiIiesNQSehPknZvDUrag2qm1YiIiI0yXHfc9yV9FLix\nxKZSPdD0A51uWEREbP4GTUK2/7s8N+4E4F0lfCtwuO2fd6Nxm7pFF09sugkdMeGwe5puQkT0iCGH\nWZdkc3SX2hIRET2mnfmEIiIiOiJJKCIiGjPUh1XPLF/f0L3mRERELxnqSuhgSQJO7lZjIiKitww1\nMOG7wEPAtpJWA6KaYVWAbW/fhfZFRMRmbNArIdvvt70j8G3b29verv51Qw8o6bmSFtaW1ZLeI+lU\nSUtq8YNr+5wsqU/SHZIOrMWnlVifpJNq8YmSrivxr0nassS3Kq/7yvYJG/o+IiJi47Uzs+p0SbtI\nen1Zxm7MAW3fYXuK7SnAi4FHgEvL5k/3b7N9OYCkycCRwPOBacDnJI2SNAr4LHAQMBk4qpQFOLPU\ntSfV1dwxJX4M8FCJf7qUi4iIhgybhMrAhOuBN1BNZne9pMNG6Pj7A3fZvneIMtOBubYftX0P0Afs\nU5Y+23fb/iMwF5he7mO9Gri47D8HOLRW15yyfjGwfykfERENaGeI9oeBl9ieYftoqj/+/zxCxz8S\nuLD2+nhJN0maLWmnEhsH3Fcrs7jEBos/A/iN7TUt8XXqKttXlfLrkDRT0gJJC5YvX74x7y8iIobQ\nThJ6mu1ltdcr2txvSOU+zSFUcxYBnAM8G5gCLGXdacW7yvYs21NtTx07dqN6HyMiYgjtzI76XUlX\nsPaK5Qjg8hE49kHAT20/AND/FUDSF4FvlZdLgN1q+40vMQaJrwB2lDS6XO3Uy/fXtbjMDLtDKR8R\nEQ1oZ2DC+4EvAC8oyyzbI/EU7aOodcVJ2rW27W+AW8r6PODIMrJtIjCJ6h7VDcCkMhJuS6quvXm2\nDVwN9N+3mgFcVqtrRlk/DLiqlI+IiAa0cyWE7W8A3xipg0raBngt8I5a+BOSplB9FmlR/zbbt0q6\nCLgNWAMcZ/uxUs/xVNOMjwJm27611PUBYK6k04GfAeeW+LnABZL6gJVUiSsiIhrSVhIaabZ/R8uA\nANtvHqL8GcAZA8QvZ4CuQdt3Uw2gaI3/gWqUX0REV7194qKmmzDivnjPhI2uIw8wjYiIxiQJRURE\nYzYoCUk6dYTbERERPWhDr4RuHNFWRERET9qgJGT7myPdkIiI6D3tPDtuvKRLJS2XtEzSJZLGd6Nx\nERGxeWvnSuhLVB/y3BV4JvDNEouIiNgo7SShsba/ZHtNWc4D8kC1iIjYaO0koRWS3tQ/h4+kN5Hn\nrUVExAhoJwm9jWoeofupnm59GPDWTjYqIiJ6w7CP7SkTzh3ShbZERESPGTQJSfqXIfaz7Y90oD0R\nEdFDhroS+t0AsW2AY6gePpokFBERG2XQJGT7iZlNJW0HvJvqXtBcGpz1NCIiNh9D3hOStDPwXuCN\nwBxgb9sPdaNhERGx+RvqntBZwN8Cs4C/tP3brrUqIiJ6wlBDtE+gekLCh4FfS1pdloclre5O8yIi\nYnM21D2hzDUUEREdlUQTERGNSRKKiIjGJAlFRERjGktCkhZJulnSQkkLSmxnSfMl3Vm+7lTiknS2\npD5JN0nau1bPjFL+TkkzavEXl/r7yr4a6hgREdF9TV8Jvcr2FNtTy+uTgCttTwKuLK8BDgImlWUm\ncA488TmmU4CXAvsAp9SSyjnA22v7TRvmGBER0WVNJ6FW06k+FEv5emgtfr4r1wI7StoVOBCYb3tl\n+RDtfGBa2ba97WttGzi/pa6BjhEREV3WZBIy8D1JN0qaWWK72F5a1u8Hdinr44D7avsuLrGh4osH\niA91jCdImilpgaQFy5cv36A3FxERwxt2KocOerntJZL+HJgv6Rf1jbYtyZ1swGDHsD2L6kkRTJ06\ntaNtiIjoZY1dCdleUr4uAy6luqfzQOlKo3xdVoovAXar7T6+xIaKjx8gzhDHiIiILmskCUnapjyZ\nG0nbAAcAtwDzgP4RbjOAy8r6PODoMkpuX2BV6VK7AjhA0k5lQMIBwBVl22pJ+5ZRcUe31DXQMSIi\nosua6o7bBbi0jJoeDXzV9ncl3QBcJOkY4F6qacUBLgcOBvqARyjTi9teKekjwA2l3Gm2V5b1dwLn\nAVsD3ykLwMcHOUZERHRZI0nI9t3ACweIrwD2HyBu4LhB6poNzB4gvgDYq91jRERE921qQ7QjIqKH\nJAlFRERjkoQiIqIxSUIREdGYJKGIiGhMklBERDQmSSgiIhqTJBQREY1JEoqIiMYkCUVERGOShCIi\nojFJQhER0ZgkoYiIaEySUERENCZJKCIiGpMkFBERjUkSioiIxiQJRUREY5KEIiKiMUlCERHRmCSh\niIhoTNeTkKTdJF0t6TZJt0p6d4mfKmmJpIVlObi2z8mS+iTdIenAWnxaifVJOqkWnyjpuhL/mqQt\nS3yr8rqvbJ/QvXceERGtmrgSWgOcYHsysC9wnKTJZdunbU8py+UAZduRwPOBacDnJI2SNAr4LHAQ\nMBk4qlbPmaWuPYGHgGNK/BjgoRL/dCkXEREN6XoSsr3U9k/L+sPA7cC4IXaZDsy1/ajte4A+YJ+y\n9Nm+2/YfgbnAdEkCXg1cXPafAxxaq2tOWb8Y2L+Uj4iIBjR6T6h0h70IuK6Ejpd0k6TZknYqsXHA\nfbXdFpfYYPFnAL+xvaYlvk5dZfuqUr61XTMlLZC0YPny5Rv1HiMiYnCNJSFJ2wKXAO+xvRo4B3g2\nMAVYCnyyqbbZnmV7qu2pY8eObaoZERGbvUaSkKQtqBLQV2x/A8D2A7Yfs/048EWq7jaAJcButd3H\nl9hg8RXAjpJGt8TXqats36GUj4iIBjQxOk7AucDttj9Vi+9aK/Y3wC1lfR5wZBnZNhGYBFwP3ABM\nKiPhtqQavDDPtoGrgcPK/jOAy2p1zSjrhwFXlfIREdGA0cMXGXEvA94M3CxpYYl9kGp02xTAwCLg\nHQC2b5V0EXAb1ci642w/BiDpeOAKYBQw2/atpb4PAHMlnQ78jCrpUb5eIKkPWEmVuCIioiFdT0K2\nfwwMNCLt8iH2OQM4Y4D45QPtZ/tu1nbn1eN/AN6wPu2NiIjOyRMTIiKiMUlCERHRmCShiIhoTJJQ\nREQ0JkkoIiIakyQUERGNSRKKiIjGJAlFRERjkoQiIqIxSUIREdGYJKGIiGhMklBERDQmSSgiIhqT\nJBQREY1JEoqIiMYkCUVERGOShCIiojFJQhER0ZgkoYiIaEySUERENCZJKCIiGtOTSUjSNEl3SOqT\ndFLT7YmI6FU9l4QkjQI+CxwETAaOkjS52VZFRPSmnktCwD5An+27bf8RmAtMb7hNERE9SbabbkNX\nSToMmGb778vrNwMvtX18rcxMYGZ5+Vzgjq439MnGAA823YhNRM7FWjkXa+VcrLUpnIs9bI8drtDo\nbrTkqcb2LGBW0+2ok7TA9tSm27EpyLlYK+dirZyLtZ5K56IXu+OWALvVXo8vsYiI6LJeTEI3AJMk\nTZS0JXAkMK/hNkVE9KSe646zvUbS8cAVwChgtu1bG25WOzap7sGG5VyslXOxVs7FWk+Zc9FzAxMi\nImLT0YvdcRERsYlIEoqIiMYkCW1CJM2WtEzSLYNsl6Szy+OGbpK0d7fb2C2SdpN0taTbJN0q6d0D\nlOmJ8yHp6ZKul/Tzci7+dYAyW0n6WjkX10ma0P2WdoekUZJ+JulbA2zrmfMAIGmRpJslLZS0YIDt\nm/zvSJLQpuU8YNoQ2w8CJpVlJnBOF9rUlDXACbYnA/sCxw3weKVeOR+PAq+2/UJgCjBN0r4tZY4B\nHrK9J/Bp4Mwut7Gb3g3cPsi2XjoP/V5le8ognwva5H9HkoQ2IbavAVYOUWQ6cL4r1wI7Stq1O63r\nLttLbf+0rD9M9UdnXEuxnjgf5f39trzcoiytI4qmA3PK+sXA/pLUpSZ2jaTxwOuA/xqkSE+ch/Ww\nyf+OJAk9tYwD7qu9XsyT/zB3/Nd4AAAEFUlEQVRvdkqXyouA61o29cz5KF1QC4FlwHzbg54L22uA\nVcAzutvKrvgMcCLw+CDbe+U89DPwPUk3lseNtdrkf0eShGKTJmlb4BLgPbZXN92epth+zPYUqid8\n7CNpr6bb1G2SXg8ss31j023ZhLzc9t5U3W7HSXpl0w1aX0lCTy099cghSVtQJaCv2P7GAEV66nwA\n2P4NcDVPvnf4xLmQNBrYAVjR3dZ13MuAQyQtonr6/aslfbmlTC+chyfYXlK+LgMupZoloG6T/x1J\nEnpqmQccXUa87Aussr206UZ1QunHPxe43fanBinWE+dD0lhJO5b1rYHXAr9oKTYPmFHWDwOu8mb2\nSXTbJ9seb3sC1eO2rrL9ppZim/156CdpG0nb9a8DBwCtI2s3+d+Rnntsz6ZM0oXAfsAYSYuBU6hu\nQmP788DlwMFAH/AI8NZmWtoVLwPeDNxc7oUAfBDYHXrufOwKzCkTMj4NuMj2tySdBiywPY8qYV8g\nqY9qcMuRzTW3u3r4POwCXFrGXYwGvmr7u5KOhafO70ge2xMREY1Jd1xERDQmSSgiIhqTJBQREY1J\nEoqIiMYkCUVERGOShCJGmCRL+mTt9fskndqB45xVnqp9Vi321vJE5YWS/lh7wvLHR/r4ESMhQ7Qj\nRpikPwBLgZfYflDS+4BtbZ86wsdZBexs+7FBti8Cptp+cCSPGzGSciUUMfLWALOAf2rdIGmCpKvK\n3C5XStp9qIrKJ93PknRLuao5osTnAdsCN/bHhiNpX0n/W+bi+R9Jk0p8G0mXqJq76WJJCyRNkTRa\n0gXluLdI+sf1PRERw8kTEyI647PATZI+0RL/D2CO7TmS3gacDRw6RD1/SzWH0AuBMcANkq6xfYik\n35aHmrbrduAVttdImgacDhwBvAu43/bfSXoh8NNS/sXAGNt/CdD/6KCIkZQroYgOKE/8Ph9ovXr4\nK+CrZf0C4OXDVPVy4MLyFO0HgB8CL9nAZu0IXKJq5t5/A55fO8bc0u6fA7eWeB/w3DIz54FU0yJE\njKgkoYjO+QzVTJ/bNN2Q4gzgCtt7UV19PX2owrZXAC8AfgQcB3yh4y2MnpMkFNEhtlcCF1Elon4/\nYe1DNd9I9Qd+KD8CjiiT2o0FXglcv4FN2oG1j/F/Sy3+P8DhAJL+Ephc1sdSDV76OvAvwN4beNyI\nQSUJRXTWJ6nu5fR7F/BWSTdRPSX83QCSDilPg251KXAT8HPgKuBE2/dvYFvOBM6S9FOgPuX1fwDj\nJN1G9eT226i63nYDrilPMf8S1VPMI0ZUhmhH9Lgy+dto238oI+a+B0wq02NHdFRGx0XEtsCVJRkJ\neEcSUHRLroQiIqIxuScUERGNSRKKiIjGJAlFRERjkoQiIqIxSUIREdGY/w/FEcxtpX0pRAAAAABJ\nRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UoGcTeZC34cf",
        "colab_type": "text"
      },
      "source": [
        "**So, it can be seen that most of the question have 2 or 3 tags.**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NsJjDW2J4P7C",
        "colab_type": "text"
      },
      "source": [
        "### Concluding Data Exploration\n",
        "**We have to clean and process our data since:**\n",
        "> It has NaN tags\n",
        "\n",
        "> It has code in the Body, which is not compatible with bag of words model.\n",
        "\n",
        "> The no of tags is too large.\n",
        "\n",
        "> The no of questions is too large.\n",
        "\n",
        "</br>\n",
        "\n",
        "It can be observed that only the top 100 tags are used frequently in the dataset.\n",
        "So, we'll use top 500 tags for safety.\n",
        "\n",
        "Also, due to limited computation power, we'll use only 500,000 examples.\n",
        "\n",
        "</br>\n",
        "\n",
        "### So,we perform following operations:\n",
        "> Get rid of Id column.\n",
        "\n",
        "> Find most frequent 500 tags.\n",
        "\n",
        "> Filter questions contaning only those of top 500 tags\n",
        "\n",
        "> Sample 500,000 examples from these."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "atOeQT4u5wKR",
        "colab_type": "text"
      },
      "source": [
        "# Data Cleaning and Data Engineering \n",
        "\n",
        "Here, it is advised to clean the RAM or restart runtime."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5flAX3K0LSWd",
        "colab_type": "code",
        "outputId": "971821af-f823-4681-e9e9-8e845beb2893",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "# Dropping all NaN tags and corresponding questions\n",
        "\n",
        "import pandas as pd\n",
        "# loading data\n",
        "df = pd.read_csv(\"data/Train.csv\")\n",
        "print(\"Shape of dataset Before: \", df.shape)\n",
        "\n",
        "# dropping nans\n",
        "df.dropna(inplace=True)\n",
        "\n",
        "# creating new index\n",
        "df.reset_index(inplace=True)\n",
        "df.drop(columns=['index'], inplace=True)\n",
        "\n",
        "print(\"Shape of dataset After: \", df.shape)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Before:  (6034195, 4)\n",
            "After:  (6034187, 4)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M8shj1-47nJc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# dropping Id column\n",
        "\n",
        "df.drop(columns=[\"Id\"], inplace=True)\n",
        "\n",
        "# Marking data checkpoint\n",
        "df.to_csv(\"data/2_Train_No_NaN.csv\", index=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nOlQOrSNPyRV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# !cp data/2_Train_No_NaN.csv drive/My\\ Drive/tcs/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yU9Uj4XU8rU6",
        "colab_type": "text"
      },
      "source": [
        "#### Now we'll find most frequent 500 tags."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ncBBSnfA_7_d",
        "colab_type": "code",
        "outputId": "566418d2-ff0d-4c63-b36c-6ac08dbc5aa4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        }
      },
      "source": [
        "# Importing new dataset\n",
        "\n",
        "import pandas as pd\n",
        "df = pd.read_csv(\"data/2_Train_No_NaN.csv\")\n",
        "\n",
        "print(df.shape)\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(6034187, 3)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Title</th>\n",
              "      <th>Body</th>\n",
              "      <th>Tags</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>How to check if an uploaded file is an image w...</td>\n",
              "      <td>&lt;p&gt;I'd like to check if an uploaded file is an...</td>\n",
              "      <td>php image-processing file-upload upload mime-t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>How can I prevent firefox from closing when I ...</td>\n",
              "      <td>&lt;p&gt;In my favorite editor (vim), I regularly us...</td>\n",
              "      <td>firefox</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>R Error Invalid type (list) for variable</td>\n",
              "      <td>&lt;p&gt;I am import matlab file and construct a dat...</td>\n",
              "      <td>r matlab machine-learning</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>How do I replace special characters in a URL?</td>\n",
              "      <td>&lt;p&gt;This is probably very simple, but I simply ...</td>\n",
              "      <td>c# url encoding</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>How to modify whois contact details?</td>\n",
              "      <td>&lt;pre&gt;&lt;code&gt;function modify(.......)\\n{\\n  $mco...</td>\n",
              "      <td>php api file-get-contents</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                               Title  ...                                               Tags\n",
              "0  How to check if an uploaded file is an image w...  ...  php image-processing file-upload upload mime-t...\n",
              "1  How can I prevent firefox from closing when I ...  ...                                            firefox\n",
              "2           R Error Invalid type (list) for variable  ...                          r matlab machine-learning\n",
              "3      How do I replace special characters in a URL?  ...                                    c# url encoding\n",
              "4               How to modify whois contact details?  ...                          php api file-get-contents\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XrIfQ35YKfo8",
        "colab_type": "code",
        "outputId": "03dafad0-12eb-4266-ea18-3b6757cccae6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# list of all tags in all questions\n",
        "all_tags = [tag for slist in df[\"Tags\"].values for tag in slist.split()]\n",
        "\n",
        "print (\"Total no of tags in all questions:\", len(all_tags))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total no of tags in all questions: 17409986\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kmRzV7OMKfrA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Counting tag frequency and selecting top 500 tags\n",
        "\n",
        "import collections\n",
        "\n",
        "# counting tag frequency\n",
        "counter=collections.Counter(all_tags)\n",
        "counter = { x:y for x, y in sorted(counter.items(), key=lambda x: x[1], reverse=True) }\n",
        "\n",
        "# selecting top 500 tags\n",
        "top_tags = set(list(counter.keys())[:500])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PechIE6eKftz",
        "colab_type": "code",
        "outputId": "e6a8db4e-52e3-42e5-f80a-fda2c1fd21d3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# finding indices of questions which have only top 500 tags\n",
        "\n",
        "req_indices=[]\n",
        "\n",
        "# question's tag set must be a subset of out top tags list\n",
        "for i in range(df.shape[0]):\n",
        "  tags = set(df[\"Tags\"][i].split())\n",
        "  if tags.issubset(top_tags):\n",
        "    req_indices.append(i)\n",
        "\n",
        "# indices of all appropriate questions\n",
        "print(req_indices[:10])\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1, 3, 7, 11, 14, 17, 21, 23, 25, 31]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nqpu3MESKfw3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# sampling 500000 examples with top 500 tags only\n",
        "\n",
        "# selecting 500,000 indices\n",
        "import random\n",
        "req_indices_5L = random.sample(req_indices, k=500000)\n",
        "\n",
        "# sampling questions from dataset\n",
        "df = df.iloc[req_indices_5L, :]\n",
        "\n",
        "# creating new index\n",
        "df.reset_index(inplace=True)\n",
        "df.drop(columns=[\"index\"], inplace=True)\n",
        "\n",
        "# saving new dataset\n",
        "df.to_csv(\"data/3_Train_5L.csv\", index=True)\n",
        "# !cp 3_Train_5L.csv drive/My\\ Drive/tcs/\n",
        "\n",
        "\n",
        "# OPTIONAL\n",
        "# saving indices list\n",
        "# req_indices_df = pd.DataFrame({\"Indices\": req_indices_5L})\n",
        "# req_indices_df.to_csv(\"data/req_indices.csv\", index=False)\n",
        "# !cp data/req_indices.csv drive/My\\ Drive/tcs/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VVhhZhkbEq1z",
        "colab_type": "text"
      },
      "source": [
        "## Our Dataset Now"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lehk5uWPEopO",
        "colab_type": "code",
        "outputId": "5872e5ea-de86-4cf4-b214-fb2443218444",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        }
      },
      "source": [
        "print(df.shape)\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(500000, 3)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Title</th>\n",
              "      <th>Body</th>\n",
              "      <th>Tags</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>graph of linked tags</td>\n",
              "      <td>&lt;p&gt;Hi I have a list of lists of linked tag, li...</td>\n",
              "      <td>python graph</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>If condition for redirection in jquery?</td>\n",
              "      <td>&lt;p&gt;Edit... &lt;/p&gt;\\n\\n&lt;p&gt;Someone suggested I do i...</td>\n",
              "      <td>javascript jquery redirect</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>git for development and svn on production server</td>\n",
              "      <td>&lt;p&gt;I have an website on development using git ...</td>\n",
              "      <td>git svn deployment integration</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>How to use HTML Image Button and PHP?</td>\n",
              "      <td>&lt;p&gt;Okay, so here's the scenario. I have a simp...</td>\n",
              "      <td>php html</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Using 'LIKE' against column names in a where c...</td>\n",
              "      <td>&lt;p&gt;I started to take a look at the following q...</td>\n",
              "      <td>sql sql-server sql-server-2008 sql-server-2005...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                               Title  ...                                               Tags\n",
              "0                               graph of linked tags  ...                                       python graph\n",
              "1            If condition for redirection in jquery?  ...                         javascript jquery redirect\n",
              "2   git for development and svn on production server  ...                     git svn deployment integration\n",
              "3              How to use HTML Image Button and PHP?  ...                                           php html\n",
              "4  Using 'LIKE' against column names in a where c...  ...  sql sql-server sql-server-2008 sql-server-2005...\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sY51xN-vEypq",
        "colab_type": "text"
      },
      "source": [
        "# Lets clean the Title, Body and Code"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pf62HcMfRVPk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.read_csv(\"data/3_Train_5L.csv\", usecols=[\"Title\", \"Body\", \"Tags\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PipelPEyRVL5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Cleaning Titles using Regular Expressions\n",
        "\n",
        "import re\n",
        "\n",
        "titles_list=[]\n",
        "for i in range(df.shape[0]):\n",
        "  # taking out html tags if any\n",
        "  x = re.sub(r\"</{0,1}[\\w]+>\", \" \", df[\"Title\"][i] )\n",
        "  # replacing c++ and c#\n",
        "  x= re.sub(r\"C\\+\\+\", \"CPP\", x)\n",
        "  x = re.sub(r\"C#\", \"CSHARP\", x)\n",
        "  # removing non alpahbets\n",
        "  x = re.sub(r\"[^a-zA-Z]+\", \" \", x)\n",
        "  # removing extra spaces\n",
        "  x = re.sub(r\"[\\s]+\", \" \", x)\n",
        "  # removing all single letters\n",
        "  x = re.sub(r\"(?:^| )+[a-bd-qs-zA-BD-QS-Z](?=$| )+\", \"\", x)\n",
        "\n",
        "  # adding to list\n",
        "  titles_list.append(x.lower())\n",
        "\n",
        "# replacing in dataset\n",
        "df[\"Title\"] = titles_list\n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zJsrfQ-VRVID",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# cleaning the body\n",
        "\n",
        "\n",
        "import re\n",
        "code_list=[]\n",
        "bodies_list=[]\n",
        "for i in range(df.shape[0]):\n",
        "  # separating the code\n",
        "  code = str( re.findall(r\"<code>(.*?)</code>\", df[\"Body\"][i], flags=re.DOTALL) ) or \"No_code\"\n",
        "  # removing code from body\n",
        "  x = re.sub(r\"<code>(.*?)</code>\", \" \", df[\"Body\"][i], flags=re.MULTILINE|re.DOTALL)\n",
        "  # removing html tags\n",
        "  x = re.sub(r\"</{0,1}[\\w]+>\", \" \", x )\n",
        "  # replacing c+ and c#\n",
        "  x = re.sub(r\"C\\+\\+\", \"CPP\", x)\n",
        "  x = re.sub(r\"C#\", \"CSHARP\", x)\n",
        "  # removing non alphabets\n",
        "  x = re.sub(r\"[^a-zA-Z]+\", \" \", x)\n",
        "  # removing extra spaces\n",
        "  x = re.sub(r\"[\\s]+\", \" \", x)\n",
        "  # removing single characters\n",
        "  x = re.sub(r\"(?:^| )+[a-bd-qs-zA-BD-QS-Z]{1,2}(?=$| )+\", \"\", x)\n",
        "\n",
        "  # adding to list\n",
        "  bodies_list.append(x.lower())\n",
        "  code_list.append(code.lower())\n",
        "\n",
        "\n",
        "# replacing in dataset\n",
        "df[\"Body\"] = bodies_list\n",
        "df[\"Code\"] = code_list\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jqRIHSioZKCJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# cleaning the code\n",
        "\n",
        "import re\n",
        "clean_code_list = []\n",
        "for i in range(df.shape[0]):\n",
        "  # removing non alphabets \n",
        "  x = re.sub(r\"[^a-zA-Z]+\", \" \", df[\"Code\"][i])\n",
        "  # removing all less than 3 letter words\n",
        "  x = re.sub(r\"\\b\\w{1,3}\\b\", \" \", x)\n",
        "  # removing all extra spaces\n",
        "  x = re.sub(r\"[\\s]+\", \" \", x)\n",
        "  # removing left and right spaces\n",
        "  x = x.strip()\n",
        "\n",
        "  # adding to list\n",
        "  clean_code_list.append(x)\n",
        "\n",
        "\n",
        "# replacing in dataset\n",
        "df[\"Code\"] = clean_code_list"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O9w9kW-sZJ_l",
        "colab_type": "code",
        "outputId": "8da7dab7-36a6-4591-cd77-c441491bf471",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 285
        }
      },
      "source": [
        "# looking at new dataset\n",
        "\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Title</th>\n",
              "      <th>Body</th>\n",
              "      <th>Tags</th>\n",
              "      <th>Code</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>graph of linked tags</td>\n",
              "      <td>have list lists linked tag like this each sub...</td>\n",
              "      <td>python graph</td>\n",
              "      <td>some tags interactive media berlin gallery eve...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>if condition for redirection in jquery</td>\n",
              "      <td>edit someone suggested differently were use h...</td>\n",
              "      <td>javascript jquery redirect</td>\n",
              "      <td>options followsymlinks nrewriteengine nrewrite...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>git for development and svn on production server</td>\n",
              "      <td>have website development using git control ve...</td>\n",
              "      <td>git svn deployment integration</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>how to use html image button and php</td>\n",
              "      <td>okay here the scenario have simple html page ...</td>\n",
              "      <td>php html</td>\n",
              "      <td>html head head body form action submit method ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>using like against column names in where clause</td>\n",
              "      <td>started take look the following question href...</td>\n",
              "      <td>sql sql-server sql-server-2008 sql-server-2005...</td>\n",
              "      <td>select orderid productid unitprice quantity di...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              Title  ...                                               Code\n",
              "0                              graph of linked tags  ...  some tags interactive media berlin gallery eve...\n",
              "1           if condition for redirection in jquery   ...  options followsymlinks nrewriteengine nrewrite...\n",
              "2  git for development and svn on production server  ...                                                   \n",
              "3             how to use html image button and php   ...  html head head body form action submit method ...\n",
              "4   using like against column names in where clause  ...  select orderid productid unitprice quantity di...\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h1hea2qiZJ9c",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Saving to new csv to mark checkpoint\n",
        "\n",
        "df.to_csv(\"data/4_Cleaned_Train.csv\", index=True)\n",
        "\n",
        "# !cp data/4_Cleaned_Train.csv drive/My\\ Drive/tcs/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "obIa9aHgLiXz",
        "colab_type": "text"
      },
      "source": [
        "## Lets add all text together\n",
        "\n",
        "Now we have cleaned the Title, Body and Code, so we can add them into a single column."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xfpa1U52ZJ0q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Creating Title + Body + Code Dataframe\n",
        "\n",
        "df_tbc = pd.DataFrame()\n",
        "\n",
        "# adding T B and C separated by space\n",
        "df_tbc[\"TBC\"] = df[\"Title\"] + \" \" + df[\"Body\"] + \" \" + df[\"Code\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7fOC32VGL3cP",
        "colab_type": "code",
        "outputId": "2a5038c1-8a41-4160-806d-a4bc95d7ea90",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 216
        }
      },
      "source": [
        "# Looking at new DF\n",
        "\n",
        "print(df_tbc.shape)\n",
        "\n",
        "df_tbc.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(500000, 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>TBC</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>graph of linked tags  have list lists linked t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>if condition for redirection in jquery   edit ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>git for development and svn on production serv...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>how to use html image button and php   okay he...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>using like against column names in where claus...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                 TBC\n",
              "0  graph of linked tags  have list lists linked t...\n",
              "1  if condition for redirection in jquery   edit ...\n",
              "2  git for development and svn on production serv...\n",
              "3  how to use html image button and php   okay he...\n",
              "4  using like against column names in where claus..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8JdCsjgsMaQS",
        "colab_type": "text"
      },
      "source": [
        "### Lets find out the total no of words and No of words per example"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U9gsPPpxbUbz",
        "colab_type": "code",
        "outputId": "08fac3eb-75de-4283-db43-1fcb5d7379d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "# Total no of words\n",
        "# Average words/ example\n",
        "\n",
        "\n",
        "len_tbc = 0\n",
        "\n",
        "for i in range(df_tbc.shape[0]):\n",
        "  l = len(df_tbc[\"TBC\"][i].split())\n",
        "\n",
        "  len_tbc += l\n",
        "\n",
        "# Total no of words\n",
        "print(\"Total Words:\",len_tbc)\n",
        "\n",
        "# Average words/ example\n",
        "print(\"Average words per example:\", len_tbc/df_tbc.shape[0])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total Words: 52692497\n",
            "Average words per example: 105.384994\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N1Zlq2MzM6Q0",
        "colab_type": "text"
      },
      "source": [
        "# Now we can Tokenize the data, Remove Stopwords and Stem the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O0XCT6pZbUXU",
        "colab_type": "code",
        "outputId": "4dd09c13-04a7-4638-d760-b2f282b9c664",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "source": [
        "# importing the libraries and downloading packages\n",
        "\n",
        "# Natural Language Toolkit\n",
        "import nltk\n",
        "# Stopwords lsist\n",
        "from nltk.corpus import stopwords\n",
        "# tokenizer\n",
        "from nltk.tokenize import word_tokenize\n",
        "# Stemmer\n",
        "from nltk.stem.snowball import SnowballStemmer\n",
        "\n",
        "# downloading necessary packages\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')\n",
        "\n",
        "# Creating Stop words and Stemmer objects\n",
        "stop_words = set(stopwords.words('english'))\n",
        "stemmer = SnowballStemmer(\"english\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0S3SGulwNvow",
        "colab_type": "text"
      },
      "source": [
        "#### Testing Tokenize + Stopword removal + Stem on single data point"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v9M_E6L3gWqz",
        "colab_type": "code",
        "outputId": "0abce298-90ce-4e77-f479-2b3e4b5e7cce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "words = word_tokenize(df_tbc.iloc[0,0])\n",
        "\n",
        "t = \" \".join(str(stemmer.stem(word)) for word in words if word not in stop_words )\n",
        "\n",
        "print(t)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "graph link tag list list link tag like sub list link tag want dictionari occur link draw graph code code work repetit exampl link art video present two time first art second video smart way make sure repetit tag interact media berlin galleri event istanbul experiment newmedia bioart seriousgam instal interact design london peopl event newmedia istanbul artist interact instal berlin german peopl video remov occur item return filter lambda item dict occur dictionari tag nfor taglist tag taglist extend taglist els taglist remov occur count nfor taglist item taglist taglist count taglist nfrom pprint import pprint npprint\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y5LoqyhzOP5w",
        "colab_type": "text"
      },
      "source": [
        "## Now lets do it on all 500,000 examples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kjXbbUYMbUTE",
        "colab_type": "code",
        "outputId": "e607fd2f-5647-4a58-be56-797ae10c28ce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "source": [
        "#  Tokenize + Stopword removal + Stem\n",
        "\n",
        "tok_stop_stem_list=[]\n",
        "\n",
        "for i in range(df_tbc.shape[0]):\n",
        "  # tokenize\n",
        "  words = word_tokenize(df_tbc.iloc[i,0])\n",
        "  # stopword removal and stemming\n",
        "  t = \" \".join(str(stemmer.stem(word)) for word in words if word not in stop_words  )\n",
        "  tok_stop_stem_list.append(t)\n",
        "\n",
        "  # to see progress\n",
        "  if not i%100000:\n",
        "    print(i)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0\n",
            "100000\n",
            "200000\n",
            "300000\n",
            "400000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yyNM9Rp6OyBB",
        "colab_type": "text"
      },
      "source": [
        "#### Now its right time to save the latest dataset on disk to mark another checkpoint"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OTzKKMTcbURp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Saving Data\n",
        "\n",
        "# replacing in df\n",
        "df_tbc[\"TBC\"] = tok_stop_stem_list\n",
        "\n",
        "# Also adding tags\n",
        "df_tbc[\"Tags\"] = df[\"Tags\"]\n",
        "\n",
        "\n",
        "# saving to csv\n",
        "df_tbc.to_csv(\"data/5_TBC_tok_stop_stem.csv\", index=True)\n",
        "\n",
        "# !cp data/5_TBC_tok_stop_stem.csv drive/My\\ Drive/tcs/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z7JeilH-PedA",
        "colab_type": "text"
      },
      "source": [
        "#### Let's look at the data now"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p_avVqdmkhON",
        "colab_type": "code",
        "outputId": "af23005e-6614-44b8-da3b-577adfed4d7b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 216
        }
      },
      "source": [
        "print(df_tbc.shape)\n",
        "\n",
        "df_tbc.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(500000, 2)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>TBC</th>\n",
              "      <th>Tags</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>graph link tag list list link tag like sub lis...</td>\n",
              "      <td>python graph</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>condit redirect jqueri edit someon suggest dif...</td>\n",
              "      <td>javascript jquery redirect</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>git develop svn product server websit develop ...</td>\n",
              "      <td>git svn deployment integration</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>use html imag button php okay scenario simpl h...</td>\n",
              "      <td>php html</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>use like column name claus start take look fol...</td>\n",
              "      <td>sql sql-server sql-server-2008 sql-server-2005...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                 TBC                                               Tags\n",
              "0  graph link tag list list link tag like sub lis...                                       python graph\n",
              "1  condit redirect jqueri edit someon suggest dif...                         javascript jquery redirect\n",
              "2  git develop svn product server websit develop ...                     git svn deployment integration\n",
              "3  use html imag button php okay scenario simpl h...                                           php html\n",
              "4  use like column name claus start take look fol...  sql sql-server sql-server-2008 sql-server-2005..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x8C5Llk2h8q8",
        "colab_type": "code",
        "outputId": "8fde79e6-2247-4ec7-c5e4-43394fc260d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "# Total no of words\n",
        "# Average words/ example\n",
        "\n",
        "\n",
        "len_tbc_new = 0\n",
        "\n",
        "for i in range(df_tbc.shape[0]):\n",
        "  l = len(df_tbc[\"TBC\"][i].split())\n",
        "\n",
        "  len_tbc_new += l\n",
        "\n",
        "print(\"OLD:\")\n",
        "print(\"Total:\", len_tbc)\n",
        "print(\"Words/row:\", len_tbc/df_tbc.shape[0])\n",
        "\n",
        "print(\"NEW:\")\n",
        "print(\"Total:\", len_tbc_new)\n",
        "print(\"Words/row:\", len_tbc_new/df_tbc.shape[0])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "OLD:\n",
            "Total: 52692497\n",
            "Words/row: 105.384994\n",
            "NEW:\n",
            "Total: 40248351\n",
            "Words/row: 80.496702\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z6ahvFQcuAMK",
        "colab_type": "text"
      },
      "source": [
        "## Vectorizing the Data\n",
        "\n",
        "In this section, we will use CountVectorizer for the TBC column, and Binary One-Hot-Encoding for the tags.\n",
        "\n",
        "Here also, it is advised to clean the RAM or restart runtime."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yclxU1V1x15Q",
        "colab_type": "text"
      },
      "source": [
        "We'll use Bag of Words model. \n",
        "\n",
        "One Parameter of Bag Of Words model is N-Gram range.</br>\n",
        "N-gram is the combination fo words to form some meaningful representation.\n",
        "\n",
        "We'll use 2 Ngram ranges:</br>\n",
        "1. (1,1) ---- for single words\n",
        "2. (1,4) ---- for 1 to 4 combinations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mplz6nr0vXCH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# loading the data into ram\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "df_tbc = pd.read_csv(\"data/5_TBC_tok_stop_stem.csv\", usecols=[\"TBC\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SZ_KdDlwvhTl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Ngrams = (1,1)\n",
        "# using count vectorizer on TBC column\n",
        "# we drop all the words with frequency leass than 0.0001\n",
        "# also to control the no of columns, we limit max_features to 25000\n",
        "\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "vectorizer = CountVectorizer(ngram_range=(1,1), tokenizer=lambda x: x.split(), min_df=0.00009, max_features=25000)\n",
        "X_data_1_1 = vectorizer.fit_transform(df_tbc[\"TBC\"])\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bEbuEHQ5vhP7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Saving the new data as a sparse matrix in npz format\n",
        "# Also saving vectorizer vocabulary for testing\n",
        "\n",
        "from scipy import sparse\n",
        "sparse.save_npz(\"data/6_X_data_1_1.npz\", X_data_1_1)\n",
        "\n",
        "# !cp 6_X_data_1_1.npz drive/My\\ Drive/tcs/\n",
        "\n",
        "# we can only save the vocabulaty dictionary\n",
        "import pickle\n",
        "pickle.dump(vectorizer.vocabulary_, open(\"vectorizers/ngram_1_1_vectorizer.pickle\", \"wb\"))\n",
        "\n",
        "# !cp ngram_1_1_vectorizer.pickle drive/My\\ Drive/tcs/\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kkw2o5krv2vy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Ngrams = (1,4)\n",
        "# using count vectorizer on TBC column\n",
        "# we drop all the words with frequency leass than 0.0001\n",
        "# also to control the no of columns, we limit max_features to 25000\n",
        "\n",
        "vectorizer = CountVectorizer( ngram_range=(1,4), tokenizer=lambda x: x.split(), min_df=0.00009, max_features=25000)\n",
        "X_data_1_4 = vectorizer.fit_transform(df_tbc[\"TBC\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3dzKTW-vv4sa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# saving the 1-4 vocabulary and spase matrix data\n",
        "\n",
        "# saving new vectorizer for testing\n",
        "pickle.dump(vectorizer.vocabulary_, open(\"vectorizers/ngram_1_4_vectorizer.pickle\", \"wb\"))\n",
        "\n",
        "# saving sparse matrix into npz\n",
        "sparse.save_npz(\"data/7_X_data_1_4.npz\", X_data_1_4)\n",
        "\n",
        "# !cp 7_X_data_1_4.npz drive/My\\ Drive/tcs/\n",
        "# !cp ngram_1_4_vectorizer.pickle drive/My\\ Drive/tcs/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a8mCfyNLzEh_",
        "colab_type": "text"
      },
      "source": [
        "### Lets look at how our data looks now.\n",
        "\n",
        "Our data is vetorized, so, we'll only look at few columns and data shape"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IItKWM1Yv4ni",
        "colab_type": "code",
        "outputId": "f4f9f718-d9dd-4cba-f761-4e0c7183875e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "source": [
        "print(\"N-gram=(1,1)\")\n",
        "print(X_data_1_1.shape)\n",
        "pd.DataFrame(X_data_1_1[:10].toarray()).head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "N-gram=(1,1)\n",
            "(500000, 13415)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "      <th>13</th>\n",
              "      <th>14</th>\n",
              "      <th>15</th>\n",
              "      <th>16</th>\n",
              "      <th>17</th>\n",
              "      <th>18</th>\n",
              "      <th>19</th>\n",
              "      <th>20</th>\n",
              "      <th>21</th>\n",
              "      <th>22</th>\n",
              "      <th>23</th>\n",
              "      <th>24</th>\n",
              "      <th>25</th>\n",
              "      <th>26</th>\n",
              "      <th>27</th>\n",
              "      <th>28</th>\n",
              "      <th>29</th>\n",
              "      <th>30</th>\n",
              "      <th>31</th>\n",
              "      <th>32</th>\n",
              "      <th>33</th>\n",
              "      <th>34</th>\n",
              "      <th>35</th>\n",
              "      <th>36</th>\n",
              "      <th>37</th>\n",
              "      <th>38</th>\n",
              "      <th>39</th>\n",
              "      <th>...</th>\n",
              "      <th>13375</th>\n",
              "      <th>13376</th>\n",
              "      <th>13377</th>\n",
              "      <th>13378</th>\n",
              "      <th>13379</th>\n",
              "      <th>13380</th>\n",
              "      <th>13381</th>\n",
              "      <th>13382</th>\n",
              "      <th>13383</th>\n",
              "      <th>13384</th>\n",
              "      <th>13385</th>\n",
              "      <th>13386</th>\n",
              "      <th>13387</th>\n",
              "      <th>13388</th>\n",
              "      <th>13389</th>\n",
              "      <th>13390</th>\n",
              "      <th>13391</th>\n",
              "      <th>13392</th>\n",
              "      <th>13393</th>\n",
              "      <th>13394</th>\n",
              "      <th>13395</th>\n",
              "      <th>13396</th>\n",
              "      <th>13397</th>\n",
              "      <th>13398</th>\n",
              "      <th>13399</th>\n",
              "      <th>13400</th>\n",
              "      <th>13401</th>\n",
              "      <th>13402</th>\n",
              "      <th>13403</th>\n",
              "      <th>13404</th>\n",
              "      <th>13405</th>\n",
              "      <th>13406</th>\n",
              "      <th>13407</th>\n",
              "      <th>13408</th>\n",
              "      <th>13409</th>\n",
              "      <th>13410</th>\n",
              "      <th>13411</th>\n",
              "      <th>13412</th>\n",
              "      <th>13413</th>\n",
              "      <th>13414</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 13415 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   0      1      2      3      4      ...  13410  13411  13412  13413  13414\n",
              "0      0      0      0      0      0  ...      0      0      0      0      0\n",
              "1      0      0      0      0      0  ...      0      0      0      0      0\n",
              "2      0      0      0      0      0  ...      0      0      0      0      0\n",
              "3      0      0      0      0      0  ...      0      0      0      0      0\n",
              "4      0      0      0      0      0  ...      0      0      0      0      0\n",
              "\n",
              "[5 rows x 13415 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6TeWNirtv2st",
        "colab_type": "code",
        "outputId": "6a199c4c-f737-43cd-e427-bfb1681d4936",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "source": [
        "print(\"N-gram=(1,4)\")\n",
        "print(X_data_1_4.shape)\n",
        "pd.DataFrame(X_data_1_4[:10].toarray()).head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "N-gram=(1,4)\n",
            "(500000, 25000)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "      <th>13</th>\n",
              "      <th>14</th>\n",
              "      <th>15</th>\n",
              "      <th>16</th>\n",
              "      <th>17</th>\n",
              "      <th>18</th>\n",
              "      <th>19</th>\n",
              "      <th>20</th>\n",
              "      <th>21</th>\n",
              "      <th>22</th>\n",
              "      <th>23</th>\n",
              "      <th>24</th>\n",
              "      <th>25</th>\n",
              "      <th>26</th>\n",
              "      <th>27</th>\n",
              "      <th>28</th>\n",
              "      <th>29</th>\n",
              "      <th>30</th>\n",
              "      <th>31</th>\n",
              "      <th>32</th>\n",
              "      <th>33</th>\n",
              "      <th>34</th>\n",
              "      <th>35</th>\n",
              "      <th>36</th>\n",
              "      <th>37</th>\n",
              "      <th>38</th>\n",
              "      <th>39</th>\n",
              "      <th>...</th>\n",
              "      <th>24960</th>\n",
              "      <th>24961</th>\n",
              "      <th>24962</th>\n",
              "      <th>24963</th>\n",
              "      <th>24964</th>\n",
              "      <th>24965</th>\n",
              "      <th>24966</th>\n",
              "      <th>24967</th>\n",
              "      <th>24968</th>\n",
              "      <th>24969</th>\n",
              "      <th>24970</th>\n",
              "      <th>24971</th>\n",
              "      <th>24972</th>\n",
              "      <th>24973</th>\n",
              "      <th>24974</th>\n",
              "      <th>24975</th>\n",
              "      <th>24976</th>\n",
              "      <th>24977</th>\n",
              "      <th>24978</th>\n",
              "      <th>24979</th>\n",
              "      <th>24980</th>\n",
              "      <th>24981</th>\n",
              "      <th>24982</th>\n",
              "      <th>24983</th>\n",
              "      <th>24984</th>\n",
              "      <th>24985</th>\n",
              "      <th>24986</th>\n",
              "      <th>24987</th>\n",
              "      <th>24988</th>\n",
              "      <th>24989</th>\n",
              "      <th>24990</th>\n",
              "      <th>24991</th>\n",
              "      <th>24992</th>\n",
              "      <th>24993</th>\n",
              "      <th>24994</th>\n",
              "      <th>24995</th>\n",
              "      <th>24996</th>\n",
              "      <th>24997</th>\n",
              "      <th>24998</th>\n",
              "      <th>24999</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 25000 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   0      1      2      3      4      ...  24995  24996  24997  24998  24999\n",
              "0      0      0      0      0      0  ...      0      0      0      0      0\n",
              "1      0      0      0      0      0  ...      0      0      0      0      0\n",
              "2      0      0      0      0      0  ...      0      0      0      0      0\n",
              "3      0      0      0      0      0  ...      0      0      0      0      0\n",
              "4      0      0      0      0      0  ...      0      0      0      0      0\n",
              "\n",
              "[5 rows x 25000 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "exhd8d5u0tCH",
        "colab_type": "text"
      },
      "source": [
        "### Now, we need to One-Hot-encode the tags, but we can use CountVectorizer with binary output for that purpose\n",
        "\n",
        "\n",
        "> If needed,  you can also reset runtime here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FeN5PSDDl023",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# loading the tags into ram and discarding TBC column to free up RAM\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "df_tbc = pd.read_csv(\"data/5_TBC_tok_stop_stem.csv\", usecols=[\"Tags\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gu4_GkUMl0x_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# vectorizing the tags and saving vectorizer to future purposes\n",
        "\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "vectorizer = CountVectorizer(tokenizer = lambda x: x.split(), binary='true')\n",
        "tags = vectorizer.fit_transform(df_tbc['Tags'])\n",
        "\n",
        "\n",
        "import pickle\n",
        "pickle.dump(vectorizer.vocabulary_, open(\"vectorizers/tags_vectorizer.pickle\", \"wb\"))\n",
        "\n",
        "# !cp tags_vectorizer.pickle drive/My\\ Drive/tcs/\n",
        "\n",
        "\n",
        "# to load\n",
        "# loaded_vec = CountVectorizer(decode_error=\"replace\",vocabulary=pickle.load(open(\"feature.pkl\", \"rb\")))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SjpOeYIs2Sgt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# saving tags vectorized sparse matrix as npz\n",
        "\n",
        "from scipy import sparse\n",
        "sparse.save_npz(\"data/Tags_vectorized.npz\", tags)\n",
        "\n",
        "# !cp data/Tags_vectorized.npz drive/My\\ Drive/tcs/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gtzwnolo31MI",
        "colab_type": "text"
      },
      "source": [
        "### Let's look at Tags in vectorized form"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7w8wK4oqobMZ",
        "colab_type": "code",
        "outputId": "25823f3d-c930-4fc9-f328-27c9e70c5ed6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 264
        }
      },
      "source": [
        "print(tags.shape)\n",
        "\n",
        "pd.DataFrame(tags[:10].toarray()).head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(500000, 500)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "      <th>13</th>\n",
              "      <th>14</th>\n",
              "      <th>15</th>\n",
              "      <th>16</th>\n",
              "      <th>17</th>\n",
              "      <th>18</th>\n",
              "      <th>19</th>\n",
              "      <th>20</th>\n",
              "      <th>21</th>\n",
              "      <th>22</th>\n",
              "      <th>23</th>\n",
              "      <th>24</th>\n",
              "      <th>25</th>\n",
              "      <th>26</th>\n",
              "      <th>27</th>\n",
              "      <th>28</th>\n",
              "      <th>29</th>\n",
              "      <th>30</th>\n",
              "      <th>31</th>\n",
              "      <th>32</th>\n",
              "      <th>33</th>\n",
              "      <th>34</th>\n",
              "      <th>35</th>\n",
              "      <th>36</th>\n",
              "      <th>37</th>\n",
              "      <th>38</th>\n",
              "      <th>39</th>\n",
              "      <th>...</th>\n",
              "      <th>460</th>\n",
              "      <th>461</th>\n",
              "      <th>462</th>\n",
              "      <th>463</th>\n",
              "      <th>464</th>\n",
              "      <th>465</th>\n",
              "      <th>466</th>\n",
              "      <th>467</th>\n",
              "      <th>468</th>\n",
              "      <th>469</th>\n",
              "      <th>470</th>\n",
              "      <th>471</th>\n",
              "      <th>472</th>\n",
              "      <th>473</th>\n",
              "      <th>474</th>\n",
              "      <th>475</th>\n",
              "      <th>476</th>\n",
              "      <th>477</th>\n",
              "      <th>478</th>\n",
              "      <th>479</th>\n",
              "      <th>480</th>\n",
              "      <th>481</th>\n",
              "      <th>482</th>\n",
              "      <th>483</th>\n",
              "      <th>484</th>\n",
              "      <th>485</th>\n",
              "      <th>486</th>\n",
              "      <th>487</th>\n",
              "      <th>488</th>\n",
              "      <th>489</th>\n",
              "      <th>490</th>\n",
              "      <th>491</th>\n",
              "      <th>492</th>\n",
              "      <th>493</th>\n",
              "      <th>494</th>\n",
              "      <th>495</th>\n",
              "      <th>496</th>\n",
              "      <th>497</th>\n",
              "      <th>498</th>\n",
              "      <th>499</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 500 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   0    1    2    3    4    5    6    ...  493  494  495  496  497  498  499\n",
              "0    0    0    0    0    0    0    0  ...    0    0    0    0    0    0    0\n",
              "1    0    0    0    0    0    0    0  ...    0    0    0    0    0    0    0\n",
              "2    0    0    0    0    0    0    0  ...    0    0    0    0    0    0    0\n",
              "3    0    0    0    0    0    0    0  ...    0    0    0    0    0    0    0\n",
              "4    0    0    0    0    0    0    0  ...    0    0    0    0    0    0    0\n",
              "\n",
              "[5 rows x 500 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FRLqnKep4Hwo",
        "colab_type": "text"
      },
      "source": [
        "## Splitting Training and Test Set\n",
        "\n",
        "We have 0.5 M examples in out Dataset, So, we can use 10,000 of them for testing.\n",
        "\n",
        "10,000 would be 2 %.  ---- test_size = 0.02"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HJn1bLG5CPdq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# loading n-gram 1,1 data\n",
        "# feel free to use ngram 1,4\n",
        "\n",
        "X_data = sparse.load_npz(\"data/6_X_data_1_1.npz\")\n",
        "y_data = sparse.load_npz(\"data/Tags_vectorized.npz\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jJNstWinukmA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# splitting Train:Test :: 98:2\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, test_size=0.02, random_state=42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m3KXujGwl0wR",
        "colab_type": "code",
        "outputId": "5c245038-e509-4820-81ca-cc7fb193305a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "print(\"Training Data:\", X_train.shape)\n",
        "print(\"Test Data:\", X_test.shape)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training Data: (490000, 13415)\n",
            "Test Data: (10000, 13415)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d1Eic1I-v6D6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# saving train and test files separately\n",
        "\n",
        "sparse.save_npz(\"data/8_X_train_1_1.npz\", X_train)\n",
        "sparse.save_npz(\"data/8_X_test_1_1.npz\", X_test)\n",
        "\n",
        "\n",
        "sparse.save_npz(\"data/8_y_train.npz\", y_train)\n",
        "sparse.save_npz(\"data/8_y_test.npz\", y_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r0q5m3p0EenA",
        "colab_type": "text"
      },
      "source": [
        "Here, we are done with all the pre processing steps.\n",
        "Now we can train our model with any of the following configuration:</br>\n",
        "> X_train_1_1 and y_train\n",
        "\n",
        "> X_train_1_4 with y_train\n",
        "\n",
        "\n",
        "Models:\n",
        "> SGDCLassifier\n",
        "\n",
        "> Logistic Regression\n",
        "\n",
        "> Gaussian Naive Bayes\n",
        "\n",
        "> Neural Network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ji4L0INhFDgy",
        "colab_type": "text"
      },
      "source": [
        "# Training\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uWjPjCjx088l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# load 8_X_train_1_1.npz\n",
        "# load 8_y_train.npz\n",
        "\n",
        "# OR\n",
        "\n",
        "# load 8_X_train_1_4.npz\n",
        "# load 8_y_train.npz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LdhoaxFZl0qM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Imports for all Classifiers\n",
        "\n",
        "from scipy import sparse\n",
        "\n",
        "# !pip install scikit-multilearn\n",
        "from sklearn.multiclass import OneVsRestClassifier\n",
        "\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.linear_model import SGDClassifier\n",
        "from sklearn.naive_bayes import GaussianNB"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BFVS4RZ4l0m7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "############################################\n",
        "\n",
        "# Logistic Regression - (1,1)\n",
        "\n",
        "############################################\n",
        "\n",
        "from scipy import sparse\n",
        "\n",
        "# feel free to replace data with ngram 1-4 if you saved it\n",
        "X_train_1_1 = sparse.load_npz(\"data/8_X_train_1_1.npz\")\n",
        "y_train = sparse.load_npz(\"data/8_y_train.npz\")\n",
        "\n",
        "# fitting the model\n",
        "clf_log_reg_1 = OneVsRestClassifier(LogisticRegression(C=0.1, penalty='l2', verbose=1))\n",
        "clf_log_reg_1.fit(X_train_1_1, y_train)\n",
        "\n",
        "# saving model to disk\n",
        "import pickle\n",
        "pickle.dump(clf_log_reg_1, open(\"models/1_log_reg_clf.pickle\", \"wb\"))\n",
        "\n",
        "# !cp models/1_log_reg_clf.pickle drive/My\\ Drive/tcs/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKAVJcipl0iD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "############################################\n",
        "\n",
        "# SGDClassifier-(1,1)\n",
        "\n",
        "############################################\n",
        "\n",
        "from scipy import sparse\n",
        "\n",
        "# feel free to replace data with ngram 1-4 if you saved it\n",
        "X_train_1_1 = sparse.load_npz(\"data/8_X_train_1_1.npz\")\n",
        "y_train = sparse.load_npz(\"data/8_Tags_vectorized.npz\")\n",
        "\n",
        "\n",
        "# fitting the model\n",
        "clf_sgd_1 = OneVsRestClassifier(SGDClassifier(loss='hinge', alpha=1, penalty='l2',n_jobs=-1))\n",
        "clf_sgd_1.fit(X_train_1_1, y_train)\n",
        "\n",
        "# saving model to disk\n",
        "import pickle\n",
        "pickle.dump(clf_sgd_1, open(\"models/2_sgd_clf.pickle\", \"wb\"))\n",
        "\n",
        "# !cp models/2_sgd_clf.pickle drive/My\\ Drive/tcs/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bL0EUn5dZ30",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "############################################\n",
        "\n",
        "# SVC - (1,1)\n",
        "\n",
        "############################################\n",
        "\n",
        "\n",
        "from scipy import sparse\n",
        "\n",
        "# feel free to replace data with ngram 1-4 if you saved it\n",
        "X_train_1_1 = sparse.load_npz(\"data/8_X_train_1_1.npz\")\n",
        "y_train = sparse.load_npz(\"data/8_y_train.npz\")\n",
        "\n",
        "# fitting the model\n",
        "clf_svc = OneVsRestClassifier(SVC())\n",
        "clf_svc.fit(X_train_1_1, y_train)\n",
        "\n",
        "# saving model to disk\n",
        "import pickle\n",
        "pickle.dump(clf_svc, open(\"models/3_svc_clf.pickle\", \"wb\"))\n",
        "\n",
        "# !cp models/3_svc_clf.pickle drive/My\\ Drive/tcs/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZiFfHW8wHP6d",
        "colab_type": "text"
      },
      "source": [
        "# Testing\n",
        "\n",
        "Now we can load the model we saved and test the model.\n",
        "\n",
        "\n",
        "Replace the ***1_log_reg_clf.pickle*** with your saved model file to test other models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1X1WOuYGdZ7H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "##############################\n",
        "# Testing\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "\n",
        "# loading the test data\n",
        "# feel free to replace ngram 1,1 with ngram 1,4\n",
        "X_test_1_1 = sparse.load_npz(\"data/8_X_test_1_1.npz\")\n",
        "y_test = sparse.load_npz(\"data/8_y_test.npz\")\n",
        "\n",
        "\n",
        "\n",
        "# loading the model\n",
        "model = pickle.load(open(\"models/1_log_reg_clf.pickle\", \"rb\"))\n",
        "\n",
        "# predicting the tags on test set\n",
        "pred = model.predict(test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rjv0fd_RMUxd",
        "colab_type": "text"
      },
      "source": [
        "### Micro average F1 Score is a good metric for Multi-Class Multi-labels problem "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "5c245348-e509-7620-90ca-cc4dk193305a",
        "id": "_yjfv35HVT-D",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "# Using Micro average F1 score for Evaluation\n",
        "\n",
        "from sklearn.metrics import precision_score, recall_score, f1_score\n",
        "\n",
        "precis = precision_score(y_test, pred, average='micro')\n",
        "rec = recall_score(y_test, pred, average='micro')\n",
        "f1 = f1_score(y_test, pred, average='micro')\n",
        "\n",
        "print(\"Micro Average F1 Score Metrics:\")\n",
        "print(\"Precision: {.6f}\", precis)\n",
        "print(\"Recall: {.6f}\", rec)\n",
        "print(\"F1 Score: {.6f}\", f1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Micro Average F1 Score Metrics:\n",
            "Precision: 0.597346\n",
            "Recall: 0.449743\n",
            "F1 Score: 0.514131\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "olahsYQeMfjC",
        "colab_type": "text"
      },
      "source": [
        "#### Viewing the Results ---- Optional\n",
        "\n",
        "\n",
        "I have added this part after already training and testing the model a couple days ago. So this part's output is for the viewer to generate."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B9sy7dJ7KGQs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# loading vectorizer for testing\n",
        "tags_vec = CountVectorizer(decode_error=\"replace\",vocabulary=pickle.load(open(\"vectorizers/tags_vectorizer.pickle\", \"rb\")))\n",
        "tags_vec._validate_vocabulary()\n",
        "\n",
        "ques_vec = CountVectorizer(decode_error=\"replace\",vocabulary=pickle.load(open(\"vectorizers/ngram_1_1_vectorizer.pickle\", \"rb\")))\n",
        "ques_vec._validate_vocabulary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6UqilN10Jmhp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"Predicted tags for Questions\")\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "Results = pd.DataFrame({\"Questions\": ques_vec.inverse_transform(X_test_1_1),\n",
        "                        \"Tags\": tags_vec.inverse_transform(pred)})\n",
        "\n",
        "Results.head(100)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t3gemU5PMuaO",
        "colab_type": "text"
      },
      "source": [
        "# Project Conclusion\n",
        "\n",
        "> It has been a wonderful (Challanging ;P ) experience to solve this problem.\n",
        "\n",
        "The model took over 7 hours of training, but it was worth it for an F1 score of .51 .\n",
        "\n",
        "In NLP problems achieving .51 F1 scores is quite a good deal. And its really great surprize that simplest of the models like Logistic Regression is the one responsible for such a score.\n",
        "\n",
        "Ther can be various thing we can do to still improve the performance of the model. Like:\n",
        "\n",
        "> We can increase the no of examples to train upon\n",
        "\n",
        "> We can reduce the no of tags to just most freaquent 100 tags\n",
        "\n",
        "> We can train Gaussian Nave Bayes, but it takes only numpy arrays as input and does not work on sparse matruces, so we'll need to either use higher ram or we will have to use a Generator. </br>\n",
        "**fit_generator**\n",
        "\n",
        ">  We can also train a Neural Network with several layers to improve the F1.\n",
        "\n",
        "> LSTM and GRU models are also good options, but I dont see any need, as in my opinion, they won't affect the score much.\n",
        "\n",
        "I am very thankful to TCS Campus Commune team to give college students chance to grow by organising such healthy competitions."
      ]
    }
  ]
}