models-and-algorithms.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Parkinson's Disease\n",
    "This notebook uses the the dataset from [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml/datasets/Parkinson%27s+Disease+Classification) to practice classification problem through popular classification models. For each model, the algorithm, and a related topic if any, will be discussed before application. Afterwards, there will be a review on modeling performance. \n",
    "\n",
    "## Overview\n",
    "\n",
    "\n",
    "### I. Data Preparation\n",
    "- Load and Oberserve Data\n",
    "- Data Preprocessing\n",
    "- Min-Max Normalization\n",
    "- Problem Definition\n",
    "\n",
    "### II. Modeling\n",
    "\n",
    "Each modeling section consists of **an brief intro, the algorithm, discussion on related topics, and application** on the dataset.\n",
    "\n",
    "1. kNN\n",
    "    - Non-parametric Models\n",
    "    - Algorithm\n",
    "2. Naive Bayes  \n",
    "    - Bayes Classifier\n",
    "    - Algorithm\n",
    "    - Generative Model vs. Discriminative Model\n",
    "3. Logistic Regression\n",
    "    - Sigmoid Function\n",
    "    - Maximum Likelihood Estimation\n",
    "    - Algorithm\n",
    "    - (Also see Appendix A & B for related topics)\n",
    "4. Support Vector Machine\n",
    "    - Convex Sets and Convex Hulls\n",
    "    - Algorithm\n",
    "    - Soft-Margin SVM\n",
    "5. Kernel SVM\n",
    "    - Kernel\n",
    "    - Mercer's theorem\n",
    "    - RBF\n",
    "    - Algorithm\n",
    "6. Decision Tree\n",
    "\n",
    "### III. Model Improvement\n",
    "- PCA\n",
    "    - Algorithm: The First Principal Component\n",
    "    - Algorithm: General\n",
    "- Pipeline\n",
    "\n",
    "### IV. Model Selection\n",
    "- ROC Curve\n",
    "- Change Threshold\n",
    "- Classification Report\n",
    "\n",
    "### V. Appendix\n",
    "- Appendix A: Concepts for Logistic Regression\n",
    "    - A1. Binary Classification\n",
    "    - A2. Log Odds\n",
    "    - A3. Linear Discriminant Analysis\n",
    "\n",
    "- Appendix B: Linear Classifiers\n",
    "    - B1. Definition\n",
    "    - B2. Linear Separability\n",
    "    - B3. Methods\n",
    "    \n",
    "### References for Model Introduction and Algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I. Data Preparation\n",
    "### Load and Observe Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('../project-parkinsons-disease-classification/data/pd_speech_features.csv', \n",
    "                 header=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(756, 755)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>gender</th>\n",
       "      <th>PPE</th>\n",
       "      <th>DFA</th>\n",
       "      <th>RPDE</th>\n",
       "      <th>numPulses</th>\n",
       "      <th>numPeriodsPulses</th>\n",
       "      <th>meanPeriodPulses</th>\n",
       "      <th>stdDevPeriodPulses</th>\n",
       "      <th>locPctJitter</th>\n",
       "      <th>...</th>\n",
       "      <th>tqwt_kurtosisValue_dec_28</th>\n",
       "      <th>tqwt_kurtosisValue_dec_29</th>\n",
       "      <th>tqwt_kurtosisValue_dec_30</th>\n",
       "      <th>tqwt_kurtosisValue_dec_31</th>\n",
       "      <th>tqwt_kurtosisValue_dec_32</th>\n",
       "      <th>tqwt_kurtosisValue_dec_33</th>\n",
       "      <th>tqwt_kurtosisValue_dec_34</th>\n",
       "      <th>tqwt_kurtosisValue_dec_35</th>\n",
       "      <th>tqwt_kurtosisValue_dec_36</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.85247</td>\n",
       "      <td>0.71826</td>\n",
       "      <td>0.57227</td>\n",
       "      <td>240</td>\n",
       "      <td>239</td>\n",
       "      <td>0.008064</td>\n",
       "      <td>0.000087</td>\n",
       "      <td>0.00218</td>\n",
       "      <td>...</td>\n",
       "      <td>1.5620</td>\n",
       "      <td>2.6445</td>\n",
       "      <td>3.8686</td>\n",
       "      <td>4.2105</td>\n",
       "      <td>5.1221</td>\n",
       "      <td>4.4625</td>\n",
       "      <td>2.6202</td>\n",
       "      <td>3.0004</td>\n",
       "      <td>18.9405</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.76686</td>\n",
       "      <td>0.69481</td>\n",
       "      <td>0.53966</td>\n",
       "      <td>234</td>\n",
       "      <td>233</td>\n",
       "      <td>0.008258</td>\n",
       "      <td>0.000073</td>\n",
       "      <td>0.00195</td>\n",
       "      <td>...</td>\n",
       "      <td>1.5589</td>\n",
       "      <td>3.6107</td>\n",
       "      <td>23.5155</td>\n",
       "      <td>14.1962</td>\n",
       "      <td>11.0261</td>\n",
       "      <td>9.5082</td>\n",
       "      <td>6.5245</td>\n",
       "      <td>6.3431</td>\n",
       "      <td>45.1780</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.85083</td>\n",
       "      <td>0.67604</td>\n",
       "      <td>0.58982</td>\n",
       "      <td>232</td>\n",
       "      <td>231</td>\n",
       "      <td>0.008340</td>\n",
       "      <td>0.000060</td>\n",
       "      <td>0.00176</td>\n",
       "      <td>...</td>\n",
       "      <td>1.5643</td>\n",
       "      <td>2.3308</td>\n",
       "      <td>9.4959</td>\n",
       "      <td>10.7458</td>\n",
       "      <td>11.0177</td>\n",
       "      <td>4.8066</td>\n",
       "      <td>2.9199</td>\n",
       "      <td>3.1495</td>\n",
       "      <td>4.7666</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.41121</td>\n",
       "      <td>0.79672</td>\n",
       "      <td>0.59257</td>\n",
       "      <td>178</td>\n",
       "      <td>177</td>\n",
       "      <td>0.010858</td>\n",
       "      <td>0.000183</td>\n",
       "      <td>0.00419</td>\n",
       "      <td>...</td>\n",
       "      <td>3.7805</td>\n",
       "      <td>3.5664</td>\n",
       "      <td>5.2558</td>\n",
       "      <td>14.0403</td>\n",
       "      <td>4.2235</td>\n",
       "      <td>4.6857</td>\n",
       "      <td>4.8460</td>\n",
       "      <td>6.2650</td>\n",
       "      <td>4.0603</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.32790</td>\n",
       "      <td>0.79782</td>\n",
       "      <td>0.53028</td>\n",
       "      <td>236</td>\n",
       "      <td>235</td>\n",
       "      <td>0.008162</td>\n",
       "      <td>0.002669</td>\n",
       "      <td>0.00535</td>\n",
       "      <td>...</td>\n",
       "      <td>6.1727</td>\n",
       "      <td>5.8416</td>\n",
       "      <td>6.0805</td>\n",
       "      <td>5.7621</td>\n",
       "      <td>7.7817</td>\n",
       "      <td>11.6891</td>\n",
       "      <td>8.2103</td>\n",
       "      <td>5.0559</td>\n",
       "      <td>6.1164</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 755 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  gender      PPE      DFA     RPDE  numPulses  numPeriodsPulses  \\\n",
       "0   0       1  0.85247  0.71826  0.57227        240               239   \n",
       "1   0       1  0.76686  0.69481  0.53966        234               233   \n",
       "2   0       1  0.85083  0.67604  0.58982        232               231   \n",
       "3   1       0  0.41121  0.79672  0.59257        178               177   \n",
       "4   1       0  0.32790  0.79782  0.53028        236               235   \n",
       "\n",
       "   meanPeriodPulses  stdDevPeriodPulses  locPctJitter  ...    \\\n",
       "0          0.008064            0.000087       0.00218  ...     \n",
       "1          0.008258            0.000073       0.00195  ...     \n",
       "2          0.008340            0.000060       0.00176  ...     \n",
       "3          0.010858            0.000183       0.00419  ...     \n",
       "4          0.008162            0.002669       0.00535  ...     \n",
       "\n",
       "   tqwt_kurtosisValue_dec_28  tqwt_kurtosisValue_dec_29  \\\n",
       "0                     1.5620                     2.6445   \n",
       "1                     1.5589                     3.6107   \n",
       "2                     1.5643                     2.3308   \n",
       "3                     3.7805                     3.5664   \n",
       "4                     6.1727                     5.8416   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_30  tqwt_kurtosisValue_dec_31  \\\n",
       "0                     3.8686                     4.2105   \n",
       "1                    23.5155                    14.1962   \n",
       "2                     9.4959                    10.7458   \n",
       "3                     5.2558                    14.0403   \n",
       "4                     6.0805                     5.7621   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_32  tqwt_kurtosisValue_dec_33  \\\n",
       "0                     5.1221                     4.4625   \n",
       "1                    11.0261                     9.5082   \n",
       "2                    11.0177                     4.8066   \n",
       "3                     4.2235                     4.6857   \n",
       "4                     7.7817                    11.6891   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_34  tqwt_kurtosisValue_dec_35  \\\n",
       "0                     2.6202                     3.0004   \n",
       "1                     6.5245                     6.3431   \n",
       "2                     2.9199                     3.1495   \n",
       "3                     4.8460                     6.2650   \n",
       "4                     8.2103                     5.0559   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_36  class  \n",
       "0                    18.9405      1  \n",
       "1                    45.1780      1  \n",
       "2                     4.7666      1  \n",
       "3                     4.0603      1  \n",
       "4                     6.1164      1  \n",
       "\n",
       "[5 rows x 755 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last column, class, is the target variable that we want to predict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "252"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.id.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary of attribute datatypes:\n",
      "float64    749\n",
      "int64        6\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('Summary of attribute datatypes:\\n', df.dtypes.value_counts(), sep=\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of null values: 0\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of null values:\", df.isnull().sum().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The data consists of 3 records for each patient. To prevent using the same patient's own data to self-predict. We should aggregate the records to make each patient as one record. \n",
    "- All the columns, except for gender, seem to be numeric. Therefore, gender will be transformed into categorical variable.\n",
    "- There's no null value in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>PPE</th>\n",
       "      <th>DFA</th>\n",
       "      <th>RPDE</th>\n",
       "      <th>numPulses</th>\n",
       "      <th>numPeriodsPulses</th>\n",
       "      <th>meanPeriodPulses</th>\n",
       "      <th>stdDevPeriodPulses</th>\n",
       "      <th>locPctJitter</th>\n",
       "      <th>locAbsJitter</th>\n",
       "      <th>...</th>\n",
       "      <th>tqwt_kurtosisValue_dec_27</th>\n",
       "      <th>tqwt_kurtosisValue_dec_28</th>\n",
       "      <th>tqwt_kurtosisValue_dec_29</th>\n",
       "      <th>tqwt_kurtosisValue_dec_30</th>\n",
       "      <th>tqwt_kurtosisValue_dec_31</th>\n",
       "      <th>tqwt_kurtosisValue_dec_32</th>\n",
       "      <th>tqwt_kurtosisValue_dec_33</th>\n",
       "      <th>tqwt_kurtosisValue_dec_34</th>\n",
       "      <th>tqwt_kurtosisValue_dec_35</th>\n",
       "      <th>tqwt_kurtosisValue_dec_36</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.823387</td>\n",
       "      <td>0.696370</td>\n",
       "      <td>0.567250</td>\n",
       "      <td>235.333333</td>\n",
       "      <td>234.333333</td>\n",
       "      <td>0.008220</td>\n",
       "      <td>0.000073</td>\n",
       "      <td>0.001963</td>\n",
       "      <td>0.000016</td>\n",
       "      <td>...</td>\n",
       "      <td>1.546500</td>\n",
       "      <td>1.561733</td>\n",
       "      <td>2.862000</td>\n",
       "      <td>12.293333</td>\n",
       "      <td>9.717500</td>\n",
       "      <td>9.055300</td>\n",
       "      <td>6.259100</td>\n",
       "      <td>4.021533</td>\n",
       "      <td>4.164333</td>\n",
       "      <td>22.961700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.415637</td>\n",
       "      <td>0.793993</td>\n",
       "      <td>0.592453</td>\n",
       "      <td>213.333333</td>\n",
       "      <td>211.000000</td>\n",
       "      <td>0.008884</td>\n",
       "      <td>0.001849</td>\n",
       "      <td>0.005790</td>\n",
       "      <td>0.000050</td>\n",
       "      <td>...</td>\n",
       "      <td>7.049367</td>\n",
       "      <td>4.918567</td>\n",
       "      <td>4.827133</td>\n",
       "      <td>6.117633</td>\n",
       "      <td>8.599667</td>\n",
       "      <td>5.737233</td>\n",
       "      <td>7.933133</td>\n",
       "      <td>5.490533</td>\n",
       "      <td>4.941833</td>\n",
       "      <td>4.467233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.801973</td>\n",
       "      <td>0.619967</td>\n",
       "      <td>0.520563</td>\n",
       "      <td>319.333333</td>\n",
       "      <td>318.333333</td>\n",
       "      <td>0.006041</td>\n",
       "      <td>0.000104</td>\n",
       "      <td>0.002217</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>...</td>\n",
       "      <td>1.581967</td>\n",
       "      <td>41.129400</td>\n",
       "      <td>31.201933</td>\n",
       "      <td>14.584467</td>\n",
       "      <td>5.446800</td>\n",
       "      <td>3.462000</td>\n",
       "      <td>4.772067</td>\n",
       "      <td>9.176633</td>\n",
       "      <td>11.848100</td>\n",
       "      <td>5.552367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.828707</td>\n",
       "      <td>0.626097</td>\n",
       "      <td>0.537183</td>\n",
       "      <td>493.000000</td>\n",
       "      <td>492.000000</td>\n",
       "      <td>0.003913</td>\n",
       "      <td>0.000042</td>\n",
       "      <td>0.000757</td>\n",
       "      <td>0.000003</td>\n",
       "      <td>...</td>\n",
       "      <td>2.382533</td>\n",
       "      <td>1.677633</td>\n",
       "      <td>1.908400</td>\n",
       "      <td>2.842167</td>\n",
       "      <td>3.493867</td>\n",
       "      <td>3.282433</td>\n",
       "      <td>3.085267</td>\n",
       "      <td>3.184433</td>\n",
       "      <td>4.032933</td>\n",
       "      <td>22.773633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.831287</td>\n",
       "      <td>0.779397</td>\n",
       "      <td>0.726717</td>\n",
       "      <td>362.666667</td>\n",
       "      <td>361.666667</td>\n",
       "      <td>0.005622</td>\n",
       "      <td>0.002023</td>\n",
       "      <td>0.003593</td>\n",
       "      <td>0.000021</td>\n",
       "      <td>...</td>\n",
       "      <td>3.881267</td>\n",
       "      <td>4.104600</td>\n",
       "      <td>4.285233</td>\n",
       "      <td>2.953200</td>\n",
       "      <td>2.799933</td>\n",
       "      <td>2.645100</td>\n",
       "      <td>2.811367</td>\n",
       "      <td>7.268333</td>\n",
       "      <td>13.338833</td>\n",
       "      <td>63.766900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 753 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    gender       PPE       DFA      RPDE   numPulses  numPeriodsPulses  \\\n",
       "id                                                                       \n",
       "0      1.0  0.823387  0.696370  0.567250  235.333333        234.333333   \n",
       "1      0.0  0.415637  0.793993  0.592453  213.333333        211.000000   \n",
       "2      1.0  0.801973  0.619967  0.520563  319.333333        318.333333   \n",
       "3      0.0  0.828707  0.626097  0.537183  493.000000        492.000000   \n",
       "4      0.0  0.831287  0.779397  0.726717  362.666667        361.666667   \n",
       "\n",
       "    meanPeriodPulses  stdDevPeriodPulses  locPctJitter  locAbsJitter  \\\n",
       "id                                                                     \n",
       "0           0.008220            0.000073      0.001963      0.000016   \n",
       "1           0.008884            0.001849      0.005790      0.000050   \n",
       "2           0.006041            0.000104      0.002217      0.000013   \n",
       "3           0.003913            0.000042      0.000757      0.000003   \n",
       "4           0.005622            0.002023      0.003593      0.000021   \n",
       "\n",
       "              ...              tqwt_kurtosisValue_dec_27  \\\n",
       "id            ...                                          \n",
       "0             ...                               1.546500   \n",
       "1             ...                               7.049367   \n",
       "2             ...                               1.581967   \n",
       "3             ...                               2.382533   \n",
       "4             ...                               3.881267   \n",
       "\n",
       "    tqwt_kurtosisValue_dec_28  tqwt_kurtosisValue_dec_29  \\\n",
       "id                                                         \n",
       "0                    1.561733                   2.862000   \n",
       "1                    4.918567                   4.827133   \n",
       "2                   41.129400                  31.201933   \n",
       "3                    1.677633                   1.908400   \n",
       "4                    4.104600                   4.285233   \n",
       "\n",
       "    tqwt_kurtosisValue_dec_30  tqwt_kurtosisValue_dec_31  \\\n",
       "id                                                         \n",
       "0                   12.293333                   9.717500   \n",
       "1                    6.117633                   8.599667   \n",
       "2                   14.584467                   5.446800   \n",
       "3                    2.842167                   3.493867   \n",
       "4                    2.953200                   2.799933   \n",
       "\n",
       "    tqwt_kurtosisValue_dec_32  tqwt_kurtosisValue_dec_33  \\\n",
       "id                                                         \n",
       "0                    9.055300                   6.259100   \n",
       "1                    5.737233                   7.933133   \n",
       "2                    3.462000                   4.772067   \n",
       "3                    3.282433                   3.085267   \n",
       "4                    2.645100                   2.811367   \n",
       "\n",
       "    tqwt_kurtosisValue_dec_34  tqwt_kurtosisValue_dec_35  \\\n",
       "id                                                         \n",
       "0                    4.021533                   4.164333   \n",
       "1                    5.490533                   4.941833   \n",
       "2                    9.176633                  11.848100   \n",
       "3                    3.184433                   4.032933   \n",
       "4                    7.268333                  13.338833   \n",
       "\n",
       "    tqwt_kurtosisValue_dec_36  \n",
       "id                             \n",
       "0                   22.961700  \n",
       "1                    4.467233  \n",
       "2                    5.552367  \n",
       "3                   22.773633  \n",
       "4                   63.766900  \n",
       "\n",
       "[5 rows x 753 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df.groupby('id').mean().drop(['class'], axis=1)\n",
    "y = df[['class', 'id']].groupby('id').mean()['class']\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    188\n",
       "0     64\n",
       "Name: class, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_origin, X_test_origin, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=91)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>PPE</th>\n",
       "      <th>DFA</th>\n",
       "      <th>RPDE</th>\n",
       "      <th>numPulses</th>\n",
       "      <th>numPeriodsPulses</th>\n",
       "      <th>meanPeriodPulses</th>\n",
       "      <th>stdDevPeriodPulses</th>\n",
       "      <th>locPctJitter</th>\n",
       "      <th>locAbsJitter</th>\n",
       "      <th>...</th>\n",
       "      <th>tqwt_kurtosisValue_dec_27</th>\n",
       "      <th>tqwt_kurtosisValue_dec_28</th>\n",
       "      <th>tqwt_kurtosisValue_dec_29</th>\n",
       "      <th>tqwt_kurtosisValue_dec_30</th>\n",
       "      <th>tqwt_kurtosisValue_dec_31</th>\n",
       "      <th>tqwt_kurtosisValue_dec_32</th>\n",
       "      <th>tqwt_kurtosisValue_dec_33</th>\n",
       "      <th>tqwt_kurtosisValue_dec_34</th>\n",
       "      <th>tqwt_kurtosisValue_dec_35</th>\n",
       "      <th>tqwt_kurtosisValue_dec_36</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.805553</td>\n",
       "      <td>0.648220</td>\n",
       "      <td>0.377633</td>\n",
       "      <td>281.666667</td>\n",
       "      <td>280.666667</td>\n",
       "      <td>0.006858</td>\n",
       "      <td>0.000056</td>\n",
       "      <td>0.001410</td>\n",
       "      <td>0.000010</td>\n",
       "      <td>...</td>\n",
       "      <td>1.651067</td>\n",
       "      <td>5.498767</td>\n",
       "      <td>65.421367</td>\n",
       "      <td>31.112067</td>\n",
       "      <td>12.535600</td>\n",
       "      <td>11.071067</td>\n",
       "      <td>20.329600</td>\n",
       "      <td>25.156433</td>\n",
       "      <td>29.161867</td>\n",
       "      <td>17.512500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.555797</td>\n",
       "      <td>0.678520</td>\n",
       "      <td>0.404007</td>\n",
       "      <td>385.666667</td>\n",
       "      <td>384.666667</td>\n",
       "      <td>0.005123</td>\n",
       "      <td>0.000797</td>\n",
       "      <td>0.001317</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>...</td>\n",
       "      <td>7.057133</td>\n",
       "      <td>9.863533</td>\n",
       "      <td>22.837800</td>\n",
       "      <td>52.240733</td>\n",
       "      <td>32.964733</td>\n",
       "      <td>16.221933</td>\n",
       "      <td>20.626933</td>\n",
       "      <td>34.765767</td>\n",
       "      <td>30.866033</td>\n",
       "      <td>79.740933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>194</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.832080</td>\n",
       "      <td>0.735363</td>\n",
       "      <td>0.253373</td>\n",
       "      <td>520.000000</td>\n",
       "      <td>519.000000</td>\n",
       "      <td>0.003712</td>\n",
       "      <td>0.000022</td>\n",
       "      <td>0.000497</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>...</td>\n",
       "      <td>45.074933</td>\n",
       "      <td>15.350233</td>\n",
       "      <td>12.990033</td>\n",
       "      <td>9.733567</td>\n",
       "      <td>6.357567</td>\n",
       "      <td>7.357533</td>\n",
       "      <td>7.033400</td>\n",
       "      <td>5.928733</td>\n",
       "      <td>5.633867</td>\n",
       "      <td>5.366900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.869927</td>\n",
       "      <td>0.786747</td>\n",
       "      <td>0.369487</td>\n",
       "      <td>399.666667</td>\n",
       "      <td>398.666667</td>\n",
       "      <td>0.004832</td>\n",
       "      <td>0.000049</td>\n",
       "      <td>0.001263</td>\n",
       "      <td>0.000006</td>\n",
       "      <td>...</td>\n",
       "      <td>1.562733</td>\n",
       "      <td>1.574833</td>\n",
       "      <td>3.143200</td>\n",
       "      <td>10.138533</td>\n",
       "      <td>6.346167</td>\n",
       "      <td>3.371900</td>\n",
       "      <td>4.953467</td>\n",
       "      <td>8.518333</td>\n",
       "      <td>8.027033</td>\n",
       "      <td>42.470633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.529193</td>\n",
       "      <td>0.774857</td>\n",
       "      <td>0.712680</td>\n",
       "      <td>237.000000</td>\n",
       "      <td>233.000000</td>\n",
       "      <td>0.008029</td>\n",
       "      <td>0.001171</td>\n",
       "      <td>0.007440</td>\n",
       "      <td>0.000064</td>\n",
       "      <td>...</td>\n",
       "      <td>5.192333</td>\n",
       "      <td>4.354967</td>\n",
       "      <td>4.360400</td>\n",
       "      <td>5.761733</td>\n",
       "      <td>4.999867</td>\n",
       "      <td>5.014533</td>\n",
       "      <td>4.847633</td>\n",
       "      <td>5.307533</td>\n",
       "      <td>6.127767</td>\n",
       "      <td>4.733533</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 753 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     gender       PPE       DFA      RPDE   numPulses  numPeriodsPulses  \\\n",
       "id                                                                        \n",
       "44      1.0  0.805553  0.648220  0.377633  281.666667        280.666667   \n",
       "59      0.0  0.555797  0.678520  0.404007  385.666667        384.666667   \n",
       "194     0.0  0.832080  0.735363  0.253373  520.000000        519.000000   \n",
       "14      0.0  0.869927  0.786747  0.369487  399.666667        398.666667   \n",
       "7       1.0  0.529193  0.774857  0.712680  237.000000        233.000000   \n",
       "\n",
       "     meanPeriodPulses  stdDevPeriodPulses  locPctJitter  locAbsJitter  \\\n",
       "id                                                                      \n",
       "44           0.006858            0.000056      0.001410      0.000010   \n",
       "59           0.005123            0.000797      0.001317      0.000007   \n",
       "194          0.003712            0.000022      0.000497      0.000002   \n",
       "14           0.004832            0.000049      0.001263      0.000006   \n",
       "7            0.008029            0.001171      0.007440      0.000064   \n",
       "\n",
       "               ...              tqwt_kurtosisValue_dec_27  \\\n",
       "id             ...                                          \n",
       "44             ...                               1.651067   \n",
       "59             ...                               7.057133   \n",
       "194            ...                              45.074933   \n",
       "14             ...                               1.562733   \n",
       "7              ...                               5.192333   \n",
       "\n",
       "     tqwt_kurtosisValue_dec_28  tqwt_kurtosisValue_dec_29  \\\n",
       "id                                                          \n",
       "44                    5.498767                  65.421367   \n",
       "59                    9.863533                  22.837800   \n",
       "194                  15.350233                  12.990033   \n",
       "14                    1.574833                   3.143200   \n",
       "7                     4.354967                   4.360400   \n",
       "\n",
       "     tqwt_kurtosisValue_dec_30  tqwt_kurtosisValue_dec_31  \\\n",
       "id                                                          \n",
       "44                   31.112067                  12.535600   \n",
       "59                   52.240733                  32.964733   \n",
       "194                   9.733567                   6.357567   \n",
       "14                   10.138533                   6.346167   \n",
       "7                     5.761733                   4.999867   \n",
       "\n",
       "     tqwt_kurtosisValue_dec_32  tqwt_kurtosisValue_dec_33  \\\n",
       "id                                                          \n",
       "44                   11.071067                  20.329600   \n",
       "59                   16.221933                  20.626933   \n",
       "194                   7.357533                   7.033400   \n",
       "14                    3.371900                   4.953467   \n",
       "7                     5.014533                   4.847633   \n",
       "\n",
       "     tqwt_kurtosisValue_dec_34  tqwt_kurtosisValue_dec_35  \\\n",
       "id                                                          \n",
       "44                   25.156433                  29.161867   \n",
       "59                   34.765767                  30.866033   \n",
       "194                   5.928733                   5.633867   \n",
       "14                    8.518333                   8.027033   \n",
       "7                     5.307533                   6.127767   \n",
       "\n",
       "     tqwt_kurtosisValue_dec_36  \n",
       "id                              \n",
       "44                   17.512500  \n",
       "59                   79.740933  \n",
       "194                   5.366900  \n",
       "14                   42.470633  \n",
       "7                     4.733533  \n",
       "\n",
       "[5 rows x 753 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_origin.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Min-Max Normalization\n",
    "For this dataset, min-max normalization, rather than z-score standardization, is used to scale the columns, because after testing both scaling methods, min-max normalization generates better prediction results in general than z-score standardization. (Specifically after applying PCA in part III.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>PPE</th>\n",
       "      <th>DFA</th>\n",
       "      <th>RPDE</th>\n",
       "      <th>numPulses</th>\n",
       "      <th>numPeriodsPulses</th>\n",
       "      <th>meanPeriodPulses</th>\n",
       "      <th>stdDevPeriodPulses</th>\n",
       "      <th>locPctJitter</th>\n",
       "      <th>locAbsJitter</th>\n",
       "      <th>...</th>\n",
       "      <th>tqwt_kurtosisValue_dec_27</th>\n",
       "      <th>tqwt_kurtosisValue_dec_28</th>\n",
       "      <th>tqwt_kurtosisValue_dec_29</th>\n",
       "      <th>tqwt_kurtosisValue_dec_30</th>\n",
       "      <th>tqwt_kurtosisValue_dec_31</th>\n",
       "      <th>tqwt_kurtosisValue_dec_32</th>\n",
       "      <th>tqwt_kurtosisValue_dec_33</th>\n",
       "      <th>tqwt_kurtosisValue_dec_34</th>\n",
       "      <th>tqwt_kurtosisValue_dec_35</th>\n",
       "      <th>tqwt_kurtosisValue_dec_36</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.884133</td>\n",
       "      <td>0.312568</td>\n",
       "      <td>0.290505</td>\n",
       "      <td>0.375000</td>\n",
       "      <td>0.386113</td>\n",
       "      <td>0.430159</td>\n",
       "      <td>0.015896</td>\n",
       "      <td>0.083782</td>\n",
       "      <td>0.089808</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000645</td>\n",
       "      <td>0.021381</td>\n",
       "      <td>0.441067</td>\n",
       "      <td>0.310147</td>\n",
       "      <td>0.138180</td>\n",
       "      <td>0.140455</td>\n",
       "      <td>0.298894</td>\n",
       "      <td>0.419466</td>\n",
       "      <td>0.588747</td>\n",
       "      <td>0.159022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.434591</td>\n",
       "      <td>0.415229</td>\n",
       "      <td>0.332193</td>\n",
       "      <td>0.577073</td>\n",
       "      <td>0.588579</td>\n",
       "      <td>0.221931</td>\n",
       "      <td>0.294102</td>\n",
       "      <td>0.076923</td>\n",
       "      <td>0.063297</td>\n",
       "      <td>...</td>\n",
       "      <td>0.024818</td>\n",
       "      <td>0.044873</td>\n",
       "      <td>0.146870</td>\n",
       "      <td>0.534226</td>\n",
       "      <td>0.418453</td>\n",
       "      <td>0.219766</td>\n",
       "      <td>0.303777</td>\n",
       "      <td>0.596052</td>\n",
       "      <td>0.626075</td>\n",
       "      <td>0.835806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.931879</td>\n",
       "      <td>0.607822</td>\n",
       "      <td>0.094084</td>\n",
       "      <td>0.838083</td>\n",
       "      <td>0.850097</td>\n",
       "      <td>0.052751</td>\n",
       "      <td>0.003006</td>\n",
       "      <td>0.016659</td>\n",
       "      <td>0.009525</td>\n",
       "      <td>...</td>\n",
       "      <td>0.194813</td>\n",
       "      <td>0.074404</td>\n",
       "      <td>0.078835</td>\n",
       "      <td>0.083418</td>\n",
       "      <td>0.053422</td>\n",
       "      <td>0.083276</td>\n",
       "      <td>0.080533</td>\n",
       "      <td>0.066127</td>\n",
       "      <td>0.073387</td>\n",
       "      <td>0.026930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.781916</td>\n",
       "      <td>0.277627</td>\n",
       "      <td>0.604275</td>\n",
       "      <td>0.615834</td>\n",
       "      <td>0.187128</td>\n",
       "      <td>0.013128</td>\n",
       "      <td>0.073003</td>\n",
       "      <td>0.053288</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000250</td>\n",
       "      <td>0.000262</td>\n",
       "      <td>0.010806</td>\n",
       "      <td>0.087713</td>\n",
       "      <td>0.053265</td>\n",
       "      <td>0.021906</td>\n",
       "      <td>0.046374</td>\n",
       "      <td>0.113715</td>\n",
       "      <td>0.125807</td>\n",
       "      <td>0.430462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.386707</td>\n",
       "      <td>0.741631</td>\n",
       "      <td>0.820119</td>\n",
       "      <td>0.288212</td>\n",
       "      <td>0.293316</td>\n",
       "      <td>0.570659</td>\n",
       "      <td>0.434922</td>\n",
       "      <td>0.526948</td>\n",
       "      <td>0.647352</td>\n",
       "      <td>...</td>\n",
       "      <td>0.016480</td>\n",
       "      <td>0.015225</td>\n",
       "      <td>0.019215</td>\n",
       "      <td>0.041295</td>\n",
       "      <td>0.034795</td>\n",
       "      <td>0.047199</td>\n",
       "      <td>0.044636</td>\n",
       "      <td>0.054711</td>\n",
       "      <td>0.084205</td>\n",
       "      <td>0.020041</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 753 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  gender       PPE       DFA      RPDE  numPulses  numPeriodsPulses  \\\n",
       "0    1.0  0.884133  0.312568  0.290505   0.375000          0.386113   \n",
       "1    0.0  0.434591  0.415229  0.332193   0.577073          0.588579   \n",
       "2    0.0  0.931879  0.607822  0.094084   0.838083          0.850097   \n",
       "3    0.0  1.000000  0.781916  0.277627   0.604275          0.615834   \n",
       "4    1.0  0.386707  0.741631  0.820119   0.288212          0.293316   \n",
       "\n",
       "   meanPeriodPulses  stdDevPeriodPulses  locPctJitter  locAbsJitter  \\\n",
       "0          0.430159            0.015896      0.083782      0.089808   \n",
       "1          0.221931            0.294102      0.076923      0.063297   \n",
       "2          0.052751            0.003006      0.016659      0.009525   \n",
       "3          0.187128            0.013128      0.073003      0.053288   \n",
       "4          0.570659            0.434922      0.526948      0.647352   \n",
       "\n",
       "             ...              tqwt_kurtosisValue_dec_27  \\\n",
       "0            ...                               0.000645   \n",
       "1            ...                               0.024818   \n",
       "2            ...                               0.194813   \n",
       "3            ...                               0.000250   \n",
       "4            ...                               0.016480   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_28  tqwt_kurtosisValue_dec_29  \\\n",
       "0                   0.021381                   0.441067   \n",
       "1                   0.044873                   0.146870   \n",
       "2                   0.074404                   0.078835   \n",
       "3                   0.000262                   0.010806   \n",
       "4                   0.015225                   0.019215   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_30  tqwt_kurtosisValue_dec_31  \\\n",
       "0                   0.310147                   0.138180   \n",
       "1                   0.534226                   0.418453   \n",
       "2                   0.083418                   0.053422   \n",
       "3                   0.087713                   0.053265   \n",
       "4                   0.041295                   0.034795   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_32  tqwt_kurtosisValue_dec_33  \\\n",
       "0                   0.140455                   0.298894   \n",
       "1                   0.219766                   0.303777   \n",
       "2                   0.083276                   0.080533   \n",
       "3                   0.021906                   0.046374   \n",
       "4                   0.047199                   0.044636   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_34  tqwt_kurtosisValue_dec_35  \\\n",
       "0                   0.419466                   0.588747   \n",
       "1                   0.596052                   0.626075   \n",
       "2                   0.066127                   0.073387   \n",
       "3                   0.113715                   0.125807   \n",
       "4                   0.054711                   0.084205   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_36  \n",
       "0                   0.159022  \n",
       "1                   0.835806  \n",
       "2                   0.026930  \n",
       "3                   0.430462  \n",
       "4                   0.020041  \n",
       "\n",
       "[5 rows x 753 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler = MinMaxScaler()\n",
    "scaler.fit(X_train_origin)\n",
    "X_train = pd.DataFrame(scaler.transform(X_train_origin))\n",
    "X_train.columns = X_train_origin.columns\n",
    "X_train['gender'] = X_train['gender'].astype('category')\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>PPE</th>\n",
       "      <th>DFA</th>\n",
       "      <th>RPDE</th>\n",
       "      <th>numPulses</th>\n",
       "      <th>numPeriodsPulses</th>\n",
       "      <th>meanPeriodPulses</th>\n",
       "      <th>stdDevPeriodPulses</th>\n",
       "      <th>locPctJitter</th>\n",
       "      <th>locAbsJitter</th>\n",
       "      <th>...</th>\n",
       "      <th>tqwt_kurtosisValue_dec_27</th>\n",
       "      <th>tqwt_kurtosisValue_dec_28</th>\n",
       "      <th>tqwt_kurtosisValue_dec_29</th>\n",
       "      <th>tqwt_kurtosisValue_dec_30</th>\n",
       "      <th>tqwt_kurtosisValue_dec_31</th>\n",
       "      <th>tqwt_kurtosisValue_dec_32</th>\n",
       "      <th>tqwt_kurtosisValue_dec_33</th>\n",
       "      <th>tqwt_kurtosisValue_dec_34</th>\n",
       "      <th>tqwt_kurtosisValue_dec_35</th>\n",
       "      <th>tqwt_kurtosisValue_dec_36</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.383671</td>\n",
       "      <td>0.533633</td>\n",
       "      <td>0.978892</td>\n",
       "      <td>0.086140</td>\n",
       "      <td>0.083063</td>\n",
       "      <td>0.567523</td>\n",
       "      <td>0.343307</td>\n",
       "      <td>1.110240</td>\n",
       "      <td>1.228684</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000172</td>\n",
       "      <td>0.000548</td>\n",
       "      <td>0.442600</td>\n",
       "      <td>0.718863</td>\n",
       "      <td>0.760094</td>\n",
       "      <td>0.681884</td>\n",
       "      <td>0.638935</td>\n",
       "      <td>0.673738</td>\n",
       "      <td>0.710756</td>\n",
       "      <td>0.766840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.763082</td>\n",
       "      <td>0.586206</td>\n",
       "      <td>0.465564</td>\n",
       "      <td>0.925518</td>\n",
       "      <td>0.937703</td>\n",
       "      <td>0.017372</td>\n",
       "      <td>0.074079</td>\n",
       "      <td>0.217785</td>\n",
       "      <td>0.102961</td>\n",
       "      <td>...</td>\n",
       "      <td>0.067715</td>\n",
       "      <td>0.062257</td>\n",
       "      <td>0.034724</td>\n",
       "      <td>0.027647</td>\n",
       "      <td>0.018053</td>\n",
       "      <td>0.032739</td>\n",
       "      <td>0.104072</td>\n",
       "      <td>0.163105</td>\n",
       "      <td>0.182861</td>\n",
       "      <td>0.580103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.515107</td>\n",
       "      <td>0.044193</td>\n",
       "      <td>0.344808</td>\n",
       "      <td>0.609456</td>\n",
       "      <td>0.620376</td>\n",
       "      <td>0.180199</td>\n",
       "      <td>0.033295</td>\n",
       "      <td>0.052425</td>\n",
       "      <td>0.038495</td>\n",
       "      <td>...</td>\n",
       "      <td>0.043537</td>\n",
       "      <td>0.006181</td>\n",
       "      <td>0.007842</td>\n",
       "      <td>0.008888</td>\n",
       "      <td>0.002388</td>\n",
       "      <td>0.028748</td>\n",
       "      <td>0.021211</td>\n",
       "      <td>0.033425</td>\n",
       "      <td>0.075417</td>\n",
       "      <td>0.115032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.315862</td>\n",
       "      <td>0.325126</td>\n",
       "      <td>0.702071</td>\n",
       "      <td>0.279145</td>\n",
       "      <td>0.290071</td>\n",
       "      <td>0.608980</td>\n",
       "      <td>0.027366</td>\n",
       "      <td>0.158501</td>\n",
       "      <td>0.202478</td>\n",
       "      <td>...</td>\n",
       "      <td>0.002921</td>\n",
       "      <td>0.002929</td>\n",
       "      <td>0.017553</td>\n",
       "      <td>0.052975</td>\n",
       "      <td>0.069480</td>\n",
       "      <td>0.029004</td>\n",
       "      <td>0.014975</td>\n",
       "      <td>0.021460</td>\n",
       "      <td>0.026557</td>\n",
       "      <td>0.155453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.163061</td>\n",
       "      <td>0.606264</td>\n",
       "      <td>0.480797</td>\n",
       "      <td>0.456606</td>\n",
       "      <td>0.467878</td>\n",
       "      <td>0.339511</td>\n",
       "      <td>0.576415</td>\n",
       "      <td>0.109995</td>\n",
       "      <td>0.107368</td>\n",
       "      <td>...</td>\n",
       "      <td>0.150706</td>\n",
       "      <td>0.050214</td>\n",
       "      <td>0.031369</td>\n",
       "      <td>0.034521</td>\n",
       "      <td>0.043515</td>\n",
       "      <td>0.053383</td>\n",
       "      <td>0.051387</td>\n",
       "      <td>0.315965</td>\n",
       "      <td>0.413887</td>\n",
       "      <td>0.448506</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 753 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  gender       PPE       DFA      RPDE  numPulses  numPeriodsPulses  \\\n",
       "0    1.0  0.383671  0.533633  0.978892   0.086140          0.083063   \n",
       "1    1.0  0.763082  0.586206  0.465564   0.925518          0.937703   \n",
       "2    0.0  0.515107  0.044193  0.344808   0.609456          0.620376   \n",
       "3    1.0  0.315862  0.325126  0.702071   0.279145          0.290071   \n",
       "4    0.0  0.163061  0.606264  0.480797   0.456606          0.467878   \n",
       "\n",
       "   meanPeriodPulses  stdDevPeriodPulses  locPctJitter  locAbsJitter  \\\n",
       "0          0.567523            0.343307      1.110240      1.228684   \n",
       "1          0.017372            0.074079      0.217785      0.102961   \n",
       "2          0.180199            0.033295      0.052425      0.038495   \n",
       "3          0.608980            0.027366      0.158501      0.202478   \n",
       "4          0.339511            0.576415      0.109995      0.107368   \n",
       "\n",
       "             ...              tqwt_kurtosisValue_dec_27  \\\n",
       "0            ...                               0.000172   \n",
       "1            ...                               0.067715   \n",
       "2            ...                               0.043537   \n",
       "3            ...                               0.002921   \n",
       "4            ...                               0.150706   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_28  tqwt_kurtosisValue_dec_29  \\\n",
       "0                   0.000548                   0.442600   \n",
       "1                   0.062257                   0.034724   \n",
       "2                   0.006181                   0.007842   \n",
       "3                   0.002929                   0.017553   \n",
       "4                   0.050214                   0.031369   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_30  tqwt_kurtosisValue_dec_31  \\\n",
       "0                   0.718863                   0.760094   \n",
       "1                   0.027647                   0.018053   \n",
       "2                   0.008888                   0.002388   \n",
       "3                   0.052975                   0.069480   \n",
       "4                   0.034521                   0.043515   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_32  tqwt_kurtosisValue_dec_33  \\\n",
       "0                   0.681884                   0.638935   \n",
       "1                   0.032739                   0.104072   \n",
       "2                   0.028748                   0.021211   \n",
       "3                   0.029004                   0.014975   \n",
       "4                   0.053383                   0.051387   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_34  tqwt_kurtosisValue_dec_35  \\\n",
       "0                   0.673738                   0.710756   \n",
       "1                   0.163105                   0.182861   \n",
       "2                   0.033425                   0.075417   \n",
       "3                   0.021460                   0.026557   \n",
       "4                   0.315965                   0.413887   \n",
       "\n",
       "   tqwt_kurtosisValue_dec_36  \n",
       "0                   0.766840  \n",
       "1                   0.580103  \n",
       "2                   0.115032  \n",
       "3                   0.155453  \n",
       "4                   0.448506  \n",
       "\n",
       "[5 rows x 753 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test = pd.DataFrame(scaler.transform(X_test_origin[X_test_origin.columns]))\n",
    "X_test.columns = X_test_origin.columns\n",
    "X_test['gender'] = X_test['gender'].astype('category')\n",
    "X_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    150\n",
       "0     51\n",
       "Name: class, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    38\n",
       "0    13\n",
       "Name: class, dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem Definition\n",
    "**Input:** $X \\in \\mathbb{R}^{d}, d = 243$\n",
    "\n",
    "**Output:** $Y = \\{0, 1\\}$\n",
    "\n",
    "**Classifier:** Calssification uses a function $f$ (called a classifier) to map input $x$ to class $y$. \n",
    "\n",
    "$y = f(x) : f$ takes in $x \\in X$ and declares its class to be $y \\in Y$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II. Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. k-Nearest Neighbors Classifier (kNN)\n",
    "KNNs classify the unseen instance based on the K points in the training set which are nearest to it. It is a **non-parametric method**. \n",
    "\n",
    "#### Non-parametric Models\n",
    "Non-parametric models differ from parametric models in that the model structure is not specified a priori but is instead determined from data. The term non-parametric is not meant to imply that such models completely lack parameters but that the number and nature of the parameters are flexible and not fixed in advance.\n",
    "\n",
    "Source: https://en.wikipedia.org/wiki/Nonparametric_statistics#Non-parametric_models\n",
    "\n",
    "#### Algorithm\n",
    "Given data $(x_1, y_1),...,(x_n, y_n)$, construct the $k$-NN classifier as follows:\n",
    "For a new input $s$,\n",
    "1. Return the $k$ points closest to $x$, indexed as $x_{i_1},...x_{i_k}$.\n",
    "2. Return the majority-vote of $y_{i_1}, y_{i_2},..., y_{i_k}$.\n",
    "The default distance for data in $\\mathbb{R}^d$ is the Euclidean one:\n",
    "$$\\|u-v\\|_2 = \\big(\\sum_{i=1}^{d}(u_i-v_i)^2\\big)^\\frac{1}{2}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Number of Neighbors: {'n_neighbors': 10}\n",
      "Accuracy on Training Set: 0.825870646766\n",
      "Accuracy on Test Set: 0.803921568627\n",
      "AUC: 0.839068825911\n"
     ]
    }
   ],
   "source": [
    "param_grid = {'n_neighbors': np.arange(3, 13)}\n",
    "gs_kNN = GridSearchCV(KNeighborsClassifier(), param_grid, cv=5)\n",
    "gs_kNN.fit(X_train, y_train)\n",
    "print(\"Best Number of Neighbors:\", gs_kNN.best_params_)\n",
    "print(\"Accuracy on Training Set:\", gs_kNN.best_score_)\n",
    "\n",
    "y_pred_prob = gs_kNN.predict_proba(X_test)[:,1]\n",
    "print(\"Accuracy on Test Set:\", gs_kNN.score(X_test, y_test))\n",
    "print(\"AUC:\", roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Observe how the accuracy change as k grows\n",
    "Note: The accuracy is different from the results from grid search because the whole training set is used to fit the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute training and test errors by k\n",
    "training_error = list()\n",
    "test_error = list()\n",
    "for k in np.arange(3, 13):\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    knn.fit(X_train, y_train)\n",
    "    training_error.append(knn.score(X_train, y_train))\n",
    "    test_error.append(knn.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3XlYVdX6wPHvYnYAHMAJnEdmRETNnCecUslKbTDLqbS6Ntzs5q/BhtvNunUd0szKBtNMc8qxnMpZHFARUZwRB1QUEJnX748FhIpyhAP7cFif5/GRc84+e78gvmfvd6/1LiGlRNM0TSsfbIwOQNM0TSs9OulrmqaVIzrpa5qmlSM66WuappUjOulrmqaVIzrpa5qmlSM66WuappUjOulrmqaVIzrpa5qmlSN2RgdwOzc3N9mgQQOjw9A0TStT9uzZc1lK6V7YdhaX9Bs0aEB4eLjRYWiappUpQojTpmynyzuapmnliE76mqZp5YhO+pqmaeWIxdX0Na28yMjIIDY2ltTUVKND0coQJycnPD09sbe3L9L7ddLXNIPExsbi7OxMgwYNEEIYHY5WBkgpuXLlCrGxsTRs2LBI+zCpvCOECBVCRAshYoQQEwt4vb4QYr0Q4oAQYpMQwjPfa8OFEMdy/gwvUpSaZoVSU1OpXr26TviayYQQVK9evVhXh4UmfSGELTAD6A14A0OFEN63bfYJ8L2U0h+YDPw7573VgLeBNkAI8LYQomqRo9U0K6MTvna/ivs7Y8qZfggQI6U8IaVMBxYAA27bxhtYn/P1xnyv9wJ+l1JelVImAL8DocWK+C6ysyUfrorizJWUkti9pmmaVTAl6XsAZ/M9js15Lr8I4OGcrwcBzkKI6ia+1yxOXbnBgl1n6Dv1L1YfPF8Sh9A0q3HlyhUCAwMJDAykVq1aeHh45D1OT083aR8jRowgOjr6ntvMmDGDefPmmSPk+7JhwwZ27NhR6sctC0y5kVvQtcTtq6m/CkwXQjwN/AmcAzJNfC9CiNHAaIB69eqZENKdGrlXZuWLHRg/fx/PzdvL8Hb1+VdfLxztbIu0P02zZtWrV2f//v0AvPPOO1SuXJlXX331lm2klEgpsbEp+Nzw22+/LfQ448aNK36wRbBhwwbc3Nxo27atIcfPlZWVha2tZeUgU870Y4G6+R57AnH5N5BSxkkpw6SULYE3c567bsp7c7adLaUMllIGu7sX2jrirupWq8gvY9ox8sGGfLf9NINnbuf0lRtF3p+mlTcxMTH4+voyduxYgoKCOH/+PKNHjyY4OBgfHx8mT56ct+2DDz7I/v37yczMpEqVKkycOJGAgADatWvHpUuXAJg0aRKff/553vYTJ04kJCSE5s2bs23bNgBu3LjBww8/TEBAAEOHDiU4ODjvAym/1157DW9vb/z9/Xn99dcBuHjxImFhYQQHBxMSEsKOHTs4fvw4c+bMYcqUKQQGBuYdJ9eOHTto164dLVu2pH379hw7dgyAzMxMJkyYgK+vL/7+/nzxxRcA7Ny5k3bt2hEQEECbNm1ISUlhzpw5/OMf/8jbZ2hoKFu2bMn7WUyaNImQkBB27drF22+/TevWrfN+rlKq896jR4/StWtXAgICCAoK4tSpUwwdOpSVK1fm7fexxx5j1apVxftHvY0pZ/q7gaZCiIaoM/ghwLD8Gwgh3ICrUsps4A3gm5yX1gIf5rt52zPn9RLjYGfDpH7etGlUnVd/iaDf1C38Z7A/ffxql+RhNa1Y3l0RyeG4RLPu07uOC2/397nv9x0+fJhvv/2WWbNmAfDRRx9RrVo1MjMz6dKlC4MHD8bb+9axHNevX6dTp0589NFHvPzyy3zzzTdMnHjHQD+klOzatYvly5czefJk1qxZw7Rp06hVqxaLFy8mIiKCoKCgO9538eJFVq1aRWRkJEIIrl27BsCLL77IP//5T9q2bcupU6fo168fhw4dYuTIkbi5ud2SmHN5eXmxZcsWbG1tWbNmDZMmTeLnn39m5syZxMXFERERga2tLVevXiU1NZUhQ4awePFigoKCuH79Oo6Ojvf8+V2/fp2goCDef/99AJo3b867776LlJJhw4axZs0aevfuzdChQ3nnnXfo378/qampZGdnM3LkSGbOnEnfvn1JSEhg9+7d/PTTT6b9w5mo0DN9KWUmMB6VwKOAhVLKSCHEZCHEQzmbdQaihRBHgZrABznvvQq8h/rg2A1MznmuxPXwrsnKFx+kcY3KPD9vL28tO0RqRlZpHFrTyrTGjRvTunXrvMfz588nKCiIoKAgoqKiOHz48B3vqVChAr179wagVatWnDp1qsB9h4WF3bHNli1bGDJkCAABAQH4+Nz5QVWtWjVsbGwYNWoUS5YsoVKlSgD88ccfjB07lsDAQAYOHEhCQgI3b9685/d37do1wsLC8PX15dVXXyUyMvKWfeWWY6pVq0ZUVBT16tXL+yBydXUttFzj4ODAoEGD8h6vX7+ekJAQAgIC2Lx5M5GRkSQkJHD58mX69+8PqAlXFStWpGvXrhw+fJgrV64wb948Hn30UbOXh0yanCWlXAWsuu25t/J9vQhYdJf3fsPfZ/6lyrNqRRaOaceUtUf46q+T7DmdwIxhQTRwq2REOJp2V0U5Iy8puQkV4NixY/zvf/9j165dVKlShSeeeKLAMeIODg55X9va2pKZmVngvnPPkvNvk1vuuBd7e3vCw8P5/fffWbBgATNnzmTdunV5Vw75j1+YN998k169evH8888TExNDaGhoXhy3D4cs6DkAOzs7srOz8x7n/5lUqFAh7z0pKSmMHz+evXv34uHhwaRJk/K2LWi/Qggef/xxfvrpJ+bOnWv2s3woB713HOxseLOvN3OeCiY24Sb9pm3htwN33FbQNK0AiYmJODs74+Liwvnz51m7dq3Zj/Hggw+ycOFCAA4ePFjglURSUhKJiYn069ePzz77jH379gHQvXt3ZsyYkbdd7r0AZ2dnkpKSCjze9evX8fBQgwjnzp2b93zPnj2ZOXMmWVmqInD16lV8fHw4ffo0e/fuBdTPIysriwYNGrBv3z6klJw6dYo9e/YUeKybN29iY2ODm5sbSUlJLF68GICqVavi5ubGihUrAPWhkZKihpuPGDGCKVOm4OTkRPPmzU34Cd4fq0/6ubp712TVSx1oWrMy43/ax6SlB3W5R9MKERQUhLe3N76+vowaNYr27dub/RgvvPAC586dw9/fn08//RRfX19cXV1v2eb69ev07duXgIAAunbtyn//+19ADQndunUr/v7+eHt789VXXwEwYMAAFi5cSMuWLe+4kfv666/z2muv3fG9jBkzhlq1auHv709AQAALFy7E0dGR+fPn89xzzxEQEEDPnj1JS0ujU6dOeHh44Ofnx8SJEwkMDCzwe6tevTrDhw/H19eXQYMG0aZNm7zX5s2bx6effoq/vz8PPvgg8fHxANSpU4dmzZoxYsSI4v1g70KYcmlVmoKDg2VJLqKSkZXNJ2uj+fLPE3jXdmHG40E01OUezQBRUVF4eXkZHYbhMjMzyczMxMnJiWPHjtGzZ0+OHTuGnV35bA1248YN/Pz8iIiIwNnZucBtCvrdEULskVIGF7b/cnOmn8ve1oY3+njxzdPBxF2/Sf9pW1gRocs9mmaU5ORk2rdvT0BAAA8//DBffvlluU34a9euxcvLiwkTJtw14RdX+fzJAl1b1GTVix14Yf4+Xpi/jx0nrvB//bxxsresiRSaZu2qVKly15p4edOrVy/OnDlToscod2f6+dWpUoEFo9syplMj5u08w6AvtnEiPtnosDRN00pMuU76kFPu6e3Ft0+35kJOuWfZ/nNGh6VpmlYiyn3Sz9WlRQ1WvtgBr9ouvLRgP2/8qkf3aJpmfXTSz6dOlQrMH92W5zo3Zv6uMwycsZXjutyjaZoV0Un/Nva2Nrwe2oJvR7TmYmIq/adtYek+Xe7RrIs5WisDfPPNN1y4cKHY8ezdu5c1a9YUez9a4XTSv4suzWuw6qUO+NRx4R8/72fi4gO63KNZjdzWyvv372fs2LFMmDAh7/H9tDSwtqR/t/YR1kQn/Xuo7VqB+aPa8nznxizYfZaBM7YSc0mXezTr9t133xESEkJgYCDPP/882dnZZGZm8uSTT+Ln54evry9Tp07l559/Zv/+/Tz22GMFXiF89tlneHt7ExAQwBNPPAGoMflPP/00ISEhtGzZkhUrVnDz5k0mT57MvHnzCAwMZNGiW9t4HT9+nA4dOtCyZUtatWrFzp0781778MMP8fPzIyAggDfffBMouGXxH3/8wcCBA/PeN3bsWH788UcAPD09ee+992jfvj1Llixh1qxZtG7dmoCAAB555JG8Bm4XLlxgwIABeTN2d+7cyRtvvHFLG4jXX389ryWzpSq34/RNZWdrwz9DWxDSsBovL4zgoelb+GCQL4Naehb+Zk0z1eqJcOGgefdZyw96f3Rfbzl06BBLlixh27Zt2NnZMXr0aBYsWEDjxo25fPkyBw+qGK9du0aVKlWYNm0a06dPL7ANwccff8zp06dxcHDIa4U8efJkQkNDmTt3LgkJCbRp04YDBw7w1ltvcejQobze+/nVrl2b33//HScnJ44cOcLw4cPZuXMnK1asYPXq1ezatYsKFSpw9apq4FtQy+KYmJh7ft+VKlVi69atgCp9jR07FoCJEycyd+5cnnvuOcaNG0ePHj0YP348mZmZpKSk4ObmxpAhQxg3bhxZWVn88ssvFj/nQCd9E3VuXoNVL3bgxfn7mPBzBDuOX+Wdh3yo4KAnc2nW448//mD37t0EB6vZ/Ddv3qRu3br06tWL6OhoXnrpJfr06UPPnj0L3ZePjw9PPPEEAwYMyDvLXrduHatXr+ajj9SHUWpqaqGTkdLS0hg/fjwRERHY2dlx/PjxvFifeeYZKlSoAKhWyAW1LDbFY489lvd17ofQtWvXSEpKol+/fgBs2rSJBQsWAKrLpouLCy4uLjg7O3Pw4EFOnz5NSEgIVatWLfAYlkIn/ftQy9WJn0a14fM/jjFjUwz7z15jxuMtaVKjZKZLa+XIfZ6RlxQpJc888wzvvffeHa8dOHCA1atXM3XqVBYvXszs2bPvua+1a9eyefNmli1bxvvvv8+hQ4eQUrJ06VIaN258y7Z//vnnXffz6aefUrduXX788UcyMjKoXLlyXqx3a098u3u1QoZb20k/9dRTrF69Gl9fX+bMmXPLWrsF7fvZZ59l7ty5nDp1ijFjxtz1+7AUuqZ/n+xsbXi1V3O+GxHC5eQ0+k/byuI9sUaHpWlm0b17dxYuXMjly5cBVeo4c+YM8fHxSCl55JFHePfdd/NaDd+thXFWVhaxsbF07dqVKVOmEB8fT0pKCr169WLq1Kl52+W2SC6sFXLt2rURQvDdd9/l9d/v2bMnX3/9dV7N/erVq3dtWVy/fn0iIyNJT08nISGBDRs23PVncOPGDWrVqkVGRsYt/ey7dOmSt5pYVlYWiYlqpbOHH36YFStWsH//frp3727CT9lYOukXUcdm7qx6qQP+nq688ksEr/0Swc10PbpHK9v8/Px4++236d69O/7+/vTs2ZOLFy9y9uxZOnbsSGBgIKNGjeLDDz8EVO/3kSNH3nEjNzMzk2HDhuHv709QUBCvv/46zs7OvP3226SkpODn54ePjw/vvPMOAF27diUiIoKWLVvecSN3/PjxzJkzh7Zt23L69Om8hVj69etHaGgowcHBBAYG8tlnnwEFtyxu2LAhAwcOxM/Pj6eeeqrAJRlzTZ48mZCQEHr06HHLspDTp09n7dq1+Pn5ERwczJEjRwBVQurYsSNDhw696yLylsSk1spCiFDgf4AtMEdK+dFtr9cDvgOq5GwzUUq5SghhD8wBglClpO+llP++17FKurWyuWVmZfO/9ceYvjGGpjUqM2NYEE1r6nKPVjjdWtk6ZGdnExgYyNKlS2nUqFGpHLNEWysLIWyBGUBvwBsYKoTwvm2zSai1c1uiFk7PHbP0COAopfQDWgFjhBANCjtmWWJna8MrPZvz/TMhXElO56HpW1mkyz2aVi4cPHiQxo0bExoaWmoJv7hMuZEbAsRIKU8ACCEWAAOA/GuaScAl52tXIC7f85WEEHZABSAdSDRD3BanQ1NV7nlpwT5e/SWCE/HJvNareYE3fjRNsw5+fn6cPHnS6DDuiykFKA/gbL7HsTnP5fcO8IQQIha1gPoLOc8vAm4A54EzwCdSyqu3H0AIMVoIES6ECM9dMqwsqunixLyRbRkaUo8vNh3n/5YdIjvbslYm0yyLpa1cp1m+4v7OmJL0CzpVvf2oQ4G5UkpPoA/wgxDCBnWVkAXUARoCrwgh7rgGklLOllIGSymD3d3d7+sbsDS2NoIPB/kyplMjftxxhpcX7icjK7vwN2rljpOTE1euXNGJXzOZlJIrV66YPP+gIKaUd2KBuvkee/J3+SbXs0BoTlDbhRBOgBswDFgjpcwALgkhtgLBwIkiR1wGCCF4o7cXrhXs+XhNNMlpmUwfFqRX5dJu4enpSWxsLGX56lYrfU5OTnh6Fr0jgClJfzfQVAjREDiHulE77LZtzgDdgLlCCC/ACYjPeb6rEOJHoCLQFrhznrWVer5zE5yd7Hlr2SGe/nYXc4a3prKjng+nKfb29jRs2NDoMLRyptDyjpQyExgPrAWiUKN0IoUQk4UQD+Vs9gowSggRAcwHnpbqmnUGUBk4hPrw+FZKeaAEvg+L9WTb+nz+WCC7TyUw7KsdXL1hettaTdM0czNpnH5pKmvj9E21Puoiz8/bS71qFfnh2TbUci16TU7TNO12Zhunr5lHN6+azB0RQty1mzzy5TZOX7lhdEiappVDOumXonaNqzN/dFuSUzMZPGs70RcK7jWiaZpWUnTSL2X+nlVYOKYdNgIe/XI7+84kGB2SpmnliE76Bmha05lFYx+gSkV7Hp+zk60xl40OSdO0ckInfYPUrVaRX8a0o27Vioz4djfrIou/zqimaVphdNI3UA0XJ34e0xbvOi48N28vv+7Vjdo0TStZOukbrEpFB+aNbEPbRmoN3rlby1bzJk3Tyhad9C1AJUc7vh7emp7eNXlnxWGmrT+m+7FomlYidNK3EE72tnzxeBBhQR58+vtRPlgZpRO/pmlmpxvBWBA7Wxs+GRyAi5M9c7acJCk1kw/D/LC10T35NU0zD530LYyNjeDt/t64VLBn6vpjJKVl8NljgTja6Q6dmqYVn076FkgIwcs9muHiZMf7K6NISg3nyydbUdFB/3NpmlY8uqZvwUZ2aMTHD/uzNeYyT369i+s3M4wOSdO0Mk4nfQv3aOu6zBgWxIHYawydvYP4pDSjQ9I0rQzTSb8M6O1Xm6+Ht+bk5Rs8+uV2zl27aXRImqaVUTrplxEdm7nz48gQLien8cjMbRyPTzY6JE3TyiCTkr4QIlQIES2EiBFCTCzg9XpCiI1CiH1CiANCiD75XvMXQmwXQkQKIQ7mrJ+rFUGr+tX4eXQ70rOyeXTWdg6du250SJqmlTGFJn0hhC1q2cPegDcwVAjhfdtmk1DLKLZEraH7Rc577YAfgbFSSh+gM6DvRhaDdx0XFo5ph5O9LUNn72D3qatGh6RpWhliypl+CBAjpTwhpUwHFgADbttGAi45X7sCcTlf9wQOSCkjAKSUV6SUWcUPu3xr5F6ZX8a2w93FkSe/3smm6EtGh6RpWhlhStL3AM7mexyb81x+7wBPCCFigVXACznPNwOkEGKtEGKvEOKfxYxXy1GnSgV+GdOOxu6VGfV9OCsPnDc6JE3TygBTkn5BPQBubwozFJgrpfQE+gA/CCFsUJO/HgQez/l7kBCi2x0HEGK0ECJcCBEeHx9/X99AeVa9siPzR7clsG4VXpi/lwW7zhgdkqZpFs6UpB8L1M332JO/yze5ngUWAkgptwNOgFvOezdLKS9LKVNQVwFBtx9ASjlbShkspQx2d3e//++iHHNxsuf7Z9rQsZk7E389yOw/jxsdkqZpFsyUpL8baCqEaCiEcEDdqF1+2zZngG4AQggvVNKPB9YC/kKIijk3dTsBh80VvKZUcLBl9pPB9PWvzYerjvDJ2mjdoVPTtAIV2sxFSpkphBiPSuC2wDdSykghxGQgXEq5HHgF+EoIMQFV+nlaqqyTIIT4L+qDQwKrpJQrS+qbKc8c7GyYOqQlLk52TN8YQ2JqBu/098FGd+jUNC0fYWlnhMHBwTI8PNzoMMosKSUfrT7Cl3+eoK9fbSb0aEaTGpWNDkvTtBImhNgjpQwubDvdttHKCCGY2LsFrhXt+XTdUVYePE9g3SoMbuVJf/86uFa0NzpETdMMpM/0rdilpFSW74/jl/BYoi8m4WBnQ0/vmgxu5UmHpu56cRZNsyKmnunrpF8OSCmJjEtk0Z5Ylu4/x7WUDGo4OzIoyIPBQZ40relsdIiaphWTTvpagdIys9h45BKL9pxjY/QlsrIlATnln4d0+UfTyiyd9LVCxSelsWz/ORbtieXIhSQcbG3okVf+ccPOVjdh1bSyQid9zWS55Z/Fe2NZtj+OqzfScXd2JKylBw+38qSZLv9omsXTSV8rkvTMbDZGX2LRnlg2HrlEZrbE39NVlX8C6lClooPRIWqaVgCd9LViu5ycxrL9cSzeE8vh84k42NrQ3bsGg1t50rGpuy7/aJoF0UlfM6vIuOss3nOOpfvPcfVGOm6VHRnUsg6DW9WleS1d/tE0o+mkr5WI9MxsNkVfYvHeWNZHqfKPn8ff5Z+qlXT5R9OMoJO+VuKuJKexPCKORXtiiYxLxN5W0K2FGv3Tqbk79rr8o2mlRid9rVRFnU9kcc7kr8vJ6bhVdmBgoBr941XbpfAdaJpWLDrpa4bIyMpmc3Q8i/bEsv7IRTKyJEH1qvBGHy9aN6hmdHiaZrV00tcMd/VGOsv2n+PLzSe4kJhKH79avB7agvrVKxkdmqZZHZ30NYtxMz2Lr/46wcxNx8nMzubpBxowvmtTXCvolg+aZi6mJn19p83c0lNgx0y4cdnoSCxGBQdbXuzWlE2vdWZQSw/mbDlJ5ykb+W7bKTKyso0OT9PKFZ30zW3nLFgzEWY9CCf/Mjoai1LTxYmPBwfw2wsP4lXbhbeXR9Lr8z/54/BFvbyjppUSk5K+ECJUCBEthIgRQkws4PV6QoiNQoh9QogDQog+BbyeLIR41VyBW6TMNNj5JdRpCQ6V4bv+sPFDyM4yOjKL4lPHlXkj2/D1cHUlOvL7cB6fs5PIuOsGR6Zp1q/QpC+EsAVmAL0Bb2CoEML7ts0mAQullC1RC6d/cdvrnwGrix+uhTu4CJIvQNf/g9GbIGAobP6PSv6JcUZHZ1GEEHTzqsnaf3Rk8gAfos4n0m/aFl77JYKLialGh6dppSrhRjrfbTvFnL9OlPixTFkuMQSIkVKeABBCLAAGAIfzbSOB3MHYrkBehhNCDAROADfMEbDFkhK2T4caPtC4KwgBg2ZCo07w28swsz0MmgXNehkdqUWxt7XhqXYNGBDowYyNMczdeorfDpxnbKfGjOrYkIoOekVPzTplZGXz51E1vPmPKDW8uX2T6ozs0KhEj1vo6B0hxGAgVEo5Mufxk0AbKeX4fNvUBtYBVYFKQHcp5R4hRCXgD6AH8CqQLKX8pIBjjAZGA9SrV6/V6dOnzfG9la6YP+DHh2HgTAgcdutrl4/BLyPg4kFoNx66vQ12ul1BQc5cSeE/a46w8uB5aro48lqvFoS19MBGL+2oWYkjFxJZFB7L0v1xXE5Oo3olBwYEevBwKw986rgWeb/mXBi9oP9tt39SDAXmSik/FUK0A34QQvgC7wKfSSmThbj7f1op5WxgNqghmybEZHm2TQPn2uA7+M7X3JrCyD/g9/9TVwOnt8Hgb6Baw9KP08LVq16RGY8HMeLUVd5bGcWrv0Qwd9tJ3uzjTbvG1Y0OT9OK5OqNdJbvP8eivbEcOpeInY2gm1cNBreqS+dSblliStKPBerme+xJvvJNjmeBUAAp5XYhhBPgBrQBBgshPgaqANlCiFQp5fRiR25Jzh+AE5ug+zt3P4O3d4I+U6BhR1g2Dr7sCP0/B9+HSzHQsiO4QTWWPPcAKw7E8fGaaIZ+tYMe3jV5o3cLGrlXNjo8TStURlY2m6LjWbTnLBuOXCIjS+Lr4cLb/b15KKAO1Ss7GhKXKeUdO+Ao0A04B+wGhkkpI/Ntsxr4WUo5VwjhBawHPGS+nQsh3uEu5Z38yuTkrF/HQNQKeDkSKlQtfPtrZ2DRsxC7C4KGQ+hH4FCx5OMso1Izsvh6y0m+2BhDWmY2T7arz0vdmuoFXTSLFHU+kUV7Ylm67xxXbpReHyqzlXeklJlCiPHAWsAW+EZKGSmEmAyESymXA68AXwkhJqBKP0/L8jLw+vo5OLQIWo8yLeEDVKkHI1bBxg9gy2dwdhc88i3U8CrZWMsoJ3tbxnVpwqPBdfnsj6N8t+0Ui/fE8mK3pjzVrgEOdnq6iWasK7kLDu21/I6zug1Dca3LqdO/uB+q1r//98eshyVjIC0Zev8Hgp5SI3+0u4q+kMQHq6L482g8DapXZGJvL3r51ORe9400zdwysrLZeEQtLbrhiPFrS+jeO6UhNRE+84Em3dWZelElXYRfR8HJzarG3+9zcNLtiAuzKfoSH66K4ujFZEIaVGNSPy/8PasYHZZm5Q7HqfLNsv255Ru1itzDrTxpUcu4/7fmHL2j3c2+HyAtER4YX/i29+JcE55coko9Gz+Ec3vV6B6PIPPEaaU6N6/Bg03c+Dn8LP9dd5SHpm8lrKUHr/ZqTp0qFYwOT7MiuetFL9oTS1QZXy9an+kXVVYmTA38uz5vLqe3w+KRkHwRekyGts/pco8JklIzmLnpOHO2nEQAozs2YmynxlRy1Oc1WtGkZ2azMVqVbzbmlG8CPF15uJUn/f0tb2lQXd4paQcXweJnYegCaN7bvPtOuQrLxkP0SmgWqiZ8VdQLkJgiNiGFj9dEszwiDndnR17t2YzBrepiqyd3aSaQUhKZU75ZHhHH1RvpuDs7EtZSjb5pVtPZ6BDvSif9kiQlzO6k2iiP2wU2JXCsFPCzAAAgAElEQVRpJyXsmg3rJkFFNxj8NdR/wPzHsVL7ziTw/soo9pxOoEUtZyb19ebBpm5Gh6VZqMvJaSzdd45Fe2I5ciEJB1sbenir0TcdmrqVifKNTvol6eRf8F0/dcM1eETJHituPywaAQmnoPMb0OEVsLEt2WNaCSklqw5e4KM1UZy9epPmNZ3x8XDBu7YL3nXU3+VhrH9WtuTk5WQi4xI5fD6Rw3GJxCbcpH9AHcZ0bFSuS2AXrqfyybpolu47p8o3daswuJUn/f1rl7nfDZ30S9K8R+HcHphwCOxL4YZhWpJq2nZwITToAGFfgUvtkj+ulUjLzGLejjP8dSyew+cTuZiYlvdaHVenvA8A9bcrdatVKLPDP2+kZXLkgkrsh88ncfh8ItEXEknNUIvV2NsKmtV0pkpFe7bGXKGGsyOv9mrOw0Ge5aoElpKeyZebTzD7zxNkZUseb1uPYSH1aGrB5ZvC6KRfUuKjYUYIdP4XdH699I4rJeyfB6teA/uKMOhLaNq99I5vRS4npxF1Pjcxqr+PxyeTnfNfwdnRDq+cDwGv2s5413alac3KONlbzhWWlJJLSWm3fA+Hzydy6soNcv9Lu1awv+WqxruOC43dK+dNZttzOoH3Vx5m35lreNd2YVJfLx5oYt0lsOxsyeK9sXyyLpqLiWn09a/NxNAW1K1W9mfE66RfUpa/AAcWwoRIqGTAf5D4aNWx81IkPPAidHsLbPVas8WVmpFF9IUk9WGQk0SjzidyI10tgGNrI2jiXvmWBOpV24VqpTCCIzMrmxOXb+Ql9twPrCs30vO2qVet4h0JvrarU6FXLFJKfjtwno9WH+HctZt096rBG328aGyF/Y22Hb/MByujiIxLJLBuFf6vnxet6lvPAAmd9EtC8iU1GavlE9DvM+PiyLgJa/8F4d+AR7C6yVu1gXHxWKnsbMmZqym3nEkfjkvkQr5FXmq7OqmrgnwJt161ikVuBZ2clsmR87eevR+5kER6pirPONja0KxWZXW82i5413GlRW1nXJyK98GfmpHFt1tPMWNjDKkZWTzeph4vdW9WKh9qJe14fDL/XnWEP6Iu4lGlAq/3bkF//9pltoR3Nzrpl4QNH8CfU2B8OLg1MToaiFwCy18EBDw0FXwGGh1RuXAlOY2o80kcPn89LzEfj79BVk59qJKDbV55yLu2uiJoXsv5lvKQlJILianq/bkfKOcTOX0lJW+bKhXt8cm3j9zyTEn2cbmcnMbnfxzlp51nqORox4tdm/LUA/VxtLOc0papEm6k87/1x/hxx2mc7G15vktjnmnf0KLKdOakk765pafAZ95Q7wEY+pPR0fwt4RQsekbdWA5+Bnp9WDo3l7VbpGZkcexi8i0fBFHnk0hOywTARkBj98o0r+VMQko6h+MSSUjJyHt//eoV8529qz+1XAovz5SUoxeT+HBVFJui46lXrSITe7egt2+tMnF2nJaZxQ/bTzN1/TGS0zIZGlKPCT2a4WZQK+PSopO+ue36Cla9CiPWQP12Rkdzq8x02DBZLeRS01e1cHBvbnRU5V52tuRsQsotpaEjF5KoVsnhluTeopYzzsUsz5SUP4/G88HKKKIvJhFcvyqT+nkTWNcy+xtJKVlz6AIfrTnC6SspdGrmzpt9vSx6QpU56aRvTtlZMK0VVKyuVsCy1LOdY7+rjp0ZN9WCLYGPW26sWpmRlS1ZGH6WT9cd5XJyGgMD6/BaaAs8LKi/UcTZa7y/8jC7TyXQrGZl3uzrTadm7kaHVap00jenqBXw8xPwyHeWXzdPPK86dp76SyX9ATN04tfMIjktk1mbjvPVXycAGNmhIc91bkJlAyd3xV27yZS10SzZdw63yg683KM5jwZ73t8M2pN/Qmw4tH1erXBXRumkb05f94SkC/DivrIxGzY7S/X53zEDRq4Hz0J/DzTNZOeu3WTKmiMs3R+HW2VHXunZjEeDS7e/Uf4PIAmM6tCQsZ0a31+ZLCvz74WMkFDLDwbPtYxBGkVgatI36eNQCBEqhIgWQsQIISYW8Ho9IcRGIcQ+IcQBIUSfnOd7CCH2CCEO5vzd9f6/FYOd3QVnd0K7cWUj4YOKs9M/wdYBDv1qdDSalfGoUoHPh7Rk2bj2NHSryBu/HqTv1L/482h8iR87K1syf9cZOk/ZxPSNMYT61mLDK514rVeL+0v4187C3D6w5b9qCPYj38H1WLV2dcTPJfcNWABT1si1Ra2R2wO1SPpuYKiU8nC+bWYD+6SUM4UQ3sAqKWUDIURL4KKUMk4I4QuslVJ63Ot4Fnem//OT6vJvQiQ4lrEJKz8NgfMRKvaSaAqnlXu5N0//vfoIZ66m0Lm5O2/28SqRdgZ/HVM3lY9cKOZN5ajfYNk4dUXc/3PwG6yev35OtTU/sw0Chqn7YmXo/7w5z/RDgBgp5QkpZTqwABhw2zYSyF0yxhWIA5BS7pNSxuU8Hwk4CSHKzripqydUPT/4mTL1j5/HNwyS4tQC7JpWAoQQ9Parze8vd2RSXy/2nE4g9H9/8eaSg1xOTit8ByY4djGJEd/u4smvd3EjPZMvHg/il7Ht7j/hZ6bBqn/Cz4+ryYxjNv+d8AFcPWD4Cuj4T4iYD7M7w4VDZvkeLIkpd2A8gLP5HscCbW7b5h1gnRDiBaASUFBTmIdRVwPm+U0oDdu/ABs7aDPG6EiKplko2DqqEk+9tkZHo1kxRztbRnZoRFiQJ1PXH+OHHadZtj+OcV2aMKJ9gyJNiMqdKDZ/11kqOtjyZh+vok8UuxyjutVeOKBu2HZ/B+wKOP+0tYOub0KDB9WAiK+6QuiHEPys1QyIMOVMv6Dv9Paa0FBgrpTSE+gD/CCEyNu3EMIH+A9QYPYUQowWQoQLIcLj40u+LmiSlKuw70fwfwycaxkdTdE4uUDTHnB4qbqU1bQSVq2SA+885MPaf3SkbaNq/GfNEbp9upnlEXGYOmgkNSOLmZuO02XKJubvOssTbeqx+bUujOrYqGgJP+Jntf7F9bNq0aPQfxec8PNr1AnGboWGHWDlK7DwKbh57f6PbYFMSfqxQN18jz3JKd/k8yywEEBKuR1wAtwAhBCewBLgKSnl8YIOIKWcLaUMllIGu7tbyNja8K8h82bx1781mm+YWnrx9DajI9HKkSY1KjNneGt+GtkG1wr2vDh/H2Ezt7HndMJd3yOlZEVEHN0+3cx/1hyhTaNqrP1HR94d4Fu0HkBpybDkOVgyGmr5qyR+P6vcVXaHYb9Aj/cgehXM6gBnd99/HBbGlKS/G2gqhGgohHAAhgDLb9vmDNANQAjhhUr68UKIKsBK4A0p5VbzhV3CMlJh52xo0h1qeBkdTfE0C1WtmCOXGB2JVg490MSNFS88yMeD/TmXcJOHZ25j/E97OXs15Zbt9pxOIGzmNl6Yvw+XCvbMG9mGOcNb06RGEe+lXTikavIR81WNfvgKVbO/XzY20P5FeGatqnl8GwpbPofs7KLFZQFMGqefMwTzc8AW+EZK+YEQYjIQLqVcnjNi5yugMqr0808p5TohxCTgDeBYvt31lFJeutuxLGL0zt7vVQvlp5ZBo87GxmIOvzytVvt6JVrVLDXNADfSMpn95wm+/PM42dkw4sEGDAz0YMbGGH47cN48C7pIqa7S1/wLKlRRCw416mSeb+DmNVjxIhxeBo27qTUtKltIZQI9OavosrPhi7Zg5wBj/rKOmzeHl6ma5JNLoXEXo6PRyrkL11OZsjaaX/fFIiU42dswumPj4i/dePOaOlmLWl5ySVlK1dJ8zRvm/1ApJlOTvj7tu13MH3A5GgbNto6ED9C0JzhUhshfddLXDFfL1YlPHw1gRPsG/HXsMgNb1qG2azH7+MSGq9E5iXHQYzK0e6Fk5qYIAa2fhbpt1PG+HwAdX4VOE8vMVbSesXO7bVPBuY66AWot7CuoG1hRKyAro/DtNa0U+Hq48lznxsVL+NnZsPV/8E0v9XjEGmj/UslPRqzlC6M3QeAwtcbGd/3V5K4yQCf9/OL2q0ZlbZ+zviUIfcLgZgKc2Gx0JJpmHsnx8NMj8Ptb0LyPKsfWbV16x3eoBAO/UFWBCwdgVnuIXl16xy8infTz2z4dHJyh1XCjIzG/Jt3A0UWVeDStrDuxWSXZk39B3//Co9+rGrsRAh6D0ZvBtS7MH6Lq/ZmWOwdVJ/1c186qmauthoOTq9HRmJ+dI7Toq/qOWPAvpKbdU1YmbHhf1dKdXGHUBlVjN/r+m1sTtdZGm7Gw4wvVmfdKgdOSDKeTfq6ds9TfbcYaG0dJ8gmDtOtwfIPRkWja/bt+TtXO/5yiaumjN6nauqWwc4Te/4EhP6llTL/sBAcXGR3VHXTSB0i9Dnu+Uzdvq9QtfPuyqlFncKqiJ2ppZU/0alXOuXBA1dAHfqFq6paoRV8YuwVq+sDiZ1VHz/QbRkeVRyd9UAk/PQnalfGWC4WxcwCv/nBklZp1rGmWLjNN1cjnD1E189GbVQ3d0lWpC0+vhA6vwL55MLsLXDxc+PtKgU76WRmqtNOgA9QJNDqakuczSH3AxfxudCSadm9Xjqva+I4vIGSMqpmXpVWtbO2g21vw5BI1cu6rLhD+rZrgZSCd9COXQOI5eOBFoyMpHQ07qQXe9YpamiU7uEjVxBNOwWPzoM/HhXfGtFSNu8BzW6FeO/jtH2pSV+p1w8Ip30lfSjUZy625aq5WHtjagddDcHSNRdUZNQ1Qv5PLxqtaeE1vVRv36md0VMVXuQY88St0exsOL1fLMp7bY0go5Tvpn9wMFw6q9snlaTlB3zDISIFj64yORNP+dvGwqn3v+1HVwp9eaV0DK2xsoMPLMGK1Wt/i656wbVqpd+wsR5muANumQaUa4Peo0ZGUrvrt1fetSzyaJZBS1bq/6qJq30/+qmrh1jYrPle9NjDmT9X2fN0kmP8Y3LhSaocvv0n/4mHVXK3NaLB3Mjqa0mVjC94D1Jl+WpLR0WjlWep1VeP+7R+q5v3cVmjc1eioSl7FavDYj9B7CpzYpIajntpSKocuv0l/+wywq6DWviyPfMMgMxWi1xgdiVZeJZxSte3Dy1Wt+4lfVe27vBBCnXSOXK/mHHzXHzZ9VOLlnvKZ9JMuwIGfoeUT6hO3PKrbVnUT1RO1NKNsngJJF1WNu8PL5eu+Wn61/dX8A79H4VJUibeUMOmnLIQIFUJECyFihBATC3i9nhBioxBinxDiQM5KW7mvvZHzvmghRC9zBl9ku2ZDdia0e97oSIxjYwM+A9V4fQOHj2nlVN6J1+Oqxl3eOVaGsC/VoixGJ30hhC0wA+gNeANDc5ZHzG8SsFBK2RK1hu4XOe/1znnsA4QCX+TszzhpybD7azUMrFojQ0MxnE8YZKWrGbqaVppyT7zaluMTr4LYFWEB+Ptkypl+CBAjpTwhpUwHFgADbttGAi45X7sCcTlfDwAWSCnTpJQngZic/Rln/zxIvVZ+JmPdi2ewmtqu2y1rpSn3xKtFX6je2Ohoyh1Tkr4HcDbf49ic5/J7B3hCCBELrAJeuI/3lp7sLHUDt24bqGvsZ49FEEKVeI5vgJSrRkejlRf6xMtQpiT9ggpMtzePGArMlVJ6An2AH4QQNia+FyHEaCFEuBAiPD4+3oSQiihqBVw7DQ+8UPi25YVPmLrMPrLS6Ei08iD3xMszRNfyDWJK0o8F8k+L8+Tv8k2uZ4GFAFLK7YAT4Gbie5FSzpZSBkspg93dzbx6/d8HUS0XqjZUS6tpSp2WULWBLvFopUOfeBnOlKS/G2gqhGgohHBA3Zhdfts2Z4BuAEIIL1TSj8/ZbogQwlEI0RBoCuwyV/D35cwO1eui3Tg1OUlThFCdN09shhuXjY5Gs2ZSqlnwVRuqer5miEKTvpQyExgPrAWiUKN0IoUQk4UQD+Vs9gowSggRAcwHnpZKJOoK4DCwBhgnpcwqiW+kUNunQ4VqEPi4IYe3aD5hILMg6vbPck0zo7M74Vy4PvEymJ0pG0kpV6Fu0OZ/7q18Xx8G2t/lvR8AHxQjxuK7HKNq1h1fBYeKhoZikWr5QfUmaqJW8DNGR6NZq23ToEJVtdShZpjyMQVuxwzVvClktNGRWCYh1Nn+qS2QfMnoaDRrdOW4OvEKftZylzksJ6w/6d+4DPt/goAh5auvx/3yDQOZDYeXGR2JZo226xMvS2H9SX/316qxmLWvf1tcNbzAvYVut6yZ343Lamy+/2PgXNPoaMo96076GTfVdO+mvcC9udHRWD6fMDizHRLvGFWraUWnT7wsinUn/YgFkHJZjwk2lW8YIHWJRzOfvBOvnlCjhdHRaFhz0s/OVnXE2oHQ4EGjoykb3JpCTT9d4tHM58DP+sTLwlhv0j+2Fq4cU79sJdyq1Kr4DoLYXXDtbOHbatq9ZGfDtulQOwAadDA6Gi2H9Sb9bdNUB0nv2xuCavfkM0j9rRdX0Yor98SrnT7xsiTWmfTP7YHTW6Htc9a7uHJJqdZIlcR00teKa9t0cPFUnVw1i2GdSX/bdHB0haCnjI6kbPINg7i9cPWk0ZFoZdW5PXB6iz7xskDWl/QTTsPhpdBqODg6Gx1N2aRLPFpxbZsOji76xMsCWV/S3zEThA20GWt0JGVXlXrgEazbLWtFk3BaDfttNRycXArfXitV1pX0bybA3u/BdzC4GrdAl1XwDYMLB1WzOk27HztnqRu3+sTLIllX0t8zFzJuwAN65l+xeefcfNMlHu1+3EyAPd+p2d2unkZHoxXApNbKZUJmOuz8Ehp1Vq2CteJx9YB67VSJp9NrRkdTfNu/gJg/jI5C9ZF/cALUf8DoSEqGPvGyeNaT9JMvgIuHnvlnTj5hsPo1uHSkbE+hP7oO1r4Bbs3UzUUjXTsDC4fDuJ1QsZqxsZhb7olXw05qQpZmkawn6VepByMt4EzOmng/BKv/qc72a/zL6GiKJuUqLH8B3L1gzGawczQ2nguHYHZnWPkKPPKtsbGY26HFkHQeHppudCTaPZhU0xdChAohooUQMUKIiQW8/pkQYn/On6NCiGv5XvtYCBEphIgSQkwVogSn5gmhZ/6Zk3Mt1bcocola37QsWv266v0yaJbxCR+gli90eUN9kFpTj6Pc9W/dvaBJN6Oj0e6h0KQvhLAFZgC9AW9gqBDCO/82UsoJUspAKWUgMA34Nee9D6CWUfQHfIHWQCezfgdayfIZBJePwsVIoyO5f4eXw8GF0PE1qBNodDR/e+Al8GilzvaTLhodjXmc2AiXIlUtX594WTRTzvRDgBgp5QkpZTqwALhXQ5uhqMXRASTgBDgAjoA9YCW/5eWE9wAQtmVvzP6Ny/DbBFVb7vCK0dHcytYOBs6CjBT47R9l9yoqv23ToHJN8HvE6Ei0QpiS9D2A/C0XY3Oeu4MQoj7QENgAIKXcDmwEzuf8WSuljCpOwFopq+QGDTuqUkRZSU5SqmSalgiDvrTMNgDuzaDbWxC9Sq37UJZdOATHN6ilEC2hhKbdkylJv6Brtbv97x8CLJJSZgEIIZoAXoAn6oOiqxCi4x0HEGK0ECJcCBEeHx9vWuRa6fEZBAkn4XyE0ZGY5uAiiFoBXd5Uy0BaqjbPQf326r7D9Vijoym67TPAviIEP2N0JJoJTEn6sUDdfI89gbutpzeEv0s7AIOAHVLKZCllMrAaaHv7m6SUs6WUwVLKYHd3d9Mi10qPV3+wsSsbJZ7E87DqFfAMsfzhuzY2MGAGZGeqEUZl5Uoqv8Q4OPgLtHzS+oagWilTkv5uoKkQoqEQwgGV2JffvpEQojlQFdie7+kzQCchhJ0Qwh51E1eXd8qaitWgURfLH8UjJax4UY0XHzhTTYSydNUaQs/3VHlkTxkcwrnzS5BZqpumViYUmvSllJnAeGAtKmEvlFJGCiEmCyEeyrfpUGCBlLdkhUXAceAgEAFESClXmC16rfT4hqmJRef2GB3J3e37AY6tgx7vglsTo6MxXfAz6kN17aSy1c46LQnCv1VXgtUaGh2NZiKTJmdJKVcBq2577q3bHr9TwPuygDHFiE+zFM37gK2DuqHrGWx0NHe6dgbW/Esty9d6lNHR3B8hYMB0+KIdLBsHw39TpR9Lt+9HSLsOD7xodCTafSgDv1maRahQBRp3U2sVZGcbHc2tsrNVskSqGnlZSJi3c/WE0I/Uim87ZxkdTeGyMlU/o3rtLPMkQLurMvi/QzOMbxgknlMLp1uS8K/h5J/Q6wOoWt/oaIoucBg06w3r34XLx4yO5t6ilsH1M9BON1Yra3TS10zXvDfYOVlW+4Arx+H3t6BJdwgabnQ0xSME9P8f2FeAJWPV2bQlklKtjFWtsfqd0MoUnfQ10zk6Q9MeOSWeLKOjUTEszVmD9aFp1jH937km9P0UzoXDtqlGR1Ow09vUGsrtxpWNEVLaLXTS1+6PzyBIvghnthe+bUnbPgPO7oTeU8CljtHRmI/vw+rnvPFDy+x5tG0aVKgGAUONjkQrAp30tfvTLFTNvjS6xHPpCGx4H1r0A/9HjY2lJPT5VN08XzJGzTuwFJePwdHVEDIKHCoaHY1WBDrpa/fHoRI066UWvjaq5pyVAUvHgmNl6Pe5dZR1blepOvSfqtYp/usTo6P52/bpYOtY9obFanl00tfun0+Y6lF/6i9jjr/lM4jbB33/C5WtuG1Hiz6qhPLnJ3Bur9HRQHI87J8PgUOt++du5XTS1+5f0x7gUNmYRdPPH4DN/wHfweAzsPSPX9pCP1Iti5c+Bxmpxsay+yvISoO244yNQysWnfS1+2dfQQ3Vi1quSi2lJTNNDWWsWB36TCm94xqpQhUYMA3ij8DGD4yLIz0Fds9R8wjcmxkXh1ZsOulrReMTBjcT4MTm0jvmpo/U6kwPTStfHR2bdIdWI9SomTM7jYkhYj6kXLH8zqVaoXTS14qmSTdwdC29dsux4bD1c2j5hLqRXN70fA+q1FU3sNNvlO6xs7PV8Ng6LaH+A6V7bM3sdNLXisbOEVr0hajfVNmlJGXcVGUd5zrQ68OSPZalcnRW7aKvnoA/3i3dYx9dDVePq7N8axwpVc7opK8Vnc8g1WXx+MaSPc769+DKMdWJ0sm1ZI9lyRo8CG2fh11flm5Zbds0cK0HXvdaGlsrK3TS14quUWdwqlKyJZ5TW2HHF2pceOMuJXecsqLbW1C9ieoqmppY8seLDVezr9s+pxZ018o8nfS1orNzUAtoHFlVMsMJ05LVUMWqDdTCKJoaOTVwlup2uu7Nkj/etmnq3k3QkyV/LK1UmJT0hRChQohoIUSMEGJiAa9/JoTYn/PnqBDiWr7X6gkh1gkhooQQh4UQDcwXvmY43zBIT4KY382/79//Ty2OMnCmmgmsKXVbQ/uXYO/3cHRdyR3n6kk1LDd4hLqnoFmFQpO+EMIWmAH0BryBoUII7/zbSCknSCkDpZSBwDQg//X+98AUKaUXEAJcMlfwmgVo0FGNmzf3RK2Y9RD+jerkWL+defdtDTq/ATW81YLqKVdL5hg7ZoKwhTZ68TtrYsqZfggQI6U8IaVMBxYA97qjMxSYD5Dz4WAnpfwdQEqZLKVMKWbMmiWxtQOvhyB6jZrAYw43r6lk5tYcuv6fefZpbewcYdAs1Q5j9evm33/KVbUcot9g6+pgqpmU9D2As/kex+Y8dwchRH2gIbAh56lmwDUhxK9CiH1CiCk5Vw6aNfENg4wbcGytefa35g1IugCDZoK9k3n2aY1qB0DHf8LBhXB4uXn3vedb9W+qV8ayOqYk/YIG5sq7bDsEWJSzIDqohdc7AK8CrYFGwNN3HECI0UKIcCFEeHx8vAkhaRalfnvVH8Yc7ZaPrIKIn6DDy+DRqvj7s3YdXobagfDbBLhx2Tz7zEyDnV9C465Qy9c8+9QshilJPxaom++xJxB3l22HkFPayffefTmloUxgKRB0+5uklLOllMFSymB3d929r8yxsQXvAXBsHaQlFX0/KVdhxUtQ00+dwWqFs7VXZZ60RPjtH2opw+I6+ItaKEef5VslU5L+bqCpEKKhEMIBldjvuJYUQjQHqgLbb3tvVSFEbibvChwuXsiaRfIJg8xUOFqMEs/KV1Q/n0Gz1HBQzTQ1vKDLmxC1Ag4uKt6+cte/reGjzvQ1q1No0s85Qx8PrAWigIVSykghxGQhxEP5Nh0KLJDy71ONnDLPq8B6IcRBVKnoK3N+A5qFqNtGtUkoaonn0K9qklfnibqkUBQPvACeIbDqFUg8X/T9xKyH+CjdcsGKCWmOy0EzCg4OluHh4UaHoRXFmjdU+93XYu6vXULSRfiiLVRrCM+s0zM/i+rKcZjZHhp2gGELi5a0v3sILh+Flw7oq60yRgixR0oZXNh2ekauZj4+YZCVrm7GmkpKVYvOSFEzTXXCL7rqjdXM5WPrYN8P9//+8wfg5GY1Ll8nfKulk75mPp7B4Fr3/iZqRcyH6FVqPL5enKP4Wo+CBh1gzb/UbOb7sX26WhGt1YiSiU2zCDrpa+YjhFrC8PgGdUO2MNdjYfVEqPeAauilFZ+NDQyYAUjVlC0727T3XT8HhxZD0FNqtS7Naumkr5mXTxhkZ6g++/cipZp1m50JA2eoYZ+aeVStr9YdOPknhH9t2nt2zgKZDW3GlmxsmuF00tfMq05L1RWzsHbLe75VVwQ9J0O1RqUSWrkS9BQ06QG/v6Vu8N5LaiLsmQveA9UHhmbVdNLXzEsIdbZ/YvPdZ4hePQlrJ0GjLhD8bOnGV14IAQ9NVZO3lj4P2Vl333bv92pyl17/tlzQSV8zP98wkFlqstDtsrNVrdnGVq2EpceClxyXOtB7Cpzdoda4LUhWhuqmWb89eNwxWV6zQjrpa+ZX01et7lRQiWfnLDi9FUI/AlfP0o+tvPF/FFr0gw3vw6Ujd75+eBkkxuqz/HJEJ33N/HJLPKe2QHK+5RMuH4P170KzUAgcZvJUczUAAAc3SURBVFx85YkQ0O9zcKwMS8eqM/tcUsK2qVC9KTTtZVyMWqnSSV8rGb5hajTI4WXqcVYmLBmrlvvr/z9d1ilNld2h738hbh9s+ezv50/9Becj1EI1NjoVlBf6X1orGTW8wN3r74la2/4H58KhzyfgXMvY2Mojn4Hg9whs/o+aeQuqsVpFNwgYYmxsWqnSSV8rOb5hcHobxPwBG/+thgT6Pmx0VOVX749Vkl8yViX+Y2shZLS6+tLKDZ30tZLjMwiQMH+omuXZ97+6rGOkitXUMM5LkfDDQLBzgtYjjY5KK2U66Wslx62pWhAlK13V8StVNzoirVkvaPkEpFxRN9P1v0m5o1saaiWrx7twKQpa9DU6Ei1Xr39DpRqqm6ZW7uikr5WsJt3UH81yOLlA97eNjkIziC7vaJqmlSMmJX0hRKgQIloIESOEmFjA658JIfbn/DkqhLh22+suQohzQojp5gpc0zRNu3+FlneEELbADKAHEAvsFkIsl1LmLXAupZyQb/sXgJa37eY9YLNZItY0TdOKzJQz/RAgRkp5QkqZDiwABtxj+6HA/NwHQohWQE1gXXEC1TRN04rPlKTvAZzN9zg257k7CCHqAw2BDTmPbYBPgdfudQAhxGghRLgQIjw+Pt6UuDVN07QiMCXpFzSbRt5l2yHAIillbvPu54FVUsqzd9le7UzK2VLKYCllsLu7uwkhaZqmaUVhypDNWKBuvseeQNxdth0CjMv3uB3QQQjxPFAZcBBCJEsp77gZrGmappU8U5L+bqCpEKIhcA6V2O/oiyuEaA5UBbbnPifl/7d3ryFSlXEcx7+/3MoLheYNS6ICqUDSTMQSlrKbltgFBMPALnQBK+tNtfQi6lVRL4peBKGWkBlmSVJkLkb4ag3XS62tJpXZlrmGXSihNH+9OM/G7uZuS7t7nnHm/4HlzMzuzP/HmZ3/zHnOnOd4Uaff3wFMj4YfQgj5/GfTt31M0gPAh8AQYIXtXZKeBrbaXp/+9DbgTds9Df30SXNz84+SvunHQ4wBejhPX2kqIQNEju4iR1eVkKMSMkB15OjTCY7Vzx5dcSRttT291jNEjshxMuSohAy1liOOyA0hhBoSTT+EEGpINTb9V3IHoDIyQOToLnJ0VQk5KiED1FCOqhvTDyGE0LNq/KQfQgihB1XR9CUNlfSJpJ2Sdkl6KnOeIZK2S3ovY4Z9kj5LM59uzZhjpKS1knZLapV0eYYMF3aaBXaHpF8lPZwhxyPp/7NF0mpJQ8vOkHIsTRl2lbkeJK2Q1C6ppdNtZ0lqlLQ3LUdlyrEgrY/jkkr5Fk8POZ5Lr5VPJa2TNHKg61ZF0wf+AGbbngJMBeZImpkxz1KgNWP9DlfZnpr5q2gvAhtsXwRMIcN6sb0nrYepwGXAEWBdmRkknQM8RHGA4mSKY14Wlpkh5ZgM3EMxkeIUYJ6kSSWVfw2Y0+22x4FNticBm9L1HDlagFuBzSXU7y1HIzDZ9iXAF0DDQBetiqbvwm/p6qnpJ8vOCkkTgRuBZTnqVxJJZwL1wHIA23/a/rn3ew26q4EvbffnAMD/qw4YJqkOGE7P05kMpouBJttHbB+jmPL8ljIK294MHO52803AynR5JXBzjhy2W23vGezafcixMT0vAE0U094MqKpo+vDPkMoOoB1otL0lU5QXgEeB45nqdzCwUVKzpHszZbgAOAS8moa7lkkakSlLh4V0mvq7LLa/A54H9gMHgF9s55huvAWolzRa0nDgBrrOrVW28bYPAKTluIxZKs1dwAcD/aBV0/Rt/5U23ycCM9JmbKkkzQPabTeXXfsEZtmeBswFlkiqz5ChDpgGvGz7UuB3ytl8PyFJpwHzgbcy1B5F8an2fOBsYISk28vOYbsVeJZiGGEDsBM41uudQukkPUHxvKwa6MeumqbfIQ0ffMy/x8rKMAuYL2kfxclmZkt6PUMObH+flu0U49czMsRoA9o6bXWtpXgTyGUusM32wQy1rwG+tn3I9lHgHeCKDDmwvdz2NNv1FMMLe3PkSA5KmgCQlu0Zs1QESYuBecCi/s5ldiJV0fQlje3Yyy1pGMULbHfZOWw32J5o+zyKYYSPbJf+aU7SCElndFwGrqPYrC+V7R+Ab9MMrFCMp3/ey10GW5ezupVsPzBT0nBJolgXWXb2SxqXludS7LzMtU4A1gOL0+XFwLsZs2QnaQ7wGDDf9pHBqNGXqZVPBhOAlel8vqcAa2xn+7pkBRgPrCt6C3XAG7Y3ZMryILAqDa18BdyZI0Qav74WuC9HfdtbJK0FtlFstm8n31Ggb0saDRwFltj+qYyiklYDVwJjJLUBTwLPAGsk3U3xxrggU47DwEvAWOB9STtsX58hRwNwOtCYXr9Ntu8f0LpxRG4IIdSOqhjeCSGE0DfR9EMIoYZE0w8hhBoSTT+EEGpINP0QQqgh0fRDCKGGRNMPIYQaEk0/hBBqyN8ZFxF/jOuLZgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x224cfd48fd0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(training_error, label = 'Training set accuracy')\n",
    "plt.plot(test_error, label = 'Test set accuracy')\n",
    "plt.legend()\n",
    "plt.xticks(np.arange(0, 10), np.arange(3, 13))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Naive Bayes \n",
    "\n",
    "With Naive Bayes classifier we predict the class of a new $x$ to be the most probable lable given model and training data $(x_1, y_1), ..., (x_n, y_n)$.\n",
    "\n",
    "#### Bayes Classifier\n",
    "Before talking about the algorithm of Naive Bayes, we need to know **Bayes Classifer**:\n",
    "\n",
    "$$f(x) = \\operatorname*{arg\\,max}_{y \\in Y} P(Y = y| X = x)$$\n",
    "\n",
    "For a particular input $x$, predict the label to be the most probable label conditioned on $x$ according to the true underlying distribution given to us from nature.\n",
    "\n",
    "#### Bayes Rule: \n",
    "$$p(y|x) = \\frac{p(x|y)p(y)}{p(x)}$$\n",
    "\n",
    "From Bayes rule we equivalently have\n",
    "$$ f(x) \\approx \\operatorname*{arg\\,max}_{y \\in Y} P(Y = y) \\times P(X = x | Y = y) $$\n",
    "- $P(Y = y)$ is called the $\\textit{class prior}$.\n",
    "- $P(X = x|Y = y)$ is called the $\\textit{class conditional distribution}$ of X.\n",
    "- In practice we don't know either of these, so we approximate them.\n",
    "\n",
    "Aside: If $X$ is a continuous-valued random variable, replace $P(X = x|Y = y)$ with class conditional density $p(x|Y=y)$.\n",
    "\n",
    "Problem: We can't construct the Bayes classifier without knowing $P(Y = y|X = x)$, or equv., $P(X = x|Y = y)$ and $P(Y = y)$. All we have are labeled examples drowm from the distribution.\n",
    "\n",
    "#### Naive Bayes Algorithm\n",
    "We have to $\\textit{define } p(X = x|Y = y)$.\n",
    "\n",
    "Naive Bayes is a Bayes classifier that makes the assumption\n",
    "$$p(X = x|Y = y) = \\prod_{j=1}^{d}p_j(x(j)|Y = y),$$\n",
    "i.e., it treats the dimension of $X$ as $\\textit{conditionally independent}$ given $y$. \n",
    "\n",
    "Note: Each dimension might not be independent with each other, but they are conditionally independent given y.\n",
    "\n",
    "#### Discriminative Model vs. Generative Model\n",
    "\n",
    "Bayes Classifer and Naive Bayes are ***generative models***. Unlike distriminative models, generative models consider the joint probability distribution on $X \\times y, P(X, y)$, make the prediction on the probability of each lable $y$ given $X$, $p(y|X = x)$, and then **pick the most likely lable $y$**.\n",
    "\n",
    "##### Discriminative Algorithms\n",
    "- Idea: model $p(y|x)$, conditional distribution of $y$ given $x$.\n",
    "- In Discriminative Algorithms: find a **decision boundary** that separates positive from negative example.\n",
    "- To predict a new example, check on which side of the decision boundary it falls.\n",
    "- **Model $p(y|x)$ directly**\n",
    "\n",
    "##### Generative Algorithms\n",
    "- Idea: Build a model for what positive examples look like and build a different model for what negative example look like.\n",
    "- To predict a new example, match it with each of the models and see which match is best.\n",
    "- Model $p(x|y)$ and $p(y)$!\n",
    "- **Use Bayes rule to obtain $p(y|x) = \\frac{p(x|y)p(y)}{p(x)}$**\n",
    "\n",
    "The definition of generative and distriminative models:\n",
    "- a generative model is a model of the conditional probability of the observable $X$, given a target $y$, symbolically, $P(X|Y=y)$\n",
    "- a discriminative model is a model of the conditional probability of the target $Y$, given an observation $x$, symbolically, $P(Y|X=x)$\n",
    "\n",
    "source: https://en.wikipedia.org/wiki/Generative_model#Definition\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on training set: 0.786069651741\n",
      "Accuracy on test set: 0.823529411765\n",
      "AUC: 0.882591093117\n"
     ]
    }
   ],
   "source": [
    "gnb = GaussianNB()\n",
    "gnb.fit(X_train, y_train)\n",
    "y_pred_prob = gnb.predict_proba(X_test)[:,1]\n",
    "\n",
    "print(\"Accuracy on training set:\", gnb.score(X_train, y_train))\n",
    "print(\"Accuracy on test set:\", gnb.score(X_test, y_test))\n",
    "print(\"AUC:\", roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Logistic Regression\n",
    "Let $(x_1, y_1),...,(x_n, y_n)$ be a set of binary labeled data with $y \\in {-1, +1}$. $\\textit{Logistic regression}$ models each $y_i$ as independently generated, with\n",
    "\n",
    "$$P(y_i = +1|x_i, w) = \\sigma(x_i^Tw), \\ \\ \\sigma(x_i; w) = \\frac{e^{x_iTw}}{1 + e^{x_iTw}}.$$\n",
    "\n",
    "#### Sigmoid Function\n",
    "From **Linear Discriminative Analysis**, we can directly plug in the **hyperplane representation** for the **log odds**. ***(See Appendix A: Concepts for Logistic Regression & Appendix B: Linear Classifiers)***\n",
    "\n",
    "$$\\ln\\frac{p(y = +1|x)}{p(y = -1|x)} = x^Tw + w_0$$\n",
    "\n",
    "Note: No restrictions on $w$ and $w_0$ compared to LDA.\n",
    "\n",
    "Setting $p(y = -1|x) = 1 - p(y = -1|x)$, solve for $p(y = +1|x)$ to find \n",
    "$$ p(y = +1|x) = \\frac{exp^{x^Tw + w_0}}{1 + exp^{x^Tw + w_0}} = \\sigma(x^tw + w_0). $$\n",
    "\n",
    "- This is called the sigmoid function\n",
    "- We have chosen $x^Tw + w_0$ as the $\\textit{link function}$ for log odds.\n",
    "- If $x^Tw > 0$, then $\\sigma(x^Tw) > 1/2$ and predict $y = +1$, and vice versa.\n",
    "- We now get **a confidence in our prediction** via the probability of $\\sigma (x^Tw)$.\n",
    "\n",
    "\n",
    "#### Maximum Likelihood Estimation\n",
    "\n",
    "Define $\\sigma_i(w) = \\sigma(x_i^Tw)$. The joint likihood of $y_1,...y_n$ is\n",
    "$$p(y_1,...,y_n|x_1,...,x_n, w) = \\prod_{i=1}^{n}p(y_i|x_i, w)\n",
    "= \\prod_{i=1}^{n}\\sigma_i(w)^{\\mathbb{I}(y_i=+1)}(1-\\sigma_i(w))^{\\mathbb{I}(y_i=-1)} \n",
    "= \\prod_{i=1}^{n}\\sigma_i(y_i \\cdot w)$$\n",
    "\n",
    "Note: here y = {+1, -1}\n",
    "\n",
    "we want to maximize this over $w$.\n",
    "\n",
    "The maximum likelihood solution for $w$ can be written \n",
    "$$ w_{ML} = \\operatorname*{arg\\,max}_{w} \\sum_{i=1}^{n}\\ln\\sigma_i(y_i\\cdot w)\n",
    "= \\operatorname*{arg\\,max}_{w} L $$\n",
    "\n",
    "We can't directly set $\\nabla_w L = 0$, so we need an iterative algorithm. At step $t$, we can update\n",
    "$$ w^{(t+1)}=w^{(t)}+\\eta\\nabla_wL \\ \\ \\nabla_wL = \\sum_{i=1}{n}(1-\\sigma_i(y_i\\cdot w))y_ix_i $$\n",
    "\n",
    "\n",
    "#### Algorithm\n",
    "\n",
    "**Input**: Training data $(x_1, y_1),...,(x_n, y_n)$ and step size $\\eta > 0$\n",
    "1. **Set** $w^{(1)} = \\overrightarrow{0}$\n",
    "2. **For step** $t = 1,2,...$ **do**\n",
    "    - Update $w^{(t+1)} = w^{(t)} + \\eta\\sum_{i=1}^{n}(1-\\sigma_i(y_i\\cdot w))y_ix_i$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Parameters: {'C': 1, 'penalty': 'l2', 'tol': 0.0001, 'max_iter': 100}\n",
      "Accuracy on Training Set: 0.8407960199\n",
      "Accuracy on Test Set: 0.862745098039\n",
      "AUC: 0.868421052632\n",
      "Wall time: 24.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Build logistic model with L2 regularization\n",
    "param_grid = {'penalty': ['l1', 'l2'],\n",
    "              'C': [0.1, 1, 10, 100], \n",
    "              'tol': [1e-4, 1e-5], \n",
    "              'max_iter': [100, 500]}\n",
    "gs_lr = GridSearchCV(LogisticRegression(), param_grid, cv=5)\n",
    "gs_lr.fit(X_train, y_train)\n",
    "print(\"Best Parameters:\", gs_lr.best_params_)\n",
    "print(\"Accuracy on Training Set:\", gs_lr.best_score_)\n",
    "\n",
    "y_pred_prob = gs_lr.predict_proba(X_test)[:,1]\n",
    "print(\"Accuracy on Test Set:\", gs_lr.score(X_test, y_test))\n",
    "print(\"AUC:\", roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.9965379   0.99630913  0.19536443  0.93574275  0.98994636]\n",
      "[1 1 0 1 1]\n",
      "id\n",
      "214    1\n",
      "108    1\n",
      "250    0\n",
      "10     1\n",
      "200    1\n",
      "Name: class, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "y_pred = gs_lr.predict(X_test)\n",
    "print(y_pred_prob[:5])\n",
    "print(y_pred[:5])\n",
    "print(y_test[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Threshold: 0.3, Accuracy: 0.8627450980392157\n",
      "Threshold: 0.4, Accuracy: 0.8627450980392157\n",
      "Threshold: 0.5, Accuracy: 0.8627450980392157\n",
      "Threshold: 0.6, Accuracy: 0.803921568627451\n",
      "Threshold: 0.7, Accuracy: 0.803921568627451\n",
      "Threshold: 0.8, Accuracy: 0.7450980392156863\n"
     ]
    }
   ],
   "source": [
    "# Moving the threshold does not improve prediction\n",
    "for t in [0.3, 0.4, 0.5, 0.6, 0.7, 0.8]:\n",
    "    y_pred_rev = [1 if i > t else 0 for i in y_pred_prob]\n",
    "    print('Threshold: {}, Accuracy: {}'.format(t, accuracy_score(y_pred_rev, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Support Vector Machine\n",
    "\n",
    "With two linearly separable classes, choose a hyperplane such that its distance to the **closest point in each class** is maximized to achieve good generalization (low prediction error).\n",
    "\n",
    "#### Convex Sets and Convex Hulls\n",
    "Where a seperating hyperplane may be placed depends on the \"outer\" points on the sets. Points in the center do not matter. In geometric terms, we can represent each class by the smallest convex set which contains all point in the class. This is called a $\\textit{convex hull}$.\n",
    "\n",
    "A convex hull is defined by all possible weighted averages of points in a set. That is, let $x_1,...x_n$ be the data coordinates. Every point $x_0$ in the convex hull can be reached by setting\n",
    "$$ x_0 = \\sum_{i=1}^{n}\\alpha_ix_i, \\ \\ \\alpha_i \\geq0, \\ \\ \\sum_{i=1}^{n}\\alpha_i = 1,$$\n",
    "for some $(\\alpha_1,...\\alpha_n)$. No point outside the convex hull can be reached this way.\n",
    "\n",
    "#### Algorithm\n",
    "For $n$ seperate points $(x_1,y_1),...,(x_n,y_n)$ with $y_i \\in {\\pm1}$, solve:\n",
    "$$ \\min_{w, w_0} \\frac{1}{2}\\|w\\|^2 $$\n",
    "subject to\n",
    "$$ y_i(x_i^Tw + w_0) \\geq 1 \\ \\ for\\ i = 1,...,n$$\n",
    "\n",
    "- If there exists a hyperplan $H$ that separates the classes, we can scale $w$ so that $y_i(x_i^Tw+w_0)>1$ for all $i$.\n",
    "- This formula only has a solution when the classes are linearly separable.\n",
    "\n",
    "Solving above foluma would require $\\textit{Lagrange multipliers}$. After derived with $\\textit{Lagrange multipliers}$, the formula will eventually turn into:\n",
    "$$\\min_{\\alpha_1,...\\alpha_n}\\bigg|\\bigg(\\sum_{i\\in S_1}\\frac{\\alpha_i}{C}x_i\\bigg) - \n",
    "\\bigg(\\sum_{j\\in S_0}\\frac{\\alpha_j}{C}x_j\\bigg)\\bigg|^2,$$\n",
    "where \n",
    "- $S_i$ and $S_0$ are the sets of $x$ in class $+1$ and $-1$\n",
    "- $C:=\\sum_{i\\in S_1}\\alpha_u = \\sum_{j\\in S_0}\\alpha_j,\\ \\ \\alpha_i\\geq0$\n",
    "\n",
    "Therefore, the algorithm is to find the closest points in the convex hulls constructed from the data in class $+1$ and $-1$.\n",
    "\n",
    "#### Soft-Margin SVM\n",
    "\n",
    "If the data isn't linearly separable, permit training data be on wrong side of hyperplane at a cost by replacing the training rule $y_i(x_i^Tw + w_0) \\geq 1$  with \n",
    "$$y_i(x_i^Tw+w_0)\\geq 1 - \\xi_i, \\ \\ with \\ \\ \\xi_i \\geq0.$$\n",
    "\n",
    "The $\\xi_i$ are also called $\\textit{slack variables}.$\n",
    "\n",
    "The function therefore becomes:\n",
    "$$ \\min_{w, w_0,\\xi_1,...\\xi_n} \\frac{1}{2}\\|w\\|^2 + \\lambda\\sum_{i=1}^{n}\\xi_i$$\n",
    "subject to\n",
    "$$ y_i(x_i^Tw + w_0) \\geq 1 - \\xi_i \\ \\ for\\ i = 1,...,n$$\n",
    "$$ \\xi_i \\geq 0 \\ \\ for \\ \\ i = 1,...,n$$\n",
    "\n",
    "- If $\\lambda$ is very small, we're happy to misclassify.\n",
    "- For $\\lambda \\rightarrow \\infty$, we recover the original SVM because we want $\\xi_i=0$.\n",
    "- We can use cross-valudation to choose $\\lambda$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.svm import LinearSVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Parameters: {'C': 1, 'tol': 0.0001}\n",
      "Accuracy on Training Set: 0.810945273632\n",
      "Accuracy on Test Set: 0.803921568627\n",
      "AUC: 0.803643724696\n",
      "Wall time: 22.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#l2 penalty\n",
    "param_grid = {'tol': [1e-4, 1e-5, 1e-6],\n",
    "              'C': [0.1, 1, 10],\n",
    "              #'max_iter': [500, 1000, 1500]\n",
    "             }\n",
    "gs_lsvc = GridSearchCV(SVC(kernel ='linear', probability=True), param_grid, cv=5)\n",
    "gs_lsvc.fit(X_train, y_train)\n",
    "print(\"Best Parameters:\", gs_lsvc.best_params_)\n",
    "print(\"Accuracy on Training Set:\", gs_lsvc.best_score_)\n",
    "\n",
    "y_pred_prob = gs_lsvc.predict_proba(X_test)[:,1]\n",
    "print(\"Accuracy on Test Set:\", gs_lsvc.score(X_test, y_test))\n",
    "print(\"AUC:\", roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Parameters: {'C': 0.1}\n",
      "Accuracy on Training Set: 0.791044776119\n",
      "Accuracy on Test Set: 0.78431372549\n"
     ]
    }
   ],
   "source": [
    "warnings.simplefilter('ignore')\n",
    "#l1 penalty\n",
    "lsvc = LinearSVC(loss='l2', penalty='l1', dual=False)\n",
    "gs_lsvc_l1 = GridSearchCV(lsvc, {'C': [0.1, 1, 10]}, cv=5)\n",
    "gs_lsvc_l1.fit(X_train, y_train)\n",
    "print(\"Best Parameters:\", gs_lsvc_l1.best_params_)\n",
    "print(\"Accuracy on Training Set:\", gs_lsvc_l1.best_score_)\n",
    "print(\"Accuracy on Test Set:\", gs_lsvc_l1.score(X_test, y_test))\n",
    "# LinearSVC does not return probability of prediction to calculate AUC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Kernelizing SVM\n",
    "\n",
    "#### Kernel\n",
    "A kernel $K(\\cdot,\\cdot) : \\mathbb{R}^d \\times \\mathbb{R}^d \\rightarrow \\mathbb{R}$ is a symmetric function defined as follows:\n",
    "\n",
    "For any set of $n$ data points $x_1,...,x_n \\in \\mathbb{R}^d$, the $n\\times n$ matrix $K$, where $K_{ij} = K(x_i, x_j)$, is $\\textit{positive semidefinite}$. (Note: The output of the kernel function is greater or equal to 0.)\n",
    "\n",
    "Intuitively, this means $K$ satisfies the properties of a covariance matrix.\n",
    "\n",
    "#### Mercer's theorem\n",
    "If the function $K(\\cdot,\\cdot)$ satisfies the above properties, then there exists a mapping $\\phi:\\mathbb{R}^d\\rightarrow\\mathbb{R}^D$ such that\n",
    "$$k(x_i, x_j) = \\phi(x_i)^T\\phi(x_j).$$\n",
    "\n",
    "(Note: If this is satisfied for every single set of n vectors in $\\mathbb{R}^d$ where $n$ is arbitrary and the vectors themselves are arbitrary, **then there exists some function** $\\phi$, which is a function that takes in any particular $x$ and it performs the same function on that particular $x$ to map it to another space. And we can then have this dot product representation of the kernel function.)\n",
    "\n",
    "If we first define $\\phi(\\cdot)$, the mapping function, and then $K$, then this is obvious. However, sometimes we first define $K(\\cdot,\\cdot)$ and avoid ever using $\\phi(\\cdot)$.\n",
    "\n",
    "(Note: We use kernel to get results of the dot products of two $x_i$ and $x_j$ mapped to a higher place without actually mapping the $x_i$ and $x_j$ and calculating the dot product.)\n",
    "\n",
    "#### RBF\n",
    "By far the most popular kernel is the Gaussian kernel, also called the radiial basis function (RBF),\n",
    "$$ K(x, x') = \\alpha \\ exp\\big\\{ - \\frac{1}{b}\\|x-x'\\|^2 \\big\\}. $$\n",
    "\n",
    "- It takes into account proximity in $\\mathbb{R}^d$. Things close together in space have larger value (as defined by kernel width $b$).\n",
    "\n",
    "In this case, the mapping $\\phi(x)$ that produces the RBF kernel is $\\textit{infinite dimensional}$ (it's a continuous function instead of a vector). Therefore\n",
    "$$K(x,x') = \\int\\phi_t(x)\\phi_t(x')dt.$$\n",
    "\n",
    "(Note: there's a function of $x$ and $t$ such that this kernel results by integratign the product of those two functions.)\n",
    "- $K(x,x')$ is like a Gaussian on $x$ with $x'$ as the mean (or vice versa).\n",
    "\n",
    "#### Algorithm\n",
    "Map the data into higher dimension using the function $\\phi(x_i)$,\n",
    "$$ \\min_{w, w_0,\\xi_1,...\\xi_n} \\frac{1}{2}\\|w\\|^2 + \\lambda\\sum_{i=1}^{n}\\xi_i$$\n",
    "subject to\n",
    "$$ y_i(\\phi(x_i)^Tw + w_0) \\geq 1 - \\xi_i \\ \\ for\\ i = 1,...,n$$\n",
    "$$ \\xi_i \\geq 0 \\ \\ for \\ \\ i = 1,...,n$$\n",
    "\n",
    "To classify a new point:\n",
    "$$ y_0 = sign\\big(\\sum_{i=1}^{n}\\alpha_iy_i\\phi(x_0)T\\phi(x_i)+w_0)\\big)=\n",
    "sign(\\sum_{i=1}^{n}\\alpha_iy_iK(x_0,x_i)+w_0\\big)$$\n",
    "\n",
    "- We're still learning a linear classifier, in the higher dimensional map space, but when we look at what the decision is in the original space, we get non-linear decision boundaries.\n",
    "- In practice, we choose a kernel function (e.g., RBF) and use cross-validation for $\\lambda$ parameter and RBF kernel width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Parameters: {'C': 10, 'tol': 0.001, 'gamma': 0.01}\n",
      "Accuracy on Training Set: 0.820895522388\n",
      "Accuracy on Test Set: 0.862745098039\n",
      "AUC: 0.878542510121\n",
      "Wall time: 2min 52s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "param_grid = {'C': [0.01, 0.1, 1, 10, 100],\n",
    "              'gamma': [1e-2, 1/753, 1e-4, 1e-5],\n",
    "              'tol': [1e-3, 1e-4, 1e-5],\n",
    "              #'max_iter': [500, 1000]\n",
    "             }\n",
    "gs_svc = GridSearchCV(SVC(probability=True), param_grid, cv=5)\n",
    "gs_svc.fit(X_train, y_train)\n",
    "print(\"Best Parameters:\", gs_svc.best_params_)\n",
    "print(\"Accuracy on Training Set:\", gs_svc.best_score_)\n",
    "\n",
    "y_pred_prob = gs_svc.predict_proba(X_test)[:,1]\n",
    "print(\"Accuracy on Test Set:\", gs_svc.score(X_test, y_test))\n",
    "print(\"AUC:\", roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far support vector machine with kernel generates the best results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Decision Tree\n",
    "A decision tree maps input $x \\in \\mathbb{R}^d$ to output $y$ using binary decision rules:\n",
    "- Each node in the tree has a splitting rule.\n",
    "- Each leaf node is associated with an output value (outputs can repeat)\n",
    "\n",
    "Each splitting rule is of the form $h(x) = \\mathbb{I}\\{x_j>t\\}$ for some dimension $j$ of $x$ and $t\\in\\mathbb{R}$. Using these transition rules, a path to a leaf node gives the prediction.\n",
    "\n",
    "The basic method for learning tree is with a top-down greedy algorithm. Measure of quality of prediction include\n",
    "1. Classificataion error: $1 - \\max_kp_k$\n",
    "2. Gini index: $1 - \\sum_{k}p_k^2$\n",
    "3. Entrophy: $-\\sum_{k}p_k\\ln p_k$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Parameters: {'criterion': 'gini', 'max_depth': 5, 'min_samples_split': 2}\n",
      "Accuracy on Training Set: 0.796019900498\n",
      "Accuracy on Test Set: 0.725490196078\n",
      "AUC: 0.644736842105\n",
      "Wall time: 37 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "param_grid = {'criterion': ['gini', 'entropy'],\n",
    "              'max_depth': [3, 5, 8, None],\n",
    "              'min_samples_split': [2, 3, 5]}\n",
    "gs_dt = GridSearchCV(DecisionTreeClassifier(), param_grid, cv=5)\n",
    "gs_dt.fit(X_train, y_train)\n",
    "print(\"Best Parameters:\", gs_dt.best_params_)\n",
    "print(\"Accuracy on Training Set:\", gs_dt.best_score_)\n",
    "\n",
    "y_pred_prob = gs_dt.predict_proba(X_test)[:,1]\n",
    "print(\"Accuracy on Test Set:\", gs_dt.score(X_test, y_test))\n",
    "print(\"AUC:\", roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III. Model Improvement\n",
    "\n",
    "Now there are 753 features, including 752 numeric features, in the dataset. Considering the massive number of features, using dimension reduction method to reduce features and leave only more discriminative ones would improve modeling speed and potentially prediction performance.\n",
    "\n",
    "### PCA\n",
    "Principle component analysis is often used for dimension reduction. Imagine we're given data in $R^d$, where $d$ is very high-dimensional, and we don't think that the information fills all of those dimensions. We can **take all the information contained in that high-dimensional space and map it to a much lower-dimensional space, where we don't lose much information**. \n",
    "\n",
    "#### Algorithm - The First Principal Component\n",
    "This is related to the problem of finding the largest eigenvalue,\n",
    "$$q = \\operatorname*{arg\\,min}_{q} \\sum_{i=1}^{n}\\|x_i - qq^Tx_i\\|^2 \\ \\ \\ \\  s.t.\\ \\ q^Tq = 1 $$\n",
    "$$ = \\operatorname*{arg\\,min}_{q} \\sum_{i=1}^{n}x_i^Tx_i - q^T\\big(\\sum_{i=1}^{n}x_ix_i^T\\big)q$$\n",
    "$$\\sum_{i=1}^{n}x_ix_i^T = XX^T$$\n",
    "\n",
    "We define $X = [x_1,...,x_n]$. Since the first term doesn't depend on $q$ and we have a negative sing in front of the second term, equivalently we solve \n",
    "$$q = \\operatorname*{arg\\,max}_{q} q^T(XX^T)q \\ \\ \\ \\ subject to \\ \\ q^Tq = 1$$\n",
    "\n",
    "This is the eigendecomposition problem:\n",
    "- $q$ is the first eigenvector of $XX^T$\n",
    "- $\\lambda = q^T(XX^T)q$ is the first eigenvalue\n",
    "\n",
    "\n",
    "#### Algorithm - General\n",
    "The general form of PCA considers $K$ eigenvectors,\n",
    "$$q = \\operatorname*{arg\\,min}_{q} \\sum_{i=1}^{n}\\|x_i - \\sum_{k=1}^{K}(x_i^Tq_k)q_k\\|^2 \\ \\ \\ \\  s.t.\\ \\ q^Tq = 1, k = k'; q^Tq = 0, k \\neq k' $$\n",
    "$$ = \\operatorname*{arg\\,min}_{q} \\sum_{i=1}^{n}x_i^Tx_i - \\sum_{k=1}^{K}q_k^T \\big(\\sum_{i=1}^{n}x_ix_i^T\\big)q_k$$\n",
    "$$\\sum_{i=1}^{n}x_ix_i^T = XX^T$$\n",
    "\n",
    "The vectors in $Q = [q_i,...,q_k]$ give us a $K$ dimensional subspace with which to represent the data:\n",
    "$$x_{proj} = \\begin{bmatrix}\n",
    "               q_1^Tx \\\\\n",
    "               \\vdots \\\\\n",
    "               q_K^Tx\n",
    "             \\end{bmatrix}, x \\approx \\sum_{k=1}^{K}(q_k^Tx)q_k = Qx_{proj}$$\n",
    "             \n",
    "The eigenvectors of $(XX^T)$ can be learned as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.groupby('id').mean().drop(['class'], axis=1)\n",
    "y = df[['id', 'class']].groupby('id').mean()['class']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=91)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PCA(copy=True, iterated_power='auto', n_components=None, random_state=None,\n",
       "  svd_solver='auto', tol=0.0, whiten=False)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler = MinMaxScaler()\n",
    "pca = PCA()\n",
    "pca.fit(scaler.fit_transform(X))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>variance_ratio</th>\n",
       "      <th>cumulated_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.200985</td>\n",
       "      <td>0.200985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.131812</td>\n",
       "      <td>0.332797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.083290</td>\n",
       "      <td>0.416087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.060048</td>\n",
       "      <td>0.476135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.039805</td>\n",
       "      <td>0.515941</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   variance_ratio  cumulated_ratio\n",
       "0        0.200985         0.200985\n",
       "1        0.131812         0.332797\n",
       "2        0.083290         0.416087\n",
       "3        0.060048         0.476135\n",
       "4        0.039805         0.515941"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pca = pd.DataFrame({'variance_ratio': pca.explained_variance_ratio_})\n",
    "df_pca['cumulated_ratio'] = df_pca['variance_ratio'].cumsum()\n",
    "df_pca.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFpxJREFUeJzt3X+QXeV93/H3J8LCTjKYX+sOkSCSx0ocxW5xLGTa1DTFv0RxETOFIEoM9pBR4wltWtep5bbGrWLPQNoprSeMa8Vg45+Y4LjeKaKqbey002KqxVBAUJVFVmEtGtYVJkwcQ2S+/eM+S66vd9lzV1datPt+zZzZc57znOc8zxy4H50f99xUFZIk/cRid0CS9OJgIEiSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUnPcYndgGKeeemqtWbNmsbshSceUu++++7tVNTZfvWMqENasWcPExMRid0OSjilJ/k+Xel4ykiQBBoIkqTEQJEmAgSBJagwESRJgIEiSmk6BkGRTkr1JJpNsm2X9e5I8mOS+JF9L8rN9665I8nCbrugrf32S+1ubH0mS0QxJkrQQ8wZCkhXA9cB5wHrg0iTrB6rdA2yoqr8M3Ar8btv2ZOCDwBuAjcAHk5zUtvkosBVY16ZNhz0aSdKCdTlD2AhMVtW+qnoWuBnY3F+hqr5eVd9vi98EVrf5twFfqaqDVfUk8BVgU5LTgBOq6s7q/ajzp4ALRzAeSdICdQmEVcBjfctTrWwuVwK3z7Ptqjbftc3DtmbbbazZdtuR3IUkHdO6vLpitmv7NWvF5NeADcDfmGfbYdrcSu/SEmecccZ8fZUkLVCXM4Qp4PS+5dXAgcFKSd4M/DPggqp6Zp5tp/iLy0pztglQVTuqakNVbRgbm/fdTJKkBeoSCLuBdUnWJlkJbAHG+yskeR3wMXph8ETfql3AW5Oc1G4mvxXYVVWPA08nObs9XXQ58OURjEeStEDzXjKqqkNJrqL34b4CuLGq9iTZDkxU1Tjwr4CfBv6gPT36aFVdUFUHk/wOvVAB2F5VB9v8u4FPAi+jd8/hdiRJi6bT66+raiewc6Ds6r75N7/AtjcCN85SPgG8pnNPJUlHlN9UliQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAR0DIcmmJHuTTCbZNsv6c5J8K8mhJBf1lf/NJPf2TT9IcmFb98kk3+5bd+bohiVJGta8P6GZZAVwPfAWYArYnWS8qh7sq/Yo8E7gvf3bVtXXgTNbOycDk8B/7qvy21V16+EMQJI0Gl1+U3kjMFlV+wCS3AxsBp4PhKra39Y99wLtXATcXlXfX3BvJUlHTJdLRquAx/qWp1rZsLYAnx8o+3CS+5Jcl+T42TZKsjXJRJKJ6enpBexWktRFl0DILGU1zE6SnAa8FtjVV/x+4NXAWcDJwPtm27aqdlTVhqraMDY2NsxuJUlD6BIIU8DpfcurgQND7udXgS9V1Z/PFFTV49XzDPAJepemJEmLpEsg7AbWJVmbZCW9Sz/jQ+7nUgYuF7WzBpIEuBB4YMg2JUkjNG8gVNUh4Cp6l3seAm6pqj1Jtie5ACDJWUmmgIuBjyXZM7N9kjX0zjD+aKDpzya5H7gfOBX40OEPR5K0UF2eMqKqdgI7B8qu7pvfTe9S0mzb7meWm9BVde4wHZUkHVl+U1mSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgR0DIQkm5LsTTKZZNss689J8q0kh5JcNLDuh0nubdN4X/naJHcleTjJF9rvNUuSFsm8gZBkBXA9cB6wHrg0yfqBao8C7wQ+N0sTf1ZVZ7bpgr7ya4Hrqmod8CRw5QL6L0kakS5nCBuByaraV1XPAjcDm/srVNX+qroPeK7LTpMEOBe4tRXdBFzYudeSpJHrEgirgMf6lqdaWVcvTTKR5JtJZj70TwG+V1WH5mszyda2/cT09PQQu5UkDeO4DnUyS1kNsY8zqupAklcCdyS5H/iTrm1W1Q5gB8CGDRuG2a8kaQhdzhCmgNP7llcDB7ruoKoOtL/7gG8ArwO+C5yYZCaQhmpTkjR6XQJhN7CuPRW0EtgCjM+zDQBJTkpyfJs/Ffhl4MGqKuDrwMwTSVcAXx6285Kk0Zk3ENp1/quAXcBDwC1VtSfJ9iQXACQ5K8kUcDHwsSR72ua/AEwk+Z/0AuCaqnqwrXsf8J4kk/TuKdwwyoFJkobT5R4CVbUT2DlQdnXf/G56l30Gt/vvwGvnaHMfvSeYJEkvAn5TWZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSUDHt50uNWu23fb8/P5rzl/EnkjSi4dnCJIkwECQJDWdAiHJpiR7k0wm2TbL+nOSfCvJoSQX9ZWfmeTOJHuS3Jfkkr51n0zy7ST3tunM0QxJkrQQ895DSLICuB54CzAF7E4y3vdTmACPAu8E3juw+feBy6vq4SQ/A9ydZFdVfa+t/+2quvVwByFJOnxdbipvBCbbT16S5GZgM/B8IFTV/rbuuf4Nq+p/980fSPIEMAZ8D0nSi0qXS0argMf6lqda2VCSbARWAo/0FX+4XUq6Lsnxw7YpSRqdLoGQWcpqmJ0kOQ34NPCuqpo5i3g/8GrgLOBk4H1zbLs1yUSSienp6WF2K0kaQpdAmAJO71teDRzouoMkJwC3Af+8qr45U15Vj1fPM8An6F2a+jFVtaOqNlTVhrGxsa67lSQNqUsg7AbWJVmbZCWwBRjv0nir/yXgU1X1BwPrTmt/A1wIPDBMxyVJozVvIFTVIeAqYBfwEHBLVe1Jsj3JBQBJzkoyBVwMfCzJnrb5rwLnAO+c5fHSzya5H7gfOBX40EhHJkkaSqdXV1TVTmDnQNnVffO76V1KGtzuM8Bn5mjz3KF6Kkk6ovymsiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCegYCEk2JdmbZDLJtlnWn5PkW0kOJbloYN0VSR5u0xV95a9Pcn9r8yPtt5UlSYtk3kBIsgK4HjgPWA9cmmT9QLVHgXcCnxvY9mTgg8AbgI3AB5Oc1FZ/FNgKrGvTpgWPQpJ02LqcIWwEJqtqX1U9C9wMbO6vUFX7q+o+4LmBbd8GfKWqDlbVk8BXgE1JTgNOqKo7q6qATwEXHu5gJEkL1yUQVgGP9S1PtbIu5tp2VZtfSJuSpCOgSyDMdm2/OrY/17ad20yyNclEkonp6emOu5UkDatLIEwBp/ctrwYOdGx/rm2n2vy8bVbVjqraUFUbxsbGOu5WkjSsLoGwG1iXZG2SlcAWYLxj+7uAtyY5qd1Mfiuwq6oeB55OcnZ7uuhy4MsL6L8kaUTmDYSqOgRcRe/D/SHglqrak2R7kgsAkpyVZAq4GPhYkj1t24PA79ALld3A9lYG8G7g48Ak8Ahw+0hHJkkaynFdKlXVTmDnQNnVffO7+dFLQP31bgRunKV8AnjNMJ2VJB05flNZkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKnp9HK7pWzNttuen99/zfmL2BNJWlyeIUiSAANBktQYCJIkwECQJDWdAiHJpiR7k0wm2TbL+uOTfKGtvyvJmlZ+WZJ7+6bnkpzZ1n2jtTmz7hWjHJgkaTjzBkKSFcD1wHnAeuDSJOsHql0JPFlVrwKuA64FqKrPVtWZVXUm8A5gf1Xd27fdZTPrq+qJEYxHkrRAXc4QNgKTVbWvqp4FbgY2D9TZDNzU5m8F3pQkA3UuBT5/OJ2VJB05XQJhFfBY3/JUK5u1TlUdAp4CThmocwk/HgifaJeLPjBLgEiSjqIugTDbB3UNUyfJG4DvV9UDfesvq6rXAm9s0ztm3XmyNclEkonp6ekO3ZUkLUSXQJgCTu9bXg0cmKtOkuOAlwMH+9ZvYeDsoKq+0/4+DXyO3qWpH1NVO6pqQ1VtGBsb69BdSdJCdAmE3cC6JGuTrKT34T4+UGccuKLNXwTcUVUFkOQngIvp3XuglR2X5NQ2/xLg7cADSJIWzbzvMqqqQ0muAnYBK4Abq2pPku3ARFWNAzcAn04ySe/MYEtfE+cAU1W1r6/seGBXC4MVwFeB3x/JiCRJC9Lp5XZVtRPYOVB2dd/8D+idBcy27TeAswfK/hR4/ZB9lSQdQX5TWZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkptOrK5aLNdtue35+/zXnL2JPJOno8wxBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqOgVCkk1J9iaZTLJtlvXHJ/lCW39XkjWtfE2SP0tyb5v+fd82r09yf9vmI0kyqkFJkoY3byAkWQFcD5wHrAcuTbJ+oNqVwJNV9SrgOuDavnWPVNWZbfqNvvKPAluBdW3atPBhSJIOV5czhI3AZFXtq6pngZuBzQN1NgM3tflbgTe90L/4k5wGnFBVd1ZVAZ8CLhy695KkkekSCKuAx/qWp1rZrHWq6hDwFHBKW7c2yT1J/ijJG/vqT83TJgBJtiaZSDIxPT3dobuSpIXoEgiz/Uu/OtZ5HDijql4HvAf4XJITOrbZK6zaUVUbqmrD2NhYh+5KkhaiSyBMAaf3La8GDsxVJ8lxwMuBg1X1TFX9P4Cquht4BPi5Vn/1PG1Kko6iLoGwG1iXZG2SlcAWYHygzjhwRZu/CLijqirJWLspTZJX0rt5vK+qHgeeTnJ2u9dwOfDlEYxHkrRA877ttKoOJbkK2AWsAG6sqj1JtgMTVTUO3AB8OskkcJBeaACcA2xPcgj4IfAbVXWwrXs38EngZcDtbZIkLZJOr7+uqp3AzoGyq/vmfwBcPMt2XwS+OEebE8BrhumsJOnI8ZvKkiTAQJAkNQaCJAkwECRJjYEgSQI6PmW0HK3Zdtvz8/uvOX8ReyJJR4dnCJIkwECQJDUGgiQJ8B5CJ95PkLQceIYgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1nQIhyaYke5NMJtk2y/rjk3yhrb8ryZpW/pYkdye5v/09t2+bb7Q2723TK0Y1qCNpzbbbnp8kaSmZ93sI7TeRrwfeAkwBu5OMV9WDfdWuBJ6sqlcl2QJcC1wCfBf421V1IMlr6P0M56q+7S5rv5wmSVpkXc4QNgKTVbWvqp4FbgY2D9TZDNzU5m8F3pQkVXVPVR1o5XuAlyY5fhQdlySNVpdAWAU81rc8xY/+K/9H6lTVIeAp4JSBOn8HuKeqnukr+0S7XPSBJBmq55KkkeoSCLN9UNcwdZL8Ir3LSH+vb/1lVfVa4I1tesesO0+2JplIMjE9Pd2hu5KkhegSCFPA6X3Lq4EDc9VJchzwcuBgW14NfAm4vKoemdmgqr7T/j4NfI7epakfU1U7qmpDVW0YGxvrMiZJ0gJ0ebndbmBdkrXAd4AtwN8dqDMOXAHcCVwE3FFVleRE4Dbg/VX132Yqt9A4saq+m+QlwNuBrx72aI4yX3onaSmZ9wyh3RO4it4TQg8Bt1TVniTbk1zQqt0AnJJkEngPMPNo6lXAq4APDDxeejywK8l9wL30gub3RzkwSdJwOr3+uqp2AjsHyq7um/8BcPEs230I+NAczb6+ezclSUeav4cwIl4+knSsMxCOAMNB0rHIdxlJkgADQZLUGAiSJMBAOCr6347qm1IlvVgZCJIkwEBYVJ4tSHoxMRBeJAwHSYvNQHgRMhwkLQYD4UXOcJB0tBgIxxDDQdKRZCAco3yUVdKo+S6jJWYmGPZfc/6s8zPLkjTIQFiG5jqbMCik5c1A0POGvexkgEhLi4GgBXuhM425Ll1JevHqdFM5yaYke5NMJtk2y/rjk3yhrb8ryZq+de9v5XuTvK1rm1qa5roZPqp5SQs37xlCkhXA9cBbgClgd5Lxqnqwr9qVwJNV9aokW4BrgUuSrAe2AL8I/Azw1SQ/17aZr01paHPdSD/SXuisyLMlHSu6XDLaCExW1T6AJDcDm4H+D+/NwL9o87cCv5ckrfzmqnoG+HaSydYeHdqUlrRhA2SY+aPhSPR7lGM2dIfXJRBWAY/1LU8Bb5irTlUdSvIUcEor/+bAtqva/HxtStKCHYlwXMzgOxpSVS9cIbkYeFtV/Xpbfgewsar+fl+dPa3OVFt+hN6ZwHbgzqr6TCu/AdhJ797FC7bZ1/ZWYGtb/Hlg78KHy6nAdw9j+2ORY14eHPPysNAx/2xVjc1XqcsZwhRwet/yauDAHHWmkhwHvBw4OM+287UJQFXtAHZ06Oe8kkxU1YZRtHWscMzLg2NeHo70mLs8ZbQbWJdkbZKV9G4Sjw/UGQeuaPMXAXdU79RjHNjSnkJaC6wD/kfHNiVJR9G8ZwjtnsBVwC5gBXBjVe1Jsh2YqKpx4Abg0+2m8UF6H/C0erfQu1l8CPjNqvohwGxtjn54kqSu5r2HsJQk2douQS0bjnl5cMzLw5Ee87IKBEnS3Hz9tSQJWEaBsBxelZHk9CRfT/JQkj1JfquVn5zkK0kebn9PWuy+jlKSFUnuSfIf2/La9gqVh9srVVYudh9HLcmJSW5N8r/a8f6ry+A4/6P23/UDST6f5KVL7VgnuTHJE0ke6Cub9bim5yPtM+2+JL90uPtfFoHQ9/qN84D1wKXttRpLzSHgH1fVLwBnA7/ZxrkN+FpVrQO+1paXkt8CHupbvha4ro33SXqvVllq/h3wn6rq1cBfoTf+JXuck6wC/gGwoapeQ+9hlJnX5CylY/1JYNNA2VzH9Tx6T26uo/ddrY8e7s6XRSDQ9/qNqnoWmHlVxpJSVY9X1bfa/NP0PiRW0RvrTa3aTcCFi9PD0UuyGjgf+HhbDnAuvVeowBIbL0CSE4Bz6D3dR1U9W1XfYwkf5+Y44GXtu04/CTzOEjvWVfVf6D2p2W+u47oZ+FT1fBM4Mclph7P/5RIIs71+Y9UcdZeE9sbZ1wF3AX+pqh6HXmgAr1i8no3cvwX+CfBcWz4F+F5VHWrLS/FYvxKYBj7RLpV9PMlPsYSPc1V9B/jXwKP0guAp4G6W/rGGuY/ryD/XlksgZJayJft4VZKfBr4I/MOq+pPF7s+RkuTtwBNVdXd/8SxVl9qxPg74JeCjVfU64E9ZQpeHZtOum28G1tJ7c/JP0btkMmipHesXMvL/1pdLIHR5/caSkOQl9MLgs1X1h634j2dOJdvfJxarfyP2y8AFSfbTuwx4Lr0zhhPbZQVYmsd6Cpiqqrva8q30AmKpHmeANwPfrqrpqvpz4A+Bv8bSP9Yw93Ed+efacgmEZfGqjHb9/Abgoar6N32r+l8tcgXw5aPdtyOhqt5fVaurag29Y3pHVV0GfJ3eK1RgCY13RlX9X+CxJD/fit5E720AS/I4N48CZyf5yfbf+cyYl/SxbuY6ruPA5e1po7OBp2YuLS3UsvliWpK/Re9fjzOvyvjwIndp5JL8deC/AvfzF9fU/ym9+wi3AGfQ+x/r4qoavHF1TEvyK8B7q+rtSV5J74zhZOAe4Nfab3IsGUnOpHcjfSWwD3gXvX/gLdnjnORfApfQe5ruHuDX6V0zXzLHOsnngV+h91bTPwY+CPwHZjmuLRh/j95TSd8H3lVVE4e1/+USCJKkF7ZcLhlJkuZhIEiSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkC4P8DA5Tvatkycs0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x224d13439e8>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar(df_pca.index[:100], df_pca['variance_ratio'][:100])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADxZJREFUeJzt3X+MZWddx/H3hy0F+VlgB4P7g13igmyIWjKWKkYr1LhtTdc/wGz9AZrq/kMBgWiWYKrWfwoaEZIVXUstEKWWSmADq9WUGoyxtVMhtdtSWUplh1a7YKlGgmXj1z/uWbxMZ/ae2TmzM/eZ9yuZzH3OfXLu9+SZfvrMM+c8m6pCktSWJ611AZKk4RnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUETwz3J9UkeSXLPEu8nyXuTHEtyd5KXD1+mJGk5+szcbwD2nOb9S4Bd3dd+4H0rL0uStBLnTOpQVZ9OsuM0XfYCH6zRo663JzkvyQuq6uHTnXfz5s21Y8fpTitJWuiuu+76SlXNTOo3Mdx72AIcH2vPd8dOG+47duxgbm5ugI+XpI0jyb/26TfEH1SzyLFFN6xJsj/JXJK5EydODPDRkqTFDBHu88C2sfZW4KHFOlbVoaqararZmZmJv1VIks7QEOF+GHhdd9fMhcBjk9bbJUmra+Kae5IPAxcBm5PMA78BPBmgqv4QOAJcChwDvg784moVK0nqp8/dMldMeL+ANwxWkSRpxXxCVZIaZLhLUoMMd0lqkOEuSQ0a4glVSdrQdhz45LL6P3jtZatUyf9z5i5JjAL6VEgv9/V6ZLhLalILAb0SLstIWveWCuAHr73sW+8tfL3RGe6SzrpWZ8vrieEuaVAG9/rgmruk3jbi2vW0MtwlAQZ3awx3aQMwuDce19ylKTYezN45onGGuzQFnF1ruQx3aY0Z3FoNhrt0lhjiOpsMd2lgi617S2ebd8tIZ8i7TrSeGe7SBIa4ppHhLnUMcbXEcNeGY4hrIzDctSEY4tpovFtGzTjd05rSRmO4a+o4A5cmM9w1FQx0aXkMd60rhrg0DMNda85Al4ZnuGtNGOjS6jLctarcZ0VaG97nrsF5T7m09gx3DcJAl9YXw11nzECX1i/DXctioEvToVe4J9mT5P4kx5IcWOT97UluS/KZJHcnuXT4UrVWDHRp+kwM9ySbgIPAJcBu4Iokuxd0+3Xgpqo6H9gH/MHQhersMtCl6dZn5n4BcKyqHqiqx4Ebgb0L+hTwrO71s4GHhitRZ4uBLrWjz33uW4DjY+154BUL+vwm8NdJ3gg8Hbh4kOq06tw5UWpTn5l7FjlWC9pXADdU1VbgUuBDSZ5w7iT7k8wlmTtx4sTyq9UgnKFL7esT7vPAtrH2Vp647HIlcBNAVf0D8FRg88ITVdWhqpqtqtmZmZkzq1hnxECXNpY+yzJ3AruS7AS+zOgPpj+zoM+XgFcDNyR5KaNwd2q+hhb+wxWSNpaJ4V5VJ5NcBdwCbAKur6qjSa4B5qrqMPA24I+TvIXRks0vVNXCpRutMmfmkk7ptXFYVR0Bjiw4dvXY63uBVw5bmiTpTLkr5JRzti5pMYb7FDLQJU3i3jKS1CBn7lPC2bqk5TDc1zEDXdKZcllGkhrkzH2dcbYuaQjO3CWpQc7c1wFn65KG5sxdkhrkzH2NOFuXtJqcuUtSgwz3s8g91SWdLYa7JDXIcF9lztYlrQXDXZIaZLhLUoMM91XgUoyktWa4S1KDDPeBOFuXtJ4Y7pLUIMNdkhpkuK+ASzGS1ivDXZIaZLhLUoMMd0lqkOG+TK6zS5oGhrskNchwl6QGGe49uBQjadoY7pLUIMNdkhpkuEtSgwz3JbjOLmma9Qr3JHuS3J/kWJIDS/T56ST3Jjma5M+GLVOStBznTOqQZBNwEPhxYB64M8nhqrp3rM8u4O3AK6vq0STPX62CJUmT9Zm5XwAcq6oHqupx4EZg74I+vwwcrKpHAarqkWHLlCQtR59w3wIcH2vPd8fGvRh4cZK/T3J7kj1DFXg2uc4uqRUTl2WALHKsFjnPLuAiYCvwd0leVlVf+7YTJfuB/QDbt29fdrGSpH76zNzngW1j7a3AQ4v0+XhVfbOqvgjczyjsv01VHaqq2aqanZmZOdOaJUkT9An3O4FdSXYmORfYBxxe0OdjwI8BJNnMaJnmgSELlST1NzHcq+okcBVwC3AfcFNVHU1yTZLLu263AF9Nci9wG/CrVfXV1SpaknR6fdbcqaojwJEFx64ee13AW7uvqXLqD6gPXnvZGlciScPxCVVJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUoA0Z7u4hI6l1GzLcJal1hrskNchwl6QGGe6S1CDDXZIaZLhLUoN67Qo57cZve3T3R0kbgTN3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoOafYjJ/dolbWTO3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUoF4bhyXZA7wH2ARcV1XXLtHvNcBHgB+oqrnBquzJzcIkaWTizD3JJuAgcAmwG7giye5F+j0TeBNwx9BFSpKWp8+yzAXAsap6oKoeB24E9i7S77eBdwHfGLA+SdIZ6BPuW4DjY+357ti3JDkf2FZVnzjdiZLsTzKXZO7EiRPLLlaS1E+fcM8ix+pbbyZPAt4NvG3SiarqUFXNVtXszMxM/yolScvSJ9zngW1j7a3AQ2PtZwIvA/42yYPAhcDhJLNDFSlJWp4+4X4nsCvJziTnAvuAw6ferKrHqmpzVe2oqh3A7cDla3G3jCRpZGK4V9VJ4CrgFuA+4KaqOprkmiSXr3aBkqTl63Wfe1UdAY4sOHb1En0vWnlZkqSV8AlVSWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqUK8nVNcz//UlSXoiZ+6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QG9Qr3JHuS3J/kWJIDi7z/1iT3Jrk7ya1JXjh8qZKkviaGe5JNwEHgEmA3cEWS3Qu6fQaYrarvBW4G3jV0oZKk/vrM3C8AjlXVA1X1OHAjsHe8Q1XdVlVf75q3A1uHLVOStBzn9OizBTg+1p4HXnGa/lcCf7mSoibZceCTq3l6SZp6fcI9ixyrRTsmPwfMAj+6xPv7gf0A27dv71miJGm5+izLzAPbxtpbgYcWdkpyMfAO4PKq+p/FTlRVh6pqtqpmZ2ZmzqReSVIPfcL9TmBXkp1JzgX2AYfHOyQ5H/gjRsH+yPBlSpKWY2K4V9VJ4CrgFuA+4KaqOprkmiSXd91+B3gG8JEkn01yeInTSZLOgj5r7lTVEeDIgmNXj72+eOC6JEkr4BOqktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhrUK9yT7Elyf5JjSQ4s8v5Tkvx59/4dSXYMXagkqb+J4Z5kE3AQuATYDVyRZPeCblcCj1bVdwPvBt45dKGSpP76zNwvAI5V1QNV9ThwI7B3QZ+9wAe61zcDr06S4cqUJC1Hn3DfAhwfa893xxbtU1UngceA5w1RoCRp+VJVp++QvBb4iar6pa7988AFVfXGsT5Huz7zXfsLXZ+vLjjXfmB/13wJcP8K698MfGWF55g2XnP7Ntr1gte8HC+sqplJnc7pcaJ5YNtYeyvw0BJ95pOcAzwb+I+FJ6qqQ8ChHp/ZS5K5qpod6nzTwGtu30a7XvCaV0OfZZk7gV1JdiY5F9gHHF7Q5zDw+u71a4BP1aRfCSRJq2bizL2qTia5CrgF2ARcX1VHk1wDzFXVYeD9wIeSHGM0Y9+3mkVLkk6vz7IMVXUEOLLg2NVjr78BvHbY0noZbIlninjN7dto1wte8+Am/kFVkjR93H5Akho0leE+aTuEFiTZluS2JPclOZrkzd3x5yb5mySf774/Z61rHVqSTUk+k+QTXXtnt63F57ttLs5d6xqHlOS8JDcn+Vw33j/Y+jgneUv3c31Pkg8neWpr45zk+iSPJLln7Nii45qR93aZdneSl6/086cu3Htuh9CCk8DbquqlwIXAG7rrPADcWlW7gFu7dmveDNw31n4n8O7umh9ltN1FS94D/FVVfQ/wfYyuvdlxTrIFeBMwW1UvY3Sjxj7aG+cbgD0Lji01rpcAu7qv/cD7VvrhUxfu9NsOYepV1cNV9U/d6/9i9B/8Fr59q4cPAD+1NhWujiRbgcuA67p2gFcx2tYCGrvmJM8CfoTRHWdU1eNV9TUaH2dGN3N8R/dczNOAh2lsnKvq0zzxeZ+lxnUv8MEauR04L8kLVvL50xjufbZDaEq3y+b5wB3Ad1bVwzD6HwDw/LWrbFX8PvBrwP927ecBX+u2tYD2xvtFwAngT7qlqOuSPJ2Gx7mqvgz8LvAlRqH+GHAXbY/zKUuN6+C5No3hvtiGZM3e8pPkGcBfAL9SVf+51vWspiQ/CTxSVXeNH16ka0vjfQ7wcuB9VXU+8N80tASzmG6deS+wE/gu4OmMliUWammcJxn853waw73PdghNSPJkRsH+p1X10e7wv5/6da37/sha1bcKXglcnuRBRsttr2I0kz+v+/Ud2hvveWC+qu7o2jczCvuWx/li4ItVdaKqvgl8FPgh2h7nU5Ya18FzbRrDvc92CFOvW2t+P3BfVf3e2FvjWz28Hvj42a5ttVTV26tqa1XtYDSun6qqnwVuY7StBbR3zf8GHE/yku7Qq4F7aXicGS3HXJjkad3P+alrbnacxyw1roeB13V3zVwIPHZq+eaMVdXUfQGXAv8CfAF4x1rXs0rX+MOMfi27G/hs93UpozXoW4HPd9+fu9a1rtL1XwR8onv9IuAfgWPAR4CnrHV9A1/r9wNz3Vh/DHhO6+MM/BbwOeAe4EPAU1obZ+DDjP6m8E1GM/MrlxpXRssyB7tM+2dGdxKt6PN9QlWSGjSNyzKSpAkMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGvR/oo0gGKrijJgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x224cfe9c6a0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar(df_pca.index[:100], df_pca['cumulated_ratio'][:100])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 26 features explains 0.8 of the variance.\n",
      "Top 39 features explains 0.85 of the variance.\n",
      "Top 58 features explains 0.9 of the variance.\n",
      "Top 92 features explains 0.95 of the variance.\n"
     ]
    }
   ],
   "source": [
    "for i in [0.8, 0.85, 0.9, 0.95]:\n",
    "    print('Top {} features explains {} of the variance.'.format(df_pca[df_pca['cumulated_ratio'] > i].index[0], i))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformer\n",
    "Build a transformer that apply PCA on numerical columns and convert gender to catetorical problem.\n",
    "\n",
    "***Note 1: Tried to build transformer but it doesn't work in pipeline. As the test set in GridSearchCV would regenerate a smaller n_component that is equal to the sample size of test set.***\n",
    "\n",
    "\n",
    "***Note 2: Tried to exclude gender to do PCA and run models in Model Selection section. However, the result is not better, and sometimes even worse, than the results of running PCA with gender included.***\n",
    "\n",
    "Therefore, in the following pipelines, no specific transformer is needed as gender will be processed as a numerical variable in scaler, PCA, and modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from sklearn.base import BaseEstimator, TransformerMixin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# class PCAexclGender(BaseEstimator, TransformerMixin):\n",
    "    \n",
    "#     def __init__( self, n_components=10 ):\n",
    "#         self.n_components = n_components\n",
    "    \n",
    "#     def fit(self, X, y=None):\n",
    "#         return self\n",
    "    \n",
    "#     def transform(self, X, y=None):\n",
    "#         X_withoutGender = X.drop(['gender'], axis=1)\n",
    "#         scaler=MinMaxScaler()\n",
    "#         pca=PCA(n_components=self.n_components)\n",
    "#         X_pca = pd.DataFrame(pca.fit_transform(scaler.fit_transform(X_withoutGender)))\n",
    "#         X_pca['gender'] = pd.Series(X.gender.values).astype('category')\n",
    "        \n",
    "#         return X_pca"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scaler_test = PCAexclGender()\n",
    "# scaler_test.fit(X_train)\n",
    "# scaler_test.transform(X_train)\n",
    "# #scaler_test\n",
    "# #X_scaler_test = scaler_test.transform(X_test)\n",
    "# #X_scaler_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipeline\n",
    "Use pipeline on standard scaler, PCA, and SVC, the best model so far, to see if there's any improvement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Parameters: {'SVC__max_iter': 500, 'PCA__n_components': 80, 'SVC__tol': 0.001, 'SVC__gamma': 0.01, 'SVC__C': 10}\n",
      "Accuracy on Training Set: 0.825870646766\n",
      "Accuracy on Test Set: 0.882352941176\n",
      "AUC: 0.872469635628\n",
      "Wall time: 24min 59s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "pipeline = Pipeline([('Scaler', MinMaxScaler()),\n",
    "                     ('PCA', PCA()),\n",
    "                     ('SVC', SVC(probability=True))])\n",
    "parameters = {'PCA__n_components': [80, 100, 120, 150],\n",
    "              'SVC__C': [0.01, 0.1, 1, 10, 100],\n",
    "              'SVC__gamma': [1e-2, 1e-3, 1e-4, 1e-5],\n",
    "              'SVC__tol': [1e-3, 1e-4, 1e-5],\n",
    "              'SVC__max_iter': [500, -1]\n",
    "             }\n",
    "gs_pca_svc = GridSearchCV(pipeline, param_grid=parameters, cv=5)\n",
    "gs_pca_svc.fit(X_train, y_train)\n",
    "print(\"Best Parameters:\", gs_pca_svc.best_params_)\n",
    "print(\"Accuracy on Training Set:\", gs_pca_svc.best_score_)\n",
    "\n",
    "y_pred_prob = gs_pca_svc.predict_proba(X_test)[:,1]\n",
    "print(\"Accuracy on Test Set:\", gs_pca_svc.score(X_test, y_test))\n",
    "print(\"AUC:\", roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After adding new features and applying PCA, the accuracy on test set improved. See next cell for improvements on other models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IV. Model Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply pipeline on all models to compare the best performance on accuracy and AUC with a narrow-lown list.\n",
    "\n",
    "***Note: Some models with best parameters roughly identified through plentiful times of running would have a shorter list of parameters to search.***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Parameters of SVM: {'SVC__max_iter': 500, 'PCA__n_components': 80, 'SVC__tol': 0.001, 'SVC__gamma': 0.01, 'SVC__C': 10}\n",
      "Accuracy on Training Set: 0.820895522388\n",
      "Accuracy on Test Set: 0.882352941176\n",
      "AUC: 0.882591093117\n",
      "\n",
      "\n",
      "Best Parameters of SVM-L1: {'PCA__n_components': 80, 'SVC__C': 0.1}\n",
      "Accuracy on Training Set: 0.835820895522\n",
      "Accuracy on Test Set: 0.823529411765\n",
      "\n",
      "\n",
      "Best Parameters of SVM-L2: {'PCA__n_components': 80, 'SVC__tol': 0.0001, 'SVC__C': 0.1}\n",
      "Accuracy on Training Set: 0.81592039801\n",
      "Accuracy on Test Set: 0.882352941176\n",
      "AUC: 0.904858299595\n",
      "\n",
      "\n",
      "Best Parameters of Decision Tree: {'PCA__n_components': 100, 'DT__criterion': 'gini', 'DT__min_samples_split': 3, 'DT__max_depth': 3}\n",
      "Accuracy on Training Set: 0.805970149254\n",
      "Accuracy on Test Set: 0.78431372549\n",
      "AUC: 0.716599190283\n",
      "\n",
      "\n",
      "Best Parameters of Naive Bayes: {'PCA__n_components': 80}\n",
      "Accuracy on Training Set: 0.791044776119\n",
      "Accuracy on Test Set: 0.823529411765\n",
      "AUC: 0.844129554656\n",
      "\n",
      "\n",
      "Best Parameters of Logistic Regression: {'LR__C': 0.1, 'PCA__n_components': 100, 'LR__tol': 0.001, 'LR__penalty': 'l2'}\n",
      "Accuracy on Training Set: 0.845771144279\n",
      "Accuracy on Test Set: 0.862745098039\n",
      "AUC: 0.888663967611\n",
      "\n",
      "\n",
      "Wall time: 4min 7s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "warnings.simplefilter('ignore')\n",
    "kNN = GridSearchCV(Pipeline([('Scaler', MinMaxScaler()), \n",
    "                             ('PCA', PCA()), ('kNN', KNeighborsClassifier())]),\n",
    "                   param_grid={'PCA__n_components': [80, 100],\n",
    "                               'kNN__n_neighbors': np.arange(3, 6)}, cv=5)\n",
    "\n",
    "NB = GridSearchCV(Pipeline([('scaler', MinMaxScaler()), ('PCA', PCA()), ('NB', GaussianNB())]),\n",
    "                  param_grid={'PCA__n_components': [80, 100]}, cv=5)\n",
    "\n",
    "LR = GridSearchCV(Pipeline([('Scaler', MinMaxScaler()), \n",
    "                            ('PCA', PCA()), ('LR', LogisticRegression())]),\n",
    "                  param_grid={'PCA__n_components': [80, 100],\n",
    "                              'LR__penalty': ['l1', 'l2'],\n",
    "                              'LR__C': [0.01, 0.1, 1],\n",
    "                              'LR__tol': [1e-3, 1e-4, 1e-5], \n",
    "                              #'LR__max_iter': [100, 150]\n",
    "                             }, cv=5)\n",
    "\n",
    "SVM2 = GridSearchCV(Pipeline([('Scaler', MinMaxScaler()), \n",
    "                              ('PCA', PCA()), ('SVC', SVC(kernel='linear', probability=True))]),\n",
    "                    param_grid={'PCA__n_components': [80, 100],\n",
    "                                'SVC__tol': [1e-4],\n",
    "                                'SVC__C': [0.01, 0.1, 1],\n",
    "                                #'SVC__max_iter': [500, 1000, 1500]\n",
    "                               }, cv=5)\n",
    "\n",
    "SVM1 = GridSearchCV(Pipeline([('Scaler', MinMaxScaler()), ('PCA', PCA()), \n",
    "                              ('SVC', LinearSVC(loss='l2', penalty='l1', dual=False))]),\n",
    "                    param_grid={'PCA__n_components': [80, 100],\n",
    "                                'SVC__C': [0.1, 1, 10]}, cv=5)\n",
    "\n",
    "SVM = GridSearchCV(Pipeline([('Scaler', MinMaxScaler()), ('PCA', PCA()),\n",
    "                             ('SVC', SVC(probability=True))]),\n",
    "                   param_grid={'PCA__n_components': [80, 100],\n",
    "                               'SVC__C': [1, 10, 100],\n",
    "                               'SVC__gamma': [1e-2, 1e-3],\n",
    "                               'SVC__tol': [1e-3],\n",
    "                               'SVC__max_iter': [500]\n",
    "                              }, cv=5)\n",
    "\n",
    "DT = GridSearchCV(Pipeline([('Scaler', MinMaxScaler()), \n",
    "                            ('PCA', PCA()), ('DT', DecisionTreeClassifier())]),\n",
    "                  param_grid={'PCA__n_components': [80, 100],\n",
    "                              'DT__criterion': ['gini', 'entropy'],\n",
    "                              'DT__max_depth': [3, 5, 8, None],\n",
    "                              'DT__min_samples_split': [2, 3, 5]}, cv=5)\n",
    "\n",
    "models = {'Naive Bayes': NB, 'Logistic Regression': LR, \n",
    "          'SVM-L2': SVM2, 'SVM-L1': SVM1, 'SVM': SVM, 'Decision Tree': DT}\n",
    "\n",
    "for k in models:\n",
    "    models[k].fit(X_train, y_train)\n",
    "    print(\"Best Parameters of {}:\".format(k), models[k].best_params_)\n",
    "    print(\"Accuracy on Training Set:\", models[k].best_score_)\n",
    "    print(\"Accuracy on Test Set:\", models[k].score(X_test, y_test))\n",
    "    if k not in ['SVM-L1']:\n",
    "        print(\"AUC:\", roc_auc_score(y_test, models[k].predict_proba(X_test)[:,1]))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%time\n",
    "# LR = GridSearchCV(Pipeline([('Scaler', MinMaxScaler()), \n",
    "#                             ('PCA', PCA()), ('LR', LogisticRegression())]),\n",
    "#                   param_grid={'PCA__n_components': [80, 100, 120],\n",
    "#                               'LR__penalty': ['l1', 'l2'],\n",
    "#                               'LR__C': np.linspace(0.05, 0.25, 15),\n",
    "#                               'LR__tol': [1/80, 0.01, 0.05, 1e-3, 1e-4, 1e-5, 1e-6], \n",
    "#                               #'LR__max_iter': [100, 150]\n",
    "#                              }, cv=5)\n",
    "# LR.fit(X_train, y_train)\n",
    "# print(\"Best Parameters:\", LR.best_params_)\n",
    "# print(\"Accuracy on Training Set:\", LR.best_score_)\n",
    "\n",
    "# y_pred_prob = LR.predict_proba(X_test)[:,1]\n",
    "# print(\"Accuracy on Test Set:\", LR.score(X_test, y_test))\n",
    "# print(\"AUC:\", roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# np.linspace(0.05, 0.25, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on iterative parameter tuning, the three best models that have potential to generate high accuracy and AUC are linear SVM with L2 penalty, SVM with kernel, and logistic regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curve\n",
    "Here only the top three best models are focused."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XlclNX+wPHPCTAtvGpCprnnCogbKuBKJmiWW3rVzKXcyFRu5pqGy81yLbVwS00rr1Jm2sKvzMosr6agRIIi5IqgAimKpgKe3x8jcxFBRp1hmJnv+/Xi9WJmzpzn+zDMd86c53m+R2mtEUIIYV8esHYAQgghzE+SuxBC2CFJ7kIIYYckuQshhB2S5C6EEHZIkrsQQtghSe5CCGGHJLkLIYQdkuQuhBB2yNlaG3Zzc9M1a9a01uaFEMImRUVFpWmt3YtqZ7XkXrNmTSIjI621eSGEsElKqROmtJNpGSGEsEOS3IUQwg5JchdCCDskyV0IIeyQJHchhLBDRSZ3pdQapdQ5pdTBQh5XSqklSqlEpVSMUqqZ+cMUQghxN0wZua8FOt/h8S5A3Zs/I4Bl9x+WEEKI+1Hkee5a651KqZp3aNId+Egb1uvbo5Qqr5SqrLVOMVOMwt4lH4CzcdaO4p6cOv83KRl/W6z/4zmpJOX8ZbH+RfG6cSOHrOtZVHWvw8CnJ1t0W+aYc38cOJXndtLN+26jlBqhlIpUSkWmpqaaYdPCLpyNg8yz1o7inqRk/M2lq1kW6z8p5y8uast9eIjik3HhAlFRUcTFxaGx/NrV5rhCVRVwX4GRa61XAisBfHx8ZGVu8T+ulaDpAGtHcddO5BjGNR19qlmk/+TELQD0qNPDIv0Ly7tw4QITJkxg1ar11KlTh1WrVtG+fXuLb9ccyT0JyPufXRVINkO/Qghh03JycvD39yc+Pp6JEycyY8YMypQpUyzbNkdy/xIYrZTaCLQCMmS+XQjhyNLT03nkkUdwcnJi9uzZVKtWDR8fn2KNocjkrpTaAHQA3JRSScB0wAVAa70ciACeBhKBK8CLlgpWWJEFD3rGpseSoHIg8WGL9J/0l+UOel66lk3ZB51xSXzEIv2n/Z2GWxk3i/QtzE9rzfr16wkJCWHOnDkMHz6cnj17WiUWU86W6V/E4xp4xWwRiZIp96CnayWzd52gckgr/TCWSmEpGX8bk7C5lX3QmcrlLPc1262MG3Ur1LVY/8J8Tp06RXBwMBEREfj6+tK6dWurxmO1kr/CBlnqoGeiIbFb6qBh1gXDQc8+FjroKcSGDRsYOXIkOTk5LFq0iNGjR+Pk5GTVmCS5CyHEfapQoQKtWrVi5cqV1KpVy9rhAJLchRDirmVnZ/Puu+9y/fp1pk6dSufOnQkKCkKpgs4Mtw4pHCaEEHfh999/x9fXl4kTJxITE4PhsCMlKrGDjNztiwXPaNl58gCRV6+SnnbN7H1nZp/H1bmCcW7c3FIzr+Hu+qBF+haO49q1a7z55pvMmTOHRx55hM8++4znnnuuxCX1XDJytycWvIw/8upVTqjSFunb1bkCj5auaZG+AdxdH6TBY/+wWP/CMSQkJDB37lyef/554uLi6N27d4lN7CAjd/tjoTNa0tOu4QrM7vSS2fsWoqTKzMxk69atDBgwAC8vLw4fPkzt2rWtHZZJZOQuhBAF+P7772nUqBEDBw7k0KFDADaT2EGSuxBC3OL8+fMMHTqUwMBASpUqxc8//0zDhg2tHdZdk2kZIYS4KScnh9atW3PkyBGmTJlCaGgopUtb5liTpUlyF0I4vLS0NGOhr7feeovq1avTrJltrxgq0zJCCIelteajjz6iXr16rFq1CoAePXrYfGIHSe5CCAd14sQJunTpwuDBg2nYsCHt2rWzdkhmJcldCOFwPvnkE7y8vPj111957733+OWXX2jQoIG1wzIrmXMXQjgcd3d3WrduzYoVK6hRo4a1w7EISe5CCLuXlZXFwoULycrK4o033iAoKIjAwMASfYXp/ZJpGSGEXTtw4ACtWrViypQpxMXFldhCX+YmyV0IYZeuXr3K66+/TosWLUhOTubzzz9nw4YNdp/Uc0lyF0LYpcTERBYsWMCgQYM4dOgQvXr1snZIxUrm3O1I7JUzJFw9Z5GFpnPL8gpRkmVmZvLFF18wcOBAvLy8iI+PLzErIxU3GbnbkYSr50jLvmyRvi1dlleI+/Xdd9/h6enJ4MGDjYW+HDWxg4zc7Y6b88MWWWjaUgtpCHG/0tPTGTduHB999BENGjTgl19+sclCX+YmyV0IYbNyC30lJiYydepUpk2bZrOFvsxNkrsQwuakpqZSsWJFnJycmDt3LjVq1KBJkybWDqtEkeRejBKif+H8sWiL9X/qylmynF35LNL8UyiyDqkoCbTWrF27lnHjxjFnzhxGjhxJ9+7drR1WiSQHVIvR+WPR5Fw8Y7H+s5xd+bt0JYv0LeuQCms7fvw4QUFBvPTSSzRq1IiAgABrh1Siyci9mDn94zFa9hxjkb6TE7cA0KNONYv0L4S1fPzxx7z88ssopVi6dCkjR47kgQdkbHonktyFECVepUqVaNeuHcuXL6d69erWDscmSHIXQpQ4WVlZzJs3j5ycHEJDQwkMDCQwMNDaYdkU+V4jhChR9u/fT4sWLZg2bRrx8fHGQl/i7pg0cldKdQYWA07AKq31nHyPVwfWAeVvtpmstY4wc6w273hOKkk5fxnnxs0t7e803Mq4WaRvISzt77//ZubMmSxYsAB3d3e++OILevQw/wV5jqLIkbtSygkIA7oAHkB/pZRHvmbTgE+11k2BfsBScwdqD5Jy/uKi/tti/buVcaNuhboW618ISzp69CjvvPMOQ4YMIS4uThL7fTJl5N4SSNRaHwVQSm0EugNxedpoIPc8uXJAsjmDtCf/UGUsUh5ACFt08eJFNm/ezJAhQ/D09CQhIcFuV0YqbqbMuT8O5L0qJunmfXnNAF5QSiUBEYBlzvUTQtiNiIgIvLy8GDp0qLHQlyR28zEluRdU2T7/EY7+wFqtdVXgaeBjpdRtfSulRiilIpVSkampqXcfrRDC5qWlpTFw4EC6du1K2bJl2bVrlxT6sgBTknsSkPeqmKrcPu0yFPgUQGu9GygN3HZkT2u9Umvto7X2cXd3v7eIhRA2K7fQ18aNGwkNDWX//v34+vpaOyy7ZMqc+z6grlKqFnAawwHT5/O1OQl0BNYqpRpiSO4yNBdCAHD27Fnc3d1xcnJiwYIF1KhRA29vb2uHZdeKHLlrrbOB0cB3wCEMZ8XEKqVmKaW63Wz2GjBcKfU7sAEYouXkVCEcntaa1atXU79+fVauXAnAs88+K4m9GJh0nvvNc9Yj8t0Xmuf3OKC1eUMTQtiyo0ePMnz4cH788Ufat2/PU089Ze2QHIpcoSqEMLt169bRqFEj9u3bx/Lly/nxxx+pU6eOtcNyKFJbRghhdlWqVOHJJ59k2bJlVK1a1drhOCRJ7kKI+3b9+nXmzJnDjRs3mDFjBp06daJTp07WDsuhybSMEOK+7Nu3j+bNmzN9+nSOHj0qhb5KCEnuQoh7cuXKFcaPH4+vry/nz5/nyy+/5KOPPkKpgq57FMVNkrsQ4p4cO3aM9957j+HDhxMbG8uzzz5r7ZBEHjLnLoQwWUZGBps3b+bFF1/E09OTxMREqlWTZR1LIhm5CyFM8s033+Dp6cmwYcM4fPgwgCT2EkySuxDijlJTUxkwYADPPPMMFSpUYPfu3TRo0MDaYYkiyLSMEKJQOTk5tGnThmPHjjFz5kwmT55MqVKlrB2WMIEkdyHEbc6cOcOjjz6Kk5MTCxcupGbNmnh5eVk7LHEXZFpGCGF048YNVqxYQb169VixYgUAzzzzjCR2GyTJXQgBQGJiIh07diQ4OJgWLVoQFBRk7ZDEfZDkLoTgww8/pFGjRuzfv58PPviA7du3U7t2bWuHJe6DzLkLIahevTpBQUGEhYXx+OP5l0gWtkiSuxAO6Nq1a7z99tvcuHGDWbNm0bFjRzp27GjtsIQZybSMEA7mt99+o3nz5sycOZOTJ09KoS87JcldCAdx+fJlxo0bh5+fHxkZGXz99desXbtWCn3ZKUnuQjiIEydOsHTpUoKDg4mNjaVr167WDklYkMy5C2HHLly4wKZNmxg2bBgeHh4kJibKykgOQkbuQtiprVu34uHhQXBwsLHQlyR2xyHJXQg7c+7cOfr160ePHj1wd3dnz549UujLAcm0TD5/JGVw+MxFi/R9NSuH0i5OFulbCDAU+mrdujUnT57kzTffZOLEibi4uFg7LGEFktzzOXzmIqmZ13B3fdDsfZd2caL8Q1JRT5hfcnIyjz32GE5OTixevJiaNWvi4eFh7bCEFUlyL4C764P08TH/IgRbMh42e5/CseUW+po0aRJz5sxh1KhRPP3009YOS5QAMucuhI06cuQIAQEBjBo1ilatWtGlSxdrhyRKEEnuQtig1atX07hxY2JiYlizZg3btm2jVq1a1g5LlCAyLSOEDapZsyZdunQhLCyMypUrWzscUQJJchfCBly7do1///vfALz55ptS6EsUSaZlhCjh/vvf/9KkSRNmz55NSkqKFPoSJpHkLkQJlZmZSUhICG3atOHKlSt8++23rF69Wgp9CZOYlNyVUp2VUvFKqUSl1ORC2vxTKRWnlIpVSv3HvGEK4XhOnjzJihUreOWVVzh48KAseyfuSpFz7kopJyAM6AQkAfuUUl9qrePytKkLTAFaa63PK6UetVTAQtiz8+fP89lnnzFixAg8PDw4evQoVapUsXZYwgaZMnJvCSRqrY9qra8DG4Hu+doMB8K01ucBtNbnzBumEPbviy++wMPDg1GjRhEfHw8giV3cM1OS++PAqTy3k27el1c9oJ5SapdSao9SqnNBHSmlRiilIpVSkampqfcWsRB25syZM/Tp04devXrx2GOPsXfvXurXr2/tsISNM+VUyIKO3uQ/XO8M1AU6AFWBX5RSXlrrC7c8SeuVwEoAHx8fOeQvHF5OTg5t27bl1KlTvPXWW4wfP14KfQmzMCW5JwF5C61UBZILaLNHa50FHFNKxWNI9vvMEqUQdiYpKYkqVarg5OTEkiVLqFWrlpTlFWZlSnLfB9RVStUCTgP9gOfztdkC9AfWKqXcMEzTHDVnoMUl+e8Ezl09jkviI2bvOy37Mm7OUjzMkd24cYOwsDCmTJnC3LlzeeWVV6QmjLCIIufctdbZwGjgO+AQ8KnWOlYpNUsp1e1ms++AdKVUHPATMEFrnW6poC3p3NXjZGaft0jfbs4PU7e0nEjkqA4fPky7du0YO3Ysbdq04ZlnnrF2SMKOmVR+QGsdAUTkuy80z+8aGHfzx+a5OlegR50e5u/40mXz9ylswqpVqxg9ejQPPfQQ69atY+DAgXIxkrAoqS0jRDF44oknePbZZ3n//fepVKmStcMRDkCSuxAWcPXqVWbNmgXAW2+9RUBAAAEBAVaOSjgSqS0jhJnt2rWLJk2a8Pbbb5OamiqFvoRVyMg9n4eunKbM1bNwwPxrqJJ5FlzlK7m9unTpEq+//jphYWHUqFGD7777jsDAQGuHJRyUjNzzKXP1LC7ZmZbp3LUSVJJFi+1VUlISq1atYsyYMfzxxx+S2IVVyci9AFnOrtB0gLXDEDYgPT2dTz/9lJdffpmGDRty9OhRWRlJlAgychfiHmit2bRpEx4eHowdO9ZY6EsSuygpJLkLcZdSUlJ47rnn6NOnD9WqVSMyMlIKfYkSR6ZlhLgLuYW+Tp8+zbx583j11Vdxdpa3kSh55L9SCBOcOnWKxx9/HCcnJ8LCwqhVqxb16tWzdlhCFEqmZYS4g5ycHJYsWUKDBg1YtmwZAEFBQZLYRYknI3chCnHo0CGGDh3K7t276dKlC88++6y1QxLCZDJyF6IAK1eupEmTJhw5coSPP/6Yb775hurVq1s7LCFMJiN3IQpQt25devbsyZIlS3j0USnTLGyPJHchgL///psZM2aglGLOnDlS6EvYPJmWEQ5v586dNG7cmHnz5pGRkSGFvoRdkOQuHNbFixcZNWoU7du3Jycnhx9++IFly5bJIhrCLkhyFw4rOTmZtWvXMm7cOGJiYnjyySetHZIQZiNz7sKhpKWl8emnnzJq1CgaNGjAsWPHZGUkYZdk5C4cgtaa8PBwPDw8+Ne//sWRI0cAJLELuyXJXdi95ORkevToQb9+/ahRowZRUVFyhamwezItI+xaTk4O7dq14/Tp0yxYsICQkBAp9CUcgvyXC7t04sQJqlatipOTE0uXLqV27drUqVPH2mEJUWxsL7knH4CzcRbr3iU707ASk7BJOTk5LF68mGnTpjFv3jxGjx4ty90Jh2R7c+5n4wwLTVtIlrMrf5eWg2y26ODBg/j7+/Paa6/RsWNHevToYe2QhLAa2xu5g2GhaQutcZqeds0i/QrLWr58OWPHjqVcuXL85z//oV+/fnIxknBotjdyFyKP3FIBDRs2pE+fPsTFxdG/f39J7MLh2ebIXTi8K1euEBoaipOTE3PnzqV9+/a0b9/e2mEJUWLIyF3YnB07duDt7c3ChQvJzMyUQl9CFECSu7AZGRkZjBw50liK98cffyQsLEymYIQogEnJXSnVWSkVr5RKVEpNvkO73koprZTyMV+IQhikpKTwySefMH78eGJiYqTeuhB3UOScu1LKCQgDOgFJwD6l1Jda67h87coCY4HfLBGocEypqals3LiRMWPG0KBBA44fP467u7u1wxKixDNl5N4SSNRaH9VaXwc2At0LaPdvYB5w1YzxCQelteY///kPDRs25LXXXjMW+pLELoRpTEnujwOn8txOunmfkVKqKVBNa/21GWMTDurUqVM8++yzDBgwgDp16nDgwAEp9CXEXTLlVMiCjlYZT09QSj0AvAsMKbIjpUYAIwBZSV4UKDs7mw4dOnDmzBneffddxowZg5OTk7XDEsLmmJLck4BqeW5XBZLz3C4LeAE7bp618BjwpVKqm9Y6Mm9HWuuVwEoAHx8fOX9NGB0/fpxq1arh7OzMihUrqF27NrVr17Z2WELYLFOmZfYBdZVStZRSpYB+wJe5D2qtM7TWblrrmlrrmsAe4LbELkRBsrOzWbBgAQ0bNmTp0qUAPPXUU5LYhbhPRY7ctdbZSqnRwHeAE7BGax2rlJoFRGqtv7xzD0IULCYmhqFDhxIZGUn37t157rnnrB2SEHbDpPIDWusIICLffaGFtO1w/2EJe7d06VJCQkKoUKEC4eHh9OnTRy5GEsKM5ApVUaxySwV4eXnRr18/4uLi+Oc//ymJXQgzk8JholhcvnyZadOm4ezszPz582nXrh3t2rWzdlhC2C0ZuQuL++GHH2jUqBGLFi3i2rVrUuhLiGIgyV1YzIULFxg2bBhPPfUUzs7O7Ny5kyVLlsgUjBDFQJK7sJizZ8+yceNGJk2axO+//07btm2tHZIQDsPm5tx/PHuc6IwUiy2Hd/ZKKpUekvol9yo3oYeEhFC/fn2OHz+Om5ubtcMSwuHY3Mg9OiOF1KxMi/Vf6SF3Wj7uabH+7ZXWmk8++QQPDw8mTpxIQkICgCR2IazE5kbuAO4urozr9JK1wxA3nTx5kuDgYP7v//4PPz8/Vq9eTd26da0dlhAOzSaTuyg5cgt9nTt3jiVLljBq1Cgp9CVECSDJXdyTo0ePUqNGDZydnfnggw944oknqFmzprXDEkLcZHNz7sK6srOzmTt3Lh4eHoSFhQHQsWNHSexClDAychcmi46OZujQoezfv5+ePXvSp08fa4ckhCiEjNyFSd5//31atGjB6dOn2bRpE5s3b6Zy5crWDksIUQhJ7uKOcksFeHt7M2DAAOLi4qQ0rxA2QKZlRIEyMzOZOnUqLi4uLFiwQAp9CWFjZOQubrNt2za8vLx47733yMrKkkJfQtggSe7C6Pz587z44osEBQVRunRpdu7cyeLFi6XQlxA2SJK7MDp37hybNm1iypQpREdH06ZNG2uHJIS4RzLn7uDOnDnDhg0bePXVV42FvipWrGjtsIQQ90lG7g5Ka826devw8PBgypQpxkJfktiFsA+S3B3Q8ePH6dy5M0OGDMHDw4Po6Ggp9CWEnZFpGQeTnZ1NQEAAaWlphIWFERwczAMPyGe8EPZGkruDSExMpFatWjg7O7NmzRpq165NjRo1rB2WEMJCZMhm57Kysnjrrbfw9PQ0FvoKCAiQxC6EnZORux3bv38/Q4cOJTo6mj59+tC3b19rhySEKCYycrdTS5YsoWXLlpw5c4bNmzfz6aefUqlSJWuHJYQoJpLc7UxuqYCmTZsyaNAg4uLi6Nmzp5WjEkIUN5mWsROXLl1iypQpPPjggyxcuJC2bdvStm1ba4clhLASGbnbgW+//RYvLy+WLl2K1loKfQkhJLnbsvT0dAYPHkyXLl14+OGH2bVrF++8844U+hJCSHK3Zenp6XzxxRe88cYbHDhwAD8/P2uHJIQoIUyac1dKdQYWA07AKq31nHyPjwOGAdlAKvCS1vqEmWMVQEpKCuvXr+e1116jXr16nDhxggoVKlg7rPuWlZVFUlISV69etXYoQpQIpUuXpmrVqri4uNzT84tM7kopJyAM6AQkAfuUUl9qrePyNDsA+GitryilXgbmAXJStRlprfnwww8ZN24c165do3v37tStW9cuEjtAUlISZcuWpWbNmjKtJBye1pr09HSSkpKoVavWPfVhyrRMSyBRa31Ua30d2Ah0zxfIT1rrKzdv7gGq3lM0okDHjh0jMDCQoUOH0rhxY37//Xe7K/R19epVKlasKIldCEApRcWKFe/rm6wp0zKPA6fy3E4CWt2h/VDg/wp6QCk1AhgBUL16dRNDdGzZ2dk8+eSTpKens2zZMkaMGGG3hb4ksQvxP/f7fjAlSxS0hQLPtVNKvQD4APMLelxrvVJr7aO19nF3dzc9SgeUkJBATk4Ozs7OfPjhh8TGxkoFRwtzdXW97z6Sk5Pp3bt3oY9fuHCBpUuXmtw+vyFDhlCrVi2aNGlC48aN+eGHH+4rXnNbvnw5H330kbXDEJiW3JOAanluVwWS8zdSSj0FTAW6aa2vmSc8x5OVlcWbb76Jl5cX77//PgAdOnSgWrVqRTxTlARVqlRh06ZNhT6eP7kX1b4g8+fPJzo6mkWLFhEcHHzPseaVnZ1tln6Cg4MZNGiQWfoS98eU5L4PqKuUqqWUKgX0A77M20Ap1RRYgSGxnzN/mI4hMjISHx8f3njjDXr16kX//v2tHZLDO3HiBB07dsTb25uOHTty8uRJAP788098fX1p0aIFoaGhxlH/8ePH8fLyAiA2NpaWLVvSpEkTvL29SUhIYPLkyfz55580adKECRMm3NI+JyeH8ePH06hRI7y9vXnvvffuGJufnx+nT5823o6KiqJ9+/Y0b96coKAgUlJSANi3bx/e3t74+fkxYcIE4/bWrl1Lnz59ePbZZwkMDAQMHxwtWrTA29ub6dOnA3D58mW6du1K48aN8fLyIjw8HIDJkyfj4eGBt7c348ePB2DGjBksWLAAgOjoaHx9ffH29qZnz56cP38eMAxWJk2aRMuWLalXrx6//PLL/bxEohBFzrlrrbOVUqOB7zCcCrlGax2rlJoFRGqtv8QwDeMKfHZznuik1rqbBeO2O4sXL2bcuHE89thjbN26lW7dHPfPtyP+HKmXzPvlz73sg3So/+hdP2/06NEMGjSIwYMHs2bNGsaOHcuWLVsICQkhJCSE/v37s3z58gKfu3z5ckJCQhgwYADXr18nJyeHOXPmcPDgQaKjowHDh0GulStXcuzYMQ4cOICzszN//fXXHWP79ttv6dGjB2D4xjdmzBi2bt2Ku7s74eHhTJ06lTVr1vDiiy+ycuVK/P39mTx58i197N69m5iYGB555BG2bdtGQkICe/fuRWtNt27d2LlzJ6mpqVSpUoVvvvkGgIyMDP766y+++OILDh8+jFKKCxcu3BbfoEGDeO+992jfvj2hoaHMnDmTRYsWAYZvCnv37iUiIoKZM2eyfft2014QYTKTznPXWkcAEfnuC83z+1NmjsthaK1RSuHj48PQoUOZN28e5cuXt3ZY4qbdu3ezefNmAAYOHMjEiRON92/ZsgWA559/3jhyzcvPz4/Zs2eTlJREr169ijzDafv27QQHB+PsbHhbPvLIIwW2mzBhAhMnTuTcuXPs2bMHgPj4eA4ePEinTp0Aw7eAypUrc+HCBS5duoS/v78x1q+//trYV6dOnYzb2bZtG9u2baNp06YAZGZmkpCQQNu2bRk/fjyTJk3imWeeoW3btmRnZ1O6dGmGDRtG165deeaZZ26JMSMjgwsXLtC+fXsABg8eTJ8+fYyP9+rVC4DmzZvf8gEnzEcKh1nJxYsXmTRpEqVLl+bdd9+ldevWtG7d2tphlQj3MsIuLndzBsPzzz9Pq1at+OabbwgKCmLVqlXUrl270Pa5H/RFmT9/Pr169WLJkiUMHjyYqKgotNZ4enqye/fuW9rmToUU5uGHH75l+1OmTGHkyJG3tYuKiiIiIoIpU6YQGBhIaGgoe/fu5YcffmDjxo28//77/Pjjj0XGnuvBBx8EwMnJyWzz/eJWcuqFFURERODp6cnKlStxdnaWQl8lmL+/Pxs3bgRg/fr1tGnTBgBfX18+//xzAOPj+R09epTatWszduxYunXrRkxMDGXLluXSpUsFtg8MDGT58uXGZHenaZkHHniAkJAQbty4wXfffUf9+vVJTU01JvesrCxiY2OpUKECZcuWNY7wC4sVICgoiDVr1pCZmQnA6dOnOXfuHMnJyTz00EO88MILjB8/nv3795OZmUlGRgZPP/00ixYtMk4z5SpXrhwVKlQwzqd//PHHxlG8KB4yci9GaWlp/Otf/2L9+vV4enqyadMmWrW60yUDojhduXKFqlX/d/3duHHjWLJkCS+99BLz58/H3d2dDz/8EIBFixbxwgsvsHDhQrp27Uq5cuVu6y88PJxPPvkEFxcXHnvsMUJDQ3nkkUdo3bo1Xl5edOnShVdeecXYftiwYRw5cgRvb29cXFwYPnw4o0ePLjTZUlnDAAATnklEQVRepRTTpk1j3rx5BAUFsWnTJsaOHUtGRgbZ2dn861//wtPTk9WrVzN8+HAefvhhOnToUGCsYPhwOXTokLFGkaurK5988gmJiYlMmDCBBx54ABcXF5YtW8alS5fo3r07V69eRWvNu+++e1t/69atIzg4mCtXrlC7dm3j304UD2WtUaOPj4+OjIy86+e9E274Zx/X931zh2RxCQkJ+Pj48Oqrr/L6669TqlQpa4dUYhw6dIiGDRtaOwyTXblyhTJlyqCUYuPGjWzYsIGtW7daO6wCZWZmGs/mmTNnDikpKSxevNjKUQlTFPS+UEpFaa19inqujNwt7PTp06xfv54JEyZQt25dTpw4IQdM7UBUVBSjR49Ga0358uVZs2aNtUMq1DfffMPbb79NdnY2NWrUYO3atdYOSRQDSe4WorVm1apVjB8/nqysLHr16kWdOnUksduJtm3b8vvvv1s7DJP07dtXFkd3QHJA1QL+/PNPOnbsyIgRI2jWrBkxMTHUqVPH2mEJIRyIjNzNLDs7m44dO/LXX3+xYsUKhg0bJvVghBDFTpK7mcTHx/PEE0/g7OzMunXreOKJJ24580IIIYqTDCnv0/Xr15k5cyaNGjUiLCwMgPbt20tiF0JYlST3+7B3716aN2/OjBkz6NOnDwMGDLB2SOI+zJ49G09PT7y9vWnSpAm//fab1WJZtGgRV65cue3+GTNmMGXKlFvui46OvqfTSKOjo4mIiCi64R0cP34cpRRvvPGG8b60tDRcXFzueI7+nRRWerk4ygnv2LHjtlIKYLgGIS4uroBnlFyS3O/RokWL8PPz4/z583z11VesX78eNzc3a4cl7tHu3bv5+uuv2b9/PzExMWzfvt1qZZZzcnIKTe79+/c3VmXMtXHjRp5//vm73s69JPeCSgXUrl37lno1n332GZ6enncdT1GsWU541apVeHh4WKx/S5RgkOR+l3Iv+mrZsiXDhw8nNja2wE96YVtSUlJwc3Mz1jxxc3OjSpUqANSsWZO0tDTAUJa5Q4cOgGEUPXDgQJ588knq1q3LBx98ABhGf+3ataNnz554eHgQHBzMjRs3ANiwYQONGjXCy8uLSZMmGbfv6upKaGgorVq1Yvbs2SQnJxMQEEBAQMAtcdavX5/y5cvf8q3i008/pV+/foCh+Jefnx/NmjWjT58+xlIC+/btw9/fn8aNG9OyZUsyMjIIDQ0lPDycJk2aEB4ezl9//UWPHj3w9vbG19eXmJgY436OGDGCwMDAApNrmTJlaNiwIbkXJYaHh/PPf/7T+PhXX31Fq1ataNq0KU899RRnz54FDBdXvfjii8YSx7nlHACmTp1K48aN8fX1NbbPW064sLLBOTk5TJgwwVi2eMWKFUW+9qbo0KGDcf9cXV0LjC81NZXnnnuOFi1a0KJFC3bt2gUYvuH7+/vTtGlT/P39iY+PBwouuWxOckDVRBkZGUycOJEyZcqwaNEi/P39jZX2hJklbIfMs+bt07US1C28eGlgYCCzZs2iXr16PPXUU/Tt29ekWigxMTHs2bOHy5cv07RpU7p27QoY3tBxcXHUqFGDzp07s3nzZvz9/Zk0aRJRUVFUqFCBwMBAtmzZQo8ePbh8+TJeXl7MmjULgDVr1vDTTz8V+G2wf//+bNy4kVatWrFnzx4qVqxI3bp1SUtL480332T79u08/PDDzJ07l3feeYfJkyfTt29fwsPDadGiBRcvXuShhx5i1qxZREZGGheFGTNmDE2bNmXLli38+OOPDBo0yFgzJioqil9//ZUyZcoU+Hfo168fGzdu5LHHHsPJyYkqVaqQnGxY06dNmzbs2bMHpRSrVq1i3rx5LFy4kH//+9+UK1eOP/74A/hfkbPLly/j6+vL7NmzmThxIh988AHTpk27bZsFlQ1evXo15cqVY9++fVy7do3WrVsTGBh4z4tMF6Sw+EJCQnj11Vdp06YNJ0+eJCgoiEOHDtGgQQN27tyJs7Mz27dv5/XXXzd+kOUtuWxuktxN8NVXXxEcHMyZM2cYP368ydX7hO1wdXUlKiqKX375hZ9++om+ffsyZ84chgwZcsfnde/enTJlylCmTBkCAgLYu3cv5cuXp2XLlsYKkP379+fXX3/FxcWFDh06kLvE5IABA9i5cyc9evTAycmJ5557zqRY+/Xrh7+/PwsXLmTjxo3GRV327NlDXFycsbro9evX8fPzIz4+nsqVK9OiRQsA/vGPfxTY76+//mpMOrnr9mZkZADQrVu3QhM7QOfOnXnjjTeoVKnSbRdMJSUl0bdvX1JSUrh+/box0W7fvv2WQmYVKlQAoFSpUsZvw82bN+f7778vcJsFlQ3etm0bMTExxtWtMjIySEhIMGtyLyy+7du33zIvf/HiRS5dukRGRgaDBw8mISEBpRRZWVnGNnlLLpubJPc7SE1NJSQkxPhVesuWLcY3iLCgO4ywLcnJyYkOHTrQoUMHGjVqxLp16xgyZAjOzs7GaZX8q9Hn/5DPvV3Q/Xeq41S6dGmcnJxMirNatWrUrFmTn3/+mc8//9xYCVJrTadOndiwYcMt7WNiYkwajBQUX+7z8pYGLkipUqVo3rw5CxcuJDY2lq+++sr42JgxYxg3bhzdunVjx44dzJgxw7i9guJycXEx3n+nksAFlQ3WWvPee+8RFBRUaKxhYWHGKbSIiAjj9JupCovvxo0b7N69+7YPwTFjxhAQEMAXX3zB8ePHjdN6UPTf9X7InPsdZGRkGL/yRUZGSmK3Y/Hx8SQkJBhvR0dHU6NGDcAw5x4VFQVwy7wwwNatW7l69Srp6ens2LHD+D+yd+9ejh07xo0bNwgPD6dNmza0atWKn3/+mbS0NHJyctiwYUOhUz93Kg0Mhm8Dr7766i3XU/j6+rJr1y4SExMBQ3GzI0eO0KBBA5KTk9m3bx8Aly5dIjs7+7ZttGvXjvXr1wOG4wZubm6FjvIL8tprrzF37lwqVqx4y/0ZGRk8/vjjgKFSZK7AwEDjlBAUXXveFEFBQSxbtsw4Oj5y5AiXL1++pc0rr7xCdHQ00dHRd53Y7yT//uROaeXd/+Ks6yPJPZ9Tp07x9ttvo7WmTp06nDhxgtDQUKngaOcyMzMZPHiwcU3QuLg44whz+vTphISE0LZt29tG1y1btqRr1674+vryxhtvGJOFn58fkydPxsvLi1q1atGzZ08qV67M22+/TUBAAI0bN6ZZs2Z07969wHhGjBhBly5dbjugmqtPnz7ExsYaD6QCuLu7s3btWvr37288KHr48GFKlSpFeHg4Y8aMoXHjxnTq1ImrV68SEBBAXFyc8YDqjBkziIyMxNvbm8mTJ9+SiE3h6enJ4MGDb7s/91Thtm3b3nIMYdq0aZw/fx4vLy8aN27MTz/9dFfbK8iwYcPw8PCgWbNmeHl5MXLkyLs+E+WHH36gatWqxp/8C6AUZsmSJca/n4eHh3H5xYkTJzJlyhRat25NTk7OXe/TvZKSvzfduHGDlStXMnHiRHJycvj999+lHkwxsrWSv2BIWq6urrctsbdjxw4WLFhwy+mBQtyL+yn5KyN3DHXWn3zySV5++WVatmzJH3/8IYldCGHTHP6AanZ2Np06deLChQusXr2aF198Uc6EESbJnbbJL/egrBDW5LDJ/dChQ9StWxdnZ2c+/vhjnnjiCbMeXBFCCGtyuGmZa9euMX36dLy9vY1Httu2bSuJXQhhVxxq5L5nzx6GDh1KXFwcAwcOZODAgdYOSQghLMJhRu4LFy7E39+fS5cuERERwUcffXTb+bhCCGEv7D65515Z6OfnR3BwMAcPHqRLly5WjkqURFLy9+7Ze8nfadOmERQUxLVr1yy63cL2+X7YbXK/cOECQ4cOJSQkBAB/f3+WLl16V1fcCcchJX9N40glf2fPns2uXbvYsmWLsdRBUSxRuvde2WVy37JlCx4eHqxbt46yZcvesaaHECAlf6Xk760WLlxIREQEX331lbFWTFRUFO3bt6d58+YEBQWRkpJijOX111+nffv2LF68mCFDhjB27Fj8/f2pXbu2sYgZwPz5842xTZ8+/Z5iM5VdHVA9d+4co0eP5rPPPqNJkyZ8/fXXNGvWzNphibv06+lfSfs7zax9upVxo83jbQp9XEr+SsnfXLt27SI+Pp6oqCjjdElWVhZjxoxh69atuLu7Ex4eztSpU1mzZg1gmCn4+eefARgyZAgpKSn8+uuvHD58mG7dutG7d2+2bdtGQkICe/fuRWtNt27d2LlzJ+3atTM5trthc8ndvWzha5NevHiR77//ntmzZzNhwgRcXFyKMTJhy6Tkr5T8zVWnTh3Onz/Ptm3b6N27N2AoLHfw4EE6deoEGL4hVK5c2fic/Pvco0cPHnjgATw8PIzfPLZt28a2bdto2rQpYPjmkpCQYN3krpTqDCwGnIBVWus5+R5/EPgIaA6kA3211sfNG6rBwKcn33L75MmTfPzxx7z++uvUqVOHkydPUrZsWUtsWhSTO42wLUlK/krJX4BKlSqxfv16OnbsSMWKFQkICEBrjaenZ6FFxPL/ffLO0ef+XbXWTJkyhZEjRxYamzkVOeeulHICwoAugAfQXymVfzHBocB5rXUd4F1grrkDze/GjRssXboUT09P3nrrLf78808ASezinkjJXyn5m1e9evXYvHkzL7zwAtHR0dSvX5/U1FRjcs/KyiI2NvauY1uzZo3xOMjp06c5d+7c3e6iyUw5oNoSSNRaH9VaXwc2AvnrlHYHcl+1TUBHZcECLfHx8XTo0IFXXnkFPz8/YmNjpdCXuC9S8ldK/ubXokULPvzwQ7p168apU6fYtGkTkyZNonHjxjRp0oT//ve/d9VfYGAgzz//PH5+fjRq1IjevXvf8QP8vmmt7/gD9MYwFZN7eyDwfr42B4GqeW7/Cbjdqd/mzZvre5GVlaVr1Kihy5cvrz/88EN948aNe+pHlCxxcXHWDuGuTZ8+Xc+fP/+2+3/66SfdtWtXK0Qk7E1B7wsgUheRt7XWJs25FzQCzz85Z0oblFIjgBEA1atXN2HTt3N2duaTTz7hiSeeuOWAhhBCiP8xJbknAXmv5qgKJBfSJkkp5QyUA/7K35HWeiWwEgyLddxLwGA4tUoIa5OSv6IkM2XOfR9QVylVSylVCugHfJmvzZdA7mRbb+DHm18fhBBCWEGRI3etdbZSajTwHYZTIddorWOVUrMwzP18CawGPlZKJWIYsfcrvEchCqYLOTVOCEd0v+Njk85z11pHABH57gvN8/tVoM99RSIcWunSpUlPT6dixYqS4IXD01qTnp5O6dKl77kPm7tCVdinqlWrkpSURGpqqrVDEaJEKF26tPEahnshyV2UCC4uLnd1ibgQ4s7ssiqkEEI4OknuQghhhyS5CyGEHVLWOh1dKZUKnLjHp7sB5i34XfLJPjsG2WfHcD/7XENr7V5UI6sl9/uhlIrUWvtYO47iJPvsGGSfHUNx7LNMywghhB2S5C6EEHbIVpP7SmsHYAWyz45B9tkxWHyfbXLOXQghxJ3Z6shdCCHEHZTo5K6U6qyUildKJSqlJhfw+INKqfCbj/+mlKpZ/FGalwn7PE4pFaeUilFK/aCUqmGNOM2pqH3O0663UkorpWz+zApT9lkp9c+br3WsUuo/xR2juZnwv11dKfWTUurAzf/vp60Rp7kopdYopc4ppQ4W8rhSSi25+feIUUo1M2sApizXZI0fDOWF/wRqA6WA3wGPfG1GActv/t4PCLd23MWwzwHAQzd/f9kR9vlmu7LATmAP4GPtuIvhda4LHAAq3Lz9qLXjLoZ9Xgm8fPN3D+C4teO+z31uBzQDDhby+NPA/2FYyc4X+M2c2y/JI/cStzB3MShyn7XWP2mtr9y8uQfDyli2zJTXGeDfwDzganEGZyGm7PNwIExrfR5Aa32umGM0N1P2WQP/uPl7OW5f8c2maK13UsCKdHl0Bz7SBnuA8kops60dWpKT++PAqTy3k27eV2AbrXU2kAFULJboLMOUfc5rKIZPfltW5D4rpZoC1bTWXxdnYBZkyutcD6inlNqllNqjlOpcbNFZhin7PAN4QSmVhGH9iDHFE5rV3O37/a6U5JK/ZluY24aYvD9KqRcAH6C9RSOyvDvus1LqAeBdYEhxBVQMTHmdnTFMzXTA8O3sF6WUl9b6goVjsxRT9rk/sFZrvVAp5YdhdTcvrfUNy4dnFRbNXyV55H43C3Nzp4W5bYgp+4xS6ilgKtBNa32tmGKzlKL2uSzgBexQSh3HMDf5pY0fVDX1f3ur1jpLa30MiMeQ7G2VKfs8FPgUQGu9GyiNoQaLvTLp/X6vSnJyd8SFuYvc55tTFCswJHZbn4eFIvZZa52htXbTWtfUWtfEcJyhm9Y60jrhmoUp/9tbMBw8RynlhmGa5mixRmlepuzzSaAjgFKqIYbkbs9Lc30JDLp51owvkKG1TjFb79Y+olzE0eangSMYjrJPvXnfLAxvbjC8+J8BicBeoLa1Yy6Gfd4OnAWib/58ae2YLb3P+druwMbPljHxdVbAO0Ac8AfQz9oxF8M+ewC7MJxJEw0EWjvm+9zfDUAKkIVhlD4UCAaC87zGYTf/Hn+Y+/9arlAVQgg7VJKnZYQQQtwjSe5CCGGHJLkLIYQdkuQuhBB2SJK7EELYIUnuQghhhyS5CyGEHZLkLoQQduj/AXnQXbv07cReAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x224d1384ac8>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred_prob_lr = LR.predict_proba(X_test)[:,1]\n",
    "y_pred_prob_lsvm = SVM2.predict_proba(X_test)[:,1]\n",
    "y_pred_prob_svm = SVM.predict_proba(X_test)[:,1]\n",
    "fpr_lr, tpr_lr, thresholds_lr = roc_curve(y_test, y_pred_prob_lr)\n",
    "fpr_lsvm, tpr_lsvm, thresholds_lsvm = roc_curve(y_test, y_pred_prob_lsvm)\n",
    "fpr_svm, tpr_svm, thresholds_svm = roc_curve(y_test, y_pred_prob_svm)\n",
    "plt.plot([0,1], [0,1], 'k--')\n",
    "plt.plot(fpr_lr, tpr_lr, alpha=0.5, label='Logistic Regression')\n",
    "plt.plot(fpr_lsvm, tpr_lsvm, alpha=0.5, label='Support Vector Machine - Linear')\n",
    "plt.plot(fpr_svm, tpr_svm, alpha=0.5, label='Support Vector Machine - Kernel')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three models have roughly the same area under ROC curve."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change Threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logistic Regression\n",
      "Threshold: 0.4, Accuracy: 0.8627450980392157\n",
      "Threshold: 0.45, Accuracy: 0.8823529411764706\n",
      "Threshold: 0.5, Accuracy: 0.8627450980392157\n",
      "Threshold: 0.55, Accuracy: 0.8235294117647058\n",
      "Threshold: 0.6, Accuracy: 0.8235294117647058\n"
     ]
    }
   ],
   "source": [
    "print(\"Logistic Regression\")\n",
    "for t in [0.4, 0.45, 0.5, 0.55, 0.6]:\n",
    "    y_pred_rev = [1 if i > t else 0 for i in y_pred_prob_lr]\n",
    "    print('Threshold: {}, Accuracy: {}'.format(t, accuracy_score(y_pred_rev, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Support Vector Machine - Linear\n",
      "Threshold: 0.4, Accuracy: 0.8823529411764706\n",
      "Threshold: 0.45, Accuracy: 0.9019607843137255\n",
      "Threshold: 0.5, Accuracy: 0.8823529411764706\n",
      "Threshold: 0.55, Accuracy: 0.8823529411764706\n",
      "Threshold: 0.6, Accuracy: 0.8627450980392157\n"
     ]
    }
   ],
   "source": [
    "print(\"Support Vector Machine - Linear\")\n",
    "for t in [0.4, 0.45, 0.5, 0.55, 0.6]:\n",
    "    y_pred_rev = [1 if i > t else 0 for i in y_pred_prob_lsvm]\n",
    "    print('Threshold: {}, Accuracy: {}'.format(t, accuracy_score(y_pred_rev, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lowering the threshold of logistic regression and linear SVM, we can improve the accuracy for around 0.02 respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Support Vector Machine - Kernel\n",
      "Threshold: 0.4, Accuracy: 0.8823529411764706\n",
      "Threshold: 0.45, Accuracy: 0.8823529411764706\n",
      "Threshold: 0.5, Accuracy: 0.8823529411764706\n",
      "Threshold: 0.55, Accuracy: 0.8823529411764706\n",
      "Threshold: 0.6, Accuracy: 0.8627450980392157\n"
     ]
    }
   ],
   "source": [
    "print(\"Support Vector Machine - Kernel\")\n",
    "for t in [0.4, 0.45, 0.5, 0.55, 0.6]:\n",
    "    y_pred_rev = [1 if i > t else 0 for i in y_pred_prob_svm]\n",
    "    print('Threshold: {}, Accuracy: {}'.format(t, accuracy_score(y_pred_rev, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changing threshold doesn't help improve the accuracy of SVM with kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification Report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logistic Regression:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.82      0.69      0.75        13\n",
      "          1       0.90      0.95      0.92        38\n",
      "\n",
      "avg / total       0.88      0.88      0.88        51\n",
      "\n",
      "AUC: 0.888663967611336\n"
     ]
    }
   ],
   "source": [
    "print(\"Logistic Regression:\")\n",
    "y_pred_rev = [1 if i > 0.45 else 0 for i in y_pred_prob_lr]\n",
    "print(classification_report(y_test, y_pred_rev))\n",
    "print(\"AUC: {}\".format(roc_auc_score(y_test, y_pred_prob_lr)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "upport Vector Machine - Linear:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.90      0.69      0.78        13\n",
      "          1       0.90      0.97      0.94        38\n",
      "\n",
      "avg / total       0.90      0.90      0.90        51\n",
      "\n",
      "AUC: 0.9048582995951417\n"
     ]
    }
   ],
   "source": [
    "print(\"Support Vector Machine - Linear:\")\n",
    "y_pred_rev = [1 if i > 0.45 else 0 for i in y_pred_prob_lsvm]\n",
    "print(classification_report(y_test, y_pred_rev))\n",
    "print(\"AUC: {}\".format(roc_auc_score(y_test, y_pred_prob_lsvm)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Support Vector Machine - Kernel:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.82      0.69      0.75        13\n",
      "          1       0.90      0.95      0.92        38\n",
      "\n",
      "avg / total       0.88      0.88      0.88        51\n",
      "\n",
      "AUC: 0.8825910931174088\n"
     ]
    }
   ],
   "source": [
    "print(\"Support Vector Machine - Kernel:\")\n",
    "print(classification_report(y_test, SVM.predict(X_test)))\n",
    "print(\"AUC: {}\".format(roc_auc_score(y_test, y_pred_prob_svm)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Best Model\n",
    "The Best model is the pipeline that combines min-max normalization, PCA, and linear SVM with L2 penalty. The best parameters of PCA and linear SVM is as below. After applying the following model and lower the threshold to 0.45 for prediction, we get the accuracy of 0.902 and AUC of 0.905."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'PCA__n_components': 80, 'SVC__C': 0.1, 'SVC__tol': 0.0001}"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# {'PCA__n_components': 80, 'SVC__C': 0.1, 'SVC__tol': 0.0001}\n",
    "SVM2.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'SVC__max_iter': 500, 'PCA__n_components': 80, 'SVC__tol': 0.001, 'SVC__gamma': 0.01, 'SVC__C': 10}\n",
      "{'LR__C': 0.1, 'PCA__n_components': 100, 'LR__tol': 0.001, 'LR__penalty': 'l2'}\n"
     ]
    }
   ],
   "source": [
    "# Second best models\n",
    "# {'SVC__max_iter': 500, 'PCA__n_components': 80, 'SVC__tol': 0.001, 'SVC__gamma': 0.01, 'SVC__C': 10}\n",
    "print(SVM.best_params_)\n",
    "# {'LR__C': 0.1, 'PCA__n_components': 100, 'LR__tol': 0.001, 'LR__penalty': 'l2'}\n",
    "print(LR.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## V. Appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Appendix A: Concepts for Logistic Regression:\n",
    "\n",
    "#### A1. Binay Classification Type\n",
    "Input $x_i \\in \\mathbb{R}^b$ and output $y_i \\in {\\pm1}$\n",
    "\n",
    "we define a $\\textit{classifier f}$, which makes prediction $y_i = f(x_i, \\Theta)$ based on a function of $x_i$ and parameters $\\Theta$. In other words $f: \\mathbb{R}^d \\rightarrow {-1, +1}$\n",
    "\n",
    "In **Bayes classificaiton** framework, $\\Theta$ contains:\n",
    "1.  class prior probabilities on $y$,\n",
    "2. parameters for calss-dependent distribution on $x$.\n",
    "\n",
    "In **linear classification** framework, the prediction is linear in the parameters $\\Theta$.\n",
    "\n",
    "**Bayes classification** and **linear classification** are connected through ***log odds***.\n",
    "\n",
    "#### A2. Log Odds\n",
    "With Bayes classifier, we declare class $y=1$ if\n",
    "$$ p(x|y = 1)P(y = 1) > p(x|y = 0)P(y = 0) $$\n",
    "\n",
    "$$ \\Updownarrow $$\n",
    "\n",
    "$$ \\ln\\frac{p(x|y = 1)P(y = 1)}{p(x|y = 0)P(y = 0)} > 0 $$\n",
    "\n",
    "The second line is referred to as the $\\textit{log odds}$.\n",
    "\n",
    "#### A3. Lineaer Discriminant Analysis\n",
    "In the case where $p(x|y) = N(x| \\mu_y, \\Sigma)$ **(a single Gaussian with a shared covariance matrix)**\n",
    "\n",
    "$$ \\ln\\frac{p(x|y = 1)P(y = 1)}{p(x|y = 0)P(y = 0)} =\n",
    "\\ln\\frac{\\pi_1}{\\pi_0} - \\frac{1}{2}(\\mu_0 + \\mu_1)^T\\Sigma^{-1}(\\mu_1 - \\mu_0) + x^T\\Sigma^{-1}(\\mu_1 - \\mu_0)$$\n",
    "\n",
    "This is also called ***lineaer discriminant analysis*** (used to be called LDA).\n",
    "\n",
    "So we ca write the decision rule for the Bayes classifer as a linear one:\n",
    "$$ f(x) = sign(x^Tw + w_0) $$\n",
    "where\n",
    "\n",
    "$$ w_0 = \\ln\\frac{\\pi_1}{\\pi_0} - \\frac{1}{2}(\\mu_0 + \\mu_1)^T\\Sigma^{-1}(\\mu_1 - \\mu_0) $$\n",
    "$$ w = \\Sigma^{-1}(\\mu_1 - \\mu_0) $$\n",
    "\n",
    "This Bayes classifier is one instance of a linear classifier.\n",
    "\n",
    "Setting $w_0$ and $w$ this way may be too restrictive - it assumes single Gaussian with shared covariance. If we relax what values $w_0$ and $w$ can take we can do better. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Appendix B: Linear Classifiers\n",
    "\n",
    "#### B1. Definition\n",
    "A $\\textit{binary linear classifier}$ is a function of the form\n",
    "$$f(x) = sing(X^Tw + w_0),$$\n",
    "where $w \\in \\mathbb{R}^d$ and $w_0 \\in \\mathbb{R}$. Since the goal is to learn $w, w_0$ from the data, we are assuming that $\\textit{linear separability}$ in $x$ is an accurate property of the classes.\n",
    "\n",
    "#### B2. Linear Separability\n",
    "Two sets $A, B \\subset \\mathbb{R}^d$ arfe called linearly separable if\n",
    "\n",
    "$$x^Tw + w_0 > 0 \\ \\ if \\ \\ x \\in A (e.g, class +1)$$\n",
    "$$x^Tw + w_0 < 0 \\ \\ if \\ \\ x \\in B (e.g, class -1)$$\n",
    "\n",
    "The pair $(w, w_0)$ defines an $\\textit{affine hyperplane}$. It is important to develop the right geometric understanding about what this is doing.\n",
    "\n",
    "#### B3. Two methods:\n",
    "\n",
    "- **Least squares:** One simple idea is to treat classification as a regression problem. However, using regression for classification problem is not robust because it's sensitive to outliers\n",
    "- **Perceptron:** The perceptron represents a first attempt at linear classification by directly learning the hyper plane defined by $w$. It is not used as much anymore because of some drawbacks: convergence issues and the assumption on linear seperability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References for Model Introduction and Algorithms\n",
    "- Applied Machine Learning Certification - Columnbia Engineering Executive Education\n",
    "- Post Graduate Diploma of Applied Machine Learning and Artificial Intelligence - Columnbia Engineering Executive Education\n",
    "\n",
    "#### Note: The coding was done through personal works and researches and not borrowed from the certification course."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}