application/SVC_on_USPS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# USPS Handwriting Dataset Recognition"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The dataset refers to numeric data obtained from the scanning of handwritten digits from envelopes by the U.S. Postal Service. The original scanned digits are binary and of different sizes and orientations; the images here have been deslanted and size normalized, resulting in 16 x 16 grayscale images (Le Cun et al., 1990). There are 7291 training observations and 2007 test observations. \n",
    "\n",
    "| Digit | Label |\n",
    "|-------|-------|\n",
    "| 0     | 10    |\n",
    "| 1     | 1     |\n",
    "| 2     | 2     |\n",
    "| 3     | 3     |\n",
    "| 4     | 4     |\n",
    "| 5     | 5     |\n",
    "| 6     | 6     |\n",
    "| 7     | 7     |\n",
    "| 8     | 8     |\n",
    "| 9     | 9     |\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy as sp\n",
    "from sklearn import svm\n",
    "from scipy import ndimage\n",
    "from sklearn import metrics\n",
    "from numpy import genfromtxt\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import preprocessing\n",
    "from sklearn.decomposition import PCA, KernelPCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train.csv and Test.csv contain a 8000x257 and a 3000x257 matrix, each entry is the pixel value in the image with the label on the first column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dimension of Training set: 8000 x 256\n",
      "Dimension of Test set : 3000 x 256\n"
     ]
    }
   ],
   "source": [
    "# Load data\n",
    "train = genfromtxt('/Users/macbookpro/Desktop/Train.csv', delimiter=',')\n",
    "test = genfromtxt('/Users/macbookpro/Desktop/Test.csv', delimiter=',')\n",
    "\n",
    "\n",
    "# Shuffle the training/test set\n",
    "np.random.shuffle(train)\n",
    "np.random.shuffle(test)\n",
    "\n",
    "\n",
    "# Extract labels from training/test set\n",
    "label_train = np.array([int(1+train[i][0]) for i in range(train.shape[0])])\n",
    "label_test = np.array([int(1+test[i][0]) for i in range(test.shape[0])])\n",
    "\n",
    "\n",
    "# Reshape training/test dataset\n",
    "train = train[:,1:257]\n",
    "test = test[:,1:257]\n",
    "\n",
    "\n",
    "# Show the dimension of training/test set\n",
    "print (\"Dimension of Training set: \" + str(train.shape[0]) + \" x \" + str(train.shape[1]))\n",
    "print (\"Dimension of Test set : \" + str(test.shape[0]) + \" x \" + str(test.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Have a glimpse of the sample images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAADuCAYAAAAp6fzCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHk9JREFUeJzt3XeQVUW+B/DvjywMGTEBSxLUpwsOaD3CwhpYJIuKgKKr\nC8ozW6JVWoIrSHBZBQWXFQUUFAkiSBLBgAj4QCgBFwRhV5AkQdKQhiD9/riXV3Rg7p0759zQ8/1U\nWeWv53fPOdNz5zeH7tt9RCkFIiLKfEVSfQFERBQMFnQiIk+woBMReYIFnYjIEyzoRESeYEEnIvKE\nVwVdRIqKyBERqRFkbmHEvkwu9nd4ClPfprSgRzvu7H9nROT4OfHd+T2eUuo3pVSWUmprkLkFJRFD\nRGSniBwUkYUicmXA5ygsfdlARBaIyD4ROe34emURmSkiR0Vki4h0Dek6Ckt/lxKR16Pv3QMiMlJE\nioV8zsLSt4HXhZQW9GjHZSmlsgBsBdDhnLaJZn7Yb6QQdQdwD4BmAKoAWAFgfJAnKER9eRLAZAAP\nnOfrbwI4CqAqgD8DeFtErgj6IgpRfz8PoAGA/wJQH8B/A3guzBMWor4NvC6k9ZCLiAwUkSkiMklE\nDgPoISJNRGRZ9C/aLyIyQkSKR/OLiYgSkZrR+P3o1+eJyGER+V8RqZXf3OjX24jIRhE5FL1LWSoi\n98X5rdQC8LVSarNS6jSAiYj8giSNL32plFqvlBoH4AfH91gOwK0A+iqljiqlFgGYA6BHwh2XIF/6\nG0AHACOUUgeUUnsAjATwl2B6KTEe9W3gdSGtC3pUZwAfACgPYAqA0wCeQOQvWjMAtwDoncfr7wLQ\nD0AlRP7av5TfXBGpCmAqgGei590M4PqzLxKRWtE30qXnOe4kAPVFpK6IlEDkznFeHtcRFh/6Mi/1\nAeQqpX46p20NkvzH8xw+9rcAqCkiWXHmh8WHvg28LmRCQV+ilJqtlDqjlDqulFqhlFqulDod/cV9\nC0DLPF4/TSm1Uil1CpG/gA0TyG0PYLVSamb0a8MB/Hr2RdG/sBWUUjvPc9wdAL4BsAnAMQCdAPSJ\n/a0Hzoe+zEsWgENGWw6AsgkcKwg+9PenAJ4UkSoicgmAx6LtF8T65kPmQ98GXhcyoaBvOzcQkStE\nZK6I7BKRHAADEPnreD67zvn/Y4j80uc399Jzr0MppQBsj+Paz+oP4FoAlwEoBWAIgC9FpFQ+jhEE\nH/oyL0cAlDPaygM4HNDx88uH/h4AYB0i/9JZAmAGgFycU7hSxIe+DbwuZEJBN7eDHA1gLYC6Sqly\nAF5A5J+BYfoFQLWzgYgIIj+EeDUEMEkptTN6BzEGwEUAAp+si8GHvszLjwAuOHeME5EJvXUBHT+/\nMr6/lVLHlFIPKaUuU0rVAXAAwMpo8UqljO9bhFAXMqGgm8oi8s/qoxL5iE9e42RBmQMgW0Q6SGRG\n/QkAF+bj9SsA3CkiVUWkiIjcH23/Ka8XJUHG9aVElAJQIhqXio4/QimVA2AmgJdEpLSItADQDsD7\nQX8TCcrE/q4mIpdE37dNEfnUy4vhXGqBZFzfIoS6kIkFvQ8ikweHEfmrPCXsEyqldgPoCmAYgH0A\n6gBYBeAEAIhIbYl8RvZ8kx+DEflUxhoABwE8CuC2aAFKpUzsyzoAjiPSl0Wj/3/uJ17+B5Fhl70A\n3gPwgFJqQwjfSiIysb8vB7AMkeGscQCeVkp9EfZ1JyAT+zbwuiCp/5dT5hGRogB2ArhDKbU41deT\nydiXycX+Dk869G0m3qGnhIjcIiIVRKQkIh9hOgXg2xRfVkZiXyYX+zs86da3LOjxa47I2NZeAK0B\ndFZKnUjtJWUs9mVysb/Dk1Z9yyEXIiJP8A6diMgTSd3URkRi/nPg7bffttqeeeYZLc7KstcAZGdn\na3Hr1q2tnIoVK8a8RpecHH3SeeDAgVbO9u3BrI1RSiX02dl4+jZRDRo00OKhQ4daOUWK6PcGb7zx\nhpUzc+bMYC8snxLtWyDc/jXVrFnTauvQoYMWN23a1MopX758IOffuHGj1davXz8tPnzYXqsV5HvX\nfD+VLWsv9jXbLrvM/gj4999/r8UlSpSwci64QF/02rx5cyvnD3/4g9Vm5uXm5lo5Zl8uWLDAypky\nRf9AzpkzZ6ycePuWd+hERJ5gQSci8gQLOhGRJ1jQiYg8kXZP+ihWzL6kHTt2aHHRokWtnJIlS2rx\nvn37rJz58+dr8cKFC60c18Tp008/rcX33XeflfPFF/pq6N697a0kgpo4DVKlSpW02PxeAeChhx7S\n4goVKsQ8ruvnmOpJ0XRwxRX2vkvmBwGuu+46K2fGjBla/M0331g55vsbcE+wmRo1aqTFrknBTZs2\naXGPHuE+M2T16tVavHLlSitn27ZtVpvptttu0+I//vGPVk7DhvrOuUePHrVy4vlAxZEjR6w2c/La\nVTt++eUXLf7qq69inut8eIdOROQJFnQiIk+woBMReSKpS/9dCwguueQSLX733Xet1/3pT3+KeWzz\ng/9r1661cg4ePBjzOC7mgg1zXA4Axo0bp8WuMfwuXbposWsMP8yFRRdffLHV9tlnn2lx5cqVrZy/\n/EV/JnCbNm2snE6dOmnxRRddZOW89957Vlv//v212JwvCVIqFhaZC9ymT59u5UyYMEGLXfMYrnHd\nZDLHkF2/k5MnTw7svXvs2LGY51uyZEm+z1Wliv0Qo65du2rx2LFjrZyrr77aarvqqqu02PX7ZS4I\nc81PmHMft9xyi5XDhUVERIUMCzoRkSdY0ImIPMGCTkTkiZRPipofqi9evLj1uuPHj2txtWrVrBxz\nIqNXr14JXWM8XItm6tWrl+f1APYueuYuhgCwe/fuwCaWzN3opk6dGvM4rn6LZ6KyVKlSWuya/Hnn\nnXesttOnT2uxubMmAEybNi3m+eMR9qSoa0J53rx5WuzaqXPWrFmJXlYoXItvzB0Jy5UrZ+UEOSlq\nTta7dlLs3r27Fps7KwJAMutb3bp1rbYvv/xSi9evX2/lmB8ocO3ayElRIqJChgWdiMgTLOhERJ5I\n+Ri6udincePG1uvMRTqusTJzTM+1+MW1oVFYRo4cabU9+uijWty3b18rZ+DAgYGNQ37yySda3KxZ\nM+t19evX1+Jdu3Ylcvq4uDZVe+2117TYNYa/dOlSLTYXOgHA1q1bY54/7DF083sBgEGDBmnx3r17\nE72EQLjmf8ynEb3wwgtWzuTJk7X4wQcftHJycnICe++ac2muhUVmf5sbegHAP/7xDy0uyMZXpksv\nvVSLv/76ayvHnP9zPfkongWPHEMnIipkWNCJiDzBgk5E5AkWdCIiT6R8UrR9+/ZaPGfOnJjHeeKJ\nJ6y2V155RYs3btxo5dx8881abC5qKogyZcpo8aeffmrlmJOSTz31lJUzfPjwwCaWTp06pcXmbo8A\n8PHHHydyutDUqFHDajMXFpmLXACgbdu2Wux6mk3Qk6KPPfaYFq9YscJ63bJlyxI9ZShcT43q2LGj\nFrveu3feeacWHz582MoJc6dQlxIlSmhxz549rRxzwnfu3LlWzvPPP6/Fe/bssXJcC6nMyXrX06HM\nnRw3bNhg5cSDk6JERIUMCzoRkSdY0ImIPJHyMfSgmE99ueeee6wcc+x9xIgRCZ2rSBH776C5wc5H\nH31k5ZhPKLrpppusnCDHIc3FNldeeaX1ulQ/CSce5iZjrgUc5uKzdu3aWTm5ubmBjqGbP2NzvBSw\nNx8LSvXq1a22u+++22ozr8l8wj0ArFy5UotbtGhh5ZgLZFySPYYeD/OJaK7FfOb8gGsez3wPAkDT\npk21uFu3blaO672aCI6hExEVMizoRESeYEEnIvIECzoRkSe8mRTt3LmzFk+fPt3KMZ8e4pqUdClf\nvrwWDxs2zMoxdwBctGiRleN6EowpyImlVatWafG1116byKHTzg033GC1mT9b146BAwYMCHRSNCcn\nR4tdC7fmz5+vxa5rN5/2ZD7ZCrAntF2Toq4JT/NYrqfhmBP6CxYssHLikY6TovEwJ4FdC6tci9km\nTZqkxa4dKI8cOVLAq4vgpCgRUSHDgk5E5AkWdCIiT7CgExF5wptJ0WrVqmnx4sWLrRxzgujGG2+0\nclwr4u6//34t7tGjh5VjrkAbPnz4ea81L5wUTYz5+LEtW7ZYOZ06dQp0UvTFF1/UYnPFIWA/8u3f\n//63lbNp0yYtdl37hx9+qMXbt2+PeRwAqFu3rhabE6AAMGvWLKstEZkwKZqVlWW1mf22f/9+K+f1\n11+32gYOHKjFrscgmqt3f/zxx7iu08RJUSKiQoYFnYjIEyzoRESe8GYM3fTdd99ZbeYY8okTJ6wc\nc8wTsJ9g8uSTT1o5U6dOze8lOnEMPTGpGENPpT59+lhtf//73622efPmabH5dCIA+O233wK5pnQc\nQ7/sssu02NzxFLB3xDSffgW430/mgsPJkydbOXXq1NFicwEiACxZssRqM3EMnYiokGFBJyLyBAs6\nEZEnWNCJiDxhzwBmqEqVKmlxmTJlYr6mZMmSVtvLL79stZmTTa6FB+nI3JGvYsWKVs6BAweSdTmB\nKV26tNVWoUIFLf7++++tHNeimkxx0UUXabH5OEUAOHjwoNVm5gU1AZopnn/+eS12PT6ycePGWnzs\n2LG4jn3o0CEt7tChg5Xz6quvarG5+yYAPP7441o8duzYuM7vwjt0IiJPsKATEXmCBZ2IyBPejKEP\nGjRIi+vVqxfzNZ988onVNmDAAKvNtWFXJqhcubIW16pVy8rJxDF0cyM0wB5Dd41D9uvXL7RrCtuz\nzz6rxa4nFrme0uTaDKwwMcfHR44caeXEO2Yei7lACbDnMC6++GIrx3wCmmvxU7x4h05E5AkWdCIi\nT7CgExF5ggWdiMgTGbHbYrly5bR4xowZVk6zZs202LVoyOR6YlFBJiSCEOSOdaNGjdLim266yXpd\nkyZNtDgdF02ZOwsOHjzYyunSpYsWu57Ck2jfAsndbbFq1apW2+7du2O+zvWeP3nyZCDXFI903G2x\nf//+Wty8eXMrp3Xr1lrsmtwMSsOGDa22ZcuWafF7771n5fTq1Yu7LRIRFSYs6EREnmBBJyLyRNot\nLHJtvGSOa2dnZyd07G3btuUZ+2bIkCFafPvtt1s5ixcv1uJ77rnHyvnXv/6lxadOnUroemrWrGm1\ntWzZUotdT4OqXbu2Frs22fr0008TuqZ0ZC4icjE3fQIS/7n4bPTo0VrsemKQ2d/m7w0Q3KZma9eu\ntdp27typxcWLF0/4+LxDJyLyBAs6EZEnWNCJiDzBgk5E5Im0W1j0yCOPWG1vvPFGvs/lWhxgTvhN\nnjw538cNW5iLM1yTie+//74WZ2VlWTlr1qzR4m+//Tbm9ZgLlgDgqquustrMp758+OGHVo65A+aO\nHTtint8lXRcW9ejRQ4v/+c9/Wjnr16/X4latWlk5Zl8mWzouLDK5dmH94IMPtHj58uVWzrRp06y2\nzZs3xzyf+SS14cOHWznmpH/37t2tnMWLF3NhERFRYcKCTkTkCRZ0IiJPpN0Yumvxifnk7lKlSlk5\n5tOH/vOf/1g58Yz9plqyxyHNpxq1aNHCymnfvr0Wu56WEw/XXMh3332nxdu3b0/o2PFI1zH05557\nToubNm1q5TzzzDNavGHDhrAuJ2GZMIbuUrRoUS2+/PLLrRxzAy8AaNu2bcxjr1u3Tovnz59v5Sxa\ntEiLc3NzrZx4+5Z36EREnmBBJyLyBAs6EZEnWNCJiDyR1ElRIiIKD+/QiYg8wYJOROQJFnQiIk+w\noBMReYIFnYjIEyzoRESeYEEnIvIECzoRkSdY0ImIPMGCTkTkCRZ0IiJPsKATEXmCBZ2IyBMs6ERE\nnmBBJyLyBAs6EZEnWNCJiDzBgk5E5AkWdCIiT7CgExF5ggWdiMgTLOhERJ5gQSci8gQLOhGRJ1jQ\niYg8wYJOROQJFnQiIk+woBMReYIFnYjIEyzoRESe8Kqgi0hRETkiIjWCzCX2bdjYv+EpTH2b0oIe\n7biz/50RkePnxHfn93hKqd+UUllKqa1B5haUiDQQkQUisk9ETod9vug5C0vf3i0iP4pIjojsFpF3\nRCQrCectLP3L925IRGSM8b2eEJEDBTqmUiqo6ysQEdkCoJdS6vM8cooppZLypgqSiFwJoAmAgwCm\nKqWKJfn8W+Bv39YAcEIptVtEygJ4G8BOpdRTSbyGLfC3f/neTRIReR/AMaXUg4keI62HXERkoIhM\nEZFJInIYQA8RaSIiy0TkoIj8IiIjRKR4NL+YiCgRqRmN349+fZ6IHBaR/xWRWvnNjX69jYhsFJFD\nIjJSRJaKyH3xfB9KqfVKqXEAfgiwewrEo77dqpTafU7TGQB1C95DBeNR//K9G1LfGt9TWQCdAYwv\nSN+kdUGP6gzgAwDlAUwBcBrAEwCqAGgG4BYAvfN4/V0A+gGoBGArgJfymysiVQFMBfBM9LybAVx/\n9kUiUiv6Rro0/99eSnnRtyLSUkQOAcgB0BHAa3lcRzJ50b9pyre+7YLIvyyXxpF7XplQ0JcopWYr\npc4opY4rpVYopZYrpU4rpX4C8BaAlnm8fppSaqVS6hSAiQAaJpDbHsBqpdTM6NeGA/j17IuUUpuV\nUhWUUjsL8o2mgBd9q5RapJQqD6A6gFcQ+aVLB170b5ryrW//jALenQNAUsfDErTt3EBErgDwKoBG\nAEoj8j0sz+P1u875/2MA8powO1/upedeh1JKicj2mFee/rzqW6XUdhH5HJE7t+tj5SeBV/2bZrzp\n2+gQTnMA9+b3taZMuEM3Z21HA1gLoK5SqhyAFwBIyNfwC4BqZwMREQCXhXzOZPCxb4sBqFPQiwqI\nj/2bLnzq23sBLFJK/VzQC8qEgm4qC+AQgKMSmYHPa5wsKHMAZItIBxEphshY3YXxvlgiSgEoEY1L\niUiJcC61QDKxb3uISPXo/9dEZHzzixCuMwiZ2L98755fgfoW+P8/AvcCeDeIC8rEgt4HkfGmw4j8\nVZ4S9gmjn6LoCmAYgH2I3AGuAnACAESktkQ+R3q+yY86AI4DWAOgaPT/0+ZTA+fIxL69BsAyETkK\nYAmAdUjOL3MiMrF/+d49jwD6FogMtVQF8FEQ15Q2n0PPJCJSFMBOAHcopRan+np8wr4NF/s3POnQ\nt5l4h54SInKLiFQQkZKIfITpFIBvU3xZXmDfhov9G55061sW9Pg1B/ATgL0AWgPorJQ6kdpL8gb7\nNlzs3/CkVd9yyIWIyBO8Qyci8kSyN9qx/jnQuHFjLe7UqVMg55o7d67Vtnr1ai3Ozc0N5FxBUkol\n9NlZV98GpXbt2lpcunRpK+fCC/VPa9WsWdPK6datm9V24IC+uVyZMmWsnBIl9E/Jbd682cpZvlxf\nQ/Lxxx9bOfv370/4c8lh9m+qNWjQQIs7dOhg5YwZM0aLd+3aZeWk43vX1LChvSC0Xr16MV/3+ef2\n3mD79+8P5JriEW/f8g6diMgTLOhERJ5gQSci8gQLOhGRJ1K+22J2drYW9+3bN5DjxnOcihUrWm0n\nT5602o4dOxbINaVasWL2j/uBBx7Q4t697VXzV155pRabk5QAcOTIES129ePRo0ettqVL9e2fzeO4\nztezZ08rx7zuqlWrWjkENGvWzGqbNGmSFlevXt3KMX9PH3744WAvLAHmz7hr165Wzr336hsYXnPN\nNVZOyZIlY55r5057B1yzbcKECVbOlCn6DgR79uyJea6C4B06EZEnWNCJiDzBgk5E5ImkLv13LSB4\n8EH9AdejR49O2vW4vPXWW1aba1w5LEEuzqhfv74WT5w40XqduQBo3bp1Vs7ChQu1eM6cOVbOli1b\ntPjXX3+1chL19NNPa/Ff//pXK2fjxo1a3L59eytn586dhW5hUaNGjbT43XfftXKuvvrqmMfZvl1/\nEI9r8dGqVatCW1hUt6793O8VK1ZocYUKFWKe67PPPrPaDh48qMWVK1e2cm688caYx3Y5cULf1uXF\nF1+0cl5++eWYx+HCIiKiQoYFnYjIEyzoRESeYEEnIvJEyhcWUXjMyZ3Zs2dbOUOGDNFi14KgRBQt\nWtRqc02+NW3aVIsff/xxK+eCCy7Q4v79+1s55mR2Tk5OXNfpkypVqlhtU6dO1WJz58x4jRo1SovN\nnUuDZi4aWrZsmZVjToKuXbvWyhk6dKgWuz4YcObMGS12vXerVatmtZmLlu68804rx3zPDx482Mp5\n9tlntdi1QCpevEMnIvIECzoRkSdY0ImIPMExdEP58uWtNnOBzs8//2zlpOPTj7755ps84yC1adNG\ni/v162flmE/GAexFSiNGjLByxo8fr8W+bJZWEK4nQn399ddWm2ujrVhcY9Hz5s3L93EKwhwzdy32\nWbVqlRa3atXKytm3b1++z/3bb79Zba7f+ZdeekmL//a3v1k5Tz75ZMwc8yldBVnsyTt0IiJPsKAT\nEXmCBZ2IyBMs6EREnkj5pKg5abFmzRorp169elpsLjQJ0s0332y1XXHFFVrcrVs3K2fDhg2hXVO6\ncU3Ivfnmm1p8++23WzkrV64M65K8Z07Mjxs3zspJZAIUAE6dOqXFrh1Hw15IZKpVq1bMnHbt2mlx\nIhOgQXItyjP71uX06dNavGDBgoSvgXfoRESeYEEnIvIECzoRkSdS/sQicxMe19iZ+ZQVc0w7SD49\nsSiZNm3apMWuxRlLly612t555x0tdi1+MjdPSlSifQskt3/N3wnA3mSrZcuWgZ3PfALUgAEDEjpO\nkO/dvXv3arFrc67u3btr8ZEjRxI5fWBq1KhhtZlza675v1mzZmlxp06drBw+sYiIqJBhQSci8gQL\nOhGRJ1jQiYg8kfKFRbfeeqsWjx49OkVXQgWRnZ2txR07drRy2rdvb7WZu/iZO+gBQIsWLQp4dZnF\nnAAFEp8E/emnn7TY9bSnadOmJXTsMDVs2FCLd+/ebeWYC3JS7frrr7fazElQ1+Ij82lQBcE7dCIi\nT7CgExF5ggWdiMgTLOhERJ5I+aSouUrqhx9+sHLGjBmjxebOc5R6hw8f1uKJEydaOa7Jt+nTp2ux\n+Sg7ALjwwgu12FxFmEluuOEGq818f9euXTuhY7t29jMnWCdMmJDQsZNtx44dqb6EfOvcuXPMnKNH\nj1pt8+fPD+waeIdOROQJFnQiIk+woBMReSLlY+i7du3KMwZSv4sa5V/ZsmWttsmTJ1ttrVq10uJH\nHnnEysnkMXNzZ9Dx48dbOYk8aeirr76y2vr27Wu1uXa4pGDUqVNHizt06BDzNcOHDw/rcgDwDp2I\nyBss6EREnmBBJyLyBAs6EZEnUj4pmm7uuOMOq61JkyZa3K1bNyvHtSCqMDEXVbgmf0Tsp2h17dpV\ni2fMmBHshaXYihUrtDgrKyuh4/z6669aPGzYMCuHE6DJ1ahRIy12fRBg//79WuyaFA8S79CJiDzB\ngk5E5AkWdCIiT3AM3VCpUqWYbeZTSHxnPjHotddes3Iuv/xyLR40aJCV8/bbb1tt+/btK+DVpY+R\nI0dabaVLl873cQ4dOmS19enTR4tnz56d7+NS4qpVq2a1DR48OObrzM3ntm7dGtg1ufAOnYjIEyzo\nRESeYEEnIvIECzoRkSc4KZqAW2+91WqrUaOGFmfKApmSJUtq8cMPP2zlmBOcM2fOtHI6duyoxdu3\nbw/g6tLbJZdcosXZ2dlWTpEi+b9n2rNnj9WWKU8a8lXr1q2tNnO3xZycHCtn7NixoV2TC+/QiYg8\nwYJOROQJFnQiIk9kxBi6+aSbTZs2WTmuDbPC4noyzLZt27TY9RSaL7/8UovXrl0b7IXF0KVLF6vN\n3OTp4MGDVk67du20eOHChcFeWIYaNWqUFjdt2jSh4yiltLhXr14JXxOFo379+jFzXIvkli1bFsbl\nnBfv0ImIPMGCTkTkCRZ0IiJPsKATEXlCzAmZUE8mEsjJzCeFAMDKlSuDOHTCVq9ercUtW7a0cnJz\nc7X45MmTVo5Syn6sTxxcfTtx4kQt/v3vf2+9ztwxbtq0aVbOqVOnErmktJNo3wLu/k3kd8e1aOiu\nu+7S4i+++CLfx00HQb53083cuXOttrZt22px7969rZy33norkPPH27e8Qyci8gQLOhGRJ1jQiYg8\nkZFj6K6nxxcvXlyLzfEtALj22mu12PVUnUSZ/ZjouHOQ45BDhw7VYteCKNc4vq+CHkPv2bOnFl93\n3XXW6/bu3avFrjFVc1FapvJ5DL158+ZW2+9+9zstXrBggZVj/vwTxTF0IqJChgWdiMgTLOhERJ5g\nQSci8kRSJ0WJiCg8vEMnIvIECzoRkSdY0ImIPMGCTkTkCRZ0IiJPsKATEXmCBZ2IyBMs6EREnmBB\nJyLyBAs6EZEnWNCJiDzBgk5E5AkWdCIiT7CgExF5ggWdiMgTLOhERJ5gQSci8gQLOhGRJ1jQiYg8\nwYJOROQJFnQiIk+woBMReYIFnYjIE/8HaNzwPONeGPgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11aa9fc88>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Variable for the location of subplot\n",
    "i = 1\n",
    "\n",
    "\n",
    "# Plotting \n",
    "for index in range(8):\n",
    "    img = test[index].reshape(16, 16)\n",
    "    rotated_img = sp.ndimage.rotate(img, 90)\n",
    "    plt.subplot(2, 4, i)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(rotated_img, cmap=plt.cm.gray, origin='lower')\n",
    "    plt.title('Training: ' + str(label_test[index]))\n",
    "    i = i + 1\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform SVC with Linear Kernel on the original images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Concatenate two matrix in order to perform PCA and Kernel PCA later\n",
    "comb = np.concatenate((train, test), axis=0)\n",
    "\n",
    "\n",
    "# Apply Scaling to X_train and X_test\n",
    "std_scale1 = preprocessing.StandardScaler().fit(train)\n",
    "train_std = std_scale1.transform(train)\n",
    "\n",
    "std_scale2 = preprocessing.StandardScaler().fit(test)\n",
    "test_std = std_scale2.transform(test)\n",
    "\n",
    "\n",
    "# SVC\n",
    "clf = svm.SVC(kernel='linear') \n",
    "clf.fit(train_std, label_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for classifier SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
      "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',\n",
      "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
      "  tol=0.001, verbose=False):\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.93      0.99      0.96       300\n",
      "          2       0.91      0.94      0.92       300\n",
      "          3       0.91      0.92      0.91       300\n",
      "          4       0.93      0.96      0.95       300\n",
      "          5       0.92      0.90      0.91       300\n",
      "          6       0.97      0.95      0.96       300\n",
      "          7       0.97      0.97      0.97       300\n",
      "          8       0.94      0.90      0.92       300\n",
      "          9       0.97      0.94      0.95       300\n",
      "         10       0.99      0.95      0.97       300\n",
      "\n",
      "avg / total       0.94      0.94      0.94      3000\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Now predict the value of the digit on the test set:\n",
    "expected = label_test\n",
    "predicted = clf.predict(test_std)\n",
    "\n",
    "\n",
    "# Print summaries of the classifier\n",
    "print(\"Classification report for classifier %s:\\n%s\\n\"\n",
    "      % (clf, metrics.classification_report(expected, predicted)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Perform SVC with polynimial kernel on the original images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='poly',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Poly SVC with default degree=3\n",
    "clf = svm.SVC(kernel='poly')\n",
    "clf.fit(train_std, label_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for classifier SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
      "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='poly',\n",
      "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
      "  tol=0.001, verbose=False):\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.98      0.99      0.98       300\n",
      "          2       0.96      0.97      0.97       300\n",
      "          3       0.98      0.98      0.98       300\n",
      "          4       0.97      0.99      0.98       300\n",
      "          5       0.95      0.99      0.97       300\n",
      "          6       0.99      0.95      0.97       300\n",
      "          7       0.99      0.98      0.98       300\n",
      "          8       0.93      0.94      0.94       300\n",
      "          9       0.98      0.98      0.98       300\n",
      "         10       0.99      0.97      0.98       300\n",
      "\n",
      "avg / total       0.97      0.97      0.97      3000\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Now predict the value of the digit on the test set:\n",
    "expected = label_test\n",
    "predicted = clf.predict(test_std)\n",
    "\n",
    "\n",
    "# Print summaries of the classifier\n",
    "print(\"Classification report for classifier %s:\\n%s\\n\"\n",
    "      % (clf, metrics.classification_report(expected, predicted)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAADuCAYAAAAp6fzCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXu0VNWx7r+ShzwFATmIIIqoKBAUwRcKim8TlRAT1Bgx\n8ai5DnK9KjcaTuJ16NGYkYd61HgyEo/AwZhoIPiCDFRQlJcawSgiooIGeT/kDSrO+0d3RphVBXvS\nu3v3Zp7vNwZjUHNX91pr9uzaa9e3qqaEEEAIIWTvZ59qnwAhhJDywIBOCCGZwIBOCCGZwIBOCCGZ\nwIBOCCGZwIBOCCGZwIBOCCGZUO8DuogsFpGtIrJJRFaIyCgRaZH42kNEZKKIrBOR5SLygIg0dPxG\nFt9/k4hsE5EdO9nzEo91lojMEZGNIvKBiFy1p9dabmo5d8NF5HUR2S4io5yfnyEi74rIFhGZKiJd\ndvE+83aayx3F+f2HPTLhPHqLyGQRWSMiXzg/bysiT4rI5uL1Dk25vkpTy7k/SkSmiMh6EXlfRL6+\nC79yrNvBO31G00Wk+55cZ7VgXNgFIYR6/Q/AYgBnFv9/EIC3Adyd+NqJAEYDaAKgA4C3APzvGl5z\nJYBX9vAc9wWwEcBVAATACQA2A+i5F8/dEACDATwEYJT6WTsA6wF8szi3PwcwK+E9XwTwr3t4DUcB\n+F7xfL5wfv4EgN8DaA5gIIANALrvresWQEMA7wG4EUADAIOKa+mICqzb7sX5Orl43J8AWACgQbXn\nr1LzW/TPNi7U+zv0nQkhfAJgEoCeiS85FMAfQwjbQgjLAfwFQI8KnFo7AC0A/HcoMBuFL+VRFThW\nSezp3IUQxocQJgBY4/x4CIB5IYQnQgjbANwGoHcl7u5CCPNDCP8F4B39MxHZD4VfOj8OIWwOIbwE\n4BkAl5f7PGrDHs59dwAdAdwTQtgRQpgCYDqA71Tg1M4F8GIIYUYI4QsAP0XhO3NKBY5VMRgX/sle\nFdBFpDOA8wHMKdq3iMgzu3nJvQCGikgzETkIwHkofHilHHuSiIzwflZcUE8A+K6INBCR/ijcNUwv\n5ViVoIS52x09ALz5DyOEsBnA+yjhSyEih4rIpyLSsYTzOBLAthDChzuNvVnKeVSSMsy9ID1Y6WPv\nct3uhpKOVS0YF/6JyRvVUyYU86frATwL4C4ACCHcXcPrpgG4BoU/Kxug8GfWhFJOIIRwXg0ujwL4\nHYAHAQQA14YQlpZyrDJT6tztjhYAVqmxDQBa7ukbhRAWAWhdi/NYX47zqBClzP0CACsB/F8RuQfA\n6SikkqaWcgI1rNvnAPy7iAwAMAvAv6EQE5qVcqwqwLig2Fvu0AeHEFqHELqEEK4LIWyt6QUisg8K\nv3XHo5BfbQdgfwA/K/fJiUgPFPK4lwJoDKAXgH8TkXPLfawS2OO5S2ATgP3UWCsU8oV1SX05j12x\nx3MfQvgchTTSVwEsB3ATgMcBLCn3yYUQ5qGgTzwEYCkKvwgXVOJYFYJxQbG3BPRSaAPgYAAPhBC2\nhxDWAHgEhT/Nyk0vAO+EEJ4PIXwZQngXhZxefQjolWAegN7/MESkOYDDiuN1yQIATUXk0J3Gelfh\nPMpKCOFvIYSBIYS2IYRzAHQF8GqFjvV4CKFHCKEdgH9H4TvzeiWOVU/IOi5kG9BDCKsBLALwfRFp\nKCKtAQwD8LcKHG4OgO4icpoUOByFBVKJY9UJxTlrgsKfpA1EpMlOj3b9GUBPEflG0ef/AXizuGDL\nfR5SPEbjot1ERBoDQAhhA4AnAdxRzIcOQOHOdmy5z6MuEZGvFK+zWTE/eyCAURU61nEiso+ItAfw\nWwDjQggLK3Gs+kDucWGvDujF50Qn7cZlCAqCxyoURLvPAdxQ4rEmi8gPvZ+FEBYAuBqFPNkGAFMA\n/AEV+hKWg4S5+zGArQBuQeGpka3FMYQQVgH4BoA7AawDcDyAS0o8j67F53p3JYoeVjz2myj8ctmK\n+ImX76OQdlkF4L8BXF2JXyzlJGHuvwNgGQq59DMAnBVC2F7isXa5bos8gEIOen7xeN8v5Tj1if/J\ncUGKz0sSQgjZy9mr79AJIYT8EwZ0QgjJBAZ0QgjJBAZ0QgjJhDqtFO3Vq5dRYL/88svInjfPPkJ8\n4oknRnaTJk2MzxFHHBHZxx13nPHZZ5/499eWLVuMz7Zt28zYp59+GtmPPfaY8Vmzxmt5EtOgQYPI\nbtq0qfFZtmyZ1PhGDr179zZzu2PHjt3agJ1/bQPAYYcdFtlXXWUbxm3evDmyx40bZ3xmzZpV4/E9\nkT5FuNdz662RJUuWlDS3APD1r3/dnMTnn38e2V98YZpBmjHPZ/v2+AGWbt26GZ8hQ4ZEdr9+/YxP\no0aNzJhe494a0HP33nvvGZ+RI+PGmN46mTNnTknz27Nnzxo/4BYtbCPFfffdN7LbtGljfN5+++3I\n1jHAGzv00EONT5cutplo586dI1t/joCNCwsWLDA+b775ZmR7n+Mnn3ySNLe8QyeEkExgQCeEkExg\nQCeEkExgQCeEkEyoU1HUE6q04Hj33bbz5fXXX7/b1wDA1q1xo7WPPvrI+Lz7blwR/uGHHxqfhg3t\nlGgB6qKLLjI+ixcvjuy//MW2V/7ss88iW6Rkjc6QMreeIKcFmL59+xqfww8/PLL/9jfbikJfW7Nm\ntgPrunXrzJgWPEsVRRs3bhzZ5ZxbAGjZ0nbkXbo07oLqiWJaZD/wwAONz9VXXx3ZHTp0MD5Tp8bd\nc5944gnj8+qrtn+XFkE9MbNr166R3b273afklltuiew33njD+JSKFjcB4Oabb45sT1DX869jAGDX\nriec6s/WezBCC+AA8P7770e29/3S8aRTp07GZ+HCuHWOfsBgT+AdOiGEZAIDOiGEZAIDOiGEZELV\nt6Br1apVZHu5T52f9YojHnnkkchesWKF8dFFFqmdJhctWhTZOi8HAGeeeWZkH3TQQcZn4sSJkV2b\nXFkKOl/qFTJ985vfjGxvTp599tnI9oosdDGGLvQCgMGDB5sxnRvdsGGD8dH5S70eAFsc4mkh5Ubn\nTL08q9YkdC4aAJ5++ukaffRaSdUa9PfJy/PqwpaZM2can2eeibfovPXWW41POdGa0OzZs42P/l6m\nFM5prQWwa1drbQCw//77m7HWreOdEz0toH379rt9DQD06tUrsl977TXjkwrv0AkhJBMY0AkhJBMY\n0AkhJBMY0AkhJBOqLoredtttke2JmbpApnnz5sanf//+ke11+/Nel4IWVrxudH//+98je9CgQcZH\nC5Be8VE50QLQBRdcYHzWr18f2VoABYCNGzdG9pIlS4yPLmrRnegA4MILLzRjvXv3jmwtHAO2Y54n\n/ulrrQtRVItwXvHRNddcE9k/+9nPjM/LL78c2V5HwJSHB7x50WvXK5rRPkcffbTx0QU5nrhXTubM\nmRPZAwcOND5aQPc6nuq59AqrdIGhJ5x6DzDo74UXX/TDAatWrTI+M2bMiGzv4YVUeIdOCCGZwIBO\nCCGZwIBOCCGZUPUcus7p3XvvvcZn06ZNkX3TTTcZnz59+kT23Llzjc/HH39c4/l4eUidK/UKSHSO\nzWtEdcwxx0T2iBEjajyf2jB06NDI1nlYAPj1r38d2V6u0MvpavS8aU0BAB588EEzNmDAgMg+/fTT\njY8u/NCNqgBbMFMXOXSdjx02bJjx0WvVWxcp56p3FfJy6Ck7Jnnr+1vf+lZk63UD2GKbsWPHGh+t\nEdWGUaNGRbZXqPe1r30tslevXm18dBzQDb0AO5fe3Hq5d53r9nQz/X3S+XKgvEVxvEMnhJBMYEAn\nhJBMYEAnhJBMYEAnhJBMqLoo+tvf/jayvQfvtZAzZcoU43PuuedGttfZ7+GHH45sXVQD+B3bdHc/\nTxTVndYOOeQQ46M7tulObOVGF/eMGTPG+GjB2UPPf0pRS4q4DNhCIm/nJf3ZeuLbc889F9meQFhu\ntHjoFYqtXbs2slOKf1JEaE+k87pQ6ocObrjhBuNz4oknRrZXOKe/O15nwXKir093VgTsrmRHHXWU\n8dFdUL2dzHRRnO7KCvifiRbwvQcKXnnlFTOmKWdRHO/QCSEkExjQCSEkExjQCSEkE6qeQ583b15k\np+QYp02bZnx08yBdxAMA3bp1i+zp06cbHy8/ro+vizwAm7/zdk7XhQ8//elPjY/eAbw26KZmXh5S\nX1vqTjg1+Xj5ci+vrf08n0mTJkX2FVdcYXwuvfTSyP7zn/+865MtE8cee2xkjx492vjo6/Ny396Y\nRs+vVySmi7QA4Pjjj49sb13qhni//OUvjY/OmVe6cCtFk9FrRe+8BNjvk/7MANs07oMPPijpHL2d\nhnRevVGjRsYnpWgsFd6hE0JIJjCgE0JIJjCgE0JIJjCgE0JIJlRdFE0RALT44IlIWlz1igwOPvjg\nyPa69qV0VTvvvPOMjy7O8Lq66Q51tdmZJIWUgqgU8SlFONU+KeIyYAUhT2zTwqInit94442R7e3O\nVG5OPfXUyNYdPwFg5syZkf2Vr3zF+OhrbteunfHRRWje2unUqZMZ08Vlnij3+9//PrI9YVoXfNVG\nuCsXKWtXF855a2f+/PmR7e2O5F2v7iiqd1AC0gRPXbSUUli2K3iHTgghmcCATgghmcCATgghmcCA\nTgghmVB1UTQFLSR4osHy5csj29vqq2PHjpHdpUsX4+MJd3379o1srwp18uTJke1Vjelqu5QKwdqQ\nIq6kbL+l8apAUzoyeoKnFo28KlzN4sWLzZjuolfO7dB2hRa5hw8fbnyuu+66yNbrFLAVvF5F7+OP\nPx7ZK1euND66cylgO3zeeeedxueNN96IbE84TXkwoZykrMOUuKDP2/PR4rbXNVFvwQcAJ510UmQf\neOCBxufFF1+M7I0bNxofPZe1mVveoRNCSCYwoBNCSCYwoBNCSCZUPYeeki9KyQXrh/q9XUd0R8Yr\nr7zS+OjdQ7z3euaZZ4yPLmxKyeeldDGsDdu3b6/Rp5RcpfcaPeblYb05KaVAxZs3vTOPp6HofHJt\nue+++yI7pXDLQxfyeDsPaZ/LLrvM+HjFdK+//vpubcB+Bt53Uusmno5STlLiQsq61LrV2WefbXz0\n5/byyy8bHy+vvmzZssgeNGiQ8TnnnHMi29vBaM2aNZFdm7jAO3RCCMkEBnRCCMkEBnRCCMkEBnRC\nCMmEqouiWuzxxBAtpnmige4+54ly2scrdPG2pdMd87Zu3VrjOZbSRbLSeIKvFuBK3YJOFwSVSwD1\n8K5Di19eV72LLrqoLMf/B1pM84RCPS8pnSo9dAdGveUeYIVhwBYbVbrbX7lI2bov5Ty9IkCN3uIw\n9Vg6dj333HPGR3dhPf/8842PLux6//33d32yNVD9T44QQkhZYEAnhJBMYEAnhJBMqHoOXechvRxj\nSsOmAQMGRLZuxOW9z3vvvWd8vKICnSvz8pApzalKaYRVG3Qesnnz5sbH0wM0KddWSiOwVPTx+/fv\nb3xatGgR2b/5zW+MT7lz6Dpn7a1dreWkaBTeeh82bFhkd+jQwfiMGTPGjHk7Z2lSmlylfAfLibdr\nUim0bds2shcuXGh8UtZ3Ct5nq/W3/fbbz/icfvrpNb5PKrxDJ4SQTGBAJ4SQTGBAJ4SQTGBAJ4SQ\nTKi6KKo7AnqCgBbzhg4dany6desW2V7xiRZaZsyYYXxK7ZhXirBUaaFJd9bT4gsAjB8/PrK9Tn+l\n7AyTSsrr+vXrF9meKDpx4sTI9nb9KTdaUPYKi3TBU8q60EIeAAwZMiSyveKXxx57zIyldEVMKQrT\nPl7hXjkp5WEJ77y1KNy1a1fjM3/+/Mgu525M+rvz5ptvGp+TTz45su+///6Sj8c7dEIIyQQGdEII\nyQQGdEIIyYSq59B1cYb3UP8ll1wS2QcccIDx0bsKbdq0yfjoMW/ndC+HnNK8SDf68nKM2qfSTZBm\nzZoV2V4xit5RZerUqcZn/fr1ke3lGFOKMVq3bm3GOnXqFNk6Xw7YIrGnnnrK+OicuaehlBu9g7un\nv+jz8NaFHvN2I1q9enVke3PgrXm9xryGdHpM5/0Bex2V1n+0tpbS+Mw7p7lz50Z2586djc+RRx4Z\n2W+99VbSOZaim+nvEmCbBh5xxBFJx/fgHTohhGQCAzohhGQCAzohhGQCAzohhGRC1UVRLSToAiHA\nFhZpgQhIEyief/75Gt8nRcz0BDf9uhTxqdKiqBaK9fUDtthIi6QAsHbt2sj25k3PycEHH2x8PFFW\nz5vXHfDRRx+NbE/8q+u5BazgtWHDBuOjz9U7L/0Z9OzZ0/jorn2jR482Pp4wrQVOT/Bs0qRJZHvr\nu65FUT1P3m5M+gEG7/p18ZcupAOAgQMHRnafPn2Mz7Jly8yYFm69edPdFfUDHoD9rtx+++3G59Zb\nbzVjHrxDJ4SQTGBAJ4SQTGBAJ4SQTKh6Dl0Xm3g7lbRs2TKyvTyk3jl71apVxkfnwbzdQ1Jy6F5+\nXOcUU3b1adasmfEpJ/r6vBzv5MmTI9sratA7p/ft29f46Gtp1aqV8fGKlj788MPIXr58ufHRc+nl\nKut6bgGrCehdkwBbEOMVyOhmc95O9T/60Y9qPJaXH9dj3tyV0pxr//33Nz7lRF+fd216Lr0GXnrt\neEVx06dPj2xd7AYAZ511lhnTDdO8XdJ0zNHfNwAYMWJEZHvXwRw6IYT8D4MBnRBCMoEBnRBCMoEB\nnRBCMkFK3WmGEEJI/YJ36IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM\n6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQ\nkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM\n6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgkM6IQQkgn1PqCLyGIR2Soim0RkhYiMEpEWia8dLiKvi8h2\nERm1G7+RxfffJCLbRGTHTva8xGM9LCLviciXInJ54uVVlVrO7VgRWS4iG4rX/a+78KvV3IrIv4jI\nDBFZKyKfFv9/0p5ea13DdVtZajm/m9S/HSJyv+NX27V7mnOsICIXlXLNSYQQ6vU/AIsBnFn8/0EA\n3gZwd+JrhwAYDOAhAKMSX3MlgFdKOM/hAAYBmAPg8mrPWx3MbU8AzYr/7w5gOYDjyj23AJoCOApA\nQwAC4BsAVgFoUO35q+Dcct1WcH7V+7QAsAnAgErMr3qPMwGsB9C0UvPSMDHu1wtCCJ+IyCQUgkmK\n/3gAEJG+ADpV+NweKB5reyWPUylKmNu3dzaL/w4D8Ncyn9dWAPMBQET2AfAlgHYAWgFYW85jVQqu\n28qyp/Or+AaAlQBeLu9ZuQwD8HhxTVeEep9y2RkR6QzgfBTuJiAit4jIM3V07EkiMqIujlUNSplb\nEfm1iGwB8C6AZQAmlnjsGue2+CfudgDjAfxnCGGvCOYA122lqeX8DgMwJhRvoUs4dtL8ikhLFP7y\nGl3KcVLZW+7QJ4jIFyj8ufIsgLsAIIRwd12dQAjhvLo6Vh1T8tyGEK4TkR8AOAnAaSgE3D0mZW5D\nCD1EpAkKd1R7y40I121lqdX8ikgXAAMBXFXqCezB/F4MYGkI4ZVSj5XC3hLQB4cQnq/2SWRKreY2\nhLADwCtFQe1/AfiPsp2ZPdY2AI8WRbw3QghJwl8V4bqtLLWd3++gkBdfVK4T2g3DAIyp9EH2ljsd\nUv9piEIOvS5oDKBrHR2L5MsVqHAKBABE5BAAp4ABvXaISMPin+kNADQQkSYiUpG/SkSkcfFYAqBR\n8VhSiWNVGxFpLyKXiEgLEWkgIucAuBTACxU41kki0l9EGolIUxEZCaANgNfKfaz6Atdt5RGRk1F4\nOuaJOjjcFQCmhRA+qvSB9uqAXnxOdNJuXH4MYCuAWwBcXvz/j0s81mQR+eFuXKYU3/94AP9V/H//\nUo5VH6hhbgMK6ZUlANYB+AWA/xNCeKrEY+1ubpui8PjeOgCfADgLwPkhhOWlHKs+wHVbWRLmFyik\nQMaHEDbW8lg1zS9QR38JAICUKO4SQgipZ+zVd+iEEEL+CQM6IYRkAgM6IYRkAgM6IYRkQp0WFp16\n6qlGgb388rjB27XXXmtet2hR/Nz/ypUrjc+KFSsi+4kn7NNIb7zxRmRv3rzZ+OzYsaPGsVKF5AYN\nGkR206ZNjc/ChQtLemTswgsvNCf1+eefR/YXX3xhXpfi06NHj8jeZx97H7D//vtHdufOnY3PJZdc\nYsaWLFkS2d6cNGnSJLL1egCA2bNnR/ZLL71kfJ5++umSH8dLmV9tA3Y+vfndvj0usP3ss8+Mjx7z\n3ifl8/XWbt++fSN76NChxuf+++NmhJs2bTI+b731Vp2u3ZS51e/Tr18/43PooYdG9pYtW4zP008/\nbcZ0HPryyy+NTwqNGjWK7BYtbNPIuXPnJs0t79AJISQTGNAJISQTGNAJISQTGNAJISQT6lQUbdas\nmRnT4ooWyQArPqxZs8b4aIFz4MCBxueEE06I7Ntuu834aIHKG/OEpRShdN99943scrbM8ISUpUuX\nRrYn+A4ePDiyzzjjjBrf2xON1q6N25Nv3GgrqidNstXYb731VmR786hF2OOPP974HHPMMZHds2cp\nex3smv3228+M6bXqrZ1t27bt1i6njyemaqGue/fuxufiiy+ObO9av/rVr0b288+Xr4mkt3aXLVsW\n2d61NW/ePLJPOeUU46NFUC3eA1Y49T5HT+TXcejFF180PtOnT4/sDRs2GB/9IEBt4gLv0AkhJBMY\n0AkhJBMY0AkhJBOqvmORzseuXr3a+Oh8rJfj0nlWnV8DbA77F7/4hfHRBSoAMH78+Mj2ctE6x+fl\nOHVhUcOGlZ1+nS+94447ajynv/7V7vGsiyqmTZtmfHS+ft26dcbHy4/rfKFXHKLzt977LF68OLLH\njRtnfM4991wzVhv0uaYUBHnFR/p6vHWh86x6Le8KvQZ++EPb6bVNmzaR7eV5jz766Mj2tK5youf2\nwAMPND733ntvZHtzq2PF66+/bnw+/fTTyPZy+r169TJjeu3272+7Dmtt6ZFHHjE+zz33XGQ3btzY\n+KTCO3RCCMkEBnRCCMkEBnRCCMkEBnRCCMmEqouiurDI6+KmRSPdfQ/wu/RptIjhPcDfvn17M6YL\nolKELU+400JWbcSPFPR5P/nkk8bnwQcfjGxPzE1Bi9Jt27Y1PocffrgZ0wVAZ599tvHRotUf//hH\n4/PKK69EtlccU260OO5129NrTHfW88ZSxGMPLW4CwMMPPxzZXnGfFrS9damLZmbOnFnj+dQGfS0P\nPfSQ8dHi8cKFC43PH/7wh8jWAqSHflAAAA444AAzpovw+vTpY3z0fF9xxRXG53vf+15kz58/v8Zz\n3BW8QyeEkExgQCeEkExgQCeEkEyoeg5d52y94gyd0/JyXDrv6OUcdW7Qe59WrVqZMZ1X1zlH7/he\nHlLn/r18ajnRjYFeeOEF46PzvqUWOw0aNCiyf/CDHxifbt26mbE//elPkT127Fjj89RTT0W2V1im\nc+aezlJuUnbM0XhrTuOtXa1ReM2idL4YsOvSW7v6vFetWmV83n333ciu9PyOGTMmsr050Tnzm266\nyfjoIilv/vV7eztyeXMyevToyP7d735nfE477bTIvuyyy4yP1pv07mt7Au/QCSEkExjQCSEkExjQ\nCSEkExjQCSEkE6ouiuqufIsWLTI+utOaJ2zoIg9P3NPih7fzjldAcMEFF0T2o48+any0uOsJnnUt\niqYUvqQUROnXeYLcyJEjI9sTRefMmWPGtm7dGtme4Kk/N6+Tpu6QVxeiaMouVdrH+wxS0ILy7bff\nbny8Bwq0wKZ39gKsCKiLtAArpno7/5QT/XCCJ0qOGDEisnUBGpD2sIS+fk/c9ooJ9Xc+Ze1686Yf\nuvA6Mg4bNsyMefAOnRBCMoEBnRBCMoEBnRBCMqHqOfQFCxZEtlf4cOmll0a21/hJ54u9XJn28XLo\nL730khmbMGFCZHs5fF1I5BUW6bx+pXcsSsmP6zlJKY7xdI6PP/44sq+55hrj8+qrr5qxSZMmRbbX\nYEk3NfPy49onpYCntqTkx/X8ertd6ffx8qzDhw+P7JYtWxofvWsTYPPq3tzpHbkmTpxofHThVqX1\nH71z2YwZM4yP3snMW7spOkfKd8DTJ7Sf19jv2muvjWzvs9WfW23iAu/QCSEkExjQCSEkExjQCSEk\nExjQCSEkE6ouih533HGRff755xsf/VC/J1poQcrz0Q/+e0UW3ut04YEnWmgRzuvYVhdC3c5oYTil\naChFOPXQXeR69+5tfLwdXe68887I9jrN3XLLLZHtzW1Kx7xqkFLcpcd+8pOfGJ8uXbpE9tq1a42P\nt3a137hx44zPrFmzIttb33U9n3pnn2XLlhkfXdiTMrfe+tbz5hUIed8B/X3Wu28BtuDNe1jC+0xK\npX6sekIIIbWGAZ0QQjKBAZ0QQjKBAZ0QQjKh6qKo7sD3wQcfGJ+LL744sjt06GB8tLDhVXbpMU/o\n8SrpUrau06RUDaZUsdUGLRSmdJrzfPSY13lOC8zTpk0zPrNnzzZjBx98cGSfeuqpxqd169a7PRZQ\nvq6Ge4I3Vxq9Vk444QTjo0VfLaQBwPr16yPbW7teF8opU6ZE9tSpU42Pvg5PuKuL+dwZ3V0x5WEF\nj5S1q68tZftKwFbLDhgwoMbj6+pWwFane1XAqfAOnRBCMoEBnRBCMoEBnRBCMqHqOXS9y4i3M4mm\nWbNmZkznVb38tM55ee/j5SF1oUVK7tRDn1NKwU5t0HlHb05S8pApnSx1rlh35wOAO+64w4z17ds3\nsu+77z7jo/PHKTtWVXpuvWN4eeauXbtG9j333GN8tG6jOw0CNofraU2jRo0yY/PmzYvsUvUfPVbp\n+U3Rl/Ta9day/g6krF1vjrxiq44dO0b2SSedtOuTLfL444+bMT23tdEreIdOCCGZwIBOCCGZwIBO\nCCGZwIBrRcABAAAFC0lEQVROCCGZUHVRVD/o74ktWsz0RAPdec0TVbT45G2jdcwxx5ixww47LLLH\njh1rfJYvX27Gqk3K3Gqxp9Rip9NPPz2yb7jhhqTX3XXXXZHtFSSlFEhp6qIQRheqecd86qmnItsT\n1/Ta8d5Hi3tPPvmk8Zk/f74ZK6VLoveaui7c0h0PveOlXFvKetYiqPcZeWvuyCOPjGxve7l169ZF\n9gsvvFDj+dSm4JB36IQQkgkM6IQQkgkM6IQQkglVz6F7TbRq8tGFJoDNDzdt2tT46FyZl6vyCmJ0\ncygv956yY05d76qzdevWGn1S8nW9evWK7Ouvv974HHLIIZE9ZswY46PzyYBtVlRqA7GU4pBys2XL\nlsi+8cYbjY/O/XqFc1rr8K7v4YcfjuwZM2YYn1LXkz7HlN22Kr12tSbm4eW6NXp9e+si5VratWtn\nxq666qoaX6d3g1qzZo3x0dpebeaWd+iEEJIJDOiEEJIJDOiEEJIJDOiEEJIJVRdFUwoItJDgdUnU\nIqgnmOjijNTiCC249ujRw/joogKvyEOfU12Lol5hkd4dZciQIcZn+PDhkT1z5kzjM3LkyMheuXJl\n8nnWhJ4n77Ota9EOsGKWt5PWxx9/HNmeoK53KFq2bJnxefbZZ2s8n5T17IngWoT15lePVVp01qKo\nd226A6V33ikdGVPo06ePGWvbtm1k686xADB58uTITunkWJu55R06IYRkAgM6IYRkAgM6IYRkQtVz\n6DrP6+3u/fLLL0e2LmIBgH79+kW2l3PTuUIvV5Wy671uROUd77XXXjM+Op+6ZMkS41NOdNGOniPA\n5se9HOu1114b2XPnzjU+KUVTKWNejlmP7bvvvsZH51O99yk33/3udyNbF6ABtvGWzpcDaQVKmzZt\niuyU4iogbX713NWH+dXXq4uvUs9Jj6V8vz0fr/GW3iXNy6EvXLgwsrXuAqRpAanwDp0QQjKBAZ0Q\nQjKBAZ0QQjKBAZ0QQjKh6qKoFm10oREATJo0KbIPOugg49OxY8fI9oSFFHGtefPmZqxVq1aRfcAB\nBxgfXQjx6quvGh8tOKbsvFMbbr755sj2Okn+/Oc/j2zdHc5DiziAFXs88cfrgKn9vM9E+3jH159t\nXXRbPPnkkyNb704DWMHzgw8+MD6/+tWvIvudd94xPnruvPn1RMGUz0XPZ8p7V3p+9fd3w4YNxkeL\nkp7ort8nZY68NajjC2CL8iZMmGB8tAiesnbZbZEQQggDOiGE5AIDOiGEZELVc+j6gX2v8ZZuKuXl\n2R966KHIHjBggPE59thjd/sawM+x6Zxayo5FXoGU9vFy2uXkrLPOiuxvf/vbxkfnIT19QOdLvTxg\nSh7Wy03qMe+9U5qa6XP0inzKzYIFCyJb7+wEAB999FFkP/DAA8ZnxYoVke01+dLz5OkRKdqG55Oy\n25Mea9OmjfEpJ+3bt49sryBLf8e85nP6vD1tTc+tt3a9nYb07lNLly41Pvq8K712eYdOCCGZwIBO\nCCGZwIBOCCGZwIBOCCGZIF53PUIIIXsfvEMnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAn\nhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBM\nYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBMYEAnhJBM+P/CVjpyohAc\n3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11acdffd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Variable for the location of subplot\n",
    "i = 1\n",
    "\n",
    "\n",
    "# Plotting\n",
    "for index in range(8):\n",
    "    img = test_std[index].reshape(16, 16)\n",
    "    rotated_img = sp.ndimage.rotate(img, 90)\n",
    "    plt.subplot(2, 4, i)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(rotated_img, cmap=plt.cm.gray, origin='lower')\n",
    "    plt.title('P: ' + str(predicted[index]) + ' T: ' + str(label_test[index]))\n",
    "    i = i + 1\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Perform Linear PCA, and SVC with Linear Kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8000, 197)\n",
      "(3000, 197)\n"
     ]
    }
   ],
   "source": [
    "dimTrain = train.shape\n",
    "dimTest = test.shape\n",
    "\n",
    "\n",
    "# Concatenate two matrix in order to perform PCA and Kernel PCA later\n",
    "comb = np.concatenate((train, test), axis=0)\n",
    "\n",
    "\n",
    "# Apply Scaling to X_train and X_test\n",
    "# Standardize features by removing the mean and scaling to unit variance\n",
    "std_scale = preprocessing.StandardScaler().fit(comb)\n",
    "comb_std = std_scale.transform(comb)\n",
    "\n",
    "\n",
    "# Perform PCA\n",
    "pca = PCA(n_components=256).fit(comb_std)\n",
    "comb_pca = pca.transform(comb_std)\n",
    "\n",
    "\n",
    "ratio = 0\n",
    "count = 0\n",
    "for i in pca.explained_variance_ratio_:\n",
    "    ratio += i\n",
    "    count +=1\n",
    "    if ratio >= 0.99:\n",
    "        p = count\n",
    "        break\n",
    "        \n",
    "\n",
    "new_train = comb_pca[0:dimTrain[0],0:p]\n",
    "new_test = comb_pca[dimTrain[0]:dimTrain[0]+dimTest[0],0:p]\n",
    "print(new_train.shape)\n",
    "print(new_test.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Poly SVC with degree=3 Polynomial Kernel\n",
    "clf = svm.SVC(kernel='linear')\n",
    "clf.fit(new_train, label_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for classifier SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
      "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',\n",
      "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
      "  tol=0.001, verbose=False):\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.92      0.99      0.95       300\n",
      "          2       0.91      0.94      0.93       300\n",
      "          3       0.90      0.93      0.92       300\n",
      "          4       0.93      0.97      0.95       300\n",
      "          5       0.95      0.89      0.92       300\n",
      "          6       0.98      0.95      0.96       300\n",
      "          7       0.97      0.96      0.97       300\n",
      "          8       0.94      0.90      0.92       300\n",
      "          9       0.96      0.94      0.95       300\n",
      "         10       0.98      0.96      0.97       300\n",
      "\n",
      "avg / total       0.94      0.94      0.94      3000\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Now predict on the test set\n",
    "expected = label_test\n",
    "predicted = clf.predict(new_test)\n",
    "\n",
    "\n",
    "print(\"Classification report for classifier %s:\\n%s\\n\"\n",
    "      % (clf, metrics.classification_report(expected, predicted)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform KernelPCA and with SVC Linear Kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Perform Kernel PCA with Polynomial Kernel \n",
    "kpca = KernelPCA(n_components=1000, kernel='poly', degree=3, fit_inverse_transform=True).fit(comb_std)\n",
    "comb_kpca = kpca.transform(comb_std)\n",
    "\n",
    "\n",
    "lamda = kpca.lambdas_\n",
    "eigVar = 0\n",
    "count = 0\n",
    "for eigen in lamda:\n",
    "    count += 1;\n",
    "    eigVar += eigen / sum(lamda)\n",
    "    if eigVar >= 0.99:\n",
    "        p = count\n",
    "        break\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8000, 922)\n",
      "(3000, 922)\n"
     ]
    }
   ],
   "source": [
    "new_train = comb_kpca[0:dimTrain[0],0:p]\n",
    "new_test = comb_kpca[dimTrain[0]:dimTrain[0]+dimTest[0],0:p]\n",
    "print(new_train.shape)\n",
    "print(new_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for classifier SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
      "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',\n",
      "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
      "  tol=0.001, verbose=False):\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.97      0.99      0.98       300\n",
      "          2       0.94      0.97      0.96       300\n",
      "          3       0.97      0.97      0.97       300\n",
      "          4       0.97      0.99      0.98       300\n",
      "          5       0.98      0.96      0.97       300\n",
      "          6       0.99      0.96      0.98       300\n",
      "          7       0.99      0.98      0.98       300\n",
      "          8       0.94      0.95      0.94       300\n",
      "          9       0.98      0.98      0.98       300\n",
      "         10       1.00      0.98      0.99       300\n",
      "\n",
      "avg / total       0.97      0.97      0.97      3000\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Perform SVC with linear Kernel\n",
    "clf = svm.SVC(kernel='linear') \n",
    "clf.fit(new_train, label_train)\n",
    "\n",
    "\n",
    "# Now predict the value of the digit on the test data:\n",
    "expected = label_test\n",
    "predicted = clf.predict(new_test)\n",
    "\n",
    "\n",
    "print(\"Classification report for classifier %s:\\n%s\\n\"\n",
    "      % (clf, metrics.classification_report(label_test, predicted)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Below are the summary of precision and recall got by SVC with Linear Kernel\n",
    "\n",
    "|                             | precision | recall | f1-score | support |\n",
    "|-----------------------------|-----------|--------|----------|---------|\n",
    "| Original images             | 0.94      | 0.94   | 0.94     | 3000    |\n",
    "| Linear PCA                  | 0.94      | 0.94   | 0.94     | 3000    |\n",
    "| Kernel PCA                  | 0.97      | 0.97   | 0.97     | 3000    |\n",
    "\n",
    "\n",
    "#### Precision :   True positives / Predicted positives   notation : P\n",
    "#### Recall:         True positives / Actual positives      notation : R\n",
    "#### F1-score:     $\\frac{2PR}{P+R}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}