text_classification_pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "source": [
  
    "## Spark Pipelines and Evaluation of Scaling of Algorithms\n",
    "\n",
    "### by Shakeel Raja, thanks to my buddy Nalan\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
 
    "## Task A - DATASET (20 NEWSGROUP)\n",
    "\n",
    "The dataset selected for this experiment is “20 Newsgroup Dataset” [1], which was originally created by Ken Lang for development of “NewsWeeder”, a system for learning user profiles based on their news article ratings [2]. The dataset is compiled from 20 different newsgroups, containing approximately one thousand documents for each newsgroup. This results as an offering of a total of  roughly 20,000 text documents from a variety of domains. The dataset is balanced having similar number of document offerings for each domain (i.e. approx. 1000). Some of these domains are very similar to each other i.e. “comp.sys.ibm.pc.hardware” and “comp.sys.mac.hardware’, while some domains are highly unrelated e.g. “sci.electronics” and “talk.religion.misc”. The variety, size, shape and presence of a ground truth makes  “20 Newsgroup Dataset”  an ideal candidate for practicing supervised multiclass text classification and other similar tasks within the big data domain.  Following table shows including newsgroups domains, grouped according to topic similarity  \n",
    "![Dataset Description](https://iscientiaqueueqa.blob.core.windows.net/samplecontainer/Capture.PNG)\n",
    "\n",
    "### DATA CLEANING\n",
    "\n",
    "Documents available in the newsgroup dataset carry headers with text information that may not be suitable or useful for analytical experimentation. Part of headers contained information generated as a part of communication protocols (i.e. IP addresses, email addresses, document locations etc. ) which would not have provided any insight into the real content of the document. The topic of document was also present within the header as text information in most cases, as shown the example below. This would have led to an artificially high accuracy rate. \n",
    "It was hence decided to remove this information from the documents prior to carrying out any further pre-processing and analysis tasks. This was achieved by iterating through all the documents in the corpus, identifying the location where the header terminates and removing the part of document upto that point. The resulting data frames were then subjected to further experiment. \n",
    "\n",
    "#### An example Header \n",
    "\n",
    ">Xref: cantaloupe.srv.cs.cmu.edu alt.atheism:49960 alt.atheism.moderated:713 news.answers:7054 alt.answers:126\n",
    "Path: cantaloupe.srv.cs.cmu.edu!crabapple.srv.cs.cmu.edu!bb3.andrew.cmu.edu!news.sei.cmu.edu!cis.ohio-state.edu!magnus.acs.ohio-state.edu!usenet.ins.cwru.edu!agate!spool.mu.edu!uunet!pipex!ibmpcug!mantis!mathew\n",
    "From: mathew <mathew@mantis.co.uk>\n",
    "Newsgroups: alt.atheism,alt.atheism.moderated,news.answers,alt.answers\n",
    "Subject: Alt.Atheism FAQ: Atheist Resources\n",
    "Summary: Books, addresses, music -- anything related to atheism\n",
    "Keywords: FAQ, atheism, books, music, fiction, addresses, contacts\n",
    "Message-ID: <19930329115719@mantis.co.uk>\n",
    "Date: Mon, 29 Mar 1993 11:57:19 GMT\n",
    "Expires: Thu, 29 Apr 1993 11:57:19 GMT\n",
    "Followup-To: alt.atheism\n",
    "Distribution: world\n",
    "Organization: Mantis Consultants, Cambridge. UK.\n",
    "Approved: news-answers-request@mit.edu\n",
    "Supersedes: <19930301143317@mantis.co.uk>\n",
    "Lines: 290\n",
    "\n",
    "### TEXT CLASSIFICATION\n",
    "For this experiment, the selected dataset was subjected to text classification task. Text classification can be described predicting a topic for given text document as below:\n",
    "\n",
    "\n",
    "### INPUT : \n",
    "#### A document ‘d’\n",
    "#### A  predefined set of ‘j’ classes  C = {c1,c2,…,cj}\n",
    "\n",
    "### OUTPUT: \n",
    "####  A predicted class c ∈ C\n",
    "\n",
    "Text classification was chosen as an experiment of choice because it provides conceptual views of text corpuses and has various applications in the real world. E.g. organizing news stories by topic or location, research publications are classified by domains and subdomains, health-care systems often index patient reports from multiple views, taxonomies of disease classification and medical procedures, client document management for businesses and spam filtering etc. \n",
    "\n",
    "### BACKGROUND\n",
    "Traditional AI based computing approach to automatic text classification involved manually defining a set of orthography rules [3] for entity recognition from given text but this proved to be time-consuming and expensive. For example, To label a sequence of tokens in the format \n",
    "“Mr. X”, where ‘X’ is a capitalized word representing the ‘Person’ entity, an expert may define following rule for identify the person: \n",
    "\n",
    "#### (token = “Mr.” orthography type = FirstCap) -> Person’s Name \n",
    "\n",
    "\n",
    "Here, the regular expression on the left side of rule identifies a given sequence of two tokens, where the first token “Mr.” is followed by orthography type “FirstCap” and the decision on right identifies the token as a person’s name.  Manually creating such rules for classification oriented entity recognition may achieve high accuracy with carefully refined rules but requires human expertise and is highly labour intensive [4].\n",
    "\n",
    "### STATISTICAL MACHINE LEARNING: \n",
    "\n",
    "Statistical tasks treat categorization problems as “ Sequence Labelling Problems”. For a given sequence of observations in the text document, expressed as x = (x1,x2, … , xn), each observation is represented using a feature vector. The task requires assignment of a class label ci to each observation xi, considering that ci depends not only on xi but also on other observations and other labels within that sequence. To achieve this goal, each word in a sentence is treated as an observation. \n",
    "A text classification problem requires the presence of a corpus with annotated document-label relations as positive training examples. Statistical machine learning algorithms are then used for training classifiers i.e. logistic regression, Naive Bayes etc. [4]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Task B - Machine Learning with Pipelines in Spark \n",
    "\n",
    "Apache Spark offers Pipelines to facilitate the creation, tuning, and testing of practical machine learning algorithm along with deployment. Pipelines offer functionality to define and monitor the entire data extraction, transformation, and algorithm training workflow. The data cleaning step (header removal) was performed at the dataset and resulting documents were labelled according to the topic name. Finally, a single dataframe with document text as rows and topic labels was generated for further experimentation. These steps and the resulting output has been shown in the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTopic Name \t\t  No of Items\n",
      "comp.sys.mac.hardware\t\t\t\t1000\n",
      "talk.politics.guns\t\t\t\t998\n",
      "rec.sport.hockey\t\t\t\t1000\n",
      "alt.atheism\t\t\t\t1000\n",
      "misc.forsale\t\t\t\t1000\n",
      "sci.space\t\t\t\t1000\n",
      "comp.sys.ibm.pc.hardware\t\t\t\t1000\n",
      "rec.autos\t\t\t\t1000\n",
      "sci.med\t\t\t\t999\n",
      "rec.sport.baseball\t\t\t\t1000\n",
      "soc.religion.christian\t\t\t\t997\n",
      "rec.motorcycles\t\t\t\t1000\n",
      "sci.crypt\t\t\t\t1000\n",
      "talk.politics.mideast\t\t\t\t998\n",
      "comp.windows.x\t\t\t\t1000\n",
      "talk.politics.misc\t\t\t\t996\n",
      "sci.electronics\t\t\t\t1000\n",
      "talk.religion.misc\t\t\t\t998\n",
      "comp.graphics\t\t\t\t1000\n",
      "comp.os.ms-windows.misc\t\t\t\t1000\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml import *\n",
    "from pyspark.ml.classification import *\n",
    "from pyspark.ml.feature import *\n",
    "from pyspark.ml.param import *\n",
    "from pyspark.ml.tuning import *\n",
    "from pyspark.ml.evaluation import *\n",
    "import os\n",
    "import re\n",
    "from operator import add\n",
    "\n",
    "#get the list of folders in the main directory\n",
    "topicList = os.listdir(\"/data/tempstore/newsgroups/\")\n",
    "#remove the non folder in the list\n",
    "topicList.remove('.DS_Store');\n",
    "\n",
    "#create a small tempory list for speedup the experiments\n",
    "#topicList = ['soc.religion.christian', 'rec.sport.baseball','comp.graphics'];\n",
    "\n",
    "#define the prefix to load the data folders\n",
    "prefix = '/data/tempstore/newsgroups/'\n",
    "\n",
    "# remove the headers, get the sender and the main text\n",
    "def parseMessage(ft): \n",
    "    \n",
    "    fn,text = ft # unpack the filename and text content   \n",
    "    #As we understand two newlines together is the most common pattern to remove the headers.\n",
    "    #almost every file header ends with two new lines. regex expressions for python gave errors and we have used normal string \n",
    "    #calculations here\n",
    "    #fist split the text by newline.\n",
    "    lines = text.split(\"\\n\"); \n",
    "    \n",
    "    #then look for the first empty item. because when 2 newlines together 2nd line dosen't contain anything \n",
    "    index = lines.index(\"\")\n",
    "    #safe check. if there are no 2 newlines \n",
    "    if(index < len(lines)):\n",
    "        #if there are then get the sub list of strings from that index\n",
    "        cleanLines = lines[index:len(lines)];\n",
    "        #joins the strings list together to make a one string \n",
    "        newText = \" \".join(cleanLines);\n",
    "        #returns file name and string\n",
    "        return (fn,newText)\n",
    "    else:\n",
    "        #if there are no new lines returns complete text\n",
    "        return (fn,text)\n",
    "    \n",
    "\n",
    "# we need a SparkSession to create DataFrames\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "\n",
    "# Make a DataFrame with labels and n-Gram vectors \n",
    "def make_dataFrame(dirPath, argLabel): \n",
    "    #create rdd from the given file group (directory)\n",
    "    ft_RDD = sc.wholeTextFiles(dirPath) \n",
    "    #parse through the remove header function. this returns a list of (fn,text)\n",
    "    ft2_RDD = ft_RDD.map(parseMessage)  \n",
    "    #rearrange the list to (text,label)\n",
    "    fl_RDD = ft2_RDD.map(lambda ft: (ft[1], argLabel))\n",
    "    \n",
    "    #create a DataFrame with two coloumns.\n",
    "    rows_DF = spark.createDataFrame(fl_RDD,schema=['text','label']) \n",
    "    \n",
    "    return rows_DF\n",
    "\n",
    "#create empty variable. we use this variable to hold the complete dataset as a dataframe. \n",
    "#we assign it to null initially\n",
    "rows_DF = None;\n",
    "\n",
    "#use the number to assign labels 0 - 19\n",
    "number = 0;\n",
    "#create a list to save the list of labels for generate charts\n",
    "labelsAndCounts = [];\n",
    "\n",
    "print(\"\\tTopic Name \\t\\t  No of Items\");\n",
    "\n",
    "for topic in topicList:\n",
    "    #make a dataframe for given topic and save it in a tempory variable\n",
    "    rows_temp_DF = make_dataFrame(prefix+topic, number)\n",
    "    \n",
    "    #get the count of files in the news category\n",
    "    count = rows_temp_DF.count()\n",
    " \n",
    "    labelsAndCounts.append((topic,count));\n",
    "    #just for printing purposes\n",
    "    print(\"{0}\\t\\t\\t\\t{1}\".format(topic,count))\n",
    "        \n",
    "    number = number + 1;\n",
    "    #main dataframe is not null except in the first case. we merge the tempory df to main df\n",
    "    if(rows_DF is not None):\n",
    "        rows_DF = rows_DF.union(rows_temp_DF)\n",
    "    else:\n",
    "        #first case main dataframe is null and just assgin tempory df to main df\n",
    "        rows_DF = rows_temp_DF;\n",
    "\n",
    "#create a view called newsgroups in the dataframe\n",
    "rows_DF.createOrReplaceTempView(\"newsgroups\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "For text classification, The end to end document classification pipeline was implemented in spark with Apache’s machine learning library, Mllib.  Creating the pipeline involved declaration of all the required stages, configuring their parameters and chaining them in a pipeline object. After the removal of noisy data as described earlier, the documents were randomly split into testing and training sets and loaded into data-frames. This allowed us to define and create a Pipeline object for text classification with stages as defined below:\n",
    "\n",
    "### Feature Transformers\n",
    "\n",
    "\n",
    "\n",
    "#### Tokenizer\n",
    "The ml tokenizer was used to convert text into words (terms) for creating vectors in later stages. \n",
    "\n",
    "#### Stop word removal\n",
    "Stop word removal stage removes such words (terms) from the document that appear often with equal probability in each document and hence do not help with classification tasks. The “StopWordRemover” transformer from machine learning library was supplied with a list of commonly occurring stopwords as an extra parameter, taken from John Bresnik’s Github repository [6]. \n",
    "\n",
    "#### Hashing TF\n",
    "Feature hashing, also known as the hashing trick, similar to the kernel trick, is a fast yet efficient way of vectorizing features. A raw feature is mapped into an index (term) by applying a hash function. Hashing  turns arbitrary features into indices in a vector or matrix by applying a hash function to the features and using their hash values as indices directly,instead of looking the indices up in an associative array.The hashingTF function in Spark Machine learning library was used for calculating hash vectors.  \n",
    "\n",
    "Following code contains transformer definitions for Spark pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline\n",
    "from pyspark.ml.classification import NaiveBayes\n",
    "from pyspark.ml.feature import HashingTF, Tokenizer, IDF\n",
    "from pyspark.ml.tuning import  ParamGridBuilder\n",
    "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\n",
    "\n",
    "rows_DF.persist()\n",
    "\n",
    "#create small dataframe from getting subset from first two groups for testing purposes\n",
    "test_small_DF = rows_DF.sampleBy( 'label', fractions={0:0.1,1:0.1}, seed=0 )\n",
    "\n",
    "#format and count of the test dataset\n",
    "#test_small_DF.groupBy('label').count().orderBy('label').show()\n",
    "\n",
    "#define the list of stopwords\n",
    "STOP_WORDS = [\n",
    "    'a','cannot','into','our','thus','about','co','is','ours','to','above',\n",
    "    'could','it','ourselves','together','across','down','its','out','too',\n",
    "    'after','during','itself','over','toward','afterwards','each','last','own',\n",
    "    'towards','again','eg','latter','per','under','against','either','latterly',\n",
    "    'perhaps','until','all','else','least','rather','up','almost','elsewhere',\n",
    "    'less','same','upon','alone','enough','ltd','seem','us','along','etc',\n",
    "    'many','seemed','very','already','even','may','seeming','via','also','ever',\n",
    "    'me','seems','was','although','every','meanwhile','several','we','always',\n",
    "    'everyone','might','she','well','among','everything','more','should','were',\n",
    "    'amongst','everywhere','moreover','since','what','an','except','most','so',\n",
    "    'whatever','and','few','mostly','some','when','another','first','much',\n",
    "    'somehow','whence','any','for','must','someone','whenever','anyhow',\n",
    "    'former','my','something','where','anyone','formerly','myself','sometime',\n",
    "    'whereafter','anything','from','namely','sometimes','whereas','anywhere',\n",
    "    'further','neither','somewhere','whereby','are','had','never','still',\n",
    "    'wherein','around','has','nevertheless','such','whereupon','as','have',\n",
    "    'next','than','wherever','at','he','no','that','whether','be','hence',\n",
    "    'nobody','the','whither','became','her','none','their','which','because',\n",
    "    'here','noone','them','while','become','hereafter','nor','themselves','who',\n",
    "    'becomes','hereby','not','then','whoever','becoming','herein','nothing',\n",
    "    'thence','whole','been','hereupon','now','there','whom','before','hers',\n",
    "    'nowhere','thereafter','whose','beforehand','herself','of','thereby','why',\n",
    "    'behind','him','off','therefore','will','being','himself','often','therein',\n",
    "    'with','below','his','on','thereupon','within','beside','how','once',\n",
    "    'these','without','besides','however','one','they','would','between','i',\n",
    "    'only','this','yet','beyond','ie','onto','those','you','both','if','or',\n",
    "    'though','your','but','in','other','through','yours','by','inc','others',\n",
    "    'throughout','yourself','can','indeed','otherwise','thru','yourselves',' ','',\"i'm\",\"I'm\"\n",
    "    ]\n",
    "\n",
    "\n",
    "#define the tokenizer transformer to split the text into words\n",
    "tokenizer = Tokenizer(inputCol=\"text\", outputCol=\"words\")\n",
    "#stop word remove transformer with list of stirngs to remove the stop words. we have used custom list of strings since we use\n",
    "#this to remove empty strings too\n",
    "remover = StopWordsRemover(inputCol=\"words\", outputCol=\"wordsFiltered\",stopWords = STOP_WORDS)\n",
    "#hashing transformer to crete hash out of words\n",
    "hashingTF = HashingTF(inputCol=remover.getOutputCol(), outputCol=\"features\", numFeatures = 2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### TEXT CLASSIFICATION WITH NAIVE BAYES\n",
    "\n",
    "Naive Bayes is one of the most efficient and effective inductive learning algorithms for machine learning and data mining[5]. Multinomial naive Bayes have been chosen as the supervised classification algorithm for the dataset described earlier. The naive Bayes classifier, based on simple (or naive) Bayes rule has long been a core technique in data mining and information retrieval tasks.  Applying Bayes rule to a document d and a class c forms equation\n",
    "\n",
    "#### P(c|d) = P(d|c)P(c) / P(d)\n",
    "\n",
    "The naive Bayes classifier combines this probability model model with the Maximum A Posteriori (MAP) decision rule to give a function that assigns a predicted label to a document as below:\n",
    "\n",
    "The classifier relies on a very simple representation of data called “ Bag of words” model which disregards grammar and even word order but keeps multiplicity. BoW representation involves creation of a vocabulary and a count of word frequency for each document. Bag of Words representation ignoring word order in a document is based upon the conditional independence property of naive Bayes classifier as mentioned in [5]. Further details of feature engineering including feature extraction, feature selection and supervised learning from the text corpus has been provided in the following section with brief descriptions and implementation in pySpark. \n",
    "\n",
    "![Naive Bayes Equation](https://iscientiaqueueqa.blob.core.windows.net/samplecontainer/naive_bayes_image.svg)\n",
    "\n",
    "#### Smoothing in Naive Bayes: \n",
    "\n",
    "With conditional independence assumption of naive Bayes, a classification task may provide a probability of zero to some observations i.e. a document containing words that do not appear in any other training documents.  The classifier will ignore the whole document because of a single rare word and give it zero probability due to point wise multiplication. The case where the count of some class is zero is just a particular case of overfitting. The reason for using a smoothing parameter with naive Bayes in is mainly to avoid such a scenario. \n",
    "\n",
    "#### Naive Bayes Classifier in Spark ML library\n",
    "Naive Bayes classifer from Spark ML library was defined as shown below with the smoothing parameter set to 1 as default value for testing the performance of resulting Pipeline. A multinomial variant of naive Bayes, as described above, was chosen to meet the requirements of multiclass text classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#define the naive bayes classifer as multinomial\n",
    "nb = NaiveBayes(smoothing=1.0, modelType=\"multinomial\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Create and Test The Pipeline\n",
    "Using above pre-processed data above, the Pipeline stages were individually tested on a small set of documents to evaluate the performance of different stages. Sample output from each stage has been shown below to see the affect of these transformers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input to the tokenizer \n",
      "\n",
      " In article <1993Apr22.055150.28215@nmt.edu>, changw@nmt.edu (willie* chang) writes: |> It happened to my classic ii once.  At startup, after the usual powerup |> tone, there were chimes (kinda like do(1st)-mi(1st)-sol(1st)-do(2nd) or |> 1st C - 1st E - 1st G - 2nd C on the music scale), and it hanged before |> giving the happy mac.  I had to re-powerup. |>  |> This was asked before but I can't remember the answer.  I've tried to |> find the answer from the faqs and other ftp sites but to no avail. |>  |> Can anyone tell me what those unusual chimes mean?  A spurious memory |> check error? |>  |> Thanks |> willie*  Those chimes indicate a hardware failure of some type during System startup.  One thing that can cause the chimes is a cable that has become a bit loose. Next time you power down the computer, check that all cables are on tight. \n"
     ]
    }
   ],
   "source": [
    "# Configure an ML pipeline, which consists of four stages: tokenizer, StopWordsRemover, hashingTF, and naive bayes.\n",
    "#we have used only naive bayes since this is one of the best algorithem to mulivariate text classification\n",
    "#create the piple line with all our transformers and naive bayes estimator.\n",
    "pipeline = Pipeline(stages=[tokenizer,remover, hashingTF, nb])\n",
    "\n",
    "#define the multinomial classifiaction since our perdiction has 20 classes\n",
    "evaluator = MulticlassClassificationEvaluator(labelCol=\"label\", predictionCol=\"prediction\",\n",
    "                                              metricName=\"accuracy\")\n",
    "\n",
    "\n",
    "#Test of pipeline \n",
    "#tokenizer is a transformer and check the tokenizer functionality\n",
    "print(\"input to the tokenizer \\n\")\n",
    "print(test_small_DF.take(1)[0][0]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### TOKENIZER TEST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output from the tokenizer and input to the stopword remover \n",
      "\n",
      "['', 'in', 'article', '<1993apr22.055150.28215@nmt.edu>,', 'changw@nmt.edu', '(willie*', 'chang)', 'writes:', '|>', 'it', 'happened', 'to', 'my', 'classic', 'ii', 'once.', '', 'at', 'startup,', 'after', 'the', 'usual', 'powerup', '|>', 'tone,', 'there', 'were', 'chimes', '(kinda', 'like', 'do(1st)-mi(1st)-sol(1st)-do(2nd)', 'or', '|>', '1st', 'c', '-', '1st', 'e', '-', '1st', 'g', '-', '2nd', 'c', 'on', 'the', 'music', 'scale),', 'and', 'it', 'hanged', 'before', '|>', 'giving', 'the', 'happy', 'mac.', '', 'i', 'had', 'to', 're-powerup.', '|>', '', '|>', 'this', 'was', 'asked', 'before', 'but', 'i', \"can't\", 'remember', 'the', 'answer.', '', \"i've\", 'tried', 'to', '|>', 'find', 'the', 'answer', 'from', 'the', 'faqs', 'and', 'other', 'ftp', 'sites', 'but', 'to', 'no', 'avail.', '|>', '', '|>', 'can', 'anyone', 'tell', 'me', 'what', 'those', 'unusual', 'chimes', 'mean?', '', 'a', 'spurious', 'memory', '|>', 'check', 'error?', '|>', '', '|>', 'thanks', '|>', 'willie*', '', 'those', 'chimes', 'indicate', 'a', 'hardware', 'failure', 'of', 'some', 'type', 'during', 'system', 'startup.', '', 'one', 'thing', 'that', 'can', 'cause', 'the', 'chimes', 'is', 'a', 'cable', 'that', 'has', 'become', 'a', 'bit', 'loose.', 'next', 'time', 'you', 'power', 'down', 'the', 'computer,', 'check', 'that', 'all', 'cables', 'are', 'on', 'tight.']\n"
     ]
    }
   ],
   "source": [
    "#apply transform on tokenizer\n",
    "results = tokenizer.transform(test_small_DF);\n",
    "#check the input to the tokenizer and output from the tokenizer\n",
    "print(\"output from the tokenizer and input to the stopword remover \\n\")\n",
    "print(results.take(1)[0][2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### STOP-WORD REMOVAL TEST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words comes from the tokenizer: 163\n",
      "Number of words comes from the remover: 90 \n",
      "\n",
      "output from the  stop word remover  and input to the hashing tf \n",
      "\n",
      "['article', '<1993apr22.055150.28215@nmt.edu>,', 'changw@nmt.edu', '(willie*', 'chang)', 'writes:', '|>', 'happened', 'classic', 'ii', 'once.', 'startup,', 'usual', 'powerup', '|>', 'tone,', 'chimes', '(kinda', 'like', 'do(1st)-mi(1st)-sol(1st)-do(2nd)', '|>', '1st', 'c', '-', '1st', 'e', '-', '1st', 'g', '-', '2nd', 'c', 'music', 'scale),', 'hanged', '|>', 'giving', 'happy', 'mac.', 're-powerup.', '|>', '|>', 'asked', \"can't\", 'remember', 'answer.', \"i've\", 'tried', '|>', 'find', 'answer', 'faqs', 'ftp', 'sites', 'avail.', '|>', '|>', 'tell', 'unusual', 'chimes', 'mean?', 'spurious', 'memory', '|>', 'check', 'error?', '|>', '|>', 'thanks', '|>', 'willie*', 'chimes', 'indicate', 'hardware', 'failure', 'type', 'system', 'startup.', 'thing', 'cause', 'chimes', 'cable', 'bit', 'loose.', 'time', 'power', 'computer,', 'check', 'cables', 'tight.']\n"
     ]
    }
   ],
   "source": [
    "#compare the output word count from tokenizer and output put word count from stop word remover to check the functionality of \n",
    "#stop word remover\n",
    "print(\"Number of words comes from the tokenizer: {}\".format(len(results.take(1)[0][2])));\n",
    "\n",
    "resultsFiltered = remover.transform(results);\n",
    "\n",
    "print(\"Number of words comes from the remover: {} \\n\".format(len(resultsFiltered.take(1)[0][3])));\n",
    "print(\"output from the  stop word remover  and input to the hashing tf \\n\")\n",
    "print(resultsFiltered.take(1)[0][3]);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### HASHING-TF TEST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ouput comes from the hashing function \n",
      "\n",
      "(2000,[4,99,103,157,172,245,257,259,263,278,293,333,388,403,409,417,446,452,458,470,485,527,529,548,646,727,730,746,773,856,865,953,987,990,995,1029,1141,1194,1230,1231,1248,1304,1310,1330,1331,1402,1458,1468,1499,1510,1526,1564,1609,1614,1639,1670,1675,1680,1685,1722,1788,1825,1839,1850,1851,1877,1878,1882,1894],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,13.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,4.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,3.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0])\n"
     ]
    }
   ],
   "source": [
    "#apply hasing transformer to the output from stop word remover and check the output \n",
    "resultsHashed = hashingTF.transform(resultsFiltered); \n",
    "print(\"ouput comes from the hashing function \\n\")\n",
    "print(resultsHashed.take(3)[0][4]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Task C - Evaluating PipeLine Performance\n",
    "\n",
    "The performance of the pipeline was evaluated by projecting the pipeline to varying sets of training and test data and calculating test and training accuracy, as well as F1 measure based on precision and recall values. A multi-class evaluator from machine learning library was chosen to obtain the evaluation metrics for monitoring the  performance of the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Ratio : 1.0\n",
      "Training Accuracy : 0.8767605633802817\n",
      "Training Accuracy F1 : 0.8758672824082925\n",
      "Training Accuracy Weighted Recall: 0.8767605633802817\n",
      "Training Accuracy Weighted Precision: 0.8811608006037032\n",
      "Testing Accuracy : 0.4802200244471608\n",
      "Testing Accuracy F1 : 0.4807148805493953\n",
      "Testing Accuracy Weighted Recall: 0.4802200244471608\n",
      "Testing Accuracy Weighted Precision: 0.5044247815294027 \n",
      "\n",
      "Training Ratio : 5.0\n",
      "Training Accuracy : 0.7078741735123222\n",
      "Training Accuracy F1 : 0.7066322130023295\n",
      "Training Accuracy Weighted Recall: 0.7078741735123221\n",
      "Training Accuracy Weighted Precision: 0.7160803269875344\n",
      "Testing Accuracy : 0.590063974410236\n",
      "Testing Accuracy F1 : 0.5885326352719783\n",
      "Testing Accuracy Weighted Recall: 0.590063974410236\n",
      "Testing Accuracy Weighted Precision: 0.5997519227510981 \n",
      "\n",
      "Training Ratio : 9.0\n",
      "Training Accuracy : 0.6853060996707405\n",
      "Training Accuracy F1 : 0.6836732027754262\n",
      "Training Accuracy Weighted Recall: 0.6853060996707406\n",
      "Training Accuracy Weighted Precision: 0.6929190477369293\n",
      "Testing Accuracy : 0.594581519109821\n",
      "Testing Accuracy F1 : 0.5926003672306978\n",
      "Testing Accuracy Weighted Recall: 0.5945815191098208\n",
      "Testing Accuracy Weighted Precision: 0.6060916867251317 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "#performance of the pipeline\n",
    "\n",
    "#create the list of ratios to devide the training and test data\n",
    "plain_trainRatios = [1.0,5.0,9.0];\n",
    "#create empty arrays to hold results. this will use to generate graphs\n",
    "plain_tr_accs = [];\n",
    "plain_tr_accs_wp = [];\n",
    "plain_tr_accs_wr = [];\n",
    "plain_tr_accs_f1 = [];\n",
    "plain_tst_accs = [];\n",
    "plain_tst_accs_wp = [];\n",
    "plain_tst_accs_wr = [];\n",
    "plain_tst_accs_f1 = [];\n",
    "\n",
    "for trainRatio in plain_trainRatios:\n",
    "    #create training and test datasets from our main dataset.\n",
    "    [training,testing] = rows_DF.randomSplit([trainRatio,10-trainRatio])\n",
    "    #cache the training and testing dataset to speed the access to datafarems\n",
    "    training.cache();\n",
    "    testing.cache();\n",
    "    print(\"Training Ratio : {0}\".format(trainRatio))\n",
    "    \n",
    "    #train the model using pipeline with training data\n",
    "    model = pipeline.fit(training)\n",
    "\n",
    "    #predict using training data on the trained model to get training accuracy\n",
    "    prediction = model.transform(training)\n",
    "\n",
    "    #here we have used weightedPrecision as the evaluation metric\n",
    "    trainAccuracy =  evaluator.evaluate(prediction)\n",
    "    trainAccuracy2 =  evaluator.evaluate(prediction, {evaluator.metricName: \"f1\"}) \n",
    "    trainAccuracy3 =  evaluator.evaluate(prediction, {evaluator.metricName: \"weightedRecall\"}) \n",
    "    trainAccuracy4 =  evaluator.evaluate(prediction, {evaluator.metricName: \"weightedPrecision\"}) \n",
    "    \n",
    "    #resutls adding to arrays\n",
    "    plain_tr_accs.append(trainAccuracy);\n",
    "    plain_tr_accs_f1.append(trainAccuracy2);\n",
    "    plain_tr_accs_wr.append(trainAccuracy3);\n",
    "    plain_tr_accs_wp.append(trainAccuracy4);\n",
    "    \n",
    "    print(\"Training Accuracy : {0}\".format(trainAccuracy))\n",
    "    print(\"Training Accuracy F1 : {0}\".format(trainAccuracy2))\n",
    "    print(\"Training Accuracy Weighted Recall: {0}\".format(trainAccuracy3))\n",
    "    print(\"Training Accuracy Weighted Precision: {0}\".format(trainAccuracy4))\n",
    "    \n",
    "    #predict using test data on the trained model to get test accuracy\n",
    "    prediction = model.transform(testing)\n",
    "\n",
    "    #here we have used weightedPrecision as the evaluation metric\n",
    "    testAccuracy =  evaluator.evaluate(prediction)\n",
    "    testAccuracy2 =  evaluator.evaluate(prediction, {evaluator.metricName: \"f1\"})\n",
    "    testAccuracy3 =  evaluator.evaluate(prediction, {evaluator.metricName: \"weightedRecall\"})\n",
    "    testAccuracy4 =  evaluator.evaluate(prediction, {evaluator.metricName: \"weightedPrecision\"})\n",
    "    \n",
    "    print(\"Testing Accuracy : {0}\".format(testAccuracy))\n",
    "    print(\"Testing Accuracy F1 : {0}\".format(testAccuracy2))\n",
    "    print(\"Testing Accuracy Weighted Recall: {0}\".format(testAccuracy3))\n",
    "    print(\"Testing Accuracy Weighted Precision: {0} \\n\".format(testAccuracy4))\n",
    "\n",
    "    #resutls adding to arrays\n",
    "    plain_tst_accs.append(testAccuracy);\n",
    "    plain_tst_accs_f1.append(testAccuracy2);\n",
    "    plain_tst_accs_wr.append(testAccuracy3);\n",
    "    plain_tst_accs_wp.append(testAccuracy4);\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3kAAAF/CAYAAAD0NgDhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8VPW9//H3JwsJgYQlGBIW2UF2AopYUePSFi3R3qoV\nVPS21vbW9uql1177a+tVW9vSWntbb3urrbalSoV7rUtBWxcU3BVlkx3ZTCAJCVvCEgiZ7++PM8lk\nssAQZnImk9fz8TgP5sycOfOZjMi88/2c79eccwIAAAAAJIYkvwsAAAAAAEQPIQ8AAAAAEgghDwAA\nAAASCCEPAAAAABIIIQ8AAAAAEgghDwAAAAASSMxDnplNM7MNZrbJzO5q5vEzzewVM1tlZq+aWZ9Y\n1wQAAAAAicpiuU6emSVJ2iTpUkm7JC2TNMM5t6HBMf8r6W/OuSfMrEDSl51zN8WsKAAAAABIYLEe\nyZssabNzbodzrkbSfElXNTpmlKRXJck5t6SZxwEAAAAAEYp1yOsrqajBfnHwvoZWSrpakszsC5K6\nmlmPGNcFAAAAAAkp1iHPmrmvcX/otyUVmNmHki6QtFPS8RjXBQAAAAAJKSXG5y+WdGaD/X7yrs2r\n55wrUWgkr4ukq51zVY1PZGaxu3gQAAAAANoB51xzA2lhYj2St0zSUDMbYGadJM2Q9LeGB5hZtpnV\nFfr/JP2hpZM559jieLvnnnt8r4GNz6c9b3xG8b/xGcX3xucT/xufUXxvfD7xv0UqpiHPOVcr6ZuS\nXpK0VtJ859x6M7vPzKYHDyuQtNHMNkjKkfSjWNYEAAAAAIks1u2acs79Q9KIRvfd0+D2XyX9NdZ1\nAAAAAEBHEPPF0NFxFBQU+F0CToDPJ/7xGcU/PqP4xucT//iM4hufT+KI6WLo0WRmrr3UCgAAAADR\nZmZyEUy8EvN2TQAAACARDRw4UDt27PC7DCSgAQMGaPv27a1+PiN5AAAAQCsER1X8LgMJqKX/tiId\nyeOaPAAAAABIIIQ8AAAAAEgghDwAAAAASCCEPAAAAAAtCgQCyszMVHFxcVSPRewQ8gAAAIAEkpmZ\nqaysLGVlZSk5OVkZGRn19z355JOnfL6kpCRVVVWpX79+UT22tR599FElJSXp2WefjdlrtHfMrgkA\nAAC0QnuYXXPw4MF67LHHdPHFF7d4TG1trZKTk9uwqtNz4YUXav369Zo6daqeeeaZNn3tQCCgpKTY\nj5MxuyYAAACAZjnnmoSFu+++WzNmzND111+vbt26ad68eXr33Xd13nnnqUePHurbt6/uuOMO1dbW\nSvJCYFJSkj755BNJ0qxZs3THHXfoiiuuUFZWls4///z69QJP5VhJ+vvf/64RI0aoR48euv322zV1\n6lT9+c9/bvH9bN26VW+//bYeeeQRvfDCC9qzZ0/Y408//bTy8/PVrVs3DR8+XK+88ookae/evfrS\nl76kPn36KDs7W9dee60kNQnAzdX/zW9+U5dffrkyMzP15ptvauHChcrPz1dWVpYGDhyo+++/P6yG\n119/Xeedd566d++uAQMG1P98+/btG3bcggULdM4555zo42s1Qh4AAADQwTz77LO68cYbdeDAAV13\n3XVKTU3VQw89pL179+qtt97Siy++qEceeaT+eLPwwaMnn3xSP/rRj7Rv3z71799fd9999ykfu3v3\nbl133XV68MEHVVFRoUGDBmnZsmUnrHvu3LmaMmWKvvCFL2jIkCFh7advv/22brnlFv3Xf/2XDhw4\noNdee00DBgyQJM2cOVM1NTXasGGDysrKdMcdd7RYb3P133fffaqqqtKUKVOUmZmpefPmqbKyUgsX\nLtRDDz2kF154QZK0bds2TZ8+XXfeeaf27t2rFStWaOzYsZoyZYqysrK0ePHi+vPOmzdPN9988wnf\nb2sR8gAAAIAOZurUqbriiiskSWlpaZo0aZLOOeccmZkGDhyoW2+9VUuXLq0/vvFo4DXXXKP8/Hwl\nJyfrhhtu0MqVK0/52Oeff175+fmaPn26kpOTNXv2bGVnZ5+w7ieeeEI33HCDJOn666/X3Llz6x/7\nwx/+oK9+9asqKCiQJPXt21fDhg1TcXGxXnvtNT388MPKyspSSkqKpk6d2uJrNK7/n/7pnzR58mRJ\nUqdOnVRQUKBRo0ZJksaOHavrrruu/mc1b948XX755br66quVlJSknj17aty4cZK8UcHHH39cklRR\nUaFXX31VM2bMOOH7ba12FfImTpT+8z+lZcukQMDvagAAAICWmUVni4X+/fuH7W/cuFHTp09XXl6e\nunXrpnvuuUcVFRUtPj83N7f+dkZGhg4ePHjKx+7atatJHSeasGXp0qUqLi6ub7WcOXOmPvzwQ61b\nt06SVFRUpCFDhjR5XlFRkXr16qWuXbu2eO4TaVzjO++8o4svvlg5OTnq3r27HnvssfqfVUs1SF7I\ne+6551RdXa358+fr4osvVq9evVpV08m0q5D3q19J1dXSTTdJ/fpJt94q/e1v0uHDflcGAAAAhHMu\nOlssNG5J/NrXvqaxY8dq69atOnDggO67776YTyqTl5enoqKisPt27tzZ4vFz585VIBDQuHHjlJeX\np6lTpyopKan+Gr7+/ftry5YtTZ7Xv39/VVRUNBtEu3TposMNwkRJSclJ2zdnzpypa6+9Vjt37tT+\n/ft1yy231P+s+vfvr48//rjZ+vv376+zzz5bzz77rJ544gnNmjWrxfd6utpVyLvgAulnP5PWr5de\nf10aNUr65S+l3Fxp+nTp4YclluQAAAAATk1VVZW6deumzp07a/369WHX48XK9OnTtWLFCj3//POq\nra3VL3/5yxZHD48cOaK//vWv+sMf/qCVK1dq1apVWrVqlX7xi1/o8ccfl3NOt9xyix599FEtXbpU\nzjnt3LlTmzZtUr9+/XTZZZfpG9/4hg4cOKDjx4/rjTfekCSNHz9eq1ev1tq1a3XkyBH94Ac/OGnd\nBw8eVI8ePZSamqp3331X8+fPr3/sxhtv1IsvvqhnnnlGtbW12rNnj1avXl3/+KxZs/STn/xEGzdu\n1FVXXXWaP8GWtauQ19DQodLs2dKrr0qffCLNmiW9+aY0frzX1nnPPdIHH9DWCQAAgI6r8ShUSx58\n8EH96U9/UlZWlr7+9a83uVas4XlOds5Ij83JydGCBQs0e/Zs9erVS9u2bVN+fr7S0tKaHPv0008r\nKytLN9xwg3Jycuq3W2+9VdXV1Xr55Zd13nnn6fe//73+9V//Vd26ddMll1xSvyj7E088Ieechg8f\nrtzcXP3617+WJI0cOVLf/e53ddFFF2nkyJG66KKLWnwvdX7729/qO9/5jrp166Y5c+bouuuuq39s\n4MCBWrhwoebMmaOePXtq0qRJWrNmTf3jV199tbZu3aprr7222fcZLQm3Tt7x49Lbb0sLF3pbZaU3\nyjd9unTZZVJGRhsUCwAAgITXHtbJa08CgYD69Omjv/71rzr//PP9LidmBg0apLlz5+rCCy9s8RjW\nyWskJUW68ELpgQekDRukpUuls84Kb+t85BHpBO2+AAAAANrAiy++qMrKSh09elQ/+MEPlJqaWj+T\nZSJasGCB0tPTTxjwoiElpmePA8OGSd/6lrft3y/94x/eCN93vysNHCgVFnrBb+JEqQ0WrwcAAAAQ\n9Oabb+qGG25QTU2NRo8ereeee06pqal+lxUTF1xwgT7++GP95S9/iflrJVy7ZqSOH5feektatCi8\nrbOwULr0Uto6AQAAcGK0ayJWTrdds8OGvMY2bw5dx/fhh9JFF3mB73Ofk/r2jdnLAgAAoJ0i5CFW\nCHkxsG9fqK3zH/+QBg3yAl9hodfWGatFKQEAANB+EPIQK4S8GKtr66wb5Tt4MLyts3PnNi8JAAAA\ncYCQh1gh5LWxTZtCgW/58lBb5/TpUp8+flcHAACAtkLIQ6wQ8nzUuK1z8OBQW2d+Pm2dAAAAiYyQ\nh1gh5MWJmprwts7Dh0OLsNPWCQAAkHgIeYgVFkOPE6mpUkGB9OCDXkvnq696a/T9/OdS797SlVdK\nv/+9VFLid6UAAABIZJmZmcrKylJWVpaSk5OVkZFRf9+TTz7Z6vOed955Ea3xduDAAXXu3FnXXHNN\nq18Lp4eQFyPDh0v//u/SkiXS9u3SjBle8Bs9WjrnHOm++7xr+vjlDwAAAKKpqqpKlZWVqqys1IAB\nA/T888/X3zdz5syYv/6CBQvUpUsXPf/889q3b1/MX6+h2traNn29eEXIawM9e0rXXy89+aRUVib9\n7Gfe4uszZkj9+0v/8i/S889LR474XSkAAAASiXOuSdtfIBDQD3/4Qw0ZMkQ5OTmaNWuWKisrJUmH\nDx/WzJkzlZ2drR49eui8887TgQMHdOedd2rZsmX6yle+oqysLH37299u8TXnzp2r2bNna8iQIU1G\nDnfs2KHPf/7zOuOMM5STkxN2nv/5n//RyJEjlZWVpfHjx2vt2rU6evSokpKStGvXrvrjZs6cqR//\n+MeSpBdffFHDhg3T/fffr9zcXN12222qqKjQFVdcoZycHPXq1Uuf//znVVZWVv/8PXv26Oabb1Ze\nXp6ys7Prg++wYcO0ePHi+uOOHj2q7t27a+PGjaf6Y/cdIa+NpaZKF18cautcvFgaOtQLfr17S1dd\nRVsnAAAAYudnP/uZXnnlFb399tsqLi5WamqqZs+eLUl69NFHVVtbq5KSEu3Zs0e//vWv1alTJ/38\n5z/XOeeco8cee0yVlZV64IEHmj335s2b9d577+n666/X9ddfrz//+c/1jx0/flyXX365Ro8eraKi\nIhUVFenqq6+WJD3++ON64IEHtGDBAlVWVuqpp55Sjx49JHnXoZ3I9u3bVVtbq+LiYj300EMKBAL6\n+te/ruLiYm3btk1mVv/+JOmLX/yiJGnTpk0qKyvTN77xDUnSzTffrMcff7z+uOeee04jRozQiBEj\nTvVH7DtCns9GjJDuvFNautRr6/ziF8PbOn/wA2nFCto6AQAAEB2/+93vNGfOHPXu3VudOnXS3Xff\nrfnz50uSUlNTVV5ers2bNyspKUmTJk1S5wYzCJ5sopm5c+dq8uTJGjRokK6//np98MEH9SNhb775\npqqqqvSjH/1I6enpSktL05QpUyRJjz32mL73ve9p3LhxkrxRtT7B9clO9prp6en6/ve/r5SUFKWl\npSknJ0eFhYXq1KmTMjMzddddd2np0qWSpG3btumtt97Sb3/7W2VmZiolJUVTp06VJM2aNUvPPvus\nqqurJXnBc9asWaf0s40XKX4XgJCePaUbbvC2mhrpzTe9mTq/+EWpujq0CPvFFzNbJwAAQLyz+6Kz\nnpa7J7q/7S8qKtIVV1xRP0JWF6L27t2rW265RaWlpbrmmmt06NAhzZo1S/fff/9JR9PqPPHEE/Ut\nmAMHDtS5556ruXPn6sc//rGKioo0aNCgFmsaPHhwq95Pbm6ukpOT6/cPHjyoO+64Q6+88ooOHDgg\n51x9cCsuLlZOTo4yMjKanGfAgAGaOHGinnvuOV166aV69dVX9cc//rFVNfmNkBen6to661o7N270\nAt9PfyrNnOnN5FlYKH3uc1Jent/VAgAAoLFoh7No6devn55++mnl5+c3+/h9992n++67T9u3b9dn\nPvMZjRkzRjNnzjxp0Hvttdf0ySef6N5779X9998vyQtcRUVF+vGPf6z+/ftr+/btzT63f//+2rJl\niy655JKw+zt16qTU1FQdPny4/r7S0lKNHTu2fr9xXXPmzNHOnTv14YcfqlevXnrvvfd0wQUX1L/O\n7t27dfjw4WaD3k033aTHH39c5eXluvTSS9WrV68Tvud4RbtmO2AmnXWW9O1ve22d27Z5o3uvvCKN\nGiVNniz98IfSypW0dQIAAODEvva1r+muu+5ScXGxJGn37t1atGiRJGnx4sVav369nHPq2rWrUlJS\nlJLijQv17t1bW7dubfG8f/rTn1RYWKh169Zp1apVWrVqlVauXKk9e/Zo8eLFmjp1qjIzM3X33Xfr\nyJEjqq6u1jvvvCNJ+spXvqI5c+Zo9erVkrxr+3bt2iUz07hx4zRv3jwFAgEtXLiw/jktqaqqUkZG\nhrKyslRRUVEfOCVvdPHCCy/UN7/5TVVWVqqmpkZvvPFG/ePXXHON3nzzTT388MO66aabWvHTjQ+E\nvHaorq1z/nxp925pzhxp717pmmukM8+Uvv516YUXvBZPAAAAdFzNjb7ddddd+vSnP61LLrlE3bp1\n09SpU7VixQpJ0s6dO3XVVVcpKytL48aN0/Tp03XttddKkmbPnq25c+cqOztb3/nOd8LOeejQIT3z\nzDO644476mfOzMnJ0ZAhQzRz5kzNnTtXKSkpeuGFF7Ry5Ur169dPAwYM0DPPPCNJuvHGG/Wtb31L\n11xzjbp166Zrr71W+/fvlyT993//t+bPn6+ePXvqueeeU2Fh4Qnf85133qny8nJlZ2frwgsv1Oc+\n97mwx5988kkdO3ZMw4YNU15enh5++OH6x7p27arCwkLt3LlTV1555Sn+tOOHnexCxnhhZq691OoX\n50JtnQsXSqtWee2edW2dubl+VwgAAJA4zOykk4Kg/fne976n8vJy/e53v/Othpb+2wref9ILJAl5\nCWzPHunvf/cC30svScOGeYGvsFAaP95rAwUAAEDrEPIST3l5uSZMmKDnnntOZ599tm91nG7Io10z\ngWVnSzfeKC1Y4C3C/pOfhNo6BwyQbrvNC4G0dQIAAKCj+81vfqNBgwbpuuuu8zXgRQMjeR2Qc9KG\nDaG2ztWraesEAAA4VYzkIVZo18Rp27PHm6hl0SKvrXP48FBb57hxtHUCAAA0h5CHWCHkIaqOHZPe\neCM0yldT44W96dO90b70dL8rBAAAiA+EPMQKIQ8x45y0fr03wlfX1nnJJaG2zt69/a4QAADAP4Q8\nxAohD22moiJ8ts6zzgqN8tHWCQAAOhpCHmKFkAdfHDsmvf56qK2zttYLe4WFUkEBbZ0AACDxDRw4\nUDt27PC7DCSgAQMGaPv27U3uJ+ShzdS1ddYFvo8+ki69NNTWmZPjd4UAAABA+0fIg28qKrzZOhcu\nlF5+OdTWWVgojR1LWycAAADQGoQ8xIXm2jrrAl9BgZSW5neFAAAAQPtAyEPccU5aty4U+Nasoa0T\nAAAAiBQhD3GvvDw0W+fLL0sjR4ZG+caMoa0TAAAAaIiQh3bl2DFp6dLQKJ9z4bN10tYJAACAjo6Q\nh3bLOWnt2tAi7GvXhto6r7iCtk4AAAB0TIQ8JIzy8tBsna+8Io0aFRrlo60TAAAAHQUhDwnp6NHw\ntk6zUOC76CLaOgEAAJC4CHlIeHVtnXWBb9066bLLQm2dZ5zhd4UAAABA9BDy0OHs3u21dS5aFGrr\nrJutc/Ro2joBAADQvhHy0KE119ZZF/guvJC2TgAAALQ/hDwgyDlv4fW6wLd+PW2dAAAAaH8IeUAL\n6to662brHDPGC3zTp9PWCQAAgPhFyAMicPSotGRJaJQvOTl8ts5OnfyuEAAAAPAQ8oBT5Jz00Ueh\nRdjXr5c+/elQW2evXn5XCAAAgI6MkAecprKyUFvn4sWhts7CQm/mTto6AQAA0JYIeUAUVVd7bZ11\no3zJyeGzddLWCQAAgFgj5AExUtfWWXcd38aNXlvn9Om0dQIAACB2CHlAGykrk55/3gt8r74qjR0b\nGuUbOZK2TgAAAEQHIQ/wQV1bZ90oX2pqaLZO2joBAABwOuIm5JnZNEm/lJQk6THn3E8bPd5f0lxJ\n3YPH/D/n3N+bOQ8hD+2Kc9Lq1V7YW7Qo1NZZN1tndrbfFQIAAKA9iYuQZ2ZJkjZJulTSLknLJM1w\nzm1ocMwjkpY75x4xs5GSXnDODWrmXIQ8tGuN2zrHjQuN8tHWCQAAgJOJNOQlxbiOyZI2O+d2OOdq\nJM2XdFWjYwKSsoK3u0vaGeOaAF/07i19+cvSM894ge+735V27JCmTZOGDpX+7d+8pRqOHfO7UgAA\nALRnsR7Ju1rSZ51zXw3u3yhpsnPu9gbH5Ep6SVIPSRmSLnPOrWjmXIzkISE1bOtcuFDatEn6zGe8\nEb7LL6etEwAAAJ5IR/JSYl1HM/c1TmozJf3ROfdfZjZF0hOSRjd3snvvvbf+dkFBgQoKCqJTJeAj\nM2n8eG/7/vel0lKvrfOpp6TbbvPur5ut86yzaOsEAADoKJYsWaIlS5ac8vNiPZI3RdK9zrlpwf3v\nSHINJ18xszXyRvt2Bve3SDrXOVfR6FyM5KHDqa6WXnstNMqXlhYKfBdc4M3eCQAAgI4hXiZeSZa0\nUd7EKyWS3pc00zm3vsExz0v6X+fc3ODEKy875/o1cy5CHjo056RVq0KBb/Nm2joBAAA6krgIecFC\npkn6lUJLKMwxs/skLXPOLQoGu99L6ipvEpZvO+cWN3MeQh7QQEmJ19a5aJE32tewrXPECNo6AQAA\nEk3chLxoIeQBLTtyJNTWuWiRlJ4eWp6Btk4AAIDEQMgDOijnpJUrQ4Fv82bps58NtXX27Ol3hQAA\nAGgNQh4ASaG2zoULvdG+CRNo6wQAAGiPCHkAmjhyRHr1VW+Eb+FCqXPnUOCbOpW2TgAAgHhGyANw\nQg3bOhculLZs8do6p0+nrRMAACAeEfIAnJJdu0JtnUuWSPn54W2dAAAA8BchD0Cr1bV11o3ydeni\nhb3p02nrBAAA8AshD0BUOCetWBEKfFu3hs/W2aOH3xUCAAB0DIQ8ADHRuK1z4sTQKB9tnQAAALFD\nyAMQc4cPh9o6Fy0KtXUWFkrnn09bJwAAQDQR8gC0Keek5ctDyzNs3SpNm+YFvmnTaOsEAAA4XYQ8\nAL7auTPU1rl0aaits7BQGj7c7+oAAADaH0IegLhx+LC0eHGorTMzM7ytMyXF7woBAADiHyEPQFwK\nBMJn69y2jbZOAACASBDyALQLO3eGruN7/XVp0qTQKN+wYX5XBwAAED8IeQDancZtnVlZ3tIMtHUC\nAAAQ8gC0c4GAN1tnXeDbvj28rbN7d78rBAAAaFuEPAAJpXFb59lnhxZhp60TAAB0BIQ8AAnr0KHw\nts5u3ULX8X3qU7R1AgCAxETIA9AhNGzrXLhQ2rFDuvxyL/B99rO0dQIAgMRByAPQIRUXh9o633gj\n1NZZWCgNHep3dQAAAK1HyAPQ4R06JL3yihf6Fi3yRvXqruOjrRMAALQ3hDwAaCAQkD78MNTW+ckn\nobbOadO86/oAAADiGSEPAE6gqCg0wvfGG9I554TaOocM8bs6AACApgh5ABChurbOutk6e/QIBb7z\nzqOtEwAAxAdCHgC0QiAgffBBqK2zuDh8tk7aOgEAgF8IeQAQBXVtnQsXSm++KU2e7E3cQlsnAABo\na4Q8AIiygwdDbZ3PPy/17Blq65wyhbZOAAAQW4Q8AIgh2joBAEBbI+QBQBv65JPwts5zzw2N8g0e\n7Hd1AAAgERDyAMAnDds6Fy2SevUKLcJ+3nlScrLfFQIAgPaIkAcAcSAQkJYtC7V17toV3taZleV3\nhQAAoL0g5AFAHNqxw5u0ZeFC6a23Qm2d06fT1gkAAE6MkAcAce7gQenll0Ozdda1ddbN1klbJwAA\naIiQBwDtSCAgvf9+6Dq+XbukK67wAt9nPkNbJwAAIOQBQLu2Y0dots633vJG9upG+QYN8rs6AADg\nB0IeACSIqqrwts6cnFDgO/dc2joBAOgoCHkAkIBqa8Nn6ywpoa0TAICOgpAHAB3A9u2hts633/bW\n4asb5Rs40O/qAABANBHyAKCDoa0TAIDERsgDgA6stjY0W+fChVJZWXhbZ2am3xUCAIBTRcgDANRr\n3Nb5qU95C7DT1gkAQPtByAMANKuqSnrpJS/wvfCC1Lt3qK1z8mTaOgEAiFeEPADASdXWSu+9Fxrl\nKyuTPvc5b5SPtk4AAOILIQ8AcMq2bQsFvnfe8do660b5BgzwuzoAADo2Qh4A4LRUVnptnYsWebN1\n5uV5YW/6dNo6AQDwAyEPABA1dW2ddbN1lpeHz9bZtavfFQIAkPgIeQCAmNm2LRT43n1XOv/80Cgf\nbZ0AAMQGIQ8A0Cbq2jrrZuvs0ye0PMPkyVJSkt8VAgCQGAh5AIA2V1vrjezVjfJVVHizdRYWSp/+\nNG2dAACcDkIeAMB3W7eGZuts2NZZWCideabf1QEA0L4Q8gAAcaWyUnrxxVBbZ9++ocB3zjm0dQIA\ncDKEPABA3KKtEwCAU0fIAwC0G1u2hNo633tPmjo1NFsnbZ0AAHgIeQCAdunAgfDZOvv1CwU+2joB\nAB0ZIQ8A0O7V1krvvBNq69y7N7yts0sXvysEAKDtEPIAAAlnyxYv7C1aJL3/vtfWWbcmX//+flcH\nAEBsEfIAAAntwIHQbJ1//3uorbOwUDr7bNo6AQCJh5AHAOgwjh8PtXUuWiTt2+e1dU6fTlsnACBx\nEPIAAB3Wxx+HZut8/33pggtCk7fQ1gkAaK8IeQAASNq/P7yt88wzQ22dkybR1gkAaD8IeQAANNKw\nrXPhQi8A1s3WedlltHUCAOIbIQ8AgJP4+ONQ4Pvgg/C2zn79/K4OAIBwhDwAAE5B47bOAQNCyzPQ\n1gkAiAeEPAAAWun4centt0OjfAcOSNOmSXl5UkaG19aZkRHaTrTfuTMBEQAQHYQ8AACiZPNm6ZVX\nvKUZDh2SDh8ObQ33m3usulpKT48sEJ5KeGx4Oz2dIAkAHUHchDwzmybpl5KSJD3mnPtpo8d/Ieli\nSU5SF0lnOOd6NnMeQh4AoN0JBLygF0kgPJXw2HC/utobMYxGeGzpuenpkp30awUAIJbiIuSZWZKk\nTZIulbRL0jJJM5xzG1o4/puSJjjnvtLMY4Q8AACaEQhIR460LiBGeuzRo16QjMVIZN1+WhpBEgBO\nJNKQlxLjOiZL2uyc2xEsar6kqyQ1G/IkzZT0nzGuCQCAhJKU5IWkWC4BUVvbcpBsab+yUiotjTxo\nHjsWm/DYcJ8gCaAjOGnIC46uzXPO7WvF+ftKKmqwXywv+DX3OmdKGijp1Va8DgAAiKHkZKlrV2+L\nldra8PAIC7ypAAAfX0lEQVQXycjjgQNSSUnko5THj8cmPDbc79SJIAnAX5GM5OVKWmZmyyX9QdKL\np9A32dz/4lp67gxJT9GTCQBAx5ScLGVmelusHD/ujUieShvrvn3Szp2RB89A4PSvgTzZfmoqQRJA\ny04a8pxz3zezuyV9RtKXJP3azP5X3iQqW07y9GJJZzbY7yfv2rzmzJB024lOdu+999bfLigoUEFB\nwUleHgAAICQlJfZBsqYm1Noa6TWQe/dKRUWRXy8ZCMRuJLJhkATgryVLlmjJkiWn/LyIJ14xs/Hy\nQt40Sa9JmiLpZefcf5zgOcmSNsqbeKVE0vuSZjrn1jc6boSkvzvnBp/gXAzyAQAAyAuSrZ2NNZLg\neeiQN1IYq2U/6vZTYj07BJBgoja7ppndLulmSRWSHpX0rHOuJjhz5mbn3JCTPH+apF8ptITCHDO7\nT9Iy59yi4DH3SEpzzn33BOch5AEAALSRmprTW9rjZMceOuS16MZqJLJuI0gikUQz5P1AXjjb0cxj\nIxuPysUKIQ8AACBxOBcakYzVGpKHDnkhL1YjkXW3k5P9/mmio4hmyJsiaa1zriq4nylplHPuvahU\nGiFCHgAAAE6Fc97SHLEYiWy4n5ISu2U/unTx1qgkSEKKbshbIWliXcIKtml+4JybGJVKI0TIAwAA\nQLxxTjp6NHYjkXW3O3U6tfB4qmEyI8NbcxPxLZqLoYelK+dcwMzobgYAAECHZyalp3tbz56xeQ3n\npOrqU59QZ9++yI89ckRKS4vdsh8ZGd6IJEGybUQS1rYGJ1/5bXD/NklbY1cSAAAAgDpmXkDq3FnK\nzo7Na9QFyVMdidyzJ/Jjq6u9MBzLyXbS0wmSUmTtmjmSHpJ0ibyFzBdL+jfn3O7YlxdWB+2aAAAA\nQDsVCIRGJGM1c2t1tReGYzESWXc7Pd0L3n6I2jV58YKQBwAAAOBEAgGv9TTa10Q2vH30qBckY7mG\nZFpa80EyatfkmVm6pFskjZaUXne/c+7Lp/QTBwAAAIAYSkryQlKXLrF7jdra8CAZSXisrJRKSyMP\nmseONR8CIxXJNXmPS9og6bOSfiDpBkltsjYeAAAAAMST5GSpa1dvi5W6INk4EJ57bmTPj2gJBedc\nvpmtds6NM7NUSW8456acfvmRo10TAAAAQEcWabtmJHPP1AT/3G9mYyR1k5RzOsUBAAAAAGIjknbN\n35lZD0nfl/Q3SV0l3R3TqgAAAAAArXLCkGdmSZIqnXP7JL0uaXCbVAUAAAAAaJUTtms65wKS/qON\nagEAAAAAnKZIJl6ZI6lC0gJJh+rud87tjW1pTepg4hUAAAAAHVbUFkM3s23N3O2cc23auknIAwAA\nANCRRS3kxQtCHgAAAICOLNKQd9LZNc3spubud879uTWFAQAAAABiJ5IlFM5pcDtd0qWSlksi5AEA\nAABAnDnldk0z6yZpgXNuWmxKavF1adcEAAAA0GFF2q55wiUUWnBY0qBWPA8AAAAAEGORXJO3UFLd\nEFqSpFGS/jeWRQEAAAAAWieSJRQuarB7XNIO51xxTKtqvg7aNQEAAAB0WFGbXVPSJ5JKnHPVwRN3\nNrOBzrntp1kjAAAAACDKIrkm7/8kBRrs1wbvAwAAAADEmUhCXopz7ljdTvB2p9iVBAAAAABorUhC\nXrmZXVm3Y2ZXSaqIXUkAAAAAgNaKZOKVIZLmSeoTvKtY0k3OuY9jXFvjOph4BQAAAECHFenEKxEv\nhm5mXYPHV51uca1ByAMAAADQkUVtMXQz+7GZdXfOHXTOVZlZDzO7PzplAgAAAACiKZJr8i53zu2v\n23HO7ZN0RexKAgAAAAC0ViQhL9nM0up2zKyzpLQTHA8AAAAA8Ekki6E/IWmxmf0xuP8lSXNjVxIA\nAAAAoLUimnjFzKZJukySSdonKc85940Y19a4BiZeAQAAANBhRW3ilaBSSQFJV0u6VNL606gNAAAA\nABAjLbZrmtlwSTMkzZS0R9ICeSN/F7dRbQAAAACAU9Riu6aZBSS9IemWuoXPzWyrc25wG9bXsB7a\nNQEAAAB0WNFo17xaXpvma2b2ezO7VN41eQAAAACAOHXSiVfMrIukz8tr27xE3syazzjnXop9eWF1\nMJIHAAAAoMOKdCQvotk1G5y0p6RrJV3nnLvkNOo7ZYQ8AAAAAB1ZTEKenwh5AAAAADqyaC+hAAAA\nAABoBwh5AAAAAJBACHkAAAAAkEAIeQAAAACQQAh5AAAAAJBACHkAAAAAkEAIeQAAAACQQAh5AAAA\nAJBACHkAAAAAkEAIeQAAAACQQAh5AAAAAJBACHkAAAAAkEAIeQAAAACQQAh5AAAAAJBACHkAAAAA\nkEAIeQAAAACQQAh5AAAAAJBACHkAAAAAkEAIeQAAAACQQAh5AAAAAJBACHkAAAAAkEAIeQAAAACQ\nQAh5AAAAAJBACHkAAAAAkEAIeQAAAACQQGIe8sxsmpltMLNNZnZXC8d80czWmtlHZvZErGsCAAAA\ngERlzrnYndwsSdImSZdK2iVpmaQZzrkNDY4ZKmmBpIudc5Vm1ss5V9HMuVwsawUAAACAeGZmcs7Z\nyY6L9UjeZEmbnXM7nHM1kuZLuqrRMbdK+o1zrlKSmgt4AAAAAIDIxDrk9ZVU1GC/OHhfQ8MljTCz\nN83sbTP7bIxrAgAAAICElRLj8zc3lNi45zJF0lBJF0o6U9IbZja6bmQPAAAAABC5WIe8YnnBrU4/\nedfmNT7mHedcQNJ2M9soaZikDxuf7N57762/XVBQoIKCgiiXCwAAAADxYcmSJVqyZMkpPy/WE68k\nS9oob+KVEknvS5rpnFvf4JjPBu/7ZzPrJS/cTXDO7Wt0LiZeAQAAANBhxcXEK865WknflPSSpLWS\n5jvn1pvZfWY2PXjMi5L2mNlaSYsl3dk44AEAAAAAIhPTkbxoYiQPAAAAgB+ccwq4QKu3Wld7Ws+v\n2z495NMRjeTF+po8AAAAIOE456L2xb02cPrniUYt0agjarXE2fuRpCRLOq0t2ZJP+xyRYiQPAADg\nNDnn5OSi/gUzoUOE2vf7kRTVL+/JSad/jqjVEifvJxp1RPNnGw8ivSaPkTwAAGLgZK098fJb6oQL\nET69Hycnk8X/l/eWalHrz5GSlNKuv7y3thazk37PBnxDyAMAoBHnnHYc2KHlJcvrtzW716j6eHXE\nIUKKvLUnkX5rnmzJrfrSH7fvJ8I6TMaXfgBxg3ZNAECHFnABfbz347BAt7xkudJT0jUxb2L9Nq73\nOHXt1DXiEMEXfgBAtEXarknIAwB0GMcDx7WhYkNYmFtZulLZGdmamDdR+bn59X/mZeb5XS4AAGEI\neQCADu3o8aNaW742LNB9tPsj9cvq543O5XojdBNyJyg7I9vvcgEAOClCHgCgwzhcc1iry1aHBboN\nFRs0pOeQsEA3Pne8stKy/C4XAIBWIeQBABJS5dFKrSxdGRbotu7bqpFnjKwPcxPzJmps77HKSM3w\nu1wAAKKGkAcAaPf2HN6jFaUrwgLdzqqdGtd7XFigG50zWp2SO/ldLgAAMUXIAwC0K6UHS5vMcLn3\nyF7l5+WHBboRvUYoJYkVgAAAHQ8hDwAQl5xzKqosahLojtYeDbt+bmLeRA3pOURJluR3yQAAxAVC\nHgDAdwEX0NZ9W5sEupSkFE3qMyks0J3Z7UzWlgMA4AQIeQCANlUbqNXGPRvDwtyK0hXqnt69yQgd\na9ABAHDqCHkAgJg5VntM68rXhQW61WWrlZeZFxbo8vPy1Sujl9/lAgCQEAh5AICoOFJzRB/t/igs\n0K0rX6dBPQY1WVS8W3o3v8sFACBhEfIAAKes6miVVpWtCgt0H+/9WCN6jQhrtxzXe5y6dOrid7kA\nAHQohDwAwAntO7KvyRp0RZVFGpMzJizQjckZo7SUNL/LBQCgwyPkAQDqlR0saxLoyg+Xa0LuhLBA\nd1avs5SanOp3uQAAoBmEPADogJxz2lm1s8mSBYdqDjWZ4XJoz6FKTkr2u2QAABAhQh4AJDjnnLbt\n39Yk0ElqsgbdwO4DWYMOAIB2jpAHAAmkNlCrTXs2hcJc6XKtKFmhzLTMJiN0fTL7EOgAAEhAhDwA\naKdqamvC16ArXa5VpavUu2vvJmvQ5XTJ8btcAADQRgh5ANAOVB+v1kdlH4UFurW712pA9wFN1qDr\n0bmH3+UCAAAfEfIAIM4cPHZQq0pX1Ye55SXLtWnPJg3PHh4W6MbnjlfXTl39LhcAAMQZQh4A+Gh/\n9X6tKFkRFuh27N+h0Tmjw66fG9t7rNJT0v0uFwAAtAOEPABoI+WHysPaLZeXLFfZwTKNzx0fFuhG\nnTGKNegAAECrEfIAIMqcc9pVtatJoKs6WlUf5PJz8zUxb6KGZw9nDToAABBVhDwAOA3OOW3fv71J\noAu4gCblTaoPdRPzJmpQ90EsWQAAAGKOkAcAEQq4gDbv2dwk0HVJ7RIW5ibmTVTfzL4EOgAA4AtC\nHgA043jguNaXrw8LdCtLV+qMjDPCwlx+br56d+3td7kAAAD1CHkAOryjx49qze41YYFuze416p/V\nPyzQTcidoJ6de/pdLgAAwAkR8gB0KIeOHdLqstVhgW5jxUYN7Tk0LNCN7z1emWmZfpcLAABwygh5\nABLWgeoDWlm6Muz6uW37tmnUGaPCAt3YnLHqnNrZ73IBAACigpAHICFUHK5osqh4SVWJxvUeFxbo\nRp0xSp2SO/ldLgAAQMwQ8gC0OyVVJU1muNxfvb9+7bm6bUT2CNagAwAAHQ4hD0Dccs7pkwOfNAl0\nNbU1TZYsGNxjsJIsye+SAQAAfEfIAxAXAi6gLXu3NAl0aclpTQJd/6z+rEEHAADQAkIegDZ3PHBc\nGys2NlmDrkd6jyZr0OVl5vldLgAAQLtCyAMQU8dqj2nt7rVhgW512Wr1zezbJNBlZ2T7XS4AAEC7\nR8gDEDVHao40WYNuffl6De4xuMmi4llpWX6XCwAAkJAIeQBapepoVZM16Lbs3aKzep0VFujG9R6n\njNQMv8sFAADoMAh5AE5q75G9TdagK64s1ticsWGBbvQZo5WWkuZ3uQAAAB0aIQ9AmNKDpaF2y+C2\n98heTcidEBbozup1llKSUvwuFwAAAI0Q8oAOyjmnosqiJoGu+nh1kyULhvYcyhp0AAAA7QQhD+gA\nAi6grfu2Ngl0KUkpmtRnkvJz8+sD3YBuA1iDDgAAoB0j5AEJpjZQq417NoaFuRWlK9Q9vbsX5HJD\nI3SsQQcAAJB4CHlAO3as9pjWla8LC3Sry1YrLzMvLNDl5+WrV0Yvv8sFAABAGyDkAe3EkZoj+mj3\nR2GBbl35Og3qMSgs0E3InaBu6d38LhcAAAA+IeQBcajqaJVWla0KC3Qf7/1YI3qNCGu3HNd7nLp0\n6uJ3uQAAAIgjhDzAZ/uO7NOK0hVhga6oskhjcsaEBboxOWNYgw4AAAAnRcgD2tDuQ7ubzHBZfrjc\nW4MuN3wNutTkVL/LBQAAQDtEyANiwDmnnVU7mwS6QzWHmsxwObTnUCUnJftdMgAAABIEIQ84Tc45\nbdu/rUmgk6RJfSaFBbqB3QeyBh0AAABiipAHnILaQK02793cZA26rp26Nhmh65PZh0AHAACANkfI\nA1pQU1uj9RXrwwLdqrJVyumS02QNupwuOX6XCwAAAEgi5AGSpOrj1Vqze01YoFuze40GdB/QZA26\nHp17+F0uAAAA0CJCHjqcQ8cONVmDbtOeTRqWPSws0I3PHa+unbr6XS4AAABwSgh5SGj7q/drZenK\nsEC3ff92jc4ZHXb93NjeY5Weku53uQAAAMBpI+QhYZQfKm+yqHjpwVKNzx0fFuhGnTGKNegAAACQ\nsAh5aHeccyo5WNJkyYLKo5XKz8sPC3TDs4ezBh0AAAA6FEIe4ppzTtv3bw9brmB5yXLVutomSxYM\n6jFISZbkd8kAAACArwh5iBsBF9DmPQ3WoCtdrhUlK9Q5tXOTQNcvqx9r0AEAAADNIOTBF8cDx7W+\nfH1YoFtVukrZGdlN1qDL7Zrrd7kAAABAu0HIQ8wdPX40fA26Um8Nuv5Z/etH5urWoOvZuaff5QIA\nAADtGiEPUXW45rBWla4KC3QbKzZqaM+hYYFufO/xykzL9LtcAAAAIOHETcgzs2mSfikpSdJjzrmf\nNnr8ZkkPSCoO3vVr59wfmjkPIa+NHKg+EFqDrtQLddv2bdOoM0aFBbqxOWPVObWz3+UCAAAAHUJc\nhDwzS5K0SdKlknZJWiZphnNuQ4NjbpY0yTl3+0nORciLgYrDFVpRsiIs0JVUlWhc73FhgW7UGaPU\nKbmT3+UCAAAAHVakIS8lxnVMlrTZObcjWNR8SVdJ2tDoOKZTbAMlVSVh7ZbLS5Zrf/V+5efma2Le\nRBUOL9Q9F92j4dnDlZIU6/80AAAAAMRCrL/J95VU1GC/WF7wa+wLZnaBvFG/bznnips5BhFyzumT\nA580CXTHao9pUt4kTcybqJljZuqBTz+gwT0GswYdAAAAkEBiHfKaG6Fr3HP5N0l/cc7VmNnXJM2V\n196JCARcQFv2bmkS6NKS0+pbLW+deKsm5k1U/6z+rEEHAAAAJLhYh7xiSWc22O8n79q8es65fQ12\nfy8pbGKWhu6999762wUFBSooKIhGje3G8cBxbazYGBboVpauVI/0HvWBbvaU2crPzVdeZp7f5QIA\nAAA4DUuWLNGSJUtO+XmxnnglWdJGeSNzJZLelzTTObe+wTG5zrnS4O1/kvRt59ynmjlXh5p45Vjt\nMa3dvTYs0H1U9pH6ZPYJmxAlPzdf2RnZfpcLAAAAIMbiYuIV51ytmX1T0ksKLaGw3szuk7TMObdI\n0u1mdqWkGkl7Jf1zLGuKR0dqjmh12eqwQLe+fL0G9xhcH+auG3OdJuROUFZalt/lAgAAAIhjLIbe\nxqqOVjVZg27L3i06q9dZYSN043qPU0Zqht/lAgAAAIgTcbFOXjS1x5C398jeJmvQFVcWa2zO2LBA\nN/qM0UpLSfO7XAAAAABxjJDXxsoOljWZ4XLP4T2akDshLNCd1ess1qADAAAAcMoIeTHinFNxZXGT\nQHek5khYmJuYN1FDew5lDToAAAAAUUHIiwLnnLbu29ok0CVZUv2i4nXbgG4DWIMOAAAAQMwQ8k5R\nbaBWm/ZsCgt0K0pWKCstq8kIXV7XPAIdAAAAgDZFyDuBmtoarStfFxboVpWuUm7X3CZr0J3R5Yyo\nvCYAAAAAnA5CXlD18Wp9VPZRWKBbu3utBnYfGBboJuROUPf07jGoHAAAAABOX4cMeQePHdSq0lVh\n189t3rNZI3qN0MTc4OhcXr7G9x6vLp26tFHlAAAAAHD6Ej7k7TuyTytKV9SP0K0oXaEd+3doTM6Y\n+tG5SXmTNDpntNJT0n2sHAAAAABOX0KGvB+9/qP6UFd+uFzje48Pa7kc2WukUpNT/S4VAAAAAKIu\n0pDXrlbl3ntkr74w8gu6/5L7NaznMCUnJftdEgAAAADElXY1ktdeagUAAACAaIt0JC+pLYoBAAAA\nALQNQh4AAAAAJBBCHgAAAAAkEEIeAAAAACQQQh4AAAAAJBBCHgAAAAAkEEIeAAAAACQQQh4AAAAA\nJBBCHgAAAAAkEEIeAAAAACQQQh4AAAAAJBBCHgAAAAAkEEIeAAAAACQQQh4AAAAAJBBCHgAAAAAk\nEEIeAAAAACQQQh4AAAAAJBBCHqJmyZIlfpeAE+DziX98RvGPzyi+8fnEPz6j+MbnkzgIeYga/scQ\n3/h84h+fUfzjM4pvfD7xj88ovvH5JA5CHgAAAAAkEEIeAAAAACQQc875XUNEzKx9FAoAAAAAMeKc\ns5Md025CHgAAAADg5GjXBAAAAIAEQsgDAAAAgAQS9yHPzB4zszIzW+13LWjKzPqZ2atmts7MPjKz\n2/2uCeHMLM3M3jOzFcHP6B6/a0JTZpZkZsvN7G9+14KmzGy7ma0K/j163+960JSZdTOz/zOz9Wa2\n1szO9bsmeMxsePDvzvLgnwf4vhB/zGy2ma0xs9VmNs/MOvldE8KZ2R3B73In/c4d99fkmdlUSQcl\n/dk5N87vehDOzHIl5TrnVppZV0kfSrrKObfB59LQgJllOOcOm1mypLck3e6c44tqHDGz2ZImScpy\nzl3pdz0IZ2ZbJU1yzu3zuxY0z8z+JGmpc+6PZpYiKcM5V+lzWWjEzJIkFUs61zlX5Hc98JhZH0lv\nSjrLOXfMzBZIet4592efS0OQmY2W9KSkcyQdl/QPSf/inNvS3PFxP5LnnHtTEv+oxinnXKlzbmXw\n9kFJ6yX19bcqNOacOxy8mSYpRVJ8/3angzGzfpKukPSo37WgRaZ28G9mR2VmmZIucM79UZKcc8cJ\neHHrMklbCHhxKVlSl7pfkkja5XM9CDdS0rvOuaPOuVpJSyX9U0sH8w8WosbMBkqaIOk9fytBY8FW\nwBWSSiW97Jxb5ndNCPNfkr4twnc8c5JeNLNlZnar38WgicGSKszsj8GWwN+ZWWe/i0KzrpM3GoE4\n4pzbJelBSZ9I2ilpv3PuFX+rQiNrJF1oZj3MLEPeL4f7t3QwIQ9REWzVfErSHcERPcQR51zAOZcv\nqZ+kc81slN81wWNmn5NUFhwRt+CG+PMp59zZ8v5R/UbwUgLEjxRJEyX9xjk3UdJhSd/xtyQ0Zmap\nkq6U9H9+14JwZtZd0lWSBkjqI6mrmV3vb1VoKHgp1E8lvSLpBUkr5bVtNouQh9MWHNZ/StLjzrnn\n/K4HLQu2Ly2RNM3nUhByvqQrg9d8PSnpYjPjGog445wrDf5ZLukZSZP9rQiNFEsqcs59ENx/Sl7o\nQ3y5XNKHwb9HiC+XSdrqnNsbbAV8WtKnfK4JjTjn/uicm+ScK5B3Odvmlo5tLyGP327Htz9IWuec\n+5XfhaApM+tlZt2CtzvL+x85E+PECefcd51zZzrnBkuaIelV59xNfteFEDPLCHYryMy6SPqMvLYZ\nxAnnXJmkIjMbHrzrUknrfCwJzZspWjXj1SeSpphZupmZvL9D632uCY2Y2RnBP8+Udz1ei3+fUtqq\nqNYys79IKpCUbWafSLqn7sJq+M/Mzpd0g6SPgtd8OUnfdc79w9/K0ECepLnBGc2SJC1wzr3gc01A\ne9Jb0jNm5uT9uznPOfeSzzWhqdslzQu2BG6V9CWf60EDDX7J+FW/a0FTzrn3zewpSSsk1QT//J2/\nVaEZfzWznvI+o9uccwdaOjDul1AAAAAAAESuvbRrAgAAAAAiQMgDAAAAgARCyAMAAACABELIAwAA\nAIAEQsgDAAAAgARCyAMAAACABELIAwD4wsx6mtkKM1tuZiVmVtxgP6J1XM3sMTMbdpJjbjOzmVGq\n+apgjSvNbI2Zffkkx19sZpNbeOwWM9sdfL/rzOxfI3j9sPNF870BABIH6+QBAHxnZv8p6aBz7hfN\nPGYuDv6xMrNOkrZJmuicKwsuuj3AOffxCZ7zQ0kVzrlfNfPYLZJGO+e+ZWa9JG2UNMo5V9aa8wEA\nUIeRPABAPLD6G2ZDzGytmT1hZmsk5ZrZI2b2vpl9ZGbfb3DsG2Y2zsySzWyfmf0kOMr2VjA4ycx+\naGa3Nzj+J2b2npmtN7MpwfszzOyp4Ojc/5nZMjMb16jGbsE/90uSc66mLuCZWY6Z/TVY47tmNtnM\nBkv6iqQ7g6N1U1p68865CklbJeUFz3dl8Dwfmtk/zKxXc+dr9N4mBp+zMvgeMlv9aQAA2jVCHgAg\nHo2Q9Avn3BjnXImku5xzkyVNkPQZMzurmed0k/Sac26CpHcltdhK6Zw7V9J/SLoneNe/Sipxzo2R\n9MPg6zR+TrmklyTtMLN5ZjbDzOrC6UOSfhqs8TpJjznntkp6VNIDzrmJzrl3W6rHzAbK+zd5TfCu\npc65Kc65SZKekfTvEZzvcUmzg+9/k6T/bOn1AACJLaJrHgAAaGNbnHPLG+zfELz+LUXeaNcoSRsa\nPeewc+6l4O0PJU1t4dxPNzhmQPD2VElzJMk5t9rM1jb3ROfcl8xsjKTLJN0l6RJJXw3uD28Q+rqZ\nWdrJ36ZuNLPL5IXarznnjgfvH2BmP5eUK6mTvNDWIjPrKSnNOfdO8K65kv4cwesDABIQI3kAgHh0\nqO6GmQ2VdLukAufceEkvSkpv5jnHGtyuVcu/yDzazDHW6JjG+/Wcc2ucc7+UNE3S1Q2OP8c5lx/c\nznTOHW3pHA084ZwbJ+kCSQ/WtZhK+o2kB4OPfUPNv9/G9bZYMwCgYyHkAQDiUcPAkiWpUtJBM8uT\n9NkInvP/27tjVo6iMI7j3ycWxRuwGwySWd6GDQOZvAHKoCw25Q2YzIySQVkYxMJskcUgpSyO4T64\n/f/4Z8Lx/dSt0+255947/rrnPPe7jmmWWRIRY8Bo1+QRQxEx1To1AVzn+IBmyedr7XgOH/L5v1RK\nOQV2aMIsec1Nfhmca5V+OF8p5Q54bO37mwGOet1XklQnQ54k6Td666aZyzav8timCWRddR3jnvN2\n2AKGs9HLKnAJ3HfUBLCcDVvOgBXe9/0tAZMRcZFzLOT5PWA6G6h82nglbQDzETEArAG7wAlw26rp\nnK/9PrPAZkSc04TU9R73kyRVyl8oSJL+vYjoA/pLKU+5PHQfGCmlPP/wo0mS9G02XpEkCQaBw9ZP\n2BcNeJKkv8oveZIkSZJUEffkSZIkSVJFDHmSJEmSVBFDniRJkiRVxJAnSZIkSRUx5EmSJElSRQx5\nkiRJklSRF1Mmsd9I1uucAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1e38f897f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "#crete the figures using collected data \n",
    "plt.figure(figsize=(15,6))\n",
    "tr_acc, = plt.plot(plain_trainRatios,plain_tr_accs,label=\"Training Accuracy\");\n",
    "test_accuracy, = plt.plot(plain_trainRatios,plain_tst_accs,label=\"Test Accuracy\");\n",
    "plt.legend(handles=[tr_acc, test_accuracy])\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Training Set Ratio')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3kAAAF/CAYAAAD0NgDhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8lOW9///3lZUEkpCFkLAYFgGRNSAIFTCACyK4WwWl\nfnvsctp69Ge/tnbzp/ZYy2lrF097Wq3a4lKlx1YtgqKgQXFXEpB9x4QlJCxJWAJZru8fdzKTYZIw\nhJncM5PX8/GYRzKTe+75JCOSN5/PfV3GWisAAAAAQHSIcbsAAAAAAEDwEPIAAAAAIIoQ8gAAAAAg\nihDyAAAAACCKEPIAAAAAIIoQ8gAAAAAgioQ85BljZhhjNhpjNhtj7m3h6+cYY5YZY1YbY94yxvQK\ndU0AAAAAEK1MKPfJM8bESNosabqkPZI+kXSztXZjs2P+Lulf1tpnjTEFkv7NWvuVkBUFAAAAAFEs\n1J288ZK2WGt3WWtrJb0g6epTjjlf0luSZK0tbOHrAAAAAIAAhTrk9ZZU0ux+aeNjzRVLul6SjDHX\nSepmjEkPcV0AAAAAEJVCHfJMC4+dOh/6PUkFxpjPJE2WtFtSXYjrAgAAAICoFBfi85dKOqfZ/T5y\nrs3zsNbulbeT11XS9dba6lNPZIwJ3cWDAAAAABABrLUtNdJ8hLqT94mkc40xecaYBEk3S/pX8wOM\nMZnGmKZCfyjpqdZOZq3lFsa3+++/3/UauPH+RPKN9yj8b7xH4X3j/Qn/G+9ReN94f8L/FqiQhjxr\nbb2kOyS9IWmdpBestRuMMQ8aY2Y1HlYgaZMxZqOkbEk/C2VNAAAAABDNQj2uKWvt65KGnPLY/c0+\n/4ekf4S6DgAAAADoDEK+GTo6j4KCArdLQBt4f8If71H44z0Kb7w/4Y/3KLzx/kSPkG6GHkzGGBsp\ntQIAAABAsBljZANYeCXk45oAAABAuOjXr5927drldhlAm/Ly8rRz5852P59OHgAAADqNxk6I22UA\nbWrtv9NAO3lckwcAAAAAUYSQBwAAAABRhJAHAAAAAFGEkAcAAABEoYaGBqWkpKi0tDSoxyL8EfIA\nAACAMJCSkqLU1FSlpqYqNjZWycnJnseef/75Mz5fTEyMqqur1adPn6Ae215PPPGEYmJi9PLLL4fs\nNdx26623KjExUampqZ737qWXXpIkPfroo7rggguUmJiob3zjGyGtg5AHAAAAhIHq6mpVVVWpqqpK\neXl5Wrx4seexOXPm+B1fX1/vQpXt9/TTTyszM1MLFizo8NduaGjokNcxxujHP/6xqqqqPO/dtdde\nK0nq06eP7r//fn31q18NeR2EPAAAACDMWGv9ltC/7777dPPNN2vu3LlKS0vTc889pw8//FATJ05U\nenq6evfurbvuussT/urr6xUTE6MvvvhCkjRv3jzdddddmjlzplJTU3XRRRd59gw8k2Ml6bXXXtOQ\nIUOUnp6uO++8U5MmTdLTTz/d6vezfft2vf/++3rssce0ZMkSHThwwOfr//znP5Wfn6+0tDQNHjxY\ny5YtkyQdPHhQX/3qV9WrVy9lZmbqxhtvlCQ9+eSTmjp1quf5LdV/xx136IorrlBKSopWrlypRYsW\nKT8/X6mpqerXr58eeughnxreeecdTZw4Ud27d1deXp7n59u7d2+f4xYuXKhx48a19fa16LrrrtPs\n2bOVnp5+xs89U4Q8AAAAIEK8/PLLuvXWW1VZWambbrpJ8fHxevTRR3Xw4EG99957Wrp0qR577DHP\n8cb4bqn2/PPP62c/+5kOHTqkvn376r777jvjY/fv36+bbrpJjzzyiCoqKtS/f3998sknbda9YMEC\nTZgwQdddd50GDhzoM376/vvv6/bbb9dvfvMbVVZW6u2331ZeXp4kac6cOaqtrdXGjRtVVlamu+66\nq9V6W6r/wQcfVHV1tSZMmKCUlBQ999xzqqqq0qJFi/Too49qyZIlkqQdO3Zo1qxZuueee3Tw4EEV\nFRVpxIgRmjBhglJTU7V8+XLPeZ977jnddtttbX6/biPkAQAAABFi0qRJmjlzpiQpMTFRY8eO1bhx\n42SMUb9+/fT1r39dK1as8Bx/ajfwhhtuUH5+vmJjY3XLLbeouLj4jI9dvHix8vPzNWvWLMXGxuru\nu+9WZmZmm3U/++yzuuWWWyRJc+fO9RnZfOqpp/SNb3xDBQUFkqTevXtr0KBBKi0t1dtvv60//elP\nSk1NVVxcnCZNmtTqa5xa/7XXXqvx48dLkhISElRQUKDzzz9fkjRixAjddNNNnp/Vc889pyuuuELX\nX3+9YmJilJGRoZEjR0pyuoLPPPOMJKmiokJvvfWWbr755lbr+PnPf66MjAylp6erV69ebf5cQiWi\nQt7WrW5XAAAAgGhnzNnfQqVv374+9zdt2qRZs2YpNzdXaWlpuv/++1VRUdHq83NycjyfJycn68iR\nI2d87J49e/zqaGvBlhUrVqi0tNQzajlnzhx99tlnWr9+vSSppKREAwcO9HteSUmJsrKy1K1bt1bP\n3ZZTa/zggw80depUZWdnq3v37nryySc9P6vWapCckPfKK6+opqZGL7zwgqZOnaqsrKxWX/eHP/yh\nDh48qEOHDmnPnj3tqv1sRVTImzxZGjpU+v73pXfflerq3K4IAAAA0cbas7+Fyqkjid/85jc1YsQI\nbd++XZWVlXrwwQf9OlrBlpubq5KSEp/Hdu/e3erxCxYsUENDg0aOHKnc3FxNmjRJMTExnmv4+vbt\nq23btvk9r2/fvqqoqGgxiHbt2lXHjh3z3N+7d+9pxzfnzJmjG2+8Ubt379bhw4d1++23e35Wffv2\n1dZWOkp9+/bVBRdcoJdfflnPPvus5s2b1+r3Gi4iKuTt3i09/bTUpYt0551STo40b570979LlZVu\nVwcAAAB0rOrqaqWlpSkpKUkbNmzwuR4vVGbNmqWioiItXrxY9fX1+u1vf9tq9/D48eP6xz/+oaee\nekrFxcVavXq1Vq9erV//+td65plnZK3V7bffrieeeEIrVqyQtVa7d+/W5s2b1adPH11yySX6zne+\no8rKStXV1endd9+VJI0aNUpr1qzRunXrdPz4cf30pz89bd1HjhxRenq64uPj9eGHH+qFF17wfO3W\nW2/V0qVL9dJLL6m+vl4HDhzQmjVrPF+fN2+efv7zn2vTpk26+uqr2/Vzq6+vV01Njerr61VXV6cT\nJ06EbNXPiAp5MTHSuHHST38qFRU5ty99SfrrX6W+faVLLpF+9ztp+3a3KwUAAADa79QuVGseeeQR\n/fWvf1Vqaqq+9a1v+V0r1vw8pztnoMdmZ2dr4cKFuvvuu5WVlaUdO3YoPz9fiYmJfsf+85//VGpq\nqm655RZlZ2d7bl//+tdVU1OjN998UxMnTtSf//xn/cd//IfS0tI0bdo0z6bszz77rKy1Gjx4sHJy\ncvT73/9ekjR06FD96Ec/0sUXX6yhQ4fq4osvbvV7afLHP/5RP/jBD5SWlqb58+frpptu8nytX79+\nWrRokebPn6+MjAyNHTtWa9eu9Xz9+uuv1/bt23XjjTe2+H0G8nN74IEHlJycrEceeUQLFixQcnKy\n5s+f3+rxZ8OEup0bLMYY21atR49Kb74pLVokLV4sZWZKs2c7twkTpNjYDiwWAAAAYckYE/Jxxs6m\noaFBvXr10j/+8Q9ddNFFbpcTMv3799eCBQs0ZcqUkL9Wa/+dNj5+2n8BiKhOXlu6dpWuuUZ68klp\nzx7pqaekuDjp2992xjpvu0168UWpqsrtSgEAAIDItnTpUlVVVenEiRP66U9/qvj4eM9KltFo4cKF\n6tKlS4cEvGCIc7uAUIiJkS680Lk99JC0a5f06qvSE09I//ZvTmevqcvXr5/b1QIAAACRZeXKlbrl\nlltUW1urYcOG6ZVXXlF8fLzbZYXE5MmTtXXrVv3tb39zu5SARc24ZqCqq33HOnv29Aa+8eMZ6wQA\nAIhmjGsiEpztuGanC3nN1ddLH3/sBL5Fi6T9+6WZM53Ad9llUju35AAAAECYIuQhEhDygmjHDmes\nc9Ei6cMPnZU7m7p855wT0pcGAABAByDkIRIQ8kKkqkp64w0n8C1ZIvXq5Q1848Y51/0BAAAgshDy\nEAkIeR2gvt7p7DWNdR44IF15pRP4Lr3UWdkTAAAA4Y+Qh0hAyHPBtm3esc6PP5YmT3YC36xZUp8+\nblcHAACA1hDyEAkIeS6rrJSWLvWOdeblecc6x4xhrBMAACCcEPIQCdgM3WVpadKXvyw984xUVib9\n9rfSsWPSrbc6Xb1vfMMJgMeOuV0pAAAAwllKSopSU1OVmpqq2NhYJScnex57/vnn233eiRMnBrTH\nW2VlpZKSknTDDTe0+7XC3dKlSxUbG6vU1FTPz/amm26SJBUVFenyyy9XVlaWkpOTXa707BDygigu\nTpoyRfrlL6WNG6UVK6TzzpN+/WspJ8fp7j3+uLRnj9uVAgAAINxUV1erqqpKVVVVysvL0+LFiz2P\nzZkzJ+Svv3DhQnXt2lWLFy/WoUOHQv56zdXX13fYaw0cOFBVVVWen+3ChQslSYmJiZo7d64ee+yx\nDqslVAh5ITRokPTd70pvvy3t2iXNnSsVFkrDh0sXXCA9+KC0apXExAAAAACas9b6jes1NDToP//z\nPzVw4EBlZ2dr3rx5qqqqkiQdO3ZMc+bMUWZmptLT0zVx4kRVVlbqnnvu0SeffKKvfe1rSk1N1fe+\n971WX3PBggW6++67NXDgQL/O4a5du3TNNdeoR48eys7O9jnP//zP/2jo0KFKTU3VqFGjtG7dOp04\ncUIxMTHa06y7MWfOHD388MOSnI7aoEGD9NBDDyknJ0ff/va3VVFRoZkzZyo7O1tZWVm65pprVFZW\n5nn+gQMHdNtttyk3N1eZmZme4Dto0CAtX77cc9yJEyfUvXt3bdq06Yx+5ueff75uu+02nXfeeWf0\nvHBEyOsg6enSnDnS3/7mjHX+6lfONg033yz17Sv9+79LixdLx4+7XSkAAADC0S9+8QstW7ZM77//\nvkpLSxUfH6+7775bkvTEE0+ovr5ee/fu1YEDB/T73/9eCQkJ+tWvfqVx48bpySefVFVVlX75y1+2\neO4tW7boo48+0ty5czV37lw9/fTTnq/V1dXpiiuu0LBhw1RSUqKSkhJdf/31kqRnnnlGv/zlL7Vw\n4UJVVVXpxRdfVHp6uiTn+rG27Ny5U/X19SotLdWjjz6qhoYGfetb31Jpaal27NghY4zn+5OkL3/5\ny5KkzZs3q6ysTN/5znckSbfddpueeeYZz3GvvPKKhgwZoiFDhpzpjzhqEPJcEB8vFRRIjzwibd4s\nLV8unXuu9ItfSD17SldfLT3xhLR3r9uVAgAAIFw8/vjjmj9/vnr27KmEhATdd999euGFFyRJ8fHx\nKi8v15YtWxQTE6OxY8cqKSnJ89zTLTazYMECjR8/Xv3799fcuXP16aefejphK1euVHV1tX72s5+p\nS5cuSkxM1IQJEyRJTz75pH784x9r5MiRkpyuWq9evQJ6zS5duugnP/mJ4uLilJiYqOzsbM2ePVsJ\nCQlKSUnRvffeqxUrVkiSduzYoffee09//OMflZKSori4OE2aNEmSNG/ePL388suqqamR5ATPefPm\ntfq627dvV0ZGhtLT05WRkaFXX321zTojUZzbBUAaMsS53XOPdPCg9NprzmIt3/ueM/LZtFrnqFHS\naf5BBAAAAGfJPHj2v3DZ+4N/PU5JSYlmzpzp6ZA1haiDBw/q9ttv1759+3TDDTfo6NGjmjdvnh56\n6KHTdtOaPPvss54RzH79+unCCy/UggUL9PDDD6ukpET9+/dvtaYBAwa06/vJyclRbGys5/6RI0d0\n1113admyZaqsrJS11hPcSktLlZ2d3eKCKHl5eRozZoxeeeUVTZ8+XW+99Zb+8pe/tPq6AwYM0ObN\nm9tVc6Qg5IWZjAzpllucW22t9O67TuC74Qbp5ElnL77Zs6WpU6UuXdyuFgAAIPqEIqAFQ58+ffTP\nf/5T+fn5LX79wQcf1IMPPqidO3fqsssu0/DhwzVnzpzTBr23335bX3zxhR544AE99NBDkpzAVVJS\noocfflh9+/bVzp07W3xu3759tW3bNk2bNs3n8YSEBMXHx+tYsyXm9+3bpxEjRnjun1rX/PnztXv3\nbn322WfKysrSRx99pMmTJ3teZ//+/Tp27FiLQe8rX/mKnnnmGZWXl2v69OnKyspq83uOdoxrhrH4\neGnaNOk3v5G2bHH24+vXT3r4YWes89prpaeecq7xAwAAQHT75je/qXvvvVelpaWSpP3793tGDZcv\nX64NGzbIWqtu3bopLi5OcXFOP6dnz57avn17q+f961//qtmzZ2v9+vVavXq1Vq9ereLiYh04cEDL\nly/XpEmTlJKSovvuu0/Hjx9XTU2NPvjgA0nS1772Nc2fP19r1qyR5Fzbt2fPHhljNHLkSD333HNq\naGjQokWLPM9pTXV1tZKTk5WamqqKigpP4JSc7uKUKVN0xx13qKqqSrW1tXr33Xc9X7/hhhu0cuVK\n/elPf9JXvvKVdvx0HSdOnNCJEydkrdWJEydUW1vb7nO5iZAXIYyRhg6Vvv99p7u3bZt03XXS6687\no54TJkg/+5m0Zg2rdQIAAES6lrpv9957ry699FJNmzZNaWlpmjRpkoqKiiRJu3fv1tVXX63U1FSN\nHDlSs2bN0o033ihJuvvuu7VgwQJlZmbqBz/4gc85jx49qpdeekl33XWXZ+XM7OxsDRw4UHPmzNGC\nBQsUFxenJUuWqLi4WH369FFeXp5eeuklSdKtt96q7373u7rhhhuUlpamG2+8UYcPH5Yk/fd//7de\neOEFZWRk6JVXXtHs2bPb/J7vuecelZeXKzMzU1OmTNGVV17p8/Xnn39eJ0+e1KBBg5Sbm6s//elP\nnq9169ZNs2fP1u7du3XVVVed4U/bsWnTJiUlJWncuHE6efKkkpOTNWrUqHady23mdBdEhgtjjI2U\nWjvayZPSO+84Y52LFkkNDd6xzoICKTHR7QoBAADCgzHmtAuCIDL9+Mc/Vnl5uR5//HG3Szlrrf13\n2vj4aS+0JORFGWul9eu9gW/dOmn6dCfwXXml1KOH2xUCAAC4h5AXncrLyzV69Gi98soruuCCC9wu\n56ydbchjXDPKGCMNGyb94AfSe+851/JddZX06qvOSp1f+pL0859La9cy1gkAAIDI94c//EH9+/fX\nTTfdFBUBLxjo5HUiJ05IK1Z4u3wxMd6xzosvlhIS3K4QAAAgtOjkIRIwrol2sdbp5jUFvg0bpEsv\ndQLfzJlSJ191FgAARClCHiIBIQ9BUVYmLVniBL7ly6URI7ybsA8dyibsAAAgOhDyEAkIeQi6mhqp\nsFD617+ca/ni472Bb8oU5z4AAEAkIuQhEhDyEFLWSqtXe8c6t2yRLr/cCXxXXCFlZLhdIQAAQOAI\neYgEhDx0qL17pcWLncD39ttSfr63yzdkiNvVAQAAtK1fv37atWuX22UAbcrLy9POnTv9HifkIeSO\nH5feessJfK++KiUnewPfRRcx1gkAAAAEEyEPHcpaqajIO9a5fbs0Y4YT+GbMkNLT3a4QAAAAiGyE\nPLhqzx6nu7dokbM339ix3i7foEFuVwcAAABEHkIewsaxY862DE1jnamp3sD3pS9JcXFuVwgAAACE\nP0IewlJDg7RqlXesc9cuZ5XOprHOtDS3KwQAAADCEyEPEaG01DvW+e670rhx3i7fwIFuVwcAAACE\nD0IeIs7Ro9KyZd6xzowMb+CbOFGKjXW7QgAAAMA9hDxEtIYG6dNPvWOdu3d7xzovv9y5rg8AAADo\nTAh5iCpffOEd61y5Upowwdvl69/f7eoAAACA0CPkIWpVV0tvvukEvsWLpexsb+C78ELGOgEAABCd\nCHnoFOrrpY8/9o51lpVJM2c6ge+yy6SUFLcrBAAAAIKDkIdOaedOb+D74APpooucwDdrlpSX53Z1\nAAAAQPsR8tDpVVVJb7zhBL4lS6RevbxjnePGSTExblcIAAAABI6QBzRTXy99+KG3y3fggHTllU7g\nu/RSqWtXtysEAAAA2kbIA9qwfbs38H38sTRpkness29ft6sDAAAA/BHygABVVkpLlzqB77XXnJDX\nNNY5dixjnQAAAAgPhDygHerqnAVbmrp8hw873b3Zs6VLLpGSk92uEAAAAJ0VIQ8Igq1bvYHv00+l\nKVO8Y529e7tdHQAAADoTQh4QZIcPS6+/7gS+11+X+vXzjnWOGSOZ0/5xAwAAANqPkAeEUF2d9N57\n3i7fkSPesc7p06WkJLcrBAAAQLQh5AEdaPNmb+BbtUoqKPCOdebmul0dAAAAokHYhDxjzAxJv5UU\nI+lJa+1/nfL1vpIWSOreeMwPrbWvtXAeQh4iwsGD3rHOpUulgQO9Y52jRzPWCQAAgPYJi5BnjImR\ntFnSdEl7JH0i6WZr7cZmxzwmaZW19jFjzFBJS6y1/Vs4FyEPEae2Vlq50tvlq6nxjnVOmyZ16eJ2\nhQAAAIgUgYa8UO8ANl7SFmvtLmttraQXJF19yjENklIbP+8uaXeIawI6THy8NHWq9OtfOyOdb74p\nDRgg/fznUs+e0jXXSE8+Ke3b53alAAAAiBah7uRdL+lya+03Gu/fKmm8tfbOZsfkSHpDUrqkZEmX\nWGuLWjgXnTxElQMHpCVLnA7fG29IQ4Z4xzpHjmSsEwAAAL4C7eTFhbqOFh47NanNkfQXa+1vjDET\nJD0raVhLJ3vggQc8nxcUFKigoCA4VQIuyMyU5s1zbidPSu+84wS+a6+V6uu9Y51Tp0qJiW5XCwAA\ngI5WWFiowsLCM35eqDt5EyQ9YK2d0Xj/B5Js88VXjDFr5XT7djfe3ybpQmttxSnnopOHTsFaaf16\n73V8a9c62zLMni1deaWUne12hQAAAHBDuCy8Eitpk5yFV/ZK+ljSHGvthmbHLJb0d2vtgsaFV960\n1vZp4VyEPHRK5eXesc5ly6ShQ71jncOHM9YJAADQWYRFyGssZIak38m7hcJ8Y8yDkj6x1r7aGOz+\nLKmbnEVYvmetXd7CeQh56PROnJBWrPB2+YzxBr6LL5YSEtyuEAAAAKESNiEvWAh5gC9rnVHOpsC3\nYYN06aVO4Js5U8rKcrtCAAAABBMhD+hk9u+XFi92At/y5dKIEd4u39ChjHUCAABEOkIe0InV1EiF\nhd4uX3y8N/BNnsxYJwAAQCQi5AGQ5Ix1rlnjDXybN0uXXeYEviuucLZyAAAAQPgj5AFo0b593rHO\nt9+WRo3ydvmGDGGsEwAAIFwR8gCc1vHjTtBr6vIlJXkD36RJzpgnAAAAwgMhD8AZsVYqLvYGvm3b\npMsv9451pqe7XSEAAEDnRsgDcFb27PGOdRYWSmPGeLt8gwe7XR0AAEDnQ8gDEDTHjklvveUEvldf\nlbp18wa+iy6S4uLcrhAAACD6EfIAhERDg1RU5B3r3LlTmjHDCXwzZkjdu7tdIQAAQHQi5AHoELt3\nO929RYukd96RLrjA2+U791y3qwMAAIgehDwAHe7oUWnZMu9YZ3q6N/BNnMhYJwAAwNkg5AFwVUOD\n9Omn3rHO0lJnlc7Zs51VO9PS3K4QAAAgshDyAISVL77wjnWuXCldeKF01VVO6Ovf3+3qAAAAwh8h\nD0DYOnJEevNNJ/AtXiz16OEd67zwQik21u0KAQAAwg8hD0BEaGiQPv7YO9a5b580c6YT+C67TEpJ\ncbtCAACA8EDIAxCRdu70jnV+8IGzYEtTly8vz+3qAAAA3EPIAxDxqqulN95wAt+SJVJOjjfwjR8v\nxcS4XSEAAEDHIeQBiCr19dJHH3nHOsvLpSuvdALfpZdK3bq5XSEAAEBoEfIARLXt271jnR99JF10\nkRP4Zs2SzjnH7eoAAACCj5AHoNOorPQd6+zTxzvWecEFjHUCAIDoQMgD0CnV1zsLtjSNdR465B3r\nvOQSqWtXtysEAABoH0IeAEjautUb+D79VJo82TvW2aeP29UBAAAEjpAHAKc4fFh6/XUn8L3+urMl\nQ9NY55gxjHUCAIDwRsgDgDbU1Unvveft8lVXO9292bOl6dOl5GS3KwQAAPBFyAOAM7B5szfwrVol\nXXyxd6yzVy+3qwMAACDkAUC7HTokvfaaE/iWLpUGDPCOdebnS+a0/2sFAAAIPkIeAARBba20cqW3\ny3f8uHesc9o0KSnJ7QoBAEBnQcgDgCCzVtq0yRv4ioulqVO9Y505OW5XCAAAohkhDwBC7MAB71jn\nG29IgwZ5xzpHjWKsEwAABBchDwA60MmT0rvvert8tbXewFdQIHXp4naFAAAg0hHyAMAl1kobNngD\n3+efO9syzJ4tXXmllJ3tdoUAACASEfIAIExUVEhLljiB7803paFDvV2+4cMZ6wQAAIEh5AFAGDp5\nUlqxwtvls9Yb+C6+WEpMdLtCAAAQrgh5ABDmrJXWrfMGvvXrpUsucQLfzJlSjx5uVwgAAMIJIQ8A\nIsz+/d6xzmXLnFHOpi7f+ecz1gkAQGdHyAOACHbihFRY6AS+f/1LiovzBr4pU6SEBLcrBAAAHY2Q\nBwBRwlppzRrvWOemTdJll3nHOjMz3a4QAAB0BEIeAESpffukxYudwPfWW87G601dvvPOY6wTAIBo\nRcgDgE6gpsYJeosWSa++6qzO2RT4Jk+W4uPdrhAAAAQLIQ8AOhlrpeJi71jn1q3S5Zc7ge+KK6SM\nDLcrBAAAZ4OQBwCd3J493rHOwkIpP9/b5RsyxO3qAADAmSLkAQA8jh+Xli/3jnV27eoNfJMmOat3\nAgCA8EbIAwC0yFpp1SrvWOeOHdKMGd6xzu7d3a4QAAC0hJAHAAjI7t1Od2/RIumdd6SxY71dvkGD\n3K4OAAA0IeQBAM7Y0aO+Y53du3sD38SJjHUCAOAmQh4A4Kw0NEiffeYd6ywpccY5Z892Vu1MS3O7\nQgAAOhdCHgAgqEpKvGOdK1dK48d7u3wDBrhdHQAA0Y+QBwAImSNHpGXLvGOdWVnewDdhghQb63aF\nAABEH0IDTgCqAAAgAElEQVQeAKBDNDRIn3ziHevcs0eaOdMJfJddJqWmul0hAADRgZAHAHDFrl3e\nsc733nMWbGnq8vXr53Z1AABELkIeAMB11dXSm286gW/xYqlnT2/gGz+esU4AAM4EIQ8AEFbq66WP\nP/aOde7f7zvW2a2b2xUCABDeCHkAgLC2Y4c38H34oXTRRd4u3znnuF0dAADhh5AHAIgYVVXS0qVO\n4FuyROrd2xv4xo2TYmLcrhAAAPcR8gAAEam+XvrgA2+X7+BB6corncB36aVS165uVwgAgDsIeQCA\nqLBtmzfwffyxNHmyt8vXp4/b1QEA0HEIeQCAqHP4sHes87XXnGv3mgLf2LGMdQIAohshDwAQ1erq\npPff93b5qqq8Y52XXCIlJ7tdIQAAwUXIAwB0Klu2eAPfZ59JU6Y4gW/WLGchFwAAIh0hDwDQaR06\nJL3+uhP4Xn9d6t9fuuoqJ/Tl50vmtH89AgAQfgh5AABIqq2V3nvP2+U7dszp7s2eLU2bJiUluV0h\nAACBIeQBANCCTZu8ga+oSJo61Ql8V14p5ea6XR0AAK0j5AEAcBoHDzqrdC5a5KzaOWiQd7XOUaMY\n6wQAhBdCHgAAZ6C2Vnr3XW+X7+RJ71jn1KlSly5uVwgA6OwIeQAAtJO10oYN3sD3+efOap25uVLX\nrs72DMnJ3s9b+9j888REOoMAgLMTNiHPGDND0m8lxUh60lr7X6d8/deSpkqykrpK6mGtzWjhPIQ8\nAIArKiqkt992xjuPHZOOHnU+Nv+8tY9Nn9fVBR4I23NMcrIUG+v2TwoAEEphEfKMMTGSNkuaLmmP\npE8k3Wyt3djK8XdIGm2t/VoLXyPkAQAiVl3d6YNhIKGxrWMTEs4sNJ5pwKQbCQDuCjTkxYW4jvGS\ntlhrdzUW9YKkqyW1GPIkzZH0/4e4JgAAOlxcnJSa6txCwVqppubMQuORI1JZWeAB83TdyLMNmHQj\nASA4ThvyGrtrz1lrD7Xj/L0llTS7Xyon+LX0OudI6ifprXa8DgAAnZoxzp5/SUlSZmZoXiOQbuSp\nAXP//vZ3I0PRlUxIoBsJIPoF0snLkfSJMWaVpKckLT2DucmW/jfa2nNvlvQiM5kAAIQnN7qRp+tK\nhqob2d5rJelGAggHpw151tqfGGPuk3SZpK9K+r0x5u9yFlHZdpqnl0o6p9n9PnKuzWvJzZK+3dbJ\nHnjgAc/nBQUFKigoOM3LAwCASBGJ3ciWQmhL3chgdiXpRgKdR2FhoQoLC8/4eQEvvGKMGSUn5M2Q\n9LakCZLetNZ+v43nxEraJGfhlb2SPpY0x1q74ZTjhkh6zVo7oI1z0eQDAABhrT3dyDM9tr7eGwJD\n0ZWkGwmEr6CtrmmMuVPSbZIqJD0h6WVrbW3jyplbrLUDT/P8GZJ+J+8WCvONMQ9K+sRa+2rjMfdL\nSrTW/qiN8xDyAABAp9fUjWzPVh6BHttWNzIY237QjQTaJ5gh76dywtmuFr429NSuXKgQ8gAAAEKv\nrW5ksLqSzbuRoVhgJymJbiSiUzBD3gRJ66y11Y33UySdb639KCiVBoiQBwAAEB3a0408085lIN3I\ns+lK0o2EG4IZ8ookjWlKWI1jmp9aa8cEpdIAEfIAAAAQiEC6kWcbMFvqRgZjlJVuJNoSzM3QfdKV\ntbbBGBPqTdQBAACAdunIlVrPJDQ2bfcRyLHHjkmJiaFbYIduZHQLJKxtb1x85Y+N978taXvoSgIA\nAADCm5v7Rrb2sfm+kWfbjQzGtZJ0I90TyLhmtqRHJU2Ts5H5ckn/n7V2f+jL86mDcU0AAAAgSGpr\npePHQ7fATvNuZCgW2ElO7nzdyKBdkxcuCHkAAABA5GjqRoZygZ1AupFnc61kuHUjg7nwShdJt0sa\nJqlL0+PW2n872yLPBCEPAAAAQHOn60ae7X6SLXUjg92VPJNuZDAXXnlG0kZJl0v6qaRbJHXI3ngA\nAAAA0Jr4eOcW6msjzyQ0Vlc710YGGjAbGgIPjYEKJOSda6290RhztbV2gTHmb5Lebe8PCgAAAAAi\nQfOVWkMl0G7k0aOBnzOQkFfb+PGwMWa4pH2Sss+8fAAAAABAc2fSjbzzzsDOGUjIe9wYky7pJ5L+\nJambpPsCOz0AAAAAoCO1GfKMMTGSqqy1hyS9I2lAh1QFAAAAAGiXmLa+aK1tkPT9DqoFAAAAAHCW\nAtlCYb6kCkkLJXku97PWHgxtaX51sIUCAAAAgE4rmPvk7WjhYWut7dDRTUIeAAAAgM4saCEvXBDy\nAAAAAHRmQdsM3RjzlZYet9Y+3Z7CAAAAAAChE8gWCuOafd5F0nRJqyQR8gAAAAAgzJzxuKYxJk3S\nQmvtjNCU1OrrMq4JAAAAoNMKdFyzzS0UWnFMUv92PA8AAAAAEGKBXJO3SFJTCy1G0vmS/h7KogAA\nAAAA7RPIFgoXN7tbJ2mXtbY0pFW1XAfjmgAAAAA6raCtrinpC0l7rbU1jSdOMsb0s9buPMsaAQAA\nAABBFsg1ef8rqaHZ/frGxwAAAAAAYSaQkBdnrT3ZdKfx84TQlQQAAAAAaK9AQl65MeaqpjvGmKsl\nVYSuJAAAAABAewWy8MpASc9J6tX4UKmkr1hrt4a4tlPrYOEVAAAAAJ1WoAuvBLwZujGmW+Px1Wdb\nXHsQ8gAAAAB0ZkHbDN0Y87Axpru19oi1ttoYk26MeSg4ZQIAAAAAgimQa/KusNYebrpjrT0kaWbo\nSgIAAAAAtFcgIS/WGJPYdMcYkyQpsY3jAQAAAAAuCWQz9GclLTfG/KXx/lclLQhdSQAAAACA9gpo\n4RVjzAxJl0gykg5JyrXWfifEtZ1aAwuvAAAAAOi0grbwSqN9khokXS9puqQNZ1EbAAAAACBEWh3X\nNMYMlnSzpDmSDkhaKKfzN7WDagMAAAAAnKFWxzWNMQ2S3pV0e9PG58aY7dbaAR1YX/N6GNcEAAAA\n0GkFY1zzejljmm8bY/5sjJku55o8AAAAAECYOu3CK8aYrpKukTO2OU3OypovWWvfCH15PnXQyQMA\nAADQaQXayQtodc1mJ82QdKOkm6y1086ivjNGyAMAAADQmYUk5LmJkAcAAACgMwv2FgoAAAAAgAhA\nyAMAAACAKELIAwAAAIAoQsgDAAAAgChCyAMAAACAKELIAwAAAIAoQsgDAAAAgChCyAMAAACAKELI\nAwAAAIAoQsgDAAAAgChCyAMAAACAKELIAwAAAIAoQsgDAAAAgChCyAMAAACAKELIAwAAAIAoQsgD\nAAAAgChCyAMAAACAKELIAwAAAIAoQsgDAAAAgChCyAMAAACAKELIAwAAAIAoQsgDAAAAgChCyAMA\nAACAKELIAwAAAIAoQsgDAAAAgCgS8pBnjJlhjNlojNlsjLm3lWO+bIxZZ4z53BjzbKhrAgAAAIBo\nZay1oTu5MTGSNkuaLmmPpE8k3Wyt3djsmHMlLZQ01VpbZYzJstZWtHAuG8paAQAAACCcGWNkrTWn\nOy7UnbzxkrZYa3dZa2slvSDp6lOO+bqkP1hrqySppYAHAAAAAAhMqENeb0klze6XNj7W3GBJQ4wx\nK40x7xtjLg9xTQAAAAAQteJCfP6WWomnzlzGSTpX0hRJ50h61xgzrKmzBwAAAAAIXKhDXqmc4Nak\nj5xr80495gNrbYOkncaYTZIGSfrs1JM98MADns8LCgpUUFAQ5HIBAAAAIDwUFhaqsLDwjJ8X6oVX\nYiVtkrPwyl5JH0uaY63d0OyYyxsf+z/GmCw54W60tfbQKedi4RUAAAAAnVZYLLxira2XdIekNySt\nk/SCtXaDMeZBY8ysxmOWSjpgjFknabmke04NeAAAAACAwIS0kxdMdPIAAAAAdGZh0ckDAAAAAHSs\nUC+8AgAAACDMWGvVYBs8Nyvf+z5fsy1/rT3Paet57XlORNQXxPMFipAHAAAQJay1nl8Oo/EXZte+\nJ0XOzyjQ+iQpxsQoxsTIyHg+P/VmTMtfa89z2npee57j9zyd/fliY2LD9vs1xuhCXRjQ/wsIeQAA\nRIDmv7xHwi+QEVdflHy/VlZGxr1fskP5S/FZni8uJi6yvt8Q12fMaS/rQgQj5AEA0IKT9Se1bv86\nFe0rUtHeIq0tX6uauhrXQoCkFn+xi9pfVAM8X/Nf3MPh+3X7Z9sU8AB0bqyuCQDo9I6cPKLV+1Zr\n1d5VTqjbV6RNFZs0IH2A8nPzlZ+TrxHZI9QtoZt7oYdf3AGg0wt0dU1CHgCgUyk/Wu7pzjUFutKq\nUg3rMUz5OfkakztG+blOqEuKT3K7XAAAPAh5AIBOzVqrXZW7fMJc0d4iHa09qtE5o5Wf43To8nPz\ndV7WeYqL4QoGAEB4I+QBADqN+oZ6bTqwyRPoVu1dpeJ9xUqKT/IJc/k5+erXvR+jjwCAiETIAwBE\npZq6Gn1e9rnPyOXa/WuVm5LrCXRjcsdodM5o9ezW0+1yAQAIGkIeACDiHa45rOJ9xT4jl9sObtPg\nzMGezlx+Tr5G5YxSamKq2+UCABBShDwAQETZW73Xpzu3au8qlR8r18ieI31GLof1GKbEuES3ywUA\noMMR8gAAYanBNmj7oe1+C6LUNdT5dOfG5I7RuRnnKjYm1u2SAQAIC4Q8AIDrautrtb58vU+HbnXZ\nanXv0t1vQZQ+qX1YEAUAgDYQ8gAAHeroyaNaU7bGJ9CtL1+vvO55zt5zjaFudM5oZSZnul0uAAAR\nh5AHAAiZA8cO+G0ovuvwLp3f43yf7tzIniPVNaGr2+UCABAVCHkAgLNmrVVJVYnf9XOVJyr9NhQf\nmjVU8bHxbpcMAEDUIuQBAM5IfUO9thzc4hfo4mLiPJ25prHL/un9FWNi3C4ZAIBOhZAHAGjViboT\nWrt/rc/I5ef7P1d212y/BVFyU3LdLhcAAIiQBwBoVHWiym9D8S0HtujcjHN9tiwYnTNaaV3S3C4X\nAAC0gpAHAJ1Q2ZEyvwVR9lbv1fDs4Z7u3JjcMRqePVxd4rq4XS4AADgDhDwAiGLWWu04vMPv+rma\nuhqf7lx+br6GZA5hQ3EAAKIAIQ8AokRdQ502VmzUqr2rPKGueF+xUhJT/K6fOyftHDYUBwAgShHy\nACACHas9ps/LPvcZuVxXvk59Uvv4BboeXXu4XS4AAOhAhDwACHOHjh/yu35ux6EdOi/rPJ8wNypn\nlLoldHO7XAAA4DJCHgCECWut9lTvUdG+ImfksjHYHTh+QKN6jvLsPZefm6/ze5yvhNgEt0sGAABh\niJAHAC5osA3aenCr34IokvwWRDk341w2FAcAAAEj5AFAiJ2sP6l1+9f5jFyuKVujzORMv+vneqX0\nYkEUAABwVgh5ABBER04e0ep9qz2BbtW+VdpUsUn90/srPyffM3I5Ome00pPS3S4XAABEIUIeALRT\n+dFyvwVRSqtKNazHMJ/u3IieI5Qcn+x2uQAAoJMg5AHAaVhrtatyl9/1c0dOHvG7fu68rPMUFxPn\ndskAAKATI+QBQDP1DfXadGCTX6DrEtfFL9D1796f6+cAAEDYIeQB6LRq6mr8NhRfu3+tclNy/RZE\n6dmtp9vlAgAABISQB6BTOFxzWMX7in06dNsObtPgzME+HbpROaOUmpjqdrkAAADtRsgDEHX2Vu/1\nWxCl7EiZRvYc6dOdG549XIlxiW6XCwAAEFSEPAARq8E2aPuh7X7Xz9U11PldPzcoY5BiY2LdLhkA\nACDkCHkAIkJtfa02VGxw9p7bu0pF+4q0umy10hLTPHvPNQW7Pql9WBAFAAB0WoQ8AGHn6MmjWlO2\nxmfkcn35euV1z/NbECUzOdPtcgEAAMIKIQ+Aqw4cO+B3/dyuw7t0fo/zfcLcyJ4j1TWhq9vlAgAA\nhD1CHoAOYa1VaVWpivZ5xy2L9hap8kSlRvUcpfycfGfsMjdfQ7OGKj423u2SAQAAIhIhD0DQ1TfU\na8vBLX4LosTFxPktiDIgfYBiTIzbJQMAAEQNQh6As3Ki7oTW7l/rM3K5pmyNenbr6Xf9XG5Krtvl\nAgAARD1CHoCAVZ2o0up9q326c5sPbNbAjIE+gW50zmh179Ld7XIBAAA6JUIegBaVHSnzWxBlT/Ue\njcge4beheFJ8ktvlAgAAoBEhD+jkrLXacXiH3/VzNXU1ftfPDc4crLiYOLdLBgAAQBsIeUAnUtdQ\np40VG30CXfG+YnWN7+oX6PLS8thQHAAAIAIR8oAodaz2mD4v+9xn5HJd+Tr1Se3jtyBKj6493C4X\nAAAAQULIA6LAoeOH/K6f23Foh4ZkDdGYnDE+G4qnJKa4XS4AAABCiJAHRBBrrfZU7/ELdBXHKjwb\nijcFumHZw5QQm+B2yQAAAOhghDwgTDXYBm09uNVvQRRJftfPnZtxLhuKAwAAQBIhDwgLJ+tPan35\neq3au8pnQ/GMpAxPoBuTO0b5OfnqldKLBVEAAADQKkIe0MGOnDzi3VC8MdBtrNio/un9/TYUz0jK\ncLtcAAAARBhCHhBC5UfL/a6fK6ks0fDs4T7Xz43oOULJ8clulwsAAIAoQMgDgsBaqy8qv1DRviJn\n5LIx2B05eUSjc0b7BLrzss5TfGy82yUDAAAgShHygDNU31CvTQc2+S2I0iWui9+CKP279+f6OQAA\nAHQoQh7Qhpq6Gr8NxdfuX6ucbjmehVCagl3Pbj3dLhcAAAAg5AFNKmsqVbyv2GfkcuvBrRqcOdin\nOzeq5yildUlzu1wAAACgRYQ8dEp7q/f6LYhSdqRMI3uO9NtQvEtcF7fLBQAAAAJGyENUa7AN2n5o\nu9/1c7UNtZ7u3JjcMcrPzdegjEGKjYl1u2QAAADgrBDyEDVq62u1oWKDJ9Ct2rtKq8tWKy0xzW9B\nlL6pfVkQBQAAAFGJkIeIdPTkUa0pW+Mzcrm+fL3yuuf5bSielZzldrkAAABAhyHkIewdOHbA7/q5\nXYd3aWiPoT4jlyN7jlTXhK5ulwsAAAC4ipCHsGGtVWlVqU+gW7V3lQ7XHPbbUHxoj6FKiE1wu2QA\nAAAg7BDy4Ir6hnptObjFb0GUuJg4v+vnBqQPUIyJcbtkAAAAICIQ8hByJ+pOaO3+tT4dujVla5Td\nNdsn0I3JHaPclFy3ywUAAAAiGiEPQVV1okqr96326c5tOrBJ52ac67cgSvcu3d0uFwAAAIg6YRPy\njDEzJP1WUoykJ621/3XK12+T9EtJpY0P/d5a+1QL5yHkdZCyI2V+C6Lsqd6j4dnDNSZnjKdLNzx7\nuJLik9wuFwAAAOgUwiLkGWNiJG2WNF3SHkmfSLrZWrux2TG3SRprrb3zNOci5AWZtVY7D+/0LITS\nFOyO1x336c7l5+RrSNYQxcXEuV0yAAAA0GkFGvJC/Vv7eElbrLW7Got6QdLVkjaechy7V4dYXUOd\nNlZs9OnOFe8rVtf4rp4g97X8ryl/Zr7y0vLYUBwAAACIUKEOeb0llTS7Xyon+J3qOmPMZDldv+9a\na0tbOAYBOl573G9D8XXl69Q7pbcn0P1w0g81Ome0srtmu10uAAAAgCAKdchrqR106szlvyT9zVpb\na4z5pqQFcsY7EYBDxw+peF+xz8jljkM7NCRriGfkct6oeRrVc5RSElPcLhcAAABAiIU65JVKOqfZ\n/T5yrs3zsNYeanb3z5J8FmZp7oEHHvB8XlBQoIKCgmDUGBGstdpTvcdvQZSKYxUa1XOU8nPyNa3/\nNP3fif9X5/c4X4lxiW6XDAAAAOAsFBYWqrCw8IyfF+qFV2IlbZLTmdsr6WNJc6y1G5odk2Ot3df4\n+bWSvmet/VIL5+o0C6802AZtPbjVb0NxK+uzIMqY3DE6N+NcNhQHAAAAOoGwWHjFWltvjLlD0hvy\nbqGwwRjzoKRPrLWvSrrTGHOVpFpJByX9n1DWFG5O1p/U+vL1KtrrHbdcU7ZGGUkZnuvn7hh3h/Jz\n89U7pTcLogAAAABoE5uhd6AjJ494NxRv7NJtrNio/un9/TYUz0jKcLtcAAAAAGEkLPbJC6ZIC3nl\nR8v9rp8rqSzRsOxhPoFuZM+RSo5PdrtcAAAAAGGOkNdBrLX6ovILv0BXfaJao3NG+2wofl7WeYqP\njXe7ZAAAAAARiJAXAvUN9dp0YJPfgiiJcYnKz3EWQmkKdf279+f6OQAAAABBQ8g7SzV1Nfq87HOf\nDt3a/WuV0y3H05lrCnQ53XI6rC4AAAAAnRMh7wxU1lR6NhRvCnVbD27VoMxBPmFuVM9RSuuSFpIa\nAAAAAKAthLxW7K3e63f9XNmRMo3oOcIn0A3PHq4ucV2CUDkAAAAAnL1OH/IabIN2HNrh2XuuKdjV\nNtT6hLn8nHwNzhys2JjYEFYPAAAAAGenU4W82vpabajY4NOdK95XrLTENL/r5/qm9mVBFAAAAAAR\nJ2pD3tGTR7WmbI3PyOX68vU6J+0cv0CXlZzldtkAAAAAEBRRGfLO+/152nV4l4b2GOq3oXi3hG5u\nlwgAAAAAIROVIa94b7GG9hiqhNgEt8sBAAAAgA4VlSEvUmoFAAAAgGALNOTFdEQxAAAAAICOQcgD\nAAAAgChCyAMAAACAKELIAwAAAIAoQsgDAAAAgChCyAMAAACAKELIAwAAAIAoQsgDAAAAgChCyAMA\nAACAKELIAwAAAIAoQsgDAAAAgChCyAMAAACAKELIAwAAAIAoQsgDAAAAgChCyAMAAACAKELIAwAA\nAIAoQsgDAAAAgChCyEPQFBYWul0C2sD7E/54j8If71F44/0Jf7xH4Y33J3oQ8hA0/I8hvPH+hD/e\no/DHexTeeH/CH+9ReOP9iR6EPAAAAACIIoQ8AAAAAIgixlrrdg0BMcZERqEAAAAAECLWWnO6YyIm\n5AEAAAAATo9xTQAAAACIIoQ8AAAAAIgiYR/yjDFPGmPKjDFr3K4F/owxfYwxbxlj1htjPjfG3Ol2\nTfBljEk0xnxkjClqfI/ud7sm+DPGxBhjVhlj/uV2LfBnjNlpjFnd+OfoY7frgT9jTJox5n+NMRuM\nMeuMMRe6XRMcxpjBjX92VjV+rOT3hfBjjLnbGLPWGLPGGPOcMSbB7ZrgyxhzV+Pvcqf9nTvsr8kz\nxkySdETS09bakW7XA1/GmBxJOdbaYmNMN0mfSbraWrvR5dLQjDEm2Vp7zBgTK+k9SXdaa/lFNYwY\nY+6WNFZSqrX2KrfrgS9jzHZJY621h9yuBS0zxvxV0gpr7V+MMXGSkq21VS6XhVMYY2IklUq60Fpb\n4nY9cBhjeklaKek8a+1JY8xCSYuttU+7XBoaGWOGSXpe0jhJdZJel/Tv1tptLR0f9p08a+1KSfyl\nGqastfustcWNnx+RtEFSb3erwqmstccaP02UFCcpvP91p5MxxvSRNFPSE27XglYZRcDfmZ2VMSZF\n0mRr7V8kyVpbR8ALW5dI2kbAC0uxkro2/SOJpD0u1wNfQyV9aK09Ya2tl7RC0rWtHcxfWAgaY0w/\nSaMlfeRuJThV4yhgkaR9kt601n7idk3w8RtJ3xPhO5xZSUuNMZ8YY77udjHwM0BShTHmL40jgY8b\nY5LcLgotuklONwJhxFq7R9Ijkr6QtFvSYWvtMnerwinWSppijEk3xiTL+cfhvq0dTMhDUDSOar4o\n6a7Gjh7CiLW2wVqbL6mPpAuNMee7XRMcxpgrJZU1dsRN4w3h50vW2gvk/KX6ncZLCRA+4iSNkfQH\na+0YScck/cDdknAqY0y8pKsk/a/btcCXMaa7pKsl5UnqJambMWauu1WhucZLof5L0jJJSyQVyxnb\nbBEhD2etsa3/oqRnrLWvuF0PWtc4vlQoaYbLpcDrIklXNV7z9bykqcYYroEIM9bafY0fyyW9JGm8\nuxXhFKWSSqy1nzbef1FO6EN4uULSZ41/jhBeLpG03Vp7sHEU8J+SvuRyTTiFtfYv1tqx1toCOZez\nbWnt2EgJefzrdnh7StJ6a+3v3C4E/owxWcaYtMbPk+T8j5yFccKEtfZH1tpzrLUDJN0s6S1r7Vfc\nrgtexpjkxmkFGWO6SrpMztgMwoS1tkxSiTFmcOND0yWtd7EktGyOGNUMV19ImmCM6WKMMXL+DG1w\nuSacwhjTo/HjOXKux2v1z1NcRxXVXsaYv0kqkJRpjPlC0v1NF1bDfcaYiyTdIunzxmu+rKQfWWtf\nd7cyNJMraUHjimYxkhZaa5e4XBMQSXpKeskYY+X8vfmctfYNl2uCvzslPdc4Erhd0lddrgfNNPtH\nxm+4XQv8WWs/Nsa8KKlIUm3jx8fdrQot+IcxJkPOe/Rta21laweG/RYKAAAAAIDARcq4JgAAAAAg\nAIQ8AAAAAIgihDwAAAAAiCKEPAAAAACIIoQ8AAAAAPh/7d1biFVVHMfx7w9FKzJBJLSHjMouNnmL\nJiGFCqme6iHoQjdKMagUupAUZYSBSSlW9BAkdLFeur+lESEVjYY21agVZQiRWUY3E+z26+GsmXbH\nc5xRpBm3vw8cZu21/2vttZiHw/+stfeukSR5ERERERERNZIkLyIiBoWkMZI+lLRR0nZJX1eOB/Qe\nV0krJU3sJ+ZmSVcdpDFfWsbYLalH0o39xJ8vqbPNuTmSvivz3Sxp/gCu/5/+DubcIiKiPvKevIiI\nGHSSFgG7bC9vcU4eAl9WkkYAXwHTbe8oL92eYPuLfbRZDOy0/WiLc3OAM2zfLmks8BkwyfaOA+kv\nIiKiV1byIiJiKFBfQTpJ0iZJqyT1AOMkPSlpvaRPJN1biX1H0mRJwyT9KGlJWWV7ryROSFosaUEl\nfomkdZK2SJpR6o+S9FJZnXtR0geSJjeNcXT5+xOA7T96EzxJx0p6uYyxS1KnpBOBucCdZbVuRrvJ\n294JbAXGl/4uKf1skPSGpLGt+mua2/TSprvMYdQB/zciIuKQliQvIiKGolOB5bY7bG8HFtruBKYC\nF0o6rUWb0cDbtqcCXUDbrZS2zwHuAu4vVfOB7bY7gMXlOs1tvgfWANskPS/pSkm9yeljwNIyxiuA\nlVzynWoAAAJLSURBVLa3Ak8BD9uebrur3XgknUDjO7mnVK21PcP2WcCrwB0D6O854LYy/8+BRe2u\nFxER9Tagex4iIiL+Z1/a3lg5vrrc/zacxmrXJODTpja7ba8p5Q3AzDZ9v1KJmVDKM4GHAGx/LGlT\nq4a2b5DUAcwGFgIXAPPK8SmVpG+0pJH9T5NrJM2mkdTeZPvPUj9B0iPAOGAEjaStLUljgJG23y9V\nzwDPDuD6ERFRQ1nJi4iIoei33oKkk4EFwHm2pwCrgSNatPm9Uv6L9j9k7mkRo6aY5uM+tntsrwAu\nBi6rxJ9te1r5HG97T7s+KlbZngzMApb1bjEFngCWlXO30Hq+zeNtO+aIiDi8JMmLiIihqJqwHAP8\nAuySNB64aABt9te7NLZZIulM4PS9OpdGSZpVqZoGbCvlN2ls+eyNnVKKv5bx75Pt9cALNJJZSptv\nysrg9ZXQlv3Z/gHYXbnv71pgbX/XjYiIekqSFxERQ1Hf0zTLts0t5fM0jYRsr7imcr/9NnkcOK48\n6OU+YDPwc1OMgLvLA1s2Avfw731/twLnSvqo9DG31L8OXF4eoNL2wSvFUmCOpCOBB4DXgHXAt5WY\n5v6q87kOWCGpm0aS+mA/14uIiJrKKxQiIuKwJ2kYMNz2nrI9dDUw0fbfgzy0iIiI/ZYHr0RERMDR\nwFuVl7DPS4IXERGHqqzkRURERERE1EjuyYuIiIiIiKiRJHkRERERERE1kiQvIiIiIiKiRpLkRURE\nRERE1EiSvIiIiIiIiBpJkhcREREREVEj/wCgI9498ncSIwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1e36fc3630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#crete the figures using collected data using f1 measure\n",
    "plt.figure(figsize=(15,6))\n",
    "tr_acc_f1, = plt.plot(plain_trainRatios,plain_tr_accs_f1,label=\"Training Accuracy F1\");\n",
    "test_accuracy_f1, = plt.plot(plain_trainRatios,plain_tst_accs_f1,label=\"Test Accuracy F1\");\n",
    "plt.legend(handles=[tr_acc_f1, test_accuracy_f1])\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Training Set Ratio')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Observation on Pipeline performance\n",
    "Above graphical outputs shows an expected behaviour while pipeline was tested under changing training and test data sets. As it can be seen, initially with only 10% training data and 90% test data, the training accuracy was very high due to overfitting. As we increase the size of training data while decreasing the size of test data, the training accuracy goes down from 87% to 70% while the test accuracy increased from 50% to 62% approximately. A similar behaviour was noticed with F1 measure values as described in the second graph above.\n",
    "\n",
    "This experiment was done without changing any further parameters just to monitor the behaviour of the pipeline and it performed as expected. Following further tasks were then performed to tune the hyper paramters for transformers and classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Task D - Parameter Grid / Tuning The Classifier\n",
    "\n",
    "Following task is about parameter tuning through ParamaterGridBuilder class from Machine Learning Library. Following values were provided for tuning purpose:\n",
    "Hash Vector size using for hashing transformer:  [1000,3000,10000,15000]\n",
    "Smoothing Factor for naive Bayes estimator: [0,0.2,0.4,0.6,0.8,1]\n",
    "\n",
    "The grid returns the best model calculated after running the classifier with given tuning paraaters. The training and test accuracies for the best were recorded as shown in the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy of best model from the gird :  0.9131647971426977\n",
      "Test accuracy of best model from the gird :  0.7822931785195936\n"
     ]
    }
   ],
   "source": [
    "#create the pramerter grid builder with parameters for hashing and naive bayes\n",
    "#we added different sizes to the hashing tf and different smoothing levels to the naive bayes\n",
    "#we have selected these parameters after few experiments \n",
    "import time;\n",
    "\n",
    "paramGrid = ParamGridBuilder() \\\n",
    "    .addGrid(hashingTF.numFeatures, [1000,3000,10000,15000]) \\\n",
    "    .addGrid(nb.smoothing, [0,0.2,0.4,0.6,0.8,1]) \\\n",
    "    .build()\n",
    "        \n",
    "trainValSpliter = TrainValidationSplit(estimator=pipeline, #our pipeline\n",
    "                               estimatorParamMaps=paramGrid, #parameter grid for grid search\n",
    "                               evaluator=evaluator, #our multiclass classification evaluator\n",
    "                               trainRatio=0.8) # we can change this ratio to change the training and validation set size\n",
    "\n",
    "# Run train validation split and choose the best set of parameters.\n",
    "tvModel = trainValSpliter.fit(training)\n",
    "\n",
    "# Make predictions on training documents to get final training accuracy \n",
    "prediction = tvModel.transform(training)\n",
    "trAcc = evaluator.evaluate(prediction)\n",
    "print(\"Train accuracy of best model from the gird : \",trAcc)\n",
    "\n",
    "# Make predictions on test documents. \n",
    "prediction = tvModel.transform(testing)\n",
    "testAcc =  evaluator.evaluate(prediction)\n",
    "print(\"Test accuracy of best model from the gird : \",testAcc)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Grid parameters\n",
    "Following code was generated to find the validation accuracy of the classifier while changing the tuning parameters. The output of the code was recorded to identify the parameters that give best performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t Accuracy \t Naive Bayes Smoothing \t HashingTF Size\n",
      "0.5109366236679753 \t\t 0 \t\t 1000\n",
      "0.511497476163769 \t\t 0.2 \t\t 1000\n",
      "0.5117779024116658 \t\t 0.4 \t\t 1000\n",
      "0.5117779024116658 \t\t 0.6 \t\t 1000\n",
      "0.5117779024116658 \t\t 0.8 \t\t 1000\n",
      "0.5123387549074594 \t\t 1 \t\t 1000\n",
      "0.6472237801458216 \t\t 0 \t\t 3000\n",
      "0.6483454851374089 \t\t 0.2 \t\t 3000\n",
      "0.6475042063937184 \t\t 0.4 \t\t 3000\n",
      "0.6469433538979249 \t\t 0.6 \t\t 3000\n",
      "0.6463825014021313 \t\t 0.8 \t\t 3000\n",
      "0.6463825014021313 \t\t 1 \t\t 3000\n",
      "0.38895120583286596 \t\t 0 \t\t 10000\n",
      "0.7462142456533931 \t\t 0.2 \t\t 10000\n",
      "0.7453729669097028 \t\t 0.4 \t\t 10000\n",
      "0.7445316881660123 \t\t 0.6 \t\t 10000\n",
      "0.7459338194054963 \t\t 0.8 \t\t 10000\n",
      "0.7462142456533931 \t\t 1 \t\t 10000\n",
      "0.25294447560291644 \t\t 0 \t\t 15000\n",
      "0.7708917554683118 \t\t 0.2 \t\t 15000\n",
      "0.7689287717330342 \t\t 0.4 \t\t 15000\n",
      "0.7697700504767246 \t\t 0.6 \t\t 15000\n",
      "0.7675266404935502 \t\t 0.8 \t\t 15000\n",
      "0.7647223780145822 \t\t 1 \t\t 15000\n"
     ]
    }
   ],
   "source": [
    "#Grid Parameter Results from the model\n",
    "\n",
    "#this function create a array of items with 3 result values\n",
    "def GetPerformanceOfParams(parameterResultItems,outputList):\n",
    "    \n",
    "    #loop trough resutls list\n",
    "    for item in parameterResultItems:\n",
    "        #create an empty array to hold 3 elements\n",
    "        localItemList = [];\n",
    "        #first element is accuracy and we can get that from item itself\n",
    "        localItemList.append(item[0]);\n",
    "        #parameter values are in a dictionary and we get items from the dictionary to loop through it \n",
    "        #here we got only 2 parameters since we have set only 2 parameters in the grid\n",
    "        for key,value in item[1].items():\n",
    "            #add the parameter to local list\n",
    "            localItemList.append(value);\n",
    "        #create list of [accuracy,param1,param2] items\n",
    "        outputList.append(localItemList);\n",
    "        \n",
    "    return;\n",
    "\n",
    "#make a list out of results from the model. here we combine the accuracy from validationmetics and parameters from estimatorparam maps\n",
    "resultItems =  list(zip(tvModel.validationMetrics, tvModel.getEstimatorParamMaps()))\n",
    "\n",
    "#create an empty array to hold the cleand results to produce graphs\n",
    "outputList = [];\n",
    "\n",
    "#call the result extract method with the combined results list\n",
    "GetPerformanceOfParams(resultItems,outputList);\n",
    "\n",
    "#print the output results to check the values\n",
    "print(\"\\t Accuracy \\t Naive Bayes Smoothing \\t HashingTF Size\")\n",
    "for i in outputList:\n",
    "    print(\"{0} \\t\\t {1} \\t\\t {2}\".format(i[0],i[2],i[1]))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### OBSERVATIONS\n",
    "\n",
    "From the above output, it can be seen that the accuracy of the classifier goes up as we increase the hash vector size within the hashingTF transformer. Best value for hashing vector size was found to be between 10,000 and 15,000 with the maximum accuracy of 77% with hash vector size of 15,000. It was also noted that a bigger hash vector size takes a longer time for the classifier to process the given data. \n",
    "\n",
    "The smoothing parameter for naive Bayes classifier also had a noticable affect on the accuracy. Different smoothing values from 0 (no smoothing) to 1 were supplied to the grid. Values greater than 1 resulted as acuracy getting lower in our previous experiments to it was decided to limit the range between 0 and 1. The best value for smoothing parameter was found to be between 0.2 and 0.4, where the accuracy was found to be 75 - 77%. \n",
    "\n",
    "Using the best values identified in the experiment above, we created an updated Pipeline with hashing value 15000 and a smoothing factor of 0.2. Rest of the values for transformers and other paramaters were kept the same as previous to their default values. Following code defines a tuned classifier for the final test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#Update the pipeline using best values\n",
    "\n",
    "#hashing transformer to crete hash out of words\n",
    "hashingTF = HashingTF(inputCol=remover.getOutputCol(), outputCol=\"features\", numFeatures = 15000)\n",
    "nb = NaiveBayes(smoothing=0.2, modelType=\"multinomial\")\n",
    "\n",
    "pipeline = Pipeline(stages=[tokenizer,remover, hashingTF, nb])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Experiment with  training and test size and their impact on time\n",
    "\n",
    "In order to identify the impact the size of training and test dataset with the final classifier, following experiment was performed where iteratively change the size of test and train data while recording the time taken for each run of the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ratio \t Training Size \t Test Size \t Training Acc. \t Training Time \t Test Acc. \t Test Time \t Total Time \n",
      "4.0\t8127\t\t 11859\t\t 95.1\t\t 1.85\t\t 74.61\t\t 4.66\t\t 9.82\n",
      "6.0\t11805\t\t 8181\t\t 93.04\t\t 2.38\t\t 76.62\t\t 3.61\t\t 10.52\n",
      "7.0\t14028\t\t 5958\t\t 92.75\t\t 2.96\t\t 77.04\t\t 2.94\t\t 11.13\n",
      "8.0\t16069\t\t 3917\t\t 91.95\t\t 3.18\t\t 77.58\t\t 2.3\t\t 11.3\n",
      "9.0\t17989\t\t 1997\t\t 91.44\t\t 3.41\t\t 78.62\t\t 1.82\t\t 11.47\n"
     ]
    }
   ],
   "source": [
    "#create an empty array to keep time results for each calculation\n",
    "trainingTimes = [];\n",
    "testTimes = [];\n",
    "trainingAccuracies = [];\n",
    "testAccuracies = [];\n",
    "totalTimes = [];\n",
    "trainingSizes = [];\n",
    "trainRatios = [4.0,6.0,7.0,8.0,9.0];\n",
    "\n",
    "#loop through list of numbers we defined as ratio for the train validation split method. in this method we devide our \n",
    "#training set further into training and validation sets. Then we do final test with our main test dataset.\n",
    "print(\"Ratio \\t Training Size \\t Test Size \\t Training Acc. \\t Training Time \\t Test Acc. \\t Test Time \\t Total Time \")\n",
    "\n",
    "for trainRatio in trainRatios:\n",
    "    #create training and test datasets from our main dataset.\n",
    "    [training,testing] = rows_DF.randomSplit([trainRatio,10-trainRatio])\n",
    "    training.cache();\n",
    "    testing.cache();\n",
    "    #get the start time for calculation\n",
    "    time1 = time.time();\n",
    "    \n",
    "    #train the classifier in pipeline\n",
    "    model = pipeline.fit(training)\n",
    "    \n",
    "    #get the time after training the model\n",
    "    time2 = time.time();\n",
    "    \n",
    "    # Make predictions on training documents to get final training accuracy \n",
    "    prediction = model.transform(training)\n",
    "    trAcc = evaluator.evaluate(prediction)\n",
    "    #get the time after prediction with training data\n",
    "    time3 = time.time();\n",
    "    \n",
    "    # Make predictions on test documents. \n",
    "    prediction = model.transform(testing)\n",
    "    testAcc =  evaluator.evaluate(prediction)\n",
    "    \n",
    "    #get the time at the end of the calculation\n",
    "    time4 = time.time();\n",
    "    \n",
    "    #calculating time differences\n",
    "    trTime = time2 - time1;\n",
    "    testTime = time4 - time3;\n",
    "    totalTime = time4 - time1;\n",
    "    trainingSize = training.count();\n",
    "    \n",
    "    #adding results to list to produce graphs\n",
    "    trainingTimes.append(trTime);\n",
    "    testTimes.append(testTime);\n",
    "    totalTimes.append(totalTime);\n",
    "    trainingAccuracies.append(trAcc);\n",
    "    trainingSizes.append(trainingSize);\n",
    "    testAccuracies.append(testAcc);\n",
    "    print(\"{0}\\t{1}\\t\\t {2}\\t\\t {3}\\t\\t {4}\\t\\t {5}\\t\\t {6}\\t\\t {7}\".format(trainRatio,trainingSize,\\\n",
    "        testing.count(),str(round(trAcc*100, 2)),str(round(trTime, 2)),str(round(testAcc * 100, 2)),str(round(testTime, 2)),str(round(totalTime, 2))))\n",
    "  \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA38AAAF/CAYAAAD5KHCmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8XGW97/HvL/c0zbVYW3oFBDZ3SrlVC6YKcjkIbqG2\npRaOorARkFePKBz3RlouB9iKutW9pUrVQmuLCAW5KBVKuG42FSiIFEopYG+AJWmS0lxnfuePmYTp\nZCaZSWcySdbn/XrlNevyrDXPKqH49fes5zF3FwAAAABgeMvLdQcAAAAAANlH+AMAAACAACD8AQAA\nAEAAEP4AAAAAIAAIfwAAAAAQAIQ/AAAAAAiArIY/M1tsZu+Z2cu9tPmJmb1hZmvN7MiY4+eb2Xoz\ne93MzstmPwEAAABguMt25e/Xkk5JdtLMTpO0n7vvL+kiSbdGj1dL+p6kYyQdJ+kaM6vMcl8BAAAA\nYNjKavhz96ckNfTS5CxJt0fb/o+kSjP7uCKBcZW7N7r7DkmrJJ2azb4CAAAAwHCW63f+xknaFLO/\nOXos/viW6DEAAAAAQD/kOvxZgn1PcFzR4wAAAACAfijI8fdvljQhZn+8pK3R47Vxxx9LdAMzIxQC\nAAAACDR3T1RA281AhD9T4kqeJP1B0iWS7jSz4yXtcPf3zOxhSTdEJ3nJk3SypKuSfYE7+Q+D04IF\nC7RgwYJcdwPogd9NDGb8fmKw4ncTg5VZn7lPUpbDn5n9VpEK3igz+7ukayQVSXJ3/4W7P2Rmp5vZ\nBkkfSvqKIicbzOw6SX9RZLjnwujELwAAAACAfshq+HP3c1Noc2mS47+R9JsMdwkAAAAAAinXE74A\nw1ptbW2uuwAkxO8mBjN+PzFY8buJoc6G+vtyZuZD/RkAAAAAoL/MbNBM+AIAAAAExuTJk/XOO+/k\nuhsYhiZNmqS3336739dT+QMAAAAyKFqFyXU3MAwl+91KtfLHO38AAAAAEACEPwAAAAAIAMIfAAAA\nAAQA4Q8AAABA2sLhsMrLy7V58+aMtkX2EP4AAACAACgvL1dFRYUqKiqUn5+vESNGdB9bvnx52vfL\ny8tTc3Ozxo8fn9G2/XXbbbcpLy9P9957b9a+Y6hjtk8AAAAgg4bCbJ/77ruvFi9erBkzZiRtEwqF\nlJ+fP4C92jMnnnii1q1bp+nTp2vlypUD+t3hcFh5edmvqzHbJwAAAIC0uHuPEHH11Vdr9uzZOvfc\nc1VZWally5bp2Wef1bRp01RdXa1x48bp8ssvVygUkhQJh3l5efr73/8uSZo3b54uv/xynX766aqo\nqNCnPvWp7vUO02krSX/84x914IEHqrq6Wt/85jc1ffp03X777UmfZ+PGjXrmmWe0aNEiPfTQQ/rg\ngw92O3/PPfdoypQpqqys1AEHHKBHHnlEklRfX6+vfOUr2nvvvTVq1CjNnDlTknoE40T9v/TSS3Xa\naaepvLxcTz31lO6//35NmTJFFRUVmjx5sq6//vrd+vDEE09o2rRpqqqq0qRJk7r/fMeNG7dbuzvv\nvFPHHHNMb//4+o3wBwAAAECSdO+99+rLX/6yGhsbNWvWLBUWFuonP/mJ6uvr9fTTT+vhhx/WokWL\nutub7V5sWr58uW644QY1NDRowoQJuvrqq9Nu+/7772vWrFm65ZZbtH37du2zzz5as2ZNr/1esmSJ\njj/+eH3xi1/Ufvvtt9sw1meeeUYXXHCBfvSjH6mxsVGPPfaYJk2aJEmaM2eOOjo69Nprr+m9997T\n5ZdfnrS/ifq/cOFCNTc36/jjj1d5ebmWLVumpqYm3X///frJT36ihx56SJL01ltv6YwzztAVV1yh\n+vp6vfjiizrssMN0/PHHq6KiQo8++mj3fZctW6bzzz+/1+ftL8IfAAAAAEnS9OnTdfrpp0uSiouL\nNXXqVB1zzDEyM02ePFlf//rX9fjjj3e3j68ennPOOZoyZYry8/M1d+5crV27Nu22Dz74oKZMmaIz\nzjhD+fn5mj9/vkaNGtVrv5cuXaq5c+dKks4991wtWbKk+9yvfvUrXXjhhaqtrZUkjRs3Tvvvv782\nb96sxx57TLfeeqsqKipUUFCg6dOnJ/2O+P7/8z//s4499lhJUlFRkWpra3XwwQdLkg477DDNmjWr\n+89q2bJlOu2003T22WcrLy9PNTU1OvzwwyVFqoh33HGHJGn79u1avXq1Zs+e3evz9hfhDwAAABhA\nZpn5yYYJEybstv/666/rjDPO0NixY1VZWalrrrlG27dvT3r9mDFjurdHjBihnTt3pt1269atPfrR\n20Qxjz/+uDZv3tw9ZHPOnDl6/vnn9eqrr0qSNm3apP3226/HdZs2bdJee+2lkSNHJr13b+L7+N//\n/d+aMWOGRo8eraqqKi1evLj7zypZH6RI+LvvvvvU2tqqFStWaMaMGdprr7361ae+EP4AAACAAeSe\nmZ9siB/aeNFFF+mwww7Txo0b1djYqIULF2Z9MpuxY8dq06ZNux3bsmVL0vZLlixROBzW4YcfrrFj\nx2r69OnKy8vrfkdwwoQJevPNN3tcN2HCBG3fvj1hQC0rK9OuXbu697dt29bnMNA5c+Zo5syZ2rJl\ni3bs2KELLrig+89qwoQJ2rBhQ8L+T5gwQUcffbTuvfdeLV26VPPmzUv6rHuK8AcAAAAgoebmZlVW\nVqq0tFTr1q3b7X2/bDnjjDP04osv6sEHH1QoFNKPf/zjpNXGlpYW3X333frVr36ltWvX6qWXXtJL\nL72kH/7wh7rjjjvk7rrgggt022236fHHH5e7a8uWLVq/fr3Gjx+vk046SZdccokaGxvV2dmpJ598\nUpJ0xBFH6OWXX9bf/vY3tbS06Nprr+2z3zt37lR1dbUKCwv17LPPasWKFd3nvvzlL+vhhx/WypUr\nFQqF9MEHH+jll1/uPj9v3jzdeOONev3113XWWWft4Z9gcoQ/AAAAIGDiq1bJ3HLLLfrNb36jiooK\nXXzxxT3eRYu9T1/3TLXt6NGjdeedd2r+/Pnaa6+99NZbb2nKlCkqLi7u0faee+5RRUWF5s6dq9Gj\nR3f/fP3rX1dra6v+/Oc/a9q0afrlL3+pyy67TJWVlfrMZz7Tvdj80qVL5e464IADNGbMGP3sZz+T\nJB100EH67ne/q09/+tM66KCD9OlPfzrps3T5+c9/rquuukqVlZW66aabNGvWrO5zkydP1v3336+b\nbrpJNTU1mjp1ql555ZXu82effbY2btyomTNnJnzOTGGdPwAAACCDhsI6f0NJOBzW3nvvrbvvvluf\n+tSnct2drNlnn320ZMkSnXjiiUnbsM4fAAAAgGHl4YcfVlNTk9ra2nTttdeqsLCwe2bN4ejOO+9U\nSUlJr8EvEwqyencAAAAASNNTTz2luXPnqqOjQ4cccojuu+8+FRYW5rpbWXHCCSdow4YN+u1vf5v1\n72LYJwAAAJBBDPtEtjDsEwAAAADQJ8IfAAAAAAQA4Q8AAAAAAoDwBwAAAAABQPgDAAAAgAAg/AEA\nAADImPb2dpWXl+vdd9/NdVcQh/AHAAAABEB5ebkqKipUUVGh/Px8jRgxovvY8uXL+33fadOm7bZG\nXVFRkZqbmzVmzJhMdDuhW2+9VXl5eXrggQey9h3DEeEPAAAACIDm5mY1NTWpqalJkyZN0oMPPth9\nbM6cObnuXlpuv/12jRo1SkuWLBnw7w6HwwP+nZlC+AMAAAACxt17LBYeDod13XXXab/99tPo0aM1\nb948NTU1SZJ27dqlOXPmaNSoUaqurta0adPU2NioK664QmvWrNHXvvY1VVRU6Nvf/rba2tqUl5en\nrVu3SpLmzJmj+fPn69RTT1VFRYVOOOEEbdq0qft7H3zwQR1wwAGqqanR/Pnze1QS461fv17PPfec\nFi1apPvvv187duzY7fxdd92lI444QpWVlTrwwAP12GOPSZI++OADnX/++Ro7dqxGjRrVHXgXLVqk\nk08+ufv6RP2//PLLdcopp6i8vFzPPvus7r33Xh155JGqrKzUPvvsoxtvvHG3PtTV1WnatGmqqqrS\n5MmTtWLFCj311FOaOHHibu2WLVumadOm9f0PLEMIfwAAAAD07//+73rkkUf0zDPPaPPmzSosLNT8\n+fMlSbfddptCoZC2bdumDz74QD/72c9UVFSkH/zgBzrmmGO0ePFiNTU16fvf/74kycx2u/fy5ct1\n8803q6GhQWPGjNE111wjSXr33Xc1e/Zs/cd//If+8Y9/aO+999YLL7zQaz+XLFmi6dOn64tf/KIm\nTZqkFStWdJ978sknddFFF+mnP/2pGhsb9eijj2rChAmSpC996UuSIuHxvffe0yWXXNJ9XXx/4/eX\nLVumG264Qc3NzTr66KNVWVmp5cuXq7GxUStXrtQtt9yiVatWSZI2bNigz3/+87ryyitVX1+v559/\nXocccoimT5+u4uJiPf7447vd9/zzz+/jn0zmEP4AAAAA6Be/+IVuuukmffzjH1dRUZGuvvrq7mBV\nWFiof/zjH3rjjTeUl5enqVOnqrS0tPva+Cpi/P6XvvQlHXHEEcrPz9e5556rtWvXSpLuv/9+HXvs\nsTrttNOUn5+vK664QlVVVUn76O5aunSp5s6dKylSlYsd+rl48WJdfPHFOvHEEyVJ48eP1yc+8Qm9\n/fbbevrpp/Xzn/9c5eXlKigo0PTp03v9nljnnHOOjj76aEmRdxpnzJihgw46SJJ05JFHaubMmd2h\nbunSpTrzzDP1hS98QXl5eRo1apQOO+wwSdK8efN0xx13SJLee+89PfHEE92hdCAUDNg3AQAAAJAt\ntL4bpcCv8b4bpWHTpk06/fTTu6teXQGovr5eF1xwgd59912dc845+vDDDzVv3jxdf/31PSpkycRO\n/jJixAjt3LlTkrR169buypwUqbiNGzcu6X1Wr16t9957T+ecc44k6dxzz9W1116r9evX64ADDtCm\nTZu6g1/8s40ePVojRoxIqb/xYvsoSU8//bT+9V//Va+++qra29vV3t6uefPmdX/Xfvvtl/A+5513\nnqZOnar/+q//0vLly3XyySerpqamX33qD8IfAAAAMIAyHdoyZfz48brnnns0ZcqUhOcXLlyohQsX\n6u2339bnPvc5HXrooZozZ07KATCRsWPH6oknnujed3dt2bIlafslS5YoHA7rkEMO6Q6neXl5uv32\n23X99ddrwoQJevPNN3tcN2HCBL3//vvatWtXjwBYVlamXbt2de9v27atz2Ggs2bN0tVXX62vfvWr\nKiws1MUXX6xQKNT9XevXr0/Y/8mTJ+vwww/XH/7wBy1dulTf/e53kz5rNjDsEwAAAIAuuugiXXnl\nldq8ebMk6f333+9eSuHRRx/VunXr5O4aOXKkCgoKVFAQqSN9/OMf18aNG/v1nWeeeaaee+45/elP\nf1IoFNItt9zSYwKXLjt37tTKlSu1ZMkSrV27Vi+99JJeeuklff/739ftt98uSfra176mRYsW6amn\nnpK7a/PmzdqwYYMmT56sE088UZdeeqmamprU0dGhJ598UlJk2OaLL76odevWadeuXbruuuv67PeH\nH36ompoaFRYW6plnntFdd93Vfe68887Tgw8+qPvuu0/hcFjbt2/XX//61+7z8+bN03XXXae33npL\nn//85/v159ZfhD8AAAAgYBJV66688kqdfPLJ+sxnPqPKykpNnz5dL774oiRpy5YtOuuss1RRUaHD\nDz9cZ5xxhmbOnClJmj9/vpYsWaJRo0bpqquu6nH/3iqDY8aM0fLly3XZZZfpYx/7mLZu3arDDjtM\nxcXFPdrefffdGjVqlGbPnq3Ro0d3/1x44YVqbm7W6tWrNX36dN166626+OKLVVlZqZNOOqk7zC5f\nvlzt7e3af//9NXbsWN16662SpEMPPVTf+c53NH36dB188MGaMWNGn39Wt956q771rW+psrJSP/jB\nD3Z7b2/ffffVfffdpxtuuEHV1dU65phj9Oqrr3afnzlzpjZs2KBZs2apsLAw6Z9NNlj8y4xDjZn5\nUH8GAAAADB9m1mPCEKQmFAppzJgxeuCBB3TcccflujtZ4e6aOHGi7rzzTn3yk59M69pkv1vR432O\nv6XyBwAAACBn/vSnP6mpqUmtra1asGCBysrKNHXq1Fx3K2t++9vfqrKyMu3glwlM+AIAAAAgZ554\n4gnNnTtXoVBIhx56qFauXNn9PuFwM23aNL3zzjtavnx5Tr6fYZ8AAABABjHsE9nCsE8AAAAAQJ8I\nfwAAAAAQAIQ/AAAAAAgAwh8AAAAABMDwnEYHAAAAyJFJkyb1urA50F+TJk3ao+uZ7RMAAAAAhjBm\n+wQAAAAAdMt6+DOzU83sNTNbb2ZXJjg/0cweMbOXzGy1me0dcy5kZi+Y2Ytmdm+2+woAAAAAw1VW\nh32aWZ6k9ZI+K2mrpDWSZrv7azFtfifpD+6+1MxqJX3V3c+Lnmty94o+voNhnwAAAAACa7AM+zxW\n0hvu/o67d0haIemsuDYHS1otSe5eF3eeN2UBAAAAIAOyPdvnOEmbYvY3KxIIY62VdLakn5rZFyWN\nNLNqd2+QVGxmz0nqlHSzu9+X6EumTZOqq6WamshPou3Yz6KizD8oAAAAAAxm2Q5/iSp38WM0vy3p\nZ2b2vyU9IWmLImFPkia6+7tmto+k1Wb2sru/FX/DW26R6uulhobIZ3299Oab0po1ux9vaIj8FBcn\nD4e9bZeXS8zaCwAAAGAoynb42yxpYsz+eEXe/evm7tsUqfzJzMokne3uzdFz70Y/3zKzOklTJPUI\nf6tWLejerq2tVW1tbdIOuUvNzbsHxdhw2BUcE51vaYmEwVSrjF3b1dWRwAkAAAAAe6qurk51dXVp\nX5ftCV/yJb2uyIQv2yQ9J2mOu6+LaTNKUr27u5ldL6nT3ReYWZWkXe7ebmZ7SXpa0lmxk8VErx+w\nCV/a26UdO3pWGePDY6Lt4uLUgmL8dkUF1UYAAAAAyaU64UtWK3/uHjKzSyWtUmRymcXuvs7MFkpa\n4+4PSKqVdKOZhRUZ9nlJ9PKDJC0ys1D02hvjg99AKyqSRo+O/KTDXdq5s/dw+NZbic/v2iVVVaUW\nFOOPUW0EAAAA0CWrlb+BMNyXeujo6L3a2FsVsrAwvSpjbLUxL+srQAIAAADIhFQrf4S/Ycpd+vDD\n/g1R/fBDqbIyvSpj12dJSa6fHAAAAAgWwh/6rbPzo2pjuuGxoCD12VNjQ2NlJdVGAAAAoD8Ifxhw\n7pF3FNOpMnZ97twZCYD9mRSHaiMAAACCjPCHIaWr2pjuENX6eik/P/33GmtqqDYCAABgeCD8IRDc\nI+sv9meI6s6dkclt+jMpTmlprp8cAAAAiCD8AX0IhZJXG3sLjx98EKkYprv0Rle1MT8/108OAACA\n4YTwB2RJV7UxnXcau441N0vl5f2bFKe0VLI+/5UGAABA0BD+gEEoFJIaG9MfolpfH7k+nSpj13ZV\nFdVGAACA4YzwBwwzsdXGdMJjU1Ok2tifSXGoNgIAAAx+hD8AkqRwuPdqY2/hMRRKf+mNmhqqjQAA\nAAOJ8Adgj3VVG9N9v7GxURo5sn+T4owYQbURAAAgHYQ/ADkTDkeGm6a7ZmNDQ2TNx3Tfa+yqNhYU\n5PrJAQAABh7hD8CQ1Nq6e7Ux1SGqO3ZIZWX9mxSnrIxqIwAAGLoIfwACpava2J8lODo60l96o+uT\naiMAAMg1wh8ApKitrWcoTCU8NjRE3lHsz6Q4VBsBAECmEP4AIMvCYam5Ob01G7u229sjYTDdSXGq\nq6XCwlw/OQAAGEwIfwAwiLW39x4Uk4XHHTsi6y/2Z1KckSOpNgIAMBwR/gBgGHJPXm3sK0i2tn4U\nBCsqIhXE+J+CgsTHc9WGsAoAQN8IfwCA3bS3RyqH9fWRyXE6OpL/dHb2fj6TbXprl5e35yFyMAXa\n/Pxc/xYAAIYjwh8AYEhzl0KhgQ2b2W7jPrTCal9tCgqozgLAYED4AwBgkAmHB0cgzdS9QqFIAMxk\noC0qiryfWl4eGZ7c9Rm7HftZVJTrf6oAkHuEPwAAkFXumQ+k7e3Shx9GhiY3NUXecY3fjj1WUNB3\nSEz1PMNyAQxVhD8AADCsuUcmMkolJPZ1rLlZKilJPUT2dmzkyMj7qgAwUAh/AAAAKXKPVBz7Exzj\nj+3aJY0Y0b/gGH+srIz3KgH0jfAHAACQA+GwtHNn/6uQscdaWyMhsL9VyNjtkhKCJDBcEf4AAACG\nuM7Oj4LknlYkOzv3fEhr13Zxca7/ZADEIvwBAACgW3t7z3DYnzDZ1BR5pzET70eWl0dmeQWwZwh/\nAAAAyDh3qa1tzyfZ6TpWVNS/2Vnjj40cyYytCC7CHwAAAAY1d6mlZc+HtDY1RSbsKS3NTEWyrIwZ\nWzG0EP4AAAAQGOHw7jO27klFsqUlEgDjQ2JZWSRglpREPjOxTbUSmUD4AwAAAPohFOo5Y2tXdbGl\nJTILa0tLZrYLCjIXJBNtJzpXVMTMr8MN4Q8AAAAYxNwjE/FkKkj2td21HwpFQmA2A2b8NlXO7CL8\nAQAAAOghFBqYsJnJKmd/A2pQqpyEPwAAAACDwmCtcmZruO1ATxhE+AMAAAAQaIOhypmtgBm7XVVF\n+AMAAACAAZWLKmdzM+EPAAAAAIa9VId9snwlAAAAAAQA4Q8AAAAAAoDwBwAAAAABQPgDAAAAgAAg\n/AEAAABAABD+AAAAACAACH8AAAAAEACEPwAAAAAIAMIfAAAAAAQA4Q8AAAAAAiDr4c/MTjWz18xs\nvZldmeD8RDN7xMxeMrPVZrZ3zLnzo9e9bmbnZbuvAAAAADBcmbtn7+ZmeZLWS/qspK2S1kia7e6v\nxbT5naQ/uPtSM6uV9FV3P8/MqiX9RdJRkkzS85KOcvfGuO/wbD4DAAAAAAxmZiZ3t77aZbvyd6yk\nN9z9HXfvkLRC0llxbQ6WtFqS3L0u5vwpkla5e6O775C0StKpWe4vAAAAAAxL2Q5/4yRtitnfHD0W\na62ksyXJzL4oaWS06hd/7ZYE1wIAAAAAUpDt8Jeo9Bg/RvPbkmrN7HlJJygS8jpTvBYAAAAAkIKC\nLN9/s6SJMfvjFXn3r5u7b9NHlb8ySWe7e7OZbZZUG3ftY4m+ZMGCBd3btbW1qq2tTdQMAAAAAIa8\nuro61dXVpX1dtid8yZf0uiITvmyT9JykOe6+LqbNKEn17u5mdr2kTndfEDfhS150e2r0/b/Y72DC\nFwAAAACBNSgmfHH3kKRLFZms5W+SVrj7OjNbaGZnRJvVSnrdzF6TNFrSDdFrGyRdp0jo+x9JC+OD\nHwAAAAAgNVmt/A0EKn8AAAAAgmxQVP4AAAAAAIMD4Q8AAAAAAoDwBwAAAAABQPgDAAAAgAAg/AEA\nAABAABD+AAAAACAACH8AAAAAEACEPwAAAAAIAMIfAAAAAAQA4Q8AAAAAAoDwBwAAAAABQPgDAAAA\ngAAg/AEAAABAABD+AAAAACAACH8AAAAAEACEPwAAAAAIAMIfAAAAAAQA4Q8AAAAAAoDwBwAAAAAB\nQPgDAAAAgAAg/AEAAABAABD+AAAAACAACH8AAAAAEACEPwAAAAAIAMIfAAAAAAQA4Q8AAAAAAoDw\nBwAAAAABQPgDAAAAgAAg/AEAAABAABD+AAAAACAACH8AAAAAEAB9hj8zu9TMqgeiMwAAAACA7Eil\n8jdG0hoz+52ZnWpmlu1OAQAAAAAyy9y970aRwPc5SV+RdLSk30la7O5vZrd7fTMzT+UZAAAAAGA4\nMjO5e59FupTe+Yumq3ejP52SqiX93sz+fY96CQAAAAAYEH1W/szsm5LOl7Rd0m2S7nX3DjPLk/SG\nu++X/W722j8qfwAAAAACK9XKX0EK99pL0hfd/Z3Yg+4eNrMz+ttBAAAAAMDASWXY50OS6rt2zKzc\nzI6TJHdfl62OAQAAAAAyJ5Vhny9KOqprbGV0uOdf3P2oAehfnxj2CQAAACDIMjnhy27pyt3DSm24\nKAAAAABgkEgl/G00s2+aWWH053JJG7PdMQAAAABA5qQS/v5F0iclbZG0WdJxki7MZqcAAAAAAJmV\n0iLvgxnv/AEAAAAIsowt9WBmJZIukHSIpJKu4+7+1T3qIQAAAABgwKQy7PMOSWMknSLpcUnjJTVn\ns1MAAAAAgMxKaakHd59iZi+7++FmVijpSXc/fmC62DuGfQIAAAAIskwu9dAR/dxhZodKqpQ0ek86\nBwAAAAAYWKmEv1+YWbWkf5P0B0mvSro51S8ws1PN7DUzW29mVyY4P8HMVpvZC2a21sxOix6fZGa7\nosdfMLP/SvU7AQAAAAC763XCFzPLk9Tk7g2SnpC0bzo3j17/M0mflbRV0hozu8/dX4tp9m+S7nT3\nRWZ2kKSHJO0TPbfB3Y9K5zsBAAAAAD31Wvlz97Ck7+zB/Y+V9Ia7v+PuHZJWSDorrk1YUkV0u0qR\n9QS79DluFQAAAADQt1SGfT5iZldEh2fWdP2keP9xkjbF7G+OHou1UNI8M9sk6QFJl8Wcm2xmz5vZ\nY2Y2PcXvBAAAAADE6XOdP0mzop+XxBxzpTYENFHlLn5qzjmSfu3uPzKz4yUtVWRNwW2SJrp7g5kd\nJeleMzvY3XfG33DBggXd27W1taqtrU2hawAAAAAw9NTV1amuri7t6/pc6mFPRMPcAnc/Nbp/lSR3\n95tj2rwi6RR33xLdf1PSce6+Pe5ej0n6lru/EHecpR4AAAAABFaqSz30Wfkzs/MSHXf321PoxxpJ\nnzCzSYpU8mYrUumL9Y6kkyQtiU74Uuzu281sL0n17h42s30lfULSxhS+EwAAAAAQJ5Vhn8fEbJco\nMnPnC5L6DH/uHjKzSyWtUuT9wsXuvs7MFkpa4+4PSLpC0i/NbL4ik7+cH738REnXmlmHpJCki9x9\nR4rPBQAAAACIkfawTzOrVGRphlOz06X0MOwTAAAAQJClOuwzldk+4+3SR+vwAQAAAACGgFTe+btf\nH83QmSfpYEm/y2anAAAAAACZ1eewTzP7dMxup6R33H1zVnuVBoZ9AgAAAAiyjM32Kenvkra5e2v0\nxqVmNtnIPMagAAAXdElEQVTd397DPgIAAAAABkgq7/zdpcgsnF1C0WMAAAAAgCEilfBX4O7tXTvR\n7aLsdQkAAAAAkGmphL9/mNmZXTtmdpak7dnrEgAAAAAg01KZ8GU/Scsk7R09tFnSee6+Ict9SwkT\nvgAAAAAIslQnfEl5kXczGxlt37ynncskwh8AAACAIMvYIu9m9v/MrMrdd7p7s5lVm9n1mekmAAAA\nAGAgpPLO32nuvqNrx90bJJ2evS4BAAAAADItlfCXb2bFXTtmViqpuJf2AAAAAIBBJpVF3pdKetTM\nfh3d/4qkJdnrEgAAAAAg01Ka8MXMTpV0kiST1CBprLtfkuW+pYQJXwAAAAAEWcYmfIl6V1JY0tmS\nPitp3R70DQAAAAAwwJIO+zSzAyTNljRH0geS7lSkUjhjgPoGAAAAAMiQpMM+zSws6UlJF3Qt6G5m\nG9193wHsX58Y9gkAAAAgyDIx7PNsRYZ7PmZmvzSzzyryzh8AAAAAYIjpc8IXMyuT9AVFhn9+RpGZ\nPle6+6rsd69vVP4AAAAABFmqlb+UZvuMuWmNpJmSZrn7Z/agfxlD+AMAAAAQZFkJf4MR4Q8AAABA\nkGV6qQcAAAAAwBBG+AMAAACAACD8AQAAAEAAEP4AAAAAIAAIfwAAAAAQAIQ/AAAAAAgAwh8AAAAA\nBADhDwAAAAACgPAHAAAAAAFA+AMAAACAACD8AQAAAEAAEP4AAAAAIAAIfwAAAAAQAIQ/AAAAAAgA\nwh8AAAAABADhDwAAAAACgPAHAAAAAAFA+AMAAACAACD8AQAAAEAAEP4AAAAAIAAIfwAAAAAQAIQ/\nAAAAAAgAwh8AAAAABADhDwAAAAACgPAHAAAAAAGQ9fBnZqea2Wtmtt7MrkxwfoKZrTazF8xsrZmd\nFnPu/5rZG2a2zsw+l+2+AgAAAMBwZe6evZub5UlaL+mzkrZKWiNptru/FtNmkaQX3H2RmR0k6SF3\n38fMDpa0TNIxksZLekTS/h7XYTOLPwQAAAAAgWFmcnfrq122K3/HSnrD3d9x9w5JKySdFdcmLKki\nul0laUt0+0xJK9y9093flvRG9H4AAAAAgDRlO/yNk7QpZn9z9FishZLmmdkmSQ9IuizJtVsSXAsA\nAAAASEG2w1+i0mP8GM05kn7t7hMk/S9JS9O4FgAAAACQgoIs33+zpIkx++MVefcv1gWSTpEkd3/W\nzErMbK8Ur5UkLViwoHu7trZWtbW1e9pvAAAAABiU6urqVFdXl/Z12Z7wJV/S64pM+LJN0nOS5rj7\nupg2D0r6nbsviU748md3Hx8z4ctxigz3/LOY8AUAAAAAdpPqhC9Zrfy5e8jMLpW0SpEhpovdfZ2Z\nLZS0xt0fkHSFpF+a2XxFJn85P3rtq2b2O0mvSuqQ9A1SHgAAAAD0T1YrfwOByh8AAACAIBssSz0A\nAAAAAAYBwh8AAAAABADhDwAAAAACgPAHAAAAAAFA+AMAAACAACD8AQAAAEAAEP4AAAAAIAAIfwAA\nAAAQAIQ/AAAAAAgAwh8AAAAABADhDwAAAAACoCDXHQAAAAAARLi7mtqaVN9Sr4bWhshnS+Qz9ljs\ndqrM3bPY9ewzMx/qzwAAAABgeGnrbEs5vMWe39G6QyMKR6i6tFo1pTWqLol89tiOOX/U3kfJ3a2v\nPhH+AAAAACCBsIfV1NaUMLz1FejaQ+2JA1tJz/AWe766pFqF+YVp9dPMCH8AAAAA0NbZ1jOkJQp0\ncee7qnCJqm29VeNqSmtUVlgmsz7zWEYQ/gAAAAAMG11VuPjw1iPQtfYMd53hzt2CWSpDKWtKa1RV\nUpV2FS4XCH8AAAAABp3WztbkQya7jrf2DHeNrY0qKypLGt6ShrvS6gGtwuUC4Q8AAABAVoQ9rMbW\nxqThLdE7cF3nO8OdCYdJ9hXoqkqqVJDHYgWJEP4AAAAA9KqloyXlCUxi2zS1NWlk0ciE4S3ZO3Bd\nx0cUjhjWVbhcIPwBAAAAARAKh9TY1tj7UMrWxOEu7OGUZqSMr8ZRhRtcCH8AAADAENLS0ZL2cgL1\nLfVqbmvWyKKRPattJcmXE+jaLi0opQo3DBD+AAAAgAHWVYXrbTmBZOHO5RpVOiqt5QSqSyJVuPy8\n/Fw/OnKI8AcAAACkIOxhtXa2alfHLrV0tKilsyXhdnN7c9IZKbsCXVNbkyqKK9JaTqDrGFU49Bfh\nDwAAAEOSu6sj3NFrEIvfbumI7sdvp3B9e6hdxQXFGlE4QqUFpSotLN1tu7Qgsh87tDJZoKssrqQK\nhwFH+AMAAEDGJKqOJQ1cKQax3q6RFAlgMeEr6XaCwJYovCXbLi4oVp7l5fhPGOg/wh8AAMAw1xHq\nSC98xW13h7EUrumqjvUZxBKEsrSuiW4X5hfm+o8XGDIIfwAAAAOsqzqWUuBKtp1GeJNSq46VFpZq\nREF64Sv++pKCEqpjwCCVavhjcQ4AADCsxVfHUn2HLFko661dW2dbwupY0iGJMe0qiit6D28Jrqc6\nBiAdVP4AAMCACHtY7aF2tXW2qbWzVW2hNrV1tqktFN2Pbqc0mUdn6uFNUq/vg+22naQ6lur1VMcA\n5AKVPwAAoM5w527BKlHwSvXYbvfpx3Ud4Q4V5RepOL9YJQUlKi4oVnF+cfdn17FE1bGu/YriitSD\nHNUxANgNlT8AADLI3ZMHpjSO9QhV/bxOUiRUJQhZXcfizxcXFKskP3k4S/te0WNF+UWsYQYAWUDl\nDwAQGJ3hzsxVtGKP9SPEtYfak1a3egtGiQLVyKKR/QpZsccK8vhPPQAggsofACBt8dWtPapoxR7r\n57BEk/UrGKVS3Ur5XlS3AAA5QuUPAAKsrbNNDa0Namhp6PFZ31Kv5vbm7lDWGkp/WGJHqEOF+YVp\nB6NEbcpKy/oVsqhuAQCQHip/ADBItYfak4a33Y4lON8Z7lR1abWqS6q7P2tKa7r3K4ordgtRVLcA\nABi6WOQdAAaBjlBHv8JbQ2uD2kPt3WEtNrhVl+we6hKdH1E4gnAGAEBAEP4AIEM6Qh3a0bpDDa3R\nYJagGtcV4uLPt4XaVFVSlXZ4qy6tVllhGQEOAAD0ifAHADE6w53a0bqj9/DW0qD61p7nWzpaegyh\njA9xicJbdUm1RhaNJMABAICsIvwBGHa6Alyv4a1rSGXc8V0du1RVUpV2eKsurVZ5UTkBDgAADFqE\nPwCDUigc6h5CmU54a2ht0IftH6qypDJhSOstvFWXVKu8uFx5lpfrxwcAAMg4wh+ArAmFQ2psa+wR\nzhIOqYwLcTvbd6qiuCLt8FZTWkOAAwAASIDwB6BXYQ+rsbUxrfDWdX5n+06VF5f3HDKZJLzFLzFA\ngAMAAMgcwh8QAGEPq6mtqffwlmQmyua2Zo0sGtmvZQQqSyoJcAAAAIME4Q8YIsIeVnNbc8+13xIN\nqYw73tTWpLKisn4tI1BZXKn8vPxcPz4AAAD2EOEPGEDuHqnApRneGlob1NjaqLKisn4tI1BZUqmC\nvIJcPz4AAAByiPAHpMnd1dze3Ht4SzKJSWNbo0oLShOGt5rSmuRrxJVWq6qkigAHAACAfiP8IZDc\nXTvbd6Yd3hpaG7SjdYdKCkp6Dpks6T281ZTWEOAAAACQM4Mm/JnZqZJ+LClP0mJ3vznu/A8lzZDk\nksokfczda6LnQpJekmSS3nH3LyS4P+FvmHF3fdjxYeK13/qYiXJH6w4V5xcnft+tj5koq0qqVJhf\nmOvHBwAAANIyKMKfmeVJWi/ps5K2Slojaba7v5ak/aWSjnT3r0X3m9y9oo/vIPwNQu6uXR27Ug5v\nsed3tO5QYX5h8vfdepnMpKqkSkX5Rbl+fAAAAGDApBr+sj1O7VhJb7j7O9FOrZB0lqSE4U/SHEnf\ni9nv8wGQPe6uls6W3sNbL5OZFOQV9BreDhx1YNLJTAhwAAAAQGZlO/yNk7QpZn+zIoGwBzObKGmy\npNUxh4vN7DlJnZJudvf7stTPYa2lo6Vf4a2htUF5lpd0psnqkmrtP2r/pOeLC4pz/egAAAAAorId\n/hJV7pKN0Zwt6fdxYzgnuvu7ZraPpNVm9rK7v5XxXg4BLR0tScNbd3BLct7Mkoa3mtIafaLmE0nf\nhyspKMn1owMAAADIgGyHv82SJsbsj1fk3b9EZkv6RuwBd383+vmWmdVJmiKpR/hbsGBB93Ztba1q\na2v3oMvZ09rZ2nt4SzITZX1LvSQlnqwkOhPlftX7JZ2RsrSwNMdPDgAAACBT6urqVFdXl/Z12Z7w\nJV/S64pM+LJN0nOS5rj7urh2B0r6o7vvG3OsStIud283s70kPS3prPjJYgZ6wpe2zrZ+hbeG1gaF\nPZx8spI+ZqIkwAEAAABIZFBM+OLuoegMnqv00VIP68xsoaQ17v5AtOlsSSviLj9I0qLocg95km5M\nNktoutpD7YnXfkthJsrOcGev4W1S5SQdOebIhDNRlhaUyow5bAAAAAAMvGGxyPszf38maXhLFO7a\nQ+3d77sleg8u2TIC1SXVGlE4ggAHAAAAYNAYFOv8DQQz8+N+eVzC8JYw3JVWq6ywjAAHAAAAYFgI\nVPgb6s8AAAAAAP2VavjLG4jOAAAAAAByi/AHAAAAAAFA+AMAAACAACD8AQAAAEAAEP4AAAAAIAAI\nfwAAAAAQAIQ/AAAAAAgAwh8AAAAABADhDwAAAAACgPAHAAAAAAFA+AMAAACAACD8AQAAAEAAEP4A\nAAAAIAAIfwAAAAAQAIQ/AAAAAAgAwh8AAAAABADhDwAAAAACgPAHAAAAAAFA+AMAAACAACD8AQAA\nAEAAEP4AAAAAIAAIfwAAAAAQAIQ/AAAAAAgAwh8AAAAABADhDwAAAAACgPAHAAAAAAFA+AMAAACA\nACD8AQAAAEAAEP4AAAAAIAAIfwAAAAAQAIQ/AAAAAAgAwh8AAAAABADhDwAAAAACgPAHAAAAAAFA\n+AMAAACAACD8AQAAAEAAEP4AAAAAIAAIfwAAAAAQAIQ/AAAAAAgAwh8AAAAABADhDwAAAAACgPAH\nAAAAAAFA+AMAAACAACD8AQAAAEAAEP4AAAAAIACyHv7M7FQze83M1pvZlQnO/9DMXjSzF8zsdTOr\njzl3fvS6183svGz3FQAAAACGq6yGPzPLk/QzSadIOkTSHDP7p9g27v5/3H2Kux8l6aeS7oleWy3p\ne5KOkXScpGvMrDKb/QUyra6uLtddABLidxODGb+fGKz43cRQl+3K37GS3nD3d9y9Q9IKSWf10n6O\npOXR7VMkrXL3RnffIWmVpFOz2lsgw/iPBAYrfjcxmPH7icGK300MddkOf+MkbYrZ3xw91oOZTZQ0\nWdLqJNduSXYtAAAAAKB32Q5/luCYJ2k7W9Lv3b3rfDrXAgAAAAB6YR9lrSzc3Ox4SQvc/dTo/lWS\n3N1vTtD2BUnfcPdno/uzJdW6+79E92+V9Ji73xl3HYEQAAAAQKC5e6Li2W6yHf7yJb0u6bOStkl6\nTtIcd18X1+5ASX90931jjlVL+oukoxSpUP5F0tTo+38AAAAAgDQUZPPm7h4ys0sVmawlT9Jid19n\nZgslrXH3B6JNZysyGUzstQ1mdp0ioc8lLST4AQAAAED/ZLXyBwAAAAAYHLK+yHu2mVledIH4P+S6\nL0AXM3vbzF4ysxfN7Llc9weIZWaVZnaXma0zs7+Z2XG57hNgZgdE/858IfrZaGbfzHW/gC5mNt/M\nXjGzl81smZkV5bpPgCSZ2eVm9tfoT69/bw75yp+ZzZc0VVKFu5+Z6/4AkmRmGxV5R7Uh130B4pnZ\nbyQ97u6/NrMCSSPcvSnH3QK6mVmeIstDHefum/pqD2Sbme0t6SlJ/+Tu7WZ2p6QH3f32HHcNAWdm\nhyiyTvoxkjol/UnSv7j7m4naD+nKn5mNl3S6pNty3RcgjmmI//uF4cnMyiWd4O6/liR37yT4YRA6\nSdKbBD8MMvmSyrr+TzNJW3PcH0CSDpL0rLu3uXtI0uOS/jlZ46H+P05/JOnbYv0/DD4u6WEzW2Nm\nX891Z4AY+0rabma/jg6v+4WZlea6U0CcWYr8P9nAoODuWyXdIunvkrZI2uHuj+S2V4Ak6RVJJ5pZ\ntZmNUKQwNiFZ4yEb/szsf0l6z93XKlJl6XNdC2AAfdLdj1bkX8BLzGx6rjsERBUosoTOf7r7UZJ2\nSboqt10CPmJmhZLOlHRXrvsCdDGzKklnSZokaW9JI83s3Nz2CpDc/TVJN0t6RNJDktYqMvwzoSEb\n/iR9StKZ0XerlkuaYWaMu8ag4O7vRj//IWmlpGNz2yOg22ZJm9z9L9H93ysSBoHB4jRJz0f//gQG\ni5MkbXT3+ujQunskfTLHfQIkSe7+a3ef6u61khokvZGs7ZANf+7+XXefGF0Yfrak1e5+Xq77BZjZ\nCDMbGd0uk/Q5RUryQM65+3uSNpnZAdFDn5X0ag67BMSbI4Z8YvD5u6TjzazEzEyRvzvX5bhPgCTJ\nzD4W/ZyoyPt+Sf8Ozeoi70BAfVzSSjNzRf4dW+buq3LcJyDWNyUtiw6v2yjpKznuDyBJir5/epKk\nC3PdFyCWuz9nZr+X9KKkjujnL3LbK6Db3WZWo8jv5jfcvTFZwyG/1AMAAAAAoG9DdtgnAAAAACB1\nhD8AAAAACADCHwAAAAAEAOEPAAAAAAKA8AcAAAAAAUD4AwAAAIAAIPwBAAYVM6sxsxfN7AUz22Zm\nm2P2U1qf1swWm9n+fbT5hpnNyVCfz4r2ca2ZvWJmX+2j/QwzOzbJuQvM7P3o875qZpel8P273S+T\nzwYAGD5Y5w8AMGiZ2fck7XT3HyY4Zz4I/iNmZkWS3pJ0lLu/Z2aFkia5+4ZerrlO0nZ3/48E5y6Q\ndIi7/x8z20vS65IOdvf3+nM/AAC6UPkDAAxm1r1htp+Z/c3MlprZK5LGmNkiM3vOzP5qZv8W0/ZJ\nMzvczPLNrMHMboxW5Z6OBiqZ2XVm9s2Y9jea2f+Y2TozOz56fISZ/T5azbvLzNaY2eFxfayMfu6Q\nJHfv6Ap+ZjbazO6O9vFZMzvWzPaV9DVJV0Sre8cne3h33y5po6Sx0fudGb3P82b2JzPbK9H94p7t\nqOg1a6PPUN7vfxoAgCGN8AcAGEoOlPRDdz/U3bdJutLdj5V0pKTPmdk/JbimUtJj7n6kpGclJR2S\n6e7HSfqOpGuihy6TtM3dD5V0XfR74q/5h6RVkt4xs2VmNtvMukLrTyTdHO3jLEmL3X2jpNskfd/d\nj3L3Z5P1x8wmK/Lf6leihx539+PdfaqklZK+lcL97pA0P/r86yV9L9n3AQCGt5TenQAAYJB4091f\niNmfG32/rkCR6tjBkl6Lu2aXu6+Kbj8vaXqSe98T02ZSdHu6pJskyd1fNrO/JbrQ3b9iZodKOknS\nlZI+I+nC6P4BMWGw0syK+35MfdnMTlIk7F7k7p3R45PM7AeSxkgqUiTMJWVmNZKK3f2/o4eWSLo9\nhe8HAAxDVP4AAEPJh10bZvYJSd+UVOvuR0h6WFJJgmvaY7ZDSv5/fLYlaGNxbeL3u7n7K+7+Y0mn\nSjo7pv0x7j4l+jPR3duS3SPGUnc/XNIJkm7pGqoq6T8l3RI9d4kSP298f5P2GQAQLIQ/AMBQEhtk\nKiQ1SdppZmMlnZLCNel6SpHhmjKzwyQd1OPmZuVmdkLMoSmS3olu/1mRoaNdbY+IbjZH+98rd39O\n0m8VCbmKXrM1Wkk8P6Zpwvu5+weSdsW8VzhP0uN9fS8AYHgi/AEAhpLu2T2jwz/XRX9+o0hQ69Eu\nbrvP+8b5qaS9oxPMXC3pVUmNcW1M0v+NThTzgqTv6qP3Ci+V9Ckzeyl6j69Fj98n6UvRiVuSTvgS\ndbOkC8ysVNJCSfdK+h9J78a0ib9f7POcJ+nHZrZWkfB6fR/fBwAYpljqAQCAJMwsX1KBu7dFh5k+\nLGl/dw/nuGsAAKSNCV8AAEhupKRHYxaXv5DgBwAYqqj8AQAAAEAA8M4fAAAAAAQA4Q8AAAAAAoDw\nBwAAAAABQPgDAAAAgAAg/AEAAABAABD+AAAAACAA/j8H5lp52OuLdwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1e36f30400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#create figures of training and test accuracies\n",
    "plt.figure(figsize=(15,6))\n",
    "f_training_accracies, = plt.plot(trainRatios,trainingAccuracies,label=\"Training Accuracy\");\n",
    "f_testing_accracies, = plt.plot(trainRatios,testAccuracies,label=\"Testing Accuracy\");\n",
    "plt.legend(handles=[f_training_accracies,f_testing_accracies])\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Training Set Ratio')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3kAAAF/CAYAAAD0NgDhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl01fWB///XJysJEAggYQ8QEgTJpihlkSZV9htcERAS\nTWx/Tmem7bHL6bTn+KW040zbGfubLjO/bzs10QRBEddcNlEaKVgqaBYQ4YYlYYlhSYAQELK9f38E\nrgETyHKTz83N83HOPeR+7ud+7useUXzxfn/eb8sYIwAAAACAb/CzOwAAAAAAwHMoeQAAAADgQyh5\nAAAAAOBDKHkAAAAA4EMoeQAAAADgQyh5AAAAAOBDOr3kWZZVYllWoWVZ+ZZlfdTCOb+zLKvYsqwC\ny7ISOjsTAAAAAPiqgC74jAZJScaYs829aFnWPElRxphoy7KmSPq/kr7WBbkAAAAAwOd0xXRN6xaf\n84CkbEkyxvxdUj/LsiK6IBcAAAAA+JyuKHlG0mbLsnZZlvWtZl4fLulYk+cnrh4DAAAAALRRV0zX\nnGaMKbcs6zZJWyzL+swYs73J61Yz7zFdkAsAAAAAfE6nlzxjTPnVX09blvWmpHskNS15xyWNbPJ8\nhKSyG69jWRbFDwAAAECPZoxpbpDsOp1a8izLCpXkZ4yptiyrt6TZklbecNo7kv5J0quWZX1N0jlj\nzMnmrmcMPQ/e6Wc/+5l+9rOf2R0D+Ap+b8Jb8XsT3ozfn/BWlnXLfiep80fyIiS9eXUULkDSy8aY\ndy3LelqSMcb8yRizwbKs+ZZlHZR0UVJ6J2cCAAAAAJ/VqSXPGHNE0lf2vTPG/PGG5//cmTkAAAAA\noKfoitU1AZ+XlJRkdwSgWfzehLfi9ya8Gb8/0d1Z3eU+N8uyTHfJCgAAAACeZlmW/QuvAAAAAGi/\n0aNHq7S01O4Y6GKRkZEqKSlp9/sZyQMAAAC81NWRG7tjoIu19M+9tSN53JMHAAAAAD6EkgcAAAAA\nPoSSBwAAAAA+hJIHAAAAwFYNDQ3q27evjh8/7tFzPenIkSMKCwvr0s9sL0oeAAAAgDbp27evwsLC\nFBYWJn9/f4WGhrqPrVmzps3X8/Pz04ULFzRixAiPntsW3/rWt9zfITg4WEFBQe7v+MADD2jMmDGq\nqqry6Gd2FlbXBAAAALxUd1hdc+zYsXrhhReUnJzc4jn19fXy9/fvwlQd8+yzz+rEiRPKzMy05fNZ\nXRMAAACAbYwxXykkzz77rJYsWaLHH39c/fr108svv6ydO3dq6tSpCg8P1/Dhw/W9731P9fX1khpL\noJ+fn44ePSpJSk1N1fe+9z3Nnz9fYWFhmj59unu/wLacK0kbN27U+PHjFR4eru9+97uaMWOGsrOz\n2/w9Dx06JD+/L+vTvffeqxUrVmjq1Knq06ePHn74YVVWVrq/89SpU6+bUrpv3z7NmjVLAwcO1MSJ\nE/XGG2+0OUNrUfIAAAAAeNxbb72l5cuX6/z581q8eLECAwP1u9/9TpWVldqxY4c2b96sP/7xj+7z\nLev6Aao1a9boueee09mzZzVy5Eg9++yzbT731KlTWrx4sZ5//nmdOXNGY8aM0a5du9r9nW783Fdf\nfVWvvPKKTpw4oc8++0zTp0/XP/zDP+js2bMaO3asfvGLX0iSqqurNXv2bD355JM6c+aMXn75ZT39\n9NMqLi5ud5aboeQBAAAA8LgZM2Zo/vz5kqTg4GDddddduvvuu2VZlkaPHq1vfetb+uCDD9zn3zga\n+OijjyoxMVH+/v5atmyZCgoK2nzu+vXrlZiYKIfDIX9/fz3zzDMaOHCgx77jU089pcjISPXr109z\n5sxRTEyMZs6cKT8/Py1atEj5+fmSpHfeeUfjx4/XsmXLZFmWEhMT9cADD2jdunUey9JUQKdcFQAA\nAECns255d1brdMZtfyNHjrzu+YEDB/SDH/xAH3/8sS5duqT6+npNmTKlxfcPGTLE/XNoaKiqq6vb\nfG5ZWdlXcnhywZaIiAj3zyEhIV95fi1HaWmptm/frgEDBkhqLKn19fV68sknPZalKUbyAAAAgG7K\nGM88OsONUxuffvppxcbG6vDhwzp//rxWrlzZ6YvKDB06VMeOHbvu2IkTJzr1M5szcuRI3X///aqs\nrFRlZaXOnj2rqqoq/e53v+uUz6PkAQAAAOh0Fy5cUL9+/RQSEqLPPvvsuvvxOovD4VB+fr7Wr1+v\n+vp6/dd//ZfOnDnT7uu1t5QuXLhQn376qdasWaO6ujrV1tZq165dcrlc7c5yM5Q8AAAAAO1244hd\nS55//nm9+OKLCgsL07e//W0tWbKkxevc6pqtPXfw4MF69dVX9cwzz2jQoEE6cuSIEhMTFRwc3KrM\n7f3cG4WFhWnz5s1atWqVhg4dqmHDhumnP/2pampq2pXjVtgnDwAAAPBS3WGfvO6koaFBw4YN0+uv\nv67p06fbHadF7JMHAAAAAC3YvHmzqqqqdOXKFf385z9XYGCg7rnnHrtjdSpKHgAAAACftX37do0d\nO1aDBw/Wu+++q7fffluBgYF2x+pUTNcEAAAAvBTTNXsmpmsCAAAAANwoeQAAAADgQyh5AAAAAOBD\nKHkAAAAA4EMoeQAAAADgQyh5AAAAALxSTU2N+vbtq/Ly8i793OLiYg0YMKBLP9OTKHkAAAAA2qRv\n374KCwtTWFiY/P39FRoa6j62Zs2adl936tSpWr16tft5UFCQLly4oCFDhngitlt6ero7b3BwsIKC\ngtzf55FHHlF0dLQqKys9+pldKcDuAG2xsXijksckq1dAL7ujAAAAAD3WhQsX3D+PHTtWL7zwgpKT\nk21M1DZZWVnKysqSJP3kJz9RRUWF/vSnP9mcynO61Ujev2//d0X8Z4QefOVB/fmTP+vzC5/bHQkA\nAADo0YwxX9m4u6GhQb/4xS8UFRWlwYMHKzU1VVVVVZKkS5cuaenSpRo4cKDCw8M1depUnT9/Xj/8\n4Q+1a9cuffOb31RYWJh+9KMf6cqVK/Lz81NZWZkkaenSpXrmmWc0d+5chYWF6d5779WxY8fcn7t+\n/XrFxMRowIABeuaZZ74yMthaBw4cUGBgoPv51KlTtXLlSk2ZMkV9+/bVo48+qsrKSi1evFj9+vXT\ntGnT3Bklae/evbrvvvs0YMAA3XHHHXr77bfbnKEjulXJ25a+TYe/e1iLJi7Se4ff08T/mai7//du\n/fyDn+uTzz9pdld4AAAAAF3r17/+td577z19+OGHOn78uAIDA/XMM89Ikv785z+rvr5en3/+uSoq\nKvSHP/xBQUFB+s///E/dfffdeuGFF1RVVaX/+I//kCRZlnXdtdesWaNf/epXOnv2rIYMGaIVK1ZI\nksrLy7VkyRL99re/1enTpzVs2DB98skn7f4ON37u2rVrtW7dOh07dkxFRUWaMWOGvvOd7+js2bMa\nNWqUnnvuOUmNo5yzZ8/Wt771LVVWVio7O1tPPfWUDh8+3O4sbdWtSp4kDQwdqGVxy/TKo6/o1A9P\n6df3/1rnL5/XknVLNOL/HaGnc59W7oFcXaq9ZHdUAAAAoEf605/+pF/+8peKiIhQUFCQnn32Wb3y\nyiuSpMDAQJ0+fVrFxcXy8/PTXXfdpZCQEPd7bxy4ufH5Y489pvj4ePn7++vxxx9XQUGBJCk3N1f3\n3HOP5s2bJ39/f/3whz9U//79PfadvvnNb2rkyJHq37+/Zs+erQkTJmjGjBny8/PTo48+qvz8fEnS\nm2++qdjYWC1ZskSSdNddd8nhcOj111/3WJZb6Vb35N0o0D9QyWOSlTwmWc/PeV6uCpecLqd+s/M3\nWvbGMs2MnKmUmBQtiFmgEWEj7I4LAAAAeJS10rr1Sa1gVnh2RtyxY8c0f/5892jYtaJWWVmpp556\nSuXl5Xr00Ud18eJFpaam6l//9V+/MnLWkqaLsISGhqq6ulqSVFZWppEjR7pfsyxLw4cP99RXUkRE\nhPvnkJCQrzy/lqO0tFQffPCBe3VOY4zq6+u7dLXObl3ybhQzMEbfn/p9fX/q93Xu8jltPrhZua5c\n/XTrTzWq3yilxKTIEePQ5GGT5Wd1u0FMAAAA4DqeLmeeMmLECL3xxhtKTExs9vWVK1dq5cqVKikp\n0ezZszVp0iQtXbq01UWvOUOHDtW2bdvcz40xOnHiRLuv114jR47UnDlzuvw+vKZ8tun079Vfiyct\n1qqHV+nkD0/qd3N/pyt1V/TkW09q2PPD9NTbT+nNz95UdU213VEBAAAAn/L000/rxz/+sY4fPy5J\nOnXqlJxOpyTp/fff12effSZjjPr06aOAgAAFBDSOPUVERLT73rWFCxfqo48+0qZNm1RfX6/nn39e\n586da/d3aO96Hw8++KDy8/O1du1a1dXVqaamRn//+99VXFzc7ixt5bMlr6kAvwDdG3mvfjXrV9r3\nT/u0I2OH4ofE6392/4+GPj9Uc1fN1X9/9N8qPVdqd1QAAACgW2lu9O3HP/6xZs2apW984xvq16+f\nZsyY4b5n7cSJE3rggQcUFhamuLg4ORwOLVq0SJL0zDPP6KWXXtLAgQP1L//yL1+5/s1G+oYMGaI1\na9boO9/5jm677TaVlZUpNjZWwcHBHf5ebRlh7N+/vzZv3qysrCwNHTpUI0aM0LPPPqu6urp25WgP\nq7usSGlZlumMrFVXqvTuoXfldDm1oXiDhvQZIkeMQ44Yh6YMnyJ/P3+PfyYAAADQGpZlsYJ8O9XX\n12vIkCFyOp2aMmWK3XHapKV/7leP37Jx9viS11R9Q70+OvGRnC6ncl25+rz6c82Pnq+UmBTNjpqt\nsOCwTv18AAAAoClKXtts2rRJ06ZNU1BQkJ577jnl5OTo4MGD7umg3QUlrxOVniuV0+WUs9ipHUd3\naMqIKXJEN47yRQ2I6tIsAAAA6HkoeW3z05/+VH/84x9VX1+vSZMm6fe//32Li794s25R8izL8pO0\nW9JxY8zCG157QtJ/SDp+9dAfjDGZzVyjy0teU9U11Xrv8HuNpc/l1ICQAe5pndNGTlOAX/f62wEA\nAAB4P0pez9RdSt4zku6SFNZCybvLGPPdW1zD1pLXVINp0MdlH7undZaeL9XccXOVEpOiOVFzFB4S\nbndEAAAA+ABKXs/k9SXPsqwRkrIkPSfp+y2UvMnGmO/c4jpeU/JudLzquNa71stZ7NQHJR/ormF3\nuad1jh803u54AAAA6KYoeT1Tdyh5r6mx4PWT9IMWSt6/STotyaXGIni8met4bclr6lLtJW09stU9\nrTM0MFSOGIdSYlI0Y9QMBfoH2h0RAAAA3QQlr2fy6pJnWdYCSfOMMf9sWVaSGkteyg3nhEuqNsbU\nWpb1tKTHjDH3NXOtblHymjLGqKC8QLmuXDldTh2sPKjZUbPliHFo3rh5Ghg60O6IAAAA8GKUvJ7J\n20vev0laLqlOUoikvpLeMMaktXC+n6RKY0z/Zl4zK1ascD9PSkpSUlJSZ8TuNOXV5e5pnVuPbFVc\nRJwc0Q6ljE/RhEET2rTJIgAAAHzf6NGjVVpaancMdLHIyEiVlJQoLy9PeXl57uMrV660v+Rd90GW\n9XU1P11ziDGm/OrPD0n6kTFmWjPv73YjeTdzue6y8kry3Iu3+Fv+7mmdMyNnKjgg2O6IAAAAALyI\nV4zkXfdBTUqeZVkrJe0yxjivjvYtlFQrqVLSt40xrmbe71MlryljjPae2uue1rnv9D7dP/Z+OWIc\nmh89X4N7D7Y7IgAAAACbeV3J6yhfLnk3OnXxlDYWb5Sz2Kkth7Zowm0T3NM6YwfHMq0TAAAA6IEo\neT6ipr5G20q3KfdArnJduao39e7tGZLHJKtXQC+7IwIAAADoApQ8H2SM0f4z+93TOgtPFip5dLIc\nMQ4tiF6goX2H2h0RAAAAQCeh5PUAFZcqtOngJuW6crX50GaNGzBOKTEpcsQ4lDgkkWmdAAAAgA+h\n5PUwtfW12nFsh3IP5MpZ7FR1TbV7Wud9Y+9TaGCo3REBAAAAdAAlr4dzVbjkdDnldDm1u2y3ZkbO\nlCOmsfSNCBthdzwAAAAAbUTJg9u5y+e0+eBm5bpytfHgRo3qN8o9rXPysMnys/zsjggAAADgFih5\naFZdQ53+duxvjaN8xU5VXKrQgugFcsQ4NCtqlvoE9bE7IgAAAIBmUPLQKocqD2l98XrlunK18/hO\nTR853T3KF9k/0u54AAAAAK6i5KHNqq5U6d1D78rpcmpD8QYN6TPEfR/flOFT5O/nb3dEAAAAoMei\n5KFD6hvq9dGJj9zTOssulGl+9Hw5oh2aM26OwoLD7I4IAAAA9CiUPHhU6blS97TO7Ue3a8rwKe5p\nnVEDouyOBwAAAPg8Sh46TXVNtd47/J6cLqfWF69XeK9w97TOaSOnKcAvwO6IAAAAgM+h5KFLNJgG\nfVz2sZwup3JduSo9X6q54+YqJSZFc6LmKDwk3O6IAAAAgE+g5MEWJ6pOuKd1flDyge4adpcc0Y2j\nfOMHjbc7HgAAANBtUfJgu0u1l7T1yNbGxVtcToUGhsoR41BKTIpmjJqhQP9AuyMCAAAA3QYlD17F\nGKOC8gL3tM7iymLNjpqtlJgUzRs3TwNDB9odEQAAAPBqlDx4tfLqcq13rZez2KmtR7YqLiJOjmiH\nUsanaMKgCbKsW/7eBQAAAHoUSh66jct1l5VXkuce5fO3/N3TOmdGzlRwQLDdEQEAAADbUfLQLRlj\ntPfUXuW6cuV0ObXv9D7dP/Z+OWIcmh89X4N7D7Y7IgAAAGALSh58wumLp7WheIOcxU5tObRFE26b\n4J7WGTs4lmmdAAAA6DEoefA5NfU12la6zT2ts66hzr09Q/KYZPUK6GV3RAAAAKDTUPLg04wx2n9m\nv3taZ+HJQiWPTpYjxqEF0Qs0tO9QuyMCAAAAHkXJQ49ScalCmw5ukrPYqc0HNytqQJRSYlLkiHEo\ncUgi0zoBAADQ7VHy0GPV1tdqx7Edyj2QK2exU9U11e5pnfeNvU+hgaF2RwQAAADajJIHXOWqcMnp\ncsrpcmp32W7NjJwpR0xj6RsRNsLueAAAAECrUPKAZpy7fE6bD26Ws9ipjcUbNbLfSPe0zsnDJsvP\n8rM7IgAAANAsSh5wC3UNddp5fKd7WmfFpQotiF4gR4xDs6JmqU9QH7sjAgAAAG6UPKCNDlUe0vri\n9XK6nNp5fKemjZwmR4xDKTEpiuwfaXc8AAAA9HCUPKADqq5UacuhLcp15WpD8QZF9IlwT+ucMnyK\n/P387Y4IAACAHoaSB3hIfUO9dpXtck/rLLtQpvnR8+WIdmjOuDkKCw6zOyIAAAB6AEoe0ElKz5Vq\nffF65bpytePoDt0z/B73KF/UgCi74wEAAMBHUfKALlBdU633D7+vXFeu1hevV3ivcPf2DNNGTlOA\nX4DdEQEAAOAjKHlAF2swDfq47GM5XU7lunJVer5Uc8fNVUpMiuZEzVF4SLjdEQEAANCNUfIAm52o\nOuGe1vlByQe6c+id7mmd4weNtzseAAAAuhlKHuBFLtVe0tYjW+V0OeV0ORUaGOrenmHGqBkK9A+0\nOyIAAAC8HCUP8FLGGBWUF7indRZXFmt21GylxKRo3rh5Ghg60O6IAAAA8EKUPKCbKK8u14biDcp1\n5Wrrka2Ki4iTI9qhlPEpmjBogizrlv8eAwAAoAeg5AHd0OW6y8oryXOP8vlb/u5pnTMjZyo4INju\niAAAALAJJQ/o5owx2ntqr7vw7Tu9T/ePvV+OGIfmR8/X4N6D7Y4IAACALuRVJc+yLD9JuyUdN8Ys\nvOG1IEnZku6SdEbSYmPM0WauQclDj3b64mltKN4gZ7FTWw5t0YTbJrindcYOjmVaJwAAgI/ztpL3\njBpLXFgzJe/bkmKNMf9oWdZiSQ8ZY5Y0cw1KHnBVTX2NtpVuc4/y1TXUyRHduAl78phk9QroZXdE\nAAAAeJjXlDzLskZIypL0nKTvN1PyNklaYYz5u2VZ/pLKjTG3NXMdSh7QDGOM9p/Zr1xXrpwupwpP\nFip5dLIcMQ4tiF6goX2H2h0RAAAAHuBNJe81NRa8fpJ+0EzJ2yNpjjGm7OrzYklTjDGVN5xHyQNa\nofKLSm0s3ihnsVObD25W1IAo97TOxCGJTOsEAADoplpb8gI6OcQCSSeNMQWWZSVJai7QjccsSbQ5\noJ0GhAzQsrhlWha3TLX1tdpxbIdyD+Rq6etLVV1T7Z7Wed/Y+xQaGGp3XAAAAHhYp47kWZb1b5KW\nS6qTFCKpr6Q3jDFpTc7ZKOlnTaZrfm6M+cqygZZlmRUrVrifJyUlKSkpqdOyA77IVeGS0+WU0+XU\n7rLdmhk5U46YxtI3ImyE3fEAAADQRF5envLy8tzPV65c6R3TNd0fZFlfV/PTNf9R0qSrC68skfQg\nC68Ane/c5XPafHCznMVObSzeqJH9RiolJkWOGIcmD5ssP8vP7ogAAABowmvuyXN/UJOSZ1nWSkm7\njDFOy7KCJeVISpRUIWmJMaakmfdT8oBOUtdQp53Hdyr3QK6cxU5VXKrQgugFcsQ4NCtqlvoE9bE7\nIgAAQI/ndSWvoyh5QNc5fPawe1rnzuM7NW3kNDliHEqJSVFk/0i74wEAAPRIlDwAHlF1pUpbDm1R\nritXG4o3KKJPhHta55ThU+Tv5293RAAAgB6BkgfA4+ob6rWrbJd7WmfZhTLNj54vR7RDc8bNUVhw\nmN0RAQAAfBYlD0CnKz1XqvXF65XrytWOozt0z/B73KN8UQOi7I4HAADgUyh5ALpUdU213j/8vnJd\nuVpfvF79e/V3F75pI6cpwK9Tt+UEAADweZQ8ALZpMA36uOzjxsVbip0qOVeiuePmKiUmRXOi5ig8\nJNzuiAAAAN0OJQ+A1zhRdcI9rfODkg9059A73aN84weNtzseAABAt0DJA+CVLtVe0l+O/EW5rlw5\nXU6FBoa6t2eYMWqGAv0D7Y4IAADglSh5ALyeMUYF5QVyupzKdeWquLJYs6NmKyUmRfPGzdPA0IF2\nRwQAAPAalDwA3U55dbk2FG9QritXW49sVVxEnBzRDjliHJp420RZ1i3/mwYAAOCzKHkAurXLdZeV\nV5LnHuXzt/zd0zpnRs5UcECw3REBAAC6FCUPgM8wxmjvqb3uwrfv9D7dP/Z+OWIcmh89X4N7D7Y7\nIgAAQKej5AHwWacvntaG4g1yFju15dAWTbhtghzRDqWMT1Hs4FimdQIAAJ9EyQPQI9TU12hb6Tb3\nKF9tfa17WmfymGT1Cuhld0QAAACPoOQB6HGMMdp/Zr+78BWeLFTy6GQ5YhxaEL1AQ/sOtTsiAABA\nu1HyAPR4lV9UamPxRjmLndp8cLOiBkS5p3UmDklkWicAAOhWKHkA0ERtfa12HNvhHuWrrql2b89w\n39j7FBoYandEAACAm6LkAcBNuCpccrqccrqc2l22WzMjZ8oR01j6RoSNsDseAADAV1DyAKCVzl0+\np80HN8tZ7NTG4o0a2W+kUmJS5IhxaPKwyfKz/OyOCAAAQMkDgPaoa6jTzuM7lXsgV85ipyouVWhB\n9AI5YhyaFTVLfYL62B0RAAD0UJQ8APCAw2cPu6d17jy+U9NGTnNv0RDZP9LueAAAoAeh5AGAh1Vd\nqdKWQ1vkLHZqvWu9IvpEuKd1Thk+Rf5+/nZHBAAAPoySBwCdqL6hXrvKdrmndZZdKNP86PlyRDs0\nZ9wchQWH2R0RAAD4GEoeAHSh0nOlWl+8Xk6XU9uPbtc9w+9xj/JFDYiyOx4AAPABlDwAsEl1TbXe\nP/y+cl25Wl+8Xv179XcXvmkjpynAL8DuiAAAoBui5AGAF2gwDfq47OPGxVuKnSo5V6K54+YqJSZF\nc6LmKDwk3O6IAACgm6DkAYAXOlF1wj2tM68kT3cOvdM9yjd+0Hi74wEAAC9GyQMAL3ep9pL+cuQv\nynXlyulyKjQw1L09w4xRMxToH2h3RAAA4EUoeQDQjRhjVFBe4J7W6apwaXbUbKXEpGjeuHkaGDrQ\n7ogAAMBmlDwA6MbKq8u1oXiDcl252npkq+Ii4uSIdsgR49DE2ybKsm7533cAAOBjKHkA4CMu113W\nByUfKNeVq1xXrvwtf/e0zpmRMxUcEGx3RAAA0AU8WvIsy5os6V5JwyR9IWmvpPeMMZUdDdpalDwA\naJzWuffUXjldTuW6crXv9D7dN/Y+pcSkaH70fA3uPdjuiAAAoJN4pORZlvWkpO9KOiLpY0mnJPWS\nFCNpuhrL3rPGmKMeyHzzoJQ8APiK0xdPa+PBjcp15WrLoS2acNsEOaIdShmfotjBsUzrBADAh3iq\n5P2TpExjzBctvJ4gaaAx5v12J20lSh4A3FxNfY22lW5zj/LV1tfq0YmPKj0hXbERsXbHAwAAHcQ9\neQDQgxljtP/Mfq3es1ovFr6oIX2GKCMhQ0tjl6p/r/52xwMAAO3g6Xvyfi3pX9V4P94mSXGSnjHG\nrOpo0Nai5AFA+9Q31Ou9w+8psyBTmw9u1oKYBcpIyFDymGT5WX52xwMAAK3k6ZJXYIxJsCzrIUkO\nSd+XtM0YE9/xqK1DyQOAjqu4VKE1e9coMz9TlV9U6smEJ/VkwpMa3X+03dEAAMAteLrkfWqMucOy\nrP+V9LoxZpNlWYWUPADovvI/z1dWQZbW7F2j+Ih4pSek6+EJDyskMMTuaAAAoBmeLnm/lPSgGqdr\n3iOpvySnMWZKR4O2FiUPADrHlboreufAO8osyNRHJz7SYxMfU0ZihiYPm8zqnAAAeBGPL7xiWVa4\npCpjTL1lWb0l9TXGlHcwZ6tR8gCg8x07f0zZhdnKKshSr4BeykjM0PK45ey/BwCAF/DUFgoP3+zN\nxpg3bhEiWNI2SUGSAiStM8asvOGcJyT9h6TjVw/9wRiT2cy1KHkA0EWMMfrr0b8qMz9Tb+1/S98Y\n8w1lJGZo7ri5CvALsDseAAA9kqdKXtbVHwdLmiZp69XnyZI+NMY4WhEk1BhzybIsf0k7JH3XGPNR\nk9efkHSXMea7t7gOJQ8AbHDhygWt/XStMgsydeTsEaXGpSo9MV23D7rd7mgAAPQorS15N1072xiT\nboxJlxQ4VeBmAAAgAElEQVQoaaIx5hFjzCOS7rh67JaMMZeu/hisxtG85poaN30AgJfqG9xXT935\nlHZk7NDWJxr/ri/5pWRNz5yuFz55QReuXLA5IQAAaKq1C698ZoyZ0OS5n6RPmx67yXv9JH0sKUrS\nfxtjfnLD609I+jdJpyW5JH3fGHO8meswkgcAXqKuoU6bDm5SZn6mth7Zqgdvf1AZiRm6d9S9LNYC\nAEAn8fTqmn+QFC1pjRpH4pZIOmiM+U4bAoVJekvSPxtj9jU5Hi6p2hhTa1nW05IeM8bc18z7zYoV\nK9zPk5KSlJSU1NqPBwB0klMXT2lV0Spl5mfqSv0VpSekKy0+TSPCRtgdDQCAbi0vL095eXnu5ytX\nrvT46poPS7r36tNtxpg32xrSsqz/o8ZC95sWXveTVGmM6d/Ma4zkAYAXM8Zod9luZeZnau2+tbpn\n+D3KSMjQwvELFRwQbHc8AAC6PY9vodDOEIMk1RpjzluWFSJps6RfGmM2NDlnyLWtGCzLekjSj4wx\n05q5FiUPALqJS7WX9OZnbyqrIEuFJwu1dNJSZSRmKGFIgt3RAADotjw9XfNhSb9S4yqb1tWHMcaE\n3eJ9sZJeUuMCL36SXjXGPGdZ1kpJu4wxTsuy/k3SQkm1kiolfdsY42rmWpQ8AOiGjpw9opcKX1JW\nQZYGhgxURmKGHo99XANCBtgdDQCAbsXTJe+gpBRjzGeeCNcelDwA6N4aTIO2HtmqrIIsrXet15xx\nc5SRkKH7x94vfz9/u+MBAOD1PF3ydhhjpnskWTtR8gDAd5z94qxe2fuKMgsyVV5drifjn9STCU8q\nakCU3dEAAPBani55v5U0RI2rY165dtwY80ZHQrYFJQ8AfNOek3uUVZClVUWrNPG2icpIzNAjEx5R\n76DedkcDAMCreLrkZTVz2BhjMtoTrj0oeQDg22rqa+R0OZWZn6kPj32oRyY8oozEDH1txNfYew8A\nAHnJ6pqeRMkDgJ6j7EKZcgpzlFmQKT/LTxkJGUqNT9WQPkPsjgYAQJeqr5dOnJCOHJGSkjw7kjdC\n0u8lTVfjZujbJX3PGHO8o6Fbi5IHAD2PMUYfHvtQmfmZemP/G5oZOVPpCelaEL1Agf6BdscDAKDD\njJEqKhpL3OHDjb9eexw+LB07Jg0cKI0ZI334oWdL3hZJqyXlXD20XNIyY8ysDn2jNqDkAUDPVl1T\nrXX71ikzP1OuCpeWxy1XekK67hh8h93RAAC4qYsXpZKSloucv780dmxjkbv2uPY8MlIKCWm8jqfv\nySswxiTc6lhnouQBAK5xVbj0YsGLyi7M1vCw4cpIyNCSSUvUr1c/u6MBAHqgurrGEbeWStz589Lo\n0c2XuDFjpPDw1n2Op0vee5JelLTm6qGlktKNMfe1Lk7HUfIAADeqb6jXu4feVWZBprYc2qKU8SlK\nT0hX0ugk+Vl+dscDAPgIY6RTp64vbk1/LiuTIiJaLnFDh0p+HvhjydMlb5SkP0iaqsZ78j5U4z15\npR0N2lqUPADAzZy5dEar96zWC/kvqOpKldIT0vVE/BOK7B9pdzQAQDdw4ULzJe7aIySk5RI3apQU\nHNz5GVldEwDQIxljlF+er8z8TL2y9xXdOfROZSRm6MHbH1SvgF52xwMA2KSmRjp6tOUplZcutVzi\nxoyRwsLs/gaeH8l7SY0jd+euPg+X9Dz75AEAvNnlust6e//byizI1O6y3VpyxxJlJGbozqF3svce\nAPiYhgapvLzlEnfypDRsWMslLiJC8vY/Gjxd8vKNMYm3OtaZKHkAgI44ev6oXip4SVkFWeoT1EcZ\niRlaFrtMt/W+ze5oAIBWOneu5fviSksbR9taKnEjR0qB3Xz3HU+XvEJJScaYs1efD5D0gTEmtsNJ\nW4mSBwDwhAbToG2l25SZn6l3Dryj+8fer4zEDM2Omq0AvwC74wFAj3b5cmNZa2k0rq6u5RI3erTU\np4/d36BzebrkpUn6iaR1alx45TFJzxljcm76Rg+i5AEAPO385fN69dNXlVWQpaPnjyotLk3piemK\nGRhjdzQA8En19Y0rUbZU4s6caRxxa67EjR3buCm4t0+p7EweX3jFsqyJkr4hyZL0vjFmX8citg0l\nDwDQmfad3qes/CzlFOUoemC00hPStWjiIvUN7mt3NADoNoyRKitbLnHHjkkDBrRc4oYPb9wYHM3r\njJI3Q1K0MSbLsqzbJPUxxhzpYM5Wo+QBALpCbX2tNhRvUGZBpraVbtNDtz+kjMQMTR85ncVaAECN\nq1CWlDR/X9yRI437wbVU4iIjG7ciQPt4errmCkmTJY03xsRYljVM0mvGmOkdj9o6lDwAQFcrry7X\nqqJVyszPVF1DndIT0pUWn6bhYcPtjgYAnaauTjp+vOXRuHPnGstacyVuzBgpPNzub+C7PF3yCiQl\nSvrk2oqalmUVGWPiOpy0lSh5AAC7GGP00YmPlJmfqdf2vaapI6cqIyFDKeNTFOQfZHc8AGgTY6TT\np1sucSdOSIMHt1zihg5tHK1D1/N0yfvIGHOPZVmfGGPutCyrt6S/UfIAAD3NpdpLen3f68osyNSn\npz7VsthlSk9MV1xEl/2RCAC3VF3d8lYDJSVScHDLJW7UqMbX4X08XfJ+KCla0ixJ/y4pQ9JqY8zv\nOxq0tSh5AABvc/jsYb1Y8KJeLHhRg3sPVkZihpZOWqrwEOYqAehctbXS0aMtj8ZdvHh9gbvxHrmw\nMLu/AdqjMxZemSVp9tWn7xpjtnQgX5tR8gAA3qq+oV7vH3lfmfmZ2nRwk+ZFz1NGQobuG3uf/Czm\nNAFoO2Ok8vKWS1x5uTRsWMslLiKiZ2814Ks8XvKuXnSgpJmSjhpjPu5Avjaj5AEAuoPKLyq1Zs8a\nZRZk6sylM3oy/kk9mfCkxoSPsTsaAC9z/nzLJa60VOrbt+WNv0eOlAID7f4G6GoeKXmWZTkl/Ysx\nZq9lWUMlfSJpt6QoSX8yxvyXpwLfCiUPANDdFJYXKqsgSy/veVmxg2OVkZihhyc8rNDAULujAegC\nV640lrWWilxtbcslbvRoqU8fu78BvI2nSt6nxpg7rv78U0m3G2PSLMvqK2kHC68AAHBrV+quKNeV\nq8z8TO08vlOLJi5SRmKG7hl+D3vvAd1YQ4NUVtZyiTtzRhoxouU94wYOZEol2sZTJa/AGJNw9ef3\nJf2vMeaVG1/rCpQ8AIAvOFF1QtmF2cosyFSQf5AyEjK0PG65IvpE2B0NwA2Mkc6ebbnEHT0qDRjQ\ncokbPlzy97f7W8CXeKrk5Up6V9JxSZmSxhhjzlmWFSJp97VRvq5AyQMA+BJjjLYf3a7Mgky9+dmb\nShqdpIzEDM0bN0+B/txoA3SVqqrGKZUlJV8+mhY5y2q5xEVGSiEhNn8B9CieKnmDJf1c0lBJ/22M\neffq8WRJdxlj/tNDeW+JkgcA8FUXrlzQa/teU2Z+pg6dPaTUuFSlJ6Rrwm0T7I4GdHtVVdcXuJKS\n60vdlSuN979de0RGXl/qwtkRBV6kU1bXtBMlDwDQExw4c0BZBVnKLsxWZP9IZSRkaPGkxQoLZlMr\noDnnz391JK7po6bmy4VMrpW4pqWO++LQnXhqJO9Pkn5njNnbzGu9JS2WdMUY83JHwrYGJQ8A0JPU\nNdRp88HNyizI1PuH39cDtz+g9IR0zYycyd576FHOn2+5wJWUSHV115e2Gx8DBlDi4Ds8VfISJP1U\nUqykvZJOS+olKVpSmBrv0/u/xpgrngh906CUPABAD3X64mmtKlqlzIJMXaq9pPSEdD0R/4RG9htp\ndzSgw86da3kq5bUS13Qk7sYROUocehKPTte0LKuPpMlqvDfvC0mfGWMOdDhlG1DyAAA9nTFGH3/+\nsTLzM/Xqp6/q7mF3Kz0hXQ/c/oB6BfSyOx7QrBtL3I2Phoabj8SFh1PigGu4Jw8AAB/2Re0XenP/\nm8oqyFL+5/laOmmpMhIzlDg00e5o6EGMuXmJKy1tLHE3G4mjxAGtR8kDAKCHKDlXopcKXlJWQZbC\nQ8KVnpCuZbHLNDB0oN3R0M1d2yeuuWmU1x7SzUfi+venxAGeQskDAKCHaTANyivJU2Z+ppwup2ZH\nzVZGYoZmjZ0lfz92ZMZXNS1xLT2u7RN3sxIHoGt0SsmzLKu3MeZih5K1EyUPAIDWO3f5nF7Z+4oy\n8zNVdqFMT8Q/ofTEdI0bMM7uaOhCxkiVlTdf2MTf/6vbClDiAO/k6YVXpkn6s6Q+xphRlmXFS3ra\nGPOPHY/aOpQ8AADaZ++pvcrKz9KqPat0+6DblZGQoUcnPqreQb3tjoYOaq7E3fgICGi5wEVGUuKA\n7sTTJe/vkh6V9I4xJvHqsb3GmEkdTtpKlDwAADqmpr5G613rlVmQqe1Ht+uRCY8oIzFDU0dMlcVN\nU17JGKmi4uYLmwQGtjwSR4kDfIvHS54xZoplWflNSl6hMSbeA1lbhZIHAIDnfH7hc+UU5SgzP1OS\nlJ6QrrT4NA3tO9TmZD2LMdKZMzdf2CQo6OYjcf362ZMdQNfzdMlbJ+k3kv4g6WuSvitpsjFmSUeD\nthYlDwAAzzPG6G/H/6as/Cyt+2ydZoyaoYyEDC2IWaAg/yC743V7TUtcS49evb66rQAlDkBzPF3y\nBkn6raT7JVmS3pX0PWNMxS3eFyxpm6QgSQGS1hljVt5wTpCkbEl3STojabEx5mgz16LkAQDQiS7W\nXNS6feuUWZCp/Wf2a1nsMmUkZmjS4C67O6PbMUY6fbr5aZTNlbjmRuLCwuzJDqD78ZotFCzLCjXG\nXLIsy1/SDknfNcZ81OT1b0uKNcb8o2VZiyU91NwIISUPAICuU1xRrBcLXtRLhS9pWN9hykjM0JJJ\nS9S/V8+6waulEte0zIWE3HwkjhIHwFM8PZI3RtJ3JI1W44icJMkYs7ANgULVOKr3bWPMribHN0la\nYYz5+9UiWG6Mua2Z91PyAADoYvUN9dpyeIsy8zP17qF3tSBmgTISMpQ8Jll+lp/d8TrMGOnUqZuX\nuNDQm4/E9e1rT3YAPY+nS16hpBck7ZHUcO24MeaDVrzXT9LHkqIk/bcx5ic3vL5H0hxjTNnV58WS\nphhjKm84j5IHAICNKi5VaPWe1Xoh/wWdu3xO6QnpeiLhCY3uP9ruaC0yRjp5suWFTUpLpd69my9v\nlDgA3qZTVtfsYKAwSW9J+mdjzL4mx/dKmt2k5B2UdLcx5uwN7zcrVqxwP09KSlJSUlJHIgEAgHbK\n/zxfmfmZWrN3jRKGJCgjMUMP3f6QQgJDujRH0xLX0khcnz43H4nr06dLIwNAq+Xl5SkvL8/9fOXK\nlR4teY9LilbjgitXrh03xnzSlpCWZf0fSdXGmN80ObZR0s+aTNf83BgzuJn3MpIHAICXuVx3We8c\neEeZ+ZnaVbZLi+9YrPSEdE0eNtkje+81NDRf4q6NypWWNo60tVTgKHEAfImnR/L+XVKqpEP6crqm\nMcZ84xbvGySp1hhz3rKsEEmbJf3SGLOhyTn/KGnS1YVXlkh6kIVXAADofo6dP6aXCl9SVkGWQgND\nlZGQoeVxy3Vb76/cau/WUom79jh6tOUSd63I9e7dyV8MALyEp0veQUkTjTE1bQwRK+klSX5XH68a\nY56zLGulpF3GGOfVbRZyJCVKqpC0xBhT0sy1KHkAAHQDDaZBfy39qzILMvX2/rf1jTHfUEZihuaO\nm6vysgCtWydt2iQdPtxY4vr1a7nEjRpFiQOAazxd8t6S9P8YY055Ilx7UPIAAOh+qq5U6Y/b1+p/\n/papExdL5L83TfOGpCt1/njdfnvjSFxoqN0pAaB78HTJy5MUJ2mXrr8nr9VbKHQUJQ8AgO7j+HFp\n3Trptdek/fulhQulKY7PVNw7Sy/vzVbUgCilxaXpsTseU3hIuN1xAaBb8HTJ+3pzx1uzhYKnUPIA\nAPBuzRW7xx6T7rtPCgr68rza+lptOrhJOUU52nxos2aNnaW0+DTNHTdXQf5BLX8AAPRwHi153oCS\nBwCA92ltsWvJucvn9Nqnrym7KFv7z+zXkjuWKDU+VXcPu9sjq3MCgC/xSMmzLGu7MWaGZVkXJDU9\n0VLj6pphHY/aOpQ8AAC8w7Vit3atdOCA9MAD0qJFrS92LTl89rBWFa1STlGO/C1/pcWnaVnsMkX2\nj/RceADoxjxV8vKNMYkeTdZOlDwAAOzTWcWuOcYY7Ty+UzlFOVr76VrFRsQqNS5Vj058VGHBXfb3\nywDgdTxV8j4xxtzp0WTtRMkDAKBrHTv25VTMzi52LblSd0Xri9crpyhHfznyF82Lnqe0uDTNipql\nAL+ArgkBAF7CUyXvuKTftPS6MabF1zyNkgcAQOfzhmLXkopLFXr101eVXZitknMlejz2caXGpSph\nSAL37wHoETxV8j6X9P+p8R68rzDGrGx3wjai5AEA0Dm8udi1xFXhUk5hjlbtWaU+QX2UGpeqZbHL\nNDxsuN3RAKDTMF0TAAC0qLli99hj0je+4b3FrjkNpkHbj25XTmGOXv/sdU0eNlmpcal6aMJD6hPU\nx+54AOBRLLwCAACu4yvFriVf1H6hdw68o5yiHG0/ul0Lxy9UWnyakkcny9/P3+54ANBhnip5A4wx\nlR5N1k6UPAAA2u5asVu7VnK5pAcfbJyK6SvFriUnq0/qlb2vKLsoWyerT2pZ7DKlxqdq0uBJdkcD\ngHZjM3QAAHqolordffdJgYF2p+t6n576VDlFOVpVtEqDew9WalyqHo99XBF9IuyOBgBtQskDAKAH\nodjdWn1DvfJK8pRTlKO39r+l6aOmKzUuVQ+Mf0AhgSF2xwOAW6LkAQDg444e/fIeu+Li61fFpNjd\n3MWai3pr/1vKLsrWrhO79NDtDyktPk33Rt4rP8vP7ngA0CxKHgAAPohi53llF8q0es9qZRdmq+pK\nlZbHLVdqXKrGDxpvdzQAuA4lDwAAH9Fcsbu2KibFzrMKywuVXZit1XtXa1S/UUqLS9PiSYs1KHSQ\n3dEAgJIHAEB3RrGzV11Dnd47/J6yC7O1oXiDkkYnKTUuVY4Yh4IDgu2OB6CHouQBANDN3Fjsmm53\nQLGzT9WVKr2+73XlFOWo8GShFk1cpLT4NE0dMVWWdcv/1wIAj6HkAQDQDVwrdmvXSgcPUuy83dHz\nR/Vy0cvKLspWbX2tUuNStTxuuaIGRNkdDUAPQMkDAMBLUey6P2OMPv78Y2UXZuuVva8oZmCM0uLT\ntGjiIoWHhNsdD4CPouQBAOBFKHa+q7a+VpsOblJOUY42H9qs2VGzlRqXqrnj5irIP8jueAB8CCUP\nAACblZZ+eY8dxa5nOPvFWb227zXlFOXowJkDWnzHYqXFp2nysMncvwegwyh5AADYoLli99hjUnIy\nxa6nOXz2sFYVrVJ2YbYC/QPd9++N6jfK7mgAuilKHgAAXYRih5sxxmjn8Z3KLszWa/teU2xErFLj\nUvXoxEcVFhxmdzwA3QglDwCATnRjsXvoocapmBQ73MyVuitaX7xe2YXZyivJ0/zo+UqNS9WsqFkK\n8AuwOx4AL0fJAwDAwyh28KQzl87o1b2vKqcoR6XnS7V00lKlxacpPiKe+/cANIuSBwCAB1wrdmvX\nSocOUezQOQ6cOaBVRauUU5SjvsF9lRaXpmVxyzSs7zC7owHwIpQ8AADaqWmxO3z4y1UxKXbobA2m\nQX8t/atyinL0xmdvaPKwyUqLT9NDtz+k3kG97Y4HwGaUPAAA2qC0tHEa5muvUezgHb6o/ULvHHhH\n2UXZ2nF0hx64/QGlxqUqeXSy/P387Y4HwAaUPAAAbqG5YvfYY1JSEsUO3uVk9Umt2btGOUU5Oll9\nUstilyktPk13DL7D7mgAuhAlDwCAZlDs0N19eupT5RTlaFXRKkX0iVBqXKqWTlqqiD4RdkcD0Mko\neQAAXFVS8uWqmIcPf7l4CsUO3Vl9Q73ySvKUXZStt/e/remjpistLk0Lxy9USGCI3fEAdAJKHgCg\nR6PYoSe5WHNRb+5/U9mF2dpdtlsPT3hYqXGpujfyXvlZfnbHA+AhlDwAQI9zrditXSsdOUKxQ890\nouqEVu9ZreyibF24ckHL45YrNS5V4weNtzsagA6i5AEAegSKHdA8Y4wKTxYqpzBHq/eu1qh+o5QW\nl6bFkxZrUOggu+MBaAdKHgDAZ7VU7JKTpYAAu9MB3qeuoU7vHX5P2YXZ2lC8QUmjk5QWn6YF0QsU\nHBBsdzwArUTJAwD4lJKSL1fFvFbsrq2KSbEDWq/qSpVe3/e6souytefkHi2auEip8amaOmKqLOuW\n/+8IwEaUPABAt0exAzpX6blSvbznZeUU5ai2vlZp8WlaHrdcY8PH2h0NQDO8ouRZljVCUrakIZLq\nJf2vMeZ3N5zzdUlvSzp89dAbxph/beZalDwA6AFuLHYPP/zlPXYUO6BzGGO0u2y3copy9MreVzR+\n0HilxqVq0cRFCg8JtzsegKu8peQNkTTEGFNgWVYfSR9LesAYs7/JOV+X9ANjzMJbXIuSBwA+imIH\neI+a+hptOrhJOUU5evfQu5odNVtpcWmaO26uAv1ZzQiwk1eUvK98mGW9Jen3xpj3mxz7uqQfGmNS\nbvFeSh4A+BCKHeD9zn5xVq/te03ZhdlyVbi0ZNISpcalavKwydy/B9jA60qeZVmjJeVJmmSMqW5y\n/OuS1kk6LqlM0o+MMfuaeT8lDwC6uSNHvlwVs7T0+u0OKHaAdztUeUirilYppyhHgf6BSo1L1fK4\n5RrVb5Td0YAew6tK3tWpmnmSfmGMebuZ1xqMMZcsy5on6bfGmJhmrkHJA4BuiGIH+BZjjP52/G/K\nKczR2n1rFR8Rr9S4VD0y8RGFBYfZHQ/waV5T8izLCpDklLTRGPPbVpx/RNJdxpjKG46bFStWuJ8n\nJSUpKSnJw2kBAJ5w5MiXUzGvFbvHHpO+/nWKHeBLrtRdkdPlVE5RjvJK8jQ/er7S4tN0/9j7FeDH\nv+xAR+Xl5SkvL8/9fOXKlV5T8rIlnTHGfL+F1yOMMSev/nyPpLXGmNHNnMdIHgB4MYod0LOduXRG\nr+59VdlF2Tp6/qgen/S40uLTFD8k3u5ogM/wipE8y7KmS9omaY8kc/XxU0mRkowx5k+WZf2TpG9L\nqpX0haRnjDF/b+ZalDwA8DI3Frtri6dQ7ICe7cCZA8opytGqolUKCw5TalyqlsUt07C+w+yOBnRr\nXlHyPImSBwDegWIHoLUaTIP+WvpX5RTl6I3P3tDdw+9WalyqHrr9IfUO6m13PKDboeQBADyGYgeg\no76o/ULvHHhH2UXZ+vDYh1o4fqHS4tKUNDpJ/n7+dscDugVKHgCgQyh2ADrLyeqTWrN3jbILs3Xq\n4iktj1uu1LhU3TH4DrujAV6NkgcAaLNrxW7tWunoUYodgM6399Re5RTm6OU9LyuiT4TS4tK0NHap\nBvcebHc0wOtQ8gAArdK02B079uWqmDNnUuwAdJ36hnr9peQvyi7M1jsH3tGMUTOUGpeqheMXKiQw\nxO54gFeg5AEAWnT48JdTMSl2ALxNdU213vzsTeUU5Wh32W49POFhpcWnacaoGfKz/OyOB9iGkgcA\nuM6Nxe7aVEyKHQBvdqLqhFbvWa3somxduHJBqXGpSo1PVczAGLujAV2OkgcAoNgB8BnGGBWeLFR2\nYbZW71mt0f1HKy0+TYvvWKyBoQPtjgd0CUoeAPQQDQ3SiRPSgQOSy3X9rxcvUuwA+J66hjptObRF\nOUU52lC8QcljkpUal6oF0QsUHBBsdzyg01DyAMDHnD//ZXlrWuSKi6WwMGn8eCkm5vpfo6Ikf7af\nAuDDqq5Uad2+dcopytGek3u0aOIipcWn6WsjvibLuuX/CwPdCiUPALqh2trGKZbNjcpduPBleWta\n5KKjpX797E4OAPYrPVeql/e8rOzCbNWbeqXGpWp53HKNDR9rdzTAIyh5AOCljJHKy5sflTt6VBo+\n/PoSd+3n4cMl/lIaAG7NGKPdZbuVXZitVz99VeMHjVdqXKoeu+Mx9e/V3+54QLtR8gDAZhcvfnU0\nzuVqfAQFNT8qFxUlBXM7CQB4TE19jTYd3KTswmxtObxFc6LmKDUuVXPHzVWgf6Dd8YA2oeQBQBeo\nr5dKSpoflauokMaNa/5euQED7E4OAD3P2S/Oau2na5VTlCNXhUtLJi1RWnya7hp6F/fvoVug5AGA\nhxjTWNhuLHEHDjTePxcR0fyo3MiRLHoCAN7qUOUhrSr6/9u78+C67vKM489ry7sd77ssWZItZ7Nj\nHLxMYideJQIzYZh2Gig0LYXSliUdaIGWoWQy6ZQyLZSlzLQpEBoIDBMIUAhFsuPY2A6OsR1jx3G8\nXMm2vG+yvEq2pLd/nKNj5eoolhNL596j72dGo7uce+97J9dSHr2/83u/r6e2P6X+ffvr4VkP6/2z\n3q+i4UVJlwZ0ipAHADeosTHYqTKuK9fa2jHElZcHm54MHpx05QCAN8vd9WLdi/re9u/pmVef0V3j\n79J7bn2PKsoqVD66nA4fcgohDwBitLZKhw51DHF79khHj0olJfFdubFj2fQEANKusblRv9r7Kz23\n5zlVZapU0KdAlWWVqiir0LLSZWzagsQR8gD0amfPdlxauWdP0KkbMaJjiJsxQ5o6lWHhAICAu2vX\nqV2q2lel6ppqrT+4XjPHzYxC39zJc1XQh18a6FmEPACpd+XKtZly2V25S5eCAJfdlSsvl4YNS7py\nAEC+aWxu1PqD66PQV9dQp6UlS1VZVqnKaZWcy4ceQcgDkAruwTLKuOHgdXVSYWH8uXKTJrG8EgDQ\nfY6eP6rqTLWqa6q1MrNSowaNirp8i6cu1pD+Q5IuESlEyAOQV86fD5ZSxnXlBg2KHw5eVhbMmwMA\nIEmt3qptx7ZFXb7NRzZr7qS5UZdv1vhZ6mN9ki4TKUDIA5BzmpuDmXJxXbn6+msz5bK7ciNHJl05\nAGxUkC0AABLmSURBVABdd77pvNYeWBuFvrONZ1VRVqGK0gpVlFVo/NDxSZeIPEXIA5AId+nkyY4h\nbvduqbZWmjAhvis3ZYrUhz9yAgBSqLa+Nlraubp2tYqHF0dLOxcWLdSAggFJl4g8QcgD0K0uX762\nvDI70EnXAlz7QDdtWrD0EgCA3qq5tVmbDm+Kunw7T+zUwqKF0dLOGaNnMJsPnSLkAXjLWlulgwfj\nu3LHj0ulpfFduTFj2PQEAICuOHP5jFbXrlbVvipVZaokKeryLS9drpGDOGcB1xDyAHRZfX38hif7\n9kmjRsUPBy8uZqYcAAA3k7tr9+ndUZdv3YF1un3s7VHom184n9l8vRwhD8DrNDVJmUzH4eC7d0uN\njfHDwadPl4YOTbpyAAB6p6bmJm2o2xB1+Q40HNDSkqWqKK1Q5bRKTR0xNekS0cMIeUAv5C4dORLf\nlTt0SCoqiu/KTZjA8koAAHLdsQvHtDKzUtU11arOVGv4gOFRl29JyRIN7c9fZtOOkAek2LlzQXDL\n7srt2SMNGRLflSspYaYcAABp0eqt2n58e7S0c9PhTXr7pLdHXb7ZE2Yzmy+FCHlAnrt69dpMueyu\nXENDsJQyuytXXi6NGJF05QAAoKddvHIxms1XlanSmctntKJsRTSbb+KwiUmXiJuAkAfkAXfpxIn4\nMQT790uTJr1+KHhboJs8mZlyAACgcwfOHohm8z1f87wKbymMlnYuKl6kgQUDky4RbwIhD8ghly5d\nmymXHej69o0fQzBtmjSQn78AAOAtam5t1uYjm6OlnTuO79C9RfdGSztvG3Mbs/nyBCEP6GEtLcFM\nubiu3MmT12bKZQe60aOTrhwAAPQmZxvPvm42X4u3RMs6l5cu1+jB/M9JriLkAd3k9On44eCZTDAE\nPK4rV1wcdOwAAAByibtr75m9UZdv7f61unXMraosq1TltErNnzxf/fr2S7pMhAh5wFvQ1BQMAo/r\nyl25ci3AtQ9006cHO1sCAADkq6bmJv320G+j0Jc5k9GSkiXR0s7SkaVJl9irEfKA62htlQ4fjh8O\nfuRI0H2L68qNH89MOQAA0DucuHhCq2pWqSpTpepMtYb0GxJ1+ZZMXaJhA4YlXWKvQsgDQg0NHbtx\ne/YEG6EMG/b6INf2vaRE6sfKBAAAgIi7a8eJHVGXb+OhjZozcU7U5ZszcQ6z+boZIQ+9ytWrUk1N\nfFfu/Pn4MQTTp0vDhyddOQAAQH66dPWS1u5fq+pMtaoyVTp56aSWly6PNnGZfMvkpEtMHUIeUsdd\nOn48fgzBgQPB7Li4rtzkySyvBAAA6G51DXXRbL5VNas0cejEaGnnoqJFGtRvUNIl5j1CHvLWxYtB\ncItbYtm/f3xXrqxMGjAg6coBAAAgSS2tLdpydEu0tHPbsW26Z8o90dLOO8bewWy+NyEnQp6ZFUp6\nStIESS2S/tvdvx5z3NclPSDpoqQ/c/dtMccQ8lKkpSXovsV15U6dCgaBZ294MmOGNGpU0pUDAADg\nRjU0Nmh17epoaWdTS5MqyipUUVqhFWUrNGbwmKRLzAu5EvImSJrg7tvMbKikLZLe7e6vtTvmAUkf\nd/d3mdl8SV9z9wUxz0XIy0OnTsWPIaipkcaNi+/KTZnCTDkAAIC0cndl6jNRl2/N/jUqH10edfkW\nFC5Q/779ky4zJ+VEyOvwYmY/k/QNd3++3W3/KekFd/9ReH2XpMXufjzrsYS8HNXY2HGmXNvllpaO\n3bjy8mDTk8GDk64cAAAASbvSckUbD22MQt/e03t1/9T7o9A3bdS0pEvMGTkX8sxsqqQ1ku509wvt\nbv+FpC+6+4vh9VWSPuPuW7MeT8hLUGurdOhQfFfu6NFg5EBcV27sWDY9AQAAQNedvHhSq2pWqbqm\nWtWZag0sGBgFviVTl2j4wN67PXpOhbxwqeYaSY+7+8+z7vulpH/OCnmfdveXs44j5PWAs2fjxxDs\n2xeMG4jrypWUSAUFSVcOAACAtHF37Ty5M+ryvVj3omZPmB2Fvrsn3q2+fXrPeT45E/LMrEDSLyX9\nn7t/Leb+7OWar0m6P2655qOPPhpdX7x4sRYvXtydpafWlSvBOXFxyysvXQqCW3ZXrrw8GBwOAAAA\nJOXy1ctad3BdFPqOnD8SzearnFapwlsKky7xplqzZo3WrFkTXX/sscdyJuQ9JemUu3+qk/vfKelj\n4cYrCyR9lY1X3jr3YBllXFeurk4qLOy4tLK8XJo0ieWVAAAAyA+Hzx3WypqVqspUaWVmpcYNGRfN\n5ruv+D4N7peuTSByopNnZvdK+o2kHZI8/PqcpGJJ7u5PhMf9h6R3KBih8MHs8/HCYwh5MS5ciJ8n\nt2ePNHBg/HDwsrJg3hwAAACQFi2tLXr52MtRl2/r0a1aULgg6vLNHDcz72fz5UTIu5l6c8hrbpb2\n74/vytXXBzPl4rpyI0cmXTkAAACQjHNN5/RC7QvRbL6LVy+qoqxClWWVWl66XOOGjEu6xBtGyMsz\n7tdmymV35WpqpAkT4rtyU6ZIffokXT0AAACQ2zJnMqrOVKu6plov1L6gslFlUZfvnin35MVsPkJe\njrp8Wdq7t+MSy927g/vbOnHtg9y0adKgQcnWDQAAAKTF1ZareunwS9HSztdOvab7iu+LQt/0UdNz\ncmknIS9Bra3B5iZxXbljx6TS0viu3JgxbHoCAAAA9LTTl04Hs/nCpZ39+vaLAt/SkqUaMXBE0iVK\nIuT1iPr6+OHg+/ZJo0Z1PEduxgypuJiZcgAAAECucnftOrUr6vJtOLhBM8fPjELf3ElzE5vNR8i7\nSZqaOp8pd/ly/HDw8nJp6NAeLxUAAADATdbY3Kj1B9dHoa+uoU7LSpdFoa9oeFGP1ULIuwHu0pEj\n8V25Q4ekoqKOSytnzAg2Q2F5JQAAANB7HD1/NNrAZWVmpUYPHh0FvvuL79eQ/kO67bUJeTHOn48f\nQ7BnjzRkSHxXrrSUmXIAAAAAOmr1Vm07ti3q8m0+slnzJs+LQt+s8bPUx27eVvi9NuQ1N0u1tfFd\nuYYGafr0jl258nJpRG6cSwkAAAAgT51vOq+1B9ZGoa+hsUErylaosqxSK0pXaPzQ8W/p+VMd8tyl\nEyfiu3L790sTJ8Z35QoLmSkHAAAAoGfU1tdGSztX167W1BFToy7fvVPu1YCCATf0fKkMeR/4gEdh\nrm/fjufIlZcHM+UGDky6WgAAAAC4prm1WZsOb1LVvipVZar06slXtah4kSrLKlVRVqEZo2dcdzZf\nKkPed77jUZgbMybpigAAAADgzTlz+Yyer3k+ms1nZlGXb1nJMo0cNLLDY1IZ8vKlVgAAAADoKnfX\n7tO7o3P51h1YpzvG3RGFvnmT56mgTwEhDwAAAADyUVNzkzbUbYiWdh5oOKClJUv17EPPEvIAAAAA\nIN8du3BMKzMr9fDshwl5AAAAAJAWXV2uyUABAAAAAEgRQh4AAAAApAghDwAAAABShJAHAAAAAClC\nyAMAAACAFCHkAQAAAECKEPIAAAAAIEUIeQAAAACQIoQ8AAAAAEgRQh4AAAAApAghDwAAAABShJAH\nAAAAAClCyAMAAACAFCHkAQAAAECKEPIAAAAAIEUIeQAAAACQIoQ8AAAAAEgRQh4AAAAApAghDwAA\nAABShJAHAAAAAClCyAMAAACAFCHkAQAAAECKEPIAAAAAIEUIeQAAAACQIt0a8szs22Z23My2d3L/\n/WZ21sy2hl+f7856AAAAACDturuT96Skyusc8xt3nxN+/VM31wN0izVr1iRdAhCLzyZyFZ9N5DI+\nn8h33Rry3H29pPrrHGbdWQPQE/hlgFzFZxO5is8mchmfT+S7XDgnb4GZvWxmz5nZ7UkXAwAAAAD5\nrCDh198iqdjdL5nZA5J+Jqk84ZoAAAAAIG+Zu3fvC5gVS/qFu8/qwrG1ku529zMx93VvoQAAAACQ\n49z9uqe79UQnz9TJeXdmNt7dj4eX5ykInR0CntS1NwMAAAAAvV23hjwz+4GkxZJGm9lBSY9K6i/J\n3f0JSX9oZn8t6aqky5Ie6s56AAAAACDtun25JgAAAACg5+TC7ppdYmZ9woHp/5t0LUAbM9tvZr8P\nd4jdlHQ9QHtmNtzMnjGzXWa208zmJ10TYGbl4c/MreH3BjN7JOm6AEkys0+a2Stmtt3Mnjaz/knX\nBLQxs78xsx3h1xv+3MybTp6ZfVLS3ZJucfcHk64HkCQzq1GwWdD15kECPc7Mvitprbs/aWYFkga7\n+7mEywIiZtZH0iFJ8929Lul60LuZ2SRJ6yXd6u5XzOxHkp5z96cSLg2Qmd0h6YeS5kpqlvRrSX/l\n7pm44/Oik2dmhZLeKelbSdcCZDHlyb8j9C5mNkzSInd/UpLcvZmAhxy0XFKGgIcc0lfSkLY/jEk6\nknA9QJvbJG109yZ3b5G0VtJ7Ojs4X/7n9N8lfVpSfrQd0Zu4pCoz+52Z/UXSxQDtlEo6ZWZPhsvi\nnjCzQUkXBWR5SMFfpoHEufsRSV+WdFDSYUln3X1VslUBkVck3WdmI81ssIIG2JTODs75kGdm75J0\n3N236Q3GMQAJucfd367gH9rHzGxh0gUBoQJJcyR9093nSLok6e+TLQm4xsz6SXpQ0jNJ1wJIkpmN\nkPRuScWSJkkaamZ/nGxVQMDdX5P0JUmrJP1K0jYFyzZj5XzIk3SvpAfDc59+KGmJmbE2GjnB3Y+F\n309K+qmkeclWBEQOSapz983h9R8rCH1ArnhA0pbw5yeQC5ZLqnH3M+FyuGcl3ZNwTUDE3Z9097vd\nfbGkekl7Ozs250Oeu3/O3YvcvVTSeyWtdveHk64LMLPBZjY0vDxEUoWCVjqQOHc/LqnOzMrDm5ZJ\nejXBkoBs7xNLNZFbDkpaYGYDzcwU/NzclXBNQMTMxobfixScj9fpz9BuHYYOpNx4ST81M1fwb+lp\nd69OuCagvUckPR0ui6uR9MGE6wEkSeH5ocslfSTpWoA27r7JzH4s6WVJV8PvTyRbFfA6PzGzUQo+\nnx9194bODsybEQoAAAAAgOvL+eWaAAAAAICuI+QBAAAAQIoQ8gAAAAAgRQh5AAAAAJAihDwAAAAA\nSBFCHgAAAACkCCEPAJAIMxtlZi+b2VYzO2pmh9pd79IcVzP7tplNv84xHzWz992kmt8d1rjNzF4x\nsz+/zvFLzGxeJ/d9yMxOhO/3VTP7RBde/3XPdzPfGwAgPZiTBwBInJl9QdIFd/9KzH3mOfDLysz6\nS6qVNMfdj4dD5ovdfd8bPOZxSafc/Wsx931I0h3u/ikzGyNpt6Tb3f34m3k+AADa0MkDAOQCiy6Y\nlZnZTjP7vpm9ImmCmf2XmW0ysx1m9vl2x64zs1lm1tfM6s3si2GXbUMYnGRmj5vZI+2O/6KZvWRm\nu8xsQXj7YDP7cdide8bMfmdms7JqHB5+PytJ7n61LeCZ2Tgz+0lY40Yzm2dmpZI+LOnvwm7dgs7e\nvLufklQjaWL4fA+Gz7PFzH5tZmPini/rvc0JH7MtfA/D3vR/DQBAXiPkAQBy0QxJX3H3O939qKTP\nuvs8SbMlVZjZrTGPGS7pBXefLWmjpE6XUrr7fEmfkfRoeNMnJB119zslPR6+TvZjTkqqlnTAzJ42\ns/eaWVs4/bqkL4U1PiTp2+5eI+lbkv7V3ee4+8bO6jGzqQp+J78S3rTW3Re4+92Sfirpb7vwfN+T\n9Mnw/e+R9IXOXg8AkG5dOucBAIAelnH3re2uvz88/61AQbfrdkmvZT3mkrtXh5e3SFrYyXM/2+6Y\n4vDyQkn/Iknuvt3MdsY90N0/aGZ3Slou6bOSlkr6SHi9vF3oG25mA67/NvUBM1uuINT+pbs3h7cX\nm9m/SZogqb+C0NYpMxslaYC7/za86X8kPdWF1wcApBCdPABALrrYdsHMpkl6RNJid79LUpWkgTGP\nudLucos6/0NmU8wxlnVM9vWIu7/i7l+V9A5Jf9Du+Lnu/rbwq8jdmzp7jna+7+6zJC2S9OW2JaaS\nvinpy+F9H1P8+82ut9OaAQC9CyEPAJCL2geWWySdk3TBzCZKquzCY27UegXLLGVmMyXd1uHJzYaZ\n2aJ2N71N0oHw8koFSz7bjr0rvHg+rP8NufsmST9QEGYVPuZI2Bn803aHxj6fu5+WdKndeX9/Imnt\n9V4XAJBOhDwAQC6KdtMMl23uCr++qyCQdTgu6/J1nzfLNyRNCjd6+UdJr0pqyDrGJP1DuGHLVkmf\n07Xz/j4u6V4z+334HB8Ob/+5pD8KN1DpdOOV0JckfcjMBkl6TNLPJL0k6Vi7Y7Kfr/37eVjSV81s\nm4KQ+k/XeT0AQEoxQgEA0OuZWV9JBe7eFC4PrZI03d1bEy4NAIAbxsYrAABIQyU9324I+0cIeACA\nfEUnDwAAAABShHPyAAAAACBFCHkAAAAAkCKEPAAAAABIEUIeAAAAAKQIIQ8AAAAAUoSQBwAAAAAp\n8v/KNG81acw+pAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1e335b0e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#create figures of training and test times\n",
    "plt.figure(figsize=(15,6))\n",
    "f_training_time, = plt.plot(trainRatios,trainingTimes,label=\"Training Time\");\n",
    "f_test_time, = plt.plot(trainRatios,testTimes,label=\"Testing Time\");\n",
    "plt.legend(handles=[f_training_time,f_test_time])\n",
    "plt.ylabel('Time (Seconds)')\n",
    "plt.xlabel('Training Set Ratio')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### OBERVATIONS \n",
    "In the above experiment, it was observed that with the increase in the size of the training data, more time was taken to train the classifier. The test accuracy also increased with the incrase in training data size as observed in the previous experiment. Going from an initial split of 40-60 (train/test) to 90-10 (train/test) and resulting graphs above re-affirm the ideas of overfitting and efficiency with respect to an increase in training data size.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Experiment  with domain grouping \n",
    "\n",
    "Following extra exeperiment was performed to further explore the text classification process using the model described above. In this experiment, we decided to group together document from similar topics as shown in section 1. Five new classes (groups) were defined as Computing, Recreation, Politics, Science, Religion and Misc. We defined new labels for these classes and performed the experiment with the optimized values for the classifier as described above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#experiment with categorized new groups\n",
    "\n",
    "#create lists of categories\n",
    "listOfCategories = {'Computing':['comp.graphics', 'comp.os.ms-windows.misc', 'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware', 'comp.windows.x'],\n",
    "                   'Recreation':['rec.autos', 'rec.motorcycles', 'rec.sport.baseball', 'rec.sport.hockey'],\n",
    "                   'Politics':['talk.politics.misc', 'talk.politics.guns', 'talk.politics.mideast'],\n",
    "                   'Science':['sci.crypt', 'sci.electronics', 'sci.med', 'sci.space'],\n",
    "                   'Religion':['talk.religion.misc', 'alt.atheism', 'soc.religion.christian'],\n",
    "                   'Misc':['misc.forsale']}; \n",
    "\n",
    "rows_DF_exp2 = None;\n",
    "\n",
    "number = 0;\n",
    "for i in listOfCategories:\n",
    "    for topic in listOfCategories[i]: \n",
    "        #make a dataframe for given topic and save it in a tempory variable\n",
    "        rows_temp_DF = make_dataFrame(prefix+topic, number)\n",
    "    \n",
    "        #main dataframe is not null except in the first case. we merge the tempory df to main df\n",
    "        if(rows_DF_exp2 is not None):\n",
    "            rows_DF_exp2 = rows_DF_exp2.union(rows_temp_DF)\n",
    "        else:\n",
    "            #first case main dataframe is null and just assgin tempory df to main df\n",
    "            rows_DF_exp2 = rows_temp_DF;\n",
    "            \n",
    "    number = number + 1;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----+\n",
      "|label|count|\n",
      "+-----+-----+\n",
      "|    0|  614|\n",
      "|    1| 1008|\n",
      "|    2|  825|\n",
      "|    3|  828|\n",
      "|    4|  648|\n",
      "|    5|  203|\n",
      "+-----+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#create a view called newsgroups in the dataframe\n",
    "rows_DF_exp2.createOrReplaceTempView(\"newsgroups\")\n",
    "\n",
    "#create training and test datasets from our main dataset.\n",
    "[train_DF_exp2,test_DF_exp2] = rows_DF_exp2.randomSplit([8.0,2.0])\n",
    "\n",
    "#format and count of the test dataset\n",
    "test_DF_exp2.groupBy('label').count().orderBy('label').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time for training the model:  110.3638117313385\n",
      "training accuracy:  0.9095208070617906\n",
      "Time for Prediction with training data:  5.197361946105957\n",
      "testing accuracy:  0.8589432864760058\n",
      "Time for Prediction with Test data:  106.62399387359619\n"
     ]
    }
   ],
   "source": [
    "#assign train and test dataframes to new variables and add them to cache to speed up the experiment\n",
    "training_exp2 = train_DF_exp2 \n",
    "testing_exp2 = test_DF_exp2\n",
    "\n",
    "#cache to speed up the access\n",
    "training_exp2.cache();\n",
    "testing_exp2.cache();\n",
    "\n",
    "#mesaure the start time\n",
    "time1 = time.time();\n",
    "\n",
    "#train the model using pipeline\n",
    "model = pipeline.fit(training_exp2)\n",
    "#measure the time after training the model\n",
    "time2 = time.time(); \n",
    "print(\"Time for training the model: \", time2 - time1)\n",
    "\n",
    "# Make predictions on training documents to get final training accuracy \n",
    "prediction = model.transform(training_exp2)\n",
    "print(\"training accuracy: \", evaluator.evaluate(prediction))\n",
    "#mesaure the time after prediction with training data\n",
    "time3 = time.time(); \n",
    "print(\"Time for Prediction with training data: \", time3 - time2)\n",
    "\n",
    "# Make predictions on test documents. \n",
    "prediction = model.transform(testing_exp2)\n",
    "print(\"testing accuracy: \", evaluator.evaluate(prediction))\n",
    "#mesaure the time after prediction with test data\n",
    "time4 = time.time(); \n",
    "print(\"Time for Prediction with Test data: \", time4 - time3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### OBSERVATIONS\n",
    "\n",
    "As shown in the results above, it was observed that the overall prediction accuracy of the classifier was much higher than what was seen with individual topic classification. Going from 20 classes to 6 classes as described above, the accuracy moved from 77% to 85%. This increase in accuracy can be explained by the idea that a smaller number of classes give a higher probability for each document to belong to a certain class i.e. with 20 classes the probablity of a document of belonging to a random class would be 1/20 = 0.05. Whereas, with 6 classes,  the probablity of a document to belong to a random class increases to 1/6 = 0.17. It can hence be concluded that with a decrease in the number of classes, and an increase in the training data size, the overall performance for balanaced data gets improved considerably. This observation is well in line with the literature mentioned earlier. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### CONCLUSION\n",
    "\n",
    "In conclusion, The set of experiments above provided an insight into the application of naive Bayes algorithm for text classification purpose in a Big Data domain. The experiment involved creating a Pipeline object in Spark and identifying different stages for data transofrmation, evaluation and parameter tuning. It can be stated that tuned parameter defined after the optimization stages, was accurate and fast enough to be deployed as an online, large scale document classification system. As future experiment, there are plans to use a similar classifier to classify RSS feeds or other streaming documents according to user prefernces to offer a personalised information delivery system. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### REFERENCES\n",
    "\n",
    "1. http://qwone.com/~jason/20Newsgroups/ (20 Newsgroup Dataset Official site)\n",
    "\n",
    "2. Lang, K., 1995, July. Newsweeder: Learning to filter netnews. In Proceedings of the 12th international conference on machine    learning (pp. 331-339).\n",
    "\n",
    "3. Venezky, R.L., 1970. The structure of English orthography (Vol. 82). Walter de Gruyter.\n",
    "\n",
    "4. Aggarwal, C.C. and Zhai, C. eds., 2012. Mining text data. Springer Science & Business Media.\n",
    "\n",
    "5. Zhang, H., 2004. The optimality of naive Bayes. AA, 1(2), p.3.\n",
    "\n",
    "6. https://github.com/brez (John Bresnik’s Github Respository) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PySpark",
   "language": "python",
   "name": "pyspark"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}