Week 07 - Data Science.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 7 - Data Science"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drill"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ Write a function `hasvowel()` which accepts a word input and returns `True` if the word contains vowels. Think about the requirements of the argument and raise them if incorrect. `string.alpha()` might be helpful. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Solution\n",
    "import string   # or from string import isalpha\n",
    "\n",
    "'''\n",
    "If the argument is not a word (i.e. with spaces or non-letter character/ not a string), then raise ValueError. \n",
    "'''\n",
    "def hasvowel(word): \n",
    "    if type(word) != str: \n",
    "        raise ValueError\n",
    "    # Check spaces and non-letter character\n",
    "    if word.isalpha() == False: \n",
    "        raise ValueError\n",
    "    \n",
    "    if ('a' or 'e' or 'i' or 'o' or 'u') in word:\n",
    "        return True   # Otherwise you will obtain False already\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ In the following, print out the results (__in 2 decimal points__) using the `.format()` method. The computation is given to you. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from random import randint\n",
    "result = randint(0,122)/100\n",
    "\n",
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.04\n"
     ]
    }
   ],
   "source": [
    "# Solution\n",
    "from random import randint\n",
    "result = randint(0,122)/100\n",
    "\n",
    "print('{0:.2f}'.format(result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ Design a pandas dataframe which has the following specification. \n",
    "* `id`: Integers of 8 digits, starting by `10000000`.\n",
    "* `name`: String characters.\n",
    "\n",
    "After this, run your code and show the first 5 records of the dataframe you have created. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         id    name\n",
      "0  10000000  Shelly\n",
      "1  10000001    Kent\n",
      "2  10000002  Martin\n",
      "3  10000003    John\n",
      "4  10000004    Myra\n"
     ]
    }
   ],
   "source": [
    "# Solution\n",
    "import pandas as pd\n",
    "df = pd.DataFrame({'id':[10000000,10000001,10000002,10000003,10000004],'name':['Shelly','Kent','Martin','John','Myra']})\n",
    "print(df.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Science Process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this week, we will head into data science. Data science is a process to analyse data and extract meanings out of it. Unlike data analytics, data science adds one layer which is to __build a model__. This week we will look into \n",
    "* The first few steps on data science, which constitute most of the data analytics process. \n",
    "* Give a practical exercise on building skills on preparing data. \n",
    "\n",
    "Note that this week we are using a range of examples that you may find bit confusing. In reality not all skills has to be required in all data analytics project. Often some area requires this skills but not others. That is why we employ examples from different areas. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In data analytics, there is a standard procedure where data scientists follow to commit a data science project. This is called __CRISP-DM__. It follows from generating the ideas, finding the ingredients to produce a data science product and report them. The following is the flowchart of the process. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"fig/CRISPDM_Process_Diagram.png\" width=\"540\"/>\n",
    "<figcaption><i>Image from https://www.datasciencecentral.com/profiles/blogs/crisp-dm-a-standard-methodology-to-ensure-a-good-outcome</i></figcaption>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Business Understanding with Domain Knowledge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A successful project should __always__ start with understanding the business you are working on. For example\n",
    "* An insurance business should know what their clients working culture (e.g. involves high risk work)\n",
    "* A restaurant should know if their customers likes the seasoning in the food or not.\n",
    "\n",
    "In data analytics project, knowing the business domain knowledge will help us focus on what analysis is required. Sometimes it tells us why the data could be flawed/ skewed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting Your Goal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first phase, the organisation understands the project objectives and the business requirements. Often they will summarise them in documents. This is the __business understanding__ phase. This is not contrainted to business, any organisation or __you__ have a goal, and this goal is where it guides you to the road of data analytics. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise:__ Think about in your career, what is your objective? Think about how this object require data to help you. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In data analytics, we try to find out what could be analysed from our business, or topic. So we need to set goals as well. To guide us on forming our goals. One way is to set the __key performance indicators__ (KPI). They are measurable values that can describe your objectives. Remember they have to be, \n",
    "* Specific\n",
    "* Measurable\n",
    "* Achievable\n",
    "* Relevant\n",
    "* Timely(-manner)\n",
    "\n",
    "Think about a typical business, their aims is to maximise profit, so KPIs are anything that can maximise their profit. For example, \n",
    "* Number of customers visiting\n",
    "* Obtain higher ratings from customers\n",
    "* Number of referrals\n",
    "\n",
    "When it comes to politics, then their KPIs could be\n",
    "* Number of votes to paries\n",
    "* Number of passed motions proposed by them\n",
    "* Number of political donations\n",
    "\n",
    "to measure the success of the politcians. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise:__ What could be KPIs of analysing the success of a student?\n",
    "\n",
    "Remember, what is the objective/ purpose of a student?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise:__ What could be KPIs of predicting a pandemic?\n",
    "\n",
    "Again, think about the objective/ purpose first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Science Team"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data analysts or scientists is the powerhorse of a data analytics project. However, they are not the only people in the team. In making sure your data analysis is correct and relevant, you need the subject matter expert alongside with you. \n",
    "\n",
    "* In health sciences, data scientists collaborate with doctors and they fact check the accuracy of the results. For example, they look into one of the patient's X-ray image and see if it is classified correctly with the disease. \n",
    "* In finances, legal or government sectors. They may need staffs that have worked for many years that understands the legal matter/ customers well to advise you if your choice of variables are correct. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Understanding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start the data science project, we need to start with the relevant data sources. This is part of the __data understanding__ process. This involves the following steps: \n",
    "* Sourcing data\n",
    "* Combining data\n",
    "* Exploratory data analysis (EDA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before that, let us briefly understanding the basics of data. Data is our ingredients of our data analysis. There are many types of data, such as:\n",
    "* __Nominal__ e.g. gender, they are labels of objects/ people. \n",
    "* __Ordinal__ e.g. age\n",
    "* __Interval__ e.g. blood pressure, they do not have meaningful reference points\n",
    "* __Ratio__ e.g. weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "__Exercise:__ From above, what is the data type of number of customers visiting?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "__Solution:__ Ordinal. Where more customer has a meaning about the success of business. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on what data we have, we can get indication on what the analysis might be. For example, \n",
    "\n",
    "* How much or how many? (Regression)\n",
    "* Which category? (Classification)\n",
    "* Which data are similar? (Clustering)\n",
    "\n",
    "The methods in brackets are machine learning methods and they will be introduced in Weeks 9 and 10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sourcing Data from Twitter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Twitter is a popular social media site where users can express themselves within some text length. It attracts a lot of attention from researchers to understand how opinion forms on Twitter. To convenient people understand what is inside the social network, Twitter offers an API that allows us to obtain the information in their server. \n",
    "\n",
    "API is the gateway to their servers. This means we need to obtain a key to get access. This is called the __public-key cryptography__. Which means we use the public key to get access into Twitter's server, and Twitter has a private key to encrypt their message back to us. This is only important when we are using the pair of keys as in below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So before to start anything, we should have a Twitter developer account beforehand. This can be obtained from [https://developer.twitter.com/](https://developer.twitter.com/). \n",
    "\n",
    "After that, you will need to obtain your __consumer keys__ and __access tokens__ (i.e. the public keys) on the developer portal. There are many guides available online, for example\n",
    "* [https://towardsdatascience.com/how-to-access-twitters-api-using-tweepy-5a13a206683b](https://towardsdatascience.com/how-to-access-twitters-api-using-tweepy-5a13a206683b)\n",
    "* [https://realpython.com/twitter-bot-python-tweepy/#creating-twitter-api-authentication-credentials](https://realpython.com/twitter-bot-python-tweepy/#creating-twitter-api-authentication-credentials)\n",
    "\n",
    "Once you have created an application, you will need Twitter officially approve you. This can take from hours to few days. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fetch tweets from Twitter, we get help from the `tweepy` package. The following code is would fetch the relevant tweets to the keyword `'pokemon'` (last line of the snippet). \n",
    "```python\n",
    "import tweepy\n",
    "\n",
    "APP_KEY = '***'\n",
    "APP_SECRET = '***'\n",
    "\n",
    "auth = tweepy.OAuthHandler('***', '***')\n",
    "auth.set_access_token(APP_KEY, APP_SECRET)\n",
    "\n",
    "api = tweepy.API(auth)\n",
    "\n",
    "tweets = api.search(q = \"pokemon\", count = 100, result_type = \"recent\")\n",
    "```\n",
    "As you can see, we use 2 pairs of keys. The `APP_KEY` represents our access to the server, and then we access the session token which allows us to stay for the time being.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tweepy` provides a lot of methods that could enable programmers to obtain relevant tweets. For example the `search()` method returns relevant tweets from a keyword. `search_users()` returns relevant users from a keyword. \n",
    "\n",
    "More of these methods, and how to use them, can be find in [https://tweepy.readthedocs.io/en/latest/api.html](https://tweepy.readthedocs.io/en/latest/api.html). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise:__ What does each line of the code mean? If possible, copy the snippet and comment out each lines. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In today, we are going to fetch some of the tweets and write them in json format. Json is a way to store complex data in text files and the program can recognise them afterwards. In python there is a native package called `json`, so we will call them at the start. \n",
    "```python\n",
    "import json\n",
    "```\n",
    "To convert the information to json, we write \n",
    "```python\n",
    "tweet_json = json.dump(tweets._json)\n",
    "```\n",
    "Thus we will need to store the data, this means we can write \n",
    "```python\n",
    "with open('tweets.json', 'w') as json_file:\n",
    "    json.dump(tweet_json, json_file)\n",
    "```\n",
    "and the tweet data is now stored in `tweets.json` in your current folder location. \n",
    "\n",
    "To read back the json file, we can use the following snippet: \n",
    "```python\n",
    "with open('data.txt') as json_file:\n",
    "    data = json.load(json_file)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ Write a code that will return tweets with the keyword `'Apple'`. Store them into the file called `apple_tweets.txt'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "solution": "hidden",
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "# Solution\n",
    "import tweepy\n",
    "\n",
    "APP_KEY = '***'\n",
    "APP_SECRET = '***'\n",
    "\n",
    "auth = tweepy.OAuthHandler('***', '***')\n",
    "auth.set_access_token(APP_KEY, APP_SECRET)\n",
    "\n",
    "api = tweepy.API(auth)\n",
    "\n",
    "tweets = api.search(q = \"Apple\", count = 100, result_type = \"recent\")\n",
    "\n",
    "with open('apple_tweets.txt', 'w') as json_file:\n",
    "    json.dump(tweets, json_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an extension, you can use a Twitter scraper to do more complex scraping. Given that one method call can fetch up to 1500 tweets, you can write a code to find tweets within a date range, and return more than 3200 at once. There are many of these online and often the authors have submitted theirs on thei Github repository. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Web Scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Internet contains a wealth of resources, so we can use python to extract information from the webpage. To scrape from the internet, we follow the following steps: \n",
    "1. Find the page that you want to scrape\n",
    "2. Inspect the page\n",
    "3. Write the code\n",
    "4. Run the code and extract the data\n",
    "5. Store the data in the required format "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, we will try to scrape from news website. Our example would be from [https://www.nbcnews.com/better/health/what-headline-stress-disorder-do-you-have-it-ncna830141](https://www.nbcnews.com/better/health/what-headline-stress-disorder-do-you-have-it-ncna830141). Which we wish to obtain the article itself. \n",
    "\n",
    "This means we should have a look at the page itself, by means of clicking in the link and inspect them. You should see the similar screenshot as below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"fig/Capture01.PNG\" width=\"620\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will see the webapage and it has not only just the news article itself. In well structured websites, there is a top bar to navigate to other pages of the news site. Often there will be some advertisements at the side of the article. If you scroll down, you will see the footer which shows the side information of the website as well. The article does not span the whole space in the middle of the page. There are spaces to put the author's information and the date of publish. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise:__ Identify the page elements in the link. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our first step is to fetch the webpage onto our computer. In this way, we could potentially extract the information from the page. In this exercise, we will look at a package called `requests`. The following is the snippet of code to extract the website and export the information. \n",
    "\n",
    "```python\n",
    "import requests\n",
    "\n",
    "htmlsource = requests.get('https://www.nbcnews.com/better/health/what-headline-stress-disorder-do-you-have-it-ncna830141').text\n",
    "tree = html.fromstring(htmlsource)\n",
    "\n",
    "with open('file/stress_news.html', mode = 'w') as fo:\n",
    "    fo.write(htmlsource)\n",
    "\n",
    "fo.close()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To extract the contents, there is a method to do so. Webpages are composed under a markup language called HTML. Each semantic contents are wrapped by a tag. For example, to make text bold, we can write `<b>I am bold.</b>` wich shows __I am bold.__ The bold find starts at the `<b>` tag and ends at `</b>` tag. A typical webpage should show \n",
    "```html\n",
    "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n",
    "   \"http://www.w3.org/TR/html4/strict.dtd\">\n",
    "<html>\n",
    "   <head>\n",
    "      <title>My first HTML document</title>\n",
    "   </head>\n",
    "   <body>\n",
    "      <p>Hello world!</p>\n",
    "   </body>\n",
    "</html>\n",
    "```\n",
    "Let us look at the tags in more detail: \n",
    "* The first line tells the computer that this is a webpage. \n",
    "* The `<head>` tag shows the meta information of the webpage. In this case, we define the title of the page under the `<head>` tag. \n",
    "* Anything showed in the `<body>` tag will appear on the screen (of your browser). In this case, there is a sentence 'Hello world!' is written and it is wrapped by the paragraph `<p>` tag. \n",
    "* Finally, every tags should wrap up by a end tag that starts with a `/`. For example, a paragrapha ends with a `</p>` tag. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We cannot assume that our contents are spans within the whole `<body>` tag, so we will inspect it. To do so, at the page opened press __F12__. You should see a portal appeared left to your page with the HTML markups showed. \n",
    "\n",
    "Then scroll to the HTML markups and scroll to the place where the article is highlighting. This is not the place contains only the article, but there is the smallest tag contained the article. So click on the triangle beside the tag, and scroll to where the article is highlighted again. Repeat this process until you are able to see only the article contents are highlighted. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ What is the HTML tag that contains the whole article? You may want to include all the information inside the tag. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden"
   },
   "source": [
    "__Solution:__\n",
    "\n",
    "It is a `<div>` tag and the whole tag reads \n",
    "```html\n",
    "<div class=\"article-body__content\">\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you read the tag that contains all the information of the article itself, you should see it contains several `<p>` tags, these mean the paragraphs and you don't need to care about them. \n",
    "\n",
    "To extract the article text, the HTML tag we have is important. If you do not have the answer, just use the solution from above exercise. We will extract the article using this information. Given we have the destination tag, how could we tell the computer where it is? We use a concept called __XPath__ (XML Path Language). It shows the address from the top most level to the tag we after. \n",
    "\n",
    "To get the XPath of the article, right click on the tag we after and click copy the 'XPath' or the 'Full XPath'. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"fig/Capture02.png\" width=\"480\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, the full XPath that stores the article itself is \n",
    "`/html/body/div[2]/div/div[5]/div/div/div/article/div/div[2]/div[1]/div[2]`, while the XPath is `//*[@id=\"content\"]/div/div[5]/div/div/div/article/div/div[2]/div[1]/div[2]`. We can see that `html` appears at the start of the XPath, then the `body`. Which appears to be the nested structure of the html tags. \n",
    "\n",
    "Furthermore, \n",
    "* `/` seperates the different levels. For example, `/html/body/` means the top layer is the `<html>` tag and the `<body>` is nested.\n",
    "* `[]` indicates which branch of the tree is chosen. For instance, if there are 2 `<div>` tags after `<body>`, `div[2]` would select the second (i.e. last) one.\n",
    "* `//` means 'arbitrary depth'. For example, `/html/body//p` means any paragraphs `<p>` under `<body`. Trilling (2019) adivses to 'start your XPATH with `//` to avoid make it shorter and avoid being too specific'.\n",
    "* `*` means 'everything'. For instance, `p[*]` means all paragraphs. \n",
    "* `//*` means everything in the next layer. \n",
    "* `[@attribute=\"whatever\"]` lets you select only those tags that contain a specific attribute. Common attributes could be `id`, `class`. Using above exercise as an example, if the `<div>` tag reads `<div class=\"reviews-single text\">`. The XPath could write \n",
    "`//div[@class=\"article-body__content\"]`. \n",
    "* `.` (a dot) means relative location path from the tag we after. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means we can fetch (or _parse_) the article by specifying the XPath. Which is done by the following code snippet. \n",
    "```python\n",
    "from lxml import html\n",
    "\n",
    "article_content = tree.xpath('//*[@id=\"content\"]/div/div[5]/div/div/div/article/div/div[2]/div[1]/div[2]')\n",
    "\n",
    "article_content_text = [r.text.strip() for r in article_content]\n",
    "print(article_content_text)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise:__ Use the snippets above to print out the texts of the article. If possible, write the texts into a `.txt` file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise:__ Use the snippets above to extract the positive words in the Havard IV-4 dictionary. After that write the list of words into a `.txt` file. \n",
    "\n",
    "This is the link: `http://www.wjh.harvard.edu/~inquirer/Pstv.html`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Some of contents come from https://github.com/info-370/eda._\n",
    "\n",
    "Exploratory data analysis (__EDA__) is the process to understand the datasets sourced. This process starts by summarising the data structures and then visualise a selected features to quickly understand univariate distributions and relationships between variables. Initial EDA questions ask basic questions, including:\n",
    "\n",
    "* How large is the dataset (rows, columns)?\n",
    "* What are the variables present in the dataset?\n",
    "* What is the data type of each variable?\n",
    "\n",
    "At the end, the goal of EDA is to find \n",
    "* statistical summary of individual columns. \n",
    "* the relationship between columns. \n",
    "* check if any missing data. \n",
    "\n",
    "In this week, we will start some work on EDA with a marks dataset and a text dataset. This will use either statistical analysis and/ or visualisations. You can source your own dataset after this class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using Jupyter to analyse data, we should maintain our habit on grouping each functions into each cell. For example, a conventional layout should include (in order): \n",
    "1. Packages/ libraries needed to be imported\n",
    "2. Custom functions you wish to use\n",
    "3. Import data\n",
    "4. EDA (and data cleaning, may separate)\n",
    "5. Analysis\n",
    "6. Report on your findings\n",
    "\n",
    "In this way it is organised, and also it is intuitive to later users to compile the early cells first so that there will be no not found errors. In the following, we have a dedicated cell for storing the packages. When we need to import a library, we will add into the next cell and rerun it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Import all needed packages in here\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, we will use a sample marks data set called `marks.csv`. It is stored in the `/files` folder. It is generated dataset where it simulates the breakdown of marks in a course. There are several columns: \n",
    "* Unikey is a unique identifier of each students. \n",
    "* Final exam marks (by question, worth 50%): \n",
    "    * `'Q1'` is out of 20.\n",
    "    * `'Q2'` is out of 20.\n",
    "    * `'Q3'` is out of 15.\n",
    "    * `'Q4'` is out of 20.\n",
    "    * `'Q5'` is out of 15.\n",
    "    * `'Q6'` is out of 10.\n",
    "* `'Quizes'` worth 10% of the course. \n",
    "* `'Lab'` worth 10% of the course. \n",
    "* `'Asgn 01'` worth 10% of the course. \n",
    "* `'Asgn 02'` worth 20% of the course. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "__Exericse:__ Read the `marks.csv` file and import the data as a `pandas` dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "# Your code below\n",
    "df = pd.read_csv('???')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes the input `.csv` file is encoded by different systems. For example, if the `.csv` file is written in Chinese or other languages, it may use unknown encoding systems to us. Sometimes, we need to add an option to the `.read_csv()` to let it read the Unicode. For example, \n",
    "\n",
    "```python\n",
    "df = pd.read_csv('???', encoding='utf8')\n",
    "```\n",
    "\n",
    "If you see the encoding error and you cannot figure out which encoding system it uses. Run the following code and it will help you. Then use the code snippet above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# This code checks what encoding the csv file has. \n",
    "with open('top50.csv') as f:\n",
    "    print(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, we don't need to worry about the encoding. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let us start exploring the data set. This means we need to print out the features inside the dataset. In the following, we are using the methods from `pandas` since last week. Which would be familiar to you. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ How can we see the first __10 records__ of the dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true,
    "solution": "hidden",
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     unikey    Q1    Q2    Q3    Q4    Q5    Q6  Quizes     Lab Asgn 01  \\\n",
      "0  pmpk3028   6.0   8.5   2.5   3.0   7.0   5.0   3.47%   3.27%  84.27%   \n",
      "1  iyqw3940  16.5  18.5   1.0  14.0  14.5   8.0  14.08%  91.72%  56.11%   \n",
      "2  mjll6065  11.5  11.5   0.5  14.5  11.0   0.0  84.66%   75.5%   48.3%   \n",
      "3  rtgm8060   6.0  14.5   5.0   8.0   7.0   0.0  59.99%   52.2%  19.31%   \n",
      "4  xfxd8275  12.0   4.0   1.5   3.5  14.5   6.0  23.46%  78.64%   2.32%   \n",
      "5  usan5260   1.5  18.0   4.5   7.5   8.0   2.5  11.53%  33.07%  53.54%   \n",
      "6  exgv5076   5.5   7.0   2.0  13.5  10.0   9.5  21.71%   1.31%   56.8%   \n",
      "7  dprj4333   1.5  20.5  10.0  13.0   1.5  10.0  76.35%  41.73%  27.52%   \n",
      "8  vera2617   4.5  15.0  11.0   1.0   5.5  10.5   69.2%  13.23%  30.51%   \n",
      "9  fuea3500   4.5  13.5   5.0   1.5   4.5   6.5  74.32%  96.75%  87.45%   \n",
      "\n",
      "  Asgn 02  \n",
      "0  50.17%  \n",
      "1   8.78%  \n",
      "2  98.73%  \n",
      "3  90.93%  \n",
      "4  91.49%  \n",
      "5  49.17%  \n",
      "6  89.46%  \n",
      "7  84.06%  \n",
      "8  30.62%  \n",
      "9  13.52%  \n"
     ]
    }
   ],
   "source": [
    "# Solution\n",
    "print(df.head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ How can we know what columns are in the data set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "solution": "hidden",
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "# Solution\n",
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ How can we find out how many rows and columns we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Your code below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(250, 11)\n"
     ]
    }
   ],
   "source": [
    "# Solution\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exericse:__ Given that we know how to find how many rows and columns we have in teh dataset? I would like to see the results printed as `'There are {} records and {} columns in this dataset.'` where the brackets are substituted by the number of rows and columns in the dataset. \n",
    "\n",
    "__Hint:__ Use the code above and think about how come the returned value able to fit in 2 values. What is the data type we obtained? So how can we extract the individual values? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 250 records and 11 columns in this dataset.\n"
     ]
    }
   ],
   "source": [
    "# Solution\n",
    "print('There are {} records and {} columns in this dataset.'.format(df.shape[0],df.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ How can we find out the data types under each column? Are they all the same under each columns?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Solution\n",
    "print(df.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of your EDA, we will start to get some statistical information from the dataset. To do so, we rely on the native `.describe()` method from `pandas`. Run the code below to see how the amrks distribution would be. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this code\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let us start with the results row by row. \n",
    "* The first column tells us how many (non null) records do we have in this dataset. Given that all counts are the same, we can say there are no missing (null) values. However, we can check that from the `.shape()` method previously. \n",
    "* The next 2 rows represents the mean and standard deviation of each questions. It would be meaningful to interpret these results given we have the total marks of each question. \n",
    "* The next rows represent their quantiles. That is, for example, how does the lowest 25% perform? We can have a percetion on where the cohort perform. Are they doing good in this question, or not?\n",
    "\n",
    "However, if we look at the last row. There is something wrong in here. If you look carefully, you should see that the maximum marks are 0.5 more than what supposed to be. We need to find out how many people have this mark and then rectify the situation. This will be done in the next section. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA: Visualising Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In EDA you will need to show your results. It can be either summarising the statistics, or visualisaing data. In this wee, we will look into few common visualisations: \n",
    "* Histogram\n",
    "* Scatter plots\n",
    "* Pairwise plots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In visualising data we use `pyplot` from `matplotlib` to show the visualisations. In this course we may use other packages such as `seaborn`. Often the `matplotlib` is used as a wrapper of the extra functionalities from these libraries and we will use the following format to visualise data \n",
    "```python \n",
    "import matplotlib.pyplot as plt \n",
    "import seanborn as sns \n",
    "\n",
    "plt.figure()  # Creating the figure objects, the visualisations are inside \n",
    "'''\n",
    "Any visualisations e.g. plt.plot() or plt.hist() in here \n",
    "'''\n",
    "plt.show() # Otherwise your visualisation will not show\n",
    "```\n",
    "\n",
    "To recap, the above code has $3$ steps: \n",
    "1. Use `plt.figure()`\n",
    "2. Insert any visualisation methods after this \n",
    "3. Use `plt.show()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first one to here is histogram, which is a type of visualisation where data are represented as bars. Often histogram are used to represent the counts of unique data values or its frequency (or probability). The following shows what this means. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAE3ZJREFUeJzt3XGQXWd93vHvE9mG2JAg0EJdSYvM1MPgELDdHUHqDpgCQgZqkTadSpMQh4FRhsEpJJ127HTGbs1kBppO0knjYFSsGlqwk9i4VRuBrcYQJyEmkoyxLRsHIRy8kVspyDE4pjgyv/5xj9rr1a727O5dXdnv9zNzZ8953/ec+7ua1XPPvvece1JVSJLa8UPjLkCSdHIZ/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGnDbuAmazatWqWrdu3bjLkKRnjb179/5lVU30GXtKBv+6devYs2fPuMuQpGeNJH/ed6xTPZLUGINfkhpj8EtSYwx+SWqMwS9JjZk3+JOsTfKFJA8m2Zfkg7OMSZLfSLI/yb1JLhzquyzJ17vHZaN+AZKkhelzOudR4J9X1d1JXgjsTbKrqh4YGnMJcG73eB3wMeB1SV4MXA1MAdVtu6OqHhvpq5Ak9TbvEX9VPVpVd3fL3wUeBFbPGLYJ+FQN3AW8KMnZwNuAXVV1pAv7XcDGkb4CSdKCLGiOP8k64ALgyzO6VgOPDK1Pd21ztUuSxqT3lbtJXgDcAnyoqr4zs3uWTeoE7bPtfyuwFWBycrJvWdJJte6K3xvL8z78kXeM5Xn13NTriD/J6QxC/9NV9dlZhkwDa4fW1wAHT9B+nKraVlVTVTU1MdHr6yYkSYvQ56yeANcDD1bVr80xbAfws93ZPa8HHq+qR4HbgA1JViZZCWzo2iRJY9Jnquci4N3AfUnu6dp+GZgEqKrrgJ3A24H9wJPAe7q+I0k+DOzutrumqo6MrnxJ0kLNG/xV9UfMPlc/PKaAD8zRtx3YvqjqJEkj55W7ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1Jh578CVZDvwTuBQVb16lv5/Afz00P5eBUx0t118GPgu8DRwtKqmRlW4JGlx+hzx3wBsnKuzqn61qs6vqvOBK4E/mHFf3Td1/Ya+JJ0C5g3+qroT6HuD9C3AjUuqSJK0rEY2x5/kTAZ/Gdwy1FzA7Un2Jtk6queSJC3evHP8C/APgT+eMc1zUVUdTPJSYFeSr3V/QRyne2PYCjA5OTnCsiRJw0Z5Vs9mZkzzVNXB7uch4FZg/VwbV9W2qpqqqqmJiYkRliVJGjaS4E/yo8Abgf821HZWkhceWwY2APeP4vkkSYvX53TOG4GLgVVJpoGrgdMBquq6bthPArdX1V8Pbfoy4NYkx57nM1X1+dGVLklajHmDv6q29BhzA4PTPofbDgCvXWxhkqTl4ZW7ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1Jh5gz/J9iSHksx6v9wkFyd5PMk93eOqob6NSR5Ksj/JFaMsXJK0OH2O+G8ANs4z5g+r6vzucQ1AkhXAtcAlwHnAliTnLaVYSdLSzRv8VXUncGQR+14P7K+qA1X1FHATsGkR+5EkjdCo5vh/IslXk3wuyY91bauBR4bGTHdts0qyNcmeJHsOHz48orIkSTONIvjvBl5eVa8F/gPwX7v2zDK25tpJVW2rqqmqmpqYmBhBWZKk2Sw5+KvqO1X1RLe8Ezg9ySoGR/hrh4auAQ4u9fkkSUuz5OBP8reSpFte3+3z28Bu4Nwk5yQ5A9gM7Fjq80mSlua0+QYkuRG4GFiVZBq4GjgdoKquA34KeH+So8D3gM1VVcDRJJcDtwErgO1VtW9ZXoUkqbd5g7+qtszT/5vAb87RtxPYubjSJEnLwSt3JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTHzBn+S7UkOJbl/jv6fTnJv9/hSktcO9T2c5L4k9yTZM8rCJUmL0+eI/wZg4wn6vwm8sapeA3wY2Daj/01VdX5VTS2uREnSKPW55+6dSdadoP9LQ6t3AWuWXpYkabmMeo7/vcDnhtYLuD3J3iRbT7Rhkq1J9iTZc/jw4RGXJUk6Zt4j/r6SvIlB8P/9oeaLqupgkpcCu5J8rarunG37qtpGN000NTVVo6pLkvRMIzniT/Ia4BPApqr69rH2qjrY/TwE3AqsH8XzSZIWb8nBn2QS+Czw7qr6s6H2s5K88NgysAGY9cwgSdLJM+9UT5IbgYuBVUmmgauB0wGq6jrgKuAlwG8lATjancHzMuDWru004DNV9flleA2SpAXoc1bPlnn63we8b5b2A8Brj99CkjROXrkrSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjekV/Em2JzmUZNZ75mbgN5LsT3JvkguH+i5L8vXucdmoCpckLU7fI/4bgI0n6L8EOLd7bAU+BpDkxQzu0fs6YD1wdZKViy1WkrR0vYK/qu4EjpxgyCbgUzVwF/CiJGcDbwN2VdWRqnoM2MWJ30AkScts3put97QaeGRofbprm6v9OEm2MvhrgcnJyUUXsu6K31v0tnr2ePgj7xh3CSeVv9dtOFm/16P6cDeztNUJ2o9vrNpWVVNVNTUxMTGisiRJM40q+KeBtUPra4CDJ2iXJI3JqIJ/B/Cz3dk9rwcer6pHgduADUlWdh/qbujaJElj0muOP8mNwMXAqiTTDM7UOR2gqq4DdgJvB/YDTwLv6fqOJPkwsLvb1TVVdaIPiSVJy6xX8FfVlnn6C/jAHH3bge0LL02StBy8cleSGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1JhewZ9kY5KHkuxPcsUs/b+e5J7u8WdJ/mqo7+mhvh2jLF6StHDz3oEryQrgWuCtDG6evjvJjqp64NiYqvrFofG/AFwwtIvvVdX5oytZkrQUfY741wP7q+pAVT0F3ARsOsH4LcCNoyhOkjR6fYJ/NfDI0Pp013acJC8HzgHuGGp+fpI9Se5K8q5FVypJGok+N1vPLG01x9jNwM1V9fRQ22RVHUzyCuCOJPdV1TeOe5JkK7AVYHJyskdZkqTF6HPEPw2sHVpfAxycY+xmZkzzVNXB7ucB4Is8c/5/eNy2qpqqqqmJiYkeZUmSFqNP8O8Gzk1yTpIzGIT7cWfnJHklsBL4k6G2lUme1y2vAi4CHpi5rSTp5Jl3qqeqjia5HLgNWAFsr6p9Sa4B9lTVsTeBLcBNVTU8DfQq4ONJfsDgTeYjw2cDSZJOvj5z/FTVTmDnjLarZqz/61m2+xLw40uoT5I0Yl65K0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY3pFfxJNiZ5KMn+JFfM0v9zSQ4nuad7vG+o77IkX+8el42yeEnSws1768UkK4BrgbcC08DuJDtmuXfub1fV5TO2fTFwNTAFFLC32/axkVQvSVqwPkf864H9VXWgqp4CbgI29dz/24BdVXWkC/tdwMbFlSpJGoU+wb8aeGRofbprm+kfJ7k3yc1J1i5wW0nSSdIn+DNLW81Y/+/Auqp6DfA/gU8uYNvBwGRrkj1J9hw+fLhHWZKkxegT/NPA2qH1NcDB4QFV9e2q+n63+h+Bv9t326F9bKuqqaqampiY6FO7JGkR+gT/buDcJOckOQPYDOwYHpDk7KHVS4EHu+XbgA1JViZZCWzo2iRJYzLvWT1VdTTJ5QwCewWwvar2JbkG2FNVO4B/luRS4ChwBPi5btsjST7M4M0D4JqqOrIMr0OS1NO8wQ9QVTuBnTParhpavhK4co5ttwPbl1CjJGmEvHJXkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGtMr+JNsTPJQkv1Jrpil/5eSPJDk3iS/n+TlQ31PJ7mne+yYua0k6eSa99aLSVYA1wJvBaaB3Ul2VNUDQ8O+AkxV1ZNJ3g/8W+Cfdn3fq6rzR1y3JGmR+hzxrwf2V9WBqnoKuAnYNDygqr5QVU92q3cBa0ZbpiRpVPoE/2rgkaH16a5tLu8FPje0/vwke5LcleRdi6hRkjRC8071AJmlrWYdmPwMMAW8cah5sqoOJnkFcEeS+6rqG7NsuxXYCjA5OdmjLEnSYvQ54p8G1g6trwEOzhyU5C3AvwIurarvH2uvqoPdzwPAF4ELZnuSqtpWVVNVNTUxMdH7BUiSFqZP8O8Gzk1yTpIzgM3AM87OSXIB8HEGoX9oqH1lkud1y6uAi4DhD4UlSSfZvFM9VXU0yeXAbcAKYHtV7UtyDbCnqnYAvwq8APjdJADfqqpLgVcBH0/yAwZvMh+ZcTaQJOkk6zPHT1XtBHbOaLtqaPktc2z3JeDHl1KgJGm0vHJXkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGtMr+JNsTPJQkv1Jrpil/3lJfrvr/3KSdUN9V3btDyV52+hKlyQtxrzBn2QFcC1wCXAesCXJeTOGvRd4rKr+DvDrwEe7bc9jcHP2HwM2Ar/V7U+SNCZ9jvjXA/ur6kBVPQXcBGyaMWYT8Mlu+WbgzRncdX0TcFNVfb+qvgns7/YnSRqTPsG/GnhkaH26a5t1TFUdBR4HXtJzW0nSSXRajzGZpa16jumz7WAHyVZga7f6RJKHetQ2m1XAXy5y2+VkXQtzwrry0ZNYyTM9K/+9xsi6FiAfXVJdL+87sE/wTwNrh9bXAAfnGDOd5DTgR4EjPbcFoKq2Adv6lT23JHuqamqp+xk161oY61oY61qY1uvqM9WzGzg3yTlJzmDwYe2OGWN2AJd1yz8F3FFV1bVv7s76OQc4F/jT0ZQuSVqMeY/4q+poksuB24AVwPaq2pfkGmBPVe0Argf+c5L9DI70N3fb7kvyO8ADwFHgA1X19DK9FklSD32meqiqncDOGW1XDS3/H+CfzLHtrwC/soQaF2rJ00XLxLoWxroWxroWpum6MpiRkSS1wq9skKTGPGeCP8n2JIeS3D/uWo5JsjbJF5I8mGRfkg+OuyaAJM9P8qdJvtrV9W/GXdOwJCuSfCXJ/xh3LcOSPJzkviT3JNkz7nqOSfKiJDcn+Vr3u/YTp0BNr+z+nY49vpPkQ+OuCyDJL3a/9/cnuTHJ88ddE0CSD3Y17Vvuf6vnzFRPkjcATwCfqqpXj7segCRnA2dX1d1JXgjsBd5VVQ+Mua4AZ1XVE0lOB/4I+GBV3TXOuo5J8kvAFPAjVfXOcddzTJKHgamqOqXO/07ySeAPq+oT3Zl3Z1bVX427rmO6r2n5C+B1VfXnY65lNYPf9/Oq6nvdySc7q+qGMdf1agbfirAeeAr4PPD+qvr6cjzfc+aIv6ruZHBG0Smjqh6tqru75e8CD3IKXLlcA090q6d3j1PiCCDJGuAdwCfGXcuzQZIfAd7A4Mw6quqpUyn0O28GvjHu0B9yGvDD3TVHZzLHtUUn2auAu6rqye7bD/4A+MnlerLnTPCf6rpvLL0A+PJ4KxnoplPuAQ4Bu6rqlKgL+PfAvwR+MO5CZlHA7Un2dleanwpeARwG/lM3PfaJJGeNu6gZNgM3jrsIgKr6C+DfAd8CHgUer6rbx1sVAPcDb0jykiRnAm/nmRe/jpTBfxIkeQFwC/ChqvrOuOsBqKqnq+p8BldTr+/+1ByrJO8EDlXV3nHXMoeLqupCBt9U+4FuenHcTgMuBD5WVRcAfw0c99Xp49JNPV0K/O64awFIspLBl0eeA/xt4KwkPzPeqqCqHmTwrca7GEzzfJXBtU/LwuBfZt0c+i3Ap6vqs+OuZ6ZuWuCLDL42e9wuAi7t5tJvAv5Bkv8y3pL+v6o62P08BNzKqfFNs9PA9NBfbDczeCM4VVwC3F1V/3vchXTeAnyzqg5X1d8AnwX+3phrAqCqrq+qC6vqDQymrZdlfh8M/mXVfYh6PfBgVf3auOs5JslEkhd1yz/M4D/D18ZbFVTVlVW1pqrWMZgeuKOqxn40BpDkrO4DerqplA0M/jwfq6r6X8AjSV7ZNb2ZwZXyp4otnCLTPJ1vAa9Pcmb3//PNDD57G7skL+1+TgL/iGX8d+t15e6zQZIbgYuBVUmmgaur6vrxVsVFwLuB+7r5dIBf7q6EHqezgU92Z1v8EPA7VXVKnTp5CnoZcOsgKzgN+ExVfX68Jf0/vwB8uptWOQC8Z8z1ANDNVb8V+Plx13JMVX05yc3A3QymUr7CqXMV7y1JXgL8DYOvt3lsuZ7oOXM6pySpH6d6JKkxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY35vzkH2mjgRUu4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt \n",
    "\n",
    "hist_data = [1,2,3,4,5,5,6,7,8,9]\n",
    "\n",
    "plt.figure()\n",
    "plt.hist(hist_data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, $5$ repeated twice and therefore it shows the bar height taller than others. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the `hist()` method from `pyplot` has an optional argument which allows to show the density instead. This is done by using the `density` argument. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEFBJREFUeJzt3X+s3XV9x/Hna62A4FSEu8VRakvsjJ0uYK51Gxku44dlmtYtEEviggtJNyOLjixLnQlkNSagy7Z/2AaRbsypFUGTZlaRCLotBuzlh2LBxktFuNaNujIdA8HCe3+cL+Zwvd393vbcnguf5yM5ud/v5/v5fM/73Jy+zvd+zvf7baoKSVIbfm7cBUiSjh5DX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktSQ5eMuYLaTTz65Vq1aNe4yJOl55c477/xBVU3M12/Jhf6qVauYmpoadxmS9LyS5Lt9+jm9I0kNMfQlqSGGviQ1xNCXpIYY+pLUkF6hn2R9kj1JppNsmWP7ZUnuS/KNJF9K8qqhbU8nuad77Bhl8ZKkhZn3lM0ky4CrgXOBGWBXkh1Vdd9Qt7uByap6PMm7gQ8D7+i2PVFVp4+4bknSYehzpL8OmK6qvVX1FLAd2Djcoapuq6rHu9XbgRWjLVOSNAp9Qv8U4OGh9Zmu7VAuAT4/tH5ckqkktyd5+2HUKEkakT5X5GaOtjn/N/Uk7wQmgTcPNa+sqn1JTgNuTXJvVT0wa9xmYDPAypUrexUujcOqLZ8by/M+eOVbx/K8euHpc6Q/A5w6tL4C2De7U5JzgA8AG6rqyWfbq2pf93Mv8GXgjNljq+raqpqsqsmJiXlvHSFJOkx9Qn8XsCbJ6iTHAJuA55yFk+QM4BoGgf/IUPuJSY7tlk8GzgSGvwCWJB1F807vVNXBJJcCNwPLgG1VtTvJVmCqqnYAHwFeAnw6CcBDVbUBeC1wTZJnGHzAXDnrrB9J0lHU6y6bVbUT2Dmr7fKh5XMOMe6rwOuPpEBJ0uh4Ra4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIb1CP8n6JHuSTCfZMsf2y5Lcl+QbSb6U5FVD2y5O8u3ucfEoi5ckLcy8oZ9kGXA1cD6wFrgoydpZ3e4GJqvqV4EbgQ93Y18BXAG8CVgHXJHkxNGVL0laiD5H+uuA6araW1VPAduBjcMdquq2qnq8W70dWNEtvwW4paoOVNWjwC3A+tGULklaqD6hfwrw8ND6TNd2KJcAnz/MsZKkRbS8R5/M0VZzdkzeCUwCb17I2CSbgc0AK1eu7FGSJOlw9DnSnwFOHVpfAeyb3SnJOcAHgA1V9eRCxlbVtVU1WVWTExMTfWuXJC1Qn9DfBaxJsjrJMcAmYMdwhyRnANcwCPxHhjbdDJyX5MTuC9zzujZJ0hjMO71TVQeTXMogrJcB26pqd5KtwFRV7QA+ArwE+HQSgIeqakNVHUjyQQYfHABbq+rAorwSSdK8+szpU1U7gZ2z2i4fWj7n/xm7Ddh2uAVKkkbHK3IlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0JakhvUI/yfoke5JMJ9kyx/azktyV5GCSC2ZtezrJPd1jx6gKlyQt3PL5OiRZBlwNnAvMALuS7Kiq+4a6PQS8C/jTOXbxRFWdPoJaJUlHaN7QB9YB01W1FyDJdmAj8NPQr6oHu23PLEKNkqQR6TO9cwrw8ND6TNfW13FJppLcnuTtC6pOkjRSfY70M0dbLeA5VlbVviSnAbcmubeqHnjOEySbgc0AK1euXMCuJUkL0edIfwY4dWh9BbCv7xNU1b7u517gy8AZc/S5tqomq2pyYmKi764lSQvUJ/R3AWuSrE5yDLAJ6HUWTpITkxzbLZ8MnMnQdwGSpKNr3tCvqoPApcDNwP3ADVW1O8nWJBsAkrwxyQxwIXBNkt3d8NcCU0m+DtwGXDnrrB9J0lHUZ06fqtoJ7JzVdvnQ8i4G0z6zx30VeP0R1ihJGhGvyJWkhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhve6n/3yyasvnxl2CFtmDV7513CUcdb6v23A03tse6UtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGtIr9JOsT7InyXSSLXNsPyvJXUkOJrlg1raLk3y7e1w8qsIlSQs3b+gnWQZcDZwPrAUuSrJ2VreHgHcBn5g19hXAFcCbgHXAFUlOPPKyJUmHo8+R/jpguqr2VtVTwHZg43CHqnqwqr4BPDNr7FuAW6rqQFU9CtwCrB9B3ZKkw9An9E8BHh5an+na+jiSsZKkEesT+pmjrXruv9fYJJuTTCWZ2r9/f89dS5IWqk/ozwCnDq2vAPb13H+vsVV1bVVNVtXkxMREz11LkhaqT+jvAtYkWZ3kGGATsKPn/m8GzktyYvcF7nldmyRpDOYN/ao6CFzKIKzvB26oqt1JtibZAJDkjUlmgAuBa5Ls7sYeAD7I4INjF7C1a5MkjcHyPp2qaiewc1bb5UPLuxhM3cw1dhuw7QhqlCSNiFfkSlJDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQ3qFfpL1SfYkmU6yZY7txyb5VLf9jiSruvZVSZ5Ick/3+PvRli9JWojl83VIsgy4GjgXmAF2JdlRVfcNdbsEeLSqXp1kE3AV8I5u2wNVdfqI65YkHYY+R/rrgOmq2ltVTwHbgY2z+mwEru+WbwTOTpLRlSlJGoU+oX8K8PDQ+kzXNmefqjoI/BA4qdu2OsndSb6S5DfneoIkm5NMJZnav3//gl6AJKm/PqE/1xF79ezzfWBlVZ0BXAZ8IslLf6Zj1bVVNVlVkxMTEz1KkiQdjj6hPwOcOrS+Ath3qD5JlgMvAw5U1ZNV9V8AVXUn8ADwy0datCTp8PQJ/V3AmiSrkxwDbAJ2zOqzA7i4W74AuLWqKslE90UwSU4D1gB7R1O6JGmh5j17p6oOJrkUuBlYBmyrqt1JtgJTVbUDuA74WJJp4ACDDwaAs4CtSQ4CTwN/VFUHFuOFSJLmN2/oA1TVTmDnrLbLh5Z/DFw4x7ibgJuOsEZJ0oh4Ra4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIb1CP8n6JHuSTCfZMsf2Y5N8qtt+R5JVQ9ve37XvSfKW0ZUuSVqoeUM/yTLgauB8YC1wUZK1s7pdAjxaVa8G/hq4qhu7FtgE/AqwHvjbbn+SpDHoc6S/Dpiuqr1V9RSwHdg4q89G4Ppu+Ubg7CTp2rdX1ZNV9R1gutufJGkM+oT+KcDDQ+szXducfarqIPBD4KSeYyVJR8nyHn0yR1v17NNnLEk2A5u71ceS7OlR16GcDPzgCMYvFutamEPWlauOciXP9bz7fY2ZdS1Arjqiul7Vp1Of0J8BTh1aXwHsO0SfmSTLgZcBB3qOpaquBa7tU/B8kkxV1eQo9jVK1rUw1rUw1rUwLdfVZ3pnF7AmyeokxzD4YnbHrD47gIu75QuAW6uquvZN3dk9q4E1wNdGU7okaaHmPdKvqoNJLgVuBpYB26pqd5KtwFRV7QCuAz6WZJrBEf6mbuzuJDcA9wEHgfdU1dOL9FokSfPoM71DVe0Eds5qu3xo+cfAhYcY+yHgQ0dQ40KNZJpoEVjXwljXwljXwjRbVwazMJKkFngbBklqyAsm9JNsS/JIkm+Ou5ZnJTk1yW1J7k+yO8l7x10TQJLjknwtyde7uv5i3DUNS7Isyd1J/mXctTwryYNJ7k1yT5KpcdfzrCQvT3Jjkm9177NfH3dNAEle0/2unn38KMn7lkBdf9K957+Z5JNJjht3TQBJ3tvVtHuxf08vmOmdJGcBjwH/VFWvG3c9AEleCbyyqu5K8vPAncDbq+q+MdcV4ISqeizJi4B/B95bVbePs65nJbkMmAReWlVvG3c9MAh9YLKqltS53UmuB/6tqj7anV13fFX997jrGtbdeuV7wJuq6rtjrOMUBu/1tVX1RHeSyc6q+sdx1dTV9ToGdzpYBzwFfAF4d1V9ezGe7wVzpF9V/8rgzKElo6q+X1V3dcv/A9zPErgiuQYe61Zf1D2WxKd/khXAW4GPjruWpS7JS4GzGJw9R1U9tdQCv3M28MA4A3/IcuDF3fVExzPHdUNj8Frg9qp6vLujwVeA312sJ3vBhP5S19159AzgjvFWMtBNodwDPALcUlVLoi7gb4A/A54ZdyGzFPDFJHd2V5AvBacB+4F/6KbDPprkhHEXNYdNwCfHXURVfQ/4S+Ah4PvAD6vqi+OtCoBvAmclOSnJ8cDv8NyLWkfK0D8KkrwEuAl4X1X9aNz1AFTV01V1OoOrpNd1f2KOVZK3AY9U1Z3jrmUOZ1bVGxjcbfY93XTiuC0H3gD8XVWdAfwv8DO3Ph+nbsppA/DpJVDLiQxuArka+CXghCTvHG9VUFX3M7gz8S0Mpna+zuC6pkVh6C+ybs78JuDjVfWZcdczWzcd8GUGt74etzOBDd38+Xbgt5P883hLGqiqfd3PR4DPsjTuFjsDzAz9lXYjgw+BpeR84K6q+s9xFwKcA3ynqvZX1U+AzwC/MeaaAKiq66rqDVV1FoNp6kWZzwdDf1F1X5heB9xfVX817nqelWQiycu75Rcz+MfwrfFWBVX1/qpaUVWrGEwJ3FpVYz8SS3JC90U83fTJeQz+JB+rqvoP4OEkr+mazmZw9ftSchFLYGqn8xDwa0mO7/5tns3ge7axS/IL3c+VwO+xiL+zXlfkPh8k+STwW8DJSWaAK6rquvFWxZnA7wP3dvPnAH/eXeE8Tq8Eru/Oqvg54IaqWjKnRy5Bvwh8dpATLAc+UVVfGG9JP/XHwMe7aZS9wB+MuZ6f6uanzwX+cNy1AFTVHUluBO5iMH1yN0vnytybkpwE/ITB7WoeXawnesGcsilJmp/TO5LUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SG/B+ZRoqYpq+PEAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt \n",
    "\n",
    "hist_data = [1,2,3,4,5,5,6,7,8,9]\n",
    "\n",
    "plt.figure()\n",
    "plt.hist(hist_data, density=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see that the heights are the same as before, but look carefully at the $y$-axis and you can see that they are less than 1. However, you should found that the frequency of $5$'s should be $0.2$ and others should be $0.1$ instead. The reason why this happens is because the `density` argument changes the $y$ values so that the area of the histogram is $1$. We can use `histplot()` from `seanborn` to visualise the density instead. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAD8CAYAAABdCyJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4XdWZ5/vvq3mwNcvW5BEbPOEZAwkxCaMJgxlMYpJUSFX6oVNP0TdVPLdvkb435AlV1V253Z3kpi83XXQgIQMBAiGYxMSQhCEU2FieJ4xl2ZZkSbYmW5MlWdJ7/zjb1EHI1pGPj4+G3+d5zqN91t5rnXdLtl7ttdZe29wdERGR85UQ7wBERGR0UyIREZGoKJGIiEhUlEhERCQqSiQiIhIVJRIREYmKEomIiERFiURERKKiRCIiIlFJincAF0NBQYFPnz493mGIiIwqW7ZsaXT3wqGOGxeJZPr06ZSXl8c7DBGRUcXMjkRynLq2REQkKkokIiISFSUSERGJihKJiIhERYlERESiEtNEYmarzGy/mVWY2cOD7H/IzPaa2U4z+6OZTQvbd7+ZHQhe94eVLzOzXUGbPzAzi+U5iIjIucUskZhZIvAYcAswD7jPzOYNOGwbsNzdFwLPA/93UDcP+BZwJbAC+JaZ5QZ1fgg8AMwOXqtidQ4iIjK0WF6RrAAq3L3S3XuAZ4DV4Qe4++vu3hm83QiUBds3A6+5e7O7twCvAavMrBjIcvd3PfSM4J8Cd8bwHEREZAixTCSlQHXY+5qg7Gy+CrwyRN3SYDvSNkVEJMZieWf7YGMXPuiBZl8ClgPXDlF3OG0+QKgLjKlTpw4V61lNmTqNmuqq864/GiUmJdPXezreYVxUsTznhLQJJKRmYqkZJKRmkpCagSV89L+e43hPF/3dHfR3d+I9nfSdaoW+3pjEBPo5jwdlU6ZSXRXRzelRiWUiqQGmhL0vA2oHHmRmNwD/J3Ctu3eH1f30gLpvBOVlA8o/1iaAuz8OPA6wfPnyQZNNJGqqq/juq/vPt/qo9NBNl+mch6Gv32np7KG5o4emjh5aT52mvauXtu5e2rt76es/739+pCcnMjEtKfRKTSYnM5n8zBTyM1NJT0k873ZBP+fx4KGbLrsonxPLRLIZmG1mM4CjwFrgC+EHmNkS4F+AVe5+PGzXBuA/hw2w3wR8w92bzazNzK4CNgFfBv5HDM9B5CP6+52mjh7qTp6i7mQXx9u6OdHZw5lcYcCEtCQmpCYxOSuVWakTyExNJDUpkZSkBFKTEkhJSiAx4aMX1+7Q09dPT2/o1d3bx6nTfR8mpBOdp6luPkVPX/+HdTJSEimYkEpRVhrF2WkUZaeRlhxdchE5HzFLJO7ea2YPEkoKicCT7r7HzB4Fyt19HfBfgQnAr4JZvFXufkeQMP6BUDICeNTdm4PtvwZ+AqQTGlN5BZEYcXca2rs50tRJdXMn9a1dnO4LZY2MlEQmZ6UxsyCT/Amhq4TcjGSSEmMz9OjudHT30dTRTVNHD03tPTS0dbP5cPOH/bu5GcmU5WYwNS+DKXnppCYpsUjsxXT1X3dfD6wfUPZI2PYN56j7JPDkIOXlwIILGKbIR3T39nGosYPDTZ1UNXVy6nQfAPkTUphbnEVxdhrF2elkpSVxMW9jMrPQ1U5aEtPyMz8s7+nt51hrF3Unu6g7eYr361vZdfQkZlCUlcb0/ExmFmaSn5lyUeOV8WNcLCMvMpSEtAnsrW3lwPE2qptP0edOenIiU/MzmJYX+gs/M3Vk/ndJSUpgSl4GU/IygNCYTf3JLo40d3CkqZN3K5t4t7KJnIxkZk+awKzCCRROTI1z1DKWjMz/GSIXQV+/c6ixg711rZQ9+HNe23eMiWlJLCzLZtakCRRnp43Kv+ATE4zS3HRKc9P5xCXQ0d3LwYZ2Ko63U36khc2HW8hOTyb7E2tpPXWarPTkeIcso5wSiYw7x1u72FvXyv76Nrp6+8lMSaS1/CUeePBvmTwxdVQmj3PJTE1iYVkOC8tyONXTx8GGdvbXt5HzqS/x43cOU5abzrziLGZNmkByjMZ3ZGzTvxoZF3r7+tlX18ozm6v45eZqdte2MjUvg9WLS/irT87gxBs/pihrdF6BDEd6SiILSrO5Z1kZNT/8K66amUdbVy+v7j3GE28f4q0DDZzo7Il3mDLK6IpExrS2rtPsqDnJ3tpWTp3uIzcjmWsvLWRu0URSx/lU2b7W41w5I58V0/OoPdHFzpoT7Kg+wbaqE0zLz2BhWTYz8jPHfHKV6CmRyJjU0NbN1qoWPjjWhgMzCzJZWJbDlNx0/WIcwOzfxlQ6unvZffQku2pP8vKOOvIyUlg6LYfLiiaSlKAODBmcEomMGe5Odcspthxpoaq5k+REY1FZDoun5pCVpgHlSGSmJnHlzHyWT8+j4ng7W4608Id9x3n3YBOLpuSwsCxb96bIxyiRyKh3JoFsrGyi7mQXGSmJfOKSfC4vzdad3ucpMcG4rGgil06eQFVzJ1uqWnjnYBNbjrSwZGoOi6fkKKHIh5RIZFSrbu5k46Emak90MSE1iU9fVsj84qyY3V0+3pgZ0/IzmZafybHWLt471MzGyma2VZ1g6dRcFk/JISVJ3+vxTolERqWalk42VjZz9MQpMlMT+fSlhcwvUQKJpclZady+qIRjrV1sOtTMu5VNbKtqYem0XBaVKaGMZ0okMqo0tnfzdkUjR5o6yUxJ5NpLC1mgBHJRTc5K445FJdS3drGpsol3DjaxvfoEV87IY0FJNgkJmsww3iiRyKjQ3t3Lxsom9ta2kpKUwDWzClhUlq0EEkdFWWmsXlxK3clTvF3RyOv7G9hefYJrZhUwo0DThscTJRIZ0Xp6+9lypIWtVS30u7N4Sg4rZuRpEH0EKc5OZ83SMiobO/jXikZe3llHaU4618wuoCgrLd7hyUWgRCIjkruzt66Vdw420dnTx6WTJvCJWQVka12oEcnMuKRwAtPzM9lTe5KNlc08u7mayyZP5JpZBUxI06+asUw/XRlxjrV28fr+4xxr7aY4O43bF5ZQlK2/bEeDxARjYVkOc4qyKD/SzNaqE1Q2trNiRh5LpuR+7IFeMjYokciIcaqnj3cONrK7tpWMlERumjeZOUUT1dc+CqUkJfCJSwqYX5LNWx808K8VofGtay8t/MizVGRsUCKRuOt3Z9fRk7x7sInTff0smZrDlTPydMPbGJCdnszti0o41NjBmx808JvttVxSmMnK2YVavn4MiemUFzNbZWb7zazCzB4eZP9KM9tqZr1mtias/DNmtj3s1WVmdwb7fmJmh8L2LY7lOUhsHW/r4tnN1byxv4HCial8YcVUVs4uVBIZY2YUZPKlq6byiUvyOdLUyc82HmHLkRb6zzzsXka1mF2RmFki8BhwI1ADbDazde6+N+ywKuArwP8eXtfdXwcWB+3kARXAq2GH/Ed3fz5WsUvsne7rZ1NlM1urW0hPTuSWBUXMnjRB3VhjWFJCAldMz2NO0UTe2N/A2xWN7K9v4/q5k5is2V2jWiy7tlYAFe5eCWBmzwCrgQ8TibsfDvb1n6OdNcAr7t4Zu1DlYjrS1MGf3j9Oa1cv80uyuGZWgabzjiMT05K5bWExFQ3tvLm/gWc3V7N4Sg5XzczX3fGjVCwTSSlQHfa+BrjyPNpZC3x3QNk/mdkjwB+Bh929+/xClIups6eXtw6E/grNzUjmnqWllOVmxDssiQMzY/akiUzNzeBfDzaxrfoEFQ3tfOayScwo0GD8aBPL9D9YH8WwOkTNrBi4HNgQVvwNYA5wBZAH/P1Z6j5gZuVmVt7Q0DCcj5UYOHCsjZ9vrOLAsTZWzMjjCyumKokIqcmJXDdnEvcuKyM5IYF1O2p5dW893af74h2aDEMsE0kNMCXsfRlQO8w2Pge86O6nzxS4e52HdAM/JtSF9jHu/ri7L3f35YWFhcP8WLlQTvX08cquOtbvrmdiWhL3rZjK1TPztbSJfERJTjr3XTmFFdPzeL++jZ9tOsKhxo54hyURimXX1mZgtpnNAI4S6qL6wjDbuI/QFciHzKzY3essNCp7J7D7QgQrF17F8Xb+9P5xunv7uHpmPsum6YY0ObukhASuviSfmYWZvLb3GOt21DKvOIuVlxZoFt8IF7NE4u69ZvYgoW6pROBJd99jZo8C5e6+zsyuAF4EcoHbzezb7j4fwMymE7qieXNA078ws0JCXWfbga/F6hzk/HSd7uON/Q3sP9ZG4cRU7ppbSuHE1HiHJaPE5Kw01q6YwqbK5g+fdnnD3Em6kXEEi+kNie6+Hlg/oOyRsO3NhLq8Bqt7mNCA/cDy6y5slHIhVTV38ureek719HHVjDyWT8/TVYgMW1JCAp+cVcAlhRN4dW89v9ley8KybD41q0DdoiOQ7myXC6K3v593DzaxteoEeRkp3HFFCZMm6t4AiU5RdhpfWDGVd4KZXTUtp1g1v0hXuCOMEolEram9m9/vqaexvYeFpdlcM7uAZP3VKBdIUmICKy8tZFp+Bq/tPcazm6v5xKx8lkzJ0Q2sI4QSiZw3d2dnzUn+XNFISmICty8qZmbBhHiHJWPUtPxMvnjlNP6w7xh/PtDI4aYObppbpCXqRwD9BOS8dHT38od9xzjc1Mm0/AxunDuZzFT9c5LYSk9J5LaFxeypbeXNDxr4xaYjXD93MrMm6Q+YeNL/fBm2I00dbNhzjJ6+fj59aSELy7LVxSAXjZmxoDSb0tx0fr+7nt/tqmN+SRafvrRQA/FxokQiEevvdzYeamLz4RbyM1O4Z0Ep+RM06CnxkZuRwueWT2FjZRPlR1qob+3i1gXF5GamxDu0cUfpWyLS3t3Lr7cdZfPhFuaXZPH5K6YoiUjcJSYYn5xVwOrFJXR09/LLzVW8X98a77DGHSUSGdKRpg6e3lTFsdYubpo3mRvmTtasLBlRpudn8oUVUymYkMqGPcf44/vH6O0716LiciGpa0vOqt+dTZXNvHe4mfzMFD57eRl56jaQEWpiWjL3LC3j3comthxpof5kF5+9vJjcDP2bjTX9WSmD6uju5cVtR3nvcDNziyfy+SumKInIiJeYYFwzq4A7FpXQ3tXLM+9V88GxtniHNeYpkcjHpJbO4+n3qqg/2cWNcydz07widWXJqDKjIJP7rpxK/oQUXtldz5v7G+jTY31jRl1b8iF3Z+fRk0y+7z+TkpjAXUtKKdCAuoxSWUFX19sVjWyvPkFDezcJGTnxDmtM0p+ZAkBvXz9/2HecN/Y3cOrQVtZeMUVJREa9xATj2ksLuXn+ZI61dlF8//eob+2Kd1hjjhKJ0NZ1ml9tqWFvXSsrZuTR8MI/kKpnqMsYMqcoi3uXl4H38/yWGvbUnox3SGOKEsk4V9PSyS/fq+ZE52luW1jM1TPzGeYTkUVGhUkT06h76u8oyUnjD/uO8/r7xzVucoFojGSccne2V5/gzxWN5KQnc9vCEs3KkjGv/1Qrdy4q5Z2DTWypaqGhvZtbLy/WOnFR0hXJONTb389r+47x1oFGZhZkamqvjCsJCcY1swu4ZUERDW3d/HJz6GZbOX8xTSRmtsrM9ptZhZk9PMj+lWa21cx6zWzNgH19ZrY9eK0LK59hZpvM7ICZPWtm+g04DJ09vby49Sj76tq4ckYet15erOdhy7h06eTQ/VGJZjy/pYYDut/kvMUskZhZIvAYcAswD7jPzOYNOKwK+Arw9CBNnHL3xcHrjrDy7wDfc/fZQAvw1Qse/BjV1N7Ns5urOdbWzS0LirhqZr5W7ZVxrWBCKp+/YgqFE1NZv7ue9w41465xk+GK5RXJCqDC3SvdvQd4BlgdfoC7H3b3nUBEi+JY6LfedcDzQdFTwJ0XLuSx63BTB8+V19Db76xZWsalkyfGOySRESEjJYm7l5Qyp2gi71Y2sWGv1ukarlgmklKgOux9TVAWqTQzKzezjWZ2JlnkAyfcvXeoNs3sgaB+eUNDw3BjHzPODKqv215Ldnoya6+YQlG2nqUuEi4pMYGb5k3m6kvy2V/fxq+3HaWzp3foigLENpEM1mcynGvGqe6+HPgC8H0zu2Q4bbr74+6+3N2XFxYWDuNjx46+fuf1/Q28+UEDMwoyWbOsjIlpyfEOS2REMjNWTM/js8Eg/DObq2ls7453WKNCLBNJDTAl7H0ZUBtpZXevDb5WAm8AS4BGIMfMzszVG1ab40l3bx/rdtSy6+hJlk3L5baFxaQkaZKeyFBmT57ImmVl9Lvzq/IaDjV2xDukES+Wv1k2A7ODWVYpwFpg3RB1ADCzXDNLDbYLgE8Cez00CvY6cGaG1/3ASxc88lGures0vyqvoaalkxvmTuKaWQUaVBcZhslZaaxdPpWcjGRe3lnL7qO6E/5cYpZIgnGMB4ENwD7gOXffY2aPmtkdAGZ2hZnVAPcC/2Jme4Lqc4FyM9tBKHH8s7vvDfb9PfCQmVUQGjN5IlbnMBo1tnfzXHkNbV29rF5cyvyS7HiHJDIqTUhL4p6lZUzNy+CP7x/n3comzeg6i5jezunu64H1A8oeCdveTKh7amC9d4DLz9JmJaEZYTJATUsnL++sIznRWLOsjMKJWnRRJBopSQncvrCEP71/nPcONdPe1ct1cyaRmKAr/HBaF2CM2F/fxmt7j5GdkczqxSVkaVBd5IJITDBumDuJiWlJbDrUTEdPL59doDHHcPpOjAFbq1r4/Z56Jmencu+yMiURkQvMzLhqZj7Xz5lEVXMnL2ytoaNb04PPUCIZxdydNz9o4M8HGpk9aQJ3LS4lTcu/i8TMgtJsbl9YQnNHD8+VV9PS2RPvkEYEJZJRqre/n1d217O9+gSLp+Rwy4IikvQ4XJGYm1GQyT3Lyjjd5zxXXk39SS34qN88o9Dpvn5e3lHHgePtfGpWAddeWqjpvSIXUVFWGp9bXkZqUiK/3lZDVXNnvEOKKyWSUabrdB8vbjtKdXMnN86dzNJpufEOSWRcyslICY1JpiezbnstFcfb4x1S3CiRjCId3b28sLWG463dfPbyYuaVZMU7JJFxLTM1iTVLQ1Pt1++qY29da7xDigslklGi9VToueonT53mjsUlzJo0Id4hiQiQlpzIXUtKKctL57W9x9hefSLeIV10SiSjQHNHD7/aUkPX6T7uWlLK1LyMeIckImFSkhK4Y1EJlxRm8uYHDWwcZ3fBK5GMcMdau/jVlmr63VmzrIzi7PR4hyQig0hKSOCzC4qZV5zFpkPNvHWgcdwkE93ZPoLVtHTy8o460pITuGtJKTkZeqqwyEiWENwFn5qUwLbqE3T39nHD3MkkjPFZlUokI1RVcycv76glKy2Zu5aUMiFNPyqR0cDM+NTsAlKTE9hY2Uxfv3PzvCISxvD6XPrtNAIdaerg5Z115GQkc/eSUjJS9GMSGU3MjCtn5JOUkMDbFY3099ezakHRmF3sUWMkI0xlYzsv76gjLzOFe5aWKYmIjGLLpuWycnYBFQ3trN9VR2//2HwWvBLJCHKwoZ3f7awjf0IKdy8pJV3rZomMekum5vLpSwupbOzgdzvr6O0be8lEiWSEOHCsjfW76pg0MY27l2jxRZGxZNGUHK6bM4nDTZ38dgwmEyWSEWB/fRuv7KlnclYady4pIVVJRGTMubw0mxvmTuJIcyfrdtRyegwlk4gSiZm9YGa3mtmwEo+ZrTKz/WZWYWYPD7J/pZltNbNeM1sTVr7YzN41sz1mttPMPh+27ydmdsjMtgevxcOJaaTZV9fKhj31lGSnc+fiUlKTlERExqr5JdncPG8yNS2neGl7LT29YyOZRJoYfgh8AThgZv9sZnOGqmBmicBjwC3APOA+M5s34LAq4CvA0wPKO4Evu/t8YBXwfTPLCdv/H919cfDaHuE5jDj76lp5de8xSnPTWb24RE9cExkH5hRncfP8ImpPnuKlHUfHxJVJRL+53P0P7v5FYClwGHjNzN4xs780s7M9jm8FUOHule7eAzwDrB7Q7mF33wn0Dyj/wN0PBNu1wHGgcBjnNeKdeTTulNx0Vi8qIVnPEhEZNy4rmsgt84uoO9E1Jrq5Iv7tZWb5hK4e/h2wDfh/CCWW185SpRSoDntfE5QNi5mtAFKAg2HF/xR0eX3PzFLPUu8BMys3s/KGhobhfmxMHTjWxoa99ZTkpHP7ohI9kEpkHJo9eSI3zQ91c432AfhIx0h+DfwZyABud/c73P1Zd/8PwNmWoR3szpthLTxjZsXAz4C/dPcz3+VvAHOAK4A84O8Hq+vuj7v7cndfXlg4ci5mDja08/s99RRlpXGHrkRExrU5RVncOG8yVc2d/HYU32cS6W+xH7n7PHf/L+5eB3DmSsDdl5+lTg0wJex9GVAbaWBmlgX8Dvi/3H3jmXJ3r/OQbuDHhLrQRoVDjR0fTvHVmIiIAMwrzuL6OZM40tTJ+l319PWPvoUeI/1N9o+DlL07RJ3NwGwzm2FmKcBaYF0kHxYc/yLwU3f/1YB9xcFXA+4EdkfSZrwdaQrdjFQwIZU7F5dodpaIfGhBaTafuayQQ40dvLK7btQlk3Ouv2FmRYTGNdLNbAn/1l2VRaib66zcvdfMHgQ2AInAk+6+x8weBcrdfZ2ZXUEoYeQCt5vZt4OZWp8DVgL5ZvaVoMmvBDO0fmFmhUEs24GvDfusL7Kq5k5e3llHbmZoAUbdJyIiAy0sy6Gv33nrQCOv7qnn5vmjZ6HHoRZyupnQAHsZ8N2w8jbgPw3VuLuvB9YPKHskbHtz0PbAej8Hfn6WNq8b6nNHktBS8LXkpCdz95Iy3bEuIme1ZGou/Q5vVzRiCce4ad7oWIL+nInE3Z8CnjKze9z9hYsU05hRfzI0tW9iWhJ3LSklPUVJRETObdm0XPrcefdgE0kJxvVzJmEjPJkM1bX1peDqYLqZPTRwv7t/d5BqAjS2d/Ob7UfJSEni7qVlZKZqFV8RicyK6Xn09vWz+XALqUkJXDOrYEQnk6F+u2UGX882xVcGcaKzhxe3HSUp0UIPpVISEZFhunpmPj29/WytOkFKUgJXzsiPd0hnNVTX1r8EX799ccIZ/dq6TvPrbUdDz1hfUkZ2+tlu/BcROTsz49pLC+np62djZTMpiQksmZob77AGNVTX1g/Otd/d/7cLG87o1tnTy4vbjtJ9up+7l5aSP2HQm+5FRCJiZtwwZzI9vf28daCRlKQE5pdkxzusjxmqz2XLRYliDOg+3cdvttXS1tXLnYtLmZyVFu+QRGQMSEgwVi0o4uUddfxx33FSEhOYPXlivMP6iEhmbckQTvf189KOWpo6url9UQmluenxDklExpCkhARuW1jMi9uO8vs99SQnJTA9P3PoihfJOe9sN7PvB19fNrN1A18XJ8SRrbe/n9/urKP+ZBer5heNqB+uiIwdyYkJrF5UQn5mKr/bWcfRllPxDulDQ3Vt/Sz4+t9iHcho1O/Oht3HqGru5Ma5k0fc5aaIjC2pyYncuaSE57fUsG5HLWuWlVE4Mf5jsee8InH3LcHXNwmtrdUCNAPvBmXjlrvz+v7jVDS086nZBcwryYp3SCIyDmSkhG5wTklK4Dfbj3KisyfeIUW8jPythJ4H8gPg/wUqzOyWWAY20m081Mzuo60sn5bL0hE6JU9ExqaJaaF1+/r7nd9sr6Wjuzeu8US6+u9/Bz7j7p9292uBzwDfi11YI9uO6hO8d6iZecVZfOKSkXuTkIiMXXmZKaxeXEpHdy8vba+lu7cvbrFEmkiOu3tF2PtKQo+/HXc+ONbGGx80MLMgc1SsgSMiY1dRdhq3LiymqaOb3+6I31MWh5q1dbeZ3Q3sMbP1ZvYVM7sfeJnQ80bGlSNNHWzYU09JThq3LBg9SzyLyNg1PT+TG+dNpubEKX6/p55+v/jPMhlq1tbtYdvHgGuD7QZCzxAZN+pbu/jdrjpyM1O4Y6Gesy4iI8ecoixO9fTx1oFGXn//ONdd5N6SoW5I/MuLFchI1tLRw7rttaQnJ3LXYj2YSkRGniVTczl1uo/Nh1vISEni6os4fhvRsrRmlgZ8FZgPfLj2h7v/VYziGjESJ+Tz4vajANy5pFTLwYvIiHX1zHw6e/p473DzRX3+UaT9Mz8Digg9MfFNQk81bBuqkpmtMrP9ZlZhZg8Psn+lmW01s14zWzNg3/1mdiB43R9WvszMdgVt/sBieP12svM0kz73bbpO97F6cQm5GSmx+igRkaiZGdddNolLCjN584MGMuauvCifG2kimeXu3wQ6gvW3bgUuP1cFM0sEHgNuAeYB95nZvAGHVRF6lO/TA+rmAd8CrgRWAN8yszNjMj8EHgBmB69VEZ7DsLg7f/P0VpJzS7ltYYkWYRSRUSEhwVg1v4jSnHQKbv07dtaciP1nRnjc6eDrCTNbAGQD04eoswKocPdKd+8BngFWhx/g7ofdfScwcM7azcBr7t7s7i3Aa8AqMysGstz9XXd34KfAnRGew7CYGQ9eN4vGl/8rU/MyYvERIiIxkZSYwO2Lijn5zrPMK479qhuRJpLHgyuCbwLrgL3Ad4aoUwpUh72vCcoicba6pcH2kG2a2QNmVm5m5Q0NDRF+7EddNTOfzg/eOa+6IiLxlJqUyMl3nrkoM0wjGjl29x8Fm28CMyNse7Cxi0gnOJ+tbsRtuvvjwOMAy5cvv/gTq0VExolI19rKN7P/EQyMbzGz75vZUHPLaoApYe/LgNoI4zpb3Zpg+3zaFBGRGIj0mucZQkui3AOsARqBZ4eosxmYbWYzzCwFWEuoWywSG4CbzCw36FK7Cdjg7nVAm5ldFczW+jLwUoRtiohIDESaSPLc/R/c/VDw+kcg51wV3L0XeJBQUtgHPOfue8zsUTO7A8DMrjCzGuBe4F/MbE9Qtxn4B0LJaDPwaFAG8NfAj4AKQisSvzKM8xURkQss0rvrXjeztcBzwfs1wO+GquTu64H1A8oeCdvezEe7qsKPexJ4cpDycmBBhHGLiEiMnTORmFkb/zbI/RDw82BXAtBO6F4PEREZx4Zaa0vPjhURkXOKeOGoYFzjzP32b7j7b2MTkoiIjCaRTv/9Z+DrhG5E3At8PSgTEZFxLtIrks8Ci929H8DMngIxXuWaAAANF0lEQVS2AR9biFFERMaX4dw7Hz7dN/tCByIiIqNTpFck/wXYZmavE5rBtRL4RsyiEhGRUWPIRBLcQf42cBVwBaFE8vfuXh/j2EREZBQYMpG4u5vZb9x9GZEvcSIiIuNEpGMkG83siphGIiIio1KkYySfAb5mZoeBDkLdW+7uC2MVmIiIjA6RJpJbYhqFiIiMWkOttZUGfA2YBewCnghW9RUREQGGHiN5ClhOKIncAvz3mEckIiKjylBdW/Pc/XIAM3sCeC/2IYmIyGgy1BXJ6TMb6tISEZHBDJVIFplZa/BqAxae2Taz1qEaN7NVZrbfzCrM7GPrcplZqpk9G+zfZGbTg/Ivmtn2sFe/mS0O9r0RtHlm36Thn7aIiFwoQz2PJPF8GzazROAx4EagBthsZuvcfW/YYV8FWtx9VvAExu8An3f3XwC/CNq5HHjJ3beH1fti8KREERGJs+Es2jhcK4AKd6909x7gGWD1gGNWExrQB3geuD5YkiXcfcAvYxiniIhEIZaJpBSoDntfE5QNekwwBnMSyB9wzOf5eCL5cdCt9c1BEo+IiFxEsUwkg/2C9+EcY2ZXAp3uvjts/xeDmWSfCl5/MeiHmz1gZuVmVt7Q0DC8yEVEJGKxTCQ1wJSw92VA7dmOMbMkQs85aQ7bv5YBVyPufjT42gY8TagL7WPc/XF3X+7uywsLC6M4DREROZdYJpLNwGwzm2FmKYSSwsDVg9cB9wfba4A/ubsDmFkCcC+hsRWCsiQzKwi2k4HbgN2IiEjcRLrW1rC5e6+ZPQhsABKBJ919j5k9CpS7+zrgCeBnZlZB6EpkbVgTK4Ead68MK0sFNgRJJBH4A/C/YnUOIiIytJglEgB3Xw+sH1D2SNh2F6GrjsHqvkHoYVrhZR3AsgseqIiInLdYdm2JiMg4oEQiIiJRUSIREZGoKJGIiEhUlEhERCQqSiQiIhIVJRIREYmKEomIiERFiURERKKiRCIiIlFRIhERkagokYiISFSUSEREJCpKJCIiEhUlEhERiYoSiYiIREWJREREohLTRGJmq8xsv5lVmNnDg+xPNbNng/2bzGx6UD7dzE6Z2fbg9T/D6iwzs11BnR+YmcXyHERE5NxilkjMLBF4DLgFmAfcZ2bzBhz2VaDF3WcB3wO+E7bvoLsvDl5fCyv/IfAAMDt4rYrVOYiIyNBieUWyAqhw90p37wGeAVYPOGY18FSw/Txw/bmuMMysGMhy93fd3YGfAnde+NBFRCRSsUwkpUB12PuaoGzQY9y9FzgJ5Af7ZpjZNjN708w+FXZ8zRBtiojIRZQUw7YHu7LwCI+pA6a6e5OZLQN+Y2bzI2wz1LDZA4S6wJg6dWrEQYuIyPDE8oqkBpgS9r4MqD3bMWaWBGQDze7e7e5NAO6+BTgIXBocXzZEmwT1Hnf35e6+vLCw8AKcjoiIDCaWiWQzMNvMZphZCrAWWDfgmHXA/cH2GuBP7u5mVhgM1mNmMwkNqle6ex3QZmZXBWMpXwZeiuE5iIjIEGLWteXuvWb2ILABSASedPc9ZvYoUO7u64AngJ+ZWQXQTCjZAKwEHjWzXqAP+Jq7Nwf7/hr4CZAOvBK8REQkTmI5RoK7rwfWDyh7JGy7C7h3kHovAC+cpc1yYMGFjVRERM6X7mwXEZGoKJGIiEhUlEhERCQqSiQiIhIVJRIREYmKEomIiERFiURERKKiRCIiIlFRIhERkagokYiISFSUSEREJCpKJCIiEhUlEhERiYoSiYiIREWJREREoqJEIiIiUVEiERGRqMQ0kZjZKjPbb2YVZvbwIPtTzezZYP8mM5selN9oZlvMbFfw9bqwOm8EbW4PXpNieQ4iInJuMXvUrpklAo8BNwI1wGYzW+fue8MO+yrQ4u6zzGwt8B3g80AjcLu715rZAkLPfS8Nq/fF4JG7IiISZ7G8IlkBVLh7pbv3AM8Aqwccsxp4Kth+HrjezMzdt7l7bVC+B0gzs9QYxioiIucplomkFKgOe1/DR68qPnKMu/cCJ4H8AcfcA2xz9+6wsh8H3VrfNDO7sGGLiMhwxDKRDPYL3odzjJnNJ9Td9e/D9n/R3S8HPhW8/mLQDzd7wMzKzay8oaFhWIGLiEjkYplIaoApYe/LgNqzHWNmSUA20By8LwNeBL7s7gfPVHD3o8HXNuBpQl1oH+Puj7v7cndfXlhYeEFOSEREPi6WiWQzMNvMZphZCrAWWDfgmHXA/cH2GuBP7u5mlgP8DviGu//rmYPNLMnMCoLtZOA2YHcMz0FERIYQs0QSjHk8SGjG1T7gOXffY2aPmtkdwWFPAPlmVgE8BJyZIvwgMAv45oBpvqnABjPbCWwHjgL/K1bnICIiQ4vZ9F8Ad18PrB9Q9kjYdhdw7yD1/hH4x7M0u+xCxigiItHRne0iIhIVJRIREYmKEomIiERFiURERKKiRCIiIlFRIhERkagokYiISFSUSEREJCpKJCIiEhUlEhERiYoSiYiIREWJREREoqJEIiIiUVEiERGRqCiRiIhIVJRIREQkKkokIiISlZgmEjNbZWb7zazCzB4eZH+qmT0b7N9kZtPD9n0jKN9vZjdH2qaIiFxcMUskZpYIPAbcAswD7jOzeQMO+yrQ4u6zgO8B3wnqzgPWAvOBVcD/Z2aJEbYpIiIXUSyvSFYAFe5e6e49wDPA6gHHrAaeCrafB643MwvKn3H3bnc/BFQE7UXSpoiIXESxTCSlQHXY+5qgbNBj3L0XOAnkn6NuJG2KiMhFlBTDtm2QMo/wmLOVD5b4BrYZatjsAeCB4G27me0/S5xDKXjopssaz7NuLBUAMYvroZsuO9+qMY0rCkPGFcU5RyOu369znPOo/TkOJUY/5xH7/TKzaOKaFslBsUwkNcCUsPdlQO1ZjqkxsyQgG2geou5QbQLg7o8Dj59v8GeYWbm7L4+2nQtNcQ2P4hoexTU84z2uWHZtbQZmm9kMM0shNHi+bsAx64D7g+01wJ/c3YPytcGsrhnAbOC9CNsUEZGLKGZXJO7ea2YPAhuAROBJd99jZo8C5e6+DngC+JmZVRC6Elkb1N1jZs8Be4Fe4G/cvQ9gsDZjdQ4iIjK0WHZt4e7rgfUDyh4J2+4C7j1L3X8C/imSNmMs6u6xGFFcw6O4hkdxDc+4jstCPUkiIiLnR0ukiIhIVJRIzsLMnjSz42a2O96xhDOzKWb2upntM7M9Zvb1eMcEYGZpZvaeme0I4vp2vGM6I1gVYZuZ/TbesYQzs8NmtsvMtptZebzjOcPMcszseTN7P/h3dvUIiOmy4Pt05tVqZn8b77gAzOzvgn/zu83sl2aWFu+YAMzs60FMe2L9vVLX1lmY2UqgHfipuy+IdzxnmFkxUOzuW81sIrAFuNPd98Y5LgMy3b3dzJKBt4Gvu/vGeMYFYGYPAcuBLHe/Ld7xnGFmh4Hl7j6i7j8ws6eAP7v7j4LZkRnufiLecZ0RLJV0FLjS3Y/EOZZSQv/W57n7qWCS0Hp3/0mc41pAaOWPFUAP8Hvgr939QCw+T1ckZ+HubxGaSTaiuHudu28NttuAfYyAu/s9pD14mxy84v5XipmVAbcCP4p3LKOBmWUBKwnNqMTde0ZSEglcDxyMdxIJkwSkB/fCZXCWe9susrnARnfvDFYNeRO4K1YfpkQyigWrJS8BNsU3kpCgC2k7cBx4zd1HQlzfB/4PoD/egQzCgVfNbEuwEsNIMBNoAH4cdAf+yMwy4x3UAGuBX8Y7CAB3Pwr8N6AKqANOuvur8Y0KgN3ASjPLN7MM4LN89GbuC0qJZJQyswnAC8DfuntrvOMBcPc+d19MaMWBFcHlddyY2W3AcXffEs84zuGT7r6U0GrWfxN0p8ZbErAU+KG7LwE6gBHzuIagq+0O4FfxjgXAzHIJLRw7AygBMs3sS/GNCtx9H6HV1F8j1K21g9A9eTGhRDIKBWMQLwC/cPdfxzuegYKukDcIPQIgnj4J3BGMRTwDXGdmP49vSP/G3WuDr8eBFwn1Z8dbDVATdjX5PKHEMlLcAmx192PxDiRwA3DI3Rvc/TTwa+ATcY4JAHd/wt2XuvtKQt30MRkfASWSUScY1H4C2Ofu3413PGeYWaGZ5QTb6YT+g70fz5jc/RvuXubu0wl1h/zJ3eP+1yKAmWUGkyUIuo5uItQdEVfuXg9Um9mZlQ2vJ7TCxEhxHyOkWytQBVxlZhnB/83rCY1bxp2ZTQq+TgXuJobft5je2T6amdkvgU8TWj2zBviWuz8R36iA0F/ZfwHsCsYjAP5TcMd/PBUDTwUzahKA59x9RE23HWEmAy+GfveQBDzt7r+Pb0gf+g/AL4JupErgL+McDwBBX/+NwL+PdyxnuPsmM3se2Eqo62gbI+cu9xfMLB84TWiZqZZYfZCm/4qISFTUtSUiIlFRIhERkagokYiISFSUSEREJCpKJCIiEhUlEhERiYoSiYiIREWJREREovL/A7p8a0aYYFo/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt \n",
    "import seaborn as sns\n",
    "\n",
    "hist_data = [1,2,3,4,5,5,6,7,8,9]\n",
    "\n",
    "plt.figure()\n",
    "sns.histplot(hist_data, stat='probability', kde=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the correct probability values, we need to ensure that each data values belongs to each bar. This is set by the `bins` argument. Bin in statistics are the range of data values bunch together. For example, often we see age brackets and this is an example of statistical bins. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAD8CAYAAABdCyJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl83XWd7/HXJ/u+p22apG1KK6W0rKGAS0dFsKgsM4KWcRQdZ1Dv4HXkjiPOXFFR78WZuTozd7hqBxBwQwSXzkwRcQBHZbHpShdK0z1N26TZ9/Vz//j9Ug4haU56enqS5v18PM7j/Lbv73wPpHnn9/1+f9+fuTsiIiKnKinRFRARkelNQSIiIjFRkIiISEwUJCIiEhMFiYiIxERBIiIiMVGQiIhITBQkIiISEwWJiIjEJCXRFTgTSkpKfMGCBYmuhojItLJhw4bj7l460XEzIkgWLFhATU1NoqshIjKtmNmBaI5T05aIiMREQSIiIjFRkIiISEwUJCIiEhMFiYiIxCSuQWJmq8xsl5nVmtmdY+y/w8x2mNlWM/tPM5sfse9WM9sdvm6N2H6pmb0UnvOfzczi+R1EROTk4hYkZpYM3AtcCywFbjGzpaMO2wRUu/sFwGPA34Vli4AvAJcDK4AvmFlhWOabwG3A4vC1Kl7fQUREJhbPK5IVQK2773X3fuAR4IbIA9z9GXfvDldfACrC5XcCT7l7s7u3AE8Bq8ysDMhz9+c9eEbww8CNcfwOIiIygXgGSTlwKGK9Ltw2no8CT0xQtjxcjvacIiISZ/EMkrH6LnzMA83+BKgG/n6CspM5521mVmNmNY2NjVFUV+TMq5w3HzNLyKty3vyJKygShXhOkVIHVEasVwD1ow8ys3cAfwv8gbv3RZR966iyz4bbK0Ztf905Adx9DbAGoLq6esywEUm0ukMH+fovdyXks++45tyEfK6cfeJ5RbIeWGxmVWaWBqwG1kYeYGYXA98Grnf3hohdTwLXmFlh2Ml+DfCkux8BOszsinC01oeAn8fxO4iIyATidkXi7oNmdjtBKCQDD7j7djO7G6hx97UETVk5wI/DUbwH3f16d282sy8ThBHA3e7eHC5/AngQyCToU3kCERFJmLjO/uvu64B1o7bdFbH8jpOUfQB4YIztNcCy01hNERGJge5sFxGRmChIREQkJgoSERGJiYJERERioiAREZGYKEhERCQmChIREYmJgkRERGKiIBERkZgoSEREJCYKEhERiYmCREREYqIgERGRmChIREQkJgoSERGJiYJERERioiAREZGYxDVIzGyVme0ys1ozu3OM/SvNbKOZDZrZTRHb32ZmmyNevWZ2Y7jvQTPbF7Hvonh+BxERObm4PWrXzJKBe4GrgTpgvZmtdfcdEYcdBD4M/FVkWXd/BrgoPE8RUAv8MuKQz7j7Y/Gqu4iIRC+ez2xfAdS6+14AM3sEuAE4ESTuvj/cN3yS89wEPOHu3fGrqoiInKp4Nm2VA4ci1uvCbZO1GvjhqG1fNbOtZvYNM0s/1QqKiEjs4hkkNsY2n9QJzMqA5cCTEZs/BywBLgOKgM+OU/Y2M6sxs5rGxsbJfKyIiExCPIOkDqiMWK8A6id5jvcBP3X3gZEN7n7EA33Adwia0F7H3de4e7W7V5eWlk7yY0VEJFrxDJL1wGIzqzKzNIImqrWTPMctjGrWCq9SMDMDbgS2nYa6iojIKYpbkLj7IHA7QbPUTuBRd99uZneb2fUAZnaZmdUBNwPfNrPtI+XNbAHBFc2vR536+2b2EvASUAJ8JV7fQUREJhbPUVu4+zpg3ahtd0Usrydo8hqr7H7G6Jx397ef3lqKiEgsdGe7iIjEREEiIiIxUZCIiEhMFCQiIhITBYmIiMREQSIiIjFRkIiISEwUJCIiEhMFiYiIxERBIiIiMVGQiIhITBQkIiISEwWJiIjEREEiIiIxUZCIiEhMFCQiIhITBYmIiMQkrkFiZqvMbJeZ1ZrZnWPsX2lmG81s0MxuGrVvyMw2h6+1EdurzOxFM9ttZj8KnwcvIiIJErcgMbNk4F7gWmApcIuZLR112EHgw8APxjhFj7tfFL6uj9j+NeAb7r4YaAE+etorLyIiUYvnFckKoNbd97p7P/AIcEPkAe6+3923AsPRnNDMDHg78Fi46SHgxtNXZRERmax4Bkk5cChivS7cFq0MM6sxsxfMbCQsioFWdx+c6JxmdltYvqaxsXGydRcRkSilxPHcNsY2n0T5ee5eb2YLgafN7CWgPdpzuvsaYA1AdXX1ZD5XREQmIZ5XJHVAZcR6BVAfbWF3rw/f9wLPAhcDx4ECMxsJwEmdU0RETr94Bsl6YHE4yioNWA2snaAMAGZWaGbp4XIJ8CZgh7s78AwwMsLrVuDnp73mIiIStbgFSdiPcTvwJLATeNTdt5vZ3WZ2PYCZXWZmdcDNwLfNbHtY/Dygxsy2EATHPe6+I9z3WeAOM6sl6DO5P17fQUREJhbPPhLcfR2wbtS2uyKW1xM0T40u9xywfJxz7iUYESYiIlOA7mwXEZGYKEhERCQmChIREYmJgkRERGKiIBERkZgoSEREJCYKEhERiYmCREREYqIgERGRmChIREQkJgoSERGJiYJERERioiAREZGYKEhERCQmChIREYmJgkRERGKiIBERkZhEFSRm9riZvdvMJhU8ZrbKzHaZWa2Z3TnG/pVmttHMBs3spojtF5nZ82a23cy2mtn7I/Y9aGb7zGxz+LpoMnUSEZHTK9pg+Cbwx8BuM7vHzJZMVMDMkoF7gWuBpcAtZrZ01GEHgQ8DPxi1vRv4kLufD6wC/tHMCiL2f8bdLwpfm6P8DiIiEgdRBYm7/8rdPwBcAuwHnjKz58zsI2aWOk6xFUCtu+91937gEeCGUefd7+5bgeFR219x993hcj3QAJRO4nuJiMgZEnVTlZkVE1w9/BmwCfgngmB5apwi5cChiPW6cNukmNkKIA3YE7H5q2GT1zfMLH2ccreZWY2Z1TQ2Nk72Y0VEJErR9pH8BPgNkAVc5+7Xu/uP3P2TQM54xcbY5pOpnJmVAd8FPuLuI1ctnwOWAJcBRcBnxyrr7mvcvdrdq0tLdTEjIhIvKVEed5+7r4vcYGbp7t7n7tXjlKkDKiPWK4D6aCtmZnnAfwD/091fGNnu7kfCxT4z+w7wV9GeU0RETr9om7a+Msa25ycosx5YbGZVZpYGrAbWRvNh4fE/BR529x+P2lcWvhtwI7AtmnOKiEh8nPSKxMzmEPRrZJrZxbzaXJVH0Mw1LncfNLPbgSeBZOABd99uZncDNe6+1swuIwiMQuA6M/tSOFLrfcBKoNjMPhye8sPhCK3vm1lpWJfNwMcn/a1FROS0mahp650EHewVwNcjtncAfzPRycPmsHWjtt0Vsbw+PPfoct8DvjfOOd8+0eeKiMiZc9IgcfeHgIfM7L3u/vgZqpOIiEwjEzVt/Ul4dbDAzO4Yvd/dvz5GMRERmUEmatrKDt/HG+IrIiIz3ERNW98O3790ZqojIiLTzURNW/98sv3u/t9Pb3VERGS6mahpa8MZqYWIiExb0YzaEhERGddETVv/6O5/aWb/xhjzZLn79XGrmYiITAsTNW19N3z/h3hXREREpqeJmrY2hO+/Due/WkJwZbIrfMaIiIjMcFHN/mtm7wa+RfBMEAOqzOxj7v5EPCsnIiJTX7TTyP8f4G3uXgtgZucQTPGuIBERmeGinUa+YSREQnsJHn8rIiIz3ESjtv4oXNxuZuuARwn6SG4meN6IiIjMcBM1bV0XsXwM+INwuZHgGSIiIjLDTTRq6yNnqiIiIjI9RTtqKwP4KHA+kDGy3d3/NE71EhGRaSLazvbvAnMInpj4a4KnGnZMVMjMVpnZLjOrNbM7x9i/0sw2mtmgmd00at+tZrY7fN0asf1SM3spPOc/h89uFxGRBIk2SBa5++eBrnD+rXcDy09WwMySgXuBa4GlwC1mtnTUYQcJHuX7g1Fli4AvAJcDK4AvmNlIn8w3gduAxeFrVZTfQURE4iDaIBkI31vNbBmQDyyYoMwKoNbd94Z3wT8C3BB5gLvvd/etwPCosu8EnnL3ZndvAZ4CVplZGZDn7s+7uwMPAzdG+R1ERCQOor0hcU14RfB5YC3BExM/P0GZcuBQxHodwRVGNMYqWx6+6sbY/jpmdhvBlQvz5s2L8mNfr3LefOoOHTzl8qcqOSWVocGBiQ88iz47kd+5onIehw4eOKWy7k5H3yCtXQO09vTT2j1Aa88Abd39dPUP0TcwTO/ga9+H3TGg+F2f5pc7jgJgGElJkJqcRGpSEqnJRmpyEinJRlpKElmpKWSkJpGZlkxGajJJsbbqWhKJaBnWz/aZE8vP9WREFSTufl+4+GtgYZTnHusn9HUzCE+ybNTndPc1wBqA6urqaD/3deoOHeTrv9x1qsVP2R3XnJuQz03kZyf6O4+nvXeAA8e7OdDcxZHWXo6293K07dX3ho5eBoZO/iOWmmykpySTkZpEekoySUngDhnzllHX0oOHxYeGncHh4QnPB5CRGoRLTkYKuSdeqeSmp5CXGbwnJZ0kKHx4Rv5/nknf+WQ/16dTtKO2ioEvAm8i+MX9G+DL7t50kmJ1QGXEegVQH2W96oC3jir7bLi94hTPKTI+S2Lf8S52HW3nlWOd7D/exf6mLg40ddPU9dr5STNTkynLz2B2XgaXVxUxKy+Dkpw08jNTKchKozArlYKsVPIyU8lJTyE9JZnkcX6hmxn/Y4xfMO7O4LAzMDTM4JDTNzhMz8AQPf1D9A4M0T0wRG//EF39g3T2DXL8eB/d/UOvOUeSQX5mKoVZaRRmpVGQFSyX5KSRnpp8+v7byYwXbdPWI8B/Ae8N1z8A/Ah4x0nKrAcWm1kVcBhYDfxxlJ/3JPC/IjrYrwE+5+7NZtZhZlcALwIfAv5vlOcUAaB/cJiGjl4aOvo43tlHU2c/lZ9+lLf9w7MnjinLz2B+cRbXnD+b+cXZLCjOZn5xFnMLMsnLSIl7k5CZnWjaitbg0DCdfYN09A7S3jtAS/cArd1BU9uBpm6G/NWrnNyMFEr/6PM8t+c4pTnplOSmU5CZmpCmLpn+og2SInf/csT6V8zspJ3c7j5oZrcThEIy8IC7bzezu4Ead19rZpcBPyW4S/46M/uSu58fBsaXeXUalrvdvTlc/gTwIJBJMGmkJo6UcQ0NO8c7+zja3sux9l6OtffRHHGFkZWWTHFOGp2bf8G37rmLc+fksnh2Dllp0f7TmDpSkpMoyEqjICvtdfuG3enoHaS5q5+mzj4aO/toKpxDzYGWE81qaSlJzMnLCF75wXtmmq5cZGLR/mt5xsxWE8y1BXATwey/J+Xu64B1o7bdFbG8ntc2VUUe9wDwwBjba4BlUdZbZpjegSHqW3s43NpDfWsvjR19J/4Sz0xNZnZeOotn5TAnL4NZeeknAuO3n72P9132r4mselwlmZGfmUp+ZipVJdkAPHXHW/m7J3bS3NVPQ2cfx8J+n/X7m090POZnpjI7L52y/EzKCzIpyUnTVYu8zkSTNnbwaif3HcD3wl1JQCfBvR4iCdPVN3giOA639nC8M7jaSDZjdl46F1UWMDsvndl5GeSegSap6SYlOYlZeRnMystg2dx8AAaGhmloD67ijrQFgfzKsU4AMlKSKC8MQqWiMEvBIsDEc23lnqmKiESjf3CYw609HGwKRlG1dAdDKlOTjbL8TK5cmEt5QSaz89JJmUT/grwqNTkMi8JMRuZmbe8Z4HBrD3UtQWDvaewCID0lifKCTCqLsphflEVBlvpZZqKoG4LN7HpgZbj6rLv/e3yqJPKqYXcaO/o40NzNwaZujrT1MOyQnGRUFGRy/tx8ygsyKc1NH3dklMQuLzMYhXZeWR4AHb0DHG7poS4Ml73Hg2DJy0hhXnEW84uyqSzKJD1FfSwzQbTDf+8BLgO+H276lJm92d1fN3+WSKz6Boc40NTNvnAIbu9AMPFBaU46F88rZF5RFnPzM3TFkUC5GaksKUtlSRgsbT0DHAiHS79ytJNth9sxg7K8DOYXZ1NVkq1msLNYtFck7wIucvdhADN7CNgEKEjktGjrGSD30uv4ycY6DrcGVx0ZqUknht1WFmaRnT79RlLNFPmZqVxQUcAFFQUMDTtH23rZ39TFweZunt/bxPN7m8jNSGFhSTYZ8y9kaNh1BXkWmcy/zAJgZAhufhzqIjOIu3O0vZe9jV3sO95FU1c/Re/4GJ19g1xcWUhVaTZl+RmxTwMiZ1xykp3oY3kTwYCIfce72Hu8i2317cxe/VXW/GYvC4qzWFiSw4KSLDWBTXPRBsn/BjaZ2TMEI7hWAp+LW63krDTsTn1rD7UNndQ2dNLVP4QZlBdk8pa5JTzyV3/Ipx59NtHVlNMsOz2FZeX5LCvPZ2BomC9+8sO89fa/Z9/xLl451kmSQXlhJotKc1g0a3rewzPTTfh/LHzex2+BKwj6SQz4rLsfjXPd5Cww7M7hlh52N3Syp7GT7v4hkpOMBcVZLJqVw4LibDLC6Tq+13okwbWVeEtNTqKn9kWuXjqbYQ+awPYe72JPYyfP7Grk2V2NlBdm8oZZuZwzK1uhMk1M+H/J3d3MfubulxLM/CtyUsPDTl1rD7sbOtjT0EXPwBApScaCkmwWh+GRlqKO8pkuyYy5BZnMLcjkTecU09TVz+5jnbzS0MHTuxp45hWoGAmV0hzdZT+FRRv3L5jZZeGd6CKv4+4c6+hj15EOdh3rOBEeVSPhUZI9qXmjZGYxM0py0inJSeeKhUUc7+xnd0MHrxzr5D9fbuDpXQ1UFmbxhtlB85f6VKaWaIPkbcDHzWw/0EXQvOXufkG8KibTQ2t3P7uOdvDy0Q5aewZITjKqirN5w5zgykPhIZNlZpTmplOam86VC4s53tnPK8c62N3Qya92NvDMrkYWlmSzpCyX+UXZGv01BUQbJNfGtRYyrXT3D7L7WCcvH+3gaHsvEDRBXLqgkMWlOZqiXE6byFB54znFHGvv4+Vwqv/dDZ1kpibzhtk5LJmTx+y8dN2nkiATzbWVAXwcWAS8BNzv7oNnomIytQwNO/uOd7G9vo0Dzd24Q0lOGm9eVMIbZueQm5Ga6CrKWc7MglmJ8zN4y+JSDjR38fKRDrbVt7Olro2CzFSWzMllSVke+Zn6eTyTJroieYjgee2/IbgqWQp8Kt6VkqnjeGcfO+rbeflo0O+RnZ7MJfMKWTInl5Kc9ERXT2ao5CRjYUkOC0ty6BscorYhuEJ+YV8zL+xrpqIwk/Pn5rGoNEczIJwBEwXJUndfDmBm9wO/j3+VJNEsPZutda3sONLOsfY+kgwWluZwflke84qzdJOgTCnpKcmcPzef8+fm0947wMtHOthe38aT24/xbEoj587J5fy5eczKzUh0Vc9aEwXJiafVhw+qinN1JFHcnbqWHrYfaafiLx7mmV2NFOeksXJxCUvm5GnopUwLeRmprKgq4rIFhcHPc3072+vb2VrXRmluOjkXv5vegaET9y7J6TFRkFxoZu3hsgGZ4frIqK28kxU2s1XAPxE8IfE+d79n1P504GHgUqAJeL+77zezDwCfiTj0AuASd99sZs8CZUBPuO8ad2+Y4HvIOHoHhthxpJ2XDrfR2j1AWkoSXS89xZ9/7BPMylXnpUxPZkZlURaVRVn0Dgyx62gH2+vbKb7mE9z3230smpXDBeX5lOVn6Gf8NJjoeSSnHNtmlgzcC1wN1AHrzWytu++IOOyjQIu7LwqfwPg1gjD5PuFMw2a2HPi5u2+OKPeB8EmJcgrcnWPtfWw93MorxzoZGnbK8jNYsbSIxbNy+OuvfovZn/l0oqspclpkpCZzYWUBF1YWcOcH38U1f/sgu452sOtoByU5aVxQUcC5s3N1k2wM4jn/wAqg1t33ApjZI8ANQGSQ3AB8MVx+DPgXMzP3kadIA3AL8MM41nPG6B8c5pVjHWw93EZjRx+pycbSsjyWl+dTmquOczn79R/bw9uXzOLNi0rYdayDrXWtPP1yA7/dfZzzynJZXp5PsQaRTFo8g6QcOBSxXgdcPt4xYR9MG1AMHI845v0EgRPpO2Y2BDwOfGVU8MgozV39bKlr5eUjHfQPDVOck8bbzi1lyZw8/RUmM1JaShLLy/NZNjePo+29bK1rY9vhYBhxRUEmF1Tks7A0Rzc7RimeQTLW/4HRv/BPeoyZXQ50u/u2iP0fcPfDZpZLECQfJOhnee2JzW4DbgOYN2/eJKs+/bk7B5u72XSolQNN3SSbsXh2DsvVLixyglnwiOay/EzesniQHfVBf+G6bUfJTkvmgooClpfna7DJBOIZJHVAZcR6BVA/zjF1ZpZC8JyT5oj9qxnVrOXuh8P3DjP7AUET2uuCxN3XAGsAqqurZ8wVy8DQMC8f6WDzoVaau/vJSkvmiqoilpXn68FQIieRlZZC9YIiLplfyIGmbrYcauX5vU38fn8z583J5aLKAjV7jSOev1nWA4vNrAo4TBAKfzzqmLXArcDzwE3A0yPNVGaWBNzMq8+JJwybAnc/bmapwHuAX8XxO0wbHb0D4eV5G72Dw5TmpnPN0tksnp1DSpKar0SilWTBZKNVJdk0dfax+VArO48Gd9DPK8ri4nkFzC/K0lV9hLgFSdjncTvwJMHw3wfcfbuZ3Q3UuPta4H7gu2ZWS3AlsjriFCuBupHO+lA68GQYIskEIfKv8foO08Gx9l42Hmxhd0MnOCwszebiykLmFqj5SiRWxTnpXHXebN54TgkvHW5jS10rP99cT1FWGhfNK+C8Obm6c574XpHg7uuAdaO23RWx3Etw1TFW2WcJHqYVua2L4J6TGW2k/6PmQAt1LT2kJSdxcWUBF1YUkKc5hkROu8y0ZFZUFXHJ/AJ2H+tk06FgtNdze45zYUUwtDhzBt/kqEbzaWR42Nnd0MmGAy00dvaRnZ7MmxeVsKw8T89nEDkDUpKSOK8sjyVzcjnc2sPGg628uK+ZDQdaWFaezyXzCmbkBKYKkmlgYGiYHfXtbDzYQnvvIIVZqbzjvFmcOydX/R8iCWBmVBRmUVGYRVNnHxsOtLClrpWtda2cOyeX6vlFFGWnJbqaZ4yCZArrHRhiS10rWw610TMwRFl+BivfUMrCkmz1f4hMEcU56Vxz/hyuWFjMpoOtbKtvY+eRDs4pzaZ6fhFz8s/+ySIVJFNQT/8QBSs/xAO/28fAkLOgOIvq+UXqQBeZwvIyU/mDc0u5rKqQLXVtbDnUyp7GQ1QUZJIx/0Lc/az996sgmUK6+gbZdLCVrYdbybviJqqKs7msqkjP/RCZRrLSUrhyYTGXzitkW30bmw62Mnv1V/nxhjourypi3lk4dFgN7FNAV98g//VKIw8+t5+NB1tYWJpD/f3/jWuXlylERKaptJQkLplXyK1vnE/Tk/fS0TvIzzbX82hNHfubujibZnbSFUkCdfQOsOFAC9vq2xl2Z8mcXC5bUERhVhq/bKpLdPVE5DRISUqic/MT3HrP19lZ38H6A838fHM9s/PSubyqmAXF0/8KRUGSAJ19g6zf18z2+nYc57yyPKrnF1KQNXNGeYjMNClJSSyvyGfp3Dx2HGln/f5m1m4JAmVFVRFVxdN3EI2C5Azq6R+i5kAzW+racHeWluVx2YIi3UQoMoMkJxnLy/NZWpbHzjBQ/m3LEWblpvPGc4qnZR+KguQM6BscYuOBVjYdamFwKGjCunxhMfkKEJEZKznJWFaez3llebx8tJ0X9zXzs831lBdkcuU5xZQXZCa6ilFTkMTRwNAwmw+1suFAC32DwyyalcOVC4tn1I1KInJyyUnG+XPzOXdOLtsPt/P7/c08tqGOBcVZXHlOMbNyp/59KAqSOBgcGualw22s399Cz8DQtPqBEJHESElK4sLKApbOzWNLXSs1+1v44e8PsXhWDldM8T9AFSSn0dCws/NIcIna2TdIRXiJOncaXaKKSGKlJidRPb+I5eX5bDzYyqaDLdQ2dLKkLJcrqoqnZJ+qguQ0cHf2NHbxuz3Hae0eYE5eBlcvnU1lYea06zQTkakhPSWZKxcWc2FFfjiXVxuvHO1keUU+K6qKptRswwqSGNW39vDb2uMcaeulMCuV91xQprmwROS0yUpL4S2LS7m4spAX9jWx5VArO+rbqV5QyEWVBaROgeehKEhOUUtXP7/bc5w9jV1kpSVz1ZJZLC3LIylJASIip19ORgrvOG82F1cW8NyeJp7b08TWujauWFjEeWV5JCXwj1cFySR19Q3y4r5mttW3kZJkXLmwmIvnTY2/CkTk7Feck851F87lcEvQGvKrnQ1sOtjKGxcVJ+ymxrgGiZmtAv6J4LG497n7PaP2pwMPEzz1sAl4v7vvN7MFwE5gV3joC+7+8bDMpcCDQCbB0xc/5Wdg0pr+wWE2Hmxh48EWhoadC8qDdsqsNGWxiJx55YWZvK+6gtrGTp6rbeLfthyhvCCTNy8qOeNT18ftt6CZJQP3AlcDdcB6M1vr7jsiDvso0OLui8xsNfA14P3hvj3uftEYp/4mcBvwAkGQrAKeiNPXgKRkttYFT0Hr7h9i8awcrjynmEJNZyIiCWZmLJ6Vy8KSHLbVt/Hi3mZ+VHOIRbNyeNM5xWesHvH8c3oFUOvuewHM7BHgBiAySG4AvhguPwb8i53kuszMyoA8d38+XH8YuJE4Bckvth1l7p/eyzO7GplbkMF7LiijLF9DeUVkaklOMi6sKOC8OXlsONjCpoMt7G3spPCqP6e1uz/u8/jFM0jKgUMR63XA5eMd4+6DZtYGjMRolZltAtqB/+nuvwmPj5wWty7cFhc/23QY92Guu6CMKo3EEpEpLi0liSsXFnNBeT4v7G1i64WraOkeiHuQxLOHeKzfuqP7MsY75ggwz90vBu4AfmBmeVGeMzix2W1mVmNmNY2NjZOo9qvuee9yjjxwOwtLcxQiIjJtZKencNV5szn8rT+lqiQ77p8XzyCpAyoj1iuA+vGOMbMUIB9odvc+d28CcPcNwB7gDeHxFROck7DcGnevdvfq0tLSU/oCBVlp4MOnVFZEJNGGu9vOyOfEM0jWA4tskDuIAAAI3ElEQVTNrMrM0oDVwNpRx6wFbg2XbwKednc3s9Kwsx4zWwgsBva6+xGgw8yuCPtSPgT8PI7fQUREJhC3PpKwz+N24EmC4b8PuPt2M7sbqHH3tcD9wHfNrBZoJggbgJXA3WY2CAwBH3f35nDfJ3h1+O8TxHPEloiITCiuN0G4+zqCIbqR2+6KWO4Fbh6j3OPA4+OcswZYdnprKiIip0q3Y4uISEwUJCIiEhMFiYiIxERBIiIiMVGQiIhITBQkIiISEwWJiIjEREEiIiIxUZCIiEhMFCQiIhITBYmIiMREQSIiIjFRkIiISEwUJCIiEhMFiYiIxERBIiIiMVGQiIhITOIaJGa2ysx2mVmtmd05xv50M/tRuP9FM1sQbr/azDaY2Uvh+9sjyjwbnnNz+JoVz+8gIiInF7dH7ZpZMnAvcDVQB6w3s7XuviPisI8CLe6+yMxWA18D3g8cB65z93ozW0bw3PfyiHIfCB+5KyIiCRbPK5IVQK2773X3fuAR4IZRx9wAPBQuPwZcZWbm7pvcvT7cvh3IMLP0ONZVREROUTyDpBw4FLFex2uvKl5zjLsPAm1A8ahj3gtscve+iG3fCZu1Pm9mdnqrLSIikxHPIBnrF7xP5hgzO5+guetjEfs/4O7LgbeErw+O+eFmt5lZjZnVNDY2TqriIiISvXgGSR1QGbFeAdSPd4yZpQD5QHO4XgH8FPiQu+8ZKeDuh8P3DuAHBE1or+Pua9y92t2rS0tLT8sXEhGR14tnkKwHFptZlZmlAauBtaOOWQvcGi7fBDzt7m5mBcB/AJ9z99+NHGxmKWZWEi6nAu8BtsXxO4iIyATiFiRhn8ftBCOudgKPuvt2M7vbzK4PD7sfKDazWuAOYGSI8O3AIuDzo4b5pgNPmtlWYDNwGPjXeH0HERGZWNyG/wK4+zpg3ahtd0Us9wI3j1HuK8BXxjntpaezjiIiEhvd2S4iIjFRkIiISEwUJCIiEhMFiYiIxERBIiIiMVGQiIhITBQkIiISEwWJiIjEREEiIiIxUZCIiEhMFCQiIhITBYmIiMREQSIiIjFRkIiISEwUJCIiEhMFiYiIxERBIiIiMYlrkJjZKjPbZWa1ZnbnGPvTzexH4f4XzWxBxL7Phdt3mdk7oz2niIicWXELEjNLBu4FrgWWAreY2dJRh30UaHH3RcA3gK+FZZcCq4HzgVXA/zOz5CjPKSIiZ1A8r0hWALXuvtfd+4FHgBtGHXMD8FC4/BhwlZlZuP0Rd+9z931AbXi+aM4pIiJnUDyDpBw4FLFeF24b8xh3HwTagOKTlI3mnCIicgaZu8fnxGY3A+909z8L1z8IrHD3T0Ycsz08pi5c30Nw1XE38Ly7fy/cfj+wjiD4TnrOiHPfBtwWrp4L7DrFr1ICHD/FsvGkek2O6jU5qtfknK31mu/upRMdlBLDB0ykDqiMWK8A6sc5ps7MUoB8oHmCshOdEwB3XwOsOdXKjzCzGnevjvU8p5vqNTmq1+SoXpMz0+sVz6at9cBiM6syszSCzvO1o45ZC9waLt8EPO3BJdJaYHU4qqsKWAz8PspziojIGRS3KxJ3HzSz24EngWTgAXffbmZ3AzXuvha4H/iumdUSXImsDstuN7NHgR3AIPAX7j4EMNY54/UdRERkYvFs2sLd1xH0bURuuytiuRe4eZyyXwW+Gs054yzm5rE4Ub0mR/WaHNVrcmZ0veLW2S4iIjODpkgREZGYKEjGYWYPmFmDmW1LdF0imVmlmT1jZjvNbLuZfSrRdQIwswwz+72ZbQnr9aVE12lEOCvCJjP790TXJZKZ7Tezl8xss5nVJLo+I8yswMweM7OXw5+zK6dAnc4N/zuNvNrN7C8TXS8AM/t0+DO/zcx+aGYZia4TgJl9KqzT9nj/t1LT1jjMbCXQCTzs7ssSXZ8RZlYGlLn7RjPLBTYAN7r7jgTXy4Bsd+80s1Tgt8Cn3P2FRNYLwMzuAKqBPHd/T6LrM8LM9gPV7j6l7j8ws4eA37j7feHoyCx3b010vUaEUyUdBi539wMJrks5wc/6UnfvCQcJrXP3BxNcr2UEM3+sAPqBXwCfcPfd8fg8XZGMw93/i2Ak2ZTi7kfcfWO43AHsZArc3e+BznA1NXwl/K8UM6sA3g3cl+i6TAdmlgesJBhRibv3T6UQCV0F7El0iERIATLDe+GyGOfetjPsPOAFd+8OZw35NfCH8fowBck0Fs6WfDHwYmJrEgibkDYDDcBT7j4V6vWPwF8Dw4muyBgc+KWZbQhnYpgKFgKNwHfC5sD7zCw70ZUaZTXww0RXAsDdDwP/ABwEjgBt7v7LxNYKgG3ASjMrNrMs4F289mbu00pBMk2ZWQ7wOPCX7t6e6PoAuPuQu19EMOPAivDyOmHM7D1Ag7tvSGQ9TuJN7n4JwWzWfxE2pyZaCnAJ8E13vxjoAqbM4xrCprbrgR8nui4AZlZIMHFsFTAXyDazP0lsrcDddxLMpv4UQbPWFoJ78uJCQTINhX0QjwPfd/efJLo+o4VNIc8SPAIgkd4EXB/2RTwCvN3MvpfYKr3K3evD9wbgpwTt2YlWB9RFXE0+RhAsU8W1wEZ3P5boioTeAexz90Z3HwB+ArwxwXUCwN3vd/dL3H0lQTN9XPpHQEEy7YSd2vcDO93964muzwgzKzWzgnA5k+Af2MuJrJO7f87dK9x9AUFzyNPunvC/FgHMLDscLEHYdHQNQXNEQrn7UeCQmZ0bbrqKYIaJqeIWpkizVuggcIWZZYX/Nq8i6LdMODObFb7PA/6IOP53i+ud7dOZmf0QeCtQYmZ1wBfc/f7E1goI/sr+IPBS2B8B8DfhHf+JVAY8FI6oSQIedfcpNdx2ipkN/DT43UMK8AN3/0Viq3TCJ4Hvh81Ie4GPJLg+AIRt/VcDH0t0XUa4+4tm9hiwkaDpaBNT5y73x82sGBggmGaqJV4fpOG/IiISEzVtiYhITBQkIiISEwWJiIjEREEiIiIxUZCIiEhMFCQiIhITBYmIiMREQSIiIjH5/2mkD3OuXKbyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt \n",
    "import seaborn as sns\n",
    "\n",
    "hist_data = [1,2,3,4,5,5,6,7,8,9]\n",
    "\n",
    "plt.figure()\n",
    "sns.histplot(hist_data, bins=10, stat='probability', kde=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now you can see that by setting more bins, the visualisation looks more detailed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next type of plot is scatter plot, where data points are represented by dots. This time we use the `scatter()` method from `pyplot` to visualise them. Using the previous example, we have "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEERJREFUeJzt3V1sZVd5xvH/W48pZwLIiJiqdpIOSJULyqgYHUWUqJGa0JovhemoF0GCC/oxvaCQIGSEe8PlqDJCILVCGiXQVqRBbTBWRduYSNAiJIjqidM6YbBK+QhzHBqj9vB52njM2wvbk/FkMt5uz/Y+a87/J1ljr9k+frRlP9pee22vyEwkSeX4uaYDSJIOxuKWpMJY3JJUGItbkgpjcUtSYSxuSSqMxS1JhbG4JakwFrckFeZIHS96/fXX57Fjx+p4aUm6Jp09e/b7mTle5dhaivvYsWMsLy/X8dKSdE2KiO9UPdapEkkqjMUtSYWxuCWpMBa3JBXG4pakwlQq7oi4OyIej4gnIuKeukNJkp7fvssBI+Jm4A+AW4BngIci4u8y89/qDidJJVhc6TC/tMZ6t8fEWIvZmSlOTE/W9vWqXHG/CvhqZv40My8A/wT8dm2JJKkgiysd5hZW6XR7JNDp9phbWGVxpVPb16xS3I8Dt0XEyyLiKPBm4MbaEklSQeaX1uhtbu0Z621uMb+0VtvX3HeqJDPPRcSfAA8DPwb+Bbhw+XERcQo4BXDTTTf1OaYkDab1bu9A4/1Q6eZkZt6Xma/NzNuA/wSeM7+dmWcys52Z7fHxSo/bS1LxJsZaBxrvh6qrSl6+8+9NwEnggdoSSVJBZmemaI2O7BlrjY4wOzNV29es+kemPhMRLwM2gXdn5n/VlkiSCrK7euQwV5VUKu7M/PXaEkhS4U5MT9Za1JfzyUlJKozFLUmFsbglqTAWtyQVxuKWpMJY3JJUGItbkgpjcUtSYSxuSSqMxS1JhbG4JakwFrckFcbilqTCWNySVBiLW5IKY3FLUmEqbaQQEe8Dfh9IYBV4V2b+d53BJGk/iyudQ915ZlDse8UdEZPAe4F2Zt4MjAB31R1Mkq5mcaXD3MIqnW6PBDrdHnMLqyyudJqOVruqUyVHgFZEHAGOAuv1RZKk/c0vrdHb3Noz1tvcYn5praFEh2ff4s7MDvBh4EngKeAHmfn5y4+LiFMRsRwRyxsbG/1PKkmXWO/2DjR+LakyVfJS4G3AK4AJ4LqIeMflx2XmmcxsZ2Z7fHy8/0kl6RITY60DjV9LqkyVvAH4VmZuZOYmsAC8vt5YknR1szNTtEZH9oy1RkeYnZlqKNHhqbKq5EngdRFxFOgBdwDLtaaSpH3srh4ZxlUl+xZ3Zj4SEQ8CjwIXgBXgTN3BJGk/J6Ynh6KoL1dpHXdmfgj4UM1ZJEkV+OSkJBXG4pakwljcklQYi1uSCmNxS1JhLG5JKozFLUmFsbglqTAWtyQVxuKWpMJY3JJUGItbkgpjcUtSYSxuSSqMxS1Jhamy5+RURDx2ydsPI+KewwgnSXquKjvgrAGvAYiIEaADfLbmXJIG2OJKZyi3DBsUlXbAucQdwL9n5nfqCCNp8C2udJhbWKW3uQVAp9tjbmEVwPI+JAed474LeKCOIJLKML+0drG0d/U2t5hfWmso0fCpXNwR8QLgTuBvnuf/T0XEckQsb2xs9CufpAGz3u0daFz9d5Ar7jcBj2bmf1zpPzPzTGa2M7M9Pj7en3SSBs7EWOtA4+q/gxT323GaRBp6szNTtEZH9oy1RkeYnZlqKNHwqXRzMiKOAr8J/GG9cSQNut0bkK4qaU6l4s7MnwIvqzmLpEKcmJ60qBvkk5OSVBiLW5IKY3FLUmEsbkkqjMUtSYWxuCWpMBa3JBXG4pakwljcklQYi1uSCmNxS1JhLG5JKozFLUmFsbglqTAWtyQVxuKWpMJU3QFnDLgXuBlI4Hcz8yt1BpN0ZYsrHXefGXKVihv4GPBQZv7Ozm7vR2vMJOl5LK50mFtYpbe5BUCn22NuYRXA8h4i+06VRMRLgNuA+wAy85nM7NYdTNJzzS+tXSztXb3NLeaX1hpKpCZUmeN+JbABfDIiViLi3oi47vKDIuJURCxHxPLGxkbfg0qC9W7vQOO6NlUp7iPAa4GPZ+Y08BPgg5cflJlnMrOdme3x8fE+x5QEMDHWOtC4rk1Vivs8cD4zH9n5+EG2i1zSIZudmaI1OrJnrDU6wuzMVEOJ1IR9izszvwd8NyJ2vzPuAL5WaypJV3RiepLTJ48zOdYigMmxFqdPHvfG5JCpuqrkPcD9OytKvgm8q75Ikq7mxPSkRT3kKhV3Zj4GtGvOIkmqwCcnJakwFrckFcbilqTCWNySVBiLW5IKY3FLUmEsbkkqjMUtSYWxuCWpMBa3JBXG4pakwljcklQYi1uSCmNxS1JhLG5JKkylv8cdEd8GfgRsARcy07/NLUkNqboDDsBvZOb3a0siPY/FlQ7zS2usd3tMjLWYnZlqZAeYQckhHaS4pUO3uNJhbmGV3uYWAJ1uj7mFVYBDLc1BySFB9TnuBD4fEWcj4lSdgaRLzS+tXSzLXb3NLeaX1oYyhwTVr7hvzcz1iHg58HBEfD0zv3TpATuFfgrgpptu6nNMDav1bu9A49d6DgkqXnFn5vrOv08DnwVuucIxZzKznZnt8fHx/qbU0JoYax1o/FrPIUGF4o6I6yLixbvvA78FPF53MAlgdmaK1ujInrHW6AizM1NDmUOCalMlvwB8NiJ2j/+rzHyo1lTSjt0bf02v5hiUHBJAZGbfX7Tdbufy8nLfX1eSrlURcbbqMzI+OSlJhbG4JakwFrckFcbilqTCWNySVBiLW5IKY3FLUmEsbkkqjMUtSYWxuCWpMBa3JBXG4pakwljcklQYi1uSCmNxS1JhLG5JKkzl4o6IkYhYiYjP1RlIknR1VXd5B7gbOAe8pKYsGjCLKx236pIGUKUr7oi4AXgLcG+9cTQoFlc6zC2s0un2SKDT7TG3sMriSqfpaNLQqzpV8lHgA8DPasyiATK/tEZvc2vPWG9zi/mltYYSSdq1b3FHxFuBpzPz7D7HnYqI5YhY3tjY6FtANWO92zvQuKTDU+WK+1bgzoj4NvBp4PaI+NTlB2XmmcxsZ2Z7fHy8zzF12CbGWgcal3R49i3uzJzLzBsy8xhwF/CFzHxH7cnUqNmZKVqjI3vGWqMjzM5MNZRI0q6DrCrRENldPeKqEmnwRGb2/UXb7XYuLy/3/XUl6VoVEWczs13lWJ+clKTCWNySVBiLW5IKY3FLUmEsbkkqjMUtSYWxuCWpMBa3JBXG4pakwljcklQYi1uSCmNxS1JhLG5JKozFLUmFsbglqTAWtyQVZt8dcCLihcCXgJ/fOf7BzPxQ3cGG3eJKx91nJF1Rla3L/ge4PTN/HBGjwJcj4h8y86s1Zxtaiysd5hZW6W1uAdDp9phbWAWwvCVV2iw4M/PHOx+O7rz1f78zXTS/tHaxtHf1NreYX1prKJGkQVJpjjsiRiLiMeBp4OHMfOQKx5yKiOWIWN7Y2Oh3zqGy3u0daFzScKlU3Jm5lZmvAW4AbomIm69wzJnMbGdme3x8vN85h8rEWOtA45KGy4FWlWRmF/hH4I21pBEAszNTtEZH9oy1RkeYnZlqKJGkQbJvcUfEeESM7bzfAt4AfL3uYMPsxPQkp08eZ3KsRQCTYy1OnzzujUlJQLVVJb8I/EVEjLBd9H+dmZ+rN5ZOTE9a1JKuaN/izsx/BaYPIYskqQKfnJSkwljcklQYi1uSCmNxS1JhLG5JKozFLUmFsbglqTAWtyQVxuKWpMJY3JJUGItbkgpjcUtSYSxuSSqMxS1JhbG4JakwVXbAuTEivhgR5yLiiYi4+zCCSZKurMoOOBeA92fmoxHxYuBsRDycmV+rOVsjFlc6zC+tsd7tMTHWYnZmyp1oJA2UKjvgPAU8tfP+jyLiHDAJXHPFvbjSYW5hld7mFgCdbo+5hVUAy1vSwDjQHHdEHGN7G7NH6gjTtPmltYulvau3ucX80lpDiSTpuSoXd0S8CPgMcE9m/vAK/38qIpYjYnljY6OfGQ/Nerd3oHFJakKl4o6IUbZL+/7MXLjSMZl5JjPbmdkeHx/vZ8ZDMzHWOtC4JDWhyqqSAO4DzmXmR+qP1JzZmSlaoyN7xlqjI8zOTDWUSJKeq8oV963AO4HbI+Kxnbc315yrESemJzl98jiTYy0CmBxrcfrkcW9MShooVVaVfBmIQ8gyEE5MT1rUkgaaT05KUmEsbkkqjMUtSYWxuCWpMBa3JBXG4pakwljcklQYi1uSCmNxS1JhLG5JKozFLUmFsbglqTAWtyQVxuKWpMJY3JJUGItbkgpTZeuyT0TE0xHx+GEEkiRd3b474AB/Dvwp8Jd1Bllc6TC/tMZ6t8fEWIvZmSl3opGkK6iyddmXIuJYnSEWVzrMLazS29wCoNPtMbewCmB5S9JlBmKOe35p7WJp7+ptbjG/tNZQIkkaXH0r7og4FRHLEbG8sbFxoM9d7/YONC5Jw6xvxZ2ZZzKznZnt8fHxA33uxFjrQOOSNMwGYqpkdmaK1ujInrHW6AizM1MNJZKkwVVlOeADwFeAqYg4HxG/1+8QJ6YnOX3yOJNjLQKYHGtx+uRxb0xK0hVUWVXy9sMIcmJ60qKWpAoGYqpEklSdxS1JhbG4JakwFrckFcbilqTCRGb2/0UjNoDv/B8//Xrg+32MUzrPx7M8F3t5Pp51LZyLX8rMSk8v1lLc/x8RsZyZ7aZzDArPx7M8F3t5Pp41bOfCqRJJKozFLUmFGcTiPtN0gAHj+XiW52Ivz8ezhupcDNwctyTp6gbxiluSdBUDVdwR8caIWIuIb0TEB5vO05SIuDEivhgR5yLiiYi4u+lMTYuIkYhYiYjPNZ2laRExFhEPRsTXd75Hfq3pTE2KiPft/Jw8HhEPRMQLm85Ut4Ep7ogYAf4MeBPwauDtEfHqZlM15gLw/sx8FfA64N1DfC523Q2cazrEgPgY8FBm/grwqwzxeYmISeC9QDszbwZGgLuaTVW/gSlu4BbgG5n5zcx8Bvg08LaGMzUiM5/KzEd33v8R2z+YQ/s3byPiBuAtwL1NZ2laRLwEuA24DyAzn8nMbrOpGncEaEXEEeAosN5wntoNUnFPAt+95OPzDHFZ7YqIY8A08EizSRr1UeADwM+aDjIAXglsAJ/cmTq6NyKuazpUUzKzA3wYeBJ4CvhBZn6+2VT1G6TijiuMDfWSl4h4EfAZ4J7M/GHTeZoQEW8Fns7Ms01nGRBHgNcCH8/MaeAnwDDfD3op27+ZvwKYAK6LiHc0m6p+g1Tc54EbL/n4BobgV57nExGjbJf2/Zm50HSeBt0K3BkR32Z7+uz2iPhUs5EadR44n5m7v4E9yHaRD6s3AN/KzI3M3AQWgNc3nKl2g1Tc/wz8ckS8IiJewPYNhr9tOFMjIiLYnsM8l5kfaTpPkzJzLjNvyMxjbH9PfCEzr/krqueTmd8DvhsRuztp3wF8rcFITXsSeF1EHN35ubmDIbhZu++ek4clMy9ExB8BS2zfGf5EZj7RcKym3Aq8E1iNiMd2xv44M/++wUwaHO8B7t+5wPkm8K6G8zQmMx+JiAeBR9lejbXCEDxF6ZOTklSYQZoqkSRVYHFLUmEsbkkqjMUtSYWxuCWpMBa3JBXG4pakwljcklSY/wXAxsYPB6KnKQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt \n",
    "import seaborn as sns\n",
    "\n",
    "hist_data = [1,2,3,4,5,5,6,7,8,9]\n",
    "\n",
    "plt.figure()\n",
    "plt.scatter(np.arange(len(hist_data)), hist_data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you carefully inspect the code above, you can see the `hist_data` spans on the $y$-axis. The value of $x$-axis is not granted but you have to specify what each $x$ values would be. If in this occassion that the data represents order, then we can use `numpy` to generate a sequence of integer array as the $x$-axis values, using `np.arange()`. \n",
    "\n",
    "Usually when we know the $x$ values then they should be stored in a separate list or array from the $y$ data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEKhJREFUeJzt3X9snHd9wPH3B8cV1wLzRMxEnIR0UrFAVJrRqXSr1LGVzoWhNEPT1kqdtAmtE6KMX/JUb9V+dEgwPG37p5vWtQw2oFUB40Wsw6BR9ktLF6cGTFKsZQUa22wNDMMYt9X1PvvD5+AYp75rzn3uvn6/pKh+vn5695F1eff8fe56kZlIksrynKoHkCR1nnGXpAIZd0kqkHGXpAIZd0kqkHGXpAIZd0kqkHGXpAIZd0kq0J6q7njv3r156NChqu5eknrSiRMnvp6Zg9udV1ncDx06xMzMTFV3L0k9KSK+2sp5bstIUoGMuyQVyLhLUoGMuyQVyLhLUoGMuyQVyLhLUoGMuyQVyLhLUoEqe4eq1G2mZheZmJ5nabnBvoEaY6PDHBkZqnos6Rkx7hJrYR+fnKOxsgrA4nKD8ck5AAOvnuS2jARMTM+fC/u6xsoqE9PzFU0kXRzjLgFLy4221qVuZ9wlYN9Ara11qdsZdwkYGx2m1t933lqtv4+x0eGKJpIujhdUJb530dRXy6gUxl1qOjIyZMxVDLdlJKlAxl2SCmTcJalAxl2SCmTcJalAxl2SCmTcJalAxl2SCmTcJalAxl2SCmTcJalAxl2SCmTcJalAxl2SCmTcJalALcU9Im6IiPmIOB0Rt2/x/YMR8VBEzEbEFyLidZ0fVZK2NzW7yDXv+QyX3/7XXPOezzA1u1j1SJXYNu4R0QfcBbwWeDlwc0S8fNNpdwAPZOYIcBPwx50eVJK2MzW7yPjkHIvLDRJYXG4wPjm3KwPfyjP3q4DTmflYZj4J3A/cuOmcBF7Q/PoHgKXOjShJrZmYnqexsnreWmNllYnp+Yomqk4rcR8Czmw4XmiubfTbwC0RsQA8CLxlqxuKiFsjYiYiZs6ePfsMxpWkC1tabrS1XrJW4h5brOWm45uB92fmfuB1wF9GxPfddmbenZn1zKwPDg62P60kPY19A7W21kvWStwXgAMbjvfz/dsubwQeAMjMfwaeC+ztxICS1Kqx0WFq/X3nrdX6+xgbHa5oouq0EvfjwBURcXlEXMLaBdOjm855HLgOICJexlrc3XeR9Kw6MjLEu99wJUMDNQIYGqjx7jdcyZGRzTvJ5duz3QmZ+VRE3AZMA33A+zLzZETcCcxk5lHgncCfRcTbWduy+cXM3Lx1I0k77sjI0K6M+Wbbxh0gMx9k7ULpxrXf3PD1KeCazo4mSXqmfIeqJBXIuEtSgYy7JBXIuEtSgYy7JBXIuEtSgYy7JBXIuEtSgYy7JBXIuEtSgYy7JBXIuEtSgYy7JBXIuEtSgYy7JBWopf+fu7QbTM0uMjE9z9Jyg30DNcZGh/3QB/Us4y6xFvbxyTkaK6sALC43GJ+cAzDw6kluy0jAxPT8ubCva6ysMjE9X9FE0sUx7hKwtNxoa13qdsZdAvYN1Npal7qdcZeAsdFhav19563V+vsYGx2uaCLp4nhBVeJ7F019tYxKYdylpiMjQ8ZcxXBbRpIKZNwlqUDGXZIKZNwlqUDGXZIKZNwlqUDGXZIKZNwlqUDGXZIKZNwlqUDGXZIKZNwlqUDGXZIKZNwlqUDGXZIKZNwlqUAtxT0iboiI+Yg4HRG3X+Ccn4uIUxFxMiI+3NkxJUnt2PaTmCKiD7gLuB5YAI5HxNHMPLXhnCuAceCazPxmRLxopwaWJG2vlWfuVwGnM/OxzHwSuB+4cdM5vwzclZnfBMjMJzo7piSpHa3EfQg4s+F4obm20UuBl0bEP0XEsYi4YasbiohbI2ImImbOnj37zCaWJG2rlbjHFmu56XgPcAXwauBm4J6IGPi+fynz7sysZ2Z9cHCw3VklSS1qJe4LwIENx/uBpS3O+avMXMnMLwPzrMVeklSBVuJ+HLgiIi6PiEuAm4Cjm86ZAn4CICL2srZN81gnB5UktW7buGfmU8BtwDTwKPBAZp6MiDsj4nDztGngGxFxCngIGMvMb+zU0JKkpxeZm7fPnx31ej1nZmYquW9J6lURcSIz69ud5ztUJalAxl2SCmTcJalAxl2SCmTcJalAxl2SCmTcJalAxl2SCmTcJalAxl2SCmTcJalAxl2SCmTcJalAxl2SCmTcJalAe6oeoBRTs4tMTM+ztNxg30CNsdFhjoxs/hxxSXp2GPcOmJpdZHxyjsbKKgCLyw3GJ+cADLykSrgt0wET0/Pnwr6usbLKxPR8RRNJ2u2MewcsLTfaWpeknWbcO2DfQK2tdUnaaca9A8ZGh6n19523VuvvY2x0uKKJJO12XlDtgPWLpr5aRlK3MO4dcmRkyJhL6hpuy0hSgYy7JBXIuEtSgYy7JBXIuEtSgYy7JBXIuEtSgYy7JBXIuEtSgYy7JBXIuEtSgYy7JBXIuEtSgYy7JBXIuEtSgVqKe0TcEBHzEXE6Im5/mvN+NiIyIuqdG1GS1K5tP6wjIvqAu4DrgQXgeEQczcxTm857PvCrwMM7Mai0W90xNcd9D59hNZO+CG5+1QHedeTKqsdSl2vlmftVwOnMfCwznwTuB27c4rzfBd4L/E8H55N2tTum5vjgscdZzQRgNZMPHnucO6bmKp5M3a6VuA8BZzYcLzTXzomIEeBAZn6ig7NJu959D59pa11a10rcY4u1PPfNiOcAfwi8c9sbirg1ImYiYubs2bOtTyntUuvP2Ftdl9a1EvcF4MCG4/3A0obj5wOvAD4bEV8BrgaObnVRNTPvzsx6ZtYHBwef+dTSLtEXWz23uvC6tK6VuB8HroiIyyPiEuAm4Oj6NzPzW5m5NzMPZeYh4BhwODNndmRiaRe5+VUH2lqX1m0b98x8CrgNmAYeBR7IzJMRcWdEHN7pAaXd7F1HruSWqw+ee6beF8EtVx/01TLaVmRFe3f1ej1nZnxyL0ntiIgTmbnte4l8h6okFci4S1KBjLskFci4S1KBjLskFci4S1KBjLskFci4S1KBjLskFci4S1KBjLskFci4S1KBjLskFci4S1KBjLskFci4S1KB9lQ9gHbW1OwiE9PzLC032DdQY2x0mCMjQ1WPpQL42Opuxr1gU7OLjE/O0VhZBWBxucH45ByAfwl1UXxsdT+3ZQo2MT1/7i/fusbKKhPT8xVNpFL42Op+xr1gS8uNttalVvnY6n7GvWD7BmptrUut8rHV/Yx7wcZGh6n19523VuvvY2x0uKKJVAofW93PC6oFW7+w5Ssa1Gk+trpfZGYld1yv13NmZqaS+5akXhURJzKzvt15bstIUoGMuyQVyLhLUoGMuyQVyLhLUoGMuyQVyLhLUoGMuyQVyLhLUoGMuyQVyLhLUoGMuyQVyLhLUoGMuyQVyLhLUoFaintE3BAR8xFxOiJu3+L774iIUxHxhYj424h4SedHlSS1attPYoqIPuAu4HpgATgeEUcz89SG02aBemZ+NyLeBLwX+PmdGFhlmJpd9FN8pB3UyjP3q4DTmflYZj4J3A/cuPGEzHwoM7/bPDwG7O/smCrJ1Owi45NzLC43SGBxucH45BxTs4tVjyYVo5W4DwFnNhwvNNcu5I3A31zMUCrbxPQ8jZXV89YaK6tMTM9XNJFUnlY+IDu2WNvyg1cj4hagDvz4Bb5/K3ArwMGDB1scUaVZWm60tS6pfa08c18ADmw43g8sbT4pIl4D/AZwODP/d6sbysy7M7OemfXBwcFnMq8KsG+g1ta6pPa1EvfjwBURcXlEXALcBBzdeEJEjAB/ylrYn+j8mCrJ2Ogwtf6+89Zq/X2MjQ5XNJFUnm23ZTLzqYi4DZgG+oD3ZebJiLgTmMnMo8AE8DzgIxEB8HhmHt7BudXD1l8V46tlpJ0TmVtun++4er2eMzMzldy3JPWqiDiRmfXtzvMdqpJUIOMuSQUy7pJUIOMuSQUy7pJUIOMuSQUy7pJUIOMuSQUy7pJUIOMuSQUy7pJUIOMuSQUy7pJUIOMuSQUy7pJUIOMuSQVq5QOyu8rU7KKf4COp5zzb7eqpuE/NLjI+OUdjZRWAxeUG45NzAAZeUteqol09tS0zMT1/7oezrrGyysT0fEUTSdL2qmhXT8V9abnR1rokdYMq2tVTcd83UGtrXZK6QRXt6qm4j40OU+vvO2+t1t/H2OhwRRNJ0vaqaFdPXVBdv/Dgq2Uk9ZIq2hWZuWM3/nTq9XrOzMxUct+S1Ksi4kRm1rc7r6e2ZSRJrTHuklQg4y5JBTLuklQg4y5JBTLuklQg4y5JBTLuklSgyt7EFBFnga9exE3sBb7eoXE6ybna041zdeNM4FztKnWul2Tm4HYnVRb3ixURM628S+vZ5lzt6ca5unEmcK527fa53JaRpAIZd0kqUC/H/e6qB7gA52pPN87VjTOBc7VrV8/Vs3vukqQL6+Vn7pKkC+i5uEfE+yLiiYj4YtWzrIuIAxHxUEQ8GhEnI+KtVc8EEBHPjYh/iYjPN+f6napn2igi+iJiNiI+UfUs6yLiKxExFxGfi4iu+cCBiBiIiI9GxJeaj7Mf7YKZhps/p/U/346It1U9F0BEvL35mP9iRNwXEc+teiaAiHhrc6aTO/2z6rltmYi4FvgO8BeZ+Yqq5wGIiBcDL87MRyLi+cAJ4Ehmnqp4rgAuy8zvREQ/8I/AWzPzWJVzrYuIdwB14AWZ+fqq54G1uAP1zOyq10dHxAeAf8jMeyLiEuDSzFyueq51EdEHLAKvysyLef9KJ2YZYu2x/vLMbETEA8CDmfn+iud6BXA/cBXwJPBJ4E2Z+a87cX8998w9M/8e+M+q59goM7+WmY80v/4v4FGg8s/+yzXfaR72N/90xX/NI2I/8NPAPVXP0u0i4gXAtcC9AJn5ZDeFvek64N+qDvsGe4BaROwBLgWWKp4H4GXAscz8bmY+Bfwd8DM7dWc9F/duFxGHgBHg4WonWdPc+vgc8ATw6czsirmAPwJ+Dfi/qgfZJIFPRcSJiLi16mGafhg4C/x5cxvrnoi4rOqhNrkJuK/qIQAycxH4feBx4GvAtzLzU9VOBcAXgWsj4oURcSnwOuDATt2Zce+giHge8DHgbZn57arnAcjM1cz8EWA/cFXzV8NKRcTrgScy80TVs2zhmsx8JfBa4M3NbcCq7QFeCfxJZo4A/w3cXu1I39PcJjoMfKTqWQAi4geBG4HLgX3AZRFxS7VTQWY+Cvwe8GnWtmQ+Dzy1U/dn3Dukuaf9MeBDmTlZ9TybNX+N/yxwQ8WjAFwDHG7ub98P/GREfLDakdZk5lLzn08AH2dtf7RqC8DCht+6Pspa7LvFa4FHMvM/qh6k6TXAlzPzbGauAJPAj1U8EwCZeW9mvjIzr2Vte3lH9tvBuHdE88LlvcCjmfkHVc+zLiIGI2Kg+XWNtQf9l6qdCjJzPDP3Z+Yh1n6d/0xmVv7MKiIua14Qp7nt8VOs/Spdqcz8d+BMRAw3l64DKr1Yv8nNdMmWTNPjwNURcWnz7+Z1rF0Hq1xEvKj5z4PAG9jBn9uenbrhnRIR9wGvBvZGxALwW5l5b7VTcQ3wC8Bcc38b4Ncz88EKZwJ4MfCB5isZngM8kJld87LDLvRDwMfXesAe4MOZ+clqRzrnLcCHmlsgjwG/VPE8ADT3jq8HfqXqWdZl5sMR8VHgEda2PWbpnnerfiwiXgisAG/OzG/u1B313EshJUnbc1tGkgpk3CWpQMZdkgpk3CWpQMZdkgpk3CWpQMZdkgpk3CWpQP8PFE8Wr7d8b8cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt \n",
    "import seaborn as sns\n",
    "\n",
    "scatter_x = [1,2,3,4,5,5,6,6,7,8,9]\n",
    "scatter_y = [0.1,0.5,0.3,0.2,0.7,0.9,0.4, 0.4,0.3,0.8,0.1]\n",
    "\n",
    "plt.figure()\n",
    "plt.scatter(scatter_x, scatter_y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have explored most of the data set. We will start find the correlations between different datasets. \n",
    "```python\n",
    "import seaborn as sns\n",
    "sns.set(style=\"ticks\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "sns.pairplot(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation: Data Cleaning and Other Tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Some of contents come from https://github.com/DataScienceUB/introduction-datascience-python-book/blob/master/ch03_Descriptive_Statistics.ipynb.*\n",
    "\n",
    "After EDA has proceed, we will realise our dataset has many issues. This then involves data cleaning means to find out the outliers, for example \n",
    "* Which data points has value too big?\n",
    "* Which data points are ilogical?\n",
    "* How many missing data do we have?\n",
    "\n",
    "The most common steps are:\n",
    "\n",
    "* __Sample the data__ We will need to extract part of the dataset to train our model, often this has to be done where there is a large dataset. \n",
    "* __Impute missing data__ It is quite common that some of the input records are incomplete in the sense that certain fields are missing or have input error. In a typical tabular data format, we need to validate each record contains the same number of fields and each field contains the data type we expect. In case the record has some fields missing, we have the following choices:\n",
    "    * Remove the row if it is imposible to impute them. \n",
    "    * Infer the missing value based on the data from other records. Usually it could be either average, or the median (e.g. if skewed). \n",
    "* __Remove duplicate data__ Clerical errors may arise some rows to have duplicated information. We need to detect them and remove them. \n",
    "* __Inconsistent data__ By business knowledge, experts may inspect which datasets are incorrect. For example they may found incorrect data which values exaggerately large. This is undetectable by computer itself. \n",
    "* __Normalise numeric value__ Transform numeric data into a uniform range is good for training them into machine learning model.\n",
    "* __Reduce dimensionality__ We often want to capture the essense of the dataset, or sometimes it is because the machine learning method prefer simpler datasets. Commonly we can use Principal component analysis (PCA) to do this.\n",
    "* __Add derived features__ In some cases, we may need to compute additional attributes from existing attributes (e.g. linear transform temperature between scales, converting a geolocation to a postcodes). \n",
    "* __Binning numeric value (into categories)__ For numeric attributes, a common way to generalize it is to bin them into ranges, which can be either constant width (variable height/frequency) or variable width (constant height). \n",
    "* __Convert categorical variables into binary variables__ Some machine learning models only take binary input (or numeric input). We may convert the catagorical variables into multiple binary variables (i.e. true and false). Where each label in the catagorical variable is encoded as binary variables for each label.  \n",
    "* __Select, combine, aggregate data__ This includes select which columns are relevant for the model, combine the columns or aggregating multiple raw data records along some dimensions. Often this happens at the end of the EDA and before we finalise which columns should go to our machine learning model. This usually aims to boost our model quality. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following, let us use some of the templates to clean the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From previously, we have found out that few people who had more marks than expected. To find out the how many people have this problem, we use the following code: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "print('=== Excessive marks awarded ===')\n",
    "print('Q1:',df.loc[df.Q1 == 20.5, 'Q1'].count(), 'students')\n",
    "print('Q2:',df.loc[df.Q2 == 20.5, 'Q2'].count(), 'students')\n",
    "print('Q3:',df.loc[df.Q3 == 15.5, 'Q3'].count(), 'students')\n",
    "print('Q4:',df.loc[df.Q4 == 20.5, 'Q4'].count(), 'students')\n",
    "print('Q5:',df.loc[df.Q5 == 15.5, 'Q5'].count(), 'students')\n",
    "print('Q6:',df.loc[df.Q6 == 10.5, 'Q6'].count(), 'students')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How would you able to diagnose the problem? Think about a cause of this incorrect data entry and write a code to see why. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason why this happens is becuase when this dataset was generated, it was based on a code that aimed to generate `0.5` marks. However, this will cause the the problem above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know that not many people have this problem, that is fine. For the purpose of this exercise, let us convert the marks into full marks. This is the code to do that. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "df.loc[df.Q1 == 20.5, 'Q1'] = 20\n",
    "df.loc[df.Q2 == 20.5, 'Q2'] = 20\n",
    "df.loc[df.Q3 == 15.5, 'Q3'] = 15\n",
    "df.loc[df.Q4 == 20.5, 'Q4'] = 20\n",
    "df.loc[df.Q5 == 15.5, 'Q5'] = 15\n",
    "df.loc[df.Q6 == 10.5, 'Q6'] = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you explain what does the code mean? You might need to see the `.loc` documentations in `pandas`. \n",
    "\n",
    "So now we have corrected the illogical data, and let us see how is the dataframe behaving. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Run it\n",
    "print('=== Excessive marks awarded ===')\n",
    "print('Q1:',df.loc[df.Q1 == 20.5, 'Q1'].count(), 'students')\n",
    "print('Q2:',df.loc[df.Q2 == 20.5, 'Q2'].count(), 'students')\n",
    "print('Q3:',df.loc[df.Q3 == 15.5, 'Q3'].count(), 'students')\n",
    "print('Q4:',df.loc[df.Q4 == 20.5, 'Q4'].count(), 'students')\n",
    "print('Q5:',df.loc[df.Q5 == 15.5, 'Q5'].count(), 'students')\n",
    "print('Q6:',df.loc[df.Q6 == 10.5, 'Q6'].count(), 'students')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is another illogical data occured in the dataset. If you look carefully, the assessment marks (rightmost of the columns) are strings. These are stored as percentages and in strings. So we will need to carefully convert the pecentage into numbers by \n",
    "1. Strip the percentage sign away before we convert the data\n",
    "2. Convert the data points using `.astype()`. \n",
    "\n",
    "So let us convert the columns in below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Complete the code and run it\n",
    "df['Quizes Marks'] = df['Quizes'].str.slice(0,-1)\n",
    "df['Quizes Marks'] = df['Quizes Marks'].astype(float)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Missing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In reality, data is rarely clean and homogeneous. In particular, realistic datasets will have some amount of data missing. We will discuss how to treat missing data, discuss how `pandas` chooses to represent it. There are 2 ways in Python to represent missing data, either: \n",
    "* `None`, or \n",
    "* `numpy.nan` (acronym for _Not a Number_). \n",
    "The first one we know it from native Python coding, while the second one comes from `numpy`. They are both equivalent when we use a dataframe. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us do an exercise about missing values. In the following, we use the marks database and, there are few students' marks that were added laterly. Run the code below to get a glimse of the new dataset (_Remember viewing the dataset itself it an important part of EDA_). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Run it\n",
    "new_data = pd.DataFrame([['qbis1318',15,18,0.5,0,0,0,'92.50%','100.00%','100.00%',None],['bzhe1028',3,5,0,0,7,0,'32.18%','0.00%','100.00%',None],['azha9208',2,0,5,0,0,0,None,None,None,None],['jwaj9208',5,8,11.5,5,1,0,'50.55%','25.18%','100.00%',None],['tbai9208',5,8,11.5,5,1,0,'72.63%',None,'100.00%',None]], columns=df.columns) \n",
    "print(new_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The missing data points occured because these students have few issues with administration. For the purpose of this exercise, we just work on the missing value through means of data analysis. Now let us add the dataset into the original one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.concat([df, new_data], ignore_index=True)\n",
    "print(df.head(15))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing we should do is to detect how many missing values we have. In `pandas`, we have a method called `.isnull()`. Either one will return a Boolean mask over the data. For example, \n",
    "```python \n",
    "df2 = pd.DataFrame([[0.5,0,2,Null]])\n",
    "df2.isnull()\n",
    "```\n",
    "will return \n",
    "\n",
    "```\n",
    "0    False\n",
    "1    False\n",
    "2    False\n",
    "3    True\n",
    "dtype: bool\n",
    "```\n",
    "In the following, the code for inspecting how many null values are partially prepared. It is to count how many null values under each column. Think about what each method means and how they are applied in here. Complete the cell below to inspect all columns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print('=== Null values ===')\n",
    "print(df['unikey'].isnull().sum())\n",
    "print(df['Q1'].isnull().sum())\n",
    "print(df['Q2'].isnull().sum())\n",
    "print(df['Q3'].isnull().sum())\n",
    "print(df['Q4'].isnull().sum())\n",
    "print(df['Q5'].isnull().sum())\n",
    "print(df['Quizes'].isnull().sum())\n",
    "# Continue the code below\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The biggest problem of having missing data is therefore we cannot compute further. Often `pandas` methods do not like missing data and we will see an exception. So we will need to eliminate missing data. There are some operations still allow missing values, however, they will be \"eaten\" by the missing value. For example, try the following: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Run this\n",
    "1 + np.nan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Run this\n",
    "0 * np.nan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will see that arithmetic operations are absorbed by the missing data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By rule of thumb, if the amount of missing data is less than 10%, then we should not be too worried of dropping them. In this exercise, we will not look at dropping rows too further. In `pandas`, the method to drop missing data is the `.dropna()`. For example, \n",
    "```python\n",
    "df.dropna()\n",
    "```\n",
    "will __drop all the rows or columns__ that contains missing value. This will lead back to the original dataframe we have. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"fig/MissingDataDrop.png\" width=\"720\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other option is to fill in the missing values. This will be the approach we pursue this time, so that each student will have a mark back. In `pandas`, the `.fillna()` method suits the needs. There are different approaches to do, here are the common appraoches: \n",
    "* Fill with `0`\n",
    "* Fill using mean/ median value\n",
    "* Bring in value from previous row (_forward-fill_)\n",
    "* Bring in value from next row (_back-fill_)\n",
    "* Based on values from the other column(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we will fill in the missing values will `0`. Which is, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Try this\n",
    "df.fillnan(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other method of treating missing values is to __impute__ the missing values. This means the missing values are replaced by another value. It could be either\n",
    "* Zeros (or a number)\n",
    "* Mean, median, mean or random sample \n",
    "* Linear interpolation\n",
    "* Regression (e.g. linear or logistic) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"fig/MissingDataImpute.png\" width=\"720\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The typical steps of imputing values in Python is to \n",
    "\n",
    "0. Generate a `SimpleImputer()` object from `sklearn.impute`. Assign a variable for this. \n",
    "1. Impute the missing values from a data frame (e.g. `df`) by \n",
    "    `imputer_df = pd.DataFrame(my_imputer.fit_transform(df))`\n",
    "2. Bring back the column names by assigning the `.columns` attribute. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ How do you call the `SimpleImputer()` object. In particular, how do you import them from `sklearn`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden"
   },
   "source": [
    "__Solution:__\n",
    "\n",
    "You should import them by \n",
    "```python\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "imputer = SimpleImputer()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the code to impute values should look like \n",
    "```python\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "# Imputation\n",
    "imputer = SimpleImputer()\n",
    "imputed_df = pd.DataFrame(imputer.fit_transform(df))\n",
    "\n",
    "# Put back column names\n",
    "imputed_df.columns = df.columns\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, there are many ways to treat missing values. Either by dropping values or imputation, they are the main catgories. Often people only choose one approach, and there is a way to benchmark either methods. This is called the scoring method. In `sklearn`, there are methods to do so (e.g. `mean_absolute_error()`). \n",
    "\n",
    "We will talk about model evaluation in week 9. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manipulating Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In data science, understanding data initiates the project. It is important to foresee how the data will be important to the deliverables. In the next process __data preparation__, it involves with the following processes: \n",
    "* Deriving new columns\n",
    "* Merging data sources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember the goal of data analysis is to explain something. Therefore, we need either __a target feature__ to explain or any quantitative features (e.g. how close the data points/ clusters). In this section, we will derive the marks column and infer from the that. Note that we will not merge data sources as we are working on only 1 data source at this point. \n",
    "\n",
    "To create a new column, we simply assign values to a column we have never seen from the data source. In particular, the marks breakdown of the course are: \n",
    "* Final exam marks  worth 50%of the course. \n",
    "* Quizes worth 10% of the course. \n",
    "* Lab worth 10% of the course. \n",
    "* Assignment 01 worth 10% of the course. \n",
    "* Assignment 02 worth 20% of the course. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ Create the `'Mark'` column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code below\n",
    "\n",
    "df['Mark']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Solution\n",
    "df['Mark'] = (df['Q1'] + df['Q2'] + df['Q3'] + df['Q4'] + df['Q5'] + df['Q6'])/2 + df['Quizes Marks'] * 0.1 + df['Lab Marks'] * 0.1 + df['Asgn 01 Marks'] * 0.1 + df['Asgn 02 Marks'] * 0.2\n",
    "df['Mark']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can visuallise the data points using `matplotlib` library. It is a Python library that presents our data visually. To call `matplotlib`, simply use the following line into our cell for calling libraries above\n",
    "```python\n",
    "from matplotlib import pyplot as plt\n",
    "% matplotlib inline\n",
    "```\n",
    "You can see that we are importing a library `pyplot` from `matplotlib`, which is commonly used in data analytics. We then shortform the library called `plt`. The second line is unique line to Jupyter, which is to enable the plot outputs show and store in this notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The usual syntax of plotting data is the following: \n",
    "```python\n",
    "plt.plot(data)\n",
    "plt.xlabel('x')\n",
    "plt.show()  # In Jupyter, it can be omitted if inline function used\n",
    "```\n",
    "Each lines above represents each part to plot a diagram. The __first line__ tells us what kind of plots to you want. `pyplot` provides several options to plot (e.g. box plot `plt.box()`, scatter plot `plt.scatter()`, line plot `plt.plot()`) data. The __second line__ is optinally for the bit and pieces (e.g. your axes label, how big is your plot) if your plot in the previous line. __Finally__, use `plt.show()` to display your work. \n",
    "\n",
    "There are plenty of exercises from their [official website](https://matplotlib.org/gallery/index.html). Click on one of the examples and there are many ideas come from the codes underneath the display. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "Let us do an example with `pyplot`, let us plot with a sample data. \n",
    "\n",
    "__Exercise:__ Complete the code below to plot a line plot. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plt_data = np.array[1,0,1,0,5,6,8,10,12,15,19]\n",
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8lOW9/vHP17BD2AMkQNh3SEAD4lLFjSKCttYFrbv9\noT1qbdW6HqvV09a21upRWw7u3VDcqkFQcam7KFsSCPsOCQn7Fsg2398fGT0pJ0DIJHkmM9f79eKV\nmWeemecahStP7jxz3+buiIhI/Dgm6AAiIlK/VPwiInFGxS8iEmdU/CIicUbFLyISZ1T8IiJxRsUv\nIhJnVPwiInFGxS8iEmcaBR2gKh07dvSePXsGHUNEpMGYN2/eVndPqs6+UVn8PXv2ZO7cuUHHEBFp\nMMxsXXX31VCPiEicUfGLiMQZFb+ISJxR8YuIxBkVv4hInFHxi4jEGRW/iEicUfGLiESBOau38cyn\na6iP5XBV/CIiAduxr4SbX1zI375cx/7S8jo/XlR+cldEJF64Oz9/JZvt+0p47coTadGk7mtZZ/wi\nIgF64fO1vLekgDvPHsjQrm3q5ZgqfhGRgCzO28WvZy7ljIGduPqknvV2XBW/iEgAikrKuGnaAtq1\nbMzvL0zHzOrt2BrjFxEJwH1vLGbN1n3840ejad+ySb0eW2f8IiL17I2Fm3h53kZuOq0vJ/TpUO/H\nV/GLiNSjddv2cc/rixjZsx0/OaNfIBlU/CIi9aSkLMRN0xaQcIzx6KQRNEoIpoI1xi8iUk8efncZ\n2Rt3MeWy4+jatnlgOXTGLyJSDz5cVsjUj1dz+egejBvaJdAsKn4RkTpWuPsAt03PYmCXRO45Z1DQ\ncY481GNmzwITgEJ3Hxre9hIwILxLW2Cnuw+v4rlrgT1AOVDm7hm1lFtEpEEIhZxbpmexr6SMly4d\nTbPGCUFHqtYY//PAE8Bfvtng7hd/c9vM/gDsOszzT3P3rTUNKCLSkE35eBWfrtzKQ+cPo2+nxKDj\nANUofnf/2Mx6VvWYVXzU7CLg9NqNJSLS8M1bt4M/vLucCWnJXDyye9BxvhXpGP93gAJ3X3GIxx14\nz8zmmdnkCI8lItJg7Npfyk+mLSClbTN+ff6wep2S4UgivZzzEmDaYR4/2d03mVknYLaZLXX3j6va\nMfyNYTJAampqhLFERILj7tz9Wg4Fuw/w8vUn0LpZ46Aj/Zsan/GbWSPgfOClQ+3j7pvCXwuB14FR\nh9l3qrtnuHtGUlJSTWOJiATuxa838FZOPreOHcCI1HZBx/k/IhnqORNY6u4bq3rQzFqaWeI3t4Gx\nwKIIjiciEvWWF+zh/jcX851+HbnulN5Bx6nSEYvfzKYBXwADzGyjmV0bfmgSBw3zmFmKmc0M3+0M\nfGpmWcBXwFvu/nbtRRcRiS4HSsu58R/zSWzWiD9clM4xx0TPuH5l1bmq55JDbL+qim15wPjw7dVA\neoT5REQajAdn5LK8YC9/uWYUnRKbBR3nkPTJXRGRWjArJ5+/z1nPdaf05pT+0f17ShW/iEiENu4o\n4o5Xs0nv3pZbxw448hMCpuIXEYlAWXmIm19ciDs8PmkETRpFf61qWmYRkQg8+t4K5q3bwX9fMoLU\nDi2CjlMt0f+tSUQkSn2+citP/mslF2V049z0lKDjVJuKX0SkBrbtLeanLy2kd8eW3H/ukKDjHBUN\n9YiIHKVQyLnt5Sx27i/lhWtG0aJJw6pSnfGLiBylZz9bw4fLtvCf5wxiUHLroOMcNRW/iMhRyNm4\ni9++vZSxgztz+egeQcepERW/iEg17S0u46Zp80lq1ZTfXZAWVVMtH42GNTAlIhKge/+5iPXbi3hx\n8gm0bdEk6Dg1pjN+EZFqeHXeRl5fsImbz+jPqF7tg44TERW/iMgRrN6yl3vfWMTxvdpz4+l9g44T\nMRW/iMhhFJeVc9O0BTRtdAyPThpOQpROtXw0NMYvInIYD81ayuK83Tx9RQbJbZoHHadW6IxfROQQ\n3l9SwHOfreWqE3ty5uDOQcepNSp+EZEqbN51gNtezmJwcmvuGj8w6Di1qjpLLz5rZoVmtqjStvvN\nbJOZLQz/GX+I544zs2VmttLM7qzN4CIidaU85Nz84gKKy0I8fukImjZKCDpSrarOGf/zwLgqtv/R\n3YeH/8w8+EEzSwCeBM4GBgOXmNngSMKKiNSHJz9cyZw123ngvKH0SWoVdJxad8Tid/ePge01eO1R\nwEp3X+3uJcCLwHk1eB0RkXrz9drtPPrecr43PIUfHNs16Dh1IpIx/pvMLDs8FNSuise7Ahsq3d8Y\n3iYiEpXydu7nJ9MW0L19C/7r+8Ma7JQMR1LT4v8z0BsYDuQDf4g0iJlNNrO5ZjZ3y5Ytkb6ciMhR\nKdh9gEue+pK9B8p48tJjadU0dq92r1Hxu3uBu5e7ewh4iophnYNtArpXut8tvO1QrznV3TPcPSMp\nKbpXqBeR2LJlTzGXPPUlW/cU88K1oxjatU3QkepUjYrfzJIr3f0+sKiK3b4G+plZLzNrAkwC3qzJ\n8URE6sq2vcX88Okvyd95gOeuHsWxqVWNXMeWI/4sY2bTgDFARzPbCNwHjDGz4YADa4HrwvumAE+7\n+3h3LzOzG4F3gATgWXdfXCfvQkSkBnYWlXDZM1+xblsRz109ssFPvlZd5u5BZ/g/MjIyfO7cuUHH\nEJEYtmt/KZc9PYdlm/fw9JUZnNK/YQ8xm9k8d8+ozr765K6IxJ09B0q58tmvWLp5N1MuP7bBl/7R\nUvGLSFzZV1zGNc9/zaJNu3jy0mM5fWDszMFTXSp+EYkb+0vKufaFr5m3bgePTRrB2CFdgo4UiNi9\nUFVEpJIDpeVM/utc5qzZzqMXD+ectOQjPylG6YxfRGJecVk5P/7bPD5ZsZXf/SCN84bH9yQCKn4R\niWklZSFu+PsCPly2hV9/fxgXZnQ/8pNinIpfRGJWWXmIm19cwHtLCnjgvCFcenxq0JGigopfRGJS\neci5ZXoWsxZt5j/PGcQVJ/QMOlLUUPGLSMwJhZzbX8nmzaw87hg3kB99p3fQkaKKil9EYkoo5Nz9\neg6vzt/ILWf158dj+gQdKeqo+EUkZrg79725mBe/3sCNp/XlJ2f0CzpSVFLxi0hMcHcenLGEv365\njutO6c2tY/sHHSlqqfhFpMFzdx56eynPfraGq0/qyZ1nD4zZ1bNqg4pfRBq8P85ezv98tJrLRqfy\niwmDVfpHoOIXkQbt8fdX8N8frOTijO48cO5QlX41qPhFpMGa8tEq/jB7Oecf25XfnD+MY45R6VeH\nil9EGqRnPl3DQ7OWMjE9hd9fkK7SPwpHLH4ze9bMCs1sUaVtvzezpWaWbWavm1nbQzx3rZnlmNlC\nM9OSWiJSK/76xVoenJHL2UO78MhF6SSo9I9Kdc74nwfGHbRtNjDU3dOA5cBdh3n+ae4+vLpLgomI\nHM6LX63n3jcWc+agzjw2aQSNEzRwcbSO+F/M3T8Gth+07V13Lwvf/RLoVgfZRET+zSvzNnLX6zmM\nGZDEkz8cQZNGKv2aqI3/atcAsw7xmAPvmdk8M5tcC8cSkTj1xsJN3P5KFif16ciUy46jaaOEoCM1\nWBGtwGVm9wBlwN8PscvJ7r7JzDoBs81safgniKpeazIwGSA1VVOnisj/mpmTzy3TsxjVqz1PXZFB\ns8Yq/UjU+IzfzK4CJgA/dHevah933xT+Wgi8Dow61Ou5+1R3z3D3jKSk+FrxXkQO7d3Fm/nJtAWM\n6N6WZ64cSfMmKv1I1aj4zWwccDtwrrsXHWKflmaW+M1tYCywqKp9RUSq8uHSQm74x3yGdm3Dc1eP\npGVTLRNeG6pzOec04AtggJltNLNrgSeARCqGbxaa2ZTwvilmNjP81M7Ap2aWBXwFvOXub9fJuxCR\nmPPJii1c97d5DOiSyAvXjCKxWeOgI8WMI377dPdLqtj8zCH2zQPGh2+vBtIjSicicemdxZu5+cUF\n9O7Ykr9dezxtmqv0a5N+bhKRqLH7QCm/fDOXV+dvZGjX1rxw9SjatmgSdKyYo+IXkajw2cqt/Pzl\nLDbvPvDtIiq6Tr9uqPhFJFD7S8p5aNYSXvhiHb07tuTVH5/IiNR2QceKaSp+EQnM/PU7uHV6Fmu2\n7uOqE3tyx7iBulyzHqj4RaTelZSFePS95Uz5aBXJbZrzjx8dz4l9OwYdK26o+EWkXi3J380t07NY\nkr+bC4/rxr0TB9Nal2rWKxW/iNSLsvIQUz9ZzR9nL6dN8yY8fUUGZw7uHHSsuKTiF5E6t2brPm6d\nvpD563cyflgX/ut7w2jfUpdpBkXFLyJ1JhRy/jZnHb+ZuZTGCcZjk4ZzbnqK1sUNmIpfROpE3s79\n3P5KNp+u3Mqp/ZP47Q/S6NKmWdCxBBW/iNQyd+e1+Zu4P3Mx5SHnV98fyqWjUnWWH0VU/CJSa7bu\nLebu13J4N7eAkT3b8fCF6fTo0DLoWHIQFb+I1Iq3F23mntdz2HOgjLvHD+Tak3trEfQopeIXkYjs\n2l/KL99czGsLNjG0a2umXTSc/p0Tg44lh6HiF5Ea+2TFFm5/JZvCPcXcfEY/bjy9L40TNLFatFPx\ni8hRKyop4zczl/LXL9fRt1MrXr/8ONK6tQ06llSTil9Ejsq8ddu5dXoW67YX8aOTe3Hbdwdo8fMG\npjpLLz5rZoVmtqjStvZmNtvMVoS/VjmHqpmNM7NlZrbSzO6szeAiUr+Ky8p5aNZSLpzyBWUhZ9r/\nG81/This0m+AqjMY9zww7qBtdwLvu3s/4P3w/X9jZgnAk8DZwGDgEjMbHFFaEQnE4rxdnPfEZ0z5\naBUXj+zO2z89hdG9OwQdS2qoOmvufmxmPQ/afB4wJnz7BeBfwB0H7TMKWBleexczezH8vNwapxWR\nelVWHmLKR6t47P0VtGvRhOeuGslpAzsFHUsiVNMx/s7unh++vRmoaoq9rsCGSvc3AsfX8HgiUs9W\nbdnLrdOzWLhhJxPTU3jg3CG008RqMSHiX+66u5uZR/o6ZjYZmAyQmpoa6cuJSA2FQs4LX6zlt28v\npVnjBB6/ZAQT01OCjiW1qKbFX2Bmye6eb2bJQGEV+2wCule63y28rUruPhWYCpCRkRHxNxIROXob\ndxTx85ez+WL1Nk4f2ImHzh9Gp9aaWC3W1LT43wSuBB4Kf32jin2+BvqZWS8qCn8ScGkNjycidcjd\neXneRh7IzMXd+e0PhnFRRndNrBajjlj8ZjaNil/kdjSzjcB9VBT+dDO7FlgHXBTeNwV42t3Hu3uZ\nmd0IvAMkAM+6++K6eRsiUlOFew5w92s5vLekkON7tefhC9Pp3r5F0LGkDpl79I2qZGRk+Ny5c4OO\nIRLzZubkc8/rOewrKef27w7gmpN6cYwmVmuQzGyeu2dUZ199clckDu0qKuUXby7ijYV5pHVrwyMX\npdO3kyZWixcqfpE4869lhdzxajbb9pZwy1n9+fGYPppYLc6o+EXixL7iMn41cwn/mLOefp1a8cyV\nIxnatU3QsSQAKn6ROPDVmu3c9nIWG3YUMfmU3txyVn/NsRPHVPwiMexAaTmPzF7OU5+spnu7Frw0\n+QRG9WofdCwJmIpfJEYt2rSLW6YvZHnBXi49PpV7xg+iZVP9kxcVv0jMKS0P8acPV/H4Byvo0KoJ\nz189kjEDNLGa/C8Vv0gMWVm4h1umZ5G9cRfnDU/hgXOH0qZF46BjSZRR8YvEgFDIefazNfz+nWW0\naJLAn354LOOHJQcdS6KUil+kgduwvYjbXs5izprtnDmoE78+fxidEjWxmhyail+kgXJ3Xvp6Aw/O\nyMXM+N0FaVx4XDdNrCZHpOIXaYAKdx/gztdy+GBpISf07sDvL0yjWztNrCbVo+IXaWAys/K4941F\n7C8p576Jg7nyhJ6aWE2OiopfpIHYsa+Ee99YxIzsfNK7t+WRi9Lpk9Qq6FjSAKn4RRqAD5cWcvur\n2ezYV8JtY/tz/al9aKSJ1aSGVPwiUWxvcRn/NSOXF7/ewIDOiTx3lSZWk8ip+EWi1Jert3Hby1nk\n7dzPdadWTKzWtJEmVpPI1bj4zWwA8FKlTb2BX7j7o5X2GUPFerxrwptec/cHanpMkXhwoLSc37+z\njGc/W0Nq+xZMv+4EMnpqYjWpPTUufndfBgwHMLMEKhZUf72KXT9x9wk1PY5IPMneuJNbpmexsnAv\nl4/uwZ1nD9TEalLrautv1BnAKndfV0uvJxJXSstDPP7BSp78cCVJrZryl2tGcUr/pKBjSYyqreKf\nBEw7xGMnmlk2FT8R3Obui2vpmCIxYXnBHm6ZvpBFm3Zz/oiu3DdxiCZWkzoVcfGbWRPgXOCuKh6e\nD6S6+14zGw/8E+h3iNeZDEwGSE1NjTSWSNQrDznPfLqah99dTqumjZhy2bGMG6qJ1aTu1cYZ/9nA\nfHcvOPgBd99d6fZMM/uTmXV0961V7DsVmAqQkZHhtZBLJGqt31YxsdpXa7dz1uDO/Pr7w0hKbBp0\nLIkTtVH8l3CIYR4z6wIUuLub2SjgGGBbLRxTpEFyd/7x1Xp+9dYSEsz4w4XpnH9sV02sJvUqouI3\ns5bAWcB1lbZdD+DuU4ALgB+bWRmwH5jk7jqbl7i0edcB7ng1m4+Wb+Gkvh343QXpdG3bPOhYEoci\nKn533wd0OGjblEq3nwCeiOQYIg2du/NmVh6/eGMxxWXlPHDeEC47vocmVpPA6AJhkTq0fV8J//nP\nHGbmbGZEalv+cGE6vTWxmgRMxS9SR97LLeDO13LYtb+En393ANed0lsTq0lUUPGL1LI9B0p5IDOX\nl+dtZGCXRP5yzSgGp7QOOpbIt1T8IrXo81Vb+fnL2eTv2s9/jOnDzWf208RqEnVU/CK1YH9JOb99\neynPf76WXh1b8vL1J3Jcj3ZBxxKpkopfJEILN+zklukLWb1lH1ee0IM7zh5Iiyb6pyXRS387RWqo\npCzE4x+s4E//WkXnxKb8/UfHc1LfjkHHEjkiFb9IDSzbvIefvbSQ3Pzd/ODYbtx37mBaN9PEatIw\nqPhFjkJ5yHnqk9U88u5yWjdvxNTLj2PskC5BxxI5Kip+kWpaUbCHu17LYe66HYwb0oVffX8oHVpp\nYjVpeFT8IoexcUcRb2XnMyM7n5xNu0hs1og/XpzO94ZrYjVpuFT8Igcp3H2At3LyyczKY/76nQCk\nd2vDPeMHcd6IFDolNgs4oUhkVPwiwLa9xcxatJkZ2XnMWbMddxjYJZGff3cAE9KS6dGhZdARRWqN\nil/i1q6iUt7J3UxmVh6fr9pGecjpk9SSn5zej4npyfTtlBh0RJE6oeKXuLK3uIz3cguYkZ3HR8u3\nUFrudG/fnOtO6c2EtBQGJSdq7F5inopfYt7+knI+XFZIZlYeHywtpLgsRHKbZlx5Qk8mpqeQ1q2N\nyl7iiopfYlJxWTkfL9/KjOw8ZucWUFRSTsdWTZk0sjsT0lM4LrWdFkKRuBXp0otrgT1AOVDm7hkH\nPW7AY8B4oAi4yt3nR3JMkUMpLQ/x+aptZGbl8c7izew5UEbbFo05b3gKE9NSOL53BxJU9iK1csZ/\nmrtvPcRjZwP9wn+OB/4c/ipSK8pDzpw125iRnc+snHx2FJWS2LQRY4d0YUJ6Mif37UhjLX4i8m/q\neqjnPOAv4QXWvzSztmaW7O75dXxciWGhkLNgww4ys/J5KyefLXuKad44gTMHd2ZiWjKn9E+iWWPN\ngS9yKJEWvwPvmVk58D/uPvWgx7sCGyrd3xjepuKXGtmwvYjLnpnDum1FNGl0DKcP6MSE9GROH9hJ\nUyGLVFOk/1JOdvdNZtYJmG1mS93945q8kJlNBiYDpKamRhhLYtWDM3LZsqeYRy5K56zBnUnUjJgi\nRy2iwU933xT+Wgi8Dow6aJdNQPdK97uFt1X1WlPdPcPdM5KSkiKJJTHq4+VbeDe3gBtP78v5x3ZT\n6YvUUI2L38xamlniN7eBscCig3Z7E7jCKowGdml8X2qitDzELzMX06NDC649uVfQcUQatEiGejoD\nr4c/+NII+Ie7v21m1wO4+xRgJhWXcq6k4nLOqyOLK/Hqhc/XsmrLPp6+IkOLl4tEqMbF7+6rgfQq\ntk+pdNuBG2p6DBGALXuKeey9FZzaP4kzBnUKOo5Ig6cLnCXqPfzOMvaXlvOLiYM1tYJILVDxS1TL\n2rCT6fM2cM3JveiT1CroOCIxQcUvUSsUcu7PXEyHlk256fS+QccRiRkqfolary/YxIL1O7lj3ABd\nuilSi1T8EpX2Fpfx0NtLSe/elh8c2y3oOCIxRZ9xl6j0+Acr2LKnmKeuyND0ySK1TGf8EnVWb9nL\ns5+u4cLjujG8e9ug44jEHBW/RJ0HZ+TStFECPx83IOgoIjFJxS9R5YOlBXy4bAs3n9GPTonNgo4j\nEpNU/BI1isvKeXDGEnonteTKE3sGHUckZqn4JWo899la1mzdxy8mDKZJI/3VFKkr+tclUaFg9wEe\nf38FZw7qzJgBmo9HpC6p+CUq/HbWUkrLnXsnDAo6ikjMU/FL4Oat28FrCzbxo+/0okeHlkHHEYl5\nKn4JVCjk3P/mYjq3bsoNp2k+HpH6oOKXQL08bwM5m3Zx9/hBtGyqD5KL1AcVvwRm1/5Sfvf2MjJ6\ntOPc9JSg44jEjUjW3O1uZh+aWa6ZLTazm6vYZ4yZ7TKzheE/v4gsrsSSx95bwfaiEu4/d4gWWBGp\nR5H8bF0G3Oru88OLrs8zs9nunnvQfp+4+4QIjiMxaEXBHv7yxVomjUxlaNc2QccRiSs1PuN393x3\nnx++vQdYAnStrWASu9ydX2bm0qJJAreN7R90HJG4Uytj/GbWExgBzKni4RPNLNvMZpnZkNo4njRs\n7+YW8OnKrdxyVn86tGoadByRuBPxZRRm1gp4Ffipu+8+6OH5QKq77zWz8cA/gX6HeJ3JwGSA1NTU\nSGNJlDpQWs6DM3Lp37kVl43uEXQckbgU0Rm/mTWmovT/7u6vHfy4u+92973h2zOBxmbWsarXcvep\n7p7h7hlJSUmRxJIo9tTHq9m4Yz/3TRxCowRdVCYShEiu6jHgGWCJuz9yiH26hPfDzEaFj7etpseU\nhi1v537+9K9VnD20Cyf1rfL7v4jUg0iGek4CLgdyzGxheNvdQCqAu08BLgB+bGZlwH5gkrt7BMeU\nBuw3s5YScufu8ZqPRyRINS5+d/8UOOzF1+7+BPBETY8hsWPO6m1kZuVx8xn96N6+RdBxROKaBlml\nzpWHnPszc0lp04zrT+0TdByRuKfilzo37av1LMnfzT3nDKZ5k4Sg44jEPRW/1KmdRSU8/O4yRvdu\nz/hhXYKOIyKo+KWOPTJ7Obv3l2o+HpEoouKXOrMkfzd/+3Idl43uwcAurYOOIyJhKn6pE+4VC6y0\nbt6YW87SfDwi0UTFL3ViZs5m5qzZzm1jB9C2RZOg44hIJSp+qXX7S8r51Vu5DEpuzSWjNO+SSLRR\n8Uut+/NHq8jbdYBfnjuEhGP0C12RaBNTxf/5yq3sKioNOka9CYWc5QV7KCopCzrKtzZsL2LKR6uY\nmJ7CqF7tg44jIlWImdWti0rKuPaFuZSFQpzSL4kJ6cmcNbgLrWJsAW93Z9Gm3WRm5/FWdj6bdu6n\neeMEzhjUiYnpKZzaP4lmjYP7kNSv3lpCghl3nT0wsAwicngx04rNGyfw4uTRzMjOY0Z2Pu8vLaRp\noxxOG1BRiKcP7NRgPzXq7iwr2MOMrHwys/NYt62IxgnGd/ol8R+n9SE3bzezFm1mRnY+iU0bcdaQ\nzkxMS+Hkfh1pXI9TH3+2citvL97MbWP7k9K2eb0dV0SOjkXjZJkZGRk+d+7cGj8/FHLmr99BZlYe\nb+VsZuveYlo0SeDMQZ2ZmJ7CKf070rRR9H8TWLVlLzOy8pmRnceKwr0cY3BS345MSEvmu0O6/NvV\nMmXlIT5fVTER2juLN7P7QBltWzRm3JAuTExPYXTvDnU63l5aHuKc//6E/aXlzP7ZqYH+1CESj8xs\nnrtnVGvfWCz+yspDXjEzZHY+sxbls7OolMRmjfjukC5MSEvmpL71e1Z8JBu2FzEjO5/MrDxy83dj\nBiN7tmdiegpnD+1Cx2osVVhcVs4ny7cyIzuP2bkF7Cspp2OrJowflsyEtBQyerTjmFr+JvDcZ2v4\nZWYuUy8/jrFDNDWDSH1T8R9CaXmIz1ZuJTMrn3cXb2ZPcRntWjRm3NBkJqYnc3yvuj0rPpTNuw58\nO0S1cMNOAEaktmVCWgrnDEumS5tmNX7tA6XlfLi0kMzsPN5fUkhxWYgurZtxTloyE9NTSO/WJuKp\nFLbtLWbMw/9iePe2/OWaUZqaQSQAKv5qKC4r5+PlW8nMyuO9JQUUlZSTlNiUc4YlMyEtmWNTa/+s\nuLKte4uZlZNPZlY+X6/bjjsMSWnNxPSKsq+LOev3Fpfx/pICMrPy+Wh5IaXlTvf2zZmQlsKEtGQG\nJ7euUWnf9VoOL8/dwNs//Q59OyXWem4ROTIV/1HaX1LOB0sLmZGdxwdLK86KU9r871nxsK6RnxVD\nxUyVb4d/Cfv5qq2EHPp3bvVt8fZOalUL76Z6du0v5d3Fm8nMzuezlVspDzm9k1oyIS2Fc9OTq13g\nizbtYuITn3LNSb24d8LgOk4tIodSb8VvZuOAx4AE4Gl3f+igxy38+HigCLjK3ecf6XXru/gr21tc\nxnu5BWRm5fHxii2Uljup7VswMb1ifHxgl8Sj+iaw50Aps8Ov98mKrZSFnJ4dWjAxPYUJaSkM6BL8\nGfL2fSXMWpTPjKx8vlyzDXcY2CUxnDGZHh1aVvk8d+eCKV+wbts+PrhtDK2bNa7n5CLyjXopfjNL\nAJYDZwEbga+BS9w9t9I+44GbqCj+44HH3P34I712kMVf2a6iUt5ZvJnM7Dw+X7WN8pDTt1MrJoR/\nEuhziDP0opIy3l9S8RPEh8u2UFIWomvb5kxIT2ZiWgpDUmo2pFIfCncfYGZOPpnZ+cxbtwOAtG5t\nmJCWzDlpKXStdJnmPxds4qcvLeS3PxjGxSM1NYNIkOqr+E8A7nf374bv3wXg7r+ptM//AP9y92nh\n+8uAMe5n8PhOAAAFEklEQVSef7jXjpbir2zb3mJmLdpMZlYeX62tGJMflNyaieEyT0psykfLt5CZ\nVfFL1P2l5XRKbPrtcNGI7m2jtuwPZdPO/bwV/qVz9sZdABzXox0T05IZM6ATF0/9gs6tm/HP/zip\nTn8fIiJHVl/FfwEwzt1/FL5/OXC8u99YaZ8ZwEPhhdkxs/eBO9z9sK0ejcVfWcHuA7yVXXF9/fz1\nFVfhNGt8DAdKQ7Rv2YSzh1ZcOz+yZ/uYmatm7dZ9vJVTcZnp0s17vt3+6o9P5Lge7QJMJiJwdMUf\nNZ/cNbPJwGSA1NToHjbo3LoZ15zci2tO7sXGHRXX3eft3M+ZgzpzYp8ONIqizwXUlp4dW3LDaX25\n4bS+rCzcQ2ZWPm1bNFbpizRAkRT/JqB7pfvdwtuOdh8A3H0qMBUqzvgjyFWvurVrwfWn9gk6Rr3q\n2ymRn50V/C+lRaRmIjk1/RroZ2a9zKwJMAl486B93gSusAqjgV1HGt8XEZG6VeMzfncvM7MbgXeo\nuJzzWXdfbGbXhx+fAsyk4oqelVRcznl15JFFRCQSEY3xu/tMKsq98rYplW47cEMkxxARkdoVe7+F\nFBGRw1Lxi4jEGRW/iEicUfGLiMQZFb+ISJyJymmZzWwLsK6GT+8IbK3FOA2B3nPsi7f3C3rPR6uH\nuydVZ8eoLP5ImNnc6s5XESv0nmNfvL1f0HuuSxrqERGJMyp+EZE4E4vFPzXoAAHQe4598fZ+Qe+5\nzsTcGL+IiBxeLJ7xi4jIYcRM8ZvZODNbZmYrzezOoPPUNTPrbmYfmlmumS02s5uDzlRfzCzBzBaE\nV3iLeWbW1sxeMbOlZrYkvOxpTDOzn4X/Xi8ys2lm1izoTLXNzJ41s0IzW1RpW3szm21mK8Jf62Sl\no5go/vDC708CZwODgUvMbHCwqepcGXCruw8GRgM3xMF7/sbNwJKgQ9Sjx4C33X0gkE6Mv3cz6wr8\nBMhw96FUTPs+KdhUdeJ5YNxB2+4E3nf3fsD74fu1LiaKHxgFrHT31e5eArwInBdwpjrl7vnuPj98\new8VZdA12FR1z8y6AecATwedpT6YWRvgFOAZAHcvcfedwaaqF42A5mbWCGgB5AWcp9a5+8fA9oM2\nnwe8EL79AvC9ujh2rBR/V2BDpfsbiYMS/IaZ9QRGAHOCTVIvHgVuB0JBB6knvYAtwHPh4a2nzaxl\n0KHqkrtvAh4G1gP5VKzc926wqepN50qrFG4GOtfFQWKl+OOWmbUCXgV+6u67g85Tl8xsAlDo7vOC\nzlKPGgHHAn929xHAPurox/9oER7XPo+Kb3opQEszuyzYVPUvvJBVnVx2GSvFX+1F3WOJmTWmovT/\n7u6vBZ2nHpwEnGtma6kYzjvdzP4WbKQ6txHY6O7f/DT3ChXfCGLZmcAad9/i7qXAa8CJAWeqLwVm\nlgwQ/lpYFweJleKvzsLvMcXMjIpx3yXu/kjQeeqDu9/l7t3cvScV/48/cPeYPhN0983ABjMbEN50\nBpAbYKT6sB4YbWYtwn/PzyDGf6FdyZvAleHbVwJv1MVBIlpzN1ocauH3gGPVtZOAy4EcM1sY3nZ3\neB1kiS03AX8Pn9SsBq4OOE+dcvc5ZvYKMJ+Kq9cWEIOf4jWzacAYoKOZbQTuAx4CppvZtVTMUHxR\nnRxbn9wVEYkvsTLUIyIi1aTiFxGJMyp+EZE4o+IXEYkzKn4RkTij4hcRiTMqfhGROKPiFxGJM/8f\nlBiN+xnACQQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9d72550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Solution\n",
    "from matplotlib import pyplot as plt\n",
    "% matplotlib inline\n",
    "\n",
    "plt_data = np.array([1,0,1,0,5,6,8,10,12,15,19])\n",
    "plt.plot(plt_data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "So now let us plot the marks of each student in the marks dataset. \n",
    "\n",
    "__Exercise:__ Plot the marks of each student using a scatter plot. What do the scatter plot mean? (Use `plt.scatter()` to help you)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Solution \n",
    "plt.plot(df['Mark'])\n",
    "plt.show()\n",
    "print('The scatter shows marks of each student, in the order of the dataframe rows.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let us plot the distribution of people who passed and failed the course. The following is the demonstration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Run it \n",
    "plt.plot(df.loc[df.Mark >= 50, 'Mark'], 'bo')\n",
    "plt.plot(df.loc[df.Mark < 50, 'Mark'], 'ro')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you explain what is in each line of code above?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label Encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often our dataset contains string data representing categories. In many analysis they require them to be numerical, and this is where label encoding comes in. For example, we can label \n",
    "* \"Male\" as $1$ and, \n",
    "* \"Female\" as $0$\n",
    "\n",
    "and this is useful in linear regression. \n",
    "\n",
    "In `skearn`, a machine learning library, there are couple of ways to encode them. For the time being, this is to let you aware of the tools and we shall encode a catagorical feature in week 9. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA with Text Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In these 2 weeks we will also look at how to infer from text files. The key of analysing text data sets is to encode them quantitatively. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "news = pd.read_csv('files/news.txt', sep='\\n', header=None)\n",
    "print(news.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our first analysis involves to find the length of each row. This is done by "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run me\n",
    "news[0].str.len()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us visualise the number of words in each line. To do that, we should make sure `matplotlib` and `pyplot` are imported. \n",
    "\n",
    "```python\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "```\n",
    "The following code does the work. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEvhJREFUeJzt3X+Mndl91/H3p97ESTcV8XYHy9gGu5IJ8kZkAyMTaFSF\nLum6LKoXhCyv1MqgRe4fbpsgJOrtP1uQLFmoVOUPtpJpAoNI4w75wVpN1eKYVKESWmd2szRrb8ya\nrF3b+Mc0IaRbkMs6X/6Ys/SuuzP3Xs9cz8zZ90uy7nnOc56533l25zNnzr3Pc1NVSJL69T2rXYAk\nabIMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1Ln7lvtAgAefPDB2rFjx2qXIUnr\nyvPPP//7VTU1bNyaCPodO3YwNze32mVI0rqS5NIo41y6kaTOGfSS1DmDXpI6Z9BLUucMeknqnEEv\nSZ0z6CWpcwa9JHVupKBP8g+TnE3yUpJPJ3lXkgeSnErySnvcNDD+qSQXkpxP8ujkypckDTP0ytgk\nW4GfAXZX1f9JMgscAHYDp6vqWJIjwBHgZ5PsbvsfAv4M8MUkf76qbk/su5AmbMeRLyy67+Kxx+5h\nJdL4Rl26uQ94d5L7gO8F/gewD5hp+2eAx1t7H3Ciqm5V1avABWDPypUsSRrH0KCvqqvALwC/B1wD\n/ldV/Udgc1Vda8OuA5tbeytweeBLXGl9kqRVMDTo29r7PmAnC0sx9yf58cExVVVAjfPESQ4lmUsy\nNz8/P86hkqQxjLJ08zeAV6tqvqr+L/A54K8BN5JsAWiPN9v4q8D2geO3tb43qarjVTVdVdNTU0Pv\nsilJukujBP3vAR9K8r1JAjwCvAycBA62MQeBZ1v7JHAgycYkO4FdwJmVLVuSNKqh77qpqueSfAZ4\nAXgd+CpwHHgPMJvkSeASsL+NP9vemXOujT/sO24kafWM9MEjVfU08PQd3bdYmN2/1fijwNHllSZJ\nWgleGStJnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0b6cpY9WWpD9EAP0hD\n6o0zeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOjc06JO8L8mLA/++k+TjSR5IcirJK+1x\n08AxTyW5kOR8kkcn+y1IkpYyNOir6nxVPVxVDwN/GfjfwOeBI8DpqtoFnG7bJNkNHAAeAvYCzyTZ\nMKH6JUlDjLt08wjw36vqErAPmGn9M8Djrb0POFFVt6rqVeACsGclipUkjW/coD8AfLq1N1fVtda+\nDmxu7a3A5YFjrrS+N0lyKMlckrn5+fkxy5AkjWrkoE/yTuDHgH9/576qKqDGeeKqOl5V01U1PTU1\nNc6hkqQxjDOj/1Hghaq60bZvJNkC0B5vtv6rwPaB47a1PknSKhgn6J/gj5dtAE4CB1v7IPDsQP+B\nJBuT7AR2AWeWW6gk6e6MdJviJPcDHwV+cqD7GDCb5EngErAfoKrOJpkFzgGvA4er6vaKVi1JGtlI\nQV9Vfwh8/x1932ThXThvNf4ocHTZ1UmSls0rYyWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ\n6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjo30t0rJUl3Z8eRLyy5/+KxxyZegzN6SeqcQS9J\nnRsp6JO8N8lnknw9yctJ/mqSB5KcSvJKe9w0MP6pJBeSnE/y6OTKlyQNM+qM/l8Av1lVfwH4APAy\ncAQ4XVW7gNNtmyS7gQPAQ8Be4JkkG1a6cEnSaIYGfZI/BfwQ8AmAqvqjqvo2sA+YacNmgMdbex9w\noqpuVdWrwAVgz0oXLkkazSgz+p3APPCvk3w1ya+0DwvfXFXX2pjrwObW3gpcHjj+SuuTJK2CUYL+\nPuAvAb9cVR8E/pC2TPOGqiqgxnniJIeSzCWZm5+fH+dQSdIYRgn6K8CVqnqubX+GheC/kWQLQHu8\n2fZfBbYPHL+t9b1JVR2vqumqmp6amrrb+iVJQwwN+qq6DlxO8r7W9QhwDjgJHGx9B4FnW/skcCDJ\nxiQ7gV3AmRWtWpI0slGvjP1p4FNJ3gl8A/j7LPySmE3yJHAJ2A9QVWeTzLLwy+B14HBV3V7xyiVJ\nIxkp6KvqRWD6LXY9ssj4o8DRZdQlSVohXhkrSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1Ln\nDHpJ6pxBL0mdM+glqXMGvSR1btSbmknSittx5AuL7rt47LF7WEnfnNFLUucMeknqnEEvSZ1zjV7d\ncv1XWuCMXpI6N1LQJ7mY5GtJXkwy1/oeSHIqySvtcdPA+KeSXEhyPsmjkypekjTcODP6v15VD1fV\nGx8peAQ4XVW7gNNtmyS7gQPAQ8Be4JkkG1awZknSGJazdLMPmGntGeDxgf4TVXWrql4FLgB7lvE8\nkqRlGDXoC/hikueTHGp9m6vqWmtfBza39lbg8sCxV1rfmyQ5lGQuydz8/PxdlC5JGsWo77r5cFVd\nTfKngVNJvj64s6oqSY3zxFV1HDgOMD09PdaxkqTRjRT0VXW1Pd5M8nkWlmJuJNlSVdeSbAFutuFX\nge0Dh29rfXoLS70FEHwboKTlG7p0k+T+JN/3Rhv4EeAl4CRwsA07CDzb2ieBA0k2JtkJ7ALOrHTh\nkqTRjDKj3wx8Pskb43+1qn4zyVeA2SRPApeA/QBVdTbJLHAOeB04XFW3J1K9JGmooUFfVd8APvAW\n/d8EHlnkmKPA0WVXNyKvgJSkxXkLBOkec2Kie81bIEhS5wx6SeqcQS9JnXONXlIXvCZlcc7oJalz\nBr0kdc6gl6TOuUavNcM1VmkynNFLUucMeknqnEEvSZ1zjV7L4rq6tPY5o5ekzjmjl/S28na8e6gz\neknq3MhBn2RDkq8m+fW2/UCSU0leaY+bBsY+leRCkvNJHp1E4ZKk0Ywzo/8Y8PLA9hHgdFXtAk63\nbZLsBg4ADwF7gWeSbFiZciVJ4xop6JNsAx4DfmWgex8w09ozwOMD/Seq6lZVvQpcAPasTLmSpHGN\nOqP/JeAfA98d6NtcVdda+zoLHyIOsBW4PDDuSuuTJK2CoUGf5G8BN6vq+cXGVFUBNc4TJzmUZC7J\n3Pz8/DiHSpLGMMqM/geBH0tyETgB/HCSfwfcSLIFoD3ebOOvAtsHjt/W+t6kqo5X1XRVTU9NTS3j\nW5AkLWVo0FfVU1W1rap2sPAi63+qqh8HTgIH27CDwLOtfRI4kGRjkp3ALuDMilcuSRrJci6YOgbM\nJnkSuATsB6iqs0lmgXPA68Dhqrq97EolSXdlrKCvqt8Gfru1vwk8ssi4o8DRZdYmSVoBXhkrSZ0z\n6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1Ln/ISpCXo7fpKNpLXHGb0kdc6gl6TOGfSS1DmDXpI6\n54ux0tuAbwx4e3NGL0mdM+glqXMu3dyFpf4MBv8UlrS2OKOXpM4NndEneRfwZWBjG/+Zqno6yQPA\nrwE7gIvA/qr6n+2Yp4AngdvAz1TVb02kekmagN7+ah9lRn8L+OGq+gDwMLA3yYeAI8DpqtoFnG7b\nJNnNwoeIPwTsBZ5JsmESxUuShhsa9LXgtbb5jvavgH3ATOufAR5v7X3Aiaq6VVWvAheAPStatSRp\nZCOt0SfZkORF4CZwqqqeAzZX1bU25DqwubW3ApcHDr/S+u78moeSzCWZm5+fv+tvQJK0tJGCvqpu\nV9XDwDZgT5L337G/WJjlj6yqjlfVdFVNT01NjXOoJGkMY73rpqq+DXyJhbX3G0m2ALTHm23YVWD7\nwGHbWp8kaRUMDfokU0ne29rvBj4KfB04CRxsww4Cz7b2SeBAko1JdgK7gDMrXbgkaTSjXDC1BZhp\n75z5HmC2qn49yX8BZpM8CVwC9gNU1dkks8A54HXgcFXdnkz5kqRhhgZ9Vf0u8MG36P8m8MgixxwF\nji67OknSsnllrCR17m1zr5vernSTpFE5o5ekzhn0ktS5t83Szaj8JB5JvXFGL0mdM+glqXMGvSR1\nzqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcF0xJGosXFa4/zuglqXMGvSR1bpSPEtye5EtJziU5m+Rj\nrf+BJKeSvNIeNw0c81SSC0nOJ3l0kt+AJGlpo8zoXwf+UVXtBj4EHE6yGzgCnK6qXcDptk3bdwB4\niIUPEX+mfQyhJGkVDA36qrpWVS+09h8ALwNbgX3ATBs2Azze2vuAE1V1q6peBS4Ae1a6cEnSaMZa\no0+yg4XPj30O2FxV19qu68Dm1t4KXB447ErrkyStgpGDPsl7gM8CH6+q7wzuq6oCapwnTnIoyVyS\nufn5+XEOlSSNYaSgT/IOFkL+U1X1udZ9I8mWtn8LcLP1XwW2Dxy+rfW9SVUdr6rpqpqempq62/ol\nSUOM8q6bAJ8AXq6qXxzYdRI42NoHgWcH+g8k2ZhkJ7ALOLNyJUuSxjHKlbE/CPwE8LUkL7a+nwOO\nAbNJngQuAfsBqupsklngHAvv2DlcVbdXvHJJ0kiGBn1V/Q6QRXY/ssgxR4Gjy6hLkrRCvDJWkjpn\n0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9\nJHXOoJekzhn0ktQ5g16SOjfKZ8Z+MsnNJC8N9D2Q5FSSV9rjpoF9TyW5kOR8kkcnVbgkaTSjzOj/\nDbD3jr4jwOmq2gWcbtsk2Q0cAB5qxzyTZMOKVStJGtvQoK+qLwPfuqN7HzDT2jPA4wP9J6rqVlW9\nClwA9qxQrZKku3C3a/Sbq+paa18HNrf2VuDywLgrre9PSHIoyVySufn5+bssQ5I0zLJfjK2qAuou\njjteVdNVNT01NbXcMiRJi7jboL+RZAtAe7zZ+q8C2wfGbWt9kqRVcrdBfxI42NoHgWcH+g8k2Zhk\nJ7ALOLO8EiVJy3HfsAFJPg18BHgwyRXgaeAYMJvkSeASsB+gqs4mmQXOAa8Dh6vq9oRqlySNYGjQ\nV9UTi+x6ZJHxR4GjyylKkrRyvDJWkjo3dEYvae3aceQLS+6/eOyxe1SJ1jJn9JLUOYNekjpn0EtS\n51yj17rjurQ0HoNeEuAv0J4Z9HpbM9z0dmDQd2ap4DK0tB6t5V/G6+XnzaDXotbL/8SSlmbQS2vQ\nWp7Fav0x6HVP+NeBtHoM+nXCGZ6ku2XQS1pxTkzWFq+MlaTOOaOXVoizWK1VE5vRJ9mb5HySC0mO\nTOp5JElLm0jQJ9kA/EvgR4HdwBNJdk/iuSRJS5vUjH4PcKGqvlFVfwScAPZN6LkkSUuYVNBvBS4P\nbF9pfZKkeyxVtfJfNPm7wN6q+gdt+yeAv1JVPzUw5hBwqG2+Dzi/Qk//IPD7K/S1VoP1ry7rX13W\nP54/V1VTwwZN6l03V4HtA9vbWt//V1XHgeMr/cRJ5qpqeqW/7r1i/avL+leX9U/GpJZuvgLsSrIz\nyTuBA8DJCT2XJGkJE5nRV9XrSX4K+C1gA/DJqjo7ieeSJC1tYhdMVdVvAL8xqa+/hBVfDrrHrH91\nWf/qsv4JmMiLsZKktcN73UhS57oK+vV+24UkF5N8LcmLSeZWu55hknwyyc0kLw30PZDkVJJX2uOm\n1axxKYvU//NJrrb/Bi8m+ZurWeNikmxP8qUk55KcTfKx1r8uzv8S9a+X8/+uJGeS/NdW/z9p/Wvy\n/HezdNNuu/DfgI+ycIHWV4AnqurcqhY2hiQXgemqWhfvI07yQ8BrwL+tqve3vn8GfKuqjrVftpuq\n6mdXs87FLFL/zwOvVdUvrGZtwyTZAmypqheSfB/wPPA48PdYB+d/ifr3sz7Of4D7q+q1JO8Afgf4\nGPB3WIPnv6cZvbdduMeq6svAt+7o3gfMtPYMCz+8a9Ii9a8LVXWtql5o7T8AXmbh6vN1cf6XqH9d\nqAWvtc13tH/FGj3/PQV9D7ddKOCLSZ5vVw6vR5ur6lprXwc2r2Yxd+mnk/xuW9pZE396LyXJDuCD\nwHOsw/N/R/2wTs5/kg1JXgRuAqeqas2e/56CvgcfrqqHWbjr5+G2tLBu1cK64HpbG/xl4AeAh4Fr\nwD9f3XKWluQ9wGeBj1fVdwb3rYfz/xb1r5vzX1W328/rNmBPkvffsX/NnP+egn7obRfWuqq62h5v\nAp9nYTlqvbnR1l/fWIe9ucr1jKWqbrQf4O8C/4o1/N+grQ1/FvhUVX2uda+b8/9W9a+n8/+Gqvo2\n8CVgL2v0/PcU9Ov6tgtJ7m8vSpHkfuBHgJeWPmpNOgkcbO2DwLOrWMvY3vghbf42a/S/QXsx8BPA\ny1X1iwO71sX5X6z+dXT+p5K8t7XfzcKbQL7OGj3/3bzrBqC9FeuX+OPbLhxd5ZJGluQHWJjFw8IV\ny7+61utP8mngIyzcse8G8DTwH4BZ4M8Cl4D9VbUmX/BcpP6PsLBsUMBF4CcH1lzXjCQfBv4z8DXg\nu63751hY517z53+J+p9gfZz/v8jCi60bWJgwz1bVP03y/azB899V0EuS/qSelm4kSW/BoJekzhn0\nktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+glqXP/D0lOBgHWqBhcAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9b0e5c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run me\n",
    "plt.bar(list(range(news[0].shape[0])), news[0].str.len())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise:__ In the plot above, add a meaningful header for it. What is it about?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explore our text files, one way is to understand how positive/ negative of the tone, and also how subjective is the piece of the text. This is called the __sentiment analysis__. `vader` in `ntlk` is a common library to proceed the sentiment analysis. To do so, you will need to download the vader lexicon (i.e. the dictionary). You will need to run this code \n",
    "```python\n",
    "import nltk\n",
    "nltk.download('vader_lexicon')\n",
    "```\n",
    "and delete the lines afterwards. Your computer has installed the `vader` lexicon and you won't need to use the code above again. \n",
    "\n",
    "After this, you are on your way to proceed with the sentiment analysis with `vader`. Which is done by the code below: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run it\n",
    "from nltk.sentiment import vader\n",
    "senti=vader.SentimentIntensityAnalyzer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second line above is to create a sentiment analysis object, which is the place to analyse and store the results from sentiment analysis. You may wonder why there is no place to put the texts (inputs) above. The reasons is you will do this once you created the object itself. For example, see below: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'neg': 0.0, 'neu': 0.455, 'pos': 0.545, 'compound': 0.6989}\n",
      "{'neg': 0.513, 'neu': 0.487, 'pos': 0.0, 'compound': -0.2755}\n",
      "{'neg': 0.183, 'neu': 0.687, 'pos': 0.13, 'compound': -0.1938}\n"
     ]
    }
   ],
   "source": [
    "# Try yourself\n",
    "print(senti.polarity_scores('I really love what I\\'ve got!'))\n",
    "print(senti.polarity_scores(\"I don't like this\"))\n",
    "print(senti.polarity_scores(\"I liked the course, but I don't like the bits and pieces of it.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's apply the sentiment analysis into the text we after. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps = news[0].apply(senti.polarity_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we can find the negative score of the first row as "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ps[0]['neg']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ Print the negative score of the $30$th row. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "solution": "hidden",
    "solution2": "hidden"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.171"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "ps[29]['neg']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution": "shown",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "__Exercise:__ Print the positive score of the $30$th row. What is your interpretation of this score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution": "shown"
   },
   "outputs": [],
   "source": [
    "# Your code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "solution": "shown",
    "solution2": "hidden"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "ps[29]['pos']\n",
    "\n",
    "# The value is 0.0, it means the 30th row has negative content. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modelling and Evalution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After all the preparation work has done, it is time to plan to make the deliverables. In data analytics, we make __models__. These are the mathematical models. You can think them as a line plot in front of the computer screen. A model is how we conceptualise a concept in a consise way. For example, \n",
    "* We describe how the population of birds grow in a nature reserve through a line plot through times.\n",
    "* After we find out why the sewage system blocked in the whole city, we could draw a map of the sewage system to show the situation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So once we have our datasets on hand, we can start modelling. There are many ways to model from data: \n",
    "* Statistical modelling (See __next week__) \n",
    "* Machine learning (See __week 9 and 10__) \n",
    "* Dynamical equation modelling\n",
    "* Agent based modelling\n",
    "\n",
    "Usually statistical modelling means to find statistical information from a large dataset. These statistical information include mean, median mode values. Or range parameters such as standard deviation or interquartile range. By bringing the context of the data into the measured statistical information, we then conclude from them. \n",
    "\n",
    "Machine learning is set of methods which means the computer finds non-trivial patterns from data. Machine learning is a category of many computational methods, it depends on the dataset and your problem to decide which method is the best. You can check more from [Google](https://developers.google.com/machine-learning/problem-framing/cases) to find out more. More often, you don't need to use machine learning to model your dataset. There is a worksheet from Google to help you whether you need machine learning or not. \n",
    "\n",
    "[https://developers.google.com/machine-learning/problem-framing/framing](https://developers.google.com/machine-learning/problem-framing/framing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"fig/MachineLearning.png\" width=\"620\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of your modelling process. You will need to evaluate your model. This will be mentioned more next week upon the technical sides. Often you will need to measure your model with __model diagnostics__. Therefore, you will able to ask yourself the following questions: \n",
    "* Does my model contribute to what I want?\n",
    "* Is there any way to optimise my computation?\n",
    "* Any surprises (both good and bad) occured when modelling?\n",
    "* Any mistakes and lessons learnt?\n",
    "\n",
    "Think about these questions once we have done next week's work. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of your data analytics cycle, you will be writing summary about your model and then use it in real life. There are many examples: \n",
    "* Recommendation systems on Youtube\n",
    "* API for sports performance prediction \n",
    "* Animation of the spread of bushfires\n",
    "\n",
    "While each dataset is unique, there is always possibilities to its applications. \n",
    "\n",
    "__Exercise:__ Think about what could be the applications from the data analysis we have just made. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you can't think of any applications on top of your head, that is fine. There are many ways to think about how to produce a useful application out of them. We call them as __deliverables__. In reality, people will: \n",
    "1. Summarise the results from their models\n",
    "2. Create a step-by-step plan for deployment\n",
    "3. You may then also wanted to see: \n",
    "    * Any factors or influences needed to follow up? \n",
    "    * Validity and accuracy of each model\n",
    "    * Does the model become unrealistic?\n",
    "\n",
    "After that you will need to write a final report to summarise your cycle. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we finish, do you remember that in the CRISP-DM flowchart, what does the arrows look like? In fact, any business process is never uni-directional. Always remember to review your work. It will help you to detect any mistakes early as possible. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this week, we have looked into the process of data analytics and \n",
    "* How to define a solid purpose for a data science project\n",
    "* How to complete EDA\n",
    "* Tools to conduct data cleaning\n",
    "* Understand data analytics process\n",
    "\n",
    "In the next weeks we will look into the other phases in the CRISP-DM: \n",
    "* Advanced EDA in week 8 \n",
    "* Select and aggregate data in week 9 and 10\n",
    "* Machine learning in weeks 9 and 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Reading\n",
    "\n",
    "* Trilling, D. (2019). _Doing Computational Social Science with Python: An Introduction_. [Online; Accessed on 16th December 2019]\n",
    "* Alvira Swalin (2018). \n",
    "[Different techniques on missing values treatment](https://towardsdatascience.com/how-to-handle-missing-data-8646b18db0d4)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}