NumPy and Pandas for 1D and 2D Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NumPy and Pandas for One Dimensional Data  \n",
    "\n",
    "## Overview\n",
    "### **NumPy (Numerical Python)**  \n",
    "- Called an array  \n",
    "- Simpler  \n",
    "\n",
    "### **Pandas**  \n",
    "- Called a series   \n",
    "- More features  \n",
    "- Built on NumPy Arrays\n",
    "\n",
    "### Similarities\n",
    "- Access elements by position, i.e. a[0]  \n",
    "- Access a range of elements, i.e. a[1:3]  \n",
    "- Use loops, i.e., for x in a:  \n",
    "\n",
    "### Differences\n",
    "- NumPy array: each element should have the same type  \n",
    "- NumPy includes a bunch of convenient functions  \n",
    "- NumPy arrays can be multi-dimensional  \n",
    "\n",
    "---\n",
    "\n",
    "## NumPy Array\n",
    "\n",
    "NumPy Arrays are like souped-up Python lists\n",
    "\n",
    "### Vectorized Operations  \n",
    "A vector is a list of numbers   \n",
    "NumPy supports Vectorized Operations between   \n",
    "- 2 NumPy arrays or   \n",
    "- 1 Numpy array and a single number  \n",
    "\n",
    "**Vector Addition**\n",
    "- NumPy: When you add two NumPy arrays, each element in the array is added together\n",
    "- Python: When you add two lists together in Python, we get list concatentation (New list w/ all elements of the 1st list followed by all elements of the 2nd list) \n",
    "\n",
    "**Multipyling by a Scaler **  \n",
    "- NumPy Arrays: Each element of the vector is multipled by 3  \n",
    "- Python: creates a list with all of the values of the original list repeated 3 times\n",
    "\n",
    "### **More Vectorized Operations**   \n",
    "\n",
    "**Math Operations**  \n",
    "Add, Subtract, Multiply, Divide, Exponentiate    \n",
    "\n",
    "**Logical Operations **  \n",
    "Make sure your arrays contain booleans  \n",
    "And & , Or | , Not ~    \n",
    "\n",
    "**Comparison Operations ** \n",
    "Greater: >  \n",
    "Greater or equal >=  \n",
    "Less <  \n",
    "Less <=  \n",
    "Equal ==    \n",
    "Not equal !=   \n",
    "\n",
    "### Slicing\n",
    "\n",
    "When slicing NumPy array just creates a different view of the original array (not a copy!) So when you make changes, it will be reflected in the original array.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "## Pandas Series  \n",
    "A series is similar to a NumPy array, but with extra functionality, e.g. s.describe()\n",
    "\n",
    "All the things we just learned how to do on NumPy arrays will also work on Pandas Series  \n",
    "- Accessing elements s[0], s[3:7]  \n",
    "- Looping for x in s  \n",
    "- Convenient functions, i.e. s.mean(), s.max()    \n",
    "- Vectorized Operations s1 + s2    \n",
    "- Implemented in C (fast!)    \n",
    "\n",
    "A Pandas series is like a cross between a list and a dictionary  \n",
    "- lists are stored in order and can be accessed by position\n",
    "- dictionary have a key and value, and you can look up values by key  \n",
    "\n",
    ".iloc access code by position  \n",
    ".loc lets you look up values by index label  \n",
    "\n",
    "If you create a series without specifying an index, then it will automaticallyl assign one starting with 0\n",
    "\n",
    "### Vectorized Operations and Series Indexes  \n",
    "When you add two Numpy arrays, you're adding by position  \n",
    "\n",
    "When you add Pandas series, you're adding by index (not position)\n",
    "- indexes can be in different order.\n",
    "- if there are indexes present in one series and not the other, get NaN\n",
    "\n",
    "### For Non Built-In Calculations  \n",
    "1. Treat the series as a list (for loops, etc)\n",
    "2. Use the function apply()  \n",
    "apply() takes a series and a function and returns a new series  \n",
    "\n",
    "Similar to python function map() but apply works on series instead of lists  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "# NumPy and Pandas for Two Dimensional Data\n",
    "  \n",
    "### **Python:** List of lists    \n",
    "- accessing elements a[1][3]  \n",
    "\n",
    "### **NumPy: **2D Array\n",
    "- more memory efficient than lists\n",
    "- accessing elements is a bit different, i.e. a[1,3] \n",
    "- mean(), std(), operate on an entire array as a whole\n",
    " - To do operations along an axis (row or column)  \n",
    "     - ridership.mean(axis = 0) function on each row\n",
    "     - ridership.mean(axis = 1) function for each column \n",
    " \n",
    " \n",
    "### **Pandas:** Dataframe \n",
    "- preferred because more functionality  \n",
    "- have indexes similar to Pandas Series\n",
    "    - Can access elements using both indexes, i.e., ridership_df.loc['05-04-11', 'R004']\n",
    "- great data structure to represent CSV file because it is a 2D data structure with a different type for each column  \n",
    "- each column is assumed to a different type so operations like .mean are performed for each column automatically\n",
    "- DataFrame vectorized operations are similar to vectorized operations for 2D NumPy arrays  \n",
    "    - But match up elements by index and column name rather than position  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "subway_df = pd.read_csv('nyc_subway_weather.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UNIT</th>\n",
       "      <th>DATEn</th>\n",
       "      <th>TIMEn</th>\n",
       "      <th>ENTRIESn</th>\n",
       "      <th>EXITSn</th>\n",
       "      <th>ENTRIESn_hourly</th>\n",
       "      <th>EXITSn_hourly</th>\n",
       "      <th>datetime</th>\n",
       "      <th>hour</th>\n",
       "      <th>day_week</th>\n",
       "      <th>...</th>\n",
       "      <th>pressurei</th>\n",
       "      <th>rain</th>\n",
       "      <th>tempi</th>\n",
       "      <th>wspdi</th>\n",
       "      <th>meanprecipi</th>\n",
       "      <th>meanpressurei</th>\n",
       "      <th>meantempi</th>\n",
       "      <th>meanwspdi</th>\n",
       "      <th>weather_lat</th>\n",
       "      <th>weather_lon</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>R003</td>\n",
       "      <td>05-01-11</td>\n",
       "      <td>00:00:00</td>\n",
       "      <td>4388333</td>\n",
       "      <td>2911002</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2011-05-01 00:00:00</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>30.22</td>\n",
       "      <td>0</td>\n",
       "      <td>55.9</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.258</td>\n",
       "      <td>55.98</td>\n",
       "      <td>7.86</td>\n",
       "      <td>40.700348</td>\n",
       "      <td>-73.887177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>R003</td>\n",
       "      <td>05-01-11</td>\n",
       "      <td>04:00:00</td>\n",
       "      <td>4388333</td>\n",
       "      <td>2911002</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2011-05-01 04:00:00</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>30.25</td>\n",
       "      <td>0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.258</td>\n",
       "      <td>55.98</td>\n",
       "      <td>7.86</td>\n",
       "      <td>40.700348</td>\n",
       "      <td>-73.887177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>R003</td>\n",
       "      <td>05-01-11</td>\n",
       "      <td>12:00:00</td>\n",
       "      <td>4388333</td>\n",
       "      <td>2911002</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2011-05-01 12:00:00</td>\n",
       "      <td>12</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>30.28</td>\n",
       "      <td>0</td>\n",
       "      <td>62.1</td>\n",
       "      <td>6.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.258</td>\n",
       "      <td>55.98</td>\n",
       "      <td>7.86</td>\n",
       "      <td>40.700348</td>\n",
       "      <td>-73.887177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>R003</td>\n",
       "      <td>05-01-11</td>\n",
       "      <td>16:00:00</td>\n",
       "      <td>4388333</td>\n",
       "      <td>2911002</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2011-05-01 16:00:00</td>\n",
       "      <td>16</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>30.26</td>\n",
       "      <td>0</td>\n",
       "      <td>57.9</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.258</td>\n",
       "      <td>55.98</td>\n",
       "      <td>7.86</td>\n",
       "      <td>40.700348</td>\n",
       "      <td>-73.887177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>R003</td>\n",
       "      <td>05-01-11</td>\n",
       "      <td>20:00:00</td>\n",
       "      <td>4388333</td>\n",
       "      <td>2911002</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2011-05-01 20:00:00</td>\n",
       "      <td>20</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>30.28</td>\n",
       "      <td>0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>10.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.258</td>\n",
       "      <td>55.98</td>\n",
       "      <td>7.86</td>\n",
       "      <td>40.700348</td>\n",
       "      <td>-73.887177</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   UNIT     DATEn     TIMEn  ENTRIESn   EXITSn  ENTRIESn_hourly  \\\n",
       "0  R003  05-01-11  00:00:00   4388333  2911002              0.0   \n",
       "1  R003  05-01-11  04:00:00   4388333  2911002              0.0   \n",
       "2  R003  05-01-11  12:00:00   4388333  2911002              0.0   \n",
       "3  R003  05-01-11  16:00:00   4388333  2911002              0.0   \n",
       "4  R003  05-01-11  20:00:00   4388333  2911002              0.0   \n",
       "\n",
       "   EXITSn_hourly             datetime  hour  day_week     ...       pressurei  \\\n",
       "0            0.0  2011-05-01 00:00:00     0         6     ...           30.22   \n",
       "1            0.0  2011-05-01 04:00:00     4         6     ...           30.25   \n",
       "2            0.0  2011-05-01 12:00:00    12         6     ...           30.28   \n",
       "3            0.0  2011-05-01 16:00:00    16         6     ...           30.26   \n",
       "4            0.0  2011-05-01 20:00:00    20         6     ...           30.28   \n",
       "\n",
       "  rain  tempi  wspdi meanprecipi  meanpressurei  meantempi  meanwspdi  \\\n",
       "0    0   55.9    3.5         0.0         30.258      55.98       7.86   \n",
       "1    0   52.0    3.5         0.0         30.258      55.98       7.86   \n",
       "2    0   62.1    6.9         0.0         30.258      55.98       7.86   \n",
       "3    0   57.9   15.0         0.0         30.258      55.98       7.86   \n",
       "4    0   52.0   10.4         0.0         30.258      55.98       7.86   \n",
       "\n",
       "   weather_lat  weather_lon  \n",
       "0    40.700348   -73.887177  \n",
       "1    40.700348   -73.887177  \n",
       "2    40.700348   -73.887177  \n",
       "3    40.700348   -73.887177  \n",
       "4    40.700348   -73.887177  \n",
       "\n",
       "[5 rows x 27 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subway_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENTRIESn</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>2.812486e+07</td>\n",
       "      <td>3.043607e+07</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.039762e+07</td>\n",
       "      <td>1.818389e+07</td>\n",
       "      <td>3.263049e+07</td>\n",
       "      <td>2.357746e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EXITSn</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>1.986993e+07</td>\n",
       "      <td>2.028986e+07</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.613712e+06</td>\n",
       "      <td>1.331609e+07</td>\n",
       "      <td>2.393771e+07</td>\n",
       "      <td>1.493782e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENTRIESn_hourly</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>1.886590e+03</td>\n",
       "      <td>2.952386e+03</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.740000e+02</td>\n",
       "      <td>9.050000e+02</td>\n",
       "      <td>2.255000e+03</td>\n",
       "      <td>3.281400e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EXITSn_hourly</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>1.361488e+03</td>\n",
       "      <td>2.183845e+03</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.370000e+02</td>\n",
       "      <td>6.640000e+02</td>\n",
       "      <td>1.537000e+03</td>\n",
       "      <td>3.482800e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hour</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>1.004675e+01</td>\n",
       "      <td>6.938928e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.000000e+00</td>\n",
       "      <td>1.200000e+01</td>\n",
       "      <td>1.600000e+01</td>\n",
       "      <td>2.000000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day_week</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>2.905719e+00</td>\n",
       "      <td>2.079231e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>3.000000e+00</td>\n",
       "      <td>5.000000e+00</td>\n",
       "      <td>6.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weekday</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>7.144364e-01</td>\n",
       "      <td>4.516877e-01</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>latitude</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>4.072465e+01</td>\n",
       "      <td>7.164979e-02</td>\n",
       "      <td>40.576152</td>\n",
       "      <td>4.067711e+01</td>\n",
       "      <td>4.071724e+01</td>\n",
       "      <td>4.075912e+01</td>\n",
       "      <td>4.088918e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>longitude</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>-7.394036e+01</td>\n",
       "      <td>5.971260e-02</td>\n",
       "      <td>-74.073622</td>\n",
       "      <td>-7.398734e+01</td>\n",
       "      <td>-7.395346e+01</td>\n",
       "      <td>-7.390773e+01</td>\n",
       "      <td>-7.375538e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fog</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>9.824380e-03</td>\n",
       "      <td>9.863108e-02</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>precipi</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>4.617693e-03</td>\n",
       "      <td>2.583151e-02</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>3.000000e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pressurei</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>2.997110e+01</td>\n",
       "      <td>1.379416e-01</td>\n",
       "      <td>29.550000</td>\n",
       "      <td>2.989000e+01</td>\n",
       "      <td>2.996000e+01</td>\n",
       "      <td>3.006000e+01</td>\n",
       "      <td>3.032000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rain</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>2.247415e-01</td>\n",
       "      <td>4.174169e-01</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tempi</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>6.310378e+01</td>\n",
       "      <td>8.455597e+00</td>\n",
       "      <td>46.900000</td>\n",
       "      <td>5.700000e+01</td>\n",
       "      <td>6.100000e+01</td>\n",
       "      <td>6.910000e+01</td>\n",
       "      <td>8.600000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wspdi</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>6.927872e+00</td>\n",
       "      <td>4.510178e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.600000e+00</td>\n",
       "      <td>6.900000e+00</td>\n",
       "      <td>9.200000e+00</td>\n",
       "      <td>2.300000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>meanprecipi</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>4.617693e-03</td>\n",
       "      <td>1.634424e-02</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.575000e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>meanpressurei</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>2.997110e+01</td>\n",
       "      <td>1.311584e-01</td>\n",
       "      <td>29.590000</td>\n",
       "      <td>2.991333e+01</td>\n",
       "      <td>2.995800e+01</td>\n",
       "      <td>3.006000e+01</td>\n",
       "      <td>3.029333e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>meantempi</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>6.310378e+01</td>\n",
       "      <td>6.939011e+00</td>\n",
       "      <td>49.400000</td>\n",
       "      <td>5.828333e+01</td>\n",
       "      <td>6.095000e+01</td>\n",
       "      <td>6.746667e+01</td>\n",
       "      <td>7.980000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>meanwspdi</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>6.927872e+00</td>\n",
       "      <td>3.179832e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.816667e+00</td>\n",
       "      <td>6.166667e+00</td>\n",
       "      <td>8.850000e+00</td>\n",
       "      <td>1.708333e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weather_lat</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>4.072856e+01</td>\n",
       "      <td>6.542018e-02</td>\n",
       "      <td>40.600204</td>\n",
       "      <td>4.068859e+01</td>\n",
       "      <td>4.072057e+01</td>\n",
       "      <td>4.075523e+01</td>\n",
       "      <td>4.086206e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weather_lon</th>\n",
       "      <td>42649.0</td>\n",
       "      <td>-7.393869e+01</td>\n",
       "      <td>5.958245e-02</td>\n",
       "      <td>-74.014870</td>\n",
       "      <td>-7.398513e+01</td>\n",
       "      <td>-7.394915e+01</td>\n",
       "      <td>-7.391203e+01</td>\n",
       "      <td>-7.369418e+01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   count          mean           std        min           25%  \\\n",
       "ENTRIESn         42649.0  2.812486e+07  3.043607e+07   0.000000  1.039762e+07   \n",
       "EXITSn           42649.0  1.986993e+07  2.028986e+07   0.000000  7.613712e+06   \n",
       "ENTRIESn_hourly  42649.0  1.886590e+03  2.952386e+03   0.000000  2.740000e+02   \n",
       "EXITSn_hourly    42649.0  1.361488e+03  2.183845e+03   0.000000  2.370000e+02   \n",
       "hour             42649.0  1.004675e+01  6.938928e+00   0.000000  4.000000e+00   \n",
       "day_week         42649.0  2.905719e+00  2.079231e+00   0.000000  1.000000e+00   \n",
       "weekday          42649.0  7.144364e-01  4.516877e-01   0.000000  0.000000e+00   \n",
       "latitude         42649.0  4.072465e+01  7.164979e-02  40.576152  4.067711e+01   \n",
       "longitude        42649.0 -7.394036e+01  5.971260e-02 -74.073622 -7.398734e+01   \n",
       "fog              42649.0  9.824380e-03  9.863108e-02   0.000000  0.000000e+00   \n",
       "precipi          42649.0  4.617693e-03  2.583151e-02   0.000000  0.000000e+00   \n",
       "pressurei        42649.0  2.997110e+01  1.379416e-01  29.550000  2.989000e+01   \n",
       "rain             42649.0  2.247415e-01  4.174169e-01   0.000000  0.000000e+00   \n",
       "tempi            42649.0  6.310378e+01  8.455597e+00  46.900000  5.700000e+01   \n",
       "wspdi            42649.0  6.927872e+00  4.510178e+00   0.000000  4.600000e+00   \n",
       "meanprecipi      42649.0  4.617693e-03  1.634424e-02   0.000000  0.000000e+00   \n",
       "meanpressurei    42649.0  2.997110e+01  1.311584e-01  29.590000  2.991333e+01   \n",
       "meantempi        42649.0  6.310378e+01  6.939011e+00  49.400000  5.828333e+01   \n",
       "meanwspdi        42649.0  6.927872e+00  3.179832e+00   0.000000  4.816667e+00   \n",
       "weather_lat      42649.0  4.072856e+01  6.542018e-02  40.600204  4.068859e+01   \n",
       "weather_lon      42649.0 -7.393869e+01  5.958245e-02 -74.014870 -7.398513e+01   \n",
       "\n",
       "                          50%           75%           max  \n",
       "ENTRIESn         1.818389e+07  3.263049e+07  2.357746e+08  \n",
       "EXITSn           1.331609e+07  2.393771e+07  1.493782e+08  \n",
       "ENTRIESn_hourly  9.050000e+02  2.255000e+03  3.281400e+04  \n",
       "EXITSn_hourly    6.640000e+02  1.537000e+03  3.482800e+04  \n",
       "hour             1.200000e+01  1.600000e+01  2.000000e+01  \n",
       "day_week         3.000000e+00  5.000000e+00  6.000000e+00  \n",
       "weekday          1.000000e+00  1.000000e+00  1.000000e+00  \n",
       "latitude         4.071724e+01  4.075912e+01  4.088918e+01  \n",
       "longitude       -7.395346e+01 -7.390773e+01 -7.375538e+01  \n",
       "fog              0.000000e+00  0.000000e+00  1.000000e+00  \n",
       "precipi          0.000000e+00  0.000000e+00  3.000000e-01  \n",
       "pressurei        2.996000e+01  3.006000e+01  3.032000e+01  \n",
       "rain             0.000000e+00  0.000000e+00  1.000000e+00  \n",
       "tempi            6.100000e+01  6.910000e+01  8.600000e+01  \n",
       "wspdi            6.900000e+00  9.200000e+00  2.300000e+01  \n",
       "meanprecipi      0.000000e+00  0.000000e+00  1.575000e-01  \n",
       "meanpressurei    2.995800e+01  3.006000e+01  3.029333e+01  \n",
       "meantempi        6.095000e+01  6.746667e+01  7.980000e+01  \n",
       "meanwspdi        6.166667e+00  8.850000e+00  1.708333e+01  \n",
       "weather_lat      4.072057e+01  4.075523e+01  4.086206e+01  \n",
       "weather_lon     -7.394915e+01 -7.391203e+01 -7.369418e+01  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subway_df.describe().transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create function to calculate Pearson's R\n",
    "def correlation(x,y):\n",
    "    std_x = (x - x.mean())/x.std(ddof = 0)\n",
    "    std_y = (y - y.mean())/y.std(ddof = 0)\n",
    "    \n",
    "    return (std_x * std_y).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.03564851577223041"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "correlation(subway_df['ENTRIESn_hourly'], subway_df['meanprecipi'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5858954707662182"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "correlation(subway_df['ENTRIESn_hourly'],subway_df['ENTRIESn'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}