notebooks/Goodness of fit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chi-square test\n",
    "```bash\n",
    "by Jeffrey Chijioke-Uche, MSIS, MSIT, CPDS (IBM Sr. Solution Architect, Hybrid Cloud & Multicloud)\n",
    "IBM Information Technology PhD Scholar at Harvard University & Walden University \n",
    "``` \n",
    "\n",
    "## Contents:    \n",
    "1. [Theory](#theory)   \n",
    "2. [Numerical example](#example)\n",
    "3. [Scipy's example](#scipy)\n",
    "\n",
    "\n",
    "<a id='theory'></a>\n",
    "Also called \"Goodness of fit chi square test\".   \n",
    "This is also used to test if observed sample frequencies are consistent with the expected frequencies.   \n",
    "\n",
    "### Assumptions    \n",
    "* The sampling method is simple random sampling.\n",
    "* The variable under study is **categorical**.\n",
    "* The expected value of the number of sample observations in each level of the variable is at least 5.    \n",
    "\n",
    "### Hypothesis    \n",
    "**Null hypothesis**: observed and expected values are consistent    \n",
    "**Alternative hypothesis**: observed and expected values are significantly different    \n",
    "\n",
    "### Test statistic    \n",
    "The test statistic (CV) is computed as    \n",
    "\n",
    "$\\chi^2 = \\sum_i (O_i - E_i)^2/E_i$\n",
    "\n",
    "where $O_i$ and $E_i$ are observed and expected frequency counts. \n",
    "\n",
    "### P-value   \n",
    "after computing CV and degrees of freedom ($= n - 1$), \n",
    "we estimate the probability of P(chi2 $\\leq$ CV).    \n",
    "This is done using the CDF for chi2, as shown in this notebook.    \n",
    "\n",
    "**Note** $\\chi^2$ is the ratio between non negative values, so we need to shift our data to the positive scale before doing chi-square tests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='example'></a>   \n",
    "### Toy exercise  \n",
    "\n",
    "Let's reproduce the example in [stattrek](http://stattrek.com/chi-square-test/goodness-of-fit.aspx?Tutorial=AP):   \n",
    "\n",
    ">Acme Toy Company prints baseball cards. The company claims that 30% of the cards are rookies, 60% veterans, and 10% are All-Stars.\n",
    "\n",
    ">Suppose a random sample of 100 cards has 50 rookies, 45 veterans, and 5 All-Stars. Is this consistent with Acme's claim? Use a 0.05 level of significance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's evaluate with the Goodness of fit, whether the observed and expected sets are significantly different.    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observed proportions [50 45  5]\n",
      "Expected proportions [30 60 10]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Observed proportions:   \n",
    "O=np.array([50,45,5])\n",
    "# Expected proportions:\n",
    "E=np.array([30,60,10])\n",
    "\n",
    "print('Observed proportions',O)\n",
    "print('Expected proportions',E)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Statistic 19.5833333333\n",
      "Critical value for 95% confidence 5.99146454711\n",
      "P value     0.000056\n"
     ]
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\n",
    "# Statistic   \n",
    "cv = np.sum((O-E)**2/E)\n",
    "print(\"Statistic\",cv)\n",
    "df=len(O)-1\n",
    "crit = stats.chi2.ppf(q = 0.95, # Find the critical value for 95% confidence*\n",
    "                      df = df)   # *\n",
    "\n",
    "\n",
    "print(\"Critical value for 95% confidence\",crit)\n",
    "\n",
    "p_value = float(1.0 - stats.chi2.cdf(x=cv, df=df))\n",
    "print(\"P value %12.6f\" % p_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our test statistic is lower than the critical value for 95% confidence, so we cannot accept the null hypothesis that the two samples are consistent.    \n",
    "Note that the P-value is close to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='scipy'></a>   \n",
    "### Scipy's $\\chi^2$ test\n",
    "\n",
    "Let's do the same test, using the scipy's implementation of $\\chi^2$ test\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Power_divergenceResult(statistic=19.583333333333336, pvalue=5.5915626856371765e-05)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.chisquare(f_obs = O,f_exp = E)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}