ab-testing-chi2-test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grocery Sign-up Rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Scenario\n",
    "\n",
    "#### In late June, we mailed out promotional content to some of our customers for our \"Delivery Club\" Signing up to the club cost dollars and gave customers free grocery deliveries for one year, starting on July 1st.\n",
    "#### We wanted to see if a nicer looking mailer would get more customers to sign-up. We randomly selected 3 groups, the first group received Mailer 1, a very basic and cheap version simple outlining the deal, the second group received Mailer 2, a really nice, colourful mailer printed on cardboard, and the third group were the control group and received to mailer at all.\n",
    "#### We're very confident that the customers who were mailed, signed up at a far higher rate than the control group... but we're unsure if there is a significant difference between signup-rate for the cheaper mailer and the more expensive mailer. At first glance, there does appear to be a slightly higher sign-up rate for the expensive version but before making any conclusions we wanted you to run your eyes over the numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Task\n",
    "\n",
    "#### Assess whether there is a difference between mailer 1 and mailer 2 in terms of sign-up rate to the club"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Required Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from scipy.stats import chi2_contingency, chi2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer_id</th>\n",
       "      <th>campaign_name</th>\n",
       "      <th>campaign_date</th>\n",
       "      <th>mailer_type</th>\n",
       "      <th>signup_flag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>74</td>\n",
       "      <td>delivery_club</td>\n",
       "      <td>2020-07-01</td>\n",
       "      <td>Mailer1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>524</td>\n",
       "      <td>delivery_club</td>\n",
       "      <td>2020-07-01</td>\n",
       "      <td>Mailer1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>607</td>\n",
       "      <td>delivery_club</td>\n",
       "      <td>2020-07-01</td>\n",
       "      <td>Mailer2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>343</td>\n",
       "      <td>delivery_club</td>\n",
       "      <td>2020-07-01</td>\n",
       "      <td>Mailer1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>322</td>\n",
       "      <td>delivery_club</td>\n",
       "      <td>2020-07-01</td>\n",
       "      <td>Mailer2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customer_id  campaign_name campaign_date mailer_type  signup_flag\n",
       "0           74  delivery_club    2020-07-01     Mailer1            1\n",
       "1          524  delivery_club    2020-07-01     Mailer1            1\n",
       "2          607  delivery_club    2020-07-01     Mailer2            1\n",
       "3          343  delivery_club    2020-07-01     Mailer1            0\n",
       "4          322  delivery_club    2020-07-01     Mailer2            1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "campaign_data = pd.read_excel(\"grocery_database.xlsx\", sheet_name = \"campaign_data\")\n",
    "campaign_data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(870, 5)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check shape of dataframe\n",
    "campaign_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter data\n",
    "campaign_data = campaign_data.loc[campaign_data[\"mailer_type\"] != \"Control\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(711, 5)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check shape of updated dataframe\n",
    "campaign_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "customer_id      711\n",
       "campaign_name    711\n",
       "campaign_date    711\n",
       "mailer_type      711\n",
       "signup_flag      711\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "campaign_data.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mailer_type  signup_flag\n",
       "Mailer1      0              252\n",
       "             1              123\n",
       "Mailer2      0              209\n",
       "             1              127\n",
       "Name: signup_flag, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check number of sign-ups\n",
    "campaign_data.groupby(\"mailer_type\")[\"signup_flag\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[252, 123],\n",
       "       [209, 127]], dtype=int64)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "observed_values = pd.crosstab(campaign_data[\"mailer_type\"], campaign_data[\"signup_flag\"]).values\n",
    "observed_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.328 0.37797619047619047\n"
     ]
    }
   ],
   "source": [
    "# compute sign-up rate\n",
    "mailer1_signup_rate = 123 / (252 + 123) \n",
    "mailer2_signup_rate = 127 / (209 + 127)\n",
    "print(mailer1_signup_rate, mailer2_signup_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## State hypotheses & set acceptance criteria"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "null_hypothesis = \"There is no relationship between mailer type and signup rate. They are independent\"\n",
    "alternate_hypothesis = \"There is a relationship between mailer type and signup rate. They are not independent\"\n",
    "acceptance_criteria = 0.05"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate expected frequencies & chi square statistic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.9414468614812481 0.16351152223398197\n"
     ]
    }
   ],
   "source": [
    "chi2_statistic, p_value, dof, expected_values = chi2_contingency(observed_values, correction = False)\n",
    "print(chi2_statistic, p_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.841458820694124\n"
     ]
    }
   ],
   "source": [
    "# find the critical value for our test\n",
    "critical_value = chi2.ppf(1 - acceptance_criteria, dof)\n",
    "print(critical_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Chi Square Statistic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As our chi-square statistic of 1.9414468614812481 is lower than our critical value of 3.841458820694124 - we retain the null hypothesis, and conclude that: There is no relationship between mailer type and signup rate. They are independent.\n"
     ]
    }
   ],
   "source": [
    "if chi2_statistic >= critical_value:\n",
    "    print(f\"As our chi-square statistic of {chi2_statistic} is higher than our critical value of {critical_value} - we reject the null hypothesis, and conclude that: {alternate_hypothesis}.\")\n",
    "else:\n",
    "    print(f\"As our chi-square statistic of {chi2_statistic} is lower than our critical value of {critical_value} - we retain the null hypothesis, and conclude that: {null_hypothesis}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### p_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As our p_value of 0.16351152223398197 is higher than our acceptance_criteria of 0.05 - we retain the null hypothesis, and conclude that: There is no relationship between mailer type and signup rate. They are independent.\n"
     ]
    }
   ],
   "source": [
    "if p_value <= acceptance_criteria:\n",
    "    print(f\"As our p_value of {p_value} is lower than our acceptance_criteria of {acceptance_criteria} - we reject the null hypothesis, and conclude that: {alternate_hypothesis}.\")\n",
    "else:\n",
    "    print(f\"As our p_value of {p_value} is higher than our acceptance_criteria of {acceptance_criteria} - we retain the null hypothesis, and conclude that: {null_hypothesis}.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:my_env] *",
   "language": "python",
   "name": "conda-env-my_env-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}