03-Lab-Solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lab: Analytics Exercise (Solution)\n",
    "\n",
    "In this mini-lab, you'll get a chance to create a Dask cluster and run a few queries on some Wikimedia traffic data, using Dask dataframe.\n",
    "\n",
    "*Hint: Copy useful code snippets from the intro notebook.*\n",
    "\n",
    "__1. Create a Client__ and request 2 workers, 1 thread, and 1GB of RAM each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask.distributed import Client\n",
    "\n",
    "client = Client(n_workers=2, threads_per_worker=1, memory_limit='1GB')\n",
    "\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__2. Read data__ from the `pageviews_small.csv` file. Use Dask's `blocksize=` parameter to set each partition to max of 10 MB.\n",
    "\n",
    "*Hint: use Pandas' sep parameter to indicate that the columns are space-separated*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.dataframe\n",
    "\n",
    "ddf = dask.dataframe.read_csv('data/pageviews_small.csv', sep=' ', blocksize=10e6)\n",
    "\n",
    "ddf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__3. Change the column names__ to `project`, `page`, `requests`, and `x` then drop the `x` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.columns = ['project', 'page', 'requests', 'x']\n",
    "\n",
    "ddf2 = ddf.drop('x', axis=1)\n",
    "\n",
    "ddf2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__4. Filter__ for `project` matching \"en\" (English Wikipedia)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf3 = ddf2[ddf2.project == 'en']\n",
    "ddf3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__5. Count__ how many pages were accessed from English Wikipedia vs. all projects in this dataset. (Note: each project/page combination appears on a unique line, so this amounts to just counting records)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf2.count().compute() #all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf3.count().compute() #English"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Extra Credit: 6. What are the record counts__ for English (en), French (fr), Chinese (zh), and Polish (pl)?\n",
    "\n",
    "*Hint: `isin` isn't supported on the Dask dataframe index, but you can `reset_index` to move the `project` into a \"regular\" column and use `isin` on that*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf4 = ddf2.groupby('project').count().reset_index()\n",
    "\n",
    "ddf4[ddf4.project.isin(['en', 'fr', 'zh', 'pl'])].compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}