getting_started_notebooks/intro_tutorials/07_Introduction_to_XGBoost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"introduction\"></a>\n",
    "## Introduction to XGBoost\n",
    "#### By Paul Hendricks\n",
    "-------\n",
    "\n",
    "In this notebook, we will show how to work with GPU accelerated XGBoost in RAPIDS.\n",
    "\n",
    "**Table of Contents**\n",
    "\n",
    "* [Introduction to XGBoost](#introduction)\n",
    "* [Setup](#setup)\n",
    "* [Load Libraries](#libraries)\n",
    "* [Generate Data](#generate)\n",
    "  * [Load Data](#load)\n",
    "  * [Simulate Data](#simulate)\n",
    "  * [Split Data](#split)\n",
    "  * [Check Dimensions](#check)\n",
    "* [Convert NumPy data to DMatrix format](#convert)\n",
    "* [Set Parameters](#parameters)\n",
    "* [Train Model](#train)\n",
    "* [Conclusion](#conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## Setup\n",
    "\n",
    "This notebook was tested using the following Docker containers:\n",
    "\n",
    "* `rapidsai/rapidsai-dev-nightly:0.10-cuda10.0-devel-ubuntu18.04-py3.7` container from [DockerHub](https://hub.docker.com/r/rapidsai/rapidsai-nightly)\n",
    "\n",
    "This notebook was run on the NVIDIA GV100 GPU. Please be aware that your system may be different and you may need to modify the code or install packages to run the below examples. \n",
    "\n",
    "If you think you have found a bug or an error, please file an issue here: https://github.com/rapidsai/notebooks-contrib/issues\n",
    "\n",
    "Before we begin, let's check out our hardware setup by running the `nvidia-smi` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:38.237293Z",
     "start_time": "2018-11-06T21:03:37.388285Z"
    }
   },
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's see what CUDA version we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:39.490984Z",
     "start_time": "2018-11-06T21:03:39.134608Z"
    }
   },
   "outputs": [],
   "source": [
    "!nvcc --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"libraries\"></a>\n",
    "## Load Libraries\n",
    "\n",
    "Let's load some of the libraries within the RAPIDs ecosystem and see which versions we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:41.067879Z",
     "start_time": "2018-11-06T21:03:40.256654Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np; print('numpy Version:', np.__version__)\n",
    "import pandas as pd; print('pandas Version:', pd.__version__)\n",
    "import sklearn; print('Scikit-Learn Version:', sklearn.__version__)\n",
    "import xgboost as xgb; print('XGBoost Version:', xgb.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"generate\"></a>\n",
    "## Generate Data\n",
    "\n",
    "<a id=\"load\"></a>\n",
    "### Load Data\n",
    "\n",
    "We can load the data using `pandas.read_csv`. We've provided a helper function `load_data` that will load data from a CSV file (and will only read the first 1 billion rows if that file is unreasonably big)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper function for loading data\n",
    "def load_data(filename, n_rows):\n",
    "    if n_rows >= 1e9:\n",
    "        df = pd.read_csv(filename)\n",
    "    else:\n",
    "        df = pd.read_csv(filename, nrows=n_rows)\n",
    "    return df.values.astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"simulate\"></a>\n",
    "### Simulate Data\n",
    "\n",
    "Alternatively, we can simulate data for our train and validation datasets. The features will be tabular with `n_rows` and `n_columns` in the training dataset, where each value is either of type `np.float32`. We can simulate data for both classification and regression using the `make_classification` or `make_regression` functions from the Scikit-Learn package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification, make_regression\n",
    "\n",
    "\n",
    "# helper function for simulating data\n",
    "def simulate_data(m, n, k=2, random_state=None, classification=True):\n",
    "    if classification:\n",
    "        features, labels = make_classification(n_samples=m, n_features=n, \n",
    "                                               n_informative=int(n/5), n_classes=k, \n",
    "                                              random_state=random_state)\n",
    "    else:\n",
    "        features, labels = make_regression(n_samples=m, n_features=n, \n",
    "                                           n_informative=int(n/5), n_targets=1, \n",
    "                                           random_state=random_state)\n",
    "    return np.c_[labels, features].astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# settings\n",
    "simulate = True\n",
    "classification = True  # change this to false to use regression\n",
    "n_rows = int(1e6)  # we'll use 1 millions rows\n",
    "n_columns = int(100)\n",
    "n_categories = 2\n",
    "random_state = np.random.RandomState(43210)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "if simulate:\n",
    "    dataset = simulate_data(n_rows, n_columns, n_categories, \n",
    "                            random_state=random_state, \n",
    "                            classification=classification)\n",
    "else:\n",
    "    dataset = load_data('/tmp', n_rows)\n",
    "print(dataset.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"split\"></a>\n",
    "### Split Data\n",
    "\n",
    "We'll split our dataset into a 80% training dataset and a 20% validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# identify shape and indices\n",
    "n_rows, n_columns = dataset.shape\n",
    "train_size = 0.80\n",
    "train_index = int(n_rows * train_size)\n",
    "\n",
    "# split X, y\n",
    "X, y = dataset[:, 1:], dataset[:, 0]\n",
    "del dataset\n",
    "\n",
    "# split train data\n",
    "X_train, y_train = X[:train_index, :], y[:train_index]\n",
    "\n",
    "# split validation data\n",
    "X_validation, y_validation = X[train_index:, :], y[train_index:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"check\"></a>\n",
    "### Check Dimensions\n",
    "\n",
    "We can check the dimensions and proportions of our training and validation dataets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check dimensions\n",
    "print('X_train: ', X_train.shape, X_train.dtype, 'y_train: ', y_train.shape, y_train.dtype)\n",
    "print('X_validation', X_validation.shape, X_validation.dtype, 'y_validation: ', y_validation.shape, y_validation.dtype)\n",
    "\n",
    "# check the proportions\n",
    "total = X_train.shape[0] + X_validation.shape[0]\n",
    "print('X_train proportion:', X_train.shape[0] / total)\n",
    "print('X_validation proportion:', X_validation.shape[0] / total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"convert\"></a>\n",
    "## Convert NumPy data to DMatrix format\n",
    "\n",
    "With out data loaded and formatted as NumPy arrays, our next step is to convert this to a `DMatrix` object that XGBoost can work with. We can instantiate an object of the `xgboost.DMatrix` by passing in the feature matrix as the first argument followed by the label vector using the `label=` keyword argument. To learn more about XGBoost's support for data structures other than NumPy arrays, see the documentation for the Data Interface:\n",
    "\n",
    "\n",
    "https://xgboost.readthedocs.io/en/latest/python/python_intro.html#data-interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:55.278322Z",
     "start_time": "2018-11-06T21:03:54.059643Z"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "dtrain = xgb.DMatrix(X_train, label=y_train)\n",
    "dvalidation = xgb.DMatrix(X_validation, label=y_validation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"parameters\"></a>\n",
    "## Set Parameters\n",
    "\n",
    "There are a number of parameters that can be set before XGBoost can be run. \n",
    "\n",
    "* General parameters relate to which booster we are using to do boosting, commonly tree or linear model\n",
    "* Booster parameters depend on which booster you have chosen\n",
    "* Learning task parameters decide on the learning scenario. For example, regression tasks may use different parameters with ranking tasks.\n",
    "\n",
    "For more information on the configurable parameters within the XGBoost module, see the documentation here:\n",
    "\n",
    "\n",
    "https://xgboost.readthedocs.io/en/latest/parameter.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:57.443698Z",
     "start_time": "2018-11-06T21:03:57.438288Z"
    }
   },
   "outputs": [],
   "source": [
    "# instantiate params\n",
    "params = {}\n",
    "\n",
    "# general params\n",
    "general_params = {'silent': 1}\n",
    "params.update(general_params)\n",
    "\n",
    "# booster params\n",
    "n_gpus = 1  # change this to -1 to use all GPUs available or 0 to use the CPU\n",
    "booster_params = {}\n",
    "\n",
    "if n_gpus != 0:\n",
    "    booster_params['tree_method'] = 'gpu_hist'\n",
    "    booster_params['n_gpus'] = n_gpus   \n",
    "params.update(booster_params)\n",
    "\n",
    "# learning task params\n",
    "learning_task_params = {}\n",
    "if classification:\n",
    "    learning_task_params['eval_metric'] = 'auc'\n",
    "    learning_task_params['objective'] = 'binary:logistic'\n",
    "else:\n",
    "    learning_task_params['eval_metric'] = 'rmse'\n",
    "    learning_task_params['objective'] = 'reg:squarederror'\n",
    "params.update(learning_task_params)\n",
    "print(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"train\"></a>\n",
    "## Train Model\n",
    "\n",
    "Now it's time to train our model! We can use the `xgb.train` function and pass in the parameters, training dataset, the number of boosting iterations, and the list of items to be evaluated during training. For more information on the parameters that can be passed into `xgb.train`, check out the documentation:\n",
    "\n",
    "\n",
    "https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model training settings\n",
    "evallist = [(dvalidation, 'validation'), (dtrain, 'train')]\n",
    "num_round = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:04:50.201308Z",
     "start_time": "2018-11-06T21:04:00.363740Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "bst = xgb.train(params, dtrain, num_round, evallist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"conclusion\"></a>\n",
    "## Conclusion\n",
    "\n",
    "In this notebook, we showed how to work with GPU accelerated XGBoost in RAPIDS.\n",
    "\n",
    "To learn more about RAPIDS, be sure to check out: \n",
    "\n",
    "* [Open Source Website](http://rapids.ai)\n",
    "* [GitHub](https://github.com/rapidsai/)\n",
    "* [Press Release](https://nvidianews.nvidia.com/news/nvidia-introduces-rapids-open-source-gpu-acceleration-platform-for-large-scale-data-analytics-and-machine-learning)\n",
    "* [NVIDIA Blog](https://blogs.nvidia.com/blog/2018/10/10/rapids-data-science-open-source-community/)\n",
    "* [Developer Blog](https://devblogs.nvidia.com/gpu-accelerated-analytics-rapids/)\n",
    "* [NVIDIA Data Science Webpage](https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}