cuml/umap_supervised_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UMAP Supervised Demo\n",
    "\n",
    "[UMAP](https://umap-learn.readthedocs.io/en/latest/) is a powerful dimensionality reduction tool, which NVIDIA recently ported to GPUs with a Python interface that matches UMAP-learn. In this notebook we will demostrate basic usage, plotting, and timing comparisons between supervised and unsupervised implementations of the CUDA (GPU) version of UMAP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports and Set Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# libraries for scoring/clustering\n",
    "from sklearn.manifold.t_sne import trustworthiness\n",
    "\n",
    "# GPU UMAP\n",
    "import cudf\n",
    "from cuml.manifold.umap import UMAP as cumlUMAP\n",
    "\n",
    "# plotting\n",
    "import seaborn as sns\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "sns.set(style='white', rc={'figure.figsize':(25, 12.5)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hide warnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sanity Checks\n",
    "\n",
    "We are going to work with the [fashion mnist](https://github.com/zalandoresearch/fashion-mnist) data set.  This is a dataset consisting of 70,000 28x28 grayscale images of clothing.  It should already be in the `data/fashion` folder, but let's do a sanity check!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('data/fashion'):\n",
    "    print(\"error, data is missing!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's make sure we have our RAPIDS compliant GPU.  It must be Pascal or higher!  You can also use this to define which GPU RAPIDS should use (advanced feature not covered here)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://github.com/zalandoresearch/fashion-mnist/blob/master/utils/mnist_reader.py\n",
    "def load_mnist(path, kind='train'):\n",
    "    import os\n",
    "    import gzip\n",
    "    import numpy as np\n",
    "\n",
    "    \"\"\"Load MNIST data from `path`\"\"\"\n",
    "    labels_path = os.path.join(path,\n",
    "                               '%s-labels-idx1-ubyte.gz'\n",
    "                               % kind)\n",
    "    images_path = os.path.join(path,\n",
    "                               '%s-images-idx3-ubyte.gz'\n",
    "                               % kind)\n",
    "\n",
    "    with gzip.open(labels_path, 'rb') as lbpath:\n",
    "        labels = np.frombuffer(lbpath.read(), dtype=np.uint8,\n",
    "                               offset=8)\n",
    "\n",
    "    with gzip.open(images_path, 'rb') as imgpath:\n",
    "        images = np.frombuffer(imgpath.read(), dtype=np.uint8,\n",
    "                               offset=16).reshape(len(labels), 784)\n",
    "\n",
    "    return images, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, train_labels = load_mnist('data/fashion', kind='train')\n",
    "test, test_labels = load_mnist('data/fashion', kind='t10k')\n",
    "data = (np.array(np.vstack([train, test]), dtype=np.float64) [:60000]/ 255.0).astype(np.float32)\n",
    "target = np.array(np.hstack([train_labels, test_labels]))[:60000].astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 60000 training images and 10000 test images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f\"Train shape: {train.shape} and Test Shape: {test.shape}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned previously, each row in the train matrix is an image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display a Nike? sneaker\n",
    "pixels = train[0].reshape((28, 28))\n",
    "plt.imshow(pixels, cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is cost with moving data between host memory and device memory (GPU memory) and we will include that core when comparing speeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "record_data = (('fea%d'%i, data[:,i]) for i in range(data.shape[1]))\n",
    "gdf = cudf.DataFrame(record_data)\n",
    "\n",
    "label_data = [('fea%0', target)]\n",
    "\n",
    "target_gdf = cudf.DataFrame(label_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`gdf` is a GPU backed dataframe -- all the data is stored in the device memory of the GPU.  With the data converted, we can apply the `cumlUMAP` the same inputs as we do for the standard UMAP.\n",
    "\n",
    "For datasets that provide a set of labels, we can pass those labels into the `fit()` and `fit_transform()` functions to have UMAP use them for better cluster separation. Supervised training can even be used with an incomplete set of labels by setting the unknown labels to -1\". It is important that the labels array be the same size as the number of samples being used to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "g_embedding_supervised = cumlUMAP(verbose = False, n_neighbors=5, init=\"spectral\", target_metric = \"categorical\").fit_transform(gdf, target_gdf)\n",
    "print(\"Took %f sec.\" % (time.time() - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "g_embedding = cumlUMAP(n_neighbors=5, init=\"spectral\").fit_transform(gdf)\n",
    "print(\"Took %f sec.\" % (time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization\n",
    "\n",
    "OK, now let's plot the output of the embeddings so that we can see the seperation of the neighborhoods.  Let's start by creating the classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = [\n",
    "    'T-shirt/top',\n",
    "    'Trouser',\n",
    "    'Pullover',\n",
    "    'Dress',\n",
    "    'Coat',\n",
    "    'Sandal',\n",
    "    'Shirt',\n",
    "    'Sneaker',\n",
    "    'Bag',\n",
    "    'Ankle boot']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as the original author of UMAP, Leland McInnes, states in the [UMAP docs](https://umap-learn.readthedocs.io/en/latest/supervised.html), we can plot the results and show the separation between the various classes defined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g_embedding_supervised_numpy = g_embedding_supervised.to_pandas().values #it is necessary to convert to numpy array to do the visual mapping\n",
    "\n",
    "fig, ax = plt.subplots(1, figsize=(14, 10))\n",
    "plt.scatter(g_embedding_supervised_numpy[:,1], g_embedding_supervised_numpy[:,0], s=0.3, c=target, cmap='Spectral', alpha=1.0)\n",
    "plt.setp(ax, xticks=[], yticks=[])\n",
    "cbar = plt.colorbar(boundaries=np.arange(11)-0.5)\n",
    "cbar.set_ticks(np.arange(10))\n",
    "cbar.set_ticklabels(classes)\n",
    "plt.title('Supervised Fashion MNIST Embedded via cumlUMAP');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison of Implementations\n",
    "\n",
    "And side-by-side we can see the effects of supervised training.  Notice how providing the labels enables the resulting model to better separation of sneakers, ankle books, and sandals while also providing a much more distinct separation of shirts, t-shirts, pullovers, and coats. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g_embedding_numpy = g_embedding.to_pandas().values #it is necessary to convert to numpy array to do the visual mapping\n",
    "\n",
    "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(20, 10))\n",
    "ax[0].scatter(g_embedding_numpy[:,1], g_embedding_numpy[:,0], s=0.3, c=target, cmap='Spectral', alpha=1.0)\n",
    "im = ax[1].scatter(g_embedding_supervised_numpy[:,1], g_embedding_supervised_numpy[:,0], s=0.3, c=target, cmap='Spectral', alpha=1.0)\n",
    "ax[0].set_title('Unsupervised Fashion MNIST Embedded via cumlUMAP ');\n",
    "ax[1].set_title('Supervised Fashion MNIST Embedded via UMAP');\n",
    "\n",
    "fig.subplots_adjust(right=0.8)\n",
    "cax,kw = mpl.colorbar.make_axes([a for a in ax.flat])\n",
    "cbar = plt.colorbar(im, cax=cax, **kw)\n",
    "cbar.set_ticks(np.arange(10))\n",
    "cbar.set_ticklabels(classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, we can also quanititaviely compare the perfomance of `cumlUMAP` (GPU UMAP) to the reference/original implementation (CPU UMAP) using the [trustworthiness score](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/manifold/t_sne.py#L395).  From the docstring:\n",
    "\n",
    "> Trustworthiness expresses to what extent the local structure is retained.  The trustworthiness is within [0, 1].\n",
    "\n",
    "\n",
    "Like `t-SNE`, UMAP tries to capture both global and local structure and thus, we can apply the `trustworthiness` of the `embedding/g_embedding` data against the original input.  With a higher score we are demonstrating that the algorithm does a better and better job of local structure retention.  As [Corey Nolet](https://github.com/cjnolet) notes:\n",
    "> Algorithms like UMAP aim to preserve local neighborhood structure and so measuring this property (trustworthiness) measures the algorithm's performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scoring ~97% shows the GPU implementation is comparable to the original CPU implementation and the training time was ~9.5X faster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cuml4",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}