cugraph/Pagerank.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PageRank\n",
    "\n",
    "In this notebook, we will use both NetworkX and cuGraph to compute the PageRank of each vertex in our test dataset.  The NetworkX and cuGraph processes will be interleaved so that each step can be compared.\n",
    "\n",
    "Notebook Credits\n",
    "* Original Authors: Bradley Rees and James Wyles\n",
    "* Last Edit: 08/13/2019\n",
    "\n",
    "RAPIDS Versions: 0.9.0    \n",
    "\n",
    "Test Hardware\n",
    "\n",
    "* GV100 32G, CUDA 10.0\n",
    "\n",
    "\n",
    "## Introduction\n",
    "Pagerank is measure of the relative importance, also called centrality, of a vertex based on the relative importance of it's neighbors.  PageRank was developed by Google and is (was) used to rank it's search results. PageRank uses the connectivity information of a graph to rank the importance of each vertex. \n",
    "\n",
    "See [Wikipedia](https://en.wikipedia.org/wiki/PageRank) for more details on the algorithm.\n",
    "\n",
    "To compute the Pagerank scores for a graph in cuGraph we use:<br>\n",
    "\n",
    "**cugraph.pagerank(G,alpha=0.85, max_iter=100, tol=1.0e-5)**\n",
    "* __G__: cugraph.Graph object\n",
    "* __alpha__: float, The damping factor represents the probability to follow an outgoing edge. default is 0.85\n",
    "* __max_iter__: int, The maximum number of iterations before an answer is returned. This can be used to limit the execution time and do an early exit before the solver reaches the convergence tolerance. If this value is lower or equal to 0 cuGraph will use the default value, which is 100\n",
    "* __tol__: float, Set the tolerance the approximation, this parameter should be a small magnitude value. The lower the tolerance the better the approximation. If this value is 0.0f, cuGraph will use the default value which is 0.00001. Setting too small a tolerance can lead to non-convergence due to numerical roundoff. Usually values between 0.01 and 0.00001 are acceptable.\n",
    "\n",
    "Returns:\n",
    "* __df__: a cudf.DataFrame object with two columns:\n",
    "    * df['vertex']: The vertex identifier for the vertex\n",
    "    * df['pagerank']: The pagerank score for the vertex\n",
    "\n",
    "\n",
    "## cuGraph Notice \n",
    "The current version of cuGraph has some limitations:\n",
    "\n",
    "* Vertex IDs need to be 32-bit integers.\n",
    "* Vertex IDs are expected to be contiguous integers starting from 0.\n",
    "\n",
    "cuGraph provides the renumber function to mitigate this problem. Input vertex IDs for the renumber function can be either 32-bit or 64-bit integers, can be non-contiguous, and can start from an arbitrary number. The renumber function maps the provided input vertex IDs to 32-bit contiguous integers starting from 0. cuGraph still requires the renumbered vertex IDs to be representable in 32-bit integers. These limitations are being addressed and will be fixed soon.    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Data\n",
    "We will be using the Zachary Karate club dataset \n",
    "*W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of\n",
    "Anthropological Research 33, 452-473 (1977).*\n",
    "\n",
    "\n",
    "![Karate Club](./img/zachary_black_lines.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: networkx in /home/seunghwak/Program/anaconda3/envs/cuda10/lib/python3.7/site-packages (2.3)\n",
      "Requirement already satisfied: decorator>=4.3.0 in /home/seunghwak/Program/anaconda3/envs/cuda10/lib/python3.7/site-packages (from networkx) (4.4.0)\n",
      "Requirement already satisfied: scipy in /home/seunghwak/Program/anaconda3/envs/cuda10/lib/python3.7/site-packages (1.3.1)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /home/seunghwak/Program/anaconda3/envs/cuda10/lib/python3.7/site-packages (from scipy) (1.17.0)\n"
     ]
    }
   ],
   "source": [
    "# The notebook compares cuGraph to NetworkX,  \n",
    "# therefore there some additional non-RAPIDS python libraries need to be installed. \n",
    "# Please run this cell if you need the additional libraries\n",
    "!pip install networkx\n",
    "!pip install scipy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import needed libraries\n",
    "import cugraph\n",
    "import cudf\n",
    "from collections import OrderedDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NetworkX libraries\n",
    "import networkx as nx\n",
    "from scipy.io import mmread"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some Prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the parameters \n",
    "max_iter = 100  # The maximum number of iterations\n",
    "tol = 0.00001   # tolerance\n",
    "alpha = 0.85    # alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the path to the test data  \n",
    "datafile='./data/karate-data.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# NetworkX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the data, this also created a NetworkX Graph \n",
    "file = open(datafile, 'rb')\n",
    "Gnx = nx.read_edgelist(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "pr_nx = nx.pagerank(Gnx, alpha=alpha, max_iter=max_iter, tol=tol)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'1': 0.09701602879679112,\n",
       " '2': 0.05288352140273713,\n",
       " '3': 0.05707824474239912,\n",
       " '4': 0.03586215186028656,\n",
       " '5': 0.021982948286246907,\n",
       " '6': 0.0291187010552997,\n",
       " '7': 0.0291187010552997,\n",
       " '8': 0.024490182143018978,\n",
       " '9': 0.02976223516027387,\n",
       " '11': 0.021982948286246907,\n",
       " '12': 0.009564948925264983,\n",
       " '13': 0.014645330577424499,\n",
       " '14': 0.02953411440759793,\n",
       " '18': 0.014558421109725606,\n",
       " '20': 0.01960235337430456,\n",
       " '22': 0.014558421109725606,\n",
       " '32': 0.037151296649534996,\n",
       " '31': 0.024585898152929067,\n",
       " '10': 0.014307076351595165,\n",
       " '28': 0.025635977707690095,\n",
       " '29': 0.019571497200047343,\n",
       " '33': 0.07169586521384512,\n",
       " '17': 0.01678918865030142,\n",
       " '34': 0.10092357454636647,\n",
       " '15': 0.01453184130992307,\n",
       " '16': 0.01453184130992307,\n",
       " '19': 0.01453184130992307,\n",
       " '21': 0.01453184130992307,\n",
       " '23': 0.01453184130992307,\n",
       " '24': 0.03151619714292456,\n",
       " '26': 0.021005086910429464,\n",
       " '30': 0.026283644274280444,\n",
       " '25': 0.021074699693936733,\n",
       " '27': 0.015041538663860602}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pr_nx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running NetworkX is that easy.  \n",
    "Let's seet how that compares to cuGraph\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# cuGraph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in the data - GPU\n",
    "cuGraph depends on cuDF for data loading and the initial Dataframe creation\n",
    "\n",
    "The data file contains an edge list, which represents the connection of a vertex to another.  The `source` to `destination` pairs is in what is known as Coordinate Format (COO).  In this test case, the data is just two columns.  However a third, `weight`, column is also possible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test file    \n",
    "datafile='./data/karate-data.csv'\n",
    "\n",
    "# Read the data file\n",
    "cols = [\"src\", \"dst\"]\n",
    "\n",
    "dtypes = OrderedDict([\n",
    "        (\"src\", \"int32\"), \n",
    "        (\"dst\", \"int32\")\n",
    "        ])\n",
    "\n",
    "gdf = cudf.read_csv(datafile, names=cols, delimiter='\\t', dtype=list(dtypes.values()) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Graph "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a Graph using the source (src) and destination (dst) vertex pairs from the Dataframe \n",
    "G = cugraph.Graph()\n",
    "G.add_edge_list(gdf[\"src\"], gdf[\"dst\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Call the PageRank algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call cugraph.pagerank to get the pagerank scores\n",
    "gdf_page = cugraph.pagerank(G)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_It was that easy!_  \n",
    "Compared to NetworkX, the cuGraph data loading might have been more steps, but using cuDF allows for a wider range of data to be loaded. \n",
    "\n",
    "\n",
    "----\n",
    "\n",
    "Let's now look at the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best vertex is 34 with score of 0.10047409\n"
     ]
    }
   ],
   "source": [
    "# Find the most important vertex using the scores\n",
    "# This methods should only be used for small graph\n",
    "bestScore = gdf_page['pagerank'][0]\n",
    "bestVert = gdf_page['vertex'][0]\n",
    "\n",
    "for i in range(len(gdf_page)):\n",
    "    if gdf_page['pagerank'][i] > bestScore:\n",
    "        bestScore = gdf_page['pagerank'][i]\n",
    "        bestVert = gdf_page['vertex'][i]\n",
    "        \n",
    "print(\"Best vertex is \" + str(bestVert) + \" with score of \" + str(bestScore))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top PageRank vertex and socre match what was found by NetworkX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A better way to do that would be to find the max and then use that values in a query\n",
    "pr_max = gdf_page['pagerank'].max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_pagerank_threshold(_df, t=0) :\n",
    "    filtered = _df.query('pagerank >= @t')\n",
    "    \n",
    "    for i in range(len(filtered)):\n",
    "        print(\"Best vertex is \" + str(filtered['vertex'][i]) + \n",
    "            \" with score of \" + str(filtered['pagerank'][i]))              "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best vertex is 34 with score of 0.10047409\n"
     ]
    }
   ],
   "source": [
    "print_pagerank_threshold(gdf_page, pr_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "a PageRank score of _0.10047_ is very low, which can be an indication that there is no more central vertex than any other.  Rather than just looking at the top score, let's look at the top three vertices and see if there are any insights that can be inferred.  \n",
    "\n",
    "Since this is a very small graph, let's just sort and get the first three records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "sort_pr = gdf_page.sort_values('pagerank', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>vertex</th>\n",
       "      <th>pagerank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>34</td>\n",
       "      <td>0.100474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0.096573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>33</td>\n",
       "      <td>0.071377</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    vertex  pagerank\n",
       "34      34  0.100474\n",
       "1        1  0.096573\n",
       "33      33  0.071377"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sort_pr.head(3).to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going back and looking at the graph with the top three vertices highlighted (illustration below) it is easy to see that the top scoring vertices also appear to be the vertices with the most connections.   \n",
    "Let's look at sorted list of degrees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = G.degree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# divide the degree by two since this is an undirected graph\n",
    "d['degree'] = d['degree'] / 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>vertex</th>\n",
       "      <th>degree</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>34</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>33</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    vertex  degree\n",
       "34      34    17.0\n",
       "1        1    16.0\n",
       "33      33    12.0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d.sort_values('degree', ascending=False).head(3).to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./img/zachary_graph_pagerank.png\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "Copyright (c) 2019, NVIDIA CORPORATION.\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");  you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n",
    "___"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}