Outlier Detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outlier Detection\n",
    "=====\n",
    "\n",
    "Outlier detection with the LOF (Local Outlier Factor) algorithm.\n",
    "\n",
    "The LOF algorithm\n",
    "-----\n",
    "`LOF` (Local Outlier Factor) is an algorithm for identifying density-based local outliers [Breunig et al., 2000]. With LOF, the local density of a point is compared with that of its neighbors. If the former is significantly lower than the latter (with an LOF value greater than one), the point is in a sparser region than its neighbors, which suggests it be an outlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "summary(cars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function `lofactor(data, k)` in packages `DMwR` and `dprep` calculates local outlier factors using the `LOF algorithm`, where `k` is the `number of neighbors` used in the calculation of the local outlier factors.\n",
    "\n",
    "Calculate Outlier scores.\n",
    "----\n",
    "\n",
    "DMwR: Functions and data for the book \"Data Mining with R\"\n",
    "\n",
    "This package includes functions and data accompanying the book \"Data Mining with R, learning with case studies\" by Luis Torgo, published by CRC Press (ISBN: 9781439810187)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "library(DMwR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "remove \"Species\", which is a categorical variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "iris2  <- iris[,1:4]\n",
    "outlier.scores  <- lofactor(iris2, k = 5)\n",
    "plot(density(outlier.scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "?lofactor: An implementation of the LOF algorithm\n",
    "\n",
    "pick top 5 as outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "outliers  <- order(outlier.scores, decreasing=T)[1:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "who are the outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print(outliers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize outliers with plots\n",
    "---------\n",
    "\n",
    "show outliers with a biplot of the first two principal components\n",
    "TODO: what is biplot and how to interpret this graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "n  <- nrow(iris2)\n",
    "labels  <- 1:n\n",
    "labels[-outliers]  <- \".\"\n",
    "biplot(prcomp(iris2), cex=0.8, xlabs=labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "show outliers with a pairs plot. The outliers are labeled with + in red."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "pch <- rep(\".\",n)\n",
    "pch[outliers]  <- \"+\"\n",
    "col  <- rep(\"black\", n)\n",
    "col[outliers]  <- \"red\"\n",
    "pairs(iris2, pch = pch, col=col)"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}