example/comparison_model_quality.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing Isolation Forest implementations\n",
    "\n",
    "This notebook performs a short comparison of the Isolation Forest implementations in different libraries ([IsoTree](https://github.com/david-cortes/isotree), [EIF](https://github.com/sahandha/eif) and [SciKit-Learn](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html)) by first fitting models to different datasets with the default hyperparameters and then performing a short tuning over a small grid of hyperparameters, depending on which ones can be varied in each library.\n",
    "\n",
    "The datasets are all taken from the [Outlier Detection DataSets (ODDS)](http://odds.cs.stonybrook.edu) webpage.\n",
    "\n",
    "The only evaluation metric used here is AUC, with outliers being the positive class and being the minority in each dataset. The outliers come already labelled and some of them are artificially produced - see the link above for details.\n",
    "\n",
    "*(For a speed comparison with larger datasets see [this other notebook](https://github.com/david-cortes/isotree/blob/master/timings/timings_python.ipynb))*\n",
    "\n",
    "**A note about results:** this notebook tries to evaluate outlier detection by producing outlier scores within the same data to which the model was fit, rather than fitting it to some data and finding outliers in new data (\"novelty detection\"). The results are nevertheless similar when evaluated as \"novelty detection\" under the 3 libraries tested here. The parameter tuning is however performed by evaluating outliers in hold-out data, and then refitting the model to the whole data with the parameters that gave the best results in the hold-out data.\n",
    "\n",
    "** *\n",
    "Datasets compared:\n",
    "* [Satellite (6435 rows, 36 columns)](#p1)\n",
    "* [Antthyroid (7200 rows, 6 columns)](#p2)\n",
    "* [Pendigits (6870 rows, 16 columns)](#p3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from isotree import IsolationForest as IsolationForestIsoTree\n",
    "from sklearn.ensemble import IsolationForest as IsolationForestSKL\n",
    "from eif import iForest\n",
    "from scipy.io import loadmat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"p1\"></a>\n",
    "## Satellite (6435 rows, 36 columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6435, 36)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "satellite = loadmat(\"satellite.mat\")\n",
    "X = np.asfortranarray(satellite[\"X\"]).astype(np.float64)\n",
    "y = satellite[\"y\"].astype(np.float64).reshape(-1)\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking isotree library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7002590560187147"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_default = IsolationForestIsoTree().fit_predict(X)\n",
    "roc_auc_score(y, pred_default)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluated as \"novelty detection\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7070890762493185"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "cv_res = cross_validate(IsolationForestIsoTree(), X, y, scoring=\"roc_auc\",\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_res[\"test_score\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'prob_pick_pooled_gain': 1, 'sample_size': 256, 'weigh_by_kurtosis': True}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_try = {\n",
    "    \"sample_size\" : [256, 1024, 5000],\n",
    "    \"prob_pick_pooled_gain\" : [0, 1],\n",
    "    \"weigh_by_kurtosis\" : [True, False],\n",
    "}\n",
    "cv_model = GridSearchCV(estimator=IsolationForestIsoTree(ntrees=100, ndim=2,\n",
    "                                                         missing_action=\"fail\",\n",
    "                                                         random_seed=1),\n",
    "                        param_grid=params_try,\n",
    "                        scoring=\"roc_auc\", refit=True,\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_model.fit(X,y)\n",
    "cv_model.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8399789803094202"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tuned = cv_model.decision_function(X)\n",
    "roc_auc_score(y, pred_tuned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking scikit-learn library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6872106805842193"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_default = IsolationForestSKL(n_jobs=-1, random_state=1).fit(X).decision_function(X)\n",
    "roc_auc_score(y, -pred_default)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As \"novelty detection\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7130569172599454"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_res = cross_validate(IsolationForestSKL(n_jobs=-1, random_state=1),\n",
    "                        X, 1-y, scoring=\"roc_auc\",\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_res[\"test_score\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_samples': 5000}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_try = {\n",
    "    \"max_samples\" : [256, 1024, 5000],\n",
    "}\n",
    "cv_model = GridSearchCV(estimator=IsolationForestSKL(n_estimators=100, random_state=1, n_jobs=-1),\n",
    "                        param_grid=params_try,\n",
    "                        scoring=\"roc_auc\", refit=True,\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_model.fit(X, 1-y)\n",
    "cv_model.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7392743305207337"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tuned = cv_model.decision_function(X)\n",
    "roc_auc_score(y, -pred_tuned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking EIF library - note that it doesn't have default arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ExtensionLevel': 1, 'sample_size': 5000}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### EIF library is not scikit-learn-compatible, needs a wrapper class\n",
    "from sklearn.base import BaseEstimator\n",
    "class EIF_sk_compat(BaseEstimator):\n",
    "    def __init__(self, sample_size=256, ntrees=100, seed=1, ExtensionLevel=0):\n",
    "        self.sample_size = sample_size\n",
    "        self.ntrees = ntrees\n",
    "        self.seed = seed\n",
    "        self.ExtensionLevel = ExtensionLevel\n",
    "    def fit(self, X, y=None):\n",
    "        self._model = iForest(X, ntrees=self.ntrees, sample_size=self.sample_size,\n",
    "                              ExtensionLevel=self.ExtensionLevel, seed=self.seed)\n",
    "        return self\n",
    "    def decision_function(self, X):\n",
    "        return self._model.compute_paths(X_in=X)\n",
    "    \n",
    "params_try = {\n",
    "    \"sample_size\" : [256, 1024, 5000],\n",
    "    \"ExtensionLevel\" : [0, 1]\n",
    "}\n",
    "cv_model = GridSearchCV(estimator=EIF_sk_compat(),\n",
    "                        param_grid=params_try,\n",
    "                        scoring=\"roc_auc\", refit=True,\n",
    "                        n_jobs=-1,\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_model.fit(X,y)\n",
    "cv_model.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7142364915048116"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tuned = cv_model.decision_function(X)\n",
    "roc_auc_score(y, pred_tuned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"p2\"></a>\n",
    "## Antthyroid (7200 rows, 6 columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7200, 6)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "annthyroid = loadmat(\"annthyroid.mat\")\n",
    "X = np.asfortranarray(annthyroid[\"X\"]).astype(np.float64)\n",
    "y = annthyroid[\"y\"].astype(np.float64).reshape(-1)\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking isotree library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8001538917936738"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_default = IsolationForestIsoTree().fit_predict(X)\n",
    "roc_auc_score(y, pred_default)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As \"novelty detection\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8043268497077755"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_res = cross_validate(IsolationForestIsoTree(), X, y, scoring=\"roc_auc\",\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_res[\"test_score\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'prob_pick_pooled_gain': 1, 'sample_size': 256, 'weigh_by_kurtosis': True}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_try = {\n",
    "    \"sample_size\" : [256, 1024, 5000],\n",
    "    \"prob_pick_pooled_gain\" : [0, 1],\n",
    "    \"weigh_by_kurtosis\" : [True, False],\n",
    "}\n",
    "cv_model = GridSearchCV(estimator=IsolationForestIsoTree(ntrees=100, ndim=2,\n",
    "                                                         missing_action=\"fail\",\n",
    "                                                         random_seed=1),\n",
    "                        param_grid=params_try,\n",
    "                        scoring=\"roc_auc\", refit=True,\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_model.fit(X,y)\n",
    "cv_model.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9817450846208218"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tuned = cv_model.decision_function(X)\n",
    "roc_auc_score(y, pred_tuned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking scikit-learn library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.836075461478732"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_default = IsolationForestSKL(n_jobs=-1, random_state=1).fit(X).decision_function(X)\n",
    "roc_auc_score(y, -pred_default)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As \"novelty detection\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8308306395401213"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_res = cross_validate(IsolationForestSKL(n_jobs=-1, random_state=1),\n",
    "                        X, 1-y, scoring=\"roc_auc\",\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_res[\"test_score\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_samples': 256}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_try = {\n",
    "    \"max_samples\" : [256, 1024, 5000],\n",
    "}\n",
    "cv_model = GridSearchCV(estimator=IsolationForestSKL(n_estimators=100, random_state=1, n_jobs=-1),\n",
    "                        param_grid=params_try,\n",
    "                        scoring=\"roc_auc\", refit=True,\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_model.fit(X, 1-y)\n",
    "cv_model.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.836075461478732"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tuned = cv_model.decision_function(X)\n",
    "roc_auc_score(y, -pred_tuned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking EIF library - note that it doesn't have default arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ExtensionLevel': 0, 'sample_size': 256}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_try = {\n",
    "    \"sample_size\" : [256, 1024, 5000],\n",
    "    \"ExtensionLevel\" : [0, 1]\n",
    "}\n",
    "cv_model = GridSearchCV(estimator=EIF_sk_compat(),\n",
    "                        param_grid=params_try,\n",
    "                        scoring=\"roc_auc\", refit=True,\n",
    "                        n_jobs=-1,\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_model.fit(X,y)\n",
    "cv_model.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.807920117854482"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tuned = cv_model.decision_function(X)\n",
    "roc_auc_score(y, pred_tuned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"p3\"></a>\n",
    "## Pendigits (6870 rows, 16 columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6870, 16)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pendigits = loadmat(\"pendigits.mat\")\n",
    "X = np.asfortranarray(pendigits[\"X\"]).astype(np.float64)\n",
    "y = pendigits[\"y\"].astype(np.float64).reshape(-1)\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking isotree library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9393651230112356"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_default = IsolationForestIsoTree().fit_predict(X)\n",
    "roc_auc_score(y, pred_default)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As \"novelty detection\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9376103836964493"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_res = cross_validate(IsolationForestIsoTree(), X, y, scoring=\"roc_auc\",\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_res[\"test_score\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'prob_pick_pooled_gain': 1, 'sample_size': 256, 'weigh_by_kurtosis': False}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_try = {\n",
    "    \"sample_size\" : [256, 1024, 5000],\n",
    "    \"prob_pick_pooled_gain\" : [0, 1],\n",
    "    \"weigh_by_kurtosis\" : [True, False],\n",
    "}\n",
    "cv_model = GridSearchCV(estimator=IsolationForestIsoTree(ntrees=100, ndim=2,\n",
    "                                                         missing_action=\"fail\",\n",
    "                                                         random_seed=1),\n",
    "                        param_grid=params_try,\n",
    "                        scoring=\"roc_auc\", refit=True,\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_model.fit(X,y)\n",
    "cv_model.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9634517999129261"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tuned = cv_model.decision_function(X)\n",
    "roc_auc_score(y, pred_tuned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking scikit-learn library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9562758262490165"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_default = IsolationForestSKL(n_jobs=-1, random_state=1).fit(X).decision_function(X)\n",
    "roc_auc_score(y, -pred_default)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As \"novelty detection\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.947638267925153"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_res = cross_validate(IsolationForestSKL(n_jobs=-1, random_state=1),\n",
    "                        X, 1-y, scoring=\"roc_auc\",\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_res[\"test_score\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_samples': 1024}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_try = {\n",
    "    \"max_samples\" : [256, 1024, 5000],\n",
    "}\n",
    "cv_model = GridSearchCV(estimator=IsolationForestSKL(n_estimators=100, random_state=1, n_jobs=-1),\n",
    "                        param_grid=params_try,\n",
    "                        scoring=\"roc_auc\", refit=True,\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_model.fit(X, 1-y)\n",
    "cv_model.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9485098111103473"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tuned = cv_model.decision_function(X)\n",
    "roc_auc_score(y, -pred_tuned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking EIF library - note that it doesn't have default arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ExtensionLevel': 0, 'sample_size': 1024}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_try = {\n",
    "    \"sample_size\" : [256, 1024, 5000],\n",
    "    \"ExtensionLevel\" : [0, 1]\n",
    "}\n",
    "cv_model = GridSearchCV(estimator=EIF_sk_compat(),\n",
    "                        param_grid=params_try,\n",
    "                        scoring=\"roc_auc\", refit=True,\n",
    "                        n_jobs=-1,\n",
    "                        cv=StratifiedKFold(n_splits=10, shuffle=True, random_state=1))\n",
    "cv_model.fit(X,y)\n",
    "cv_model.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9579447461484994"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_tuned = cv_model.decision_function(X)\n",
    "roc_auc_score(y, pred_tuned)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}