k-fold_processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0986987c-cdc2-449b-9a72-1b900edd93e8",
   "metadata": {},
   "source": [
    "# **Pre-processing data : Kfold**\n",
    "The aim of this jupyter notebook was creating a csv file that contains 10 different spilts of samples. Each of them must be at least one time a validation sample and training sample. The notebook has been created in the framework of an intership.\\\n",
    "This notebook has been inspired from \n",
    "Author : Julien Govoorts\\\n",
    "Organisation : ANAGEO ULB (Université libre de Bruxelles)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ca47dd0-b498-4614-b1d9-917478aaf8e8",
   "metadata": {},
   "source": [
    "## Importing libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b250867d-acb4-4f09-980c-5cc1325d70da",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os, sys\n",
    "import pandas as pd\n",
    "import glob\n",
    "import re\n",
    "import natsort\n",
    "import sklearn\n",
    "import datetime\n",
    "import cv2\n",
    "import shutil\n",
    "import time\n",
    "import h5py\n",
    "\n",
    "from osgeo import gdal\n",
    "from osgeo import gdal_array\n",
    "from natsort import natsorted\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import KFold, StratifiedKFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d8fb140d-3f87-471c-9593-3cb6a96a2573",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b07ab9a9-2d2c-4c74-a02d-aa967c1ff591",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function allows the jupyter notebook to find python modules in the SRC folder\n",
    "import sys\n",
    "sys.path.insert(0, \"/home/jgovoort/SRC\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "42a4ae20-8d8c-45a5-902d-49d925f7d7b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import functions for processing time information\n",
    "from processing_time import start_processing, print_processing_time\n",
    "# Import function that checks and creates folder\n",
    "from mkdir import check_create_dir\n",
    "# Import functions for plots\n",
    "from plots import plot_loss, plot_pred_test, plot_pred_train, plot_loss_multirun\n",
    "# Import functions for metrics to monitoring accuracy\n",
    "from metrics import coeff_determination\n",
    "# Import functions for display\n",
    "from display import Norma_Xpercentile\n",
    "# Import functions for exportation of results\n",
    "from export_results import save_predictions, write_run_metrics_file\n",
    "# Import LRFinder function\n",
    "from LRFinder import LRFinder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02841cac-7ba3-4939-a7e4-d4e282df720b",
   "metadata": {},
   "source": [
    "## Importing data\n",
    "This step import name of the different samples and its label (lulc). The data are store in a hdf5 file create in the YModel notebook from the chunckage module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "afb5ca61-1256-4e2f-b1f4-639fc912f8a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = h5py.File('/home/jgovoort/Images/big_raster/tr_BNIR_patch/tr_Ymodel_S1BNIR.hdf5', 'r')\n",
    "S1_patch_id = np.array(f[\"id_sar\"])\n",
    "S2_patch_id = np.array(f[\"id_rgb\"])\n",
    "S1_patch_lulc = np.array(f[\"lulc_sar\"])\n",
    "S2_patch_lulc = np.array(f[\"lulc_rgb\"])\n",
    "S1_patch_path = f[\"patch_sar_path\"]\n",
    "S2_patch_path = f[\"patch_rgb_path\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0d6c297d-88c7-41fe-b9ad-fc522a1d0235",
   "metadata": {},
   "outputs": [],
   "source": [
    "lulc = S1_patch_lulc.reshape(S1_patch_lulc.size,1)\n",
    "patch_id = S1_patch_id.reshape(S1_patch_lulc.size,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b9e6bee-803e-4541-8cb1-c80afc1a428f",
   "metadata": {},
   "source": [
    "## Kfold : Dividing all the samples in 10 splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0579050a-90a6-47d9-9f16-4b004305af51",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "[5537, 5537, 5538, 5538, 5538]\n",
      "5\n",
      "[1385, 1385, 1384, 1384, 1384]\n"
     ]
    }
   ],
   "source": [
    "kf_SEED = 101\n",
    "kf = KFold(n_splits=5, random_state=kf_SEED, shuffle=True)\n",
    "idx_train = []\n",
    "idx_val = []\n",
    "for train_index, val_index in kf.split(np.zeros(len(lulc)),lulc):\n",
    "    idx_train.append(train_index)\n",
    "    idx_val.append(val_index)\n",
    "print(len(idx_train))\n",
    "print([len(x) for x in idx_train])\n",
    "print(len(idx_val))\n",
    "print([len(x) for x in idx_val])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "17b5ebf1-7651-4014-84bd-e9f4201c40ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "kfold_nairobi_id = []\n",
    "for i in range(len(idx_train)):\n",
    "    kfold_nairobi_id.append((i+1,[x[0] for x in patch_id[idx_train[i]].tolist()],[x[0] for x in patch_id[idx_val[i]].tolist()]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cf39799-c6c6-4ebf-8675-0f514823f78f",
   "metadata": {},
   "source": [
    "### Store the Kfold in a csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a7df3623-f2f1-4577-bf7e-ac36573c533c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>kfold</th>\n",
       "      <th>ID_train</th>\n",
       "      <th>ID_val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>[10508, 10509, 10763, 10767, 11016, 11017, 112...</td>\n",
       "      <td>[10255, 16578, 17321, 17338, 17596, 17850, 181...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>[10255, 10508, 10509, 10763, 10767, 11016, 112...</td>\n",
       "      <td>[11017, 14525, 15571, 16579, 16834, 17088, 175...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>[10255, 10508, 10509, 10763, 11016, 11017, 112...</td>\n",
       "      <td>[10767, 11270, 11523, 11776, 16305, 16580, 168...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>[10255, 10508, 10767, 11017, 11270, 11275, 115...</td>\n",
       "      <td>[10509, 10763, 11016, 12533, 14023, 15572, 168...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>[10255, 10509, 10763, 10767, 11016, 11017, 112...</td>\n",
       "      <td>[10508, 11275, 11781, 12795, 14021, 14022, 142...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   kfold                                           ID_train  \\\n",
       "0      1  [10508, 10509, 10763, 10767, 11016, 11017, 112...   \n",
       "1      2  [10255, 10508, 10509, 10763, 10767, 11016, 112...   \n",
       "2      3  [10255, 10508, 10509, 10763, 11016, 11017, 112...   \n",
       "3      4  [10255, 10508, 10767, 11017, 11270, 11275, 115...   \n",
       "4      5  [10255, 10509, 10763, 10767, 11016, 11017, 112...   \n",
       "\n",
       "                                              ID_val  \n",
       "0  [10255, 16578, 17321, 17338, 17596, 17850, 181...  \n",
       "1  [11017, 14525, 15571, 16579, 16834, 17088, 175...  \n",
       "2  [10767, 11270, 11523, 11776, 16305, 16580, 168...  \n",
       "3  [10509, 10763, 11016, 12533, 14023, 15572, 168...  \n",
       "4  [10508, 11275, 11781, 12795, 14021, 14022, 142...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_csv_kfold_split = '/home/jgovoort/Nairobi_10Fold_Split.csv'\n",
    "a = pd.DataFrame(kfold_nairobi_id, columns=['kfold','ID_train','ID_val'])\n",
    "a.to_csv(path_csv_kfold_split, index=False)\n",
    "a"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}