zizibee.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# From Jupyter to CCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains an example of how a scheduled job may be launched from a local Jupyter notebook to the CCV cluster.\n",
    "\n",
    "It uses `zizibee` to determine which functions have been defined in the running notebook, and puts them together with other requirements in a single .py file that is then uploaded to CCV using sftp.\n",
    "\n",
    "For this to work one needs to be either on campus, or connected through the VPN.\n",
    "\n",
    "It assumes that there's a function that needs to be executed in an enumerated set of parameters. To provide these input values to the function of interest a necessary helper function needs to be defined. This function takes an integer and returns a tuple with the corresponding parameters.\n",
    "\n",
    "The output of the function can be a numpy array of arbitrary shape.\n",
    "\n",
    "For giving the function of interest a command-line interface, which is useful when setting up the sbatch shell script, `zizibee` uses [fire](https://github.com/google/python-fire).\n",
    "\n",
    "To execute some of the remote commands necessary to set this up, `zizibee` established an SSH connection to CCV using [paramiko](https://www.paramiko.org).\n",
    "\n",
    "Over at CCV the output of the function saves its result in .h5 format.\n",
    "\n",
    "As an additional convenience zizibee also provides a convenience function that calls rsync to pull all the files from a remote directory at CCV.\n",
    "\n",
    "Most of this assumes that passwordless SSH login to CCV has been already configured."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## run_at_ccv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "# These are the imports required for running the function at CCV.\n",
    "# one needs to make sure that these dependencies will be met in the remote environment.\n",
    "# If there's any custom import that needs to be met, this can be \n",
    "# satisfied by providing the adequate path in the extra_py argument\n",
    "# to send_to_ccv.\n",
    "imports = '''\n",
    "from itertools import product\n",
    "import numpy as np\n",
    "import h5py\n",
    "import fire\n",
    "import os\n",
    "'''\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import zizibee as zzb\n",
    "from time import sleep\n",
    "import ast\n",
    "exec(imports)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_function_data(node):\n",
    "    \"\"\"Extract the function name, parameters and docstring.\"\"\"\n",
    "    name = node.name\n",
    "    params = [arg.arg for arg in node.args.args]\n",
    "    docstring = ast.get_docstring(node)\n",
    "    \n",
    "    return name, params, docstring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fun1(i,j,k,l):\n",
    "    '''\n",
    "    This one and the next are just to emphasize the fact that \n",
    "    that all the defined functions are packaged and uploaded.\n",
    "    So that any functional dependenices of the target function\n",
    "    are met.\n",
    "    '''\n",
    "    return i - j + k - l\n",
    "\n",
    "def fun2(i,j,k,l):\n",
    "    return i + j + k + l\n",
    "\n",
    "def input_params(idx):\n",
    "    '''\n",
    "    This is a function that maps the job index to the input parameters\n",
    "    of the function to be run.\n",
    "    '''\n",
    "    ranger = range(1,5)\n",
    "    inputs = list(product(ranger,ranger,ranger,ranger))\n",
    "    return inputs[idx]\n",
    "\n",
    "def complex_fun(job_index):\n",
    "    '''\n",
    "    A silly function to debug things.\n",
    "    The important thing is that the function is made so that the \n",
    "    function value is saved to scratch_dir in h5 format.\n",
    "    This function assumes that scratch_dir will be an available\n",
    "    symbol when the function is run at CCV. This part is taken care of by run_at_ccv.\n",
    "    '''\n",
    "    ins = input_params(job_index)\n",
    "    (i, j, k, l) = ins\n",
    "    zrange = fun1(i,j,k,l) + fun2(i,j,k,l)\n",
    "    out = np.linspace(0, abs(zrange), 100)\n",
    "    fname = '%d.h5' % job_index\n",
    "    fname = os.path.join(scratch_dir, fname)\n",
    "    with h5py.File(fname, 'w') as f:\n",
    "        f.create_dataset('out', data=out, compression='gzip')\n",
    "        f.create_dataset('in', data=ins, compression='gzip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'function' object has no attribute 'name'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[160], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m extract_function_data(complex_fun)\n",
      "Cell \u001b[0;32mIn[159], line 3\u001b[0m, in \u001b[0;36mextract_function_data\u001b[0;34m(node)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mextract_function_data\u001b[39m(node):\n\u001b[1;32m      2\u001b[0m \u001b[39m    \u001b[39m\u001b[39m\"\"\"Extract the function name, parameters and docstring.\"\"\"\u001b[39;00m\n\u001b[0;32m----> 3\u001b[0m     name \u001b[39m=\u001b[39m node\u001b[39m.\u001b[39;49mname\n\u001b[1;32m      4\u001b[0m     params \u001b[39m=\u001b[39m [arg\u001b[39m.\u001b[39marg \u001b[39mfor\u001b[39;00m arg \u001b[39min\u001b[39;00m node\u001b[39m.\u001b[39margs\u001b[39m.\u001b[39margs]\n\u001b[1;32m      5\u001b[0m     docstring \u001b[39m=\u001b[39m ast\u001b[39m.\u001b[39mget_docstring(node)\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'function' object has no attribute 'name'"
     ]
    }
   ],
   "source": [
    "extract_function_data(complex_fun)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Establishing an SSH connection to CCV and launching a shell ...\n",
      "Saving Python script to file ...\n",
      "Uploading script to CCV ...\n",
      "Composing the sbatch script ...\n",
      "Writing sbatch script ...\n",
      "Sending sbatch script ...\n",
      "Mac folder does not exist, creating ...\n",
      "Progress: |██████████████████████████████| 100.0% Complete Elapsed Time: 63.15s Remaining Time: -0.25s\n"
     ]
    }
   ],
   "source": [
    "job_config = {'username':'jlizaraz',\n",
    "              'numCores':1, \n",
    "              'numJobs': 4**4, \n",
    "              'memInGB': 2,\n",
    "              'import_block': imports,\n",
    "              'extra_py': [],\n",
    "              'theglobals': globals(),\n",
    "              'fun_name': 'complex_fun',\n",
    "              'job_name': 'reboot'}\n",
    "job_config = zzb.run_at_ccv(job_config, verbose=False)\n",
    "# at this point the\n",
    "mac_folder, ccv_folder = job_config['scratch_dir_at_mac'], job_config['scratch_dir_at_CCV']\n",
    "numJobs = job_config['numJobs']\n",
    "ccv_fruit = zzb.get_ccv_values(mac_folder, ccv_folder, numJobs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.        , 0.08080808, 0.16161616, 0.24242424, 0.32323232,\n",
       "       0.4040404 , 0.48484848, 0.56565657, 0.64646465, 0.72727273,\n",
       "       0.80808081, 0.88888889, 0.96969697, 1.05050505, 1.13131313,\n",
       "       1.21212121, 1.29292929, 1.37373737, 1.45454545, 1.53535354,\n",
       "       1.61616162, 1.6969697 , 1.77777778, 1.85858586, 1.93939394,\n",
       "       2.02020202, 2.1010101 , 2.18181818, 2.26262626, 2.34343434,\n",
       "       2.42424242, 2.50505051, 2.58585859, 2.66666667, 2.74747475,\n",
       "       2.82828283, 2.90909091, 2.98989899, 3.07070707, 3.15151515,\n",
       "       3.23232323, 3.31313131, 3.39393939, 3.47474747, 3.55555556,\n",
       "       3.63636364, 3.71717172, 3.7979798 , 3.87878788, 3.95959596,\n",
       "       4.04040404, 4.12121212, 4.2020202 , 4.28282828, 4.36363636,\n",
       "       4.44444444, 4.52525253, 4.60606061, 4.68686869, 4.76767677,\n",
       "       4.84848485, 4.92929293, 5.01010101, 5.09090909, 5.17171717,\n",
       "       5.25252525, 5.33333333, 5.41414141, 5.49494949, 5.57575758,\n",
       "       5.65656566, 5.73737374, 5.81818182, 5.8989899 , 5.97979798,\n",
       "       6.06060606, 6.14141414, 6.22222222, 6.3030303 , 6.38383838,\n",
       "       6.46464646, 6.54545455, 6.62626263, 6.70707071, 6.78787879,\n",
       "       6.86868687, 6.94949495, 7.03030303, 7.11111111, 7.19191919,\n",
       "       7.27272727, 7.35353535, 7.43434343, 7.51515152, 7.5959596 ,\n",
       "       7.67676768, 7.75757576, 7.83838384, 7.91919192, 8.        ])"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ccv_fruit(3, 1, 1, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Uploading files to CCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mSignature:\u001b[0m\n",
      "\u001b[0mzzb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupload_to_ccv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mfolder\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0musername\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'jlizaraz'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mtransferhost\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'sshcampus.ccv.brown.edu'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mDocstring:\u001b[0m\n",
      "This  function can be used to upload local files to a folder\n",
      "at CCV.\n",
      "\n",
      "Parameters\n",
      "----------\n",
      "filename  (str):  path  of the file to be uploaded, can be a\n",
      "full or relative path.\n",
      "folder (str): file will be uploaded to this folder\n",
      "username (str): username at CCV\n",
      "transferhost(str): the hostname of the transfer node at CCV\n",
      "verbose (bool): whether to print out the command executed to\n",
      "upload the file\n",
      "\n",
      "Returns\n",
      "-------\n",
      "None\n",
      "\u001b[0;31mFile:\u001b[0m      ~/ZiaLab/Codebase/zizibee/zizibee.py\n",
      "\u001b[0;31mType:\u001b[0m      function"
     ]
    }
   ],
   "source": [
    "?zzb.upload_to_ccv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Progress: |████████████████████| 100.0% Complete\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "def progress_bar(iteration, total, prefix = '', suffix = '', decimals = 1, length = 50, fill = '█'):\n",
    "    \"\"\"\n",
    "    Call in a loop to create terminal progress bar\n",
    "    @params:\n",
    "        iteration   - Required : current iteration (Int)\n",
    "        total       - Required : total iterations (Int)\n",
    "        prefix      - Optional : prefix string (Str)\n",
    "        suffix      - Optional : suffix string (Str)\n",
    "        decimals    - Optional : positive number of decimals in percent complete (Int)\n",
    "        length      - Optional : character length of bar (Int)\n",
    "        fill        - Optional : bar fill character (Str)\n",
    "    \"\"\"\n",
    "    percent = (\"{0:.\" + str(decimals) + \"f}\").format(100 * (iteration / float(total)))\n",
    "    filled_length = int(length * iteration // total)\n",
    "    bar = fill * filled_length + '-' * (length - filled_length)\n",
    "    print(f'\\r{prefix} |{bar}| {percent}% {suffix}', end = '\\r')\n",
    "    # Print New Line on Complete\n",
    "    if iteration == total: \n",
    "        print()\n",
    "\n",
    "# Test\n",
    "for i in range(101):\n",
    "    time.sleep(0.1)\n",
    "    progress_bar(i, 100, prefix = 'Progress:', suffix = 'Complete', length = 20)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Progress: |██████████████████████████████████████████████████| 100.0% Complete Elapsed Time: 10.33s Remaining Time: -0.10s\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "def progress_bar(iteration, total, prefix = '', suffix = '', decimals = 1, length = 50, fill = '█'):\n",
    "    \"\"\"\n",
    "    Call in a loop to create terminal progress bar\n",
    "    @params:\n",
    "        iteration   - Required : current iteration (Int)\n",
    "        total       - Required : total iterations (Int)\n",
    "        prefix      - Optional : prefix string (Str)\n",
    "        suffix      - Optional : suffix string (Str)\n",
    "        decimals    - Optional : positive number of decimals in percent complete (Int)\n",
    "        length      - Optional : character length of bar (Int)\n",
    "        fill        - Optional : bar fill character (Str)\n",
    "    \"\"\"\n",
    "    # Calculate elapsed time and estimated remaining time\n",
    "    global start_time\n",
    "    if iteration == 0:\n",
    "        start_time = time.time()\n",
    "    elapsed_time = time.time() - start_time\n",
    "    remaining_time = (elapsed_time * (total / (iteration + 1))) - elapsed_time if iteration > 0 else 0\n",
    "\n",
    "    percent = (\"{0:.\" + str(decimals) + \"f}\").format(100 * (iteration / float(total)))\n",
    "    filled_length = int(length * iteration // total)\n",
    "    bar = fill * filled_length + '-' * (length - filled_length)\n",
    "    print(f'\\r{prefix} |{bar}| {percent}% {suffix} Elapsed Time: {elapsed_time:.2f}s Remaining Time: {remaining_time:.2f}s', end = '\\r')\n",
    "    # Print New Line on Complete\n",
    "    if iteration == total: \n",
    "        print()\n",
    "\n",
    "# Test\n",
    "for i in range(101):\n",
    "    time.sleep(0.1)\n",
    "    progress_bar(i, 100, prefix = 'Progress:', suffix = 'Complete', length = 50)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "foundation",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}