versions/2022/tools/python/wandb_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weights and Biases (`wandb`) Demo\n",
    "\n",
    "In deep learning, we perform a lot of model training especially for novel neural architectures. The problem is deep learning frameworks like PyTorch do not provide sufficient tools to visualize input data, track the progress of our experiments, log data, and visualize the outputs. \n",
    "\n",
    "`wandb` addresses this problem. In this demo, we will train a ResNet18 model from scratch. We show how to use `wandb` to visualize input data, prediction, and training progress using loss function value and accuracy. \n",
    "\n",
    "**Note**: Before running this demo, please make sure that you have `wandb.ai` free account. \n",
    "\n",
    "Let us install `wandb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Collecting wandb\n",
      "  Downloading wandb-0.13.11-py3-none-any.whl (2.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m505.4 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting GitPython!=3.1.29,>=1.0.0\n",
      "  Downloading GitPython-3.1.31-py3-none-any.whl (184 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m184.3/184.3 kB\u001b[0m \u001b[31m1.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting docker-pycreds>=0.4.0\n",
      "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
      "Collecting setproctitle\n",
      "  Downloading setproctitle-1.3.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n",
      "Requirement already satisfied: protobuf!=4.21.0,<5,>=3.19.0 in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from wandb) (3.20.1)\n",
      "Requirement already satisfied: requests<3,>=2.0.0 in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from wandb) (2.28.2)\n",
      "Requirement already satisfied: appdirs>=1.4.3 in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from wandb) (1.4.4)\n",
      "Requirement already satisfied: PyYAML in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from wandb) (6.0)\n",
      "Requirement already satisfied: psutil>=5.0.0 in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from wandb) (5.9.4)\n",
      "Requirement already satisfied: setuptools in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from wandb) (65.6.3)\n",
      "Collecting sentry-sdk>=1.0.0\n",
      "  Downloading sentry_sdk-1.16.0-py2.py3-none-any.whl (184 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m184.3/184.3 kB\u001b[0m \u001b[31m1.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting pathtools\n",
      "  Downloading pathtools-0.1.2.tar.gz (11 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: Click!=8.0.0,>=7.0 in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from wandb) (8.1.3)\n",
      "Requirement already satisfied: six>=1.4.0 in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from docker-pycreds>=0.4.0->wandb) (1.16.0)\n",
      "Collecting gitdb<5,>=4.0.1\n",
      "  Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m1.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: charset-normalizer<4,>=2 in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from requests<3,>=2.0.0->wandb) (2.1.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from requests<3,>=2.0.0->wandb) (2022.12.7)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from requests<3,>=2.0.0->wandb) (1.26.14)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages (from requests<3,>=2.0.0->wandb) (3.4)\n",
      "Collecting smmap<6,>=3.0.1\n",
      "  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n",
      "Building wheels for collected packages: pathtools\n",
      "  Building wheel for pathtools (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8791 sha256=7810ccccff55dabd00705b5d372e0586637fc50f0f77408811c78f70df11530a\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-vh68sdn0/wheels/44/1b/54/249c94316d4e1030e2d0683fba1d8ea06197de866f5a4de738\n",
      "Successfully built pathtools\n",
      "Installing collected packages: pathtools, smmap, setproctitle, sentry-sdk, docker-pycreds, gitdb, GitPython, wandb\n",
      "Successfully installed GitPython-3.1.31 docker-pycreds-0.4.0 gitdb-4.0.10 pathtools-0.1.2 sentry-sdk-1.16.0 setproctitle-1.3.2 smmap-5.0.0 wandb-0.13.11\n"
     ]
    }
   ],
   "source": [
    "!pip install wandb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Import** the required modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "import wandb\n",
    "import datetime\n",
    "from torch.optim import SGD\n",
    "from torch.optim.lr_scheduler import CosineAnnealingLR\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets, transforms\n",
    "from ui import progress_bar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Login to and initialize** `wandb`. You will need to use your `wandb` API key to run this demo.\n",
    "\n",
    "As the config indicates, we will train our model using `cifar10` dataset, learning rate of `0.1`, and batch size of `128` for `100` epochs. \n",
    "\n",
    "epochs means a complete sampling of the dataset (train). In the `wandb` plots, step is the term used instead of epoch.  \n",
    "batch size is the number of samples per training step.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mrowel\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mrowel\u001b[0m (\u001b[33mupeee\u001b[0m). Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e966e66f138544ea883c0bd16f6a1229",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='Waiting for wandb.init()...\\r'), FloatProgress(value=0.016681649799769122, max=1.0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.13.11"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>/home/rowel/github/roatienza/Deep-Learning-Experiments/versions/2022/tools/python/wandb/run-20230308_135144-k6a0jiqo</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/upeee/wandb-project/runs/k6a0jiqo' target=\"_blank\">hardy-haze-6</a></strong> to <a href='https://wandb.ai/upeee/wandb-project' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/upeee/wandb-project' target=\"_blank\">https://wandb.ai/upeee/wandb-project</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/upeee/wandb-project/runs/k6a0jiqo' target=\"_blank\">https://wandb.ai/upeee/wandb-project/runs/k6a0jiqo</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wandb.login()\n",
    "config = {\n",
    "  \"learning_rate\": 0.1,\n",
    "  \"epochs\": 100,\n",
    "  \"batch_size\": 128,\n",
    "  \"dataset\": \"cifar10\"\n",
    "}\n",
    "run = wandb.init(project=\"wandb-project\", entity=\"upeee\", config=config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the model\n",
    "\n",
    "Use a ResNet18 from `torchvision`. Remove the last layer that was used for 1k-class ImageNet classification. Since we will use CIFAR10, the last layer is replaced by a linear layer with 10 outputs. We will train the model from scratch, so we set `pretrained=False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.\n",
      "  warnings.warn(\n",
      "/home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=None`.\n",
      "  warnings.warn(msg)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "model = torchvision.models.resnet18(pretrained=False, progress=True)\n",
    "\n",
    "model.fc = torch.nn.Linear(model.fc.in_features, 10)  \n",
    "model.to(device)\n",
    "\n",
    "# watch model gradients during training\n",
    "wandb.watch(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss function, Optimizer, Scheduler and DataLoader\n",
    "\n",
    "The appropriate loss function is cross entropy for multi-category classfications. We use `SGD` or stochastic gradient descent for optimization. Our learning rate that starts at `0.1` decays to zero at the end of total number of epochs. The decay is controlled by a cosine learning rate decay scheduler. \n",
    "\n",
    "Finally, we use `cifar10` dataset that is available in `torchvision`. We will discuss datasets and dataloaders in our future demo. For the meantime, we can treat dataloader as a data strcuture that dispenses batch size data from either the train or test split of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./data/cifar-10-python.tar.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "578666283b7048949fd461dca7565500",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/170498071 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./data/cifar-10-python.tar.gz to ./data\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "loss = torch.nn.CrossEntropyLoss()\n",
    "optimizer = SGD(model.parameters(), lr=wandb.config.learning_rate)\n",
    "scheduler = CosineAnnealingLR(optimizer, T_max=wandb.config.epochs)\n",
    "\n",
    "x_train = datasets.CIFAR10(root='./data', train=True, \n",
    "                           download=True, \n",
    "                           transform=transforms.ToTensor())\n",
    "x_test = datasets.CIFAR10(root='./data',\n",
    "                          train=False, \n",
    "                          download=True, \n",
    "                          transform=transforms.ToTensor())\n",
    "train_loader = DataLoader(x_train, \n",
    "                          batch_size=wandb.config.batch_size, \n",
    "                          shuffle=True, \n",
    "                          num_workers=2)\n",
    "test_loader = DataLoader(x_test, \n",
    "                         batch_size=wandb.config.batch_size, \n",
    "                         shuffle=False, \n",
    "                         num_workers=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing sample data from test split\n",
    "\n",
    "We can visualize data from the test split by getting a batch sample: `image, label = iter(test_loader).next()`. We use `wandb` table to create a column for image, grount truth label and initial model predicted label. The `wandb` table will show up when we run `wandb.log()` during training. \n",
    "\n",
    "CIFAR10 dataset is made of small `32x32` RGB images. Each image belongs to one of the 10 categories or classes. Below are sample images from CIFAR10 and their corresponding human labels.\n",
    "\n",
    "<img src=\"cifar10-samples.png\" width=\"600\" height=\"600\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cat vs.  automobile\n",
      "ship vs.  airplane\n",
      "ship vs.  airplane\n",
      "airplane vs.  automobile\n",
      "frog vs.  dog\n",
      "frog vs.  dog\n",
      "automobile vs.  ship\n",
      "frog vs.  automobile\n"
     ]
    }
   ],
   "source": [
    "\n",
    "label_human = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\", \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]\n",
    "\n",
    "table_test = wandb.Table(columns=['Image', \"Ground Truth\", \"Initial Pred Label\",])\n",
    "\n",
    "image, label = iter(test_loader).next()\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "  pred = torch.argmax(model(image.to(device)), dim=1).cpu().numpy()\n",
    "\n",
    "for i in range(8):\n",
    "  table_test.add_data(wandb.Image(image[i]),\n",
    "                      label_human[label[i]], \n",
    "                      label_human[pred[i]])\n",
    "  print(label_human[label[i]], \"vs. \",  label_human[pred[i]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The train loop\n",
    "\n",
    "At every epoch, we will run the train loop for the model. At every iteration, we will get a batch of data from the train split. We will use the data to update the model parameters. We will use the loss function to calculate the loss value. We will use the optimizer to update the model parameters. We will use the scheduler to update the learning rate. Later, we will use the `wandb` table to visualize the loss and accuracy.\n",
    "\n",
    "We use `progress_bar` to show the progress of the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(epoch):\n",
    "  model.train()\n",
    "  train_loss = 0\n",
    "  correct = 0\n",
    "  train_samples = 0\n",
    "\n",
    "  # sample a batch. compute loss and backpropagate\n",
    "  for batch_idx, (data, target) in enumerate(train_loader):\n",
    "    optimizer.zero_grad()\n",
    "    target = target.to(device)\n",
    "    output = model(data.to(device))\n",
    "    loss_value = loss(output, target)\n",
    "    loss_value.backward()\n",
    "    optimizer.step()\n",
    "    scheduler.step(epoch)\n",
    "    train_loss += loss_value.item()\n",
    "    train_samples += len(data)\n",
    "    pred = output.argmax(dim=1, keepdim=True)\n",
    "    correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "    if batch_idx % 10 == 0:\n",
    "      accuracy = 100. * correct / len(train_loader.dataset)\n",
    "      progress_bar(batch_idx,\n",
    "                   len(train_loader),\n",
    "                  'Train Epoch: {}, Loss: {:.6f}, Acc: {:.2f}%'.format(epoch+1, \n",
    "                  train_loss/train_samples, accuracy))\n",
    "  \n",
    "  train_loss /= len(train_loader.dataset)\n",
    "  accuracy = 100. * correct / len(train_loader.dataset)\n",
    "\n",
    "  return accuracy, train_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The validation loop\n",
    "\n",
    "After every epoch, we will run the validation loop for the model. In this way, we can track the progress of our model training. Both the average loss and accuracy are calculated. During training, we will use the `wandb` table to visualize the loss and accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test():\n",
    "  model.eval()\n",
    "  test_loss = 0\n",
    "  correct = 0\n",
    "  with torch.no_grad():\n",
    "    for data, target in test_loader:\n",
    "      output = model(data.to(device))   \n",
    "      target = target.to(device)\n",
    "\n",
    "      test_loss += loss(output, target).item()\n",
    "      pred = output.argmax(dim=1, keepdim=True)\n",
    "      correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "\n",
    "  test_loss /= len(test_loader.dataset)\n",
    "  accuracy = 100. * correct / len(test_loader.dataset)\n",
    "\n",
    "  print('\\nTest Loss: {:.4f}, Acc: {:.2f}%\\n'.format(test_loss, accuracy))\n",
    "\n",
    "  return accuracy, test_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `wandb` plots\n",
    "\n",
    "Finally, we will use `wandb` to visualize the training progress. We will use the following plots:\n",
    "- Model gradients (`wandb.watch(model)`)\n",
    "- Train and test losses (`\"train loss\": train_loss,` `\"test loss\": test_loss,`)\n",
    "- Train and validation accuracies (`\"Train accuracy\": train_acc, \"Test accuracy\": test_acc,`)\n",
    "- Learning rate which decreases over epochs (`\"Learning rate\": optimizer.param_groups[0]['lr']`)\n",
    "\n",
    "We re-use the earlier `table_test` to see the final prediction.\n",
    "\n",
    "We save the best peforming model to `./resnet18_best_acc.pth`. This can be used as a pretrained model like the pre-trained model in `torchvision`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages/IPython/core/display.py:431: UserWarning: Consider using IPython.display.IFrame instead\n",
      "  warnings.warn(\"Consider using IPython.display.IFrame instead\")\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe src='https://wandb.ai/upeee/wandb-project/runs/k6a0jiqo?jupyter=true' style='border:none;width:100%;height:1000px;'></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/rowel/anaconda3/envs/speech/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:156: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.\n",
      "  warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " [=============================>]  Step: 167ms | Tot: 6s671ms | Train Epoch: 1, Loss: 0.012777, Acc: 44.41 391/391 \n",
      "\n",
      "Test Loss: 0.0131, Acc: 43.43%\n",
      "\n",
      " [=============================>]  Step: 172ms | Tot: 6s669ms | Train Epoch: 2, Loss: 0.008770, Acc: 60.21 391/391 \n",
      "\n",
      "Test Loss: 0.0144, Acc: 41.04%\n",
      "\n",
      " [=============================>]  Step: 169ms | Tot: 6s766ms | Train Epoch: 3, Loss: 0.007061, Acc: 68.31 391/391 \n",
      "\n",
      "Test Loss: 0.0130, Acc: 48.72%\n",
      "\n",
      " [=============================>]  Step: 172ms | Tot: 6s653ms | Train Epoch: 4, Loss: 0.005908, Acc: 73.41 391/391 \n",
      "\n",
      "Test Loss: 0.0136, Acc: 51.99%\n",
      "\n",
      " [=============================>]  Step: 171ms | Tot: 6s748ms | Train Epoch: 5, Loss: 0.005025, Acc: 77.38 391/391 \n",
      "\n",
      "Test Loss: 0.0138, Acc: 53.59%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 6s728ms | Train Epoch: 6, Loss: 0.004174, Acc: 81.16 391/391 \n",
      "\n",
      "Test Loss: 0.0096, Acc: 61.33%\n",
      "\n",
      " [=============================>]  Step: 173ms | Tot: 6s818ms | Train Epoch: 7, Loss: 0.003460, Acc: 84.40 391/391 \n",
      "\n",
      "Test Loss: 0.0083, Acc: 67.56%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 6s777ms | Train Epoch: 8, Loss: 0.002897, Acc: 86.84 391/391 \n",
      "\n",
      "Test Loss: 0.0096, Acc: 66.08%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 6s909ms | Train Epoch: 9, Loss: 0.002314, Acc: 89.51 391/391 \n",
      "\n",
      "Test Loss: 0.0112, Acc: 65.24%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 6s872ms | Train Epoch: 10, Loss: 0.001943, Acc: 91.17 391/391 \n",
      "\n",
      "Test Loss: 0.0097, Acc: 68.87%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 6s938ms | Train Epoch: 11, Loss: 0.001504, Acc: 93.29 391/391 \n",
      "\n",
      "Test Loss: 0.0098, Acc: 68.82%\n",
      "\n",
      " [=============================>]  Step: 179ms | Tot: 6s894ms | Train Epoch: 12, Loss: 0.001264, Acc: 94.41 391/391 \n",
      "\n",
      "Test Loss: 0.0225, Acc: 53.63%\n",
      "\n",
      " [=============================>]  Step: 179ms | Tot: 6s965ms | Train Epoch: 13, Loss: 0.001018, Acc: 95.50 391/391 \n",
      "\n",
      "Test Loss: 0.0113, Acc: 67.96%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s11ms | Train Epoch: 14, Loss: 0.000886, Acc: 96.05 391/391  \n",
      "\n",
      "Test Loss: 0.0132, Acc: 67.24%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 6s935ms | Train Epoch: 15, Loss: 0.000746, Acc: 96.69 391/391 \n",
      "\n",
      "Test Loss: 0.0153, Acc: 63.35%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 6s974ms | Train Epoch: 16, Loss: 0.000648, Acc: 97.11 391/391 \n",
      "\n",
      "Test Loss: 0.0115, Acc: 72.00%\n",
      "\n",
      " [=============================>]  Step: 174ms | Tot: 6s928ms | Train Epoch: 17, Loss: 0.000512, Acc: 97.70 391/391 \n",
      "\n",
      "Test Loss: 0.0120, Acc: 71.33%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 7s2ms | Train Epoch: 18, Loss: 0.000470, Acc: 97.87 391/391 1 \n",
      "\n",
      "Test Loss: 0.0120, Acc: 71.79%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s10ms | Train Epoch: 19, Loss: 0.000375, Acc: 98.33 391/391  \n",
      "\n",
      "Test Loss: 0.0129, Acc: 71.69%\n",
      "\n",
      " [=============================>]  Step: 180ms | Tot: 6s948ms | Train Epoch: 20, Loss: 0.000303, Acc: 98.69 391/391 \n",
      "\n",
      "Test Loss: 0.0136, Acc: 71.11%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 7s12ms | Train Epoch: 21, Loss: 0.000337, Acc: 98.47 391/391  \n",
      "\n",
      "Test Loss: 0.0126, Acc: 72.56%\n",
      "\n",
      " [=============================>]  Step: 179ms | Tot: 6s993ms | Train Epoch: 22, Loss: 0.000272, Acc: 98.81 391/391 \n",
      "\n",
      "Test Loss: 0.0166, Acc: 68.96%\n",
      "\n",
      " [=============================>]  Step: 174ms | Tot: 6s947ms | Train Epoch: 23, Loss: 0.000281, Acc: 98.80 391/391 \n",
      "\n",
      "Test Loss: 0.0123, Acc: 72.79%\n",
      "\n",
      " [=============================>]  Step: 181ms | Tot: 7s10ms | Train Epoch: 24, Loss: 0.000219, Acc: 99.11 391/391  \n",
      "\n",
      "Test Loss: 0.0189, Acc: 63.55%\n",
      "\n",
      " [=============================>]  Step: 179ms | Tot: 7s | Train Epoch: 25, Loss: 0.000171, Acc: 99.26 391/391 /391 \n",
      "\n",
      "Test Loss: 0.0134, Acc: 73.63%\n",
      "\n",
      " [=============================>]  Step: 179ms | Tot: 7s29ms | Train Epoch: 26, Loss: 0.000099, Acc: 99.58 391/391  \n",
      "\n",
      "Test Loss: 0.0139, Acc: 73.89%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 6s991ms | Train Epoch: 27, Loss: 0.000156, Acc: 99.30 391/391 \n",
      "\n",
      "Test Loss: 0.0139, Acc: 73.29%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 7s18ms | Train Epoch: 28, Loss: 0.000125, Acc: 99.49 391/391  \n",
      "\n",
      "Test Loss: 0.0135, Acc: 74.14%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s27ms | Train Epoch: 29, Loss: 0.000082, Acc: 99.66 391/391  \n",
      "\n",
      "Test Loss: 0.0144, Acc: 73.67%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 7s2ms | Train Epoch: 30, Loss: 0.000071, Acc: 99.69 391/391 1 \n",
      "\n",
      "Test Loss: 0.0152, Acc: 73.00%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 7s79ms | Train Epoch: 31, Loss: 0.000059, Acc: 99.78 391/391  \n",
      "\n",
      "Test Loss: 0.0149, Acc: 74.24%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 6s995ms | Train Epoch: 32, Loss: 0.000091, Acc: 99.62 391/391 \n",
      "\n",
      "Test Loss: 0.0149, Acc: 72.70%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 6s988ms | Train Epoch: 33, Loss: 0.000064, Acc: 99.74 391/391 \n",
      "\n",
      "Test Loss: 0.0149, Acc: 73.93%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 7s41ms | Train Epoch: 34, Loss: 0.000051, Acc: 99.78 391/391  \n",
      "\n",
      "Test Loss: 0.0147, Acc: 74.92%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 6s995ms | Train Epoch: 35, Loss: 0.000054, Acc: 99.77 391/391 \n",
      "\n",
      "Test Loss: 0.0162, Acc: 73.16%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s25ms | Train Epoch: 36, Loss: 0.000041, Acc: 99.83 391/391  \n",
      "\n",
      "Test Loss: 0.0158, Acc: 73.34%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 7s19ms | Train Epoch: 37, Loss: 0.000025, Acc: 99.91 391/391  \n",
      "\n",
      "Test Loss: 0.0164, Acc: 73.50%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 6s982ms | Train Epoch: 38, Loss: 0.000018, Acc: 99.92 391/391 \n",
      "\n",
      "Test Loss: 0.0153, Acc: 74.62%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s34ms | Train Epoch: 39, Loss: 0.000010, Acc: 99.96 391/391  \n",
      "\n",
      "Test Loss: 0.0157, Acc: 74.76%\n",
      "\n",
      " [=============================>]  Step: 179ms | Tot: 7s11ms | Train Epoch: 40, Loss: 0.000004, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0160, Acc: 74.79%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 7s30ms | Train Epoch: 41, Loss: 0.000005, Acc: 99.98 391/391  \n",
      "\n",
      "Test Loss: 0.0180, Acc: 72.26%\n",
      "\n",
      " [=============================>]  Step: 181ms | Tot: 6s999ms | Train Epoch: 42, Loss: 0.000005, Acc: 99.98 391/391 \n",
      "\n",
      "Test Loss: 0.0160, Acc: 74.97%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 6s988ms | Train Epoch: 43, Loss: 0.000003, Acc: 99.99 391/391 \n",
      "\n",
      "Test Loss: 0.0160, Acc: 75.36%\n",
      "\n",
      " [=============================>]  Step: 180ms | Tot: 7s16ms | Train Epoch: 44, Loss: 0.000004, Acc: 99.98 391/391  \n",
      "\n",
      "Test Loss: 0.0163, Acc: 74.97%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 7s13ms | Train Epoch: 45, Loss: 0.000002, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0166, Acc: 75.05%\n",
      "\n",
      " [=============================>]  Step: 179ms | Tot: 7s29ms | Train Epoch: 46, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0167, Acc: 75.17%\n",
      "\n",
      " [=============================>]  Step: 173ms | Tot: 6s882ms | Train Epoch: 47, Loss: 0.000002, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0165, Acc: 75.10%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s21ms | Train Epoch: 48, Loss: 0.000002, Acc: 99.99 391/391  \n",
      "\n",
      "Test Loss: 0.0165, Acc: 75.05%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 7s80ms | Train Epoch: 49, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0167, Acc: 74.96%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 7s15ms | Train Epoch: 50, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0168, Acc: 75.13%\n",
      "\n",
      " [=============================>]  Step: 185ms | Tot: 7s12ms | Train Epoch: 51, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0167, Acc: 75.15%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s16ms | Train Epoch: 52, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0169, Acc: 75.09%\n",
      "\n",
      " [=============================>]  Step: 174ms | Tot: 7s27ms | Train Epoch: 53, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0171, Acc: 75.07%\n",
      "\n",
      " [=============================>]  Step: 180ms | Tot: 6s990ms | Train Epoch: 54, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0172, Acc: 75.10%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s9ms | Train Epoch: 55, Loss: 0.000001, Acc: 100.00 391/391  \n",
      "\n",
      "Test Loss: 0.0173, Acc: 75.12%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 7s20ms | Train Epoch: 56, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0173, Acc: 74.94%\n",
      "\n",
      " [=============================>]  Step: 179ms | Tot: 7s31ms | Train Epoch: 57, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0170, Acc: 75.06%\n",
      "\n",
      " [=============================>]  Step: 174ms | Tot: 6s966ms | Train Epoch: 58, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0170, Acc: 75.09%\n",
      "\n",
      " [=============================>]  Step: 185ms | Tot: 7s40ms | Train Epoch: 59, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0174, Acc: 75.07%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 6s989ms | Train Epoch: 60, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0171, Acc: 74.87%\n",
      "\n",
      " [=============================>]  Step: 179ms | Tot: 7s4ms | Train Epoch: 61, Loss: 0.000000, Acc: 100.00 391/391  \n",
      "\n",
      "Test Loss: 0.0171, Acc: 75.17%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 7s18ms | Train Epoch: 62, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0176, Acc: 75.02%\n",
      "\n",
      " [=============================>]  Step: 174ms | Tot: 6s988ms | Train Epoch: 63, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.11%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 7s39ms | Train Epoch: 64, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.08%\n",
      "\n",
      " [=============================>]  Step: 174ms | Tot: 7s11ms | Train Epoch: 65, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0172, Acc: 75.12%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 6s963ms | Train Epoch: 66, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0176, Acc: 75.14%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 7s24ms | Train Epoch: 67, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.09%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 7s9ms | Train Epoch: 68, Loss: 0.000000, Acc: 100.00 391/391  \n",
      "\n",
      "Test Loss: 0.0172, Acc: 75.18%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 7s11ms | Train Epoch: 69, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0173, Acc: 75.16%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 7s16ms | Train Epoch: 70, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.19%\n",
      "\n",
      " [=============================>]  Step: 179ms | Tot: 6s995ms | Train Epoch: 71, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0173, Acc: 75.30%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 6s942ms | Train Epoch: 72, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.16%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 6s973ms | Train Epoch: 73, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 74.84%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 6s991ms | Train Epoch: 74, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0173, Acc: 75.14%\n",
      "\n",
      " [=============================>]  Step: 180ms | Tot: 7s32ms | Train Epoch: 75, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.17%\n",
      "\n",
      " [=============================>]  Step: 172ms | Tot: 6s948ms | Train Epoch: 76, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0172, Acc: 75.14%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 7s24ms | Train Epoch: 77, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0173, Acc: 75.09%\n",
      "\n",
      " [=============================>]  Step: 180ms | Tot: 7s40ms | Train Epoch: 78, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0174, Acc: 75.23%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s28ms | Train Epoch: 79, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0174, Acc: 75.24%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s30ms | Train Epoch: 80, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.04%\n",
      "\n",
      " [=============================>]  Step: 174ms | Tot: 6s976ms | Train Epoch: 81, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.06%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 6s880ms | Train Epoch: 82, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.08%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 6s984ms | Train Epoch: 83, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.21%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 7s14ms | Train Epoch: 84, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.21%\n",
      "\n",
      " [=============================>]  Step: 180ms | Tot: 7s50ms | Train Epoch: 85, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0176, Acc: 75.18%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 7s12ms | Train Epoch: 86, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0176, Acc: 75.09%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 7s16ms | Train Epoch: 87, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0172, Acc: 75.31%\n",
      "\n",
      " [=============================>]  Step: 182ms | Tot: 7s17ms | Train Epoch: 88, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0171, Acc: 75.28%\n",
      "\n",
      " [=============================>]  Step: 177ms | Tot: 6s892ms | Train Epoch: 89, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0174, Acc: 75.32%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 7s16ms | Train Epoch: 90, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0174, Acc: 75.29%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 7s15ms | Train Epoch: 91, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.08%\n",
      "\n",
      " [=============================>]  Step: 180ms | Tot: 6s992ms | Train Epoch: 92, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0171, Acc: 75.09%\n",
      "\n",
      " [=============================>]  Step: 184ms | Tot: 7s48ms | Train Epoch: 93, Loss: 0.000001, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0174, Acc: 75.22%\n",
      "\n",
      " [=============================>]  Step: 178ms | Tot: 6s991ms | Train Epoch: 94, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0176, Acc: 75.17%\n",
      "\n",
      " [=============================>]  Step: 185ms | Tot: 7s7ms | Train Epoch: 95, Loss: 0.000000, Acc: 100.00 391/391  \n",
      "\n",
      "Test Loss: 0.0174, Acc: 75.09%\n",
      "\n",
      " [=============================>]  Step: 185ms | Tot: 7s11ms | Train Epoch: 96, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0178, Acc: 75.16%\n",
      "\n",
      " [=============================>]  Step: 182ms | Tot: 7s20ms | Train Epoch: 97, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0179, Acc: 75.16%\n",
      "\n",
      " [=============================>]  Step: 175ms | Tot: 7s9ms | Train Epoch: 98, Loss: 0.000000, Acc: 100.00 391/391  \n",
      "\n",
      "Test Loss: 0.0176, Acc: 75.32%\n",
      "\n",
      " [=============================>]  Step: 176ms | Tot: 6s996ms | Train Epoch: 99, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0175, Acc: 75.32%\n",
      "\n",
      " [=============================>]  Step: 182ms | Tot: 7s51ms | Train Epoch: 100, Loss: 0.000000, Acc: 100.00 391/391 \n",
      "\n",
      "Test Loss: 0.0177, Acc: 75.20%\n",
      "\n",
      "Elapsed time: 0:12:39.240327\n",
      "cat vs.  cat\n",
      "ship vs.  ship\n",
      "ship vs.  ship\n",
      "airplane vs.  airplane\n",
      "frog vs.  frog\n",
      "frog vs.  frog\n",
      "automobile vs.  cat\n",
      "frog vs.  frog\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[32m\u001b[41mERROR\u001b[0m Control-C detected -- Run data was not synced\n"
     ]
    }
   ],
   "source": [
    "run.display(height=1000)\n",
    "\n",
    "start_time = datetime.datetime.now()\n",
    "best_acc = 0\n",
    "for epoch in range(wandb.config[\"epochs\"]):\n",
    "    train_acc, train_loss = train(epoch)\n",
    "    test_acc, test_loss = test()\n",
    "    if test_acc > best_acc:\n",
    "        wandb.run.summary[\"Best accuracy\"] = test_acc\n",
    "        best_acc = test_acc\n",
    "        torch.save(model, \"resnet18_best_acc.pth\")\n",
    "    wandb.log({\n",
    "        \"Train accuracy\": train_acc,\n",
    "        \"Test accuracy\": test_acc,\n",
    "        \"Train loss\": train_loss,\n",
    "        \"Test loss\": test_loss,\n",
    "        \"Learning rate\": optimizer.param_groups[0]['lr']\n",
    "    })\n",
    "\n",
    "elapsed_time = datetime.datetime.now() - start_time\n",
    "print(\"Elapsed time: %s\" % elapsed_time)\n",
    "wandb.run.summary[\"Elapsed train time\"] = str(elapsed_time)\n",
    "\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "  pred = torch.argmax(model(image.to(device)), dim=1).cpu().numpy()\n",
    "\n",
    "final_pred = []\n",
    "for i in range(8):\n",
    "    final_pred.append(label_human[pred[i]])\n",
    "    print(label_human[label[i]], \"vs. \",  final_pred[i])\n",
    "\n",
    "table_test.add_column(name=\"Final Pred Label\", data=final_pred)\n",
    "\n",
    "wandb.log({\"Test data\": table_test})\n",
    "\n",
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the best performing model\n",
    "\n",
    "In the following code, we load the best performing model. The model is saved in `./resnet18_best_acc.pth`. The average accuracy of the model is the same as the one in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Test Loss: 0.0160, Acc: 75.36%\n",
      "\n",
      "Best accuracy: 75.36\n"
     ]
    }
   ],
   "source": [
    "model = torch.load(\"resnet18_best_acc.pth\")\n",
    "accuracy, _ = test()\n",
    "print(\"Best accuracy: %.2f\" % accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "975196f818cef4d8418ca9caaf131be767e393cdee268c01707e7bb8d73c6de2"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}