11_training_deep_neural_networks.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chương 11 – Huấn luyện Mạng Nơ-ron Sâu**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Notebook này sẽ chứa tất cả các mã lập trình mẫu và lời giải cho các bài tập trong chương 11._"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/mlbvn/handson-ml2-vn/blob/master/11_training_deep_neural_networks.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://kaggle.com/kernels/welcome?src=https://github.com/mlbvn/handson-ml2-vn/blob/master/11_training_deep_neural_networks.ipynb\"><img src=\"https://kaggle.com/static/images/open-in-kaggle.svg\" /></a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cài đặt"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Đầu tiên hãy nhập một vài mô-đun thông dụng, đảm bảo rằng Matplotlib sẽ vẽ đồ thị ngay trong notebook, và chuẩn bị một hàm để lưu đồ thị. Ta cũng kiểm tra xem Python phiên bản từ 3.5 trở lên đã được cài đặt hay chưa (mặc dù Python 2.x vẫn có thể hoạt động, phiên bản này đã không còn sử dụng nên chúng tôi rất khuyến khích việc sử dụng Python 3), cũng như Scikit-Learn ≥ 0.20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "%load_ext tensorboard\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"deep\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vấn đề Tiêu biến/Bùng nổ Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logit(z):\n",
    "    return 1 / (1 + np.exp(-z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure sigmoid_saturation_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABNSklEQVR4nO3dd3gUVffA8e9JJQkdlBpACVV4QYIKghK6NKWKiEqRIvxUQIVXXkVQsTcQpagg0jsqIkWEoHSRjiCiVOklkJC+e39/TBJSNgXYZDfJ+TzPPLAzd+4chsme3Jk794oxBqWUUsrdeLg6AKWUUsoRTVBKKaXckiYopZRSbkkTlFJKKbekCUoppZRb0gSllFLKLWmCUi4lIo1FZLCr40hORAaKSAtXx6FUfqcJSrna78CTItLf1YEAiMgAYCCwNQtlQ0Xksxusv7eIRNxsfK7gTjFnJRYReUlEjuZQSCobaYJSTici00XkBwfr64uIEZFKieuMMVFAB+A5EamTjTGl+WITkSAR+UdEVolIgIjcCwwB2hhjwrMrlpxwI0lFRPqJyE4RiRCRKyKyR0TGJisyH7gzeyK9Ye4Ui8pmXq4OQCljzAXgPzl5TBGpC6wEQoGnjDGxwDbgrpyMw9VEpC/wKTAM+BnwwToHDRPLJPwSEeWSAFNxp1hU9tMWlHIZEfEUkakickREokTkLxEZISIeycpMF5EfROS/InIm4Tf8d0XEQ0TGiMi5hPX/vYHjPoCVmBYDjyckJ0QkJKGFVzJZ2QYislZEriUc+2cRKZusOg8ReVtELiTE8mHy+DOIobmI7Euod52I3JFq+0AROSwisQl/9k+1/YWEls41EflXRL4SkaKJ/w7gayAg4d9jRGRMOqE8DCwxxkwxxhw2xvxhjFlojHkh2bEctT5HisjZhFbXDBEZnfy22s3+v4lIBRFZKiLhCcsSESmfSSwjEuqKEJEZQMGMz77KLTRBKVfyAP4FHgVqAK8A/wP6pCr3IHAHEAI8A4wAfgR8gcbAGOBdEQnO7IAi0gFYBXxqjPk/Y4w9g7J1gHXAYaAR0ABYQMo7Dz2BeOB+4FlgKNA9kzB8gZFAX6yWSlFgcrLjdgI+A8YBtYDxwMSE2BPZE451F/A4cC8wIWHbpoRtkUCZhOXDdGI5A9wrIlm+bSYijwGjsf6/6gEHgBccFL2h/zcREeBboBTQDGgKlAW+TdjmKJZHgbEJ8dQD/kwnFpUbGWN00cWpCzAd60s7ItUSCRigUgb7vgusSVXXCcAz2brtwJ5U+x0FXsqg3t6ADYgDPkinTEhCfCUTPs8GtmRQZyiwOdW6n4CvMonDANWSresJxAIeCZ83AtMcnNMNGdT7EBCTrI7eQEQW/q/KAJsTYvoLmAU8BXinijki2efNwORU9awGjt7K/xvQMuH/qFKy7XdiJeMW6cSyCfgyVZ1rkseiS+5dtAWlsssvQN1Uy+OpC4nIMyKyXUTOJ9y6GQZUSFXsD2OMLdnns8DeVGXOArdnElMM1nOn3lnskHE31nOZjOxJ9flUVuIwxvyZah9vrJYUWK3Jjan22QDUTPwgIs1E5CcROSki4cASrOdHpTM5dgrGmNPGmIZAbawWmwBTgG0i4p/ObtWxntcl56jX443+v9UAThljjiaL7x+s81MTx2pgJczkUn9WuZQmKJVdIo31TCNpAU4mLyAi3bG+FKcDrbGS2ESsL9rk4lJ9Numsy+x6tgNdsL7s14rI3ZmUd3hbKQuxZRZHvIN9SLWfo3lwDICIVASWY91a6wYEY90uhLTnLkuMMfuMMZ8bY3pitWTqYt16TXeXLFR7o/9vkkG9Oi9QPqQJSrlSY2CrMeYzY8yOhCRWOTsPaKwOEd2AtcDPmTy32oH1LCSnHcA6N8k1Bv5I+Ht9rEQ0zBiz2RhzCOtZTXKxgOdNHj/xOOl1NjiI9cwrudSfb/a45STZawgJz8bKJosptQNYzwaTS/1Z5VLazVy50iGs221tsDoiPAY0AS5n50GNMfEi0gOYCawRkdbGmNS3rAA+ALaIyBfA50A08ACw2hhzPBtD/ABYKCK/Yz3beQjrOVXnhO1/Yf1yOVRElmB9IQ9NVcdRoICItAR2YrVoI1MfSEQmYd1CW4vVwi0DvIr1vHB1OvGNB74Wkd+AX4FOwH3c+v/bGmA3MFtEnsdqUU3A+kVhbQaxzEiIJRTomhDLpVuMRbkBbUEpV5qC1StuDvAbUAn4KCcObIyJB54AfgB+EpE0v3UbY3YBLbCeuWzBes7yGGlvUzk7tm+B57Cex/2B9fLwYGPMsoTtexLWvZCwvR/wUqo6NmH1DJwLnMfqQefIT1hf6AuwfmFYmrC+ZULLzFF884A3sTq07MTqaTgZK4HfNGOMATomxBuK1YPyDNAxYZujfeZj9QZ8KyGW2sDHtxKHch+Szv+7UkplmYgsBbyMMR0yLaxUFmkLSqlcSkTuTng5NXWvx+w+rr+IvCgid4lINRH5H/AIMC0n41B5n1MTlIg8m9BlOEZEpmdQrpeI/C4iVxO6yb4vIvo8TKkb8wdWb7tTOXxcA7TBepVgJ9aLyU8aY5ZmuJdSN8ipt/hEpDNWV97WgJ8xpnc65QYB+7Du6d8GfA8sNMa867RglFJK5WpObbUYY5aANWo1UD6DcpOSffxXRGZjDWuilFJKAe7TzfxBYL+jDWLNzzMAwM/PLzgwMDAn48oSu92Oh4c+zssKPVeZO3HiBMYYKlTI0UdLuVZOX1MR8RH4ePjg43FT70S7lLv+/B06dOiCMea21OtdnqBEpA/Wi4f9HG03xnwBfAFQv359s3379hyMLmtCQ0MJCQlxdRi5gp6rzIWEhBAWFsauXbtcHUqukJPX1MrDK+kwtwMPV3uYxY8uzpFjOpO7/vyJyDFH612aoESkI9a7FC2MNSeQUkq5pa0nt9JlQRdq3V6LaQ9rh8Wc4LIEJSIPAV8C7YwxqQeQVEopt3Hg/AHazmlLmYJlWNlzJUUKFHF1SPmCUxNUQldxL6wxwDxFpAAQn/DWfvJyzbCmMuiUzhAzSinlNt785U18PH1Y/eRqShUs5epw8g1nPy17FWs65pexhpGJAl5NmCUz+QuFo4AiwI8J6yNEZIWTY1FKKaeY+vBU1vdez53Fsjyvo3ICpyYoY8wYY4ykWsYYY44bYwomDrBpjGlqjPFKWJe4tHFmLEopdSsiYiN47sfnCIsOw8/bj6olqro6pHzH/fobKqWUi8XEx9B5fmcmbZ/Eb//+5upw8i2XdzNXSil3YrPbeOrbp/jpn5/4+pGvaVm5patDyre0BaWUUgmMMTy/4nkW7F/ABy0/oHfd3q4OKV/TBKWUUgkuRF5g2aFlDL9/OC/d/1LmO6hspbf4lFIqwW0Bt7Fj4A5K+JVwdSgKbUEppRQL9i/g2R+fxWa3UdK/JCLi6pAUmqCUUvnc6r9X88SSJ9hzdg9x9jhXh6OS0QSllMq3tp7cSuf5nalxWw2+7/E9BbwKuDoklYwmKKVUvnTg/AHazWnH7QG3s7LnSooWKOrqkFQqmqCUUvnS8SvHKeRbiNVPrqZMoTKuDkc5oL34lFL5is1uw9PDk9ZBrfnz2T/x8cx9Ew/mF9qCUkrlGxGxETw4/UGm7bTmc9Lk5N40QSml8oVYWyxdFnRhy8ktFPcr7upwVBboLT6lVJ5nN3Z6fduL1X+v5qsOX9GxekdXh6SyQFtQSqk8zRjDkBVDmLdvHu82f5en6z3t6pBUFmmCUkrleWUKleGlhi8xotEIV4eiboDe4lNK5VlXoq9QpEAR/vfA/zDG6BBGuYy2oJRSedLC/Qup/Gll9pzdA6DJKRfSBKWUynPW/LOGnkt6UuO2GgQVD3J1OOomaYJSSuUp209tp9P8TlQvWZ1lPZbh7+3v6pDUTdIEpZTKM/65/A9tZrehpH9JVj2xSsfXy+U0QSml8oxyhcrRpUYXVj+h4+vlBU5NUCLyrIhsF5EYEZmeSdlhInJGRK6IyDQR8XVmLEqp/ONS1CUuRV3C18uXye0nU6VEFVeHpJzA2S2oU8BYYFpGhUSkNfAy0ByoBNwJvO7kWJRS+UCULYp2c9rRamYrbHabq8NRTuTUBGWMWWKM+Ra4mEnRXsBUY8x+Y8xl4E2gtzNjUUrlfbG2WMbsH8O2f7fxygOv4Onh6eqQlBO56kXdu4Dvkn3eDZQSkRLGmHST259//klISEiKdY8++iiDBw8mMjKStm3bptmnd+/e9O7dmwsXLtC1a9c02wcNGkT37t05ceIETz75ZJrtL774Ih06dODPP/9k4MCBaba/+uqreHl5sWvXLoYOHZpm+9tvv83999/Ppk2b+N///pdm+7hx46hbty5r1qxh7NixabZPmTKFatWqsWzZMj766KM022fOnElgYCDz589n0qRJabYvWrSIkiVLMn36dKZPn55m+48//oi/vz8TJ05kwYIFabaHhoYC8OGHH/LDDz+k2Obn58eKFSsAePPNN/n5559TbC9RogSLFy8GYOTIkWzevJmwsDCKFi0KQPny5Zk1axYAQ4cOZdeuXSn2r1q1Kl988QUAAwYM4NChQym2161bl3HjxgHwxBNPcPLkyRTbGzZsyDvvvANAly5duHgx5aXVvHlzRo0aBUCbNm2IiopKsb19+/a89NJLAGmuO8i+a2/Xrl3Ex8ezbNmyTK+9Fi1a5Ntrb/mPy+nzXR+2Xd5G1YNVGT9oPOMZDzi+9pLLr9de4s+fM773snLtbdy4iZEjX8Nu98Vu98Vm88Fu92XYsP9RoUI1Nm7cwcyZi9Lsn8hVCaogcCXZ58S/FyJV60tEBgADALy9vQkLC0tR0aFDhwgNDSU6OjrNNoCDBw8SGhrKlStXHG7fv38/oaGhnDt3zuH2vXv3UqhQIY4fP+5w++7du6lWrRqHDx92uH3Hjh3Exsayb98+h9u3b99OWFgYu3fvdrh969atnD59mr179zrcvnnzZv7++2/279/vcPvGjRspUqQIBw8edLj9l19+oUCBAhw6dMjh9sQvib///jvN9qioqKTtR44cSbPdbrcnbU88fzabLamct7d30vaTJ0+m2f/UqVNJ20+dOpVm+8mTJ5O2nz17Ns3248ePJ20/f/48V69eTbH9yJEjSdsvXbpETExMiu1///130nZH5ya7rr34+HiMMVm69ry8vPLttddvRj/mHJ1DnQt14ACEcb2Mo2svufx67SX+/KV37VlJpDBr157i339/5+jRixw7Vhe73R+bLQC7PQCbLYDXXy/ORx+d5vz5wvz559vY7f7Y7QUwxkpErVsXxWaLJzq6AcasSRNfz56Jf6uXsLyTpgyAGGMcbrgVIjIWKG+M6Z3O9t3AW8aYBQmfSwAXgJIZtaDq169vtm/f7vR4b1VoaKjD33JUWnquMhcSEkJYWFia3+pVSqfCTzFv3zzujr6bpk2bujoct2IMhIXB2bNw5gxcuACXLsH27f9QpMidXLqEwyU62vmxeHqCnx/4+1t/Olp++EF+N8bUT72vq1pQ+4E6QGK7vg5wNqPkpJRSAKFHQ2lcoTFlC5XlhYYvJLU08gNj4PJlOHECjh+3ljNnri+JCensWYiNdVTDnRnW7+MDxYpB4cJQqND1JfXn1OsLFrQSkKMk5O2d9jg2mw0PD4+k4afSG4XKqQlKRLwS6vQEPEWkABBvjIlPVXQGMF1EZgOngVeB6c6MRSmV9yz6YxGPLnyUt5q9xcgHRro6HKczBs6dg7/+gn/+uZ6Eki/XrmWtroIFoXRpKFUKbrsNSpSAa9eOU7duBYoXx+Hi759+snCGyMhIxo0bx7vvvsvixYtp2bJlhuWd3YJ6FRid7PMTwOsiMg34A6hpjDlujFkpIu8D6wA/YHGq/ZRSKoW1R9bSc0lPGgY2ZEiDIa4O55aEh8P+/XDokJWMki/h4RnvW7AgVKhgLYGBULbs9USU+GepUhAQkHbf0NB/CAmpkD3/qAzExsYyefJkRo8eTWxsLPHx8ZQpk/mL1E5NUMaYMcCYdDYXTFX2Y+BjZx5fKZU3/X7qdx6Z9whVilfJVePrxcbCn3/C3r2wb5/15969cOxY+vsULQpVqkDlylCx4vVklLgUKZK9rRxnio+PZ8aMGbz88stERkZyLaH55+npSZUqmb9MrfNBKaXcWpwtjm4Lu1HCrwSrnlhFcb/irg7Jofh4q1W0bRv89pu17NtnrU/Nxwdq1IDq1a1klHwpUSL3JKD02O12Fi1axIsvvsjly5eTElOiUqVK4eub+eBBmqCUUm7N29ObeV3nUbRAUcoVLufqcJKcPAm//molpG3bYOdOSPU6EyJW0qlVC2rXtv6sVcta55UHv32NMaxYsYKhQ4dy6tSpNIkpUY0aNbJUXx48RUqpvOBS1CVW/LWCnv/pyb3l7nVpLMbAkSOwfj388ou1/PNP2nKVK8M998C991p/3n2342dBedH69esZMmQIhw8fTjcxgTVx5D333JOlOjVBKaXcTmRcJO3ntOf307/TqEIjKhWtlOMxhIXBzz/DqlXWcvx4yu2FCkGjRnD//VZCql/fuj2XH3311Vf0798/S2ULFizIf/7znyyV1QSllHIrcbY4ui7oypaTW1jYbWGOJSdjYPduWLYMVq6ErVvBlmzs2eLF4cEHry916uTN23Q3o3v37qxevZrly5cTGRmZaXm9xaeUynXsxk7f7/uy4vAKprSfQpeaXbL1eDYbbNoES5day9Gj17d5eVmJqHVra7n7bvDQGfQcKlSoEAsWLGDmzJkMGjSIqKgo7Ha7w7JRUVFUq1YtS/VqglJKuY31R9cza88sxjYdy4DgAdlyDLvdeoY0Zw58+y2cP399W6lS8PDD0LYtNGtmjZSgsu7RRx/lzTff5PDhw+mWKVGiBH5+flmqTxOUUsptNL2jKVue3pItnSL27oVZs6zElHzw8cqVoVMna7nvPmvsOHVzXnzxRU6ePElGY7xm9fYeaIJSSrmB6bumc0fRO2hSqQn3lb/PafVevAjffGMte/ZcX1+xIjzxBDz6qNX9O7e/d+QOVqxYwbRp09JMHQLW6PFxcXGICPXrpxkTNl2aoJRSLrXkwBKe/v5pHqn2CE0qNbnl+oyxnitNngwLF0LiTBbFilkJ6YknrJ53+jzJec6ePUuPHj3SJCc/Pz+WLl3K+++/z5YtW7Db7VnuwQeaoJRSLrTuyDp6LO7BveXuZWanmbdUV3i41VKaPNka0QGsltFDD0H//tCuHWRh8AJ1g+x2O127dk3Te8/f359Ro0bRunVrWrVqxaeffsrw4cOpU6dOluvWBKWUcokdp3fwyLxHCCoexPLHlxPgc3NvtF644MN//wtTpsCVhKlPb78dnn7aSkx33OHEoFUaH374ITt27CAuLi5pnZeXF3fffTcjRowArJdzhwwZwtNPP03BggXTqyoNTVBKKZeYvms6xfyK3fT4evv2wUcfwaxZDZLGu2vcGJ57Djp2tMa7U9lrx44djBkzJs2tPX9/fxYtWoRHqvuoN5KcQBOUUspFxj00jpGNR1KmUObTLiS3bRu8/jr8+KP12cND6NoVXnrJ6oWnckZERAQPP/yww+dOc+bMoXTp0rd8DH1MqJTKMZejLtN1QVeOhR3DQzxuKDnt3Gm9o3TffVZy8vOD//s/mDFjKwsXanLKaf379+fixZSToPv5+dGrVy/atWvnlGNoglJK5YjIuEg6zO3AskPL+Pvy31neb+9e6NwZ6tWzhiHy94f//tcaG++zz6BcuehsjFo5MnfuXL7//nuio6+fexGhfPnyjBs3zmnH0Vt8SqlsF2eL49GFj7LpxCbmd51PszuaZbrP8eMwcqT1Yi1AgQIweLCVnG6/PZsDVuk6evQoAwYMSNNrz8/Pj++//z5L8zxllSYopVS2shs7/Zb1Y/lfy5nUbhLd7uqWYfmICHj3XasDRHS01dlh4EArWWVhlnCVjeLj43nkkUccdin/5JNPqF69ulOPpwlKKZWtImIj2HduH2+EvMEz9Z9Jt5zdbr3H9L//wZkz1rru3a1kValSzsSqMvbKK69w+PDhFAPB+vj40LRp0yxPt3EjNEEppbKNMYbCvoXZ0GcDBbwKpFtu82arw8POndbne++FTz6xRnxQ7mH9+vVMmDAhTa+9IkWKMGvWLCQbxovSThJKqWwxbec02s9tT2RcJH7efg6/wMLCrOdKjRpZyal8eWtA182bNTm5k0uXLtGlS5d0hzIqWrRothxXE5RSyum+Pfgt/Zf1x2a34eWR9kaNMdY4eTVqwKRJ1gjiI0fCn39Cz546Tp47McbQo0cPwsPDU6z39/dn2LBhNGrUKNuOrbf4lFJOtf7oeh5b9Bj3lL2HxY8uxscz5ZAOx45Zt/OWL7c+N2xoDVNUu7YLglWZmjhxIhs2bCA2NjZpnaenJ9WrV+f111/P1mM79fcUESkuIktF5JqIHBORx9MpJyIyVkT+FZErIhIqInc5MxalVM7bdWYXD897mDuL3ZlmfD1j4Msv4a67rORUpIjVetqwQZOTu9q/fz/Dhw932KV86dKleGXznPfObkh/DsQCpYCewKR0Ek83oC/wAFAc2Azc2lDGSimXsxs7VUtUZdUTqyjhXyJp/Zkz0KEDDBgA165B165w4AA884zeznNnAwcOdDjO3tdff02FChWy/fhOuzREJADoAowyxkQYYzYA3wNPOih+B7DBGPOPMcYGzAJqOisWpVTOioyzfsOuV6Ye2/ptI7BIYNK2xYuhVi2r1VS0qPXi7cKF+k5TbvDJJ59QpkyZpJdvCxQoQOfOnenatWuOHF8ympr3hioSuRvYZIzxS7buJaCJMaZDqrIVgaXAY8AR4C2gqjGmo4N6BwADAEqVKhU8b948p8TrTBERETc8Sm9+pecqc0OHDsVmszFhwgRXh5IlEfERDNk1hAdLPkivSr2ur4/w4tNPg/jpJ2vQ0Pr1LzFixEFuuy02vapu7vh6TWXZzZyryMhI3nvvPTZv3kyxYsX45ptvKFAg/VcGbkbTpk1/N8aknWrXGOOUBet23ZlU6/oDoQ7K+gDjAQPEYyWpOzI7RnBwsHFH69atc3UIuYaeq8w1adLE1KlTx9VhZElkbKR5YNoDxvsNb7P68Oqk9du2GVOpkjFgjJ+fMZ99Zozdnj0x6DWVdbdyrhYvXmwOHjzovGCSAbYbB9/5znzCFQEUTrWuMBDuoOxo4B4gEDgDPAGsFZG7jDGRDsorpdxMvD2e7ou6s+H4BuZ1nUfLyi0xxhrA9cUXIS4OgoNh9myoVs3V0apb1blz5xw/pjMfTx4CvESkSrJ1dYD9DsrWAeYbY04aY+KNMdOBYuhzKKVyjYHLBrLs0DI+a/sZj971KFeuQLdu8PzzVnJ69lnYuFGTk7p5TmtBGWOuicgS4A0R6QfUBR4BHL0P/hvQTUTmAeexevx5A4edFY9SKnu1uLMFlYpWYvA9g9mxAx59FP7+GwoVgqlTrWSl1K1wdif2wcA04BxwERhkjNkvIhWAP4CaxpjjwHvA7cAuIAArMXUxxoQ5OR6llJOduHKCwCKB9KjdA4Cvv7a6i8fGQt26Vg+9oCDXxqhg5syZlChRgrZt27o6lJvm1DcQjDGXjDEdjTEBxpgKxpg5CeuPG2MKJiQnjDHRxpj/M8aUMcYUNsbUM8asdGYsSinn+3rn1wRNCGLj8Y3Ex8OQIdC3r5WcBg60xtDT5HTd0aNHCQwMpE2bNpw5c8Yp01GMGTOGWrVqZVquQYMGDBw4kJ2JI/DmQvqKnFIqS77/83v6L+tPk4pNuLPAPTz0EHz6KXh7WyNETJ5sTSqYH/Tu3RsRSVpKlixJ+/btOXjwYIpya9asoW/fvrRp04aGDRvSt2/fHIuxSpUqLFy4kKeeeoqwsLAcO64z6Vh8SqlM/XLsF7ov6k69MvV4q/a3NG7owz//WDPbLllijUae37Ro0YKZM60BcE6dOsXw4cPp1KkTBw4cSCrTr1+/pL8///zzOR5jgwYN2Lt3b44f11m0BaWUytDRsKM8PPdhKhWtxPNF19DsAX/++Qfq1YPt2/NncgLw9fWldOnSlC5dmnr16jFs2DAOHjyYYmigl19+mWrVquHn50elSpUYMWIE0dHRKeqZMmUKQUFB+Pj4EBQUxJdffpml48+bN4/KlStTqFAhOnbsyIULF5K22e123nzzTQIDA/H19aV27dp89913KfbPSmyupglKKZWhikUqMqLRCB69uomnuhcmIgJ69IBff4XAwMz3zw/Cw8OZP38+tWvXxs8vaTAdAgICmDZtGgcOHGDixInMmzePt956K2n70qVLefbZZxk6dCj79u1jyJAhDB48mGXLlmV4vKNHjzJ//nyWLl3K6tWr2blzJ6+88krS9vHjx/PBBx/w3nvvsXfvXjp16kTnzp05fPh6R+nMYnMLjt7edddFR5LI/fRcZc5dRpI4HX7aHL542MTHGzNkiDUqBBgzdmz2jQpxM1xxTfXq1ct4enqagIAAExAQYAATGBho9u7dm+F+kyZNMpUrV076fP/995s+ffqkqbtRo0bp1jF69Gjj6+trwsLCktaNHTs2Rb1ly5Y1r7/+eor9mjRpYlq0aJHl2HISOTCShFIqj7gSfYWHZj3E1Yg46mzay7dLPfD2trqU9+zp6ujcw4MPPsgXX3wBWDPOTpw4kVatWrF161YCE5qWixYtYty4cRw+fJiIiAhsNhs2my2pjgMHDqTpONG4cWO+//77DI9dsWJFihQpkvS5bNmynDt3DoCrV69y6tSpNBMJNm7cmAULFiR9ziw2d6C3+JRSKUTHR/PwvIfZf/QcBeZs4NulHhQpAqtWaXJKzt/fn6CgIIKCgrj33nuZOnUqV69eTUpaW7Zs4bHHHqN169YsW7aMnTt3MnbsWOLi4lLUIyJp6na0Ljlvb+805e12e5bryGpsrqYtKKVUknh7PI8teoxfdpyi1Hd/cuB4ISpUgB9/tCYaVOkTETw8PJIm99u4cSPlypVj1KhRSWWOHTuWYp8aNWqwYcOGFK2oDRs2ULPmzY/6VrhwYcqWLcuGDRto1qxZinorVaqU5djcgSYopVSSDzZ+wHfrj1BowS7OXgrg7rvhhx+gbFlXR+Z+YmJiOHPmDACXL1/ms88+IyIigg4drNmFqlatyr///svs2bNp2LAhq1atYu7cuSnqGD58ON26dSM4OJhWrVqxcuVKZs+ezZIlS24ptuHDh/Paa69RpUoVgoODmTVrFr/++itTpkzJcmzuQBOUUipJfdsQ/Ge/SHi4D82bw9Kl1th6Kq01a9ZQJmHWxUKFClG9enUWLlxISEgIAB06dGD48OEMHTqUqKgoWrVqxRtvvMHgwYOT6ujYsSMTJkzgww8/ZOjQoVSsWJGJEycmJbmb9fzzzxMeHs6IESM4e/Ys1apVY/HixRQtWjTLsbkDp01YmBPq169vtm/f7uow0ggNDU26KFXG9FxlLiQkhLCwMHbt2pVjx1z0xyI8/2lLz0f9iYqCzp2tmW8TJlJ1a3pNZZ27nisRcThhobaglMrnZuyeQa+3luGxtCP2eOjd2xq6yEu/HZSLaS8+pfKxHw79QO/XNsCiedjjvRg2zJoqQ5OTcgeaoJTKpzYc30CnF9Zhvv8CjCdvvgkffQQe+q2g3IT+nqRUPmSz2+gydBPxyz8CrFHJn3vOxUEplYomKKXyoQ8/8OTc0hEATJkCAwa4OCClHNAEpVQ+cjbiLE8OO8hPXzVBBL76yppwUCl3pAlKqXwiLOoKdXss5cwPzyBi+PproVcvV0elVPo0QSmVD0TFRVO3+3ec+eEZPDwMM2aIjqun3J4mKKXyuHh7PME9l3Js2VN4eNqZM9uD7t1dHZVSmdMOpUrlcS+8doYDC3sgHnZmz9LkpHIPTVBK5WHjx8OEt8ojYpj+tQePPebqiJTKOk1QSuVRj45Yy9Ch1t+nTBGeesql4Sh1w5yaoESkuIgsFZFrInJMRB7PoOydIvKDiISLyAURed+ZsSiVn/V/fRMLP7DmAvr0U0P//i4OSKmb4OwW1OdALFAK6AlMEpE005yJiA/wE7AWKA2UB2Y5ORal8qXhH+3kq9cbAPDu+/E891zGs7Mq5a6clqBEJADoAowyxkQYYzYA3wNPOijeGzhljPnYGHPNGBNtjNnjrFiUyq/GTjrIhyNqg/Hg1THR/He4dtRVuZczr96qgM0YcyjZut1AEwdlGwBHRWQFcA+wD3jOGLM3dUERGQAMAChVqhShoaFODNk5IiIi3DIud6TnKnNhYWHYbLYbPk+bNxfntVE1we5Ft8cP0rzJGfLDqdZrKuty3bkyxjhlAR4AzqRa1x8IdVB2NRAHtAF8gOHAP4BPRscIDg427mjdunWuDiHX0HOVuSZNmpg6derc0D7r19tNgQLGgDHPD4sxdnv2xOaO9JrKOnc9V8B24+A735nPoCKAwqnWFQbCHZSNAjYYY1YYY2KBD4ESQA0nxqNUvrB28yWaPXSN6Ghr0NdxH/kg+thJ5QHOTFCHAC8RqZJsXR1gv4Oye4DcM9e8Um5q5/4IWrc22KIK0rTtBSZORJOTyjOclqCMMdeAJcAbIhIgIo2AR4CZDorPAhqISAsR8QSGAheAA86KR6m87u9j0dzfJIL48BLc3eg8K5aUxNPT1VEp5TzO7mY+GPADzgFzgUHGmP0iUkFEIkSkAoAx5k/gCWAycBkrkT2ccLtPKZWJ8xds1Gt8juiLpbmz1gV+WXkbvr6ujkop53JqH1RjzCWgo4P1x4GCqdYtwWpxKaVuQEQEtGsnXD1ZgVJ3XGRbaEkKFsx8P6VyGx3qSKlcJCYGOnay89s2DypWNPz+awlKlHB1VEplD01QSuUSNhs0aHuIn9d4UPI2Oz/9JJQr5+qolMo+mqCUygWMgVaP/cWutVXx8rvGjyvsVKmS+X5K5WaaoJTKBXoMPszaRVXw8I5hxXIv7gnWIYxU3qcJSik3N+S1o8yfHAQe8cyZZ6NFU+2up/IHTVBKubHp0+HTNysBMGHyNbp39ndpPErlJE1QSrmpGfOv0K+fNeDKuHHwbP8irg1IqRymN7KVckNXou+m9xMFMDZh1CgYMsTVESmV87QFpZSbCbtWmaNHxmPifen05Clef93VESnlGpqglHIje/fHsmffexBbmAfb/cui6WV18FeVb2mCUspNnDgBDULCMdElKXDbOn5aUg4P/QlV+Zhe/kq5gQsXoFUriLxQggIlfyeo1Eh8fFwdlVKupQlKKRcLD4cmLa5x8CDUqgX1qo7G0zM6afuVK1f44osv+Oeff1wYpVI5TxOUUi4UEwP1m53gj90BlC4fyerV4O0dkaLM1atXGThwIDVq1KBixYqMGDGCrVu3YrfbXRS1UjlDE5RSLhIfD43bnuDQ9kB8i14idK0PZcqkLRcYGEjdunWJjY3l+PHjfPLJJ7Rs2ZLixYvz1FNPsXz5cqKjo9PuqFQupwlKKRew26Ft93/ZvjYQL/9wfv05gGpV0n8tcdCgQQQEBAAQHx9PeHg4V65cYebMmTz++OMULVqUli1b8s0333DhwoWc+mcola00QSmVw4yBZ56LsHrp+USx/AfhnnoZj6/XrVs34uPjHW67evUqMTExrFmzhmeffZZy5cpRp04d1q9fnx3hK5VjNEEplcPGjoUvJxbE08vGrPlRtGqa+XS4xYoVIyQkJNNyERERxMbGcuDAATy0j7rK5fQKVioHvfHBJV57DTw8YN5cT3p0LJ7lfZ955hkKFSqUaTk/Pz8WL17MAw88cCuhKuVymqCUyiGTpoYzeoSVkD6bGEfXrje2f5s2bTLtuefr68uUKVPo0KHDzYaplNvQBKVUDliwJIrBA/wAGPS/fxg00PuG6/D19aVr164Z3rrz8PDg66+/Jioq6qZjVcpdaIJSKput/jmWHo95gt2LrgMOMfGtO2+6rv79++Pvn/6cUFFRUWzevJmmTZsSERGRbjmlcgOnJigRKS4iS0XkmogcE5HHs7DPWhExIqJTf6g8Z/t2ePhhgz3Oh6ZdD7JgctVbqu/+++9Pk6C8vVO2xqKjo9m1axeNGjUiLCzslo6nlCs5uwX1ORALlAJ6ApNE5K70CotIT3ROKpVHHTgADz0EMZG+hLQ/zU/zqt/yyOQiQt++fZOSkr+/P82aNcPPzy9FuZiYGA4ePMh9993H+fPnb+2gSrmI0xKUiAQAXYBRxpgIY8wG4HvgyXTKFwFGAyOcFYNS7uLwYWj44DUuXoS2bWH1kjJ4ejqn7j59+uDl5YW/vz+9evVixYoVTJ06NU2Sio2N5ciRI9SvX59Tp0455+BK5SBntqCqAjZjzKFk63YD6bWg3gYmAWecGINSLnfsGNz7QDhXLgQQ+J+/WbgQvG+8T0S6qlatStWqVencuTOff/45IkKPHj2YO3dumiQVFxfHv//+S/369Tl69KjzglAqB4gxxjkViTwALDTGlE62rj/Q0xgTkqpsfeAroD5QHjgCeBtj0rwqLyIDgAEApUqVCp43b55T4nWmiIgIChbM/GVLlffP1fnzvgx4thph54pT6I49zPnsEgXT79Pg0NChQ7HZbEyYMCHdMrGxsXh7eyOp7hlu376dV199lZiYmBTrRYRChQoxYcIEKlSocGMBubm8fk05k7ueq6ZNm/5ujKmfZoMxxikLcDcQmWrdi8CyVOs8gG1Ak4TPlQADeGV2jODgYOOO1q1b5+oQco28fK5OnTKmXKUIA8YUvOMPc/p85E3V06RJE1OnTp2bjmPDhg0mICDAJPxcJS0iYooWLWp2795903W7o7x8TTmbu54rYLtx8J3vzFt8hwAvEamSbF0dYH+qcoWxWk7zReQM8FvC+pMJrTClcp3z56FFC8O/RwMoUP4gezeWpXRJv8x3zAaNGjVi/fr1FC5cOMV6YwxhYWE0btyY3377LZ29lXIfTktQxphrwBLgDREJEJFGwCPAzFRFrwBlgboJS9uE9cHAVmfFo1ROuXQJWrSAP/4Qqte0sf3XYlQqU8SlMQUHB7Np0yaKFSuW5jZgeHg4TZs25ZdffnFRdEpljbO7mQ8G/IBzwFxgkDFmv4hUEJEIEamQ0KI7k7gAiX1gzxpjYp0cj1LZ6soVCGkey549ULWandC1ntxVqZSrwwLgrrvuYtu2bZQsWTLN6BPXrl2jTZs2rFy50kXRKZU5pyYoY8wlY0xHY0yAMaaCMWZOwvrjxpiCxpjjDvY5aowR46CDhMqamTNn8uOPP7o6jHzn6lVo2TqOvbt8kOJHmLLgb0q5R25KEhQUxPbt2yldujReXilfOYyMjKRz584sXbrURdEplTEd6shJjh49SmBgIG3atOHMmTNUr179luscM2YMtWrVyrRcgwYNGDhwIDt37rzlY6qsuXIFWrSy8dtWbyhynLnfnSXkP1Uy39EFKlSowPbt2ylfvnyaUSeioqLo2bMnM2emvhOvlOtpgspE7969EZGkpWTJkrRv356DBw+mKLdmzRr69u1LmzZtaNiwIX379s2xGKtUqcLChQt56qmndGibHHD5MrRoYee3rZ5Q9ChTFv1J98YNXB1WhsqUKcP27dupXLkyPj4+KbZFRUUxcOBAJk+e7KLolHJMhxnKghYtWiT9hnnq1CmGDx9Op06dOHDgQFKZfv36Jf39+eefz/EYGzRowN69e3P8uPnNpUvQsiXs2OGBR/GjvDNjKwNadHd1WFlSokQJtmzZQtOmTfnjjz9SvCsVFRXFCy+8QEREBC+99JILo1TqOm1BZYGvry+lS5emdOnS1KtXj2HDhnHw4MEUUxq8/PLLVKtWDT8/PypVqsSIESOIjo5OUc+UKVMICgrCx8eHoKAgvvzyyywdf968eVSuXJlChQrRsWNHLly4kLTNbrfz5ptvEhgYiK+vL7Vr1+a7775LsX9WYlOZu3ABmjeHHTugcmXYtaUoI9rljuSUqEiRIvz6668EBwenGXUiKiqK0aNHM3r06MR3FpVyKU1QNyg8PJz58+dTu3btFD/gAQEBTJs2jQMHDjBx4kTmzZvHW2+9lbR96dKlPPvsswwdOpR9+/YxZMgQBg8ezLJlyzI83tGjR5k/fz5Lly5l9erV7Ny5k1deeSVp+/jx4/nggw9477332Lt3L506daJz587s2rUry7GpzJ07B82awa5dUCLwAqGhhtpViro6rJsSEBDAzz//TKNGjdIkqcjISD788EOGDRumSUq5nqO3d911ccVIEr169TKenp4mICAg6e38wMBAs3fv3qQyjt7OnjRpkqlcuXLS5/vvv9/06dMnTd2NGjVK99ijR482vr6+JiwsLGnd2LFjU9RbtmxZ8/rrr6fYr0mTJqZnz57p1ps6tpzkrm+yZ+TkSWNq1DAGjKHkH6br1OeM3W7PtuPd6kgSWRUbG2seeeQR4+/vn2bUCX9/f9O3b19js9myPY5blRuvKVdx13NFDowkkWc9+OCD7Nq1i127drF161aaNWtGq1atOHHiRFKZRYsW0bhxY0qXLk3BggUZNmwYx49f71V/4MABGjVqlKLexo0b88cff2R47IoVK1KkyPWXPsuWLcu5c+cAuHr1KqdOncq03sxiU+k7fBgaNbKmzuC2fbR8823m9PoozcuvuZG3tzeLFy+mY8eOaeaYioyMZN68eQwaNMhF0Smlt/iyxN/fn6CgIIKCgrj33nuZOnUqV69e5YsvvgDgjz/+4LHHHqN169YsW7aMnTt3MnbsWOLi4lLU4+hLLbMvutTdgkUEu92e5Xq3bNmSpdhUWrt3Q+PG1ujkUn4r940awbdPT8Hb04lDk7uYp6cnM2fO5IknnnA4U2/NmjVdEJVSFu3FdxNEBA8PDyIjIwHYt28f5cqVY9SoUUlljh07lmKfGjVqsGHDhhTdzzds2HBLXwCFCxembNmybNiwgWbNmjmsd+PGjZnGptLatAnatYOwMKh532m8erzAin4/4O99g0OT5wIeHh5MnjyZwoULM3HiRCIjI/H392f48OEMGTLE1eGpfEwTVBbExMRw5ow1bdXly5f57LPPiIiIoEOHDgCUL1+ef//9l9mzZ9OwYUNWrVrF3LlzU9QxfPhwunXrRnBwMK1atWLlypXMnj2bJUuW3FJsw4cP57XXXqNKlSoEBwcza9Ysfv31V37//XfAmjsos9hUSqtWQefOEBkJnTrB3Lll8PRej5dH3v1xERE++OADChcuzGuvvcbAgQMZPXq0q8NS+Z2jB1PuuriqkwTJHh4XKlTI3HPPPWbRokVJZdatW2defvllU7JkSRMQEGA6depkJk6caKzTe11i5wQvLy9TuXJl88UXX2R47NGjR5u77rorxbqvv/7aBAQEJH222WzmjTfeMOXLlzfe3t6mVq1aZunSpSn2yUpsOcVdH9ImmjfPGG9vq0NE4fsWm+/2L8/xGHKqk0R69u7dm62dQJzN3a8pd+Ku54p0Okk4bcLCnFC/fn2zfft2V4eRRmhoKCEhIa4OI1dw13NlDHz4IYwYYX0u3uwbYps9zy99Q7m7zN05GktISAhhYWEpXhVQ6XPXa8odueu5EhGHExbm3XsWSmWRzQbPPw8TJ1qfK3T7lNO1h7O656ocT05Kqes0Qal87do16NEDli0DX19DjQFvs6fkayzquoiQSiGuDk+pfE27mat86+xZaNrUSk7Fi8PqnwyNHzrDlPZT6FSjk6vDU6mEhYXxxhtvcOnSJVeHonKItqBUvrR/P3ToAEeOwB13GOYuvcJ9dYryIBNcHZpKx6BBg6hYsSLFixd3et12u52PP/6YkJAQ6tdP8yhEuYi2oFS+89130KCBlZzq14fuH4+n69ranAo/5erQVDJxcXEEBwdTtmxZ1q9fT8OGDXn77bez5Vivvvoqa9eupW7dulkq37t3b9q3b58tsajrNEGpfMMYGDsWOnaEiAjr2VOvcVN5d/cwQiqFULpgaVeHmCsknx/N0dK7d2+nHOf111+nQ4cOLFmyhJEjRzJo0KA0U9ffTOyLFi1Kse7bb79l3bp1LFy4MM2sw8q18v3/xuTJk1m6dCmTJk3izjvvdHU4KptcuwZ9+sDChSAC77wDldsvpvviAbSt0pZpD0/DQ/T3taw4ffp00t9/+OEH+vfvn2Jd6hHSb9aQIUMoWbIkIsLy5cuzbfzDjh070rFjx2ypW92afP0Tefr0aV566SXWrFlDrVq1GDlyZIo5nlTecOyYNeDrwoVQuLDVKaJxj430XPo4Dco3YGG3hXlqfL3sljg3WunSpSlatGiKdQcPHqRo0aIp5iw7evQoIsL27dsxxhAUFMSHH36Yos6//voLEWHHjh0AXLlyhVdeeYVSpUpRqFAhHn74YX777bek8tOnT6dgwYL8/PPP9OnTh4CAAJo2bcqRI0fSjbtSpUoAdOvWDRFJ+pxYV6IxY8ZQq1atDOdhSzR+/HjKlStHsWLF6NOnT9LwZ8o58nWCevbZZ4mNjcVutxMVFcX48eOpVauWq8NSTrR8OdSrZw38WqUKbNlijbFX6/Za9Knbhx965M3x9dyViPD0008zbdq0FOunTZtG3bp1qVevHsYY2rVrx5kzZ/jxxx/ZuXMnzZs3p2XLlinGkYyJieGdd95hxIgRbN68mbCwMJ555pl0j52Y4L788ktOnz6dIuGlltk8bAC//vor+/btY82aNUllx48ffzOnRaUj3yaoX3/9lRUrVqQY1dtms6WZukLlTnFx8N//Qvv21jTtbdvC1q3gV+YokXGRFClQhMntJ1PMr5irQ813+vTpw19//cWWLVsA6+duxowZPP300wCsW7eOPXv2sHDhQurXr09QUBBjxoyhatWqzJkzJ6me+Ph4Pv/8c2rUqMF//vMfXnrpJdatW5dmtP9Et912GwBFixaldOnSSZ8diY+PZ/r06fznP/+hYcOGDBgwgJ9//jlFmcKFCzNp0iRq1KhBq1at6NatW5oy6tbkywQVHx9P796909zO8/Hx4ZNPPnFRVMpZTp603m96/33w9IR337Vu60V5nSJkegg9l/R0dYj5WunSpWnfvn1SK2rlypVcvHiRnj2t/5fff/+d8PBwChQokKLzxc6dO/n777+T6vH19aVatWpJn8uWLUtcXBxhYWG3HGNG87AlqlmzZopOFY7KqFvj1AQlIsVFZKmIXBORYyLyeDrleonI7yJyVUROisj7IpJjHTYmTJiQNDp5ooCAAN5//31KlCiRU2GobLBiBdStCxs3QrlyEBpqtaSuxFym9azWXIq6xKsPvOrqMPOsxF52ycf4dDT3WL9+/Zg/fz6RkZFMmzaNzp07U6yY1Zq12+2ULVvW4YDRX331VVIdqXvcJXaiSK8FdSOyMg9bVsqoW+PsFtTnQCxQCugJTBKRuxyU8weGAiWB+4DmwEtOjsWhs2fPMmrUqDQPMwMDAxkwYEBOhKCyQXQ0vPiidSvv4kVo3Rp27rQmHIyMi6T93PYcuniIbx/7luCywa4ON89KvG2WvFefo0FvH3roIQoXLszkyZNZtmxZinnS6tWrx+nTpzlw4IDT4/P29sZmszm9XpU9nJagRCQA6AKMMsZEGGM2AN8DT6Yua4yZZIz51RgTa4z5F5gN5MjDn+eff57Y2NgU6/z8/Pjmm2/w9PTMiRCUk+3YAcHB8PHH1i29t96CH3+ExEcMg5YPYsvJLczpPIdmdzTLuDJ1S4KCgggMDGTMmDEcOnSI1atXM3bs2DTlPD096du3LyNHjqRcuXI0b948aVuLFi1o1KgRXbp0Yfny5Rw5coTNmzfzyiuvEBoaekvxVapUiZ9//pkzZ85w+fLlW6pLZT9n3larCtiMMYeSrdsNNMnCvg8C+x1tEJEBwACAUqVK3dIFunfvXr777rsUtxy8vLxo1KgRkZGRN113RETELf/g5BfOPFc2mzBnTgW++aYiNpsHgYGRjBx5gBo1wvnll+vlWvi0oHy18pQ4V4LQc845dnYKCwvDZrPlimtq/37rxzZ5rCNGjGDcuHHUrl2boKAgnnjiCfbs2cPvv/9OREREUrkaNWoQGxtLSEgI69evT1HvyJEjmTp1Kr179yYsLIxixYpRq1YtatasSWhoKAcPHkw6R4nXVGJLbePGjSmeHyXXu3dvJk6cyLRp0yhZsiTz5s1LURdYPfiuXbuW4t+UusyZM2e4cuVKijKO9nM3ue67ytF93ptZgAeAM6nW9QdCM9mvD3ASKJnZMW5lwsK4uDgTFBSUYvJBwBQsWNCcP3/+pus1xn0nAXNHzjpXhw4Z06CBNbEgGPP888Zcu3Z9u91uN8sPLc9VE+8lcvWEhTlly5YtxtPT0xw7duyW6tGfv6xz13NFOhMWOvMZVARQONW6wkB4ejuISEfgXaCNMSbtW3BO9Pnnn3PqVMqx1vz9/XnnnXcoWbJkdh5aOVFcHLz3HtSpY73TVL48/PQTjB8P/sleZ3pv43u0m9OOhX8sdF2wyqGYmBgOHz7Mq6++SqdOnahQoYKrQ1JuypkJ6hDgJSJVkq2rQ/q37h4CvgQ6GGP2OjGONM6dO8crr7ySpmNE+fLlGTRoUHYeWjnRli3Ws6aXX4aoKHjySdi7F1q0SFnuqx1fMfLnkfSs3ZOuNbu6JliVrrlz51KtWjUuXrzIxx9/7OpwlBtzWoIyxlwDlgBviEiAiDQCHgFmpi4rIs2wOkZ0McZsc1YM6RkyZEiarq5+fn5Mnz5dO0bkAleuwP/9H9x/v5WQ7rwTVq2CGTMgYaSdJEsOLGHgDwNpE9SGrx/5WsfXc0O9e/fGZrOxY8cOAgMDXR2OcmPO/ukdDPgB54C5wCBjzH4RqSAiESKS2JYfBRQBfkxYHyEiK5wcCwCbN2/mu+++S9Fzz8fHh0ceeYSGDRtmxyGVkxgD8+dDzZrWdOyenjByJOzbB61apS1/MfIivb7txX3l7tPx9ZTKA5z6cqwx5hLQ0cH640DBZJ+bOvO46bHZbA5HjPD29tYxs9zctm0wbBhs2mR9btgQpkyB2rXT36eEfwm+e+w76pauS4BPQM4EqpTKNnn6/sekSZP4999/U6wLCAjgrbfe4vbbb3dRVCojJ07AE0/AffdZyalUKfjyS9iwIf3k9NfFv/j24LcANLujGcX9nD/jqlIq5+XZ+aAuXLjAyJEjuXbtWor1ZcqU4f/+7/9cFJVKz9Wr8OGH1hIVBb6+8MIL1i29QoXS3+9U+ClazWpFVFwULe5sQUGfgukXVkrlKnk2QQ0dOtThiBHTp0/XWTPdSEQEfPYZfPCBNeo4wKOPWl3JE6brSdflKGt8vQuRF1jXa50mJ6XymDz5Tb1t2zaWLFmSIkF5e3vToUMHnU7DTURGwqRJViI6f95a17ixNfJ4Vv6LIuMi6TC3A4cuHmL548upX7Z+9gaslMpxeS5B2Ww2evXq5XAqjU8//dRFUalEUVGejBtnJabEAeXvuw/efNN6nymrs3ov2L+ATSc2Mb/rfFrc2SLzHZRSuU6eS1BffvklJ06cSLEuICCAN998k1KlSrkoKnXmDEyYABMmNCA8YWyR4GB44w1o0ybriSlR77q9qVu6LnVL13V6rEop95CnevFdvHiRESNGpOkYUapUKZ577jkXRZW/HTgA/fpBxYrw9tsQHu7N/ffDt9/Cb79Z02NkNTkZYxj7y1h2n9kNoMlJqTwu17agjh49SmxsLFWrVk1a98ILLxATE5OinHaMyHlxcfDDDzB5Mqxeba0TgU6doFmzHTz7bL2bqvf9je8zat0owmPCqVO6jhMjVkq5o1zbgnrnnXeoUaMGzz33HOHh4Wzfvp2FCxem6RjRpk0bHnjgARdGmn8cPw6vvWa1ljp3tpJTgQLwzDNw8CAsWQK1al29qbqn7pjKyz+/zGO1HuOdFu84OXKllDvKtc2KU6dOYbfb+eqrr5g1axZFihRx2DHis88+c1GE+UN0tNVa+uYba5LAxBmvq1WzEtNTT0HxW3xv9tuD3zLghwG0qtyKbzp+o+PrKZVP5NoEde7cOQCio6OJjo4mPDzlrB7+/v6MGTOGMmXKuCK8PM1uh/XrYdYsWLTIeskWwNvbeodp4EBo0uTGOz44Yozh611fc0/Ze1j86GJ8PH1uvVKlVK6QaxPUxYsXU3y22WwpPt9+++0MHTo0ByPK2+x22LwZli61BnA9efL6tuBg6NnTWpw9gpSIsKDrAiLjIvVFXKXymVyboMLCwtLd5unpyeXLl5kzZw5PPvkk4oxf5fOh2FhYu9ZKSt99B2fPXt9WqZI1Zl7PnlC9uvOP/felvxm2ahjTHplGSf+S+Hr5Ov8gSim3lmsTVERERLrbbDYbV65cYfDgwYwbN46ZM2dy11135WB0udfx49ZcS6tWWTPVXk3Wp6FSJasnXpcu1txM2ZX3T4efpuXMllyNucrFyIuU9NcZj5XKj3JlgoqJiUkzAaEj165dY8+ePUyZMkVHkUjH1avWSOGrV1tJ6eDBlNtr17aSUqdO1jTr2d0YDYsO46HZD3Hu2jnW9lpLtZLVsveASim3lSsT1MWLF/H19U3Tay85Hx8fChQowOeff07Pnj1zMDr3dumSlZDWr4dffoEdO673vANr5PDmzaF1a2u5446ciy0qLoqH5z7MgfMHWP74cu4td2/OHVwp5XZybYLK6MVbf39/2rZty6RJkyhZMv/eHoqJgT17rBEbfvvNmgTwwAFrptpEXl5wzz3Xk1LDhlZvPFe4EHmBMxFnmNV5Fi0rt3RNEEopt5FrE5SHR9p3YQoUKECRIkWYMWMGrRzNCZ6HRUVZt+cSE9K2bbB7t9XRITlfX2tw1gcftLqCN2gABV3cOc4Yg8EQWCSQvYP2aocIpRSQixOUSd4MwBrS6Omnn+bdd98lICDvTvcdHw9//w379sHevdf/PHw45a26RNWrw733Wq2ke+6xniMVKJDzcWfkv2v+y7lr55j68FRNTkqpJLk2QSV2kvD396dcuXLMnTuX4OBgF0fmHDabNfX5X3+lXY4csca6S83T00pGtWpB/fpWMgoOhiJFcj7+G/HBxg/4YNMH/N89/6cjRCilUsiVCer8+fNER0fj5+fHK6+8wvDhw/F21YOTm3DtmpWAjh93vJw4kfbWXHKBgVbvulq1rv9Zvbr7tYwyM33XdEasGUH3u7rzaZtP9X01pVQKuTJBFShQgIYNGzJjxgwqV67s6nAA6/baxYvWvEdnz6b8M/nfT52yymWmdGmoUiXtUrky5IU7mMv+XEa/7/vR8s6WzOg0Q1tPSqk0cmWCevHFF3nxxRedXm98PEREQHi49X7Q1atw+bLVNTuj5ezZRkREOH4G5IiPD1SoYC2Bgdf/nnxdXkhCGfH39iekUghLui/R8fWUUg45NUGJSHFgKtAKuACMNMbMSafsMOC/gB+wGBhkjIlxVDa5iAirx5qjJTIy/W3XrlmJJ/ly9WrKzxm8VpUJ6/Zi8eJWy6dUqfT/LFPGGq/OQSfEfCEyPhKA5nc2p9kdzfS2nlIqXc5uQX0OxAKlgLrAchHZbYzZn7yQiLQGXgaaAaeApcDrCevS9fvv1ouk2UXEqj9xKVzYSjqZLQcObKRdu0Yue38ot/jn8j/0+q0X7xZ/l6frPa3JSSmVIUndXfumKxIJAC4DtYwxhxLWzQT+Nca8nKrsHOCoMeZ/CZ+bA7ONMaUzPkZ94+HxKx4esXh6xuDhkbjE4uERk2xdLJ6e0am2RePpGYWnZySenpF4eUUm/d36HIWHR9RNDeUTFhZG0aJFb3zHfCTWJ5add+8kziuOu3feTUBkHr+HeQt27dpFfHw89evXd3UouYL+/GWdu56r9evX/26MSXPBO7MFVRWwJSanBLuBJg7K3gV8l6pcKREpYYxJ0YVARAYAA8CaIbdmzYZOC9hms5Zbr8eW4ejq+Z3Ny8bhBw4T6xNLpV8qEXcljjDCXB2W24qPj8cYo9dUFunPX9bltnPlzARVELiSat0VwNFNudRlE/9eCEiRoIwxXwBfANSvX99s377dKcE6U2hoKCEhIa4Owy3F2+NpPqM58Sfi+bHHj/g299VzlYmQkBDCwsLYtWuXq0PJFfTnL+vc9Vyld7vfmY/qI4DCqdYVBsKzUDbx747KqlzMy8OLTtU7MaPjDFoHtXZ1OEqpXMSZLahDgJeIVDHG/JWwrg6w30HZ/QnbFiQrdzb17T2VexljOBp2lDuK3cHQBkNdHY5SKhdyWgvKGHMNWAK8ISIBItIIeASY6aD4DOBpEakpIsWAV4HpzopFud7In0dSZ3Id/r70t6tDUUrlUs5+G2cw1ntN54C5WO827ReRCiISISIVAIwxK4H3gXXAsYRltJNjUS7y0aaPeG/je/Ss3ZM7i93p6nCUUrmUU9+DMsZcAjo6WH8cq2NE8nUfAx878/jK9WbsnsFLP71Et5rd+KztZ/quk1LqpuXT8QxUdth0YhN9v+tLiztbMLPTTDw9PF0dklIqF9MEpZzmnrL38FqT11jy6BKd10kpdcs0Qalbtv/cfs5GnMXb05vXmrxGId9sHI9KKZVvaIJSt+TI5SO0mNmCxxY/5upQlFJ5jCYoddPORpyl1axWxMTH8Fmbz1wdjlIqj8mV80Ep17sac5U2s9twKvwUa55cw1233+XqkJRSeYwmKHVTXlz1InvP7WVZj2U0DHTeAL5KKZVIE5S6Ke+0eIeO1TvyUNBDrg5FKZVH6TMolWXGGKbumEqsLZaS/iVpV7Wdq0NSSuVhmqBUlr269lX6LevHnL1zXB2KUiof0ASlsmTclnG8veFtBgYPpFedXq4ORymVD2iCUpmatWcWw1YNo0uNLnze9nMdX08plSM0QakMXY25ytCVQ2l2RzNmd56t4+sppXKM9uJTGSrsW5jQ3qFULFJRx9dTSuUobUEph/ad28f4LeMBqHV7LR1fTymV47QFpdI4GnaU1rNaA/BknScp7lfcxREppfIjTVAqhXPXztFqZiui4qL4pc8vmpyUUi6jCUolSRxf7+TVk6x5ag21bq/l6pCUUvmYJiiVZN2Rdew7t4+l3Zdyf+D9rg5HKZXPaYJSSR6p/giHnztMYJFAV4eilFLaiy+/M8YwbOUwVh1eBaDJSSnlNjRB5XOvrXuNcVvHseH4BleHopRSKWiCysc+3fopY38dS7+7+/FG0zdcHY5SSqXglAQlIsVFZKmIXBORYyLyeAZle4nI7yJyVUROisj7IqLPwnLY7D2zGbJyCJ2qd2JS+0k6vp5Syu04qwX1ORALlAJ6ApNEJL05wP2BoUBJ4D6gOfCSk+JQWbTxxEZCKoUwp8scvDz09wOllPu55W8mEQkAugC1jDERwAYR+R54Eng5dXljzKRkH/8VkdlA01uNQ2WNMQYR4fO2nxMVH0UBrwKuDkkppRxyxq/OVQGbMeZQsnW7gSZZ3P9BYH96G0VkADAg4WOEiPx5U1Fmr5LABVcHkUvoucqakiKi5ylr9JrKOnc9VxUdrXRGgioIXEm17gqQ6eiiItIHqA/0S6+MMeYL4ItbCTC7ich2Y0x9V8eRG+i5yho9T1mn5yrrctu5yvQZlIiEiohJZ9kARACFU+1WGAjPpN6OwLtAG2OMO2Z0pZRSLpRpC8oYE5LR9oRnUF4iUsUY81fC6jpkfNvuIeBLoJ0xZm/Ww1VKKZVf3HIvPmPMNWAJ8IaIBIhII+ARYKaj8iLSDJgNdDHGbLvV47sJt74F6Wb0XGWNnqes03OVdbnqXIkx5tYrESkOTANaAheBl40xcxK2VQD+AGoaY46LyDrgASA6WRW/GmPa3HIgSiml8gynJCillFLK2XSoI6WUUm5JE5RSSim3pAnKyUSkiohEi8gsV8fijkTEV0SmJozZGC4iO0VEnz8muJFxLfMzvY5uTm77ftIE5XyfA7+5Ogg35gWcwBpppAgwClggIpVcGZQbuZFxLfMzvY5uTq76ftIE5UQi8hgQBvzs4lDcljHmmjFmjDHmqDHGboz5ATgCBLs6NldLNq7lKGNMhDFmA5A4rqVKRq+jG5cbv580QTmJiBQG3gBedHUsuYmIlMIazzHdF7vzkfTGtdQWVCb0OspYbv1+0gTlPG8CU40xJ1wdSG4hIt5YL21/Y4w56Op43MBNj2uZn+l1lCW58vtJE1QWZDYeoYjUBVoAn7g4VJfLwtiNieU8sEYbiQWedVnA7uWmxrXMz/Q6ylxu/n7SmeqyIAvjEQ4FKgHHE2amLQh4ikhNY0y97I7PnWR2rgDEOklTsToCtDXGxGV3XLnEIW5wXMv8TK+jLAshl34/6UgSTiAi/qT8zfclrAtikDHmvEuCcmMiMhmoC7RImORSJRCReYDBmoKmLvAjcL8xRpNUKnodZU1u/n7SFpQTGGMigcjEzyISAUS7+3++K4hIRWAgEAOcSfiNDmCgMWa2ywJzH4OxxrU8hzWu5SBNTmnpdZR1ufn7SVtQSiml3JJ2klBKKeWWNEEppZRyS5qglFJKuSVNUEoppdySJiillFJuSROUUkopt6QJSimllFvSBKWUUsot/T+E5l2+ZtFZygAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "z = np.linspace(-5, 5, 200)\n",
    "\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [1, 1], 'k--')\n",
    "plt.plot([0, 0], [-0.2, 1.2], 'k-')\n",
    "plt.plot([-5, 5], [-3/4, 7/4], 'g--')\n",
    "plt.plot(z, logit(z), \"b-\", linewidth=2)\n",
    "props = dict(facecolor='black', shrink=0.1)\n",
    "plt.annotate('Bão hòa', xytext=(3.5, 0.7), xy=(5, 1), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.annotate('Bão hòa', xytext=(-3.5, 0.3), xy=(-5, 0), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.annotate('Tuyến tính', xytext=(2, 0.2), xy=(0, 0.5), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.grid(True)\n",
    "plt.title(\"Hàm Kích hoạt Sigmoid\", fontsize=14)\n",
    "plt.axis([-5, 5, -0.2, 1.2])\n",
    "\n",
    "save_fig(\"sigmoid_saturation_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Khởi tạo Xaiver và He"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Constant',\n",
       " 'GlorotNormal',\n",
       " 'GlorotUniform',\n",
       " 'HeNormal',\n",
       " 'HeUniform',\n",
       " 'Identity',\n",
       " 'Initializer',\n",
       " 'LecunNormal',\n",
       " 'LecunUniform',\n",
       " 'Ones',\n",
       " 'Orthogonal',\n",
       " 'RandomNormal',\n",
       " 'RandomUniform',\n",
       " 'TruncatedNormal',\n",
       " 'VarianceScaling',\n",
       " 'Zeros',\n",
       " 'constant',\n",
       " 'deserialize',\n",
       " 'get',\n",
       " 'glorot_normal',\n",
       " 'glorot_uniform',\n",
       " 'he_normal',\n",
       " 'he_uniform',\n",
       " 'identity',\n",
       " 'lecun_normal',\n",
       " 'lecun_uniform',\n",
       " 'ones',\n",
       " 'orthogonal',\n",
       " 'random_normal',\n",
       " 'random_uniform',\n",
       " 'serialize',\n",
       " 'truncated_normal',\n",
       " 'variance_scaling',\n",
       " 'zeros']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[name for name in dir(keras.initializers) if not name.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7f8a00956880>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"relu\", kernel_initializer=\"he_normal\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7f8a0009fca0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "init = keras.initializers.VarianceScaling(scale=2., mode='fan_avg',\n",
    "                                          distribution='uniform')\n",
    "keras.layers.Dense(10, activation=\"relu\", kernel_initializer=init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Các Hàm Kích hoạt Không bão hòa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Leaky ReLU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def leaky_relu(z, alpha=0.01):\n",
    "    return np.maximum(alpha*z, z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure leaky_relu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAj40lEQVR4nO3deXxU9fX/8ddhJ6zyRVPbCtgWELGCkNrFLXWrAq6oLG6ULwKudaGidSk/961fseCGgrggrYgCrlXUoKACoaJVRFBcERHEACEJIeHz++NO6jDZZiYzuffOvJ+PxzyYuXPn3pObYd65937mHnPOISIiEjRN/C5ARESkJgooEREJJAWUiIgEkgJKREQCSQElIiKBpIASEZFAUkBJ3MzsYDM7z+86qpjZGDM70u86RCQ9FFCSiGXAmWZ2TjpXYmYTzOz9euYZDYwBFsexvAIzm5xgDSPMrDiR14SdmU03s2f9rkOkigIqy9X2oWRmeWbmzKxb1TTnXClwHHChmfVpxDJjazsQ+BNwrHNuq191pEK8QWhm+ZHfR+fGqCsVzKxbpOaq22Yze9vMjktwObVuo9r+mIlad16y9Yv/FFCSEOfcRufc/s65d32sYYlzrrdzbr1fNUhCjgH2BH4NLAFmm9l+/pYkYaCAkriYWVMzm2pmn5pZqZmtNrPLzaxJ1DzTzexZMxtvZt9E/mK+xcyaRP7S/TYyfXyC6+5iZivN7GEza1bT3oSZ/cbMXjWzbZH1vmJmP45aTBMzu8nMNkbquCO69jrWfYSZvR9Z7mtmtnfM82PM7GMzK4/8e07M85ea2XuR1681swfNrGPkuXzgIaBN1F7GhES2TdR6WpjZrWb2VWRdS83sD1HP1/v7q2GZfcxsnZndaGbTYve0I7/XL8zs0nrK+845941zbiVwFdAc+H3Ucn5iZv8ws+8jt+fMrHsy20EyiwJK4tUEWAucBvTC+6D5C/DHmPkOBfYG8oGxwOXA80BL4GBgAnCLmfWPZ6Vm1gtYFFnGCOdcRQ3z9AFeAz4GDgJ+AzwBNIua7XSgAvgdcAFwMTCkntW3BK4ERgK/BToC90Wt9yRgMjAR2A+4C7gn5hDWzsi6egPDgQOBSZHn3ow8V4K3h7EncEc9NdXmIeCwyDp+CTwMPBN1KDbe31/Vz3YI3ja9zTl3FfAAcIyZ7Rk121HAj4BH4ynQzJoDVQG+IzItJ7Keskj9vwXWAfMjz0k2c87plsU3YDreB3dxzK0EcEC3Ol57CzA/ZllfAk2jphUC78W87jNgXB3LnQC8j/dhvhG4Kub5/EhtnSOPZwBv17G8AuCtmGkvAw/W8ZoRkXX0jJp2OlAONIk8XgRMq2F7LqxjuccA26OWMQIojuP3tMvPHPPcz/GCsEvM9DnAPQn+/p4FBgFbgLNi5n8fuCLq8T+BJ+tYfrdIzSWR91Rl5PEaoFNknpHAasCiXtcU+A44rb5tVPVeqWPdeX7939Kt4bfovzAle70OjI6Zth/wdPQEMxsLjAK6Aq3xDtV8HvO6Fc65yqjH64GimHnWA3vUU9NPgFeA65xzt9cz7wGxtdbgvZjHX8dRw3bn3Ecxr2mOtye1CW9PZFrMaxYCx1c9MLPD8fbCegEd8D58W+DteXxdz/rj1Q8wYIWZRU9vCbwaVUs8v7/+eNtyuHNuVsxzDwDn4e0BdwJOAE6Ko77hwAdAD+BOYLRzblPU+vYGtsbUnoMXvJLFFFACUOKc+zh6QtV5kqjHQ/AOZY3DOzS1BTif6h9QO2Ieu1qm1Xd4eSPentZQM3vQOfd9HfNaHc/VVVd9NcQeTqzqTdOkhmnV5jOzrsBzeB/s1+LtFfQDZuKFVKo0iazzV1T/OUsjtcT7+/sU+BYYaWbznHPbo557FLjVzA7G+6NgI/BSHPV95ZxbDayOjMabZWb7Ouc2RmpfDgyt4XWbapgWawte8MfqGPl3cxzLkIDSOSiJ18HAYufcZOfcvyOBls6/cLfj7Yl8D7xsZrvVMe+/gcPTWEttPsTbLtEOBlZE7ufhBdElzrm3nHOrgB/HzF+Ot1fVEO/ghfSPnHMfx9zWRtUVz+9vE3BEpM6nzaxl1RORvZ6n8A7LjQSmx+wt18s5twBv+1wbmfRv4BfAxhpqjyegPgL2jDk3Bt4fAuV4gSshpYCSeK0C+pnZsWbW3cyuwTupnTbuh+9dbcYLqY61zHo7cICZTYmMPOtpZqPMrEs664us90wzOz+yTS7EO091W+T51Xj/xy42s73NbBjeoIhonwGtzOwoM+scx8CA/cysb/QNb3DIDGC6mZ1iZj8z73ts48zs5Mjr4v79RfZsjgB+CjwVHVJ4e4OnA33wBmYk42/AaDPbK1L3emCumR0W2U6HmtnfYkbyNYn9uc0bqv4vYCUw08wOivzsg4EbgImuhkE1Eh4KKInX/Xgj4x4HluKdhP5bulcaCalB1BFSzrnlwJHAPsDbeFeXGEr1w12prm0OcCFwCd5ewZ+A85xzz0Sefy8y7dLI86PwDrFFL+NNvJGBM4ENeKMe6/Ia3h5T9C0HbzTeQ3jhuBJvsMOh/HCOKaHfXySkDgf2wvveUlVIFQBfAQXOuU/qqbU2z+IF8zXOuZJInWuAWZHaHwZ2w9t7rtK6hp+7IBJAR0eWNwNvIMf1eCMqr06yPgkIc04t30UkPmbWGm+4+oXOuRl+1yOZTXtQImlmZgeYWXEjHHJMm8iXcvcE/h/ewIvYEX4iKadRfCLptwLoS+qGlfuhC96Ag6+APzrnyn2uR7KADvGJiEgg6RCfiIgEUtoO8XXu3Nl169YtXYtvkG3bttGmTRu/ywgtbb/kfPTRR1RWVrLvvvv6XUoo6X2XvJq23c6dsHIllJZCu3bQvTtYPF95T4Nly5ZtdM7tHjs9bQHVrVs3CgsL07X4BikoKCA/P9/vMkJL2y85+fn5FBUVBfb/RdDpfZe82G3nHAwbBu+8A7/4BSxZArvV9VX4NDOz2EtuATrEJyKSdW69Ff75T2jbFubO9Tec6qKAEhHJIs89B3/5i3d/xgwI8hFnBZSISJZYuRKGD/cO8V1/PRx/fP2v8ZMCSkQkCxQVwQknwJYtcMopcNVVfldUv4QCKnKRyTIzeyxdBYmISGpVVnp7TqtWwf77w0MP+TdiLxGJ7kHdjXehSRERCYmpU3/GCy/A//wPzJnjDY4Ig7gDysyG4nVGfSVt1YiISErNnAkzZ3ahaVOYNQv23tvviuIXV0CZWXvgOuCy9JYjIiKpsmwZjBzp3Z84EX7/e1/LSVi8X9S9HpjqnPvS6jhwaWajgdEAubm5FBQUNLjAdCguLg5sbWGg7ZecoqIiKisrte2SpPddYjZtas7Ysf0pK2vFUUd9Qe/eawjb5qs3oCIdO48EDqhvXufcFGAKQF5engvqt771jfSG0fZLTseOHSkqKtK2S5Led/ErL4cjjoANG+C3v4Vx4z7l97/P97mqxMWzB5WP133zi8jeU1ugqZnt65zrl77SREQkGRddBAsXwk9+ArNnw0cfhbNrRTznoKYAP8frZ9MXrz31c8Af0laViIgk5b774P77oWVLePpp2HNPvytKXr17UM65EqCk6rGZFQNlzrkN6SxMREQS8/rrcOGF3v0HHoBf/crfehoq4auZO+cmpKEOERFpgM8/964QUVEBl10GZ57pd0UNp0sdiYiEXEkJnHiiNyji6KPhllv8rig1FFAiIiHmnPddp+XLvd5O//gHNEtbp7/GpYASEQmxsPR2SoYCSkQkpMLU2ykZCigRkRAKW2+nZCigRERCJoy9nZKhgBIRCZGw9nZKhgJKRCRErrqKUPZ2SoYCSkQkJGbO9EbthbG3UzIUUCIiIRD23k7JUECJiATc+vXelSLKyuB//xfOP9/vihqHAkpEJMDKy72Rel995fV2uvvuzB0UEUsBJSISYNG9nZ56ymujkS0UUCIiAVXV26lVK2/E3o9+5HdFjUsBJSISQAsW7NrbKS/P33r8oIASEQmY6N5O48bBGWf4XZE/FFAiIgGybZs3Ym/jxszq7ZQMBZSISEDU1NupaVO/q/KPAkpEJCBuuQWeeALatcu83k7JUECJiATAs89619kzy8zeTsnIkMbAIiLh9eGHP/R2uuEGOO44vysKBu1BiYj4qKq309atcOqpP3TIFQWUiIhvKith2DBYvTrzezslQwElIuKTv/wFXnzR6+00dy60aeN3RcGigBIR8cHjj8Ntt3nDyJ98Erp187ui4FFAiYg0smXLvLYZAHfdBfn5vpYTWAooEZFGFN3badQoOO88vysKLgWUiEgjKS+HwYO93k6/+x1MnqxBEXVRQImINALn4IILYNEir7fT7NnZ1dspGQooEZFGcN99XtuMbO3tlAwFlIhImi1Y4HXGhezt7ZQMBZSISBqpt1PyFFAiImmi3k4No4ASEUkD9XZqOAWUiEgaqLdTwymgRERSTL2dUkP9oEREUki9nVJHe1AiIimi3k6ppYASEUkB9XZKvbgCysweM7N1ZrbFzFaZ2ah0FyYiEibq7ZR68e5B3Qx0c861B44HbjCz/ukrS0QkPNTbKT3iCijn3AfOue1VDyO3n6etKhGRkFBvp/SJexSfmd0DjABaA+8Az9cwz2hgNEBubi4FBQUpKTLViouLA1tbGGj7JaeoqIjKykptuyQF8X23aVNzxo7tT1lZKwYO/Jp9911FwEoEgrnt4mHOufhnNmsK/BbIB251zu2obd68vDxXWFjY4ALToaCggHz9mZM0bb/k5OfnU1RUxPLly/0uJZSC9r4rL4fDD/faZxx0ELz6KrRo4XdVNQvatotlZsucc9UuoZvQKD7nXKVzbiHwU+DcVBUnIhIm0b2dfvpTr7dTUMMpzJIdZt4MnYMSkSx1770/9HZ6+mnIzfW7osxUb0CZ2R5mNtTM2ppZUzP7AzAMeDX95YmIBEtBAfzpT979Bx9Ub6d0imeQhMM7nHcfXqB9DlzsnJubzsJERILms8+8K0RUVMCf/wynn+53RZmt3oByzm0ADmuEWkREAiu6t9Mxx8DNN/tdUebTpY5EROrhHPzxj/Duu9C9u/fFXPV2Sj8FlIhIPW6+GWbNUm+nxqaAEhGpwzPPwNVX/9DbqVcvvyvKHuoHJSJSiw8/9AZCOAc33qjeTo1Ne1AiIjX4/vtdeztdeaXfFWUfBZSISIzo3k59+qi3k18UUCIiMa68Ev71L+jcGebMUW8nvyigRESizJgBt98OzZqpt5PfFFAiIhGFhTAq0i/8rrvgMF2iwFcKKBER4Jtv4KSToKwMzjkHzlW/Bt8poEQk623fDoMHw1dfeb2dJk/WoIggUECJSFar6u305pvq7RQ0CigRyWr33uu1zVBvp+BRQIlI1lJvp2BTQIlIVlJvp+BTQIlI1lFvp3BQQIlIVlFvp/BQQIlIVlFvp/BQQIlI1lBvp3BRPygRyQrq7RQ+2oMSkYyn3k7hpIASkYwW3dupb1/1dgoTBZSIZDT1dgovBZSIZKzY3k5du/pdkSRCASUiGSm6t9Pf/67eTmGkgBKRjPPNN96VIsrKYPRoGDvW74okGQooEckoVb2d1q71ejtNmqRBEWGlgBKRjOEcnH++ejtlCgWUiGSMe+6BqVO93k5z5qi3U9gpoEQkI0T3dpo6Ffr397UcSQEFlIiE3mefwSmneF/KvfxyGD7c74okFRRQIhJq27Z5lzH67juvt9NNN/ldkaSKAkpEQss5GDEC3nvP6+00c6Z6O2USBZSIhNZNN3lXiGjfHubNg44d/a5IUkkBJSKhNG/err2d9tnH74ok1RRQIhI6K1bAGWd492+8EQYN8rceSQ8FlIiESnRvpyFD4Ior/K5I0kUBJSKhUVEBQ4fCxx97vZ2mTtVljDKZAkpEQuPKK+Gll9TbKVvUG1Bm1tLMpprZ52a21czeMbNjG6M4EZEqL7+cyx13qLdTNolnD6oZ8CVwGNABuAZ4wsy6pbEuEZH/KiyE22/vCai3UzZpVt8MzrltwISoSc+a2adAf+Cz9JQlIuKp6u20Y0cT9XbKMvUGVCwzywV6AB/U8NxoYDRAbm4uBQUFDa0vLYqLiwNbWxho+yWnqKiIyspKbbsElJcbl17al7VrO9Cr1yZOOeU/LFjg/C4rdML6fzahgDKz5sAM4GHn3MrY551zU4ApAHl5eS4/Pz8VNaZcQUEBQa0tDLT9ktOxY0eKioq07eLkHJxzDnzwgdfb6YYbPuSoo3RsLxlh/T8bd0CZWRPgUaAcuCBtFYmIUL2309atO/wuSRpZXMPMzcyAqUAuMNg5p3eKiKSNejsJxL8HdS/QCzjSOVeaxnpEJMupt5NUied7UF2BMUBf4BszK47cTk93cSKSXdTbSaLFM8z8c0AXExGRtFJvJ4mlSx2JSCCot5PEUkCJiO+iezs9/rh6O4lHASUivoru7XTTTTBwoL/1SHAooETEN7G9ncaP97siCRIFlIj4Irq30wEHwLRp6u0ku1JAiYgvqno77b67d6WInBy/K5KgUUCJSKN77DF26e3UpYvfFUkQKaBEpFEtXQqjRnn3J02CQw/1tx4JLgWUiDSadeu83k7bt8OYMertJHVTQIlIo9i+HQYPhq+/hoMP9jrjitRFASUiaeccnH8+vPUW7LWXd96pRQu/q5KgU0CJSNrdffeuvZ1yc/2uSMJAASUiafXaa3Dxxd79adOgXz9fy5EQUUCJSNp8+imceqrX22n8eBg2zO+KJEwUUCKSFsXF3oi9776DY4+FG2/0uyIJGwWUiKRcdG+nHj28K5Srt5MkSgElIil3440we7bX22nuXPV2kuQooEQkpebOhWuuUW8naTgFlIikzAcfqLeTpI4CSkRSYtMmr7dTcbF6O0lqKKBEpMGqejt98ol6O0nqKKBEpMGuuAJeflm9nSS1FFAi0iCPPgp/+5t6O0nqKaBEJGlLl8I553j31dtJUk0BJSJJUW8nSTcFlIgkTL2dpDEooEQkIertJI1FASUiCVFvJ2ksCigRiZt6O0ljUkCJSFzU20kamwJKROoV3dtpwAD1dpLGoYASkTpF93bq2VO9naTxKKBEpE6xvZ06dPC7IskWCigRqVV0b6eZM709KJHGooASkRpF93a6+Wbv3JNIY1JAiUg10b2dhg6Fyy/3uyLJRgooEdlFbG+nqVPV20n8oYASkV2MH6/eThIMCigR+a9HHoH/+z+vt9Ps2ertJP6KK6DM7AIzKzSz7WY2Pc01iYgPliyB0aO9+5MnwyGH+FuPSLM45/sauAH4A9A6feWIiB/WrYOTTvLaaIwd6/V3EvFbXAHlnHsKwMzygJ+mtSIRaVTbt8PJJ3u9nQ45BO66y++KRDzx7kHFxcxGA6MBcnNzKSgoSOXiU6a4uDiwtYWBtl9yioqKqKysDNS2cw5uu60nb7+9J3vsUcYllyzjzTd3+F1WjfS+S15Yt11KA8o5NwWYApCXl+fy8/NTufiUKSgoIKi1hYG2X3I6duxIUVFRoLbdpEnw4ovQujW88EIr+vU7yO+SaqX3XfLCuu00ik8kS736KlxyiXdfvZ0kiBRQIlkourfTFVd4X8wVCZq4DvGZWbPIvE2BpmbWCqhwzlWkszgRSb3iYu8yRps2edfXu+EGvysSqVm8e1BXA6XAFcAZkftXp6soEUmPnTu93k7/+Y96O0nwxTvMfAIwIa2ViEjaqbeThInOQYlkiblz4dpr1dtJwkMBJZIF1NtJwkgBJZLh1NtJwkoBJZLB1NtJwkwBJZLB1NtJwkwBJZKh1NtJwk4BJZKB1NtJMoECqpGYGU8++aTfZUgWUG8nyRQKqIgRI0YwaNAgv8sQaRD1dpJMooASyRDOwbnnwttvw157wZNPQosWflclkjwFVBxWrFjBwIEDadeuHXvssQfDhg3jm2+++e/zS5cu5eijj6Zz5860b9+egw8+mLfeeqvOZd5666107tyZxYsXp7t8yRKTJ8NDD3m9nebOhT328LsikYZRQNVj3bp1HHrooey3334sWbKE+fPnU1xczPHHH8/OnTsB2Lp1K2eeeSZvvPEGS5YsoW/fvgwYMICNGzdWW55zjnHjxjFp0iQWLFjAr3/968b+kSQDRfd2eugh7ztPImGX0o66mejee++lT58+3Hrrrf+d9sgjj9CpUycKCws58MADOfzww3d5zaRJk5g9ezYvvvgiZ1RdXwaorKxk5MiRLFq0iIULF9KtW7fG+jEkg0X3drryShgyxO+KRFJDAVWPZcuW8frrr9O2bdtqz33yyScceOCBfPvtt1xzzTW89tprrF+/nsrKSkpLS/niiy92mX/cuHE0a9aMxYsXs4eOv0gKRPd2GjgQrr/e74pEUkcBVY+dO3cycOBA7rjjjmrP5ebmAnD22Wezfv167rzzTrp160bLli054ogjKC8v32X+o446ipkzZ/L8888zYsSIxihfMlhsb6cZM9TbSTKLAqoe/fr144knnqBr1640b968xnkWLlzI3//+dwYOHAjA+vXrWbduXbX5BgwYwMknn8ypp56KmXH22WentXbJbFW9nTp0UG8nyUwaJBFly5YtLF++fJfbwIED2bx5M0OGDGHx4sWsWbOG+fPnM3r0aLZu3QpAjx49eOyxx1ixYgVLly5l6NChtKhlfO+gQYOYNWsWY8eO5ZFHHmnMH08yiHo7STbQHlSUN954gwNihj8NHjyYRYsWceWVV3LMMcdQVlZGly5dOProo2nZsiUA06ZNY/To0fTv358f//jHTJgwgQ0bNtS6nkGDBvHEE09w2mmnAXDWWWel74eSjBPd2+mWW+DYY/2tRyRdFFAR06dPZ/r06bU+X9dlivr06VPt+0xnnnnmLo+dc7s8Pu644ygtLU28UMlq0b2dhg2DP//Z74pE0keH+ERCoqLCG0L+ySfQrx88+KB6O0lmU0CJhMTll8P8+d4VIp5+Wr2dJPMpoERC4OGH4c47oXlz9XaS7JHxAbVu3Tqee+45v8sQSdrixT+0zJg8GQ4+2N96RBpLRgfUM888wz777MPgwYMpKCjwuxyRhH399Q+9nc4994cmhCLZICMDqqysjDFjxjBkyBC2bNnC9u3bGTx4MN99953fpYnErazM6+20bh0ceihMnOh3RSKNK+MC6sMPP+SXv/wljz766C7DuIuLixk2bFi14d4iQVTV22nxYu9806xZ6u0k2SdjAso5x3333UdeXh6ffPJJte8Y7dy5k8LCQkpKSnyqUCR+kybB9Oleb6c5c9TbSbJTRgTU999/z6BBg7jssssoKSmptpeUk5PDQQcdxMqVK2nTpo1PVYrE55VX4NJLvfvq7STZLPQBtXDhQnr06MH8+fNr3Dtq3bo11113Ha+99ppaXEjgrVkDp52m3k4iEOJLHVVUVHDttdcyceLEGi8Z1KpVKzp37sy8efOqXV9PJIjU20lkV6EMqC+++IITTzyRjz76qMZwysnJ4cQTT2TKlCk6pCehsHMnnHUWvP++ejuJVAldQM2aNYuRI0dSWlpKZWXlLs81adKEnJwcHnjgAYYOHepThSKJu+EG7/JF6u0k8oPQBNS2bds499xzmT17do3nmnJycujevTtz586la9euPlQokpw5c+Cvf1VvJ5FYoRgk8e6779KrVy9mzZpV60CIiy66iMLCQoWThMr770NVZxb1dhLZVaD3oJxzTJw4kauuuqrGc00tWrSgXbt2zJ49m8MOO8yHCkWSp95OInXzNaAqKipYvXo1vXr1qvbchg0bGDJkCEuWLKl1IMQhhxzC448/TqdOnRqjXJGUqerttGaNejuJ1MbXQ3z33HMP+++/P++8884u01955RV69uzJwoUL2bZtW7XX5eTkcPvtt/PCCy8onCSU1NtJpH6+7UGVlJTw17/+lYqKCo4//nhWrlxJixYtuPzyy7n//vtr3Gtq3bo1e+65J/PmzaN3794+VC3ScOrtJBKfuPagzKyTmT1tZtvM7HMzG97QFd91112Ul5cDsHHjRoYPH84BBxzAlClTaj2kN3z4cN5//32Fk4RWSUlT9XYSiVO8e1B3A+VALtAXeM7M3nXOfZDMSjdv3sxNN9303xF5ZWVlvPzyy2zfvp2dO3fuWmCzZuTk5PDII49wwgknJLM6kUDYuhU+/bQNFRXq7SQSD6uv/YSZtQG+B/Zzzq2KTHsUWOucu6K217Vr187179+/xufWrFnD2rVrq4VRrCZNmtCmTRt69+5Ny5Yt6/5JElBUVETHjh1Ttrxso+1Xs8pKb/BDTbdt22DduuUAdOjQlz59NCgiUXrfJS/o227BggXLnHN5sdPj2YPqAVRWhVPEu0C1cd1mNhoYDdC8eXOKioqqLayiooKvvvqq3r5MZsbuu+9Obm4upaWlNR72S1ZlZWWNtUl8MnH7OQc7dxqVlfHdapo3Hs2a7aRLl81s3qy+ZInKxPddYwnrtosnoNoCm2OmbQbaxc7onJsCTAHIy8tzhYWF1RZ20UUX8fHHH7N9+/ZaV9iuXTteeuklfvOb38RRXuIKCgrIz89Py7KzQRC3X3k5bN4MRUU//Ft1i+fxli0Nr6FNG+8SRR07/nCrerzbbvDii/lUVhaxfPk7dS9IahTE911YBH3bWS2HE+IJqGKgfcy09sDWRIv4+uuvefDBB+sMJ/D2sqoGUEjmc85rb55IoMQ+TkUfyg4dag+Y+h536OCNyqvLokVevSISn3gCahXQzMy6O+dWR6b1ARIeIHH11VdTUVFR73ylpaWcfPLJrFq1St9zCgHnvKshNCRgGvr3SNOmiQVK7ON27XT1cJGgqTegnHPbzOwp4DozG4U3iu8E4HeJrGjNmjXMnDmTHTt2xDX/999/z5gxY5g1a1Yiq5EkVFZ6h7hqCo+apn3+eR9g1+frGe9SrxYtvMNgyQZMmzYadCCSaeIdZn4eMA34FvgOODfRIebjx4+vNZzatm1LkyZNKC0tpX379vTo0YP+/fszYMCARFaRtRr//Mtu1abUdf4lnsetWiVag4hkurgCyjm3CTgx2ZWsXLmS2bNn07p1a5o1a0ZJSQmdOnWiZ8+e9O/fn759+7Lvvvuyzz770K5dtbEXGS2M518++2w5+fl9Ezr/IiKSqEa51FGLFi0YP348vXv3pnfv3vTs2ZOcDLn4WPT5l3gOj2XC+ZeCgiJq+YqbiEjKNEpA/exnP+Pmm29ujFUlrK7zL/E+1vkXEZHUC3Q/qHgkc/5l7do8duxI7fdfGrIHo/MvIiLV+RpQ/p1/abvLo3R//0VERBKXtoD69lu48cZgnn9ZvXopRx75K33/RUQkwNIWUF9+CVdfXf98fpx/cW4b3bol9hoREWlcaQuo3XeHUaN0/kVERJKTtoDq0gVuuildSxcRkUwXV0ddERGRxqaAEhGRQFJAiYhIICmgREQkkBRQIiISSAooEREJJAWUiIgEkgJKREQCSQElIiKBpIASEZFAMudcehZstgH4PC0Lb7jOwEa/iwgxbb/kadslT9sueUHfdl2dc7vHTkxbQAWZmRU65/L8riOstP2Sp22XPG275IV12+kQn4iIBJICSkREAilbA2qK3wWEnLZf8rTtkqdtl7xQbrusPAclIiLBl617UCIiEnAKKBERCSQFlIiIBJICCjCz7mZWZmaP+V1LGJhZSzObamafm9lWM3vHzI71u64gM7NOZva0mW2LbLfhftcUBnqvpUZYP+MUUJ67gaV+FxEizYAvgcOADsA1wBNm1s3PogLubqAcyAVOB+41s97+lhQKeq+lRig/47I+oMxsKFAEvOJzKaHhnNvmnJvgnPvMObfTOfcs8CnQ3+/agsjM2gCDgWucc8XOuYXAPOBMfysLPr3XGi7Mn3FZHVBm1h64DrjM71rCzMxygR7AB37XElA9gErn3Kqoae8C2oNKkN5riQn7Z1xWBxRwPTDVOfel34WElZk1B2YADzvnVvpdT0C1BTbHTNsMtPOhltDSey0pof6My9iAMrMCM3O13BaaWV/gSOBOn0sNnPq2XdR8TYBH8c6tXOBbwcFXDLSPmdYe2OpDLaGk91riMuEzrpnfBaSLcy6/rufN7GKgG/CFmYH3V25TM9vXOdcv3fUFWX3bDsC8jTYV76T/AOfcjnTXFWKrgGZm1t05tzoyrQ86TBUXvdeSlk/IP+Oy9lJHZpbDrn/VjsP7ZZ7rnNvgS1EhYmb3AX2BI51zxT6XE3hm9g/AAaPwttvzwO+ccwqpeui9lpxM+IzL2D2o+jjnSoCSqsdmVgyUheUX5ycz6wqMAbYD30T+OgMY45yb4VthwXYeMA34FvgO70NC4VQPvdeSlwmfcVm7ByUiIsGWsYMkREQk3BRQIiISSAooEREJJAWUiIgEkgJKREQCSQElIiKBpIASEZFAUkCJiEgg/X8dDYyn05afpgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, leaky_relu(z, 0.05), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([0, 0], [-0.5, 4.2], 'k-')\n",
    "plt.grid(True)\n",
    "props = dict(facecolor='black', shrink=0.1)\n",
    "plt.annotate('Leak', xytext=(-3.5, 0.5), xy=(-5, -0.2), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.title(\"Hàm kích hoạt Leaky ReLU\", fontsize=14)\n",
    "plt.axis([-5, 5, -0.5, 4.2])\n",
    "\n",
    "save_fig(\"leaky_relu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['deserialize',\n",
       " 'elu',\n",
       " 'exponential',\n",
       " 'gelu',\n",
       " 'get',\n",
       " 'hard_sigmoid',\n",
       " 'linear',\n",
       " 'relu',\n",
       " 'selu',\n",
       " 'serialize',\n",
       " 'sigmoid',\n",
       " 'softmax',\n",
       " 'softplus',\n",
       " 'softsign',\n",
       " 'swish',\n",
       " 'tanh']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[m for m in dir(keras.activations) if not m.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LeakyReLU', 'PReLU', 'ReLU', 'ThresholdedReLU']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[m for m in dir(keras.layers) if \"relu\" in m.lower()]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hãy huấn luyện một mạng nơ-ron trên Fashion MNIST bằng Leaky ReLU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.fashion_mnist.load_data()\n",
    "X_train_full = X_train_full / 255.0\n",
    "X_test = X_test / 255.0\n",
    "X_valid, X_train = X_train_full[:5000], X_train_full[5000:]\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.LeakyReLU(),\n",
    "    keras.layers.Dense(100, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.LeakyReLU(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 1.6314 - accuracy: 0.5054 - val_loss: 0.8886 - val_accuracy: 0.7160\n",
      "Epoch 2/10\n",
      "1719/1719 [==============================] - 2s 892us/step - loss: 0.8416 - accuracy: 0.7247 - val_loss: 0.7130 - val_accuracy: 0.7656\n",
      "Epoch 3/10\n",
      "1719/1719 [==============================] - 2s 879us/step - loss: 0.7053 - accuracy: 0.7637 - val_loss: 0.6427 - val_accuracy: 0.7898\n",
      "Epoch 4/10\n",
      "1719/1719 [==============================] - 2s 883us/step - loss: 0.6325 - accuracy: 0.7908 - val_loss: 0.5900 - val_accuracy: 0.8066\n",
      "Epoch 5/10\n",
      "1719/1719 [==============================] - 2s 887us/step - loss: 0.5992 - accuracy: 0.8021 - val_loss: 0.5582 - val_accuracy: 0.8200\n",
      "Epoch 6/10\n",
      "1719/1719 [==============================] - 2s 881us/step - loss: 0.5624 - accuracy: 0.8142 - val_loss: 0.5350 - val_accuracy: 0.8238\n",
      "Epoch 7/10\n",
      "1719/1719 [==============================] - 2s 892us/step - loss: 0.5379 - accuracy: 0.8217 - val_loss: 0.5157 - val_accuracy: 0.8304\n",
      "Epoch 8/10\n",
      "1719/1719 [==============================] - 2s 895us/step - loss: 0.5152 - accuracy: 0.8295 - val_loss: 0.5078 - val_accuracy: 0.8284\n",
      "Epoch 9/10\n",
      "1719/1719 [==============================] - 2s 911us/step - loss: 0.5100 - accuracy: 0.8268 - val_loss: 0.4895 - val_accuracy: 0.8390\n",
      "Epoch 10/10\n",
      "1719/1719 [==============================] - 2s 897us/step - loss: 0.4918 - accuracy: 0.8340 - val_loss: 0.4817 - val_accuracy: 0.8396\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bây giờ hãy thử PReLU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.PReLU(),\n",
    "    keras.layers.Dense(100, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.PReLU(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 1.6969 - accuracy: 0.4974 - val_loss: 0.9255 - val_accuracy: 0.7186\n",
      "Epoch 2/10\n",
      "1719/1719 [==============================] - 2s 990us/step - loss: 0.8706 - accuracy: 0.7247 - val_loss: 0.7305 - val_accuracy: 0.7630\n",
      "Epoch 3/10\n",
      "1719/1719 [==============================] - 2s 980us/step - loss: 0.7211 - accuracy: 0.7621 - val_loss: 0.6564 - val_accuracy: 0.7882\n",
      "Epoch 4/10\n",
      "1719/1719 [==============================] - 2s 985us/step - loss: 0.6447 - accuracy: 0.7879 - val_loss: 0.6003 - val_accuracy: 0.8048\n",
      "Epoch 5/10\n",
      "1719/1719 [==============================] - 2s 967us/step - loss: 0.6077 - accuracy: 0.8004 - val_loss: 0.5656 - val_accuracy: 0.8182\n",
      "Epoch 6/10\n",
      "1719/1719 [==============================] - 2s 984us/step - loss: 0.5692 - accuracy: 0.8118 - val_loss: 0.5406 - val_accuracy: 0.8236\n",
      "Epoch 7/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5428 - accuracy: 0.8194 - val_loss: 0.5196 - val_accuracy: 0.8314\n",
      "Epoch 8/10\n",
      "1719/1719 [==============================] - 2s 983us/step - loss: 0.5193 - accuracy: 0.8284 - val_loss: 0.5113 - val_accuracy: 0.8316\n",
      "Epoch 9/10\n",
      "1719/1719 [==============================] - 2s 992us/step - loss: 0.5128 - accuracy: 0.8272 - val_loss: 0.4916 - val_accuracy: 0.8378\n",
      "Epoch 10/10\n",
      "1719/1719 [==============================] - 2s 988us/step - loss: 0.4941 - accuracy: 0.8314 - val_loss: 0.4826 - val_accuracy: 0.8398\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ELU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def elu(z, alpha=1):\n",
    "    return np.where(z < 0, alpha * (np.exp(z) - 1), z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure elu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAfzklEQVR4nO3dfXRU9b3v8fc3EOVRHkRzq4ixKi4tergadalVY+VYsXqtpa1tUQ+918LVclusXeeoF1apUjz26Km2VQoeerBoj7Cwter12TpaFcVYUEADgsiDqBBxgEAgJPzuH79JmEwmmUyyk733zOe11l7J7L1nzze/7Mwne8+e75hzDhERkagpCbsAERGRbBRQIiISSQooERGJJAWUiIhEkgJKREQiSQElIiKRpIASEZFIUkBJl5lZqZn9i5mdFHYtAGZWZmY/M7PDwq5FRDpPASVd5pzbB7wPLDKzAd31OGaWMLPf5linN7AAaHDObe3q9oK4T6EzsyFm9qmZHRt2LdmY2SIz+0nYdUj+FFDSJjObZ2ZPZJlfYWbOzMqb5jnn/gTMSU1h+jdguXPuFyHX0SUdDcLU78hlmV7PWKfV7zHXY5nZBDOr7UC5twBPOufWdmDdQJnZeWb2mJl9lPq5J2RZ7efAVDMb1MPlSRf1DrsAKRzOubsjUMMNYdcQgueBqzPm1ffEA5tZP+Ba4LKeeLwsBgArgD+kplacc8vN7APgKuDeHqxNukhHUNJlZnaxmf3NzD43s21m9oyZnZi2PGFms8zsrtTyrWb2YzM72MzuNbOkmW0ws8wn2VyPe2HqvpPS5jUfLZh3o5m9b2Z7zWyTmd2esZkSM5tpZjVmtsXM7jSzXH8X7d4n9XPdnTrttcfMXjezL3d0zMxsHnA+8MO0I6LydurZ65z7JGPalnMAg3EJsB94NX2mmZ1mZi+YWZ2ZrUkd6XzbzF7NvpnOcc496Zy7xTm3KFVHWx4DvhvkY0v3U0BJEPoDdwNnAJXAduBxMzsobZ3xwE7gTOBfU+s/CqwGKoAHgP8wsyM68oBmNg74MzDROTe7jdVmAtOA24EvAd8CNmasMx5oAM4GJgNTgCtzPHyu+/wydft/Av8dWA48bWZfSFunvTH7MbAY+E/gC6kps+6oOBd4y6V1nTaz04G/AS8CpwCv40+z/V/876MFM7vFzGpzTOd2sc4lwBlm1reL25Ge5JzTpCnrBMzDPxHXZky7AQeUt3G//kAj8OXU7QSwOG25AVuBx9LmleJPS32znXoSwG+Bifgn9IvaqPkJ/KmfPcD/zrG9xRnzngP+o7P3Sf3s9cA1act7AWuBGe1sN9uY/bYLv6M7Msck17hmmT8BqM3x+I8CD2TMexlYkHb7ktTP9mIb2xgKHJdj6tuBsagFJrSx7JTUPntsGH9Lmjo36TUoyeVlfCCkG4U/egEgdfXWbfijo8PwR+YlwIi0+7zT9I1zzpnZFvyRRdO8fWb2OXB4jnouByYB5znnFrez3knAwcALObb3TsbtzR2oob37HIsP2+ZTWc65RjNbnKoJ6PCYdVS231GyE9vpjL7Ap003zOy/4Y+qLkhbpx7/s7U6egJw/nRkd5+SrEt91RFUjCigJJfdzrk16TPMbHDGOo8DH+GD4yP8f/TvAumn+PZl3Me1MS/Xaed3Uuv9LzN73aX+Pc7Ccmynvbpy1dDefSxtXqb0eR0Zs45q9TvK0w4g2xVug/FHqu2pAYak3W567fHNtHknAKucc69k24CZ3YK/ErA9Y51zf8uxTnuGpr62+9YDiRYFlHSJmR2Kf1L6oXPuxdS8U+m+fWsd8H/wp6XmmNnENkLqXWAvcCH+PVo9ZQ3+iOHLwAcAZtYLOAv4Y+p2R8asHn9qsCesAi4xM8sYy1NTy9qzFH8qsMlgfBDvBzCzgfjXnj5pZxu/AxbmeJyPcizPZRSw2Tn3ac41JTIUUNJVn+P/i/6BmW0EjsS/F6mhux7QOfeBmV1AOyHlnNtpZvcAt5vZXvxpsEOB05xzs7qxtl1mNgv4VzOrwQfqDUAZcF9qtY6M2Yf4F/XL8a+tbHPOtXWV2sGpU2vpGl3LNyofYmajM9ZJOuc+BGbhL/b4jZndj3/t7hL8VW+X5/iRnwHuMLNDnXOfAcvwR5E3m9lDqZ/rY+A4MzveOdfqn4WunOIz/8bw41I3S4ARqZ9zm3NuQ9qq5wJPd+YxJDy6ik+6JPWkeSX+RegV+PeZTMMfvXTn467FX/12MTDbzLKd0rsZuCNVz3vAI8Dw7qwr5V/wRwT/iX/CPgW42Dn3MXR4zO7EH0W9iz8t1d5rU2PwIZA+Lc1Y59zUvPTpzlQ9HwDnAccDz+KvePsO8C3n3JPt/aDOueVp6+OcW4c/YroOeBt/5eaY1M/5Wnvb6qQKDvw8ffFXCy4Fbm1awcz6AFcA93fD40s3srZP4YuI5GZmFwP3ACc55xrDrieTmf0QuNw5d1HYtUh+dAQlEiIz+8TMxoddR1c4557GHwX2xNFpZ+zDv24pMaMjKJEQmdkXga3OuZ1h1yISNQooERGJJJ3iExGRSFJAiYhIJIX+Pqhhw4a58vLysMtoZdeuXfTv3z/sMmJH45afVatW0djYyEknReLDiGMjTvvZnj1QXQ2NjTB0KBxzTDh1RHnM3nrrrRrnXKtPwA49oMrLy6mqqgq7jFYSiQSVlZVhlxE7Grf8VFZWkkwmI/k3EGVx2c82boSzz/bh9LWvwZ//DKWl4dQS5TEzs/XZ5usUn4hIN6ipgYsugk2b4JxzYOHC8MIprhRQIiIBq631R0zV1TBqFDz+OPTrF3ZV8aOAEhEJUH09jBsHS5ZAeTk88wwMGZLzbpJFoAFlZg+a2cdmtsPMVpvZtUFuX0Qkyhob4Zpr4Nln4fDD/dcjOvQZ0ZJN0EdQt+M/ZfUQ4H8AM8zstIAfQ0QkcpyDH/8YFiyAgQPhqafg+OPDrireAg0o59xK51xTR2aXmo4N8jFERKLo1lvh3nvhoIPgL3+BU08Nu6L4C/wyczO7D/8BZn3xbe9btes3s4mkPqK6rKyMRCIRdBldVltbG8m6ok7jlp9kMkljY6PGLE9R288effQI7rlnJCUljqlTV2JWQ4TKA6I3Zh3RLb340j5BtBK4wzmX+RHZzSoqKlwU3wMS5fcMRJnGLT9N74NatmxZ2KXESpT2swUL4Lvf9af47r8fro3oK+9RGrNMZvaWc64ic363XMXnnGt0zr2Cb79/XXc8hohI2J59Fq6+2ofTzJnRDae46u7LzHuj16BEpAC98QZ84xuwbx/ccAPcdFPYFRWewALKzA43s++Y2QAz62VmXwW+C/w1qMcQEYmC997zb8TdtcsfQd15J5iFXVXhCfIiCYc/nfc7fPCtB6Y45/4S4GOIiIRq40bfwuizz3xIzZ0LJWp50C0CCyjn3Fbg/KC2JyISNeqv17OU+yIiHaD+ej1PASUikoP664VDASUi0g711wuPAkpEpA3qrxcuBZSISBvUXy9cCigRkSzuuw+mT/eXkD/8MFxwQdgVFR8FlIhIhgULYPJk//3s2XDFFeHWU6wUUCIiadRfLzoUUCIiKeqvFy0KKBER1F8vihRQIlL01F8vmvQrEJGipv560aWAEpGipf560aaAEpGipP560aeAEpGio/568aCAEpGiov568aGAEpGiov568aGAEpGiof568aKAEpGioP568aOAEpGCp/568aSAEpGCpv568aWAEpGCpf568aaAEpGCpP568adfl4gUHPXXKwwKKBEpKOqvVzgUUCJSMNRfr7AooESkIKi/XuFRQIlI7Km/XmFSQIlI7Km/XmFSQIlIrKm/XuFSQIlIbKm/XmFTQIlILKm/XuELLKDM7GAzm2tm681sp5ktNbOxQW1fRKTJu+8OVH+9IhDkEVRvYCNwPjAImAYsNLPyAB9DRIrce+/BzTefov56RaB3UBtyzu0CpqfNesLM1gGnAR8G9TgiUrya+uvt2FGq/npFILCAymRmZcBIYGWWZROBiQBlZWUkEonuKqPTamtrI1lX1Gnc8pNMJmlsbNSYdcD27aX86Eej2bSpPyeeuI3Jk1fw6qv7wy4rNuL4t9ktAWVmpcBDwAPOuerM5c65OcAcgIqKCldZWdkdZXRJIpEginVFncYtP4MHDyaZTGrMcqithQsvhA0bfH+9mTPf5eKLzwu7rFiJ499m4AfHZlYCzAfqgclBb19Eiku2/noDBzaEXZb0gEADyswMmAuUAeOcc/uC3L6IFBf11ytuQZ/imwWcCIxxztUFvG0RKSLqrydBvg/qaGASMBr4xMxqU9P4oB5DRIqH+utJkJeZrwf0bgQR6TL11xNQqyMRiRj115MmCigRiQz115N0CigRiYQ33kD99aQFBZSIhO699+BrX0P99aQFBZSIhKqpv95nn6H+etKCdgMRCU1NjQ+nTZvgnHNg4UIoLQ27KokKBZSIhKK21h8xVVf7/nqPPw79+oVdlUSJAkpEely2/npDhoRdlUSNAkpEepT660lHKaBEpMeov57kQwElIj1G/fUkHwooEekR6q8n+VJAiUi3U3896QwFlIh0K/XXk85SQIlIt1F/PekKBZSIdAv115OuUkCJSODUX0+CoF1GRAKl/noSFAWUiARG/fUkSAooEQmE+utJ0BRQItJl6q8n3UEBJSJdov560l0UUCLSJeqvJ91FASUinab+etKdFFAi0inqryfdTQElInlTfz3pCQooEcmL+utJT1FAiUiHqb+e9CQFlIh0iPrrSU/T7iUiOam/noRBASUi7VJ/PQmLAkpE2qT+ehKmQAPKzCabWZWZ7TWzeUFuW0R6lvrrSdh6B7y9zcAM4KtA34C3LSI9RP31JAoCDSjn3J8AzKwCGB7ktkWk56i/nkRB0EdQHWJmE4GJAGVlZSQSiTDKaFdtbW0k64o6jVt+kskkjY2NkRqzRx89gnvuGUlJiWPq1JWY1RCh8gDtZ50RxzELJaCcc3OAOQAVFRWusrIyjDLalUgkiGJdUadxy8/gwYNJJpORGbMFC+DXv/bfz55tXHvtqHALaoP2s/zFccx0FZ+IAOqvJ9GjgBIR9deTSAr0FJ+Z9U5tsxfQy8z6AA3OuYYgH0dEgqP+ehJVQR9BTQXqgJuAq1LfTw34MUQkIOqvJ1EW9GXm04HpQW5TRLqH+utJ1Ol/JZEipP56EgcKKJEio/56EhcKKJEiov56EicKKJEiof56EjcKKJEiof56EjcKKJEicN99MH26v4T84YfhggvCrkgkNwWUSIFbsAAmT/bfz54NV1wRbj0iHaWAEilg6q8ncaaAEilQ6q8ncaeAEilA6q8nhUABJVJg1F9PCoV2W5ECov56UkgUUCIFQv31pNAooEQKgPrrSSFSQInEnPrrSaFSQInEmPrrSSFTQInEmPrrSSFTQInElPrrSaFTQInEkPrrSTFQQInEjPrrSbFQQInEiPrrSTFRQInEhPrrSbFRQInEgPrrSTHSLi4SceqvJ8VKASUSYeqvJ8VMASUSUeqvJ8VOASUSQeqvJ6KAEokc9dcT8RRQIhGj/noingJKJELUX0/kAAWUSESov55ISwookQhQfz2R1gINKDMbamZ/NrNdZrbezL4X5PZFCtHu3b3UX08ki94Bb+9eoB4oA0YD/8/M3nbOrQz4cUQKwu7d8MEHA2hsVH89kUzmnAtmQ2b9gc+BUc651al584GPnHNt/k84cOBAd9pppwVSQ5CSySSDBw8Ou4zY0bh13J49sGTJMpyDoUNHM2qUwqmjtJ/lL8pj9tJLL73lnKvInB/kEdRIoLEpnFLeBs7PXNHMJgITAUpLS0kmkwGWEYzGxsZI1hV1GreO2bevhDVrBuAclJQ4jjxyO9u3B/PPYjHQfpa/OI5ZkAE1ANieMW87MDBzRefcHGAOQEVFhauqqgqwjGAkEgkqKyvDLiN2NG651dTA+ef7VkYDBlRSXr6dd95ZGnZZsaL9LH9RHjNr49RBkAFVCxySMe8QYGeAjyESa8kkjB0L774LX/oSDBoEu3bpyEkkmyCv4lsN9Daz9KYs/wDoAgkR/JHTV74CVVXwxS/6S8v1sRkibQssoJxzu4A/AbeaWX8zOwe4HJgf1GOIxNUnn0BlJSxdCscdB4mEmr+K5BL0G3WvB/oCW4D/Aq7TJeZS7DZuhPPOg5Ur4aST4OWX4aijwq5KJPoCfR+Uc24b8PUgtykSZ9XV/jWnDz+E0aP9ab3DDgu7KpF4UKsjkW7y0ktw1lk+nM48E/76V4WTSD4UUCLd4I9/hIsu8lftXX65Dyd9Gq5IfhRQIgFyDn7xCxg/3r/P6Uc/gkcegX79wq5MJH6C7sUnUrR27IDvfx/+9CffsuhXv/KfjCsinaOAEglAdbX//Kbqav/m2wcfhEsvDbsqkXjTKT6RLlq0CE4/3YfTqFH+jbgKJ5GuU0CJdFJtrf9gwW99y3//ne/A66/7N+KKSNfpFJ9IJ7z5Jnzve7BmDfTp4z/H6frr9XEZIkHSEZRIHurr4ec/h7PP9uF0yin+lN4Pf6hwEgmajqBEOuj11/0pvZWp5l033AAzZ/ojKBEJno6gRHKorYUpU/xR08qV/jWmF1+Ef/93hZNId1JAibTBOX+5+AknwD33QEkJ/PM/wzvv+M7kItK9dIpPJIuqKt8FYvFif7uiAmbPhlNPDbcukWKiIyiRNGvX+jZFZ5zhw6msDH7/e3jjDYWTSE/TEZQIsHkz3HorzJ0LDQ1w0EH+CGraNDjkkLCrEylOCigpahs2+Pcw3X8/7NnjX2f6/vfhZz+Do48OuzqR4qaAkqK0ahXccQfMn++PmAC+8Q2YMQNOPDHc2kTEU0BJ0XAOnnsOfv1rePJJf7ukxHeEuOkmOPnksCsUkXQKKCl4n38ODz0E997rG7qCf//SNdf4y8aPPTbc+kQkOwWUFKTGRnjhBX8F3qOPwt69fv6RR/q2RD/4AQwbFmqJIpKDAkoKyurV8Ic/wAMPwKZNfp4Z/OM/+jZFV1wBpaXh1igiHaOAklhzDt59138m0yOPwPLlB5Z98YswYQL80z/BiBGhlSginaSAktjZvx/eegsee8wHU9PrSuA/zfbrX/eXip97rr8IQkTiSQElsfDpp/DMM3569lmoqTmwbOhQf+pu3Di48EL/JlsRiT8FlETSZ5/BK6/Ayy/7zuFLl7ZcXl4OY8f69y6df75eVxIpRAooCZ1zsG4dLFkCf/ubD6UVK1qu06cPXHABXHyxn44/Xh8QKFLoFFDSo5yDTz7xryEtWeI/Ov3NN/0RU7o+feDMM+G88/x0zjnQt284NYtIOBRQ0m127vRHQsuXH/i6fHnrMAI47DA4/XT48pd9IFVUwMEH93zNIhIdCijpkn37YP16eP99WLMGXnzxOH75S3/p9/r12e8zaBCMHu0/0uKMM3wwjRihU3Yi0pICStq1d69/w+vGjQemDRt8+KxZ4187amxMv8fw5u8OOghOOglGjfJ97k4+2X8/fLjCSERyU0AVqbo62LLFX77dNG3Z4l8fagqkDRv8vPaY+aOf445runBhLV/96rGccIK/3Vt7mIh0kp4+Yq6hwTdDTZ+2bcs+r6bmQBjt3Nmx7ffq5fvXHXVUy2nECB9AxxzT8uKFRGIjlZXqvioiXRdIQJnZZGACcDLwX865CUFst1A4B/X1/qilrs5/MF7m19paP+3c2f7X9O937Oh40GQqLfUfZ3744f5r+vfDhx8Ioi98wYeUiEhPC+oIajMwA/gqkPfFwPv3+9cxmqbM20HPa2jwL+7X17f8mv79+++X8/TTree39X1TAGULnz17fEh1h5ISGDwYhgzx09ChB77PvD1s2IEgGjxYrwOJSLSZC/CZ08xmAMPzOYIyG+jgtIy53wauB3YDl2S514TUVAN8M8vy64ArgY3A1VmW3whcBqwCJmVZPhUYAywDpmRZPhM4G3gNuCXL8ruB0cDz+Nz2YVBS4qfDD5/NwIEnUF//ODU1d1FS4o9SmqZLL53PEUccxZo1C1iyZBa9evnXcpqW/+Y3ixgxYhiPPTaPhQvntXr0J598kn79+nHfffexcOHCVssTiQQAd955J0888USLZX379uWpp54C4LbbbuOFF15osfzQQw/lkUceAeDmm29m8eLFLZaXlpby3HPPATBlyhSWLVvWYvnIkSOZM2cOABMnTmT16tUtlo8ePZq7774bgKuuuopNTS3JU8466yxuv/12AMaNG8dnGdesX3jhhUybNg2AsWPHUldX12L5pZdeyk9/+lMAKisryfTtb3+b66+/nt27d3PJJa33vQkTJjBhwgRqamr45jdb73vXXXcdV155JRs3buTqq1vvezfeeCOXXXYZq1atYtKkSSxbtoyGhgYqKioAmDp1KmPGjGHZsmVMmTKl1f1nzpzJ2WefzWuvvcYtt7Te9+6++25Gjx7N888/z4wZM1otnz17NieccAKPP/44d911V6vl8+fP56ijjmLBggXMmjWr1fJFixYxbNgw5s2bx7x581ot76l9b/z48Xz00Uctlg8fPpwHH3wQ0L6Xbd+76KKLuOWWW5r3vUxh7nsvvfTSW865isz7hPIalJlNBCb6WwPS5gM4evdu5KCDGjDbR12da57f9B9///71DBiwG+d2UVPT2Dy/6evQobsZMmQ7DQ3b2bSpIW37fhtHHrmdsrIt7N1bQ3V1PWYHlpnBKad8SlnZKurqNvH663XN85vW+cpX1nHMMYP46KO1PP/8rub5JSWOkhK45prlHH/8bt57bwWLFm1vXt7kJz95gxEjPua115azcGGy1fiMGbOYww9fy/79K1mxouXyxkbYuvVV6usHsWFDNclk6/u//PLL9OnTh9WrV2dd3vQksXbt2lbL6+rqmpevW7eu1fL9+/c3L9+wYUOr5UOGDGlevmnTplbLN2/e3Lx88+bNrZZv2rSpefmnn37aavmGDRual2/dupUdO3a0WL5u3brm5du2bWNv0wdBpaxdu7Z5ebaxWb16NYlEgj179mRdXl1dTSKRYPv27VmXr1y5kkQiwZYtW7IuX758OQMHDmweu4aGBpxzzeu+/fbb9O7dmzVr1mS9/9///nfq6+tZsWJF1uVVVVUkk0nefvvtrMvfeOMNPv74Y5YvX551+eLFi1m7di0rV67MuvzVV19l0KBBVFeHu+/V19e3Wl5aWqp9r519b8+ePSQSiax/txD+vpdN6EdQFRUVrqqqKrAagpJIJLL+lyPt07jlp7KykmQy2eq/fWmf9rP8RXnMzCzrEVTODyMws4SZuTamV7qnXBERKXY5T/E55yp7oA4REZEWgrrMvHdqW72AXmbWB2hwzjW0f08REZHsgvq80alAHXATcFXq+6kBbVtERIpQIEdQzrnpwPQgtiUiIgLBHUGJiIgESgElIiKRpIASEZFIUkCJiEgkKaBERCSSFFAiIhJJCigREYkkBZSIiESSAkpERCJJASUiIpGkgBIRkUhSQImISCQpoEREJJIUUCIiEkkKKBERiSQFlIiIRJICSkREIkkBJSIikaSAEhGRSFJAiYhIJCmgREQkkhRQIiISSQooERGJJAWUiIhEkgJKREQiSQElIiKRpIASEZFIUkCJiEgkKaBERCSSFFAiIhJJCigREYkkBZSIiERSlwPKzA42s7lmtt7MdprZUjMbG0RxIiJSvII4guoNbATOBwYB04CFZlYewLZFRKRI9e7qBpxzu4DpabOeMLN1wGnAh13dvoiIFKcuB1QmMysDRgIr21lnIjARoKysjEQiEXQZXVZbWxvJuqJO45afZDJJY2OjxixP2s/yF8cxM+dccBszKwWeAtY65yZ15D4VFRWuqqoqsBqCkkgkqKysDLuM2NG45aeyspJkMsmyZcvCLiVWtJ/lL8pjZmZvOecqMufnfA3KzBJm5tqYXklbrwSYD9QDkwOtXkREik7OU3zOucpc65iZAXOBMuAS59y+rpcmIiLFLKjXoGYBJwJjnHN1AW1TRESKWBDvgzoamASMBj4xs9rUNL6r2xYRkeIVxGXm6wELoBYREZFmanUkIiKRpIASEZFICvR9UJ0qwGwrsD7UIrIbBtSEXUQMadzypzHLn8Ysf1Ees6Odc4dlzgw9oKLKzKqyvXFM2qdxy5/GLH8as/zFccx0ik9ERCJJASUiIpGkgGrbnLALiCmNW/40ZvnTmOUvdmOm16BERCSSdAQlIiKRpIASEZFIUkCJiEgkKaA6yMyON7M9ZvZg2LVEmZkdbGZzzWy9me00s6VmNjbsuqLIzIaa2Z/NbFdqvL4Xdk1Rpn2ra+L4HKaA6rh7gTfDLiIGegMbgfOBQcA0YKGZlYdZVETdi/+AzzJgPDDLzL4UbkmRpn2ra2L3HKaA6gAz+w6QBF4IuZTIc87tcs5Nd8596Jzb75x7AlgHnBZ2bVFiZv2BccA051ytc+4V4DHg6nAriy7tW50X1+cwBVQOZnYIcCtwY9i1xJGZlQEjgZVh1xIxI4FG59zqtHlvAzqC6iDtWx0T5+cwBVRutwFznXMbwy4kbsysFHgIeMA5Vx12PREzANieMW87MDCEWmJH+1ZeYvscVtQBZWYJM3NtTK+Y2WhgDPCrkEuNjFxjlrZeCTAf/xrL5NAKjq5a4JCMeYcAO0OoJVa0b3Vc3J/DuvyJunHmnKtsb7mZTQHKgQ1mBv6/3l5mdpJz7tTuri+Kco0ZgPnBmot/8f8S59y+7q4rhlYDvc3seOfc+6l5/4BOV7VL+1beKonxc5haHbXDzPrR8r/cn+J/2dc557aGUlQMmNnvgNHAGOdcbcjlRJaZPQw44Fr8eD0JnO2cU0i1QftWfuL+HFbUR1C5OOd2A7ubbptZLbAnDr/YsJjZ0cAkYC/wSeq/NoBJzrmHQissmq4Hfg9sAT7DP2konNqgfSt/cX8O0xGUiIhEUlFfJCEiItGlgBIRkUhSQImISCQpoEREJJIUUCIiEkkKKBERiSQFlIiIRJICSkREIun/A98BnzSvKR5WAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, elu(z), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [-1, -1], 'k--')\n",
    "plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
    "plt.grid(True)\n",
    "plt.title(r\"Hàm kích hoạt ELU ($\\alpha=1$)\", fontsize=14)\n",
    "plt.axis([-5, 5, -2.2, 3.2])\n",
    "\n",
    "save_fig(\"elu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sử dụng ELU trong TensorFlow rất đơn giản, chỉ cần chỉ định hàm kích hoạt này khi xây dựng từng tầng:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7ff1b25afad0>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"elu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SELU"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chức năng kích hoạt này đã được giới thiệu trong [bài báo tuyệt vời](https://arxiv.org/pdf/1706.02515.pdf) này của Günter Klambauer, Thomas Unterthiner và Andreas Mayr, được xuất bản vào tháng 6 năm 2017. Trong quá trình huấn luyện, một mạng nơ-ron chỉ bao gồm một chồng các tầng dày đặc sử dụng hàm kích hoạt SELU và khởi tạo LeCun sẽ tự chuẩn hóa: đầu ra của mỗi tầng sẽ có xu hướng duy trì cùng một giá trị trung bình và phương sai trong quá trình huấn luyện, giúp giải quyết vấn đề tiêu biến/bùng nổ gradient. \n",
    "\n",
    "Kết quả là, hàm kích hoạt này vượt trội hơn rất nhiều so với các hàm kích hoạt khác đối với kiểu mạng nơ-ron như trên, cho nên bạn thực sự nên dùng thử nó. Thật không may, tính năng tự chuẩn hóa của hàm kích hoạt SELU dễ bị hỏng: bạn không thể sử dụng chuẩn hóa ℓ<sub>1</sub> hoặc ℓ<sub>2</sub>, dropout, định mức tối đa, kết nối tắt hoặc các cấu trúc liên kết không tuần tự khác (vì vậy các mạng nơ-ron hồi tiếp sẽ không tự chuẩn hóa). Tuy nhiên, trong thực tế, nó hoạt động khá tốt với các CNN tuần tự. Nếu bạn phá vỡ quá trình tự chuẩn hóa, SELU sẽ không nhất thiết hoạt động tốt hơn so với các hàm kích hoạt khác."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.special import erfc\n",
    "\n",
    "# alpha and scale to self normalize with mean 0 and standard deviation 1\n",
    "# (see equation 14 in the paper):\n",
    "alpha_0_1 = -np.sqrt(2 / np.pi) / (erfc(1/np.sqrt(2)) * np.exp(1/2) - 1)\n",
    "scale_0_1 = (1 - erfc(1 / np.sqrt(2)) * np.sqrt(np.e)) * np.sqrt(2 * np.pi) * (2 * erfc(np.sqrt(2))*np.e**2 + np.pi*erfc(1/np.sqrt(2))**2*np.e - 2*(2+np.pi)*erfc(1/np.sqrt(2))*np.sqrt(np.e)+np.pi+2)**(-1/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def selu(z, scale=scale_0_1, alpha=alpha_0_1):\n",
    "    return scale * elu(z, alpha)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure selu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAlSElEQVR4nO3deXgV1f3H8feXsC8SFExFUGoVFTfEPLao1SjUBcFdUXGhVkGsVhDrgmBRcC1t0QoIFksFF3CXRVuXpj8tVguF0qKCCyDuoAQIe5Lz++PclJAEkptMcube+3k9zzxc7kzmfjkZ8snMnDnHnHOIiIjETYPQBYiIiFRGASUiIrGkgBIRkVhSQImISCwpoEREJJYUUBJLZnacmV0Tuo5SZjbQzHqGrkMkkyigJK7mA5ea2VV1+SFmNtLM/lvFNgOAgcA71dhfvpk9lGQN/c2sMJmvEckECiipN2Y2xcxmVfJ+rpk5M+tU+p5zbhPQB7jOzI6oxzLL13Y0cD1wmnNufag6opBMEJrZlWa2wMwKzWytmS0ys9Hl9uV2sjRNbFPp9zuxrtIgV1hLWQ1DFyCyM8651cDhgWt4FzgkZA31zcyuAB4EhgCvA43xbdC93KYbgR+U/3rn3Oa6rlEyg86gJHbMLMvMJpvZMjPbZGYfmtlNZtagzDZTzGyWmd1sZl8lfsu/18waJC7bfZN4/+YkP3sfM/vAzP5kZg3NLC9xVtC2zDY/MrM3zGxD4nNfN7P2ZXbTwMzuNrPViTrGlK19F5/dw8z+m9jvX83s++XWDzSzj8xsa+LPq8qtvyFxprPBzD43sz+YWXZiXR7wR6BFmTOdkTsp5QzgOefcROfcR86595xzTzvnbii3nXPOfVV+qerfKVJdCiiJowbA58AFwMHAbcAw4Kfltjse+D6QB1wN3ATMAZoAxwEjgXvN7KjqfKiZHQz8PbGP/s65okq2OQL4K/ARcCzwI2AGO16N6AcUAccA1wKDgb5VfHwT4FbgCvyZSjbwcJnPPRt4CBgLHAo8AIw3sz5l9lGS+KxDgIuBo4HfJ9bNTazbCOyVWMbspJavgKPNbL8qahapU7rEJ/Xt1EruMezwi5Jzbhtwe5m3lptZN+AiYHKZ99cCP3fOFQMfmNlQoL1z7tTE+qVmdgtwIr7TxU4l7jXNAX7nnLtrF5veBPzbOTegzHvvl9vmPedcaf1LE2c6PYAnd7Hfhol/y5JEPWOAP5pZA+dcCXAjMNU5V3rfZmkieG8GZgI458aW2d9yM7sJeNHMLnfObTWztSTOenZRB8AdwBHAx2b2Eb5zyF+AJxPfm1ItKvleLnLOHVPF/kWqRQEl9e3/gAHl3jsUeL7sG2Z2NXAlsC/QDGgErCj3de8lwqnU10BBuW2+Bvasoqa98fda7nTO/bqKbY8sX2slFpX7+xfVqGFLaTiV+ZpG+DOp7/Bnko+W+5q38JfjADCzk/BnYQcDrYEs/P2j7yX2Vy3OuS+B7mZ2KHAC/kxwIjDEzI51zm1MbLoR6Fr+31HdzxGpii7xSX3bmLiv8b8F+KzsBmbWF38pawpwCv6H4Hj8D9uytpX7u9vJe1Ud56uBfwAXmlmbKra1KtbvrK6qaih/ObF0moEGlbxXYTsz2xeYjT+bOx84Cn+5ECq2W7U45/7rnBvnnOsH/AT/fbhgx012/F4651ZWc/fr8CFaXjb+zFhEASWxdBzwjnPuIefcvxIhVqG3WIS24M9E1gCvVhFS/wJOqsNaduZ9fLuUdRzwXuJ1Lj6Ihjjn3nbOLQXal9t+K/6sqiZKP6dlDb++vCVANzMrH/jdEutEdIlPYmkp0N/MTsN3RrgQf6lpTV19oHNuU6LDwSx8SPV0zhVUsumvgX+Y2SRgHLAZ+DHwF+fcp3VVX+Jznzaz+fj7QafiO2Ock1j/If4XzsFm9hy+88bgcvtYDjQ1s58AC/BnsxvLbYOZTcBfEnwDf3a7FzAcf0nvLztuat+rpNZVZS697mZmXcutLwAm4DuQ/N7MHsG3Yy/8fcYzd9YIkll0BiVxNBHfM+4J4J9AJ+A3df2hiYeDe+MvMb1a2kW73DYLgZ7AQfjLgu/gA7T8Zb2oa3sBuA7/bNJ7+IeHr3HOlXaQWJR474bE+ivxHSvK7mMuvmfgk8AqfIePyrwK/BD/PVjK9ntuP0mcmZVqDnxZyVK2e/yP8WFYdhnjnPsE3wvzAHzovYtvx/Odc3Oq1yqS7kwz6oqISBzpDEokjZjZkYnhifYJXYtIbekMSiSNmFkToCOwvLIHjUVSiQJKRERiSZf4REQkloJ3M2/btq3r1KlT6DIq2LBhAy1atAhdRspRuyVnyZIlFBcX06VLl9ClpJS4HmeFhbB0KTgHHTpATk7oiraLa5sBzJ8/f7Vzrl3594MHVKdOnZg3b17oMirIz88nLy8vdBkpR+2WnLy8PAoKCmL5fyDO4nicLV0K3bv7cLr2WnjwQajwGHJAcWyzUmZWfhgzQJf4RERqbdUq6NULvvsOeveGsWPjFU6pSgElIlILmzbBmWfCxx9Dt27w5JOQVdMBpWQHCigRkRoqKYHLL4e334aOHWHWLGgZ1WiFooASEampYcPg6aehVSuYPRv22it0Rekl0oAys2lm9qWZrTOzpWZ2ZZT7FxGJi0cegfvu85fznnkGDjssdEXpJ+ozqHuATs653fDTF4yu7nTbIiKp4s9/hkGD/OuHH4aTTw5bT7qKNKCcc4udc6UzarrEUpfz+IiI1KtFi+D886G4GG69Fa7UdaI6E/lzUGY2HuiPn6Z7AVBh6HwzG0Bi2u+cnBzy8/OjLqPWCgsLY1lX3KndklNQUEBxcbHaLEmhjrPVqxtzzTXdWL++KSee+A09e75HqnzrUvH/Zp2MxWdmWUB3IA+4zzm307lycnNzXRwfUozzQ21xpnZLTumDugsXLgxdSkoJcZwVFsLxx8OCBXDssfDaa9C0ab2WUCtx/r9pZvOdc7nl36+TXnzOuWLn3FtAB2BQXXyGiEh9KSqCCy/04bT//vDCC6kVTqmqrruZN0T3oEQkhTkH11/vu5HvsQfMmQNt24auKjNEFlBmtqeZXWhmLc0sy8xOAS4C3ojqM0RE6tvvfgfjx0Pjxv7M6YADQleUOaLsJOHwl/MexgffCmCwc+7FCD9DRKTePP883Hijf/2nP8Fxx4WtJ9NEFlDOuVXACVHtT0QkpHfegX79/CW+u+/296CkfmmoIxGRcpYtgz59/ECwP/sZ3HJL6IoykwJKRKSMNWv81BmrVsFPfgITJmjqjFAUUCIiCVu3wrnnwgcfwKGH+oFgGzUKXVXmUkCJiODvNV11Ffz1r/C97/lu5a1bh64qsymgRESAO++Exx6D5s39vE777BO6IlFAiUjGmzoVRo6EBg3gqafgKM3BEAsKKBHJaPn5vqcewNixvveexIMCSkQy1vvvw9lnw7ZtMHgwXHdd6IqkLAWUiGSkb76B00+HggI480wYMyZ0RVKeAkpEMs6mTXDGGf6B3NxcePxxP3W7xIsCSkQySkkJXHqpH8po331h5kxo0SJ0VVIZBZSIZJSbb4Znn/XPOM2e7Z95knhSQIlIxpgwwd9ratjQh9Qhh4SuSHZFASUiGWHOHLj2Wv/6kUegR4+w9UjVFFAikvYWLoS+ff39p+HDoX//0BVJdSigRCStffaZ705eWAgXX+yHNJLUoIASkbS1bp0Ppy++gOOPh0cf1dQZqUQBJSJpqajIX9ZbtAg6d/bTtzdpEroqSYYCSkTSjnO+Q8Qrr0Dbtr6DxO67h65KkqWAEpG0M2YMTJzoz5heegl+8IPQFUlNKKBEJK08/TTcdJN/PXUqdO8eth6pOQWUiKSNt9/2wxgB3HcfnH9+2HqkdhRQIpIWPv7YDwC7ZQsMGAC//GXoiqS2FFAikvK++w569YLVq+HUU2HcOHUnTwcKKBFJaVu2wFlnwdKlcPjhMH26H2tPUp8CSkRSlnNwxRXw5pvQvr0fnXy33UJXJVFRQIlIyvrVr+CJJ/x8TrNmQYcOoSuSKCmgRCQlTZkCo0ZBgwYwYwYceWToiiRqCigRSTnz52dz1VX+9UMP+Q4Skn4UUCKSUt57D371q0MpKoKhQ2HQoNAVSV1RQIlIyvjqK3+2tGFDQ849F+6/P3RFUpcUUCKSEjZsgD59YMUKOPjgdUyd6u8/SfqK7NtrZk3MbLKZrTCz9Wa2wMxOi2r/IpK5iouhXz+YNw++/324667/0KxZ6KqkrkX5+0dDYCVwAtAaGAHMMLNOEX6GiGSgG2+EF1+E7Gz/rFObNttClyT1ILKAcs5tcM6NdM4td86VOOdmAcuAo6L6DBHJPA89BGPHQqNGftLBgw8OXZHUlzobEMTMcoDOwOJK1g0ABgDk5OSQn59fV2XUWGFhYSzriju1W3IKCgooLi5Wm+3E3Ll7MGLEoYDxy1++D3xNfr6Os5pIxTYz51z0OzVrBLwMfOycG7irbXNzc928efMir6G28vPzycvLC11GylG7JScvL4+CggIWLlwYupTYmT8fjj8eNm6EkSP9qBGldJwlL85tZmbznXO55d+PvA+MmTUApgJbgWuj3r+IpL9PP4XevX04XXYZ3H576IokhEgv8ZmZAZOBHKCXc053MkUkKWvXwumn+2eeTjwRHnlEU2dkqqjvQU0ADgZ6Ouc2RbxvEUlz27b5WXD/+1846CB49llo3Dh0VRJKlM9B7QsMBLoCX5lZYWLpF9VniEj6cs4PW/Tqq7DnnjBnDrRpE7oqCSmyMyjn3ApAJ+IiUiP33guTJ0PTpvDSS/6BXMlsGihERIJ76ikYNszfa3r8cfjhD0NXJHGggBKRoN56C/r396/HjIFzzglajsSIAkpEgvnwQzjzTNiyBa65BoYMCV2RxIkCSkSCWL3aT53x3Xf+zwceUHdy2ZECSkTq3ebNcNZZ8NFHfqr26dOhYZ0NvCapSgElIvWqpAR++lP4+9+hQweYNQtatgxdlcSRAkpE6tXw4b7XXqtWfuqM9u1DVyRxpYASkXrzhz/APfdAVhY8/TQcfnjoiiTOFFAiUi9efRWuvtq/Hj8eTjklbD0SfwooEalz//kPnHeen7r95pthwIDQFUkqUECJSJ364gs/Ovm6dXDBBXD33aErklShgBKROlNYCH36wMqV0L07TJkCDfRTR6pJh4qI1IniYrj4YvjXv+AHP4AXX4RmzUJXJalEASUikXMOBg+GmTNh99391Bnt2oWuSlKNAkpEIvfAA/DQQ36ywRdegM6dQ1ckqUgBJSKReuEFuOEG//qPf4Qf/zhoOZLCFFAiEpl//tPfd3IORo/2r0VqSgElIpFYvtz32Nu0Ca64wk9AKFIbCigRqbWCAj9lxtdfQ48e8PDDmjpDak8BJSK1snUrnHsuvP8+dOkCzzwDjRqFrkrSgQJKRGrMORg4EN54A773Pd+dPDs7dFWSLhRQIlJjd93lR4do3tw/87TvvqErknSigBKRGnn8cRgxwt9reuIJyM0NXZGkGwWUiCTt//7P99QD+N3v4Mwzw9Yj6UkBJSJJWbIEzjrLd474xS/g+utDVyTpSgElItW2apXvTr5mDZxxBvz2t6ErknSmgBKRatm0yYfSJ5/AUUf5+05ZWaGrknSmgBKRKpWUwGWXwT/+Afvs43vstWgRuipJdwooEanSrbf6B3B32w1mz4a99gpdkWQCBZSI7NLEiXD//dCwITz7LBx6aOiKJFMooERkp155BX7+c/964kTo2TNsPZJZFFAiUql//xvOP99P3X7bbdufexKpLwooEang88/h9NOhsBAuughGjQpdkWSiSAPKzK41s3lmtsXMpkS5bxGpH+vXQ+/ePqSOO87PiqupMySEhhHv7wtgNHAK0CzifYtIHSsqgr59YeFCOOAAP317kyahq5JMFWlAOeeeAzCzXKBDlPsWkbrlnB+66OWXYY89/NQZe+wRuirJZFGfQVWLmQ0ABgDk5OSQn58fooxdKiwsjGVdcad2S05BQQHFxcWxaLMZMzowYcL+NGpUwsiRC/nss3V89lnoqiqn4yx5qdhmQQLKOTcJmASQm5vr8vLyQpSxS/n5+cSxrrhTuyUnOzubgoKC4G327LN+mnaAadMacMEF3YLWUxUdZ8lLxTZTLz6RDPePf8All/hLfPfcAxdcELoiEU8BJZLBPvnEDwC7eTNcdRXcfHPoikS2i/QSn5k1TOwzC8gys6ZAkXOuKMrPEZHaW7PGP+u0ahWcfDKMG6fu5BIvUZ9BDQc2AbcAlyReD4/4M0SklrZsgXPOgQ8+gMMOg6efhkaNQlclsqOou5mPBEZGuU8RiZZz/nJefr4flXz2bD9KuUjc6B6USIa54w6YOtXP5zRrFnTsGLoikcopoEQyyGOP+YBq0ACmT4du8e5NLhlOASWSIf76V7jySv/6wQd9BwmROFNAiWSA99+Hs8+GbdtgyJDtczyJxJkCSiTNff019OoFa9f6kPr1r0NXJFI9CiiRNLZxo38Qd/lyOPpomDYNsrJCVyVSPQookTRVXOyHMHr3XejUCV56CZo3D12VSPUpoETS1E03wfPPQ+vW/lmnnJzQFYkkRwElkobGj4ff/taPDvHcc9ClS+iKRJKngBJJM7Nnw3XX+dePPAInnRS2HpGaUkCJpJEFC/yU7SUlcPvtcPnloSsSqTkFlEiaWLnSP3y7YYPvHDFyZOiKRGpHASWSBtat8+H05Zdwwgnwhz9o6gxJfQookRS3bZufBfc//4EDD/Q995o0CV2VSO0poERSmHN+2KI//xnatYM5c6BNm9BViURDASWSwu6/3/fUa9rUP4i7336hKxKJjgJKJEXNmAG33OLvNU2bBj/6UeiKRKKlgBJJQXPnwmWX+df33w/nnhu2HpG6oIASSTEffQRnnglbtsDVV8PQoaErEqkbCiiRFPLtt37qjNWr4bTT4Pe/V3dySV8KKJEUsWWLn8/pww/hiCP8lO0NG4auSqTuKKBEUoBzcMUV8OabsPfefry9Vq1CVyVStxRQIing9tvhiSegZUsfTnvvHboikbqngBKJuUcfhdGj/Uy4M2b4y3simUABJRJjr70GAwf61+PG+Y4RIplCASUSU4sX++ebiorgl7/cHlQimUIBJRJDX33lu5OvWwfnnQf33hu6IpH6p4ASiZkNG6B3b/j0Uz980WOPQQP9T5UMpMNeJEaKi+Hii2H+fD/w64svQrNmoasSCUMBJRIjQ4f6UcnbtPFTZ+y5Z+iKRMJRQInExIMPwgMPQKNGftLBAw8MXZFIWAookRh46SUYPNi/fvRRP227SKaLNKDMbHcze97MNpjZCjO7OMr9i6SjjRuzuOgiP5zRnXfCJZeErkgkHqIeanIcsBXIAboCs83s3865xRF/jkha2LIFli1rQVER9O8Pw4eHrkgkPsw5F82OzFoAa4BDnXNLE+9NBT53zt2ys69r1aqVO+qooyKpIUoFBQVkZ2eHLiPlqN2S8/e/L6SoCLKzu3L44Zo6o7p0nCUvzm32t7/9bb5zLrf8+1GeQXUGikvDKeHfQIWr6WY2ABgA0KhRIwoKCiIsIxrFxcWxrCvu1G7Vt2ZNY4qK/Ov27dexdm1J2IJSiI6z5KVim0UZUC2BteXeWwtUmBTAOTcJmASQm5vr5s2bF2EZ0cjPzycvLy90GSlH7VY9q1fDQQcB5NGhw0YWL343dEkpRcdZ8uLcZraTSwdRdpIoBHYr995uwPoIP0MkLYwa5WfHzc6GPfbYGrockViKMqCWAg3N7IAy7x0BqIOESBmffAITJvj7TfvvH7oakfiKLKCccxuA54A7zayFmR0LnAlMjeozRNLBbbfBtm1w6aXQokXoakTiK+oHda8BmgHfAE8Cg9TFXGS7d9+Fp56CJk38ZT4R2blIn4Nyzn0HnBXlPkXSRUkJ/OIX/vXgwbDPPkHLEYk9DXUkUk+mToV33oG99vKX+URk1xRQIvVg3Tq4+Wb/+v77oVWFhy9EpDwFlEg9GDUKvv4auneHfv1CVyOSGhRQInXsgw9g7Fjfrfz3v9dwRiLVpYASqUPOwZAhUFQEV14JMRx2UiS2FFAidWjGDHjlFWjdGu66K3Q1IqlFASVSR1avhuuu869//Wto1y5sPSKpRgElUkeGDIFVq+DEE/3lPRFJjgJKpA68/DJMmwZNm8KkSeoYIVITCiiRiK1fDwMH+tejRmlAWJGaUkCJROzWW2HlSt9jb/Dg0NWIpC4FlEiEXnkFxo2Dhg1h8mT/p4jUjAJKJCLffAP9+/vXd9wBRxwRtByRlKeAEomAc/Czn/nhjE44Yfu4eyJScwookQiMHw+zZvkp3KdOhays0BWJpD4FlEgtLV4MN97oXz/yCHTsGLYekXShgBKphcJC6NsXNm+GK66A884LXZFI+lBAidRQ6X2nxYvhoIPggQdCVySSXhRQIjU0ZowfDLZVK3jhBWjZMnRFIulFASVSA6+9Brfc4l8/9hgceGDYekTSkQJKJEkrVsCFF0JJCdx2G5x1VuiKRNKTAkokCevWwRlnwLffwqmn+gdyRaRuKKBEqmnbNt9Lb9Ei6NwZHn9czzuJ1CUFlEg1OOdHKH/1VT/x4Msvw+67h65KJL0poESqYdQo+OMfoVkzP2LEfvuFrkgk/SmgRKoweTL86lfQoAE89RQcfXToikQygwJKZBeeeAKuusq/fvBB30FCROqHAkpkJ555Bi67zN9/uusu+PnPQ1ckklkUUCKVmDkTLroIiothxAgYNix0RSKZRwElUs7s2b47eVGRH6VczzqJhKGAEinjySf9yBBbt8K118L994NZ6KpEMpMCSiTh4YehXz9/5nTTTb5ThMJJJBwFlAhw770waJDvEHHPPXDffQonkdAiCSgzu9bM5pnZFjObEsU+RepDURFcdx3ceqsPpPHjt49SLiJhNYxoP18Ao4FTgGYR7VOkTq1d62fD/fOfoXFj+NOf/CjlIhIPkQSUc+45ADPLBTpEsU+RurRsGfTuDe+958fWe/55OPbY0FWJSFlRnUElxcwGAAMAcnJyyM/PD1HGLhUWFsayrrhLhXabPz+b0aO7UFDQmH333cA99/yHbds2E6LsgoICiouLY99mcZMKx1ncpGKbBQko59wkYBJAbm6uy8vLC1HGLuXn5xPHuuIuzu1WUgJ33w233+47Q5xyCkyf3oLWrX8UrKbs7GwKCgpi22ZxFefjLK5Ssc2q7CRhZvlm5nayvFUfRYrU1urVcPrpflQI8CE1eza0bh22LhHZuSrPoJxzefVQh0ideeMNuPxy+OwzP4fT44/72XBFJN6i6mbe0MyaAllAlpk1NbMglw9FSm3aBEOGQI8ePpx++ENYsEDhJJIqonpQdziwCbgFuCTxenhE+xZJ2vz5cNRRMHasn5Z95Eh4803YZ5/QlYlIdUXVzXwkMDKKfYnURmGhD6OxY/1I5AcdBFOnQm5u6MpEJFka6kjSxsyZ0KUL/OY3vpfe4MHwr38pnERSle4TScr76CM/LcaLL/q/d+sGkyb5S3wikrp0BiUpa80auOEGf9b04ovQsqW/tPfOOwonkXSgMyhJOZs2wcSJMHo0fPutH+T1pz/1f2/fPnR1IhIVBZSkjC1bYPJkuOsu+OIL/94JJ8Bvf+sv64lIelFASext3AhTpvg5mj791L/XtSvceacf8FXzNomkJwWUxNa33/r5mR580A9VBHDIIXDHHXD22dBAd1BF0poCSmJnwQI//fq0af7sCXxX8Ztv9sGUlRW2PhGpHwooiYWNG2HGDB9M77yz/f1TT4WbboK8PF3KE8k0CigJxjn/IO3UqX4224IC/352th/cdeBAOPjgkBWKSEgKKKl3S5fCk0/CE0/416WOPhquvtpPw968ebj6RCQeFFBSLz780D9MO306zJu3/f099/SBdPnlerhWRHakgJI6UVQEc+f68fFmzoQlS7ava9UKzjkHLr4YTjoJGuooFJFK6EeDRGbZMpg9ey8mToS//AW++277uuxs6NXL98I7/XRo1ixYmSKSIhRQUmMrV/o5ll5/3c9au3w5wIH/W3/AAdCnj1+OPRYaNQpVqYikIgWUVMuWLf75pLff9svcufD55ztu06YNHHroKvr2bUfPnnDggZXvS0SkOhRQUsHGjbBokQ+k0mXRIti6dcftWreG7t39faQePeCII+DNNxeTl5cXpG4RSS8KqAy2aZPvvPD++3557z2/LFkCJSUVt+/SxQdS6XLQQRpuSETqjgIqzW3e7O8NLVsGn3zilw8+8IG0fLl/WLa8rCw47DA48ki/dOvmz45at67v6kUkkymgUlhJiR9E9fPPd1w+/XR7GJW/T1RWVpbvyHDwwTsuXbqol52IhKeAihnn/JA/q1btuKxeDd98s2MQffEFbNu26/1lZcE++8B++21fSkNp//2hceN6+WeJiCRNAVUHSkpg/XofNAUFsHbt9teVLWvW+KklSoOoqKj6n9WmDey9945Lx47bw6hjRz0IKyKpKWN+dDnne6Ft3uy7TG/evH2p7O8LFuTw0Uf+7xs2QGFh9f/ctKl2tbZqBe3aVb60b789iNq315h1IpK+ggfUl1/CiBH+rGHbtp3/uat1O9umNJBKQyc5tRtGu0ULf3aTnV310ro1tG3rl3btoEmTWn20iEhaMFdZN676LMBaOSg/SugFwDXARqBXJV/VP7GsBs6rZP0goC+wEri0zGf5btEtWgyldes+NGiwhFWrBtKgATssXboMp0mTw2jZ8kv++c/BZGX597Oy/HLhhXdz5JHHsGLFXB57bFiF9Q88MJZu3bry2muvMXr06ArVTZw4kQMPPJCZM2fym9/8psL6qVOn0rFjR6ZPn86ECRMqrH/mmWdo27YtU6ZMYcqUKRXWz5kzh+bNmzN+/HhmzJhRYX1+fj4AY8aMYdasWTusa9asGS+//DIAo0aN4vXXX99h/R577MGzzz4LwK233srbb7+9w/pGjRrx6quvAjB48GAWLly4w/rOnTszadIkAAYMGMDSssOZA127dmXs2LEAXHLJJXz22Wc7rO/evTv33HMPAOeeey7ffvvtDut79OjBiBEjADjttNPYVO50tnfv3tx4440AlT6vdcEFF3DNNdewceNGevWqeOz179+f/v37s3r1as47r+KxN2jQIPr27cvKlSu59NJLK6wfOnQoffr0YcmSJQwcOJCFCxdSVFREbm4uAMOHD6dnz54sXLiQwYMHV/j6u+++m2OOOYa5c+cybNiwCuvHjh1L167pf+z169ePz8v1AOrQoQPTpk0DdOxVduydfPLJDBs27H/HXnkhj72//e1v851zueW/JvgZVOPG/lKV2falWzf/4GdJiZ/uu+w6Mzj5ZD+eW2EhjBxZcf0ll8BZZ/l7Otdfvz14Sg0d6offWbLEzzlU3vDh0LDh+2RnZ1PJ94lTT4VjjvGjKbzwQsX1ejZIRKT2gp9B5ebmunll51+Iifz8fI2IUANqt+Tk5eVRUFBQ4bd92TUdZ8mLc5uZWaVnUPpdX0REYkkBJSIisaSAEhGRWFJAiYhILCmgREQklmodUGbWxMwmm9kKM1tvZgvM7LQoihMRkcwVxRlUQ/wTsScArYERwAwz6xTBvkVEJEPV+kFd59wGYGSZt2aZ2TL88BDLa7t/ERHJTJGPJGFmOUBnYPEuthkADADIycn53/AncVJYWBjLuuJO7ZacgoICiouL1WZJ0nGWvFRss0hHkjCzRsDLwMfOuUoGEapII0mkF7VbcjSSRM3oOEtenNusxiNJmFm+mbmdLG+V2a4BMBXYClwbafUiIpJxqrzE55zLq2obMzNgMpAD9HLOVTHPq4iIyK5FdQ9qAn4CpZ7OuVpO1yciIhLNc1D7AgOBrsBXZlaYWPrVdt8iIpK5ouhmvgKwCGoRERH5Hw11JCIisaSAEhGRWAo+o66ZrQJWBC2icm2B1aGLSEFqt+SpzZKnNktenNtsX+dcu/JvBg+ouDKzeZU9OCa7pnZLntoseWqz5KVim+kSn4iIxJICSkREYkkBtXOTQheQotRuyVObJU9tlryUazPdgxIRkVjSGZSIiMSSAkpERGJJASUiIrGkgKomMzvAzDab2bTQtcSZmTUxs8lmtsLM1pvZAjM7LXRdcWRmu5vZ82a2IdFeF4euKc50bNVOKv4MU0BV3zjgn6GLSAENgZXACUBrYAQww8w6hSwqpsbhJ/jMAfoBE8zskLAlxZqOrdpJuZ9hCqhqMLMLgQLg9cClxJ5zboNzbqRzbrlzrsQ5NwtYBhwVurY4MbMWwLnACOdcoXPuLeAl4NKwlcWXjq2aS9WfYQqoKpjZbsCdwNDQtaQiM8sBOgOLQ9cSM52BYufc0jLv/RvQGVQ16diqnlT+GaaAqtooYLJzbmXoQlKNmTUCHgf+5Jz7IHQ9MdMSWFvuvbVAqwC1pBwdW0lJ2Z9hGR1QZpZvZm4ny1tm1hXoCfwucKmxUVWbldmuATAVf4/l2mAFx1chsFu593YD1geoJaXo2Kq+VP8ZVusZdVOZcy5vV+vNbDDQCfjUzMD/1ptlZl2cc93qur44qqrNAMw31mT8zf9ezrltdV1XCloKNDSzA5xzHybeOwJdrtolHVtJyyOFf4ZpqKNdMLPm7Phb7o34b/Yg59yqIEWlADN7GOgK9HTOFQYuJ7bM7CnAAVfi22sOcIxzTiG1Ezq2kpPqP8My+gyqKs65jcDG0r+bWSGwORW+saGY2b7AQGAL8FXitzaAgc65x4MVFk/XAI8C3wDf4n9oKJx2QsdW8lL9Z5jOoEREJJYyupOEiIjElwJKRERiSQElIiKxpIASEZFYUkCJiEgsKaBERCSWFFAiIhJLCigREYml/weXNl1Ng1Jz8QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, selu(z), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [-1.758, -1.758], 'k--')\n",
    "plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
    "plt.grid(True)\n",
    "plt.title(\"Hàm kích hoạt SELU\", fontsize=14)\n",
    "plt.axis([-5, 5, -2.2, 3.2])\n",
    "\n",
    "save_fig(\"selu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mặc định, siêu tham số SELU (`scale` và `alpha`) được điều chỉnh sao cho đầu ra trung bình của mỗi nơ-ron gần bằng 0 và độ lệch chuẩn gần bằng 1 (giả sử đầu vào được chuẩn hóa với giá trị trung bình bằng 0 và độ lệch chuẩn 1 nữa). Sử dụng hàm kích hoạt này, ngay cả mạng nơ-ron sâu 1.000 tầng cũng duy trì giá trị trung bình gần bằng 0 và độ lệch chuẩn 1 trên tất cả các tầng, tránh được vấn đề độ dốc bùng nổ/tiêu biến:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer 0: mean -0.00, std deviation 1.00\n",
      "Layer 100: mean 0.02, std deviation 0.96\n",
      "Layer 200: mean 0.01, std deviation 0.90\n",
      "Layer 300: mean -0.02, std deviation 0.92\n",
      "Layer 400: mean 0.05, std deviation 0.89\n",
      "Layer 500: mean 0.01, std deviation 0.93\n",
      "Layer 600: mean 0.02, std deviation 0.92\n",
      "Layer 700: mean -0.02, std deviation 0.90\n",
      "Layer 800: mean 0.05, std deviation 0.83\n",
      "Layer 900: mean 0.02, std deviation 1.00\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "Z = np.random.normal(size=(500, 100)) # standardized inputs\n",
    "for layer in range(1000):\n",
    "    W = np.random.normal(size=(100, 100), scale=np.sqrt(1 / 100)) # LeCun initialization\n",
    "    Z = selu(np.dot(Z, W))\n",
    "    means = np.mean(Z, axis=0).mean()\n",
    "    stds = np.std(Z, axis=0).mean()\n",
    "    if layer % 100 == 0:\n",
    "        print(\"Layer {}: mean {:.2f}, std deviation {:.2f}\".format(layer, means, stds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sử dụng SELU khá dễ:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7ff190832b10>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"selu\",\n",
    "                   kernel_initializer=\"lecun_normal\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hãy tạo một mạng nơ-ron cho Fashion MNIST với 100 tầng ẩn, sử dụng hàm kích hoạt SELU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"selu\",\n",
    "                             kernel_initializer=\"lecun_normal\"))\n",
    "for layer in range(99):\n",
    "    model.add(keras.layers.Dense(100, activation=\"selu\",\n",
    "                                 kernel_initializer=\"lecun_normal\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bây giờ hãy huấn luyện nó. Đừng quên chia tỷ lệ đầu vào thành giá trị trung bình 0 và độ lệch chuẩn 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "pixel_means = X_train.mean(axis=0, keepdims=True)\n",
    "pixel_stds = X_train.std(axis=0, keepdims=True)\n",
    "X_train_scaled = (X_train - pixel_means) / pixel_stds\n",
    "X_valid_scaled = (X_valid - pixel_means) / pixel_stds\n",
    "X_test_scaled = (X_test - pixel_means) / pixel_stds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1719/1719 [==============================] - 12s 6ms/step - loss: 1.3556 - accuracy: 0.4808 - val_loss: 0.7711 - val_accuracy: 0.6858\n",
      "Epoch 2/5\n",
      "1719/1719 [==============================] - 9s 5ms/step - loss: 0.7537 - accuracy: 0.7235 - val_loss: 0.7534 - val_accuracy: 0.7384\n",
      "Epoch 3/5\n",
      "1719/1719 [==============================] - 9s 5ms/step - loss: 0.7451 - accuracy: 0.7357 - val_loss: 0.5943 - val_accuracy: 0.7834\n",
      "Epoch 4/5\n",
      "1719/1719 [==============================] - 9s 5ms/step - loss: 0.5699 - accuracy: 0.7906 - val_loss: 0.5434 - val_accuracy: 0.8066\n",
      "Epoch 5/5\n",
      "1719/1719 [==============================] - 9s 5ms/step - loss: 0.5569 - accuracy: 0.8051 - val_loss: 0.4907 - val_accuracy: 0.8218\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bây giờ hãy xem điều gì sẽ xảy ra nếu chúng ta sử dụng hàm kích hoạt ReLU thay thế:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"relu\", kernel_initializer=\"he_normal\"))\n",
    "for layer in range(99):\n",
    "    model.add(keras.layers.Dense(100, activation=\"relu\", kernel_initializer=\"he_normal\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1719/1719 [==============================] - 11s 5ms/step - loss: 2.0460 - accuracy: 0.1919 - val_loss: 1.5971 - val_accuracy: 0.3048\n",
      "Epoch 2/5\n",
      "1719/1719 [==============================] - 8s 5ms/step - loss: 1.2654 - accuracy: 0.4591 - val_loss: 0.9156 - val_accuracy: 0.6372\n",
      "Epoch 3/5\n",
      "1719/1719 [==============================] - 8s 5ms/step - loss: 0.9312 - accuracy: 0.6169 - val_loss: 0.8928 - val_accuracy: 0.6246\n",
      "Epoch 4/5\n",
      "1719/1719 [==============================] - 8s 5ms/step - loss: 0.8188 - accuracy: 0.6710 - val_loss: 0.6914 - val_accuracy: 0.7396\n",
      "Epoch 5/5\n",
      "1719/1719 [==============================] - 8s 5ms/step - loss: 0.7288 - accuracy: 0.7152 - val_loss: 0.6638 - val_accuracy: 0.7380\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Có vẻ không khả quan lắm, ta đang gặp phải vấn đề tiêu biến/bùng nổ gradient."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chuẩn hóa theo Batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(300, activation=\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_4 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization (BatchNo (None, 784)               3136      \n",
      "_________________________________________________________________\n",
      "dense_212 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 300)               1200      \n",
      "_________________________________________________________________\n",
      "dense_213 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "batch_normalization_2 (Batch (None, 100)               400       \n",
      "_________________________________________________________________\n",
      "dense_214 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 271,346\n",
      "Trainable params: 268,978\n",
      "Non-trainable params: 2,368\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('batch_normalization/gamma:0', True),\n",
       " ('batch_normalization/beta:0', True),\n",
       " ('batch_normalization/moving_mean:0', False),\n",
       " ('batch_normalization/moving_variance:0', False)]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bn1 = model.layers[1]\n",
    "[(var.name, var.trainable) for var in bn1.variables]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "#bn1.updates #deprecated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 1.2287 - accuracy: 0.5993 - val_loss: 0.5526 - val_accuracy: 0.8230\n",
      "Epoch 2/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5996 - accuracy: 0.7959 - val_loss: 0.4725 - val_accuracy: 0.8468\n",
      "Epoch 3/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5312 - accuracy: 0.8168 - val_loss: 0.4375 - val_accuracy: 0.8558\n",
      "Epoch 4/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4884 - accuracy: 0.8294 - val_loss: 0.4153 - val_accuracy: 0.8596\n",
      "Epoch 5/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4717 - accuracy: 0.8343 - val_loss: 0.3997 - val_accuracy: 0.8640\n",
      "Epoch 6/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4420 - accuracy: 0.8461 - val_loss: 0.3867 - val_accuracy: 0.8694\n",
      "Epoch 7/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4285 - accuracy: 0.8496 - val_loss: 0.3763 - val_accuracy: 0.8710\n",
      "Epoch 8/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4086 - accuracy: 0.8552 - val_loss: 0.3711 - val_accuracy: 0.8740\n",
      "Epoch 9/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4079 - accuracy: 0.8566 - val_loss: 0.3631 - val_accuracy: 0.8752\n",
      "Epoch 10/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3903 - accuracy: 0.8617 - val_loss: 0.3573 - val_accuracy: 0.8750\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Đôi khi áp dụng BN trước hàm kích hoạt cho kết quả tốt hơn (có một cuộc tranh luận về chủ đề này). Ngoài ra, tầng trước tầng `BatchNormalization` không cần phải có các hệ số điều chỉnh. Tầng `BatchNormalization` cũng có một số hệ số, do đó có thể dẫn đến lãng phí các tham số. Để tránh điều này ta có thể đặt `use_bias=False` khi tạo các tầng đó:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(300, use_bias=False),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Activation(\"relu\"),\n",
    "    keras.layers.Dense(100, use_bias=False),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Activation(\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 1.3677 - accuracy: 0.5604 - val_loss: 0.6767 - val_accuracy: 0.7812\n",
      "Epoch 2/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.7136 - accuracy: 0.7702 - val_loss: 0.5566 - val_accuracy: 0.8184\n",
      "Epoch 3/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.6123 - accuracy: 0.7990 - val_loss: 0.5007 - val_accuracy: 0.8360\n",
      "Epoch 4/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5547 - accuracy: 0.8148 - val_loss: 0.4666 - val_accuracy: 0.8448\n",
      "Epoch 5/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5255 - accuracy: 0.8230 - val_loss: 0.4434 - val_accuracy: 0.8534\n",
      "Epoch 6/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4947 - accuracy: 0.8328 - val_loss: 0.4263 - val_accuracy: 0.8550\n",
      "Epoch 7/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4736 - accuracy: 0.8385 - val_loss: 0.4130 - val_accuracy: 0.8566\n",
      "Epoch 8/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4550 - accuracy: 0.8446 - val_loss: 0.4035 - val_accuracy: 0.8612\n",
      "Epoch 9/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4495 - accuracy: 0.8440 - val_loss: 0.3943 - val_accuracy: 0.8638\n",
      "Epoch 10/10\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4333 - accuracy: 0.8494 - val_loss: 0.3875 - val_accuracy: 0.8660\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gọt Gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tất cả các trình tối ưu hóa Keras đều chấp nhận các đối số `clipnorm` hoặc `clipvalue`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(clipvalue=1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(clipnorm=1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tái sử dụng các Tầng Tiền huấn luyện"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tái sử dụng một mô hình Keras"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hãy chia tập huấn luyện thời trang MNIST thành hai phần:\n",
    "* `X_train_A`: tất cả hình ảnh của tất cả các mặt hàng trừ xăng-đan và áo (lớp 5 và lớp 6).\n",
    "* `X_train_B`: một tập huấn luyện nhỏ hơn nhiều chỉ gồm 200 hình ảnh đầu tiên về xăng-đan hoặc áo.\n",
    "\n",
    "Tập kiểm định và tập kiểm tra cũng được phân chia theo cách này nhưng không hạn chế số lượng hình ảnh.\n",
    "\n",
    "Chúng ta sẽ huấn luyện một mô hình trên tập A (tác vụ phân loại có 8 lớp) và cố gắng sử dụng lại nó để giải quyết tập B (phân loại nhị phân). Ta hy vọng sẽ chuyển một chút kiến thức từ tác vụ A sang tác vụ B, vì các lớp trong nhóm A (giày thể thao, giày cổ cao, áo khoác, áo phông, v.v.) có phần giống với các lớp trong nhóm B (xăng-đan và áo sơ mi). \n",
    "\n",
    "Tuy nhiên, vì chúng ta đang sử dụng các lớp `Dense`, nên chỉ các khuôn mẫu xảy ra ở cùng một vị trí thì mới có thể được sử dụng lại (ngược lại, các tầng tích chập sẽ chuyển kiến thức tốt hơn nhiều, vì các khuôn mẫu đã học có thể được phát hiện ở bất kỳ đâu trên hình ảnh, như chúng ta sẽ thấy trong chương CNN)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_dataset(X, y):\n",
    "    y_5_or_6 = (y == 5) | (y == 6) # sandals or shirts\n",
    "    y_A = y[~y_5_or_6]\n",
    "    y_A[y_A > 6] -= 2 # class indices 7, 8, 9 should be moved to 5, 6, 7\n",
    "    y_B = (y[y_5_or_6] == 6).astype(np.float32) # binary classification task: is it a shirt (class 6)?\n",
    "    return ((X[~y_5_or_6], y_A),\n",
    "            (X[y_5_or_6], y_B))\n",
    "\n",
    "(X_train_A, y_train_A), (X_train_B, y_train_B) = split_dataset(X_train, y_train)\n",
    "(X_valid_A, y_valid_A), (X_valid_B, y_valid_B) = split_dataset(X_valid, y_valid)\n",
    "(X_test_A, y_test_A), (X_test_B, y_test_B) = split_dataset(X_test, y_test)\n",
    "X_train_B = X_train_B[:200]\n",
    "y_train_B = y_train_B[:200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(43986, 28, 28)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_A.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200, 28, 28)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_B.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 0, 5, 7, 7, 7, 4, 4, 3, 4, 0, 1, 6, 3, 4, 3, 2, 6, 5, 3, 4, 5,\n",
       "       1, 3, 4, 2, 0, 6, 7, 1], dtype=uint8)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_A[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 0., 0., 0., 0., 1., 1., 1., 0., 0., 1., 1., 0., 0., 0., 0.,\n",
       "       0., 0., 1., 1., 0., 0., 1., 1., 0., 1., 1., 1., 1.], dtype=float32)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_B[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A = keras.models.Sequential()\n",
    "model_A.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "for n_hidden in (300, 100, 50, 50, 50):\n",
    "    model_A.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
    "model_A.add(keras.layers.Dense(8, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "                optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "                metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1375/1375 [==============================] - 2s 1ms/step - loss: 0.9249 - accuracy: 0.6994 - val_loss: 0.3896 - val_accuracy: 0.8662\n",
      "Epoch 2/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.3651 - accuracy: 0.8745 - val_loss: 0.3288 - val_accuracy: 0.8827\n",
      "Epoch 3/20\n",
      "1375/1375 [==============================] - 1s 989us/step - loss: 0.3182 - accuracy: 0.8897 - val_loss: 0.3013 - val_accuracy: 0.8991\n",
      "Epoch 4/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.3048 - accuracy: 0.8954 - val_loss: 0.2896 - val_accuracy: 0.9021\n",
      "Epoch 5/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2804 - accuracy: 0.9029 - val_loss: 0.2773 - val_accuracy: 0.9061\n",
      "Epoch 6/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2701 - accuracy: 0.9075 - val_loss: 0.2735 - val_accuracy: 0.9066\n",
      "Epoch 7/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2627 - accuracy: 0.9093 - val_loss: 0.2721 - val_accuracy: 0.9081\n",
      "Epoch 8/20\n",
      "1375/1375 [==============================] - 1s 997us/step - loss: 0.2609 - accuracy: 0.9122 - val_loss: 0.2589 - val_accuracy: 0.9141\n",
      "Epoch 9/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2558 - accuracy: 0.9110 - val_loss: 0.2562 - val_accuracy: 0.9136\n",
      "Epoch 10/20\n",
      "1375/1375 [==============================] - 1s 997us/step - loss: 0.2512 - accuracy: 0.9138 - val_loss: 0.2544 - val_accuracy: 0.9160\n",
      "Epoch 11/20\n",
      "1375/1375 [==============================] - 1s 1000us/step - loss: 0.2431 - accuracy: 0.9170 - val_loss: 0.2495 - val_accuracy: 0.9153\n",
      "Epoch 12/20\n",
      "1375/1375 [==============================] - 1s 995us/step - loss: 0.2422 - accuracy: 0.9168 - val_loss: 0.2515 - val_accuracy: 0.9126\n",
      "Epoch 13/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2360 - accuracy: 0.9181 - val_loss: 0.2446 - val_accuracy: 0.9160\n",
      "Epoch 14/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2266 - accuracy: 0.9232 - val_loss: 0.2415 - val_accuracy: 0.9178\n",
      "Epoch 15/20\n",
      "1375/1375 [==============================] - 1s 988us/step - loss: 0.2225 - accuracy: 0.9239 - val_loss: 0.2447 - val_accuracy: 0.9195\n",
      "Epoch 16/20\n",
      "1375/1375 [==============================] - 1s 995us/step - loss: 0.2261 - accuracy: 0.9216 - val_loss: 0.2384 - val_accuracy: 0.9198\n",
      "Epoch 17/20\n",
      "1375/1375 [==============================] - 1s 1ms/step - loss: 0.2191 - accuracy: 0.9251 - val_loss: 0.2412 - val_accuracy: 0.9175\n",
      "Epoch 18/20\n",
      "1375/1375 [==============================] - 1s 991us/step - loss: 0.2171 - accuracy: 0.9254 - val_loss: 0.2429 - val_accuracy: 0.9158\n",
      "Epoch 19/20\n",
      "1375/1375 [==============================] - 1s 992us/step - loss: 0.2180 - accuracy: 0.9252 - val_loss: 0.2330 - val_accuracy: 0.9205\n",
      "Epoch 20/20\n",
      "1375/1375 [==============================] - 1s 994us/step - loss: 0.2112 - accuracy: 0.9274 - val_loss: 0.2333 - val_accuracy: 0.9200\n"
     ]
    }
   ],
   "source": [
    "history = model_A.fit(X_train_A, y_train_A, epochs=20,\n",
    "                    validation_data=(X_valid_A, y_valid_A))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A.save(\"my_model_A.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_B = keras.models.Sequential()\n",
    "model_B.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "for n_hidden in (300, 100, 50, 50, 50):\n",
    "    model_B.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
    "model_B.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_B.compile(loss=\"binary_crossentropy\",\n",
    "                optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "                metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "7/7 [==============================] - 0s 29ms/step - loss: 1.0360 - accuracy: 0.4975 - val_loss: 0.6314 - val_accuracy: 0.6004\n",
      "Epoch 2/20\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.5883 - accuracy: 0.6971 - val_loss: 0.4784 - val_accuracy: 0.8529\n",
      "Epoch 3/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.4380 - accuracy: 0.8854 - val_loss: 0.4102 - val_accuracy: 0.8945\n",
      "Epoch 4/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.4021 - accuracy: 0.8712 - val_loss: 0.3647 - val_accuracy: 0.9178\n",
      "Epoch 5/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.3361 - accuracy: 0.9348 - val_loss: 0.3300 - val_accuracy: 0.9320\n",
      "Epoch 6/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.3113 - accuracy: 0.9233 - val_loss: 0.3019 - val_accuracy: 0.9402\n",
      "Epoch 7/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.2817 - accuracy: 0.9299 - val_loss: 0.2804 - val_accuracy: 0.9422\n",
      "Epoch 8/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.2632 - accuracy: 0.9379 - val_loss: 0.2606 - val_accuracy: 0.9473\n",
      "Epoch 9/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.2373 - accuracy: 0.9481 - val_loss: 0.2428 - val_accuracy: 0.9523\n",
      "Epoch 10/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.2229 - accuracy: 0.9657 - val_loss: 0.2281 - val_accuracy: 0.9544\n",
      "Epoch 11/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.2155 - accuracy: 0.9590 - val_loss: 0.2150 - val_accuracy: 0.9584\n",
      "Epoch 12/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.1834 - accuracy: 0.9738 - val_loss: 0.2036 - val_accuracy: 0.9584\n",
      "Epoch 13/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1671 - accuracy: 0.9828 - val_loss: 0.1931 - val_accuracy: 0.9615\n",
      "Epoch 14/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1527 - accuracy: 0.9915 - val_loss: 0.1838 - val_accuracy: 0.9635\n",
      "Epoch 15/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1595 - accuracy: 0.9904 - val_loss: 0.1746 - val_accuracy: 0.9686\n",
      "Epoch 16/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.1473 - accuracy: 0.9937 - val_loss: 0.1674 - val_accuracy: 0.9686\n",
      "Epoch 17/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1412 - accuracy: 0.9944 - val_loss: 0.1604 - val_accuracy: 0.9706\n",
      "Epoch 18/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.1242 - accuracy: 0.9931 - val_loss: 0.1539 - val_accuracy: 0.9706\n",
      "Epoch 19/20\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.1224 - accuracy: 0.9931 - val_loss: 0.1482 - val_accuracy: 0.9716\n",
      "Epoch 20/20\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.1096 - accuracy: 0.9912 - val_loss: 0.1431 - val_accuracy: 0.9716\n"
     ]
    }
   ],
   "source": [
    "history = model_B.fit(X_train_B, y_train_B, epochs=20,\n",
    "                      validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_7\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_5 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_28 (Dense)             (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "dense_29 (Dense)             (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "dense_30 (Dense)             (None, 50)                5050      \n",
      "_________________________________________________________________\n",
      "dense_31 (Dense)             (None, 50)                2550      \n",
      "_________________________________________________________________\n",
      "dense_32 (Dense)             (None, 50)                2550      \n",
      "_________________________________________________________________\n",
      "dense_33 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 275,801\n",
      "Trainable params: 275,801\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model_B.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A = keras.models.load_model(\"my_model_A.h5\")\n",
    "model_B_on_A = keras.models.Sequential(model_A.layers[:-1])\n",
    "model_B_on_A.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lưu ý rằng `model_B_on_A` và `model_A` hiện tại chia sẻ các tầng với nhau, vì vậy khi chúng ta huấn luyện một tầng, nó sẽ cập nhật cả hai mô hình. Nếu muốn tránh điều này, ta cần xây dựng `model_B_on_A` trên một *bản sao* của `model_A`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A_clone = keras.models.clone_model(model_A)\n",
    "model_A_clone.set_weights(model_A.get_weights())\n",
    "model_B_on_A = keras.models.Sequential(model_A_clone.layers[:-1])\n",
    "model_B_on_A.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "for layer in model_B_on_A.layers[:-1]:\n",
    "    layer.trainable = False\n",
    "\n",
    "model_B_on_A.compile(loss=\"binary_crossentropy\",\n",
    "                     optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "                     metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "7/7 [==============================] - 0s 29ms/step - loss: 0.2575 - accuracy: 0.9487 - val_loss: 0.2797 - val_accuracy: 0.9270\n",
      "Epoch 2/4\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.2566 - accuracy: 0.9371 - val_loss: 0.2701 - val_accuracy: 0.9300\n",
      "Epoch 3/4\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.2473 - accuracy: 0.9332 - val_loss: 0.2613 - val_accuracy: 0.9341\n",
      "Epoch 4/4\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.2450 - accuracy: 0.9463 - val_loss: 0.2531 - val_accuracy: 0.9391\n",
      "Epoch 1/16\n",
      "7/7 [==============================] - 1s 29ms/step - loss: 0.2106 - accuracy: 0.9524 - val_loss: 0.2045 - val_accuracy: 0.9615\n",
      "Epoch 2/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.1738 - accuracy: 0.9526 - val_loss: 0.1719 - val_accuracy: 0.9706\n",
      "Epoch 3/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.1451 - accuracy: 0.9660 - val_loss: 0.1491 - val_accuracy: 0.9807\n",
      "Epoch 4/16\n",
      "7/7 [==============================] - 0s 9ms/step - loss: 0.1242 - accuracy: 0.9717 - val_loss: 0.1325 - val_accuracy: 0.9817\n",
      "Epoch 5/16\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.1078 - accuracy: 0.9855 - val_loss: 0.1200 - val_accuracy: 0.9848\n",
      "Epoch 6/16\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.1075 - accuracy: 0.9931 - val_loss: 0.1101 - val_accuracy: 0.9858\n",
      "Epoch 7/16\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.0893 - accuracy: 0.9950 - val_loss: 0.1020 - val_accuracy: 0.9858\n",
      "Epoch 8/16\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.0815 - accuracy: 0.9950 - val_loss: 0.0953 - val_accuracy: 0.9868\n",
      "Epoch 9/16\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.0640 - accuracy: 0.9973 - val_loss: 0.0892 - val_accuracy: 0.9868\n",
      "Epoch 10/16\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.0641 - accuracy: 0.9931 - val_loss: 0.0844 - val_accuracy: 0.9878\n",
      "Epoch 11/16\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.0609 - accuracy: 0.9931 - val_loss: 0.0800 - val_accuracy: 0.9888\n",
      "Epoch 12/16\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.0641 - accuracy: 1.0000 - val_loss: 0.0762 - val_accuracy: 0.9888\n",
      "Epoch 13/16\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.0478 - accuracy: 1.0000 - val_loss: 0.0728 - val_accuracy: 0.9888\n",
      "Epoch 14/16\n",
      "7/7 [==============================] - 0s 10ms/step - loss: 0.0444 - accuracy: 1.0000 - val_loss: 0.0700 - val_accuracy: 0.9878\n",
      "Epoch 15/16\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.0490 - accuracy: 1.0000 - val_loss: 0.0675 - val_accuracy: 0.9878\n",
      "Epoch 16/16\n",
      "7/7 [==============================] - 0s 11ms/step - loss: 0.0434 - accuracy: 1.0000 - val_loss: 0.0652 - val_accuracy: 0.9878\n"
     ]
    }
   ],
   "source": [
    "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=4,\n",
    "                           validation_data=(X_valid_B, y_valid_B))\n",
    "\n",
    "for layer in model_B_on_A.layers[:-1]:\n",
    "    layer.trainable = True\n",
    "\n",
    "model_B_on_A.compile(loss=\"binary_crossentropy\",\n",
    "                     optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "                     metrics=[\"accuracy\"])\n",
    "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=16,\n",
    "                           validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vậy kết quả cuối cùng là như thế nào?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63/63 [==============================] - 0s 714us/step - loss: 0.1408 - accuracy: 0.9705\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.1408407837152481, 0.9704999923706055]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_B.evaluate(X_test_B, y_test_B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63/63 [==============================] - 0s 751us/step - loss: 0.0562 - accuracy: 0.9940\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.0561506561934948, 0.9940000176429749]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_B_on_A.evaluate(X_test_B, y_test_B)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tuyệt vời! Chúng ta đã nhận được khá nhiều chuyển giao: tỷ lệ lỗi giảm xuống 4,9 lần!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.916666666666718"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(100 - 97.05) / (100 - 99.40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Các Bộ tối ưu nhanh hơn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tối ưu Momentum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tăng tốc Gradient Nesterov"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(learning_rate=0.001, momentum=0.9, nesterov=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AdaGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adagrad(learning_rate=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RMSProp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adam Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adamax Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adamax(learning_rate=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nadam Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Nadam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Định thời Tốc độ học"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Định thời Lũy thừa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```lr = lr0 / (1 + steps / s)**c```\n",
    "* Keras uses `c=1` and `s = 1 / decay`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(learning_rate=0.01, decay=1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5980 - accuracy: 0.7933 - val_loss: 0.4031 - val_accuracy: 0.8598\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 2s 954us/step - loss: 0.3829 - accuracy: 0.8636 - val_loss: 0.3714 - val_accuracy: 0.8720\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 2s 943us/step - loss: 0.3491 - accuracy: 0.8771 - val_loss: 0.3746 - val_accuracy: 0.8738\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 2s 954us/step - loss: 0.3277 - accuracy: 0.8814 - val_loss: 0.3502 - val_accuracy: 0.8798\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 2s 934us/step - loss: 0.3172 - accuracy: 0.8856 - val_loss: 0.3453 - val_accuracy: 0.8780\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 2s 919us/step - loss: 0.2922 - accuracy: 0.8940 - val_loss: 0.3419 - val_accuracy: 0.8820\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 2s 921us/step - loss: 0.2870 - accuracy: 0.8973 - val_loss: 0.3362 - val_accuracy: 0.8872\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 2s 925us/step - loss: 0.2720 - accuracy: 0.9032 - val_loss: 0.3415 - val_accuracy: 0.8830\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 2s 929us/step - loss: 0.2730 - accuracy: 0.9004 - val_loss: 0.3297 - val_accuracy: 0.8864\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 2s 928us/step - loss: 0.2585 - accuracy: 0.9068 - val_loss: 0.3269 - val_accuracy: 0.8888\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 2s 932us/step - loss: 0.2529 - accuracy: 0.9100 - val_loss: 0.3280 - val_accuracy: 0.8878\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 2s 954us/step - loss: 0.2485 - accuracy: 0.9101 - val_loss: 0.3343 - val_accuracy: 0.8822\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 2s 964us/step - loss: 0.2420 - accuracy: 0.9148 - val_loss: 0.3266 - val_accuracy: 0.8890\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2373 - accuracy: 0.9144 - val_loss: 0.3299 - val_accuracy: 0.8890\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2363 - accuracy: 0.9154 - val_loss: 0.3255 - val_accuracy: 0.8874\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2309 - accuracy: 0.9181 - val_loss: 0.3217 - val_accuracy: 0.8910\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2235 - accuracy: 0.9211 - val_loss: 0.3248 - val_accuracy: 0.8914\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2247 - accuracy: 0.9194 - val_loss: 0.3202 - val_accuracy: 0.8934\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2235 - accuracy: 0.9218 - val_loss: 0.3243 - val_accuracy: 0.8906\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2227 - accuracy: 0.9225 - val_loss: 0.3224 - val_accuracy: 0.8900\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2193 - accuracy: 0.9230 - val_loss: 0.3221 - val_accuracy: 0.8912\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2163 - accuracy: 0.9227 - val_loss: 0.3195 - val_accuracy: 0.8948\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 2s 997us/step - loss: 0.2127 - accuracy: 0.9252 - val_loss: 0.3208 - val_accuracy: 0.8908\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2076 - accuracy: 0.9273 - val_loss: 0.3226 - val_accuracy: 0.8902\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 2s 999us/step - loss: 0.2104 - accuracy: 0.9250 - val_loss: 0.3225 - val_accuracy: 0.8924\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAzyklEQVR4nO3deXxU1f3/8dcnCyEkbGEnEkBEZFFEq6JopWKLrbbyVbuqdS2trb+21qpYta6trVq/XbQLX6UuRVutoFZUXBAVrAtVEdkRBWRfA4EEQvj8/rg3dJhMkhvIzCSZ9/PxmAcz954593OvMZ+ce849x9wdERGRxpaV7gBERKRlUoIREZGkUIIREZGkUIIREZGkUIIREZGkUIIREZGkUIIRaUbM7EIzK0tS3R+a2U0N/M4nZvbT2j5LZlOCkWbHzB4wMw9flWa21MzuMrOCdMdWHzPra2Z/M7NPzWynma0ysylmNizdsTWSY4A/pjsIaRpy0h2AyH56CTgfyAVOAu4DCoDL0hlUNTPLdffK+G3Ai8BHwNeAlUAx8HmgKOVBJoG7r093DNJ0qAUjzdVOd1/j7ivc/RFgIjAGwMzyzOy3ZrbWzCrM7E0zO7H6i2b2lpldE/N5Ytga6h5+bmNmu8xsRPjZzOxqM/vIzMrNbI6ZnRfz/T7h979pZtPMrBz4boKYBwP9gB+4+xvuviz892Z3fzmmvnZm9iczWx3GP9/Mvh5bkZmNCm9pbTezV8ysb9z+L5vZf8Lvf2xmvzCzVjH7u5rZU+H5LDOzi+ODDc/pnLhtdd4CS3DLzM1srJk9Hsa6NPbahWWOM7N3w1jfM7Mvhd8bWdtxpHlQgpGWopygNQNwB/B14GJgGDAHeN7MeoT7pwOfi/nuycAGYGT4eQRQCbwdfr4NuAT4ATAIuB34i5mdHhfD7QS3hwYBTyaIcT2wBzjbzBLePTAzA54LY7oorOsnwK6YYnnAteH5HQ90AP4cU8dogoR7D0FSuxg4B/hlTB0PAIcApxIk5m8DfRLF1Ah+DjwFDAX+AUwws95hrIXAM8AC4GjgauDOJMUhqebueunVrF4Evxyfifl8LEGC+AfBbbJdwLdj9mcT3Ja6Lfz8RaCM4BZxf2Ab8AvgL+H+XwAvhu8LCJLXSXEx/BZ4NnzfB3Dgygix/wDYHh7/VeBWYHDM/s8TJKGBtXz/wvBYA2K2nRuec1b4+TXghrjvjQmPacChYR0jYvb3BqqAm2K2OXBOXD2fAD9twGcHbo/5nAPsAM4LP38X2ATkx5T5Vvi9ken+WdPrwF5qwUhzdZqZlZlZBfBvgl+q/4/gFlQuMLO6oLtXhWUGhZteJ2gFHEPQanmdoE9nZLh/JEErh/A7rQlaQGXVL4K+nn5xMc2qL2h3vxfoTvBLdAZwJvC+mZ0fFhkGrHb3+XVUs9PdF8Z8XhWec4fw89HAdXHxPkKQLLsDAwmSWHULDXdfFtaTDB/EHGc3QUuua7jpMOBDdy+PKf9WkuKQFFMnvzRXrwFjCW5lrfKwQz3mNliiacKDP6ndy8zsXYLbZIOBVwgSUG8z60+QeK4Ov1P9R9iXgeVx9VXGfd4eJXB33wY8DTxtZtcDUwlaMg8TtDDqszu+yrhYs4CbgccTfHd9xGNU1xtfNjdRwXrEXyfnv7Eaif9bSQugBCPN1Q53X5Jg+xKC20UnAksBzCyboK/ikZhy0wkSzEDgt+5eYWZvAdexb//LPGAn0NvdpzX2Sbi7m9kC4Khw07tADzMbWE8rpi7vAofVcn0ws/kEv+CPAd4It5UAPeOKrgd6xHyvW+znRjIf+LaZ5ce0Yo5t5GNImijBSIvi7tvN7E/Ar8xsA/AxcAXQjX2fz5gOXEnQ6ng3Ztt1wCvVLSJ332ZmdwF3hR3wrwGFwHBgj7uPjxqbmR1J0LJ4mCBx7SLozL8YeDQs9jLBLaInzOwKYBFBZ3yBuz8Z8VC3AM+Y2TLgMYIWzxDgWHe/2t0XmtnzBAMVxhL0Md0d/htrGvADM3uDoH/ml0BF1PONaCLBIIr/M7NfEiS5n4X71LJp5tQHIy3RNQS/WP8KvA8cAZzm7qtjyrxO8Avs9bCPBoJbZdn8t/+l2g3ATcBPgbkEz7KcTZC8GuJTglbVz4E3w9iuBO4i6D/C3fcQDEKYCfyN4C/83wGtalaXmLtPBU4naKG9Hb7Gse8tvgvD+KcB/yJo3X0SV9WVYbzTgX8SPGu0LmocEWMtI7j9OBh4j2AE2U3h7sZOZpJi5q4/EkSk6TCzM4HJQFd335DueGT/6RaZiKSVmV1A0FJaQXAr77fAv5Rcmr+U3iIzsyIzmxw+0bvMzL5VR9krzGyNmZWa2QQzy4vZd7mZzbJgLqcHEnx3lJktMLMd4VPOvZN0SiJy4LoR9EstBO4leND0vDq/Ic1CSm+RmdmjBEntEuBIYApwgrvPjSs3GngIOIVgbP5k4E13HxfuP4tgHP9ogge0Loz5bmeCh+ouJbi3fCvBQ3LDk3luIiKyr5QlGAtmut0MDHH3ReG2h4GV1YkjpuwjwCfu/rPw8yhgort3jyt3G3BQXIIZC1zo7ifEHHcDMMzdFyTr/EREZF+p7IM5FKiqTi6h2QTDNOMNJpi7KLZcNzPr5O4b6znO4LA8sHfY6kfh9n0STJiMxgJk5bc7Oqd91737+rTTADuAPXv2kJWlaxFP1yUxXZeaWvo1WbRo0QZ375JoXyoTTCFQGretFGgboWz1+7ZAfQmmkOABsXqPEz7DMB4gr0d/73HBbwEo7pDPzHGn1HOYzDB9+nRGjhyZ7jCaHF2XxHRdamrp1yR83iqhVKbVMqBd3LZ2BBMN1le2+n2isgdynBrycrK4avSAKEVFRKQOqUwwi4CccK6nakMJHlyLNzfcF1tubYTbYzW+G/bB9KvlOPswYEjPdowZVhzhMCIiUpeUJRh33w5MAm4xswILFnM6k2B4YryHgEvMbJCZdQSuJ5iiHQAzyzGz1gRPXWebWeuY9TUmA0PM7OywzM+BD+rr4O/TLouLRvRl9qelrN2qB4hFRA5Uqnuevg/kE0w38ShwmbvPNbOScFrxEgB3f55g0ahXgGXh68aYeq4nmDdpHMF4+fJwGx4s2Xo2wZoem4HjgG9ECe7CE/pQ5c7D/671lqKIiESU0if53X0T4bK2cduXE3TOx267m2ACvkT13MR/5ytKtP8lgnUmGqSkUxs+P7AbE99axuWnHELr3OyGViEiIqGWO3ZuP118Yl8276jkyfdWpjsUEZFmTQkmznF9ixjUox0TZn6MJgIVEdl/SjBxzIyLT+zLorVlzFwSZdCaiIgkogSTwJeH9qBzYSsmzGzoch8iIlJNCSaBvJxszhvem2kL1rF0fVm6wxERaZaUYGpx7nG9aZWdxQNvfJLuUEREmiUlmFp0aZvHV47syeOzPqV0R2W6wxERaXaUYOpw0Yg+lFdW8Y9Zy+svLCIi+1CCqcPgnu0ZfnARD76xjN1Ve9IdjohIs6IEU4+LR/Rl5ZZyXpi3Nt2hiIg0K0ow9Rg1sBslRW2YMENDlkVEGkIJph7ZWcaFJ/Rh1rLNzF6xJd3hiIg0G0owEXztmF60zcvhr3rwUkQkMiWYCArzcvjaMb145oPVWitGRCQiJZiILjyhD3u0VoyISGRKMBH1KmrD5wcFa8VUVFalOxwRkSZPCaYBLh4RrBUzWWvFiIjUSwmmAY7tW8Tgnu2YMENrxYiI1EcJpgHMjItH9GXxujJmLNmQ7nBERJo0JZgGOmNoDzoX5unBSxGReijBNFBeTjbnD+/NKwvX85HWihERqZUSzH44d3hJsFbMzE/SHYqISJOVk+4AmqPOhXkc2as9f3tzGX97cxk9O+Rz1egBjBlWnO7QRESaDCWY/fDkeyuZ/Wkp1ePIVm4p59pJcwCUZEREQrpFth/unLqQnbv3XR+mvLKKO6cuTFNEIiJNjxLMfli1pbxB20VEMpESzH7o2SG/QdtFRDKREsx+uGr0APJzs/fZlp1lXDV6QJoiEhFpetTJvx+qO/LvnLqQVVvKaZOXzfadVfTrUpjmyEREmg4lmP00Zljx3kSztaKSU+6azo1Pf8gTl52AmaU5OhGR9NMtskbQrnUuV592GO8u36KZlkVEQkowjeScow5iaK8O3P7cAsp27k53OCIiaacE00iysoybvzKY9dt28oeXF6c7HBGRtFOCaURH9urAV48+iAkzP9ZEmCKS8VKaYMysyMwmm9l2M1tmZt+qo+wVZrbGzErNbIKZ5UWtx8y+ZmbzzWybmc0zszFJPK19XH3aYbTOyeaWf83TomQiktFS3YK5F9gFdAPOBf5kZoPjC5nZaGAcMAroAxwM3BylHjMrBv4G/ARoB1wFPGJmXZNzSvvq0jaPH53an1cXrefl+etScUgRkSYpZQnGzAqAs4Eb3L3M3WcATwPnJyh+AXC/u891983ArcCFEes5CNji7s95YAqwHeiXxNPbN/gT+nBI10JueWYeFZVVqTqsiEiTksrnYA4Fqtx9Ucy22cDJCcoOBp6KK9fNzDoBJfXUMwuYb2ZfAaYAXwZ2Ah/EH8TMxgJjAbp06cL06dP347QS+5+SKu6cVcF1D73Ml/u1arR6U62srKxRr0tLoeuSmK5LTZl8TVKZYAqB0rhtpUDbCGWr37etrx53rzKzh4BHgNYEt9K+6u7b4w/i7uOB8QADBgzwkSNHNuB06jYSmFP+H55dtJ4rzz6u2c5TNn36dBrzurQUui6J6brUlMnXJJV9MGUEfSKx2gHbIpStfr+tvnrM7FTgDoLf8a0IWjb3mdmR+x/6/rnu9IHscef25xak+tAiImmXygSzCMgxs/4x24YCcxOUnRvuiy231t03RqjnSOA1d5/l7nvc/R3gLeDUxjmN6HoVteF7J/fjX7NX8ebSjak+vIhIWqUswYS3qCYBt5hZgZmNAM4EHk5Q/CHgEjMbZGYdgeuBByLW8w5wUnWLxcyGASeRoA8mFb53cj+KO+Rz09Nz2V21p/4viIi0EKkepvx9IB9YBzwKXObuc82sxMzKzKwEwN2fJ7jN9QqwLHzdWF894XdfBW4C/mlm24AngF+6+wspOL8a8ltlc/3pA1mwZhuPvL08HSGIiKRFSmdTdvdNwJgE25cTdN7HbrsbuLsh9cTsvwe45wBCbVSnDenOCf068ZsXFnHGET0pKmi+o8pERKLSVDEpYGbc9JXBlO3czV0vLEx3OCIiKaEEkyKHdmvLt4/vzaNvL+fDlfGjrEVEWh4tOJZCPz71UB57ZwVn/fENKqv20LNDPleNHrB34TIRkZZECSaFXlmwjl1Ve6isCibBXLmlnGsnzQFQkhGRFke3yFLozqkL9yaXauWVVdw5Vf0yItLyKMGk0Kot5Q3aLiLSnCnBpFBt85E113nKRETqogSTQleNHkB+bnaN7Wcdpf4XEWl5lGBSaMywYm4/63CKO+RjQI/2relckMtjs1awoWxnusMTEWlUGkWWYmOGFe8zYmzeqq38zx9n8uO/v8+DFx9LdpalMToRkcajFkyaDerZjlvOHMyMJRv4/cuL0x2OiEijUYJpAr72mV6cdVQxv5+2mNcXr093OCIijSJygjGzL5rZM2Y2z8x6hdsuNbNRyQsvM5gZt40ZQv+uhfz47++zprQi3SGJiBywSAnGzM4FHgMWA32B3HBXNnB1ckLLLG1a5fDHc4+mvLKKyx95l0qtHSMizVzUFszVwHfc/Qpgd8z2NwlWkJRGcEjXQm4/63BmLdvMXXq6X0SauagJpj/w7wTby4B2jReOnHlkMecNL+Evry3lxXlr0x2OiMh+i5pgVgGHJtj+WeCjxgtHAG44YxCHF7fnysfeZ8WmHekOR0Rkv0RNMOOB35vZiPBzLzO7gGBZ4z8lJbIMlpeTzb3fOgoHvj/xXXburkp3SCIiDRYpwbj7HcAk4EWgAHgF+DPwZ3e/N3nhZa6STm34zVeHMmdlKbc9Mz/d4YiINFjkYcrufh3QGTgWGA50cfcbkhWYwBcGd2fsZw/m4TeX8fTsVekOR0SkQSJNFWNmE4Afufs2YFbM9gLgD+5+cZLiy3hXjR7Au8s2c+Vj73PbM/NYv22nVsIUkWYhagvmAiDRnPL5wLcbLxyJl5udxRlDe1BZ5azbthPnvythPvneynSHJyJSqzoTjJkVmVknwICO4efqVxfgDEBjaZPs/177uMY2rYQpIk1dfbfINgAevuYl2O/AjY0dlOxLK2GKSHNUX4L5HEHrZRpwNrApZt8uYJm7q/c5yXp2yGdlgmSilTBFpCmrM8G4+6sAZtYXWOHumiArDa4aPYBrJ82hvHLf52GO69sxTRGJiNQv0igyd18GYGY9gRKgVdz+1xo/NKlWPVrszqkLWbWlnB4dWlPUphWT3lvFcQd34uvHlKQ5QhGRmqIOU+4JPEIwNYwT3DbzmCI1F5qXRhW/EubO3VV856H/MG7SHFrnZnPmkRqyLCJNS9Rhyr8FqoBBwA7gJOCrwHzgtKREJnXKy8nmL+cdzTF9ivjJY7N5Ye6adIckIrKPqAnmZOAad19A0HJZ7+6TgGuAW5MVnNQtv1U2Ey48hsOL23P5I+/x2iKthikiTUfUBJNPMGQZgpFkXcP384AjGjsoia4wL4cHLzqWfl0LGfvwLN5aujHdIYmIANETzALgsPD9+8D3zKw38ANAj5OnWfs2uTx8ybEUd8jnkgdn8f6KLekOSUQkcoL5HdA9fH8L8AVgKfB94GdJiEsaqHNhHhMvHU7HglwumPA281dvTXdIIpLhok7XP9HdHwjfvwv0AY4BStz98agHC6eYmWxm281smZl9q46yV5jZGjMrNbMJZpYXtR4za2NmfzSzDeH3M2IYdff2rXnk0uG0aZXN+fe/xZJ1ZekOSUQyWOTp+mO5+44w0Ww3s3EN+Oq9BDMAdAPOBf5kZoPjC5nZaGAcMIogmR0M3NyAesYDRcDA8N8rGhBjs9arqA1/u/Q4AM677y2tiCkiaVPvczBm1hk4DqgEXnb3KjPLJeh/uZbgGZhfRaingGC6mSHuXgbMMLOngfMJkkmsC4D73X1u+N1bgYnAuPrqMbMBwFeAg9y9+j7Rf+qLryXp16WQhy85jm+Mf5Mz751Bq+ws1m7VNP8iklp1JhgzOwGYArQnGJ78jpldCEwGcgmGKE+IeKxDgSp3XxSzbTbBEOh4g4Gn4sp1C2d2LqmnnuOAZcDNZnY+sBq4yd2fSHB+Y4GxAF26dGH69OkRT6V5OKXYmbxk997PK7eUc/Xj7zNv/jxO6JkbqY6ysrIWd10ag65LYrouNWXyNamvBXMrMBW4DbgY+DHwDEFH/8Pu7rV/tYZCoDRuWynQNkLZ6vdtI9RzEDAEeALoCRwPTDGzee6+z9rD7j6e4HYaAwYM8JEjRzbgdJq+696cBuzeZ9uuPTBleTY/+9bISHVMnz6dlnZdGoOuS2K6LjVl8jWprw9mKHCru38IXE/QirnW3R9qYHIBKAPaxW1rB2yLULb6/bYI9ZQT3M67zd13hRN2vkIw8i2jaJp/EUmn+hJMEbAego59gmli3tvPYy0Ccsysf8y2ocDcBGXnhvtiy611940R6vlgP+NrcWqbzr9jQauE20VEGlOUUWTVK1l2ImjBtItb2bIoyoHcfTswCbjFzArMbARwJvBwguIPAZeY2SAz60jQenogYj2vAcuBa80sJ9w/kuBWX0a5avQA8nP3nYfUDDZt38WEGR/T8EaoiEh0URLMPIJWzDqC/o93ws/rCaaPacgEWN8nmHZmHfAocJm7zzWzEjMrM7MSAHd/HriD4NbWsvB1Y331hN+tJEg4XyLom/k/4NvhPGoZZcywYm4/63CKO+RjQHGHfH591uGMHtyNW56Zx41Pz2V3lZb4EZHkiLKiZaNx903AmATblxMkr9htdwN3N6SemP1zCTr3M178NP8A5xzdi18/v4C/vLaU5Zt28IdvDqNt62ijykREooq0oqW0LFlZxrVfGkifzgVc/+SHfPXP/+b+C4+hWEswi0gj2q8n+aVl+OaxJTx40bGs3FLOmffMZLYmyRSRRqQEk+FO7N+ZSZedQOvcLL4+/t88/+HqdIckIi2EEozQv1tbnvzBCAb2aMdlE9/lL69+pBFmInLA6p2LTDJD58I8Hv3OcH76+Gxuf24B0xeuY9mmHazaUkHxm9M0h5mINJgSjOzVOjeb339jGLt2V/HCvHV7t6/cUs61k+YAKMmISGSREoyZ1TahpQMVwBLgH+6+qrECk/TIyjLmrqo5e095ZRV3Tl2oBCMikUVtwXQBTgL2AB+G24YARjAV/lkET9af5O7vN3aQklqaw0xEGkPUTv6ZwHMEa6x81t0/SzBr8bPAC0Bvgmn9f5OUKCWlapvDrFVOFqtLlWREJJqoCeZHwC3hhJfA3skvfwFc4e67gF8DRzZ6hJJyieYwy8029uxxTvvt6zw7R0OZRaR+URNMIdAjwfbu/HeKl61o0ECLEDuHGQRzmN15zlBe+MnJ9OnUhu9PfJerHp9N2c7d9dQkIpksakKYDNxvZlcTTHbpwLEEE1JOCsscSzCVvrQA1XOYxS+W9M/LTuB3Ly3mj9OX8PYnm/jt149kWEnH9AUqIk1W1BbM9wimu/8b8BGwNHz/PMHMxgDzge80doDStORmZ/HT0QP4+9jj2V3lnPPnf/P7lxdrVmYRqSFSgnH3He7+PYIFyIYBRwFF7n5ZuD4L7v6+RpBljmP7FvHcj0/ijCN6cPeLi/jG+DdZsWlH/V8UkYzRoD6TMJloxUgBoF3rXH73jWF8bkBXbnjyQ774u9c5c1gPpi9Yz6otFfTskK8ZAEQyWNQHLVsTjCQbBXQlruXj7kc0fmjSXIwZVszRvTvy7QlvMfHNFXu3awYAkcwWtQXzR+B/gMeBNwg6+UX26lXUhp27a/bDaAYAkcwVNcGMAb7q7i8lMRZp5lZvqUi4XTMAiGSmqKPIdgAr6i0lGa22GQAc+OWz8/XcjEiGiZpg7gB+YmZaP0ZqlWgGgNa5WRzXtyPjX1vKqN9M5+nZq7TWjEiGiHqL7PMEk12eZmbzgMrYne7+lcYOTJqf6n6WO6cuZNWW8n1Gkb27fDM/f+pDfvjoezz61nJuPnMwh3Zrm+aIRSSZoiaYDQRP84vUqXoGgHhHlXTkqR+cyCNvL+euqQv50u9e56IRffjRqYdSmKcZhkRaokj/Z7v7RckORFq+7Czj/OG9Of3wHtzx/ALum/ExT72/iutOH8iePc5dLyyq0fIRkeZLfzpKyhUVtOJXZx/BN44t4edPfciP/v4+WQZ7wq4ZPT8j0jLU2mlvZh+YWcfw/Zzwc8JX6sKVluTIXh2Y/P0RdMjP3ZtcqlU/PyMizVddLZgngJ3h+3+mIBbJQNlZRml5ZcJ9en5GpHmrNcG4+82J3os0tp4d8llZSzK5+4WFXHLSwbTPz01xVCJyoPRci6Rdoudn8nKyGHpQe34/bQkn/Xoaf3h5sR7UFGlmok52WUSwPHJtk122a/zQJFPU9fzM3FWl/O+Li/nNi4uYMPNjvntyP759fG/atNL4FJGmLur/pfcTrAMzHliFJruURlbb8zODe7bnvgs+w+wVW/jflxbxq+cWcN/rS7ls5CGce1wJz3+4JmFiEpH0i5pgRgGfd/e3khmMSG2G9urAAxcdy3+WbeLuFxdx6zPz+N1LCymv3ENlVfD3joY3izQtUftg1gFlyQxEJIqjexcx8dLhPPqd4VRU+t7kUk3Dm0WajqgJ5jrgFjMrTGYwIlEd368TlVU1158BDW8WaSqiJpjrgS8A68xsvh60lKagruUBrnxsNvNWbU1tQCKyj6gJ5p/AXcCvgb8TPIQZ+4rEzIrMbLKZbTezZWb2rTrKXmFma8ys1MwmmFleQ+sxsxvNzM3s1KgxSvNR2/Dmk/p35rkPV/Ol37/ON8e/yUvz1rInfqoAEUm6ejv5zSwXKADudfdlB3i8e4FdQDfgSGCKmc1297lxxxwNjANOIRi1Nhm4OdwWqR4z6wecA6w+wJiliapreHNpeSV/f3s5D77xCZc+NIuDOxdw0Yg+nH30QbRplcOT763U6DORJKs3wbh7pZldBvzxQA5kZgXA2cAQdy8DZpjZ08D5/DdxVLsAuL86YZjZrcBEYFwD6rkHuOZA45amrbbhze3zc/nuyf24+MS+PPfhGu6f8TE3PDWXu15YxGf6dGTm4g1U7A76cDT6TCQ5og5TfoGgNTHhAI51KFDl7otits0GTk5QdjDwVFy5bmbWCSiprx4z+yqwy92fNbNaAzKzscBYgC5dujB9+vQGnVAmKCsra/bXpR3w40HOkp6tmfpJJS/PX1ejTHllFbc+NZsOpYsj1dkSrksy6LrUlMnXJGqCeRn4pZkdAfwH2B67090nRaijECiN21YKJFrWML5s9fu29dUTjnT7JcGghDq5+3iCh0cZMGCAjxw5sr6vZJzp06fTUq7L54DvAH3HTUn4pPCmCo98ri3pujQmXZeaMvmaRE0w94T//jDBPgeyE2yPV0bwx2SsdsC2CGWr32+LUM/NwMPu/nGEmCQD1Ta5pgM/mPguXzumFyce0pnsrNpbvyJSv0ijyNw9q45XlOQCsAjIMbP+MduGAnMTlJ0b7ostt9bdN0aoZxTww3AE2hqgF/CYmV0TMU5p4WobfXZy/8688dEGLpjwNif9ehp3v7iIFZt2pClKkeYvZTMGuvt2M5tE8MDmpQSjv84ETkhQ/CHgATObSDAK7HrggYj1jAJi53Z/B/gJ8Fwjn5I0U3WNPtu5u4qX5q3jH7NW8Idpi/nDtMWM6NeZrx3Tiy8M6rZ37rOVW8opfnOaRp+J1CFygglnVD6NoJO9Vew+d78lYjXfJxgosA7YCFzm7nPNrASYBwxy9+Xu/ryZ3QG8AuQTPGtzY331hLFsjIu7CtgcjjgTAWoffZaXk83pR/Tg9CN6sHJLOY/PWsHjsz7lh4++R35uFruqnKo9mvtMJIqo0/UPB6YQrHDZBVgJ9Ag/fwJESjDuvgkYk2D7coLO+9htdwN3N6SeWsr2iVJOJF5xh3x+fOqh/PCU/sz8aANjH5q1N7lUq577TAlGpKaoT/LfSfAcSjFQQTBkuQSYRfB0v0iLlZVlnNS/CxWViec+W7mlnPteX8rqUs2BJhIraoI5ArjH3R2oAvLcfS3Bg4w3JSk2kSaltrnPcrON26bM5/jbp/HVP7/Bg298wrptFSmOTqTpidoHsyvm/VqgNzCfYMhwz8YOSqQpumr0AK6dNIfyyqq92/Jzs7n9rMMZ2qsDz8xexTMfrObGp+dy87/mMvzgTpxxRE++OKQ7ry5ar6lpJONETTDvAscQDBGeDtxmZt2A8wDNpiwZIXb02cot5RTHJYr/N6o//29Ufxat3bY32fxs8hyumzwHM6juvtHgAMkUDVkPZlX4/npgPfAHoCPhVCsimWDMsGJmjjuFB04rYOa4UxImiEO7teUnXxjAy1eezJQfnkhBXg7xkzmXV1Zx+3PzUxS1SHpEasG4+6yY9+uBLyYtIpEWwswY3LM923fuTrh/7dadnHr3q5w6sBunDuzKsJKOmj1AWpQGPWhpZp8B+gHPhA88FgA73T3x/0EiUuvUNO3zc+jerjX3vb6UP7/6EUUFrfjcgK58flBXTurfhYI8LSsgzVvU52C6AU8T9MM40B9YSvCcSgXwo2QFKNLc1TY44OavDGHMsGK2VlTy2qL1vDRvLS/NX8sT735Kq+ws+nYpYOn6Miqr9GCnNE9RWzD/C6wBOgHLY7Y/TtAXIyK1qGtqGoB2rXM544ienHFET3ZX7WHWss28PH8tE2Z+kvDBzl8/v0AJRpqFqAlmFDDK3TfHra/yEcEDlyJSh9qmpomXk53F8IM7MfzgTtz3euIJwVeXVvDN8W9yYv/OnNS/M4N7tlffjTRJURNMPvs+C1OtC8EtMhFpZLX13RTm5bClvJI7py7kzqkL6dAmlxH9gmRzYv/OHNSxjfpupEmImmBeAy4EfhZ+djPLJniS/+UkxCWS8Wrru7ltTNB3s37bTmYu2cDrizcwY8l6psxZDUDnwlZs3lGpSTkl7aImmKuBV83sGCAP+A3BssbtgRFJik0ko9XXd9Olbd7eW2/uzpJ1Zby2eAN3PL8gYd/NLf+ay0n9O9OpMC/l5yKZKepzMPPM7HDgMoIZlFsTdPDf6+6rkxifSEaL2ndjZvTv1pb+3dpy2zPzEpbZtKOSo297iUO6FnJs3yKO61vEcX070b19671ldGtNGlPk52DcfQ37rsmCmfU2s8fc/WuNHpmI7Jfa+m46F+ZxyYl9efvjjfzr/VU88lYwILSkqA3H9i0iN9uY9O5Kdu4OZo3WrTU5UAe6omUH4OxGiENEGkltfTfXnz6QMcOKuWxkP6r2OPNXb+Wtjzfx1tKNvDx/LZt3VNaoq7yyijs0LFr2U8qWTBaR1Kiv7wYgO8sYUtyeIcXtueTEvuzZ4/T72bN4gvpWlVYw5t6ZHFXSkWElHRhW0oHiDvnEPrJQfWtNS0lLLCUYkRYoat9Ntawsq3NYdKvsLB55exkTZgbP5nRpm8dRJR0YVtKRsopK7pvx8d4F2XRrTaopwYgIUP+w6MqqPSxcs433lm/m3eVbeG/5ZqbOXZuwLt1aE6gnwZjZ0/V8v10jxiIiaVTfrbXc7Ky9t9XOPz74zqbtuzjq1hcT1reqtIIv/2EGQ4rbc3j4GtC9La1y/rtKiEattWz1tWA2RtifeD4LEWl2GnprraigFcV13Fprl5/DlA9W8ejbwYi1VtlZDOjeliHF7dm9Zw9Pv79Ko9ZasDoTjLtflKpARKR5qu/WmruzfNMO5qwsZc7KUj5cWcqUD1axtaLmKh/llVXc+sy8eh8IVcuneVAfjIgckPqWkjYzencqoHenAs44oicA7s7B1yYetbZx+y6Ovu0lurbN47Ae7RjYoy0Du7djYI92HNylgCkfrN4noanl03QpwYjIAau+tTZ9+nRGjhxZb3mz2ketdS5sxfdO7sf81duYv3orf/1oI7uqgttorbKz2OPO7gRT4dw5daESTBOjBCMiaVH7A6GD9kkUlVV7WLp+O/NXb2X+mq385dWlCetbuaWccU98EEyZ07WQQ7u1pVu7vITP6+jWWmoowYhIWkR5IBSC0WsDurdlQPe2jKGYZ2avTtjyaZWdxQvz1vL3d1bs3da2dc7eZFNRWcWzc9bsbQ3p1lryKcGISNo0dNQa1N7yuf2swxkzrJiNZTtZtLaMxeu2sXhtGYvWbuOFeWvZtL3mklbllVXc8NSH5GQbB3cu5OAuBbTOza5RTi2f/aMEIyLNSn0tn06FeRxfmMfx/Trt872+46YkHFSwrWI3lz/yHgBm0LN9Pgd3KaBfl0L6dSlgdWkFE2Z8TIWGUzeYEoyINDv70/KpbVBBz/atue+CY/hofRlL129n6YYyPlpfxmOzVrBjV1WCmoKWz01Pz6VH+9b06VxA17b79vWA5mcDJRgRyRC13Vq7+rTDGNSzHYN67jsxibuzdutOht+eeNHeLeWVfH38m3vr6d2pDb07taFP5wI2b9/Fk++vYleGt3qUYEQkI0QdVFDNzOjevnWtMxV0a5fHnecMZdnG7XyycQefbNjOknVlvLJg/d6BBLHKK6u4bvIcNpTt5KCObSgpakOvonzats7dp1xL6u9RghGRjNGYgwqu/eJAPntoF6DLPuWr9jiH1LL0wfZdVdw2Zf4+2zq2yaWkqA0HFbWhorKK1xatp7Iq+HZDWj5NMTEpwYiI1KGhLZ/sOpY+KO7Qmik/PInlm3awYlN58O/mHazYtIO5K0v5ZOOOGt8pr6zi6n9+wLQF6yjumE9xh3yKO+ZzUPhvm1Y5PPneyiY5u0FKE4yZFQH3A18ANgDXuvsjtZS9ArgGyAeeAC5z95311WNmw4FbgaOBKmA68EN3X528MxORlqyhLZ/aWj1XjT6MDm1a0aFNK444qEON79U20m1X1R7eX7GFZ+esrjGLQVFBK7ZVVO5t9VQrr6ziV88t4CtDe5KVte8AhFjJbPmkugVzL7AL6AYcCUwxs9nuPje2kJmNBsYBpwCrgMnAzeG2+urpCIwHpgK7gXuAvwKnJfPERESq1Tc/W21qb/nk89rVn6Nqj7NuWwUrN5ezcks5n4b/PvLW8oT1rdlawYAbnqN7+9b0aJ9Pz/at6dEh/Ld9PgvWbuWeaUv2a7G46sTUqvshR9dWJmUJxswKgLOBIe5eBswI15s5n/8mjmoXAPdXJx4zuxWYCIyrrx53fy7uuPcArybx1EREamjo/GxQV8tnABDcfuvRPp8e7fP5TMz3Xl24PmFiap+fyzePLWF1aTmrt1Qwa9lm1s5ZXaO1E6u8sorrn/yQzTt20b1da7q1b033dq3p2jaPnOxgLZ/4W3K1MffaD9SYzGwY8Ia758ds+ylwsrt/Oa7sbOCX7v6P8HNnYD3QGSiJWk+478fAN9x9eIJ9Y4GxAF26dDn6scceO+DzbGnKysooLCxMdxhNjq5LYrouNTX0mryxqpInFlWyscLp1No4+9BcTuiZW+93HvhwF7tiBq+1yoILh7Sq8d097mzd5WyqcG75d0XkuAxol2d0zDNWlu0hbPSw+sEfs3P14oT34FJ5i6wQKI3bVgq0jVC2+n3bhtRjZkcAPwfOTBSQu48nuJ3GgAEDPOpfGZmkIX99ZRJdl8R0XWpq6DUZCfysgccYCQzaj76U++dPq3UwwtOXn8iarRWs3VrBmtKdwfvSCtZsreCTResjxZXKBFNGzSWW2wHbIpStfr8taj1mdgjwHPAjd399P2MWEWkWGnMI9lWjD6NTYR6dCvMY3LN9je+N+FXixBQvq94SjWcRkGNm/WO2DQXmJig7N9wXW26tu2+MUo+Z9QZeAm5194cbKX4RkRZlzLBibj/rcIo75GMEgwmqJw2ty1WjB5CfYFLQeClrwbj7djObBNxiZpcSjP46EzghQfGHgAfMbCKwGrgeeCBKPWZWDEwD7nX3PyfznEREmrv9afnEjpKr6/mPVLZgAL5P8FzLOuBRgmdb5ppZiZmVmVkJgLs/D9wBvAIsC1831ldPuO9S4GDgxrDOMjMrS8G5iYhkjDHDipk57hR2rVnyn9rKpPQ5GHffBIxJsH05Qed97La7gbsbUk+472aCZ2ZERCSNUt2CERGRDKEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSZHSBGNmRWY22cy2m9kyM/tWHWWvMLM1ZlZqZhPMLC9qPWY2yswWmNkOM3vFzHon87xERKSmVLdg7gV2Ad2Ac4E/mdng+EJmNhoYB4wC+gAHAzdHqcfMOgOTgBuAImAW8I/knI6IiNQmZQnGzAqAs4Eb3L3M3WcATwPnJyh+AXC/u891983ArcCFEes5C5jr7o+7ewVwEzDUzA5L3tmJiEi8nBQe61Cgyt0XxWybDZycoOxg4Km4ct3MrBNQUk89g8PPALj7djP7KNy+IPYgZjYWGBt+3GlmHzb4rFq+zsCGdAfRBOm6JKbrUlNLvya1dkGkMsEUAqVx20qBthHKVr9vG6GeQmB9lOO4+3hgPICZzXL3z9R9CplH1yUxXZfEdF1qyuRrkso+mDKgXdy2dsC2CGWr32+LUE9DjiMiIkmSygSzCMgxs/4x24YCcxOUnRvuiy231t03Rqhnn++GfTb9ajmOiIgkScoSjLtvJxjddYuZFZjZCOBM4OEExR8CLjGzQWbWEbgeeCBiPZOBIWZ2tpm1Bn4OfODuC+IPEmf8gZ1hi6XrkpiuS2K6LjVl7DUxd0/dwcyKgAnA54GNwDh3f8TMSoB5wCB3Xx6W/QlwDZAPPAF8z9131lVPzHFOBe4h6Hx6C7jQ3T9JyUmKiAiQ4gQjIiKZQ1PFiIhIUijBiIhIUmR8gmnI/GiZxMymm1mFmZWFr4XpjikdzOxyM5tlZjvN7IG4fRk5511t18TM+piZx/zMlJnZDWkMNaXMLM/M7g9/j2wzs/fM7Isx+zPu5yXjEwwR50fLUJe7e2H4GpDuYNJkFXAbwaCSvTJ8zruE1yRGh5ifm1tTGFe65QArCGYVaU/ws/FYmHgz8ucllU/yNzkx85oNcfcyYIaZVc9rNi6twUmT4O6TAMzsM8BBMbv2znkX7r8J2GBmh0UYEt+s1XFNMlr4CMVNMZueMbOPgaOBTmTgz0umt2Bqmx9NLZjA7Wa2wcxmmtnIdAfTxNSY8w6onvMu0y0zs0/N7K/hX+4Zycy6EfyOmUuG/rxkeoJpyPxomeYagmUSigkeFPuXmfVLb0hNin52atoAHEPw/NnRBNdiYlojShMzyyU49wfDFkpG/rxkeoLRvGW1cPe33H2bu+909weBmcCX0h1XE6KfnTjh8hmz3H23u68FLge+YGbx16lFM7MsgplFdhFcA8jQn5dMTzANmR8t0zlg6Q6iCdGcd/Wrfoo7Y35uzMyA+wkGDZ3t7pXhroz8ecnoBNPA+dEyhpl1MLPRZtbazHLM7Fzgs8DUdMeWauH5twaygezqa8L+z3nX7NV2TczsODMbYGZZ4dpNvwemu3v8raGW7E/AQODL7l4esz0zf17cPaNfBEMGnwS2A8uBb6U7pnS/gC7AOwTN9y3Am8Dn0x1Xmq7FTQR/ice+bgr3nUqwiF05MB3ok+5403lNgG8CH4f/L60mmLS2e7rjTeF16R1eiwqCW2LVr3Mz9edFc5GJiEhSZPQtMhERSR4lGBERSQolGBERSQolGBERSQolGBERSQolGBERSQolGJEWKlyb5Zx0xyGZSwlGJAnM7IHwF3z86810xyaSKhm9HoxIkr1EsLZQrF3pCEQkHdSCEUmene6+Ju61CfbevrrczKaES+guM7PzYr9sZoeb2UtmVm5mm8JWUfu4MheY2Zxw+eK18cs6A0Vm9ni4JPjS+GOIJJMSjEj63Aw8DRxJsObOQ+EqkZhZG+B5grmsjgX+BziBmGWKzey7wF+AvwJHECynED8778+Bpwhm8v0HMCET1oKXpkFzkYkkQdiSOI9g4sNY97r7NWbmwH3u/p2Y77wErHH388zsO8BdwEHuvi3cPxJ4Bejv7kvM7FPgb+6ecHnv8Bi/cvdrw885wFZgrLv/rfHOViQx9cGIJM9rwNi4bVti3v87bt+/gdPD9wMJpnOPXZDqDWAPMMjMthKsNvpyPTF8UP3G3Xeb2Xqga6ToRQ6QEoxI8uxw9yX7+V3jvwt2xWvI4m+VcZ8d3RqXFNEPmkj6DE/weX74fh4w1Mxi12w/geD/2fkeLEm8EhiV9ChF9pNaMCLJk2dm3eO2Vbn7+vD9WWb2DsHiU+cQJIvjwn0TCQYBPGRmPwc6EnToT4ppFf0C+F8zWwtMAdoAo9z9N8k6IZGGUIIRSZ5TCVZ2jLUSOCh8fxNwNsHSwuuBi9z9HQB332Fmo4HfAm8TDBZ4CvhRdUXu/icz2wVcCfwa2AQ8m6RzEWkwjSITSYNwhNdX3f2f6Y5FJFnUByMiIkmhBCMiIkmhW2QiIpIUasGIiEhSKMGIiEhSKMGIiEhSKMGIiEhSKMGIiEhS/H9sdrylM8/uCQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "learning_rate = 0.01\n",
    "decay = 1e-4\n",
    "batch_size = 32\n",
    "n_steps_per_epoch = math.ceil(len(X_train) / batch_size)\n",
    "epochs = np.arange(n_epochs)\n",
    "lrs = learning_rate / (1 + decay * epochs * n_steps_per_epoch)\n",
    "\n",
    "plt.plot(epochs, lrs,  \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.01])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Tốc độ học\")\n",
    "plt.title(\"Hoạch định Lũy thừa\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Định thời Cấp số nhân"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```lr = lr0 * 0.1**(epoch / s)```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay_fn(epoch):\n",
    "    return 0.01 * 0.1**(epoch / 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay(lr0, s):\n",
    "    def exponential_decay_fn(epoch):\n",
    "        return lr0 * 0.1**(epoch / s)\n",
    "    return exponential_decay_fn\n",
    "\n",
    "exponential_decay_fn = exponential_decay(lr0=0.01, s=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 1.1122 - accuracy: 0.7363 - val_loss: 0.8947 - val_accuracy: 0.7496\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.7354 - accuracy: 0.7825 - val_loss: 0.6059 - val_accuracy: 0.8122\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.5973 - accuracy: 0.8175 - val_loss: 0.8195 - val_accuracy: 0.7754\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.6040 - accuracy: 0.8148 - val_loss: 0.6135 - val_accuracy: 0.8398\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.5462 - accuracy: 0.8323 - val_loss: 0.5075 - val_accuracy: 0.8490\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4479 - accuracy: 0.8555 - val_loss: 0.4538 - val_accuracy: 0.8502\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4225 - accuracy: 0.8622 - val_loss: 0.4792 - val_accuracy: 0.8524\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3873 - accuracy: 0.8678 - val_loss: 0.5517 - val_accuracy: 0.8448\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.3635 - accuracy: 0.8767 - val_loss: 0.5312 - val_accuracy: 0.8600\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.3353 - accuracy: 0.8840 - val_loss: 0.4671 - val_accuracy: 0.8660\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3108 - accuracy: 0.8927 - val_loss: 0.4885 - val_accuracy: 0.8670\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2895 - accuracy: 0.8987 - val_loss: 0.4698 - val_accuracy: 0.8636\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2660 - accuracy: 0.9071 - val_loss: 0.4558 - val_accuracy: 0.8820\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2442 - accuracy: 0.9153 - val_loss: 0.4325 - val_accuracy: 0.8774\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2375 - accuracy: 0.9177 - val_loss: 0.4703 - val_accuracy: 0.8800\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2196 - accuracy: 0.9231 - val_loss: 0.4657 - val_accuracy: 0.8870\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2013 - accuracy: 0.9312 - val_loss: 0.5023 - val_accuracy: 0.8760\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1938 - accuracy: 0.9331 - val_loss: 0.4782 - val_accuracy: 0.8856\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1774 - accuracy: 0.9394 - val_loss: 0.4815 - val_accuracy: 0.8898\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1703 - accuracy: 0.9418 - val_loss: 0.4674 - val_accuracy: 0.8902\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.1611 - accuracy: 0.9462 - val_loss: 0.5116 - val_accuracy: 0.8930\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1530 - accuracy: 0.9481 - val_loss: 0.5326 - val_accuracy: 0.8934\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.1436 - accuracy: 0.9519 - val_loss: 0.5297 - val_accuracy: 0.8902\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1326 - accuracy: 0.9560 - val_loss: 0.5526 - val_accuracy: 0.8930\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1308 - accuracy: 0.9560 - val_loss: 0.5699 - val_accuracy: 0.8928\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.LearningRateScheduler(exponential_decay_fn)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1kklEQVR4nO3deXhU5dnH8e+dBEgIS9gMEAVFEAUUcUOlVhQV21rFpbXVWm21tLV2sa1r3XCvtmpbl0rdXre644ZiRYyKKyiChFVlk32RQEICIdzvH+cEx2GSnGBmJmR+n+uaKzPnPPPMfQ5D7pzzbObuiIiINLasdAcgIiLNkxKMiIgkhRKMiIgkhRKMiIgkhRKMiIgkhRKMiIgkhRKMSAqY2VlmVtbA9xSb2e3Jiin8jPlm9uck1HuKmTVoDET8OdqecyZNixKMJJWZPWBmnuDxXrpjS5bw+E6J2/w40CsJn3WOmU0xszIzKzWzaWZ2bWN/Tpok5ZxJ6uSkOwDJCOOBM+K2bUpHIOni7hVARWPWaWY/B/4JnA+8BrQE+gOHNObnpEsyzpmklq5gJBU2uvuyuMcaADM73MyqzGxoTWEz+5WZrTOzXuHrYjP7t5n9w8y+DB83m1lWzHs6mNn/hfsqzGy8mfWP2X9W+Ff+MDObbmblZva6me0WG6iZfd/MPjSzSjObZ2bXmVnLmP3zzewyM7s7jPELM7sgdn/49MnwSmZ+7OfHlNvdzJ4zs2VhLB+Z2XENPK/HA8+4+93u/qm7z3D3J939j3HH9D0zez88L6vN7AUzy40pklvb8YTvb29mo81shZmtN7M3zOyAuDI/NbMFZrbBzF4ECuP2X2Vm0+O21XkLLME5uyr8t/uRmX0WxvKsmXWOKZNjZrfGfE9uNbO7zKy4/tMpjU0JRtLK3d8AbgYeMrOOZrYn8Hfgt+7+eUzR0wm+r4cAvwRGAn+I2f8AMBg4ATgI2ACMM7O8mDKtgEuAn4f1FAD/rtlpZsOBR4DbCa4Efg6cAlwfF/b5wCfAfsBfgZvMrOaq4cDw5y+AbjGv47UBXgaOBgYCTwPPhMcf1TLgoJpEnIiZHQs8B7wK7A8cAbzB1//v13o8ZmbAWKAIOA4YBLwJTDCzbmGZwQTnfzSwL/ACcHUDjqMhdgVOBU4EjgnjuS5m/5+Bs4BzgIMJjvO0JMUi9XF3PfRI2oPgF89moCzu8deYMi2AScAzwEfA43F1FANzAIvZdhnwRfi8D+DAt2P2twdKgXPC12eFZfrGlDmd4FZdVvj6TeDyuM8eEcZr4ev5wH/jyswFLot57cApcWXOAsrqOVfvxdVTDNxeR/luwLvh580FHgZ+CrSIKfM28FgdddR5PMCR4fHnxZX5GLgwfP4o8Grc/nuCXy9bX18FTK/rnER4fRVQCbSP2fYX4NOY10uBi2NeGzALKE73/4VMfOgKRlLhTYK/bGMfN9fsdPcqgr8yjwN2IrhCifeeh78xQu8CRWbWDtgL2BJuq6mzlOCv8n4x79no7rNjXi8hSG4F4ev9gb+Et9LKwtszjwL5QNeY902Li21JGHdkZpZvZjeZ2YzwVk4ZcADQI2od7r7U3Q8B9gZuI/hlejfwgZm1DosNImifqUtdx7M/0BpYGXdeBgC7h2X2Iubch+JfN5YF4b/tNrGaWXuCf6cPanaG35lJSYpF6qFGfkmFDe7+aT1lam5nFABdgLUNqN/q2BeblDbXsi8r5uco4MkE9ayMeV6VoJ6G/rH2N+BYgls6cwlu6T1I0FDfIO4+HZgO3GFm3wLeAn5IcPUYRV3HkwUsBw5L8L514c+6zn+NLQnKtYgYX6wo515TxDcRuoKRtDOzXQnaPX5D0FbwiJnF//EzOGwPqHEwsMTd1wEz+Kp9pqbOdgR/2c9oQCgfAXt60GAe/4hPTnWpArLrKfMt4EF3f9rdpwFf8NUVwTdRc7xtwp9TgGHfoL6PCBrstyQ4JytiPvPguPfFv14JFMb9G+77DeLaRnhls4ygDQ7Y2oZUWzuYJJmuYCQVWplZ17ht1e6+0syyCdoO3nD3u83sKYJbW1cCl8eU7w7cZmZ3EiSOC4BrAdx9rpk9B9xtZiMJrn6uI/gL+9EGxHk18KKZLQCeILjiGQAc5O4XNqCe+cAwM3uD4LbclwnKzAFODOOuIjje3ATlamVmdxHcIppAkKC6EbRNbQD+Fxa7DnjBzD4lOBdG0Dh+t7tviPAx4wnacZ4zswsJ2jO6Elx9jXf3twi6Sr9jZpcATwFDCRrhYxUDHYFLzeyxsEz8WKHG8A/gQjObQ5D4fklwXpYm4bOkHrqCkVQ4iuA/eOxjSrjvUqA3cDaAu68GzgQuDm/31HiE4KrgfeA/wL3ArTH7f0Zw7/358Gdr4FgPxlJE4u6vAN8j6Gn1Qfi4GFgY/VAB+FNYxyK+Os54fwRWENzOepmggf+tBn7OqwQ9554gSFhjwu1Hu/scAHd/ieCX/XfCWN4IY9sS5QPCNozvEiSx/wCzw8/rS5DccPf3CP79fk3QnnMSQYN8bD0zw/0jwzJHs23vvMbwN+Ah4H6CcwrBealMwmdJPWp6xog0WeEYhunufl66Y5Edj5l9BLzt7r9NdyyZRrfIRKTZMLOewHCCK7UcgiumgeFPSTElGBFpTrYQjAW6maAJYAbwHXefnNaoMpRukYmISFKokV9ERJJCt8hCBQUF3rt373SH0eSUl5eTn5+f7jCaHJ2XxHRettXcz8mHH364yt27JNqnBBMqLCxk8mTdpo1XXFzM0KFD0x1Gk6PzkpjOy7aa+zkJx40lpFtkIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFEowIiKSFClNMGbW0czGmFm5mS0ws9PqKHu+mS0zs1Izu8/MWsXsO8/MJpvZRjN7IMF7h5nZLDPbYGavm1nP+mKbv24LQ26cwLNTFm/38YmIyFdSfQVzB7AJKAROB+4ys/7xhcxsOHAxMAzYFegFjIopsgS4FrgvwXs7A88AlwMdgcnA41GCW7y2gkue+URJRkSkEaQswZhZPnAycLm7l7n7ROB54IwExc8E7nX3Enf/ErgGOKtmp7s/4+7PAqsTvPckoMTdn3T3SuAqYKCZ7Rklzoqqam5+ZXb0AxMRkYRSuWTyHkC1u8+J2TYVODxB2f7Ac3HlCs2sk7snSirx751a88Ldy83ss3D7rNiCZjYSGAnQsmvvrdsXr62guLi4vuPJCGVlZToXCei8JKbzsq1MPiepTDBtgNK4baVA2whla563JfFVS/x7V0b5HHcfDYwGaNWtj9dsLyrIa9ZraDdEc19PfHvpvCSm87KtTD4nqWyDKQPaxW1rB6yPULbmeaKy3+RztmHAH47qE6WoiIjUIZUJZg6QY2axv70HAiUJypaE+2LLLY9we2yb94ZtP7vX8jlf07lNSxz4bGV5hI8REZG6pCzBuHs5Qe+uq80s38yGACcADyUo/iBwtpn1M7MOwGXAAzU7zSzHzHKBbCDbzHLNrOZ23xhggJmdHJa5Apjm7rOow67tsph82dGcesAu/Oetz/nki/i7eSIi0hCp7qZ8LpAHrAD+C/za3UvMrIeZlZlZDwB3HwfcBLwOLAgfV8bUcxlQQdCV+Sfh88vC964k6K12HfAlMBj4UdQAL/3eXnTKb8mFT0+jqnrLNzlWEZGMltIE4+5r3H2Eu+e7ew93fzTcvtDd27j7wpiyt7h7obu3c/efufvGmH1XubvFPa6K2T/e3fd09zx3H+ru86PG2D6vBdeMGMDMpeu4+43PGufARUQykKaKSWB4/658b+9u/PO1T/l0RaS+ASIiEkcJphZXHd+f1q2yufCpaVRv8frfICIiX6MEU4subVtxxXH9+GjhWh56d366wxER2eEowdThxEFFHL5HF256ZTaL1mxIdzgiIjsUJZg6mBnXnTgAAy4d8wnuulUmIhKVEkw9du7Qmou+sydvzV3F0x9plmURkaiUYCL4yeCeHNCzA9e8OIMV6yvTHY6IyA5BCSaCrCzjr6fsQ0VVNVc+V++MMyIighJMZLt3acPvh/Xh5enLePmTpekOR0SkyVOCaYCR3+5Fv27tuPy5Eko3VKU7HBGRJk0JpgFaZGdx0yn78OWGTVw7dka6wxERadKUYBpoQFF7fvntXjz54Re8NTd+XTMREamRyhUtm43fDevDk5MXcdb9k9iyxelekMcFw/syYlBRukMTEWkylGC2w7jpyyit3Lx1jrLFayu45JlPAJRkRERCukW2HW5+ZTabNn99rZiKqmpufmV2miISEWl6lGC2w5K1FQ3aLiKSiZRgtkP3grxatuemOBIRkaZLCWY7XDC8L3ktsrfZPrhXpzREIyLSNCnBbIcRg4q44aS9KSrIw4Ciglz26tqWF6ctZcaSdekOT0SkSVAvsu00YlDR13qMrS7byLH/eIvf/vcjXvjtt2jdUqdWRDKbrmAaSac2rbjt1H35fFU5V7+gUf4iIkowjWhI7878+vDdeWzSIl6YuiTd4YiIpJUSTCM7/+g9GNSjgEuf+UTLLItIRlOCaWQtsrP4548GAfC7x6ZQVb2lnneIiDRPSjBJsEvH1lx/0t5MWbiW28bPSXc4IiJpoQSTJN8f2J0fHrAzdxZ/xjufrkp3OCIiKacEk0RXHd+f3Trn84fHP2Z12cZ0hyMiklJKMEnUumUO//rxINZuqOKCp6bh7ukOSUQkZZRgkqx/9/Zc+t09mTBrBQ+8Mz/d4YiIpIwSTAqceeiuHLXXTtzw0iymLy5NdzgiIimR0gRjZh3NbIyZlZvZAjM7rY6y55vZMjMrNbP7zKxV1HrM7IdmNtPM1pvZDDMbkcTDqpeZcdMpA+mQ34Lf/XcK5Rs3pzMcEZGUSPUVzB3AJqAQOB24y8z6xxcys+HAxcAwYFegFzAqSj1mVgQ8DPwRaAdcADxqZjsl55Ci6ZjfklvDqWQOvG48u108liE3TuDZKYvTGZaISNKkLMGYWT5wMnC5u5e5+0TgeeCMBMXPBO519xJ3/xK4BjgrYj07A2vd/WUPjAXKgd2TeHiRrFi3kZwsY8OmapyvllpWkhGR5iiVU/7uAVS7e+zIw6nA4QnK9geeiytXaGadgB711DMZmGlmxwNjge8DG4Fp8R9iZiOBkQBdunShuLh4Ow4rumuKN7B5y9d7klVUVXPNc1MpKJ2b1M/eXmVlZUk/LzsinZfEdF62lcnnJJUJpg0Q38JdCrSNULbmedv66nH3ajN7EHgUyCW4lfYDdy+P/xB3Hw2MBujbt68PHTq0AYfTcGvGjU28vdJJ9mdvr+Li4iYbWzrpvCSm87KtTD4nqWyDKSNoE4nVDlgfoWzN8/X11WNmRwE3AUOBlgRXNveY2b7bH3rjqG2p5W5aallEmqHICcbMvmNmL4a9snYJt51jZsMiVjEHyDGzPjHbBgIlCcqWhPtiyy1399UR6tkXeNPdJ7v7FnefBLwPHBUxzqSpbanlPQsTXcSJiOzYIiUYMzsdeAKYC+wGtAh3ZQMXRqkjvEX1DHC1meWb2RDgBOChBMUfBM42s35m1gG4DHggYj2TgMNqrljMbBBwGAnaYFJt26WW8xiyeycmzF7JmClfpDs8EZFGFbUN5kLgF+7+mJmdE7P9PeDqBnzeucB9wApgNfBrdy8xsx7ADKCfuy9093FmdhPwOpAHPA1cWV89AO7+hpldBTxlZoXASuB6d/9fA+JMmvillquqt3DGve9z0dOf0KtzGwbuUpC+4EREGlHUBNMHeDfB9kTtIbVy9zXAiATbFxI03sduuwW4pSH1xOy/Hbg9alzp1CI7iztP35/jb5/IyIcm88J532KndmqTEZEdX9Q2mCUE3YzjfRv4rPHCyUwd81vyn58ewPrKzYx86EMqq6rTHZKIyDcWNcGMBv4ZtncA7GJmZxL01rorKZFlmL26teOWHw7k40Vr+cuY6Zp5WUR2eJFukbn7TWbWHniVYGzJ6wSDF//m7nckMb6McuyAbvx+WB/+8dpc9urWlnMO65XukEREtlvkgZbu/hczuw7oR3DlM8Pdy5IWWYb6/bA+zF62nutfmskehW359h5d0h2SiMh2idpN+T4za+vuG8LxJR+4e1nYTfi+ZAeZSbKyjL//cCB7FLblvEc/Yt6qbSYgEBHZIURtgzmToLtwvDzgp40XjgDkt8rhPz89gOws4xcPTmZ9ZVW6QxIRabA6E0y47konwIAO4euaRxfgOGB5KgLNNLt0bM2dp+/P/FXl/OGxj6neokZ/Edmx1HcFs4pgMKMTDIRcGfNYBtwD3JnMADPZIbt34srv9+O1WSv4+/9mpzscEZEGqa+R/wiCq5cJBGuwrInZtwlY4O5LkhSbAD85uCczlq7nzuLP+O8HC1m7oYruBXlcMLzv12YEEBFpaupMMO7+BoCZ7QYscvctKYlKtjIz9u9RwOOTFvLlhqAtpmahMkBJRkSarKjjYBYAmFl3ggW/Wsbtf7PxQ5Mat46fS3wTTEVVNTe/MlsJRkSarEgJJkwsjxJMDeMEt81if+VtOwe9NJolaysatF1EpCmI2k35NqCaYJDlBoLp738AzASOTUpkslVtC5V1ba9JMUWk6YqaYA4HLnL3WQRXLivd/RngIuCaZAUngdoWKmuX14JNm9UsJiJNU9QEk0fQZRmCnmQ7hc9nAPs0dlDydYkWKvvRgbswe9l6/vzkVLZojIyINEFR5yKbBewJzAc+Bn5lZouA3wCLkxKZfE38QmUAPTvl89dxs+jQugVXHd8fM0tTdCIi24qaYP4BdA2fXw2MA35MMKPymUmISyL41eG9WFO+kf+8NY9ObVrxu2F90h2SiMhWUbspPxLz/CMz25Xgimahu6+q9Y2SVGbGJd/Zi9Xlm7jl1Tl0yG/JGQf3THdYIiJA9DaYrwlnVf4IKDezixs5JmmArCzjryfvw7A9d+KK56YzdtrSdIckIgJESDBm1tnMvmdmx5hZdrithZn9gaBN5s/JDVHq0yI7i9tP248DenbgD49PYeJcXVSKSPrVN5vyocBc4AXgZeBtM9sTmAacR9BFuUeyg5T65bXM5p4zD2T3Lm0Y+dBkpi5am+6QRCTD1XcFcw3wCkFX5H8ABwEvAjcAfdz9dnffkNwQJar2eS148OcH0TG/JWfd/wGfrtCCoyKSPvUlmIHANe4+HbiMYJDlJe7+oLtr8EUTtFO7XB4+ezDZWcZP732fpaWaTkZE0qO+XmQdCdZ+wd03mNkGYErSo5JvZNfO+Tzws4P40ej3OOH2t8nOMpaVVmqafxFJqSi9yDrErGzpQLu4lS07JjlG2Q4Ditpz5qE9WbF+I0tLK3G+mub/2SkaGysiyRclwdSsZLkCaANM4qtVLVeFP6UJenbKtmvB1UzzLyKSbFFWtJQdlKb5F5F0irSipeyYuhfksThBMtE0/yKSCts1kl92DLVN85+bk8X6yqo0RCQimSSlCSbsFDDGzMrNbIGZnVZH2fPNbJmZlZrZfWbWKmo9ZtbazO40s1Xh+zNySedE0/yfeUhPFn1ZwU/u/YDSDUoyIpI8UWdTbix3AJuAQmBfYKyZTXX3kthCZjYcuBg4ElgCjAFGhdui1DOa4Nj2Ili/Zt+kHVETl2ia/yG9O/ObRz/itHve4+GzB9Mhv2WaohOR5ixlVzBmlg+cDFzu7mXuPhF4HjgjQfEzgXvdvcTdvySYUeCsKPWYWV/geGCku69092p3/zDJh7dDOaZ/V0b/9ADmrijjx/95j1VlG9Mdkog0Q6m8gtkDqHb3OTHbphIsxxyvP/BcXLnCcCxOj3rqGQwsAEaZ2RnAUuAqd386/kPMbCQwEqBLly4UFxdvz3HtkAz4/b4t+cdH6zn+1te48MBcCnK3/XujrKwso85LVDoviem8bCuTz0mkBGNm99Wyy4FK4FPgcXffduDFV9oApXHbSoG2EcrWPG8boZ6dgQHA00B34BCCW2gz3H3m14J3H01wO42+ffv60KFD6wi/+RkK7L/fan7+wCT+MT2LR38xmG7t875Wpri4mEw7L1HovCSm87KtTD4nUW+RdQFOAkYAvcPHiHBbX+BCYLaZ7VtHHWVAu7ht7YD1EcrWPF8foZ4KoAq41t03hV2tXweOqSO2jHVwr048dPZBrFq/kR/e/S6L1mjuUhFpHFETzNsE0/Xv7O7fdvdvE1wpvAT8D+gJjAX+Xkcdc4AcM4td13cgUJKgbEm4L7bccndfHaGeaRGPSUL79+zIw+cMpnRDFafe/S7zV5WnOyQRaQaiJpjfA1fHTs0fPr8OON/dNwF/pY7eWu5eDjwDXG1m+WY2BDgBeChB8QeBs82sn5l1IJjJ+YGI9bwJLAQuMbOccP9QgmUHpBYDdyngvyMPpqKqmlNHv6up/kXkG4uaYNoA3RJs7xruA1hH/W065wJ5BPOa/Rf4tbuXmFkPMyszsx4A7j4OuIng1taC8HFlffWE760iSDjfJWib+Q/wU3efFfFYM1b/7u15bOQhVG9xRtwxkYOuG89Z48oZcuMETZApIg0WtRfZGOBeM7uQYLJLJ1h87CaCqwnC13MSvz3g7msI2m7ity/kq0RVs+0W4JaG1BOzv4SgcV8aqG/XtvzisF7c8PIsyjZWA1/Nwgxoqn8RiSzqFcyvCG4xPQx8BnwePh9HcDUBMBP4RWMHKKn34LsLttmmWZhFpKEiXcGE7S2/MrM/AbsTDKP4NGwPqSnzcVIilJTTLMwi0hgaNNAyTCjqpdXM1TYLc5vcHNwdM0tDVCKyo4l0i8zMcs3sIjP7n5l9bGbTYh/JDlJSK9EszNlmrK/czJ+fnMamzVvSFJmI7EiiXsHcCZwIPAm8Q9DIL81UTUP+za/MZvHaCooK8vjzMXuwcE0Ft46fw5K1Ffz7jP1pn9cizZGKSFMWNcGMAH7g7uOTGIs0ITWzMMdPc7FLxzwuenoap9z1DveddSC7dGydviBFpEmL2otsA7AomYHIjuGk/XbmwZ8PZvm6Sk688x2mfbE23SGJSBMVNcHcBPzRzLQCpnDI7p145txDyW2Rxal3v8erM5anOyQRaYKiJoyjgVOBeWb2spk9H/tIYnzSRPXeqS1jzh3CHoVtGPnQZO5/e166QxKRJiZqgllFMJp/ArAMWB33kAzUpW0rHht5CEfvVcioF2Yw6oUSqreo/4eIBKIOtPxZsgORHVNey2zu+sn+XDd2Jve9PY9J89awpnwTS0sr6V6QxwXD+2p6GZEMlcoVLaWZys4yrvh+P9Zu2MgzU75ac05zmIlktloTTDiA8nB3/9LMPqGOsS/uvk8ygpMdy/vzvtxmW80cZkowIpmnriuYp4GN4fOnUhCL7OA0h5mIxKo1wbj7qETPRWpT2xxmuS2yqdhUTV7L7ATvEpHmSuNapNEkmsMsJ8uoqKrmxDvfZp6WYhbJKFEnu+xoZneZ2RwzW2tm62IfyQ5SdgwjBhVxw0l7U1SQhwFFBXn87QcD+b+fH8TydZUc/6+JjJu+NN1hikiKRO1Fdi8wCBgNLEGTXUotauYwi/fi7w7j3Ec+4lcPf8TIb/fiwuF9ycnWBbRIcxY1wQwDjnb395MZjDRfRQV5PPHLg7n2xZmMfvNzPl60ltt/PIid2uWmOzQRSZKof0KuAMqSGYg0f61ysrlmxABuO3VfPvmilO/9ayLvf66JIESaq6gJ5i/A1WbWJpnBSGYYMaiIZ38zhLatcjjtnvcZ/eZnuOuuq0hzE/UW2WXArsAKM1sAVMXu1EBLaai+Xdvy3HlDuPCpaVz/0ixenLqElWWbWKYpZkSajagJRgMtpdG1zW3Bnafvx/mPf8yzH2uKGZHmpt4EY2YtgHzgDndfkPyQJJOYGZPma4oZkeao3jYYd68Cfg1Y8sORTKQpZkSap6iN/P8DjkxmIJK5uhfk1brvicmL1AFAZAcVNcG8BlxvZreZ2RlmdlLsI5kBSvOXaIqZVjlZ9Oqcz4VPTeMXD37IyvUba3m3iDRVURv5bw9//i7BPgc0i6Fst5p2lptfmc2StRVbe5EdP7A79709j5temc2xt73JdSfuzbEDuqY5WhGJKuqKlprTQ5KqtilmzjmsF4fv0YXzn/iYXz38ISfvtzNXHt+Pdrkt0hCliDSEEoc0eX0K2zLm3CH87sjePPvxYo699U3e+XRVusMSkXpETjDhjMqnmdnFZnZF7KOBdYwxs3IzW2Bmp9VR9nwzW2ZmpWZ2n5m1amg9ZnalmbmZHRU1RmmaWmRn8cdj+vLUrw4ht0U2p93zPqNeKOHJyYsYcuMEdrt4LENunMCzUxanO1QRCUW6RWZmBwNjCVa47AIsBrqFr+cDV0f8vDuATUAhsC8w1symuntJ3OcNBy4m6Lm2BBgDjAq3RarHzHYHTgE0P3wzMqhHB8b+7jBufHkm9789H+Orqb01QFOkaYl6BXMz8AhQBFQS/OLvAUwG/hqlAjPLB04GLnf3MnefCDwPnJGg+JnAve5e4u5fAtcAZzWwntuBiwgSkTQjeS2zGXXCADrlt9xm3YiaAZoikn5Re5HtA5zt7m5m1UArd//czC4CHiVIPvXZA6h29zkx26YChyco2x94Lq5coZl1IkhsddZjZj8ANrn7S2a1jw81s5HASIAuXbpQXFwc4TAyS1lZWZM9L6vLE//tsHhtRdJjbsrnJZ10XraVyeckaoKJ/Z+8HOgJzCSYwr97xDraAKVx20qBthHK1jxvW1894YzP1wPH1BeQu48mWESNvn37+tChQ+t7S8YpLi6mqZ6XovcmsDjBaP+8Flnstd/BFCZxrZmmfF7SSedlW5l8TqLeIvsIODB8Xgxca2ZnAv8EpkWsowxoF7etHbA+Qtma5+sj1DMKeMjd50WMS3ZQiQZo5mQZmzZvYdjf3+C+ifPYXL0lTdGJSEPWg6mZ7vYyYCXwL6AD4S2mCOYAOWbWJ2bbQKAkQdmScF9sueXuvjpCPcOA34U90JYBuwBPhLfzpBkZMaiIG07am6KCPIxg1cy//WAgE/48lP17duDqF2dwwh1vM2XhtpNpikjyRR1oOTnm+UrgOw39IHcvN7NnCBYuO4eg99cJwKEJij8IPGBmjxD0ArsMeCBiPcOA2FF4k4A/Ai83NGZp+moboPnAzw7k5enLGPVCCSfd9Q4/PqgHFw3fk/atNUBTJFUaNNDSzA4ws1PDnlyYWb6ZRW3HATgXyCNYgvm/wK/dvcTMephZmZn1AHD3ccBNwOvAgvBxZX31hO9d7e7Lah5ANfClu2vJ5wxiZnx372689qeh/HzIbjz2wUKO/HsxT3/4hSbPFEmRqONgCgm6Ah9IMOygD/A5cAtBt+XfR6nH3dcAIxJsX0jQeB+77Zaw/sj11FJ21yjlpHlq0yqHy4/rx0n7FXHZs9P505NTefLDRRzRtwsPvrvwa3OfaeyMSOOKevVxK7AM6AQsjNn+JEFbjEiT1r97e57+1aE8NmkR17xYwnufr9m6TwM0RZIj6i2yYcBfwkGPsT4jGJci0uRlZRmnDe5B+9Ytt9mnAZoijS9qgskj8Yj4LgS3yER2GMtLE39lF6+tUPuMSCOKmmDeJJyqJeRmlk0wFctrjR2USDLVtYLmGfd+wPTF8eN4RWR7RE0wFwK/MLNXgVbA34EZwBDgkiTFJpIUiQZo5rbI4qRB3SlZUspx/5rI+Y9/zBdfbkhThCLNQ9RxMDPMbG/g1wQzKOcSNPDf4e6arVh2KLWtoDliUBHrKqv4d/Fn3DtxHmOnLeWsIbvym6G9NX5GZDtEHsMSjimJHYuCmfU0syfc/YeNHplIEtU2QLNdbgsuPHZPzjikJ7f8bw7/eetzHp+0iPOO6M0Zh/Rk3PRl3PzKbBavraDovQnq3ixSh4YMkkykgGDqfJFmpVv7PG7+wUB+/q3d+Ou4WVz30kzueP1Tyjdtpqo66Aig7s0iddOSySJ12KtbOx742UE8cs7gryWXGureLFI7JRiRCIb07szm6sRdmJckWDJARJRgRCKrrXtzdpbx0idL2bJFY2hEYtXZBmNmz9fz/vh1WUSarQuG9+WSZz6hoqp667YW2UZBXgvOfeQjeu/UhvOO6M1x+3QjJ1t/u4nU18i/OsJ+LewlGSG2e/PitRUUhd2bvz+wO2M/WcrtE+byh8c/5rbxczj3iN6cOKiIFko0ksHqTDDu/rNUBSKyI6jp3hy/DO7xA7tz3N7d+N+M5fxrwlwufGoa/xg/l18P3Z0fHLAzL3+yLOG4G5Hm7Jt2UxaRUFaWceyArgzvX8jrs1fwz9c+5bJnp3PTuFlUVFWre7NkHF2/izQyM+PIPQsZc+6hPHz2YCqrtqh7s2QkJRiRJDEzvtWnM1XVWxLuV/dmae6UYESSrLbuzQ6cfs97jJ+xXF2cpVlSghFJstpmbz5un658vrKccx6czBF/L+a+ifNYX1mVpihFGp8a+UWSrK7Zm6uqt/BKyTLumziPq1+cwS2vzuEHB+zMWYfuSs9O+Tw7ZbF6n8kOSwlGJAVqm725RXYWx+3TneP26c7URWu5/+15PPzeAh54Zz79urVl7vJyNoVtOOp9Jjsa3SITaSIG7lLAbT8axNsXHclvj+jNzKXrtyaXGup9JjsSJRiRJmandrn88Zi+eC3t/kvWVuC17RRpQpRgRJqounqfHfG3Yu54/VOWr6tMbVAiDaAEI9JE1db77PTBu1DYLpebX5nNoTdO4OwHJvFKybJax9uIpIsa+UWaqLp6nwHMX1XOE5MX8dSHX/DarBV0btOKk/cr4ocH7sInX5Sq95mknRKMSBNWW+8zgF0753PhsXvyx6P3oHj2Sh6fvIh7Js7j7jc/J8ugZuymep9JuugWmcgOLic7i6P6FfKfnx7Au5ccSbvcHOInBqioquamcbPSE6BkLCUYkWZkp7a5rK/cnHDfktJK/vTEVIpnr1B7jaREShOMmXU0szFmVm5mC8zstDrKnm9my8ys1MzuM7NWUeoxs4PN7FUzW2NmK83sSTPrluxjE2kqaut91rplNv+bsYyz7p/EQdeN59Ixn/De56upjrnceXbKYobcOIHdLh7LkBsn8OyUxakKW5qhVLfB3AFsAgqBfYGxZjbV3UtiC5nZcOBi4EhgCTAGGBVuq6+eDsBo4BVgM3A7cD9wbDIPTKSpSLS0c16LbK4/cW++s3dX3pyzihemLmHMR4t59P2FFLZrxff27k67vBzufuMzKqo0c4A0jpQlGDPLB04GBrh7GTDRzJ4HzuCrxFHjTODemsRjZtcAjwAX11ePu78c97m3A28k8dBEmpT6ep8d3a+Qo/sVsmHTZl6buYIXpi7h4fcWbDNrAHw1c4ASjGwPS9WIYDMbBLzj7nkx2/4MHO7u348rOxW43t0fD193BlYCnYEeUesJ9/0B+JG7H5xg30hgJECXLl32f+KJJ77xcTY3ZWVltGnTJt1hNDnN7byUVzm/eW1DrfvvOaY1OVlWbz3N7bw0huZ+To444ogP3f2ARPtSeYusDVAat60UaBuhbM3ztg2px8z2Aa4ATkgUkLuPJridRt++fT12jXUJxK89L4HmeF6u/3ACi2tZBO0Pb2zi8L5dOKZfIUP32In2rVskLNccz8s3lcnnJJUJpgxoF7etHbA+Qtma5+uj1mNmvYGXgd+7+1vbGbNIxkjUdpPbIoufHNyT8o2beXXGCsZOW0pOlnHQbh05ul8hR+1VyC4dW29dVmDx2gqK3puggZ0CpDbBzAFyzKyPu88Ntw0EShKULQn3PRFTbrm7rzazyvrqMbOewHjgGnd/KAnHItLs1Nd2c90I5+Mv1jJ+xnJenbGcUS/MYNQLM+jWrhUryzaxOeyNps4BUiNlCcbdy83sGeBqMzuHoPfXCcChCYo/CDxgZo8AS4HLgAei1GNmRcAE4A53/3cyj0mkualr5oCsLGO/Hh3Yr0cHLjx2T+avKmf8zOXcNG721uRSo6KqmhtfnqUEk+FSPdDyXCAPWAH8F/i1u5eYWQ8zKzOzHgDuPg64CXgdWBA+rqyvnnDfOUAv4MqwzjIzK0vBsYlklF0753POYb1qHbS5bF0lx972Jje8NJOJc1dRGXPrrYbG3TRvKR0H4+5rgBEJti8kaLyP3XYLcEtD6gn3jSIYMyMiKdC9IC9h54B2uTl0zG/J/W/P5+43P6dVThaDe3Xi2306c1ifLsxYUsqlY6ZvbfPRrbXmR5Ndisg3UtvAzqtPGMCIQUVs2LSZ9+et4a05q3hz7kquHTsTmPm1CTlraNxN86IEIyLfSGzngMVrKyiK6xzQumUOR/TdiSP67gQEK3JOnLuKC5+elrC+xWsrttYjOzYlGBH5xmo6B0QZ89G9II8fHrgL/3htbq3jbobcOIGigjwG9+rI4N06Mni3TvTs1BqzYLBnTbdorXfTtCnBiEhaJL61lsV5R/Yhv2U2789bwxuzV/LMR0HDf2G7Vhy0Wyda5WTxwtQlbNysOdOaOiUYEUmL+sbdnDVkN9ydz1aW8f68Nbz/+Rren7ea5es2blNXRVU1fx2nbtFNjRKMiKRNXeNuAMyM3ju1pfdObTl9cE/cnV6XvESiGRSXllYy/NY3GdSjgH13KWBQjw703qkN2TFzqOnWWmopwYjIDsPMau0W3TY3h24Fubw8fRmPTVoEQJtWOeyzc3sG9ShgY9UWHn5/AZVajiBllGBEZIdSW7foa8Ju0e7OvFXlfLxoLVMWrmXKoi/59xuff21htRoVVdXcqFtrSaMEIyI7lPrabsyMXl3a0KtLG07ab2cAKjZV0++KcQlvrS0rreTg61+jf/d29C9qz4DwZ/f2ueq19g0pwYjIDqe+tpt4eS2za7211j4vh0N278T0xaW8PnvF1sGfHVq3YEBRe1pkG2/NXUVVtSbzbCglGBHJCLXdWht1/ICtiaJiUzUzl62jZHEp0xevo2Rp8DNeRVU1Vzw3nQ75Lelb2JbCdq22Xu3U0BIGSjAikiHqu7UGwZVOzYzRNXa7eGzCW2vrKjdz5n0fANA+rwV9C9uyR9c29C1sy/J1ldwzcV7GdyhQghGRjNHQW2tQ+2Se3drncuup+zJn+XpmLVvPnGXree7jJayv3Jywnoqqaq4dO4ODe3VKeMVTozm19yjBiIjUobZbaxcduycH9+rEwb06bd3u7ixbV8khN0xIWNeqsk0cfMNr5LfMpleXNuzeJT/82YZeXfKZvriUK54raTYzTCvBiIjUIcqttRpmRrf2eRTVctXTuU1Lfj+sD5+tLOezlWVMmv8lz368pM7Pr6iq5vqXZvKdvbvSKie71nJN8cpHCUZEpB4NvbVW21XPZd/rt009GzZtZt6qcj5fWc5v/zslYX0r1m9kz8vH0b19Hj07taZnp9b06Jgf/mzN9MVrGfXCzCZ35aMEIyLSyOpbwiBW65Y59O/env7d23Pjy7MSXvl0aN2CMw7ZlYWry1mwZgP/K1nO6vJNdcZQUVXNNS/OYO+d21NUkEdui8RXP8m88lGCERFJgoYsYVCjtiufK7/ff5tf+usrq1iwegML12zg3Ec+Sljf6vJNDPv7GwB0btOKog557Nwhj50L8ijqkMeiNRt48N0F2zUzdU1iatm19/61lVGCERFpIhrS3tM2NxgIOqCofZ1tPpd+dy8Wfxks4vbFlxXMWLKOV2csZ1OYVOJVVFVz6ZhPWLB6A93a59K1fe7Wn21zWwBBcolPhIkowYiINCHb05W6IW0+AFu2OKvKNjL4+tcSjvHZsKmaW8fP2WZ7m1Y5dG2fy6I1G7Ze9dRFCUZEZAfXkCsfgKwsY6d2ubWO8SkqyGPCnw9nxbqNLFtXydLSSpaVVoQ/K/l0RVmkuJRgRESagca88rlgeF9a5WSzS8fW7NKx9TbvG3LjhFqXu46V1aBoRESk2RgxqIgbTtqbooI8jODK5YaT9q43UV0wvC95tfRKi6UrGBGRDLY9Vz6xt+SW1lFOVzAiItJgIwYV8fbFR7Jp2acf1lZGCUZERJJCCUZERJJCCUZERJJCCUZERJJCCUZERJIipQnGzDqa2RgzKzezBWZ2Wh1lzzezZWZWamb3mVmrqPWY2TAzm2VmG8zsdTPrmczjEhGRbaX6CuYOYBNQCJwO3GVm/eMLmdlw4GJgGLAr0AsYFaUeM+sMPANcDnQEJgOPJ+dwRESkNilLMGaWD5wMXO7uZe4+EXgeOCNB8TOBe929xN2/BK4BzopYz0lAibs/6e6VwFXAQDPbM3lHJyIi8VI5kn8PoNrdY6fonAocnqBsf+C5uHKFZtYJ6FFPPf3D1wC4e7mZfRZunxX7IWY2EhgZvtxoZtMbfFTNX2dgVbqDaIJ0XhLTedlWcz8ntTZBpDLBtAFK47aVAm0jlK153jZCPW2AlVE+x91HA6MBzGyyux9Q9yFkHp2XxHReEtN52VYmn5NUtsGUAe3itrUD1kcoW/N8fYR6GvI5IiKSJKlMMHOAHDPrE7NtIFCSoGxJuC+23HJ3Xx2hnq+9N2yz2b2WzxERkSRJWYJx93KC3l1Xm1m+mQ0BTgAeSlD8QeBsM+tnZh2Ay4AHItYzBhhgZiebWS5wBTDN3WfFf0ic0d/sCJstnZfEdF4S03nZVsaeE3NPtGBmkj7MrCNwH3A0sBq42N0fNbMewAygn7svDMv+EbgIyAOeBn7l7hvrqifmc44CbidofHofOMvd56fkIEVEBEhxghERkcyhqWJERCQplGBERCQpMj7BNGR+tExiZsVmVmlmZeFjdrpjSgczO8/MJpvZRjN7IG5fRs55V9s5MbNdzcxjvjNlZnZ5GkNNKTNrZWb3hr9H1pvZFDP7Tsz+jPu+ZHyCIeL8aBnqPHdvEz76pjuYNFkCXEvQqWSrDJ/zLuE5iVEQ8725JoVxpVsOsIhgVpH2BN+NJ8LEm5Hfl1SO5G9yYuY1G+DuZcBEM6uZ1+zitAYnTYK7PwNgZgcAO8fs2jrnXbj/KmCVme0ZoUv8Dq2Oc5LRwiEUV8VsetHM5gH7A53IwO9Lpl/B1DY/mq5gAjeY2Soze9vMhqY7mCZmmznvgJo57zLdAjP7wszuD/9yz0hmVkjwO6aEDP2+ZHqCacj8aJnmIoJlEooIBoq9YGa7pzekJkXfnW2tAg4kGH+2P8G5eCStEaWJmbUgOPb/C69QMvL7kukJRvOW1cLd33f39e6+0d3/D3gb+G6642pC9N2JEy6fMdndN7v7cuA84Bgziz9PzZqZZRHMLLKJ4BxAhn5fMj3BNGR+tEzngKU7iCZEc97Vr2YUd8Z8b8zMgHsJOg2d7O5V4a6M/L5kdIJp4PxoGcPMCsxsuJnlmlmOmZ0OfBt4Jd2xpVp4/LlANpBdc07Y/jnvdni1nRMzG2xmfc0sK1y76Z9AsbvH3xpqzu4C9gK+7+4VMdsz8/vi7hn9IOgy+CxQDiwETkt3TOl+AF2ASQSX72uB94Cj0x1Xms7FVQR/icc+rgr3HUWwiF0FUAzsmu5403lOgB8D88L/S0sJJq3tmu54U3heeobnopLglljN4/RM/b5oLjIREUmKjL5FJiIiyaMEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEIyIiSaEEI9JMhWuznJLuOCRzKcGIJIGZPRD+go9/vJfu2ERSJaPXgxFJsvEEawvF2pSOQETSQVcwIsmz0d2XxT3WwNbbV+eZ2dhwCd0FZvaT2Deb2d5mNt7MKsxsTXhV1D6uzJlm9km4fPHy+GWdgY5m9mS4JPjn8Z8hkkxKMCLpMwp4HtiXYM2dB8NVIjGz1sA4grmsDgJOBA4lZpliM/slcDdwP7APwXIK8bPzXgE8RzCT7+PAfZmwFrw0DZqLTCQJwiuJnxBMfBjrDne/yMwcuMfdfxHznvHAMnf/iZn9AvgbsLO7rw/3DwVeB/q4+6dm9gXwsLsnXN47/Iwb3f2S8HUOsA4Y6e4PN97RiiSmNhiR5HkTGBm3bW3M83fj9r0LfC98vhfBdO6xC1K9A2wB+pnZOoLVRl+rJ4ZpNU/cfbOZrQR2ihS9yDekBCOSPBvc/dPtfK/x1YJd8Rqy+FtV3GtHt8YlRfRFE0mfgxO8nhk+nwEMNLPYNdsPJfg/O9ODJYkXA8OSHqXIdtIVjEjytDKzrnHbqt19Zfj8JDObRLD41CkEyWJwuO8Rgk4AD5rZFUAHggb9Z2Kuiq4DbjWz5cBYoDUwzN3/nqwDEmkIJRiR5DmKYGXHWIuBncPnVwEnEywtvBL4mbtPAnD3DWY2HLgN+ICgs8BzwO9rKnL3u8xsE/An4K/AGuClJB2LSIOpF5lIGoQ9vH7g7k+lOxaRZFEbjIiIJIUSjIiIJIVukYmISFLoCkZERJJCCUZERJJCCUZERJJCCUZERJJCCUZERJLi/wGhnHaPiXVH9QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, history.history[\"lr\"], \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Tốc độ học\")\n",
    "plt.title(\"Hoạch định cấp số mũ\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hàm hoạch định có thể lấy tốc độ học hiện tại làm đối số thứ hai:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay_fn(epoch, lr):\n",
    "    return lr * 0.1**(1 / 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nếu bạn muốn cập nhật tốc độ học tập ở mỗi lần lặp thay vì ở mỗi epoch, bạn phải viết lớp gọi lại của riêng mình:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 1.1153 - accuracy: 0.7390 - val_loss: 0.9588 - val_accuracy: 0.7338\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.6929 - accuracy: 0.7934 - val_loss: 0.5328 - val_accuracy: 0.8318\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.6317 - accuracy: 0.8097 - val_loss: 0.7656 - val_accuracy: 0.8278\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5827 - accuracy: 0.8258 - val_loss: 0.5585 - val_accuracy: 0.8382\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5041 - accuracy: 0.8407 - val_loss: 0.5367 - val_accuracy: 0.8574\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.4595 - accuracy: 0.8588 - val_loss: 0.6000 - val_accuracy: 0.8516\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4490 - accuracy: 0.8644 - val_loss: 0.4605 - val_accuracy: 0.8648\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3925 - accuracy: 0.8783 - val_loss: 0.5076 - val_accuracy: 0.8616\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4085 - accuracy: 0.8797 - val_loss: 0.4577 - val_accuracy: 0.8650\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3440 - accuracy: 0.8927 - val_loss: 0.5309 - val_accuracy: 0.8762\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3267 - accuracy: 0.8948 - val_loss: 0.4652 - val_accuracy: 0.8792\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3046 - accuracy: 0.9033 - val_loss: 0.4863 - val_accuracy: 0.8692\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2811 - accuracy: 0.9087 - val_loss: 0.4726 - val_accuracy: 0.8770\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2684 - accuracy: 0.9145 - val_loss: 0.4526 - val_accuracy: 0.8760\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2478 - accuracy: 0.9209 - val_loss: 0.4926 - val_accuracy: 0.8838\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2315 - accuracy: 0.9253 - val_loss: 0.4686 - val_accuracy: 0.8840\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2164 - accuracy: 0.9318 - val_loss: 0.4845 - val_accuracy: 0.8858\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2093 - accuracy: 0.9346 - val_loss: 0.4923 - val_accuracy: 0.8834\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1929 - accuracy: 0.9396 - val_loss: 0.4779 - val_accuracy: 0.8880\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1852 - accuracy: 0.9439 - val_loss: 0.4886 - val_accuracy: 0.8868\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1740 - accuracy: 0.9470 - val_loss: 0.5097 - val_accuracy: 0.8852\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1668 - accuracy: 0.9474 - val_loss: 0.5161 - val_accuracy: 0.8898\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1571 - accuracy: 0.9530 - val_loss: 0.5381 - val_accuracy: 0.8886\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.1444 - accuracy: 0.9575 - val_loss: 0.5415 - val_accuracy: 0.8910\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1447 - accuracy: 0.9569 - val_loss: 0.5833 - val_accuracy: 0.8880\n"
     ]
    }
   ],
   "source": [
    "K = keras.backend\n",
    "\n",
    "class ExponentialDecay(keras.callbacks.Callback):\n",
    "    def __init__(self, s=40000):\n",
    "        super().__init__()\n",
    "        self.s = s\n",
    "\n",
    "    def on_batch_begin(self, batch, logs=None):\n",
    "        # Note: the `batch` argument is reset at each epoch\n",
    "        lr = K.get_value(self.model.optimizer.learning_rate)\n",
    "        K.set_value(self.model.optimizer.learning_rate, lr * 0.1**(1 / self.s))\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        logs = logs or {}\n",
    "        logs['lr'] = K.get_value(self.model.optimizer.learning_rate)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "lr0 = 0.01\n",
    "optimizer = keras.optimizers.Nadam(learning_rate=lr0)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "\n",
    "s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\n",
    "exp_decay = ExponentialDecay(s)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[exp_decay])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_steps = n_epochs * len(X_train) // 32\n",
    "steps = np.arange(n_steps)\n",
    "lrs = lr0 * 0.1**(steps / s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA/UElEQVR4nO3dd3wUdfrA8c+TQhLSICSE3puAFAEVUYroWU5FxXLqeaIi6qnncWe9s7ffnd07+x2KXVCk2JWToBTpNfTeEggEUkkI8Pz+mAku6ybZhOxuQp7367Wv7M585zvPfLO7z87Md74jqooxxhhT3cJCHYAxxpjjkyUYY4wxAWEJxhhjTEBYgjHGGBMQlmCMMcYEhCUYY4wxAWEJxtR6IjJCRPIruUyaiLwcqJjcdWwSkbsCUO9lIlKp6wu826gqbXYsROQREXkrWOvzsX4VkctCsN4K21lEbheRKcGKKZgswdRiIjLW/eB4P34OdWyBUsYXxTigXQDWNVJEFolIvojkiMhSEXmiutcTIgFpM19EpDHwF6BWt52bJJcHoOr/AH1F5IwA1B1SEaEOwByzqcC1XtMOhCKQUFHV/cD+6qxTRG4A/gWMBv4H1AO6Af2rcz2hEog2K8dIYK6qbgj0ikQkUlVLAr2e6qSqxSLyIfAn4KdQx1OdbA+m9itW1UyvRzaAiAwSkRIRGVxaWERuEZFcEWnnvk4TkddF5CUR2es+nhGRMI9lGorIO+68/SIyVUS6ecwf4f7KHyoiy0WkQESmiUhbz0BF5EIRWSAiRSKyUUSeFJF6HvM3icgDIvKGG+M2Ebnbc7779BN3T2aT5/o9yrUXkckikunGslBELqhku14EfKaqb6jqOlVdoaqfqOpfvLbptyIyx22XPSLyuYhEexSJLmt73OUTReRNEdklInkiMl1E+nqV+YOIbBaRQhH5Akj1mv+rX9YVHZrx0WaPuP+734nIejeWSSKS7FEmQkRe8HifvCAir4lIWgVteTVw1CEgP9939UTkn267FYjIPBE5x2P+YPd9cL6IzBWRA8A5lK2JiHzptuNmEfm9V0z/EJHV7v9yk4g8Xfq/FJERwMNAN/nlSMEId16C2w4Z7nt7pYhc6VV3uZ8Nt30uEpH6FbRl7aKq9qilD2As8EUFZZ4CtgJJQBegALjOY34akAf8251/BZAD/MWjzGRgFTAQOBHnw7AViHHnjwBKcPamTgZ6AIuAbz3qOAfIBa4H2gNDgNXAsx5lNgF7gNuBDsAdgAL93fkp7uuRQBMgxWP9+R719ARucWPtAPwdZ6+ui9d2v1xOu70OrAHalVPmXOAgzqGfru523wXU93N7BJgBfOm2WwfgcbedmrplTgEOu9vQCbjZrVM94ngEWO4Vm3ebVPT6ESAfmOhuR39gM/CGR5n7gL3AcKAz8JL7Xkkrp42S3PhP85qeRsXvuw+An3Hed+3cdjwA9HTnD3bbcxnwG7dMShlxqNtuN7vt+Hc3rr4eZR4EBgBtgPOBLcDj7rwY4Fmcz0ET9xHj/g9nAivc90M74DzgEn8/G265+sAhYGiov1eq9Tsq1AHY4xj+eU6COeh+MXg+/ulRJhKYB3wGLATGedWRhvNFKh7THgC2uc87uh/OgR7zE90vg5Hu6xFumc4eZa5xvwzC3Nc/Ag96rftiN15xX28CPvIqsxZ4wOO1Apd5lRmBx5dlGW31s1c9aZSfYJoCs931rQXeB/4ARHqUmQl8XE4d5W4PcKa7/TFeZRYD97jPPwS+95r/XwKTYIqARI9pfwfWebzOAO7zeC04X7hp5bRBL7cN21byfdceJwG08lpuEvCq+3ywW/dwPz4rCvzHa9pU4P1ylrnFa/t9tfPZbpwnlFHHCCr4bHhMzwZurGhbatPDDpHVfj/ifIg9H8+UzlTnePTVwAVAY5xfcN5+Vvcd7poNNBeRBOAEnA/QbI86c3B+NXb1WKZYVVd7vN6Bk9wauK/7AH93D6Xlu4dnPgRicX4NllrqFdsON26/iUise3hjhXvoJR/oC7Tytw5VzVDV/jh7QS/ifJm+Acz1OIzRG+f8THnK254+OL9cs7zapTvOFyw47T/bqw7v19Vls/u//VWsIpKI83+aWzrTfc/Mq6DOGPdvkY955b3vTsJp8xVebfNbfmmbUvMriMGzfu/XR97D4vTOm+EeWs0HXqDi90xvIENVV5ZTpqLPRqn9/NJexwU7yV/7FarqugrKnIpzvq0BzmGmfZWoX8qZ5/nlcLCMeWEefx8FPvFRT5bHc+8TtErlzxU+i3O44i6cPYZC4F2cE/WVoqrLgeXAKyJyOs5J2Ctw9h79Ud72hAE7AV+9h3Ldv+W1f6nDPspF+hmfJ3/avrLDr+92/zbE2QPyV5i7rn4+4vLunFBQyZh+RUROBT7GeY+OxvmMXITzXip3UT+qr+izUSqJoz8LtZ7twRznRKQN8DJwG/A98IGIeP+wOEVEPD8opwI7VDUX59hyGB69p9xfmCe68/y1EOccyDofD+8PYHlKgPAKypwOvKuqE1R1KbCNX//qrYrS7Y1z/y4Chh5DfQtxTtgf9tEmuzzWearXct6vs4BUr/9hr2OI61fcPZtMnPMIALjr61fBoutxkmVXH/PKe98twvnybuKjbbZXcTN8tWPpnscAYLuqPq6q81R1LdDaq/wBfv3eWwg0FZETqhgT4HRMAaLd+o4btgdT+0WJSBOvaYdUNUtEwnHOHUxX1TdE5FOcQ1sP45zQLNUMeFFEXsVJHHfjXrOgqmtFZDLwhoiMwvll9yTOl8aHlYjzMeALEdkMjMf5VdcdOFlV76lEPZuAoSIyHefQw14fZdYAl7hxl+Bsb7SPcmUSkddwDmX8gJOgmuKcIygEvnOLPQl8LiLrcNpCcE42v6GqhX6sZirOeZzJInIPv5xAPheYqqo/4XSVniUi9wOf4px3uMSrnjScX79/E5GP3TKBuKjwJeAeEVmDk/huxmmXMvdMVPWwiEzFSfqfes0u7323RkQ+AMaKyF9xvniTcLZtg6p+VoX4LxWReTjtdRnOj4NT3HlrcA7PXYNz6Owc4Cqv5TcBrUXkJJwOAHk4h0jnABNEZLRbTwcgVlUnVSK2M3C2a23lN6vmsj2Y2u8snA+452ORO+9vOG/2GwFUdQ9wHXCfe7in1Ac4v8zm4Fz0NQbn+HOp63GOvU9x/9YHzlXnWgq/qOq3OMfPh7h1zMXplbTF/00F4K9uHVv5ZTu9/QXYhXM462ucE/yVvb7ge5wvn/E4XxoT3elnq+oaAFX9CufL/jw3lulubIf9WYF7/uF8nCT2H5xedeNxemjtcMv8jPP/uxXnfM6lOCebPetZ6c4f5ZY5G6f3YHV7FngPeBunTcFpF1/nVzy9CVzp/uDx5M/77m3gaZzk+wVOj7LNVYz/EZwecEtx2ut6VZ0HoKqf45y7fJFf2vAhr+UnAF/hJJUs4CpVPYzz/5+J82NuJU4iruzh2Ktw2uC4Utp7x9RR7jUMy1X19lDHYmofEVkIzFTVOyooNxun99d77us07H0HgIh0x0lanbw6WdR6dojMGOMXEWmNc+hoOs53xyica45G+bH4zTg9rsyvNQP+cLwlF7AEY4zx32Gca4GewTm8vgI4T1Ur7Cbsdrbw7rJtAFX9ruJStZMdIjPGGBMQdpLfGGNMQNghMleDBg20Q4cOoQ7Dp4KCAmJjY0Mdhk8WW9VYbFVjsVVNIGNbsGDBblVN8Tkz1GPV1JRHp06dtKaaNm1aqEMok8VWNRZb1VhsVRPI2ID5amORGWOMCSZLMMYYYwLCEowxxpiAsARjjDEmICzBGGOMCQhLMMYYYwLCEowxxpiAsARjjDEmICzBGGOMCQhLMMYYYwLCEowxxpiAsARjjDEmICzBGGOMCQhLMMYYYwLCEowxxpiACGqCEZEkEZkoIgUisllEri6n7GgRyRSRHBF5S0SiPObdLiLzRaRYRMb6WHaoiKwSkUIRmSYirQO0ScYYY8oQ7D2YV4ADQCpwDfCaiHTzLiQi5wD3AUOBNkA74FGPIjuAJ4C3fCybDHwGPAgkAfOBcRUFdlgrtyHGGGPKF7QEIyKxwHDgQVXNV9UZwBTgWh/FrwPGqGq6qu4FHgdGlM5U1c9UdRKwx8eylwLpqvqJqhYBjwA9RaRLefFtzTvMnA2+qjPGGFMV4tzxMggrEukNzFLVGI9pdwGDVPVCr7JLgKdUdZz7OhnIApJVdY9HuSeAFqo6wmPaS0A9Vb3VY9py4GFVneC1nlHAKIB6TTr06XfbSzxyWgwRYVJdm10t8vPziYuLC3UYPllsVWOxVY3FVjWBjG3IkCELVLWvr3kRAVmjb3FAjte0HCDej7Klz+PxvdfivWyWP+tR1TeBNwFim3XUbfnKpsjWjDyjXQWrCK60tDQGDx4c6jB8stiqxmKrGoutakIVWzDPweQDCV7TEoA8P8qWPvdV9ljWc0RSjLPX8uLUtezMLfJjNcYYY8oTzASzBogQkY4e03oC6T7KprvzPMvt9Dw8Vo6jlnXP/bQvYz1H1I8Qzu6aSn7xQZ78cqUfqzHGGFOeoCUYVS3A6d31mIjEisgAYBjwno/i7wI3ikhXEWkIPACMLZ0pIhEiEg2EA+EiEi0ipYf7JgLdRWS4W+YhYKmqrqooxocu6Ep0ZBhTluxg1vrdx7C1xhhjgt1N+Y9ADLAL+Ai4VVXTRaSViOSLSCsAVf0GeBqYBmx2Hw971PMAsB+nK/Pv3ecPuMtm4fRWexLYC5wC/M6f4Fom1ef2IR0AeGhyOiWHDh/TxhpjTF0WzJP8qGo2cLGP6VtwTs57TnseeL6Meh7B6X5c1nqmAuV2Sy7LTQPb8emCbazblc/bMzcyamD7qlRjjDF1ng0V4yUqIpxHh3UHnBP+GTn7QxyRMcbUTpZgfBjUKYXzujeh8MAhnvjCTvgbY0xVWIIpwwMXdCUmMpwvl2UwbdWuUIdjjDG1jiWYMjRvEMNfzu4EwAOTllN44GCIIzLGmNrFEkw5rh/Qhm7NEti+bz8vfL8m1OEYY0ytYgmmHBHhYfzj0h6ECYyZsZHl271HujHGGFMWSzAVOLFFIjcMaMthhfs+W8pBuzbGGGP8YgnGD6PP7kTzBjEs357L2FmbQh2OMcbUCpZg/BAbFcETFzvXxjz33Rq2ZheGOCJjjKn5LMH4aUiXxlzQoyn7Sw7xwKTlBOs+OsYYU1tZgqmEhy7sSkJ0BNPXZDFlyY5Qh2OMMTWaJZhKaBwfzd/OPwGAR6akszu/OMQRGWNMzWUJppKu7NeS0zsks7ewhIcmLw91OMYYU2NZgqkkEeH/Lj2R2HrhfLUsky+XZoQ6JGOMqZEswVRBy6T63O8eKnto8nL22KEyY4z5FUswVXT1ya3o364RewoO8PCUcu/GbIwxdZIlmCoKCxOevqwH9euF88XSDL5ZbofKjDHGkyWYY9AyqT73nefcOPOBScvZW3AgxBEZY0zNYQnmGP3+lNac0jaJ3fkHeORzO1RmjDGlLMEco9JDZdGRYUxevIOvltmhMmOMAUsw1aJ1o9gjF2D+beIyduUWhTgiY4wJPUsw1eTaU1szsFMK+wpLuPvTpTZWmTGmzrMEU01EhGcu60GD+pFMX5PF+3O2hDokY4wJKUsw1Sg1IZqnLjkRgCe/XMH6rPwQR2SMMaFjCaaanX9iUy7t3ZyiksP8ZdxiSuwOmMaYOsoSTAA8MqwbzRvEsGRbDi//sC7U4RhjTEhYggmAhOhInr28JyLw8rR1LNqyN9QhGWNM0FmCCZD+7Rsx8vS2HDqs/HncYvKKSkIdkjHGBJUlmAD6628607VpApv3FPLQZLvK3xhTt1iCCaDoyHD+dVVvYiLDmbhoOxMWbAt1SMYYEzRBTTAikiQiE0WkQEQ2i8jV5ZQdLSKZIpIjIm+JSJS/9YjIFSKyUkTyRGSFiFwcwM0qV4fGcTx6UTcAHpy8nA3WddkYU0cEew/mFeAAkApcA7wmIt28C4nIOcB9wFCgDdAOeNSfekSkOfA+8BcgAbgb+FBEGgdmkyp2ed8WXNizGYUHDnHHR4soPngoVKEYY0zQBC3BiEgsMBx4UFXzVXUGMAW41kfx64AxqpquqnuBx4ERftbTAtinql+r40ugAGgfwM0rl4jw5CXdaZkUQ/qOXP759epQhWKMMUEjwRozS0R6A7NUNcZj2l3AIFW90KvsEuApVR3nvk4GsoBkoFV59YhIOPAD8BzwJXAh8DLQWVULvNYzChgFkJKS0mf8+PHVvNVH27DvEE/OKeKQwp9PiqJX4wi/lsvPzycuLi6gsVWVxVY1FlvVWGxVE8jYhgwZskBV+/qcqapBeQBnAJle024C0nyUXQ+c6/E6ElCcw2UV1gPcCOQDB4FC4LcVxdepUycNhtfS1mnre7/Q3o99pxn79vu1zLRp0wIb1DGw2KrGYqsai61qAhkbMF/L+F4N5jmYfJxzIp4SgDw/ypY+z6uoHhE5C3gaGAzUAwYB/xWRXlUPvfqMOqMdZ3RMJrvgAHd8tNCGkjHGHLf8TjAicp6IfOH2ymrpThspIkP9rGINECEiHT2m9QR8XSCS7s7zLLdTVff4UU8v4EdVna+qh1V1HjAHOMvPOAMqLEx44cpepCZEMW/TXp751s7HGGOOT34lGBG5BhgPrAXa4hyyAggH7vGnDnXOf3wGPCYisSIyABgGvOej+LvAjSLSVUQaAg8AY/2sZx5wRukei3vu5wxgqT9xBkNyXBQvX30S4WHCmz9u4JvlmaEOyRhjqp2/ezD3ADep6mic8xqlfsbZY/DXH4EYYBfwEXCrqqaLSCsRyReRVgCq+g3OYa5pwGb38XBF9bjLTgceAT4VkTxgAk6Hge8qEWfA9WuTxP3ndQHg7k+WsGl3QQVLGGNM7eJfNyboCMz2Md3X+ZAyqWo2cLGP6VuAOK9pzwPPV6Yej/kv4/Qcq9FuPL0t8zZl8236Tm79YCET/3ga0ZHhoQ7LGGOqhb97MDuATj6mD8Tp8WWqQER45vKetGlUn5UZuTxs45UZY44j/iaYN4F/uec7AFqKyHU4h7FeC0hkdURCdCSvXtOHqIgwxs3fyvj5W0MdkjHGVAu/EoyqPo1zYv17IBbn3MjrwOuq+krgwqsbujZL4PGLuwPw4KTlLN+eE+KIjDHm2PndTVlV/45zJf3JwKlAiqo+GKjA6por+rbkd/1aUnzwMKPenc/u/OJQh2SMMcfE327Kb4lIvKoWuteXzFXVfLeb8FuBDrKueHRYN3q3asCOnCJu+8AuwjTG1G7+7sFch9Mt2FsM8IfqC6dui4oI5/Xf96FxfBRzNmbzxBcrQh2SMcZUWbkJxr3vSiNAgIbu69JHCnABsDMYgdYVqQnRvH5tH+qFh/HO7M2Mn2cn/Y0xtVNFezC7cS5mVGAFzojGpY9M4L/Aq4EMsC46qVVDHr/YuU3OA5OWs26f3T/GGFP7VHSh5RCcvZcfcO7Bku0x7wCwWVV3BCi2Ou3Kfq1I35HLu7M38/KiYi46s4jGCdGhDssYY/xWboJxh11BRNoCW1XVzjoH0YMXdGVVZh5zN2Zz8/sL+OimU+1Kf2NMreHvdTCbVfWwiDQTkVNFZKDnI9BB1lWR4WG8es1JJEULi7bs455Pl5be78YYY2o8v8YiE5FmwIc4Q8MozmEzz286+1kdIMlxUYzuE80/5h1gypIdtEuJ5c9n+Rq1xxhjahZ/uym/CBwCuuLcIfIM4HJgJXBuQCIzR7SMD+PfV/cmTODFqWuZvHh7qEMyxpgK+ZtgBgH3quoqnD2XLFX9DLgXeDxQwZlfnNkllQd+2xWAuz9dyoLN2RUsYYwxoeVvgonB6bIMTk+yxu7zFUCP6g7K+Hb9gDb8/tRWHDh4mFHvLmBrdmGoQzLGmDL5m2BWAV3c54uBW0SkNXAbYMdrgkREePjCbpzRMZk9BQe4Yew8cotKQh2WMcb45G+CeQlo4j5/DPgNsAHnzpJ/C0BcpgyR4WG8fPVJdGgcx9pd+dz2wUIOHLTe48aYmsffbsofqOpY9/lCoA3QD2ilqp8ELDrjU2JMJG9d149GsfX4ae1u7ptg3ZeNMTWP38P1e3JHVV4IFIjIfdUck/FDq0b1efv6ftSvF85ni7bzzLerQx2SMcYcpcIEIyLJIvJbEfmNiIS70yJF5M/AJuCuwIZoytKjRQNeueYkwsOEV9PW8+7sTaEOyRhjjqhoNOXTgLXA58DXwEwR6QIsBW7H6aLcKtBBmrIN6dyYf1x6IgAPT0nnm+WZIY7IGGMcFe3BPA58i9MV+SWcu1l+Afwf0FFVX1ZV6ysbYpf3bcldv+mEKvzp40XM22TXyBhjQq+iBNMTeFxVlwMP4Fxkeb+qvqt2VrlGuW1IB645xblGZuQ781m7My/UIRlj6riKEkwSzr1fcPdUCoFFgQ7KVJ6I8Niw7pzdNZWc/SVcO2auXYhpjAkpf3qRNfS4s6UCCV53tkwKcIzGT+Fhwr9+15uT2ySRmVvEtWPmsCuvKNRhGWPqKH8STOmdLHcBccA8frmr5W73r6khYuqF898RfenePIFNewr5w5i57Cs8EOqwjDF1kD93tDS1TEJ0JO9cfzJXvDGbVZl5jHh7Hh+MPIXYKL/uzmCMMdXCrztamtqnUVwU7488hctem83irfsY9d58xlzXz+6IaYwJmipdyW9qh6aJMXww8hSS46KYuW4Pd3y0iIOHbNwyY0xwBDXBuJ0CJopIgYhsFpGryyk7WkQyRSRHRN4SkSh/6xGR+iLyqojsdpf/MZDbVZO1SY7l/ZEnkxgTyfcrdvLXT5Zw6LD1MDfGBF6w92BeAQ4AqcA1wGsi0s27kIicA9wHDMUZWLMd8Ggl6nkTp4v1Ce7f0dW9IbVJlyYJvH19P2LrhTN58Q7utiRjjAmCoCUYEYkFhgMPqmq+qs4ApgDX+ih+HTBGVdNVdS/OiAIj/KlHRDoDFwGjVDVLVQ+p6oIAb16Nd1Krhoy94eQjg2PeO2Ephy3JGGMCSIJ1Qb6I9AZmqWqMx7S7gEGqeqFX2SXAU6o6zn2djNMdOhln7LMy6xGRPwB3A1Nxkk4G8IiqTvAR0yhgFEBKSkqf8ePHV+cmV5v8/Hzi4uKqpa7V2Yd4bkERBw7BwBYRjOhWjzCRGhFbdbPYqsZiq5q6GtuQIUMWqGpfX/P86rcqIm+VMUuBImAdME5Vd5RTTRyQ4zUtB4j3o2zp83g/6mkBdAcmAM2A/sCXIrJCVVceFbzqmziH0+jcubMOHjy4nPBDJy0tjeqKbTDQo+cerh87lx+3HaR5s2Y8eXF3wsKqlmSqM7bqZrFVjcVWNRbbr/l7iCwFuBS4GOjgPi52p3UG7gFWi0ivcurIBxK8piUAvgbN8i5b+jzPj3r2AyXAE6p6wO1qPQ3nLpwG6N++EWOu60dURBgfzd3CQ1OW2w3LjDHVzt8EMxNnuP4WqjpQVQfi7Cl8BXwHtAa+BJ4rp441QISIdPSY1hNI91E23Z3nWW6nqu7xo56lfm5TnTagQzL/+UNf6kWE8f7PW3hw8nI7J2OMqVb+Jpg7gcc8h+Z3nz8JjFbVA8A/gV5lVaCqBcBnwGMiEisiA4BhwHs+ir8L3CgiXUWkIc5IzmP9rOdHYAtwv4hEuPMH49x2wHgY2CmFN6/tcyTJ3DthqfUuM8ZUG38TTBzQ1Mf0Ju48gFwqPqfzRyAGZ1yzj4BbVTVdRFqJSL6ItAJQ1W+Ap3EObW12Hw9XVI+7bAlOwjkf59zMf4A/qOoqP7e1ThncuTFvXdePmMhwPlmwjT+PW0yJXYxpjKkG/g5ONREYIyL34Ax2qTg3H3saZ28C9/Wa8ipR1Wycczfe07fwS6IqnfY88Hxl6vGYn45zct/44fSOybxzw8ncMHYeny/ZQXHJIf59dW+iImxYGWNM1fm7B3MLziGm94H1wAb3+Tc4exMAK4GbqjtAExwnt03i/ZGnkBAdwXcrdjLq3QUUlRwKdVjGmFrMrwSjqoWqegvOVfG9gZOAJFW91T0ngqouVtXFAYvUBFyvlg34aNSpJMXWY/qaLK5/ex4FxQdDHZYxppaq1JX8qlqgqktVdUlpYjHHl27NEhk36lQax0cxe8MervnvHPYW2P1kjDGV51eCEZFoEblXRL4TkcUistTzEeggTXB1TI1n/M39ad4ghsVb93H5G7PZsW9/qMMyxtQy/u7BvIoz+OQmYBLOVfKeD3OcaZMcy4RbT6NTahzrduUz/LVZrNvl65pYY4zxzd9eZBcDl6vq1ADGYmqYJonRjL+5Pze+M58Fm/dy2euzeXtEP3q3ahjq0IwxtYC/ezCFwNZABmJqpgb16/H+jacwtEtj9hWWcPV/5pC2eleowzLG1AL+Jpingb+IiN0Bsw6KqRfO69f2YfhJLdhfcoiR78xn0qLtoQ7LGFPD+XuI7GzgDOBcEVmBM5jkEap6UXUHZmqWyPAwnr28B8lx9Xjjxw38edxitu/bT1dsaBljjG/+JpjdOFfzmzpMRLj//BNIiY/iya9W8sy3qxnYIoLTBx4mMtx2bo0xR/Mrwajq9YEOxNQeI89oR4uG9fnzuEX8uO0g1789j1d/fxIJ0ZGhDs0YU4PYz05TJed2b8LHo/qTUA9mrNvNZa/NYtvewooXNMbUGWUmGPciyobu82XeF1fahZamV8sGPHhqDB0ax7FmZz6XvDqLZdu8bzZqjKmryjtENgEodp9/GoRYTC2UUj+MCbecxi3vL2D2hj1c8cZsnr+iJ+ed6OvuDsaYuqTMBKOqj/p6boy3xPqRvHPDydz/2TImLNzGrR8s5M6hHblzaEfCwiTU4RljQsTOwZhqUS/C6cb8t/O7ECbw0v/W8scPFtpozMbUYf4OdpkkIq+JyBoR2SciuZ6PQAdpagcRYdTA9owZ0Y/46Ai+Sc9k+Guz2JptJ/+NqYv8vQ5mDM59YN4EdoBdXWfKNqRzYybdNoCb3pnPqsw8hr0yk1evOYlT2zUKdWjGmCDyN8EMBc5W1TmBDMYcP9qnxDHxtgH86aNFTF+Txe//O4eHLuzKtae2RsTOyxhTF/h7DmYXkB/IQMzxJzEmkrdG9GPUwHYcPKw8NDmd0eMWU3jAzssYUxf4m2D+DjwmInGBDMYcf8LDhL+dfwL/uqo39euFM2nxDi5+ZSbrs+z3ijHHO38TzAPAb4BdIrLSLrQ0lXVRz2ZMvm0A7VNiWbMzn2Evz+TrZRmhDssYE0D+noOxCy3NMeuYGs/k20/n3glL+XJpBrd+sJCbzmjLPed2scEyjTkOVZhgRCQSiAVeUdXNgQ/JHM/ioiJ4+are9GnVkKe+Wsl/ftrIkq05vHRVL5omxoQ6PGNMNarwZ6OqlgC3Atb1x1QLEeGG09vy8ahTSU2IYu6mbM5/6SemrtgZ6tCMMdXI3+MS3wFnBjIQU/f0bZPEl386g8GdU9hbWMLId+fzyJR0ig8eCnVoxphq4O85mP8BT4lID2ABUOA5U1U/q+7ATN2QHBfFW9f1Y8yMjTz97SrGztrE3I3Z/Pvq3rRPsU6LxtRm/iaYl92/f/IxT4Hw6gnH1EVhYcJNA9txSrsk7vhoESsycrnw3zN4bFh3hp/U3C7MNKaW8usQmaqGlfOw5GKqRY8WDfjijtMZ1qsZhQcOcdcnS7jz48XkFJaEOjRjTBVY31BTo8RHR/Lilb145rIexESGM2XJDs558UdmrN0d6tCMMZXkd4JxR1S+WkTuE5GHPB+VrGOiiBSIyGYRubqcsqNFJFNEckTkLRGJqmw9IvKwiKiInOVvjCb0RITL+7bkqzvPoHerBmTmFvH7MXN4ZEo6+w9YBwBjagt/h+s/FVgLPAs8DtyAM3zMXcBllVjfK8ABIBW4BnhNRLr5WN85wH04g2y2AdoBnjc9q7AeEWnvxmaXi9dSbZNj+eTm/tx9TmciwoSxszbx23//xJKt+0IdmjHGD/7uwTwDfAA0B4pwuiy3AuYD//SnAhGJBYYDD6pqvqrOAKYA1/oofh0wRlXTVXUvTlIbUcl6XgbuxUlEppaKCA/jtiEdmHTbADo2jmNDVgGXvjaLF75fQ8mhw6EOzxhTDlGt+NYuIpID9FPVNSKyD+ivqitFpB/woap29KOO3sAsVY3xmHYXMEhVL/QquwR4SlXHua+TgSwgGSexlVuPiFwO/F5Vh4nIJmCkqk71EdMoYBRASkpKn/Hjx1fYFqGQn59PXFzN7LIbzNgOHFImrD3Ad5sOokCr+DBuPLEerRN89zOxdqsai61q6mpsQ4YMWaCqfX3N87ebsudewE6gNbASZwj/Zn7WEQfkeE3LAeL9KFv6PL6ietwRn5/CGZyzXKr6Js5N1OjcubMOHjy4okVCIi0tDYvN8ZuhMHv9Hu7+dAlb9u7nsZ+LuXlgO/40tCPRkUcnGmu3qrHYqsZi+zV/D5EtBPq5z9OAJ0TkOuBfgL+jKecDCV7TEoA8P8qWPs/zo55HgfdUdaOfcZlapn/7Rnz754FcP6ANh1V5NW09v/3XTyzYnB3q0IwxHipzP5gd7vMHcA5X/RtoiHuIyQ9rgAgR8Tyc1hNI91E23Z3nWW6nqu7xo56hwJ/cHmiZQEtgvIjc62ecphaIjYrg4Qu78ekt/WmfEsv6rAIue302j0xJp6DYbmhmTE3g74WW81V1mvs8S1XPU9UEVe2rqsv8rKMA+AznxmWxIjIAGAa856P4u8CNItJVRBriJLWxftYzFOgO9HIfO4CbcXqemeNMn9bOeGa3DWlPmDg9zc558Uemrd4V6tCMqfMqdaGliPQVkSvdnly4X/D+nscB+CMQg3ML5o+AW1U1XURaiUi+iLQCUNVvgKeBacBm9/FwRfW4y+5R1czSB3AI2KuqdgvF41R0ZDh3n9OFybcNoGvTBLbt3c/1b8/j5UVFZOTsD3V4xtRZfiUHEUnF6QrcD2fssY7ABuB5nG7Ld/pTj6pmAxf7mL4F5+S957Tn3fr9rqeMsm38KWdqv+7NE5l8+wDGztzEC1PXMH/nIc56bjqjz+7EiNPaEGE3NTMmqPz9xL0AZAKNgEKP6Z/gR28tY4IlMjyMmwa2Y+pfBtEnNZyCA4d44suVXPDvGSzYvDfU4RlTp/ibYIYCf3cvevS0Hue6FGNqlGYNYrijdzRvjehLi4YxrMrMY/hrs7hvwlKyC+zaW2OCwd8EE4PvK+JTcA6RGVMjndklle9HD+K2Ie2JDBc+nreVwc9M4+2ZG20kAGMCzN8E8yPuUC0uFZFwnKFY/lfdQRlTnWLqOZ0Avr7zDM7omExu0UEe/XwF5730E9PXZIU6PGOOW/4mmHuAm0TkeyAKeA5YAQwA7g9QbMZUqw6N43n3hpP5zx/60qZRfdbtyue6t+Zy49h5bMiyTobGVDd/r4NZAZwIzAK+A6JxTvD3VtX1gQvPmOolIpzdNZVvRw/k/vO6EBcVwf9W7eKcF3/kyS9XkFtkNzczprr43W/Tva7kYVW9QFXPV9UHgHoiUjNHiDSmHFER4dw8qD0/3DWIK/q24OBh5T8/bWTQ09MYM2MjxQftvjPGHKtjvTCgAc7Q+cbUSo3jo3n6sp5Mue10Tm6TxN7CEh7/YgVDn5vOpEXbOXy44tHGjTG+2ZVnxgAntkhk3M2nMua6vnRKjWPb3v38edxiLnx5Bj+ttY4AxlSFJRhjXCLC0BNS+frOgTw9vAdNEqJJ35HLtWPmcu2YOSzf7n2XCGNMeSzBGOMlPEy4ol9Lpt01mHvO7Ux8dAQ/rd3NBf+ewS3vLWBVZm6oQzSmVih3LDIRmVLB8t73ZTHmuBFTL5w/Du7AVf1a8cq0dbz382a+Sc/km/RMftujKaPP6kiHxr7ul2eMgYoHu9zjx3y7sZc5rjWMrccDF3TlpoHteC1tPR/O2cKXSzP4alkGw3o2486zOtE2OTbUYRpT45SbYFT1+mAFYkxNl5oQzSMXdWPUwHa8mraOcfO2MmnxDj5fmsElvZtz+5AOtLFEY8wRdg7GmEpq1iCGJy4+kR/+Opjf9WsJwKcLtnHmc2nc8dEiVmbYORpjwBKMMVXWMqk+/xjegx/+Oogr+7YkPEz4fMkOznvpJ24YO48Fm7NDHaIxIWUJxphj1LpRLP+8rAfT7x7C9QPaEB0Zxg+rdjH8tdlc+cZsflyThapdsGnqHkswxlSTZg1iePjCbsy890zuOLMD8dERzNmYzR/emstFL89k8uLtdosAU6dYgjGmmjWKi+Kvv+nMrPvO5N5zu5AcV49l23O48+PFDHx6Gm9MX09Bie3RmONfRd2UjTFVFB8dya2D23P9gDZ8tnA7Y2ZsYH1WAf/39Sqiw2F+UTrXn9aWVo3qhzpUYwLC9mCMCbDoyHCuPqUV348exNsj+jGgQyOKDsHbMzcx+Nlp3Pr+AuZvyrbzNOa4Y3swxgRJWJgwpEtjhnRpzLtT/seSomSmLNnO18sz+Xp5Jl2bJnBt/9YM69WM+vXso2lqP9uDMSYEWiWE89wVPZl575ncPqQDSbH1WJGRy/2fLeOUp/7Ho5+ns97usmlqOUswxoRQ44Ro7jqnM7PvP5MXruzJSa0akFd0kLdnbmLoc9P5/X/n8M3yTA5a7zNTC9l+uDE1QFREOJf0bsElvVuwfHsO7/+8mUmLtzNj3W5mrNtN08RoLu/Tgsv7tqRlknUKMLWD7cEYU8N0b57IP4b3YM7fzuKhC7rSLjmWjJwi/vXDOs54ehrX/PdnJi/eTlGJ3dbZ1Gy2B2NMDZUYE8kNp7dlxGlt+HnjHsbP28rXyzOZuW4PM9ftITEmkot7NeOKfi3p1iwx1OEa8yuWYIyp4cLChNPaJ3Na+2QeLSxhypLtjJu/leXbc3ln9mbemb2Z7s0TuLxPSy7o0ZRGcVGhDtkYwBKMMbVKYv1Iru3fhmv7tyF9Rw7j521l4qLtLN+ey/Lt6Tz+xQoGdkrhkt7NOeuEVGLqhYc6ZFOHBfUcjIgkichEESkQkc0icnU5ZUeLSKaI5IjIWyIS5U89InKqiHwvItkikiUin4hI00BvmzHB1q1ZIo8O687cv5/FS7/rxZDOKSjww6pd3PHRIvo9OZW7PlnCzHW7OXTYLuI0wRfsPZhXgANAKtAL+FJElqhqumchETkHuA84E9gBTAQedadVVE9D4E3gW+Ag8DLwNnBuIDfMmFCJjgxnWK/mDOvVnN35xXyxZAcTF+9gydZ9fLpgG58u2EZqQhTDejXnwh7N6N48AREJddimDghaghGRWGA40F1V84EZIjIFuJZfEkep64AxpYlHRB4HPgDuq6geVf3aa70vA9MDuGnG1BjJcVGMGNCWEQPasiErn0mLdzBp0Xa2ZBfy5o8bePPHDbRMiuH8E5vy2xObcmLzREs2JmAkWOMfiUhvYJaqxnhMuwsYpKoXepVdAjylquPc18lAFpAMtPK3Hnfen4HfqeqpPuaNAkYBpKSk9Bk/fvwxb2cg5OfnExcXF+owfLLYqiaYsakq6/cdZnbGQebvPERO8S+f+ZQYoV+TCPo1CadNQhgiYu1WRXU1tiFDhixQ1b6+5gXzEFkckOM1LQeI96Ns6fP4ytQjIj2Ah4BhvgJS1TdxDqfRuXNnHTx4cLkbECppaWlYbJVnsf1iCDASOHRYmb8pm6+WZfDV8kyy8or5amMJX20scfZsujchpeQQNwwcRFhYzduzsf9p1YQqtmAmmHwgwWtaApDnR9nS53n+1iMiHYCvgTtV9acqxmzMcSU8TDilXSNOadeIhy7sdlSy2Zq9nzd+3ADA6+n/46wTGnN211QGdEgmOtJ6o5nKC2aCWQNEiEhHVV3rTusJpPsom+7OG+9Rbqeq7hGRoorqEZHWwFTgcVV9LwDbYkyt5yvZfL08k88XbmZ3fjEfz9vKx/O2EhMZzhkdkzm7aypDT0glKbZeqEM3tUTQEoyqFojIZ8BjIjISp/fXMOA0H8XfBcaKyAdABvAAMNafekSkOfAD8Iqqvh7IbTLmeOGZbAbF7yK1cx++X7GTqSt3smx7Dt+t2Ml3K3YSJtC3dRJnntCYwZ1T6Jwab50ETJmC3U35j8BbwC5gD3CrqqaLSCtgBdBVVbeo6jci8jQwDYgBJgAPV1SPO28k0A54WESOLKOqNfPsmzE1jIjQtVkCXZslcOdZHcnI2c9UN8H8vGEPczdlM3dTNv/4ehVNE6MZ1CmFQZ1SGNAxmYToyFCHb2qQoCYYVc0GLvYxfQvOyXvPac8Dz1emHnfeozjXzBhjqkHTxJgjowfkFZUwfU0WaauzmL4mi4ycoiOH0iLChJNaN2RQpxQGd06ha1O73qaus6FijDF+i4+O5IIezbigRzMOH1ZWZOQyfU0W01dnsWDLXuZuzGbuxmye+XY1jeOjGNAhmdPaN2JAh2SaNYipeAXmuGIJxhhTJWFhQvfmiXRvnshtQzqQs7+EWet2k7Y6i7Q1u9iZW8zERduZuGg7AG0a1ee0DskMaJ9M//aNrLNAHWAJxhhTLRJjIjnvxKacd2JTVJU1O/OZuW43s9bvZs6GbDbtKWTTni18OGcLACc0TWBA+0ac1qERfdsk2fmb45AlGGNMtRMROjeJp3OTeG44vS0HDx1m2fYcZq3fw8x1u5m/eS8rM3JZmZHLf2dsJEychNOvTRInt02iX5skUuLttgO1nSUYY0zARYSH0btVQ3q3ashtQzpQVHKIhZv3MnP9bmat38OybTmk78glfUcuY2dtAqBtciz92jQ8knRa2a2iax1LMMaYoIuODOe0Dsmc1iEZgP0HDrFo617mbdzLvE3ZLNyyl427C9i4u4Dx87cB0Dg+itaxB1kXvoHerRrQrVmijTBQw1mCMcaEXEy98CN37QQoOXSYFTtymbfJ6ZU2f/NeduUVsysP5n25EoCIMOd6nd4tG9CrVQN6t2xI60b1rWt0DWIJxhhT40SGh9GzZQN6tmzAyDPaOSNCZ+Xz4Xc/s79+Kou27GP1zjyWbsth6bYc3pm9GYCG9SPp1bIBvVs1pGfLBpzYPNF6q4WQJRhjTI0nInRoHM/AFpEMHtwDgPzigyzduo9FW/exaMs+Fm/dy+78A0xbncW01VlHlm3eIIZuzRI4sXki3Vsk0r1ZonUgCBJLMMaYWikuKuKo8ziqyra9+1m0dR8LN+9l2fYcVuzIZfu+/Wzft5/vVuw8smyThGi6N0+ge/NEJ/E0T6RxfJQdXqtmlmCMMccFEaFlUn1aJtXnop7NAOf+Nxuy8lm+I4dl23JZvsNJOpm5RWTmFjF15a4jyyfF1qNLk3i6NElw/jaNp2PjeGLqWUeCqrIEY4w5boWHCR1T4+mYGs8lvZ1phw8rm/YUsGy70zXa6SKdQ3bBAWat38Os9XuOLB8m0KZRLF2aOomnc5N4TmiSQIuGMTXyhmw1jSUYY0ydEhYmtEuJo11KHMN6NQecw2sZOUWsysxlZUYeqzPzWJWZy/qsAjbsdh5fLcs8UkdsvXA6NI6jfeM4OjSOo0NKHNkFhzl46DAR4WGh2rQaxxKMMabOExGaNYihWYMYzuySemR68cFDrNuV7yacPFZm5LIqM4+svGKWbMthybaj797+0KxvaZsceyT5dHQTUNvk2Dp5zY4lGGOMKUNURDjdmiXSrVniUdOzCw6wblc+63bls3ZXHut25ZO+dQ/ZRYdZvTOP1TuPvhN8mECLhvVpkxxL20bO3zaNYmmTHEuLhjFEHqd7PZZgjDGmkpJi63FyW2cIm1JpaWn0638667PyWbszn3VZTgJavyufzdmFbHEfP3rVFR4mtGwYQ+tGsbRNjqVNo/q0To6lbSMn+dTmQ26WYIwxpprERkXQo0UDerRocNT04oOH2LKn0BlRencBG/cUsGl3AZv3FLIjZ7870nQh09dkHbVcRJjQtEE0LRvWdx5JMbRw/7ZsWJ/kuKga3dnAEowxxgRYVET4kd5s3opKDrElu5CNuwvYvKeAjbudJLRpTwEZOUVszd7P1uz9OHeH9643jOYNnWTTomGM003bfd68YQyNYuuF9NoeSzDGGBNC0ZHhdEqNp1MZyWfb3v1s3VvItr372ZZdeOT51uxC9haWsCGrgA1ZBT7rrhceRpPEaGK0iMk7F9M0MZqmDWJolhhN08QYmiZG06B+ZMCSkCUYY4ypoaIjne7QHRrH+ZyfV1TiJB434Xgmn4ycInL2l7AluxCA1Xu3+6wjJjLcTTxO0mmWGE1qYjSp8dE0TogiNSGaRrH1qnQuyBKMMcbUUvHRkZzQNJITmib4nF9QfJCMnCK++XEOjVt3IiOniIyc/ezIKSJj334ycorILz545FqfsoQJNIqLIjUh6kjiaRwfTWpCdLnxWYIxxpjjVGxUBB0ax9E9OZzB/Vr6LJNXVEJGThE73ISTsW8/O3OL2ZVXdOTv7vwDZOUVk5VXzHJy/V6/JRhjjKnD4qMjiY+O9HkOqFTJocPszi9mZ24xO3OLnHvz5BaxM7eIZ8qp2xKMMcaYckWGh7mdAmJ+Na+8BFN7r+AxxhhTo1mCMcYYExCWYIwxxgSEJRhjjDEBYQnGGGNMQFiCMcYYExBBTTAikiQiE0WkQEQ2i8jV5ZQdLSKZIpIjIm+JSJS/9YjIUBFZJSKFIjJNRFoHcruMMcb8WrD3YF4BDgCpwDXAayLSzbuQiJwD3AcMBdoA7YBH/alHRJKBz4AHgSRgPjAuMJtjjDGmLEFLMCISCwwHHlTVfFWdAUwBrvVR/DpgjKqmq+pe4HFghJ/1XAqkq+onqloEPAL0FJEugds6Y4wx3oJ5JX8n4JCqrvGYtgQY5KNsN2CyV7lUEWkEtKqgnm7uawBUtUBE1rvTV3muRERGAaPcl8UisrzSWxUcycDuUAdRBoutaiy2qrHYqiaQsZV5CiKYCSYOyPGalgP4GgDHu2zp83g/6okDssqZf4Sqvgm8CSAi81W1b/mbEBoWW9VYbFVjsVWNxfZrwTwHkw94jymdAOT5Ubb0eZ4f9VRmPcYYYwIkmAlmDRAhIh09pvUE0n2UTXfneZbbqap7/KjnqGXdczbty1iPMcaYAAlaglHVApzeXY+JSKyIDACGAe/5KP4ucKOIdBWRhsADwFg/65kIdBeR4SISDTwELFXVVd4r8fLmsW1hQFlsVWOxVY3FVjUWmxdR1eCtTCQJeAs4G9gD3KeqH4pIK2AF0FVVt7hl/wLcC8QAE4BbVLW4vHo81nMW8DLOyac5wAhV3RSUjTTGGAMEOcEYY4ypO2yoGGOMMQFhCcYYY0xgqGqdfuAMJzMRKAA2A1cHeH1pQBFOd+p8YLXHvKE4F4MWAtOA1h7zBPgnzjmnPcDTuIc43flt3GUK3TrO8iOW23GG0ikGxnrNC1gswNVuWxcAk4Akf2Nz61aP9svHGdUhKLEBUcAYt0wesAg4rya0W3mxhbrd3DLvAxlALk5v0JE1od3Ki60mtJtH2Y443x3v15R2q/A7prILHG8P4COcscrigNNxLsrsFsD1pXl+sDymJ7vrvhyIxrnV9c8e828GVgMtgOY4nSJu8Zg/G3gep1PEcGAfkFJBLJcCFwOvcfSXeMBiwRlRIQ8Y6Lb5h8DHlYitDc4HPqKMbQpobEAszvBDbXCOAFzgLtMm1O1WQWwhbTePclHu8y5AJtAn1O1WQWwhbzePur4DfsJNMDWh3Sr8vgvUF2lteOB8IA8AnTymvQf8I4DrTMN3ghkFzPKKbT/QxX09CxjlMf/G0jcTzjA8xUC8x/yfPN9MFcT0BEd/iQcsFuAp4EOPee3d/0G8n7FV9IEPWmwe5Za6H9Aa024+YqtR7QZ0xtljuKKmtZtXbDWi3YDfAeNxfkCUJpga1W6+HnX9HExZ46P9aoTnavZ/IrJbRGaKyGB32q/GUANKx1D71XyvOLsBG1Q1r4z5lRXIWLzrXo+b5CsZ42YR2SYib7sjaPuMPdCxiUiqOz/dx/IhbTev2EqFtN1E5FURKT0kkwF85WP5kLRbGbGVClm7iUgC8BjwV6+Qa0S7laeuJ5jKjI9WXe7Fuf1Ac5yLnz4XkfZ+xOJrfLY4ERE/lq2sQMZyrLHuBvrhXOPUx13ug3JiD1hsIhLprvsddS7krTHt5iO2GtFuqvpHd/oZOBdMF1eh/mDGVhPa7XGc0eW3ek2vEe1WnrqeYII+bpmqzlHVPFUtVtV3gJnA+X7E4mt8tnx19l+rezsCGcsxxarOLRrmq+pBVd2J0xngN+6vvKDFJiJhOIdTD7gx+LN8yGKrKe3mxnJIndtstABurUL9QYst1O0mIr2As4AXfIRbY9qtLHU9wVRmfLRAUZzeHhWNoeZrfDbPee1EJL6M+ZUVyFi8626H0/vJ8zBlZWhpVcGKzf0FOAbnhnfDVbWkjOWD3m7lxOYt6O3mQwS/tE9Ne7+VxuYt2O02GOc80BYRyQTuAoaLyEIfy9eEdjtaZU7YHI8P4GOcnmSxwAAC2IsMaACcg9PjIwLnbpwFOCcVU9x1D3fn/5Oje4TcAqzEObTWzH0DePYI+Rl41l32EvzrRRbhlv8/nF+8pXEFLBacY7u5OIchYnG6h/rq1VNWbKe47RUGNMLpATgtyLG97tYT5zW9JrRbWbGFtN2AxjgnquOAcJzPQQHOOIIhbbcKYgt1u9UHmng8ngU+ddss5O+3Cr/zAvFFWpseONfBTHLfUFsI4HUw7htiHs5u5j73H3y2x/yzcE4w7sfpbdbGY57g9GPPdh+++rSnucuuxr/rYB7B+UXm+Xgk0LHg9K/f4rb5ZHxfl+AzNuAqYKO7bAbOwKhNghUbzrF45ehrmfKBa0LdbuXFVgPaLQWYjvO+zwWWATcF471/LLGFut3K+Fy8XxPazZ+HjUVmjDEmIOr6ORhjjDEBYgnGGGNMQFiCMcYYExCWYIwxxgSEJRhjjDEBYQnGGGNMQFiCMeY4IyIjRCQ/1HEYYwnGmAARkbEioh6P3SLyhYh0qUQdj4jI8kDGaUygWIIxJrCmAk3dx29wbu40MaQRGRMklmCMCaxiVc10HwtxRsXtIiIxACLyDxFZLSL7RWSTiDwtItHuvBHAw0A3j72gEe68BBF5TUQyRKRIRFaKyJWeKxaRoSKyXEQKRGSaiLQN5oYbExHqAIypK9yRa68ElqnqfndyAXADsB3oijNYZTHwIM7Ait1xbn082C2f446Y/DXQELgeZ4TbzjiDFpaKAu536y4C3nHrPicwW2fMr1mCMSawzvU44R4LbMW5/w8Aqvq4R9lNIvIUzpDsD6rqfnfZg6qaWVpIRM4G+uOM+r3SnbzBa70RwG2qutpd5lngbREJU9XD1bh9xpTJDpEZE1g/Ar3cxynAD8B3ItISQEQuE5EZIpLpJpMXgFYV1NkbyPBILr4UlyYX1w4gEueWEcYEhSUYYwKrUFXXuY+5wI04dwYcJSKn4tyP6FvgQpzE8QBOIiiPVDAf4KDX69Jh0+0zb4LGDpEZE1wKHMa5kdQAYLvnYTIRae1V/gDOTbA8LQSaisgJFezFGBNSlmCMCawoEWniPm+Ic0/3OOBzIB5oLiLXALNxTsBf5bX8JqC1iJyEc/OnPOB/wBxggoiMxjnJ3wGIVdVJAd0aYyrBdpeNCayzcO6EmIGTFPoBl6tqmqp+DjwDvAgsBc4GHvJafgLwFU5SyQKuck/SnwfMxLmV7UrgJaBeoDfGmMqwO1oaY4wJCNuDMcYYExCWYIwxxgSEJRhjjDEBYQnGGGNMQFiCMcYYExCWYIwxxgSEJRhjjDEBYQnGGGNMQPw/48JuhV1aIY8AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(steps, lrs, \"-\", linewidth=2)\n",
    "plt.axis([0, n_steps - 1, 0, lr0 * 1.1])\n",
    "plt.xlabel(\"Batch\")\n",
    "plt.ylabel(\"Tốc độ học\")\n",
    "plt.title(\"Hoạch định cấp số mũ (trên mỗi batch)\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Định thời Từng phần Cố định"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "def piecewise_constant_fn(epoch):\n",
    "    if epoch < 5:\n",
    "        return 0.01\n",
    "    elif epoch < 15:\n",
    "        return 0.005\n",
    "    else:\n",
    "        return 0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "def piecewise_constant(boundaries, values):\n",
    "    boundaries = np.array([0] + boundaries)\n",
    "    values = np.array(values)\n",
    "    def piecewise_constant_fn(epoch):\n",
    "        return values[np.argmax(boundaries > epoch) - 1]\n",
    "    return piecewise_constant_fn\n",
    "\n",
    "piecewise_constant_fn = piecewise_constant([5, 15], [0.01, 0.005, 0.001])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 1.1511 - accuracy: 0.7326 - val_loss: 0.8456 - val_accuracy: 0.7410\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.7371 - accuracy: 0.7786 - val_loss: 0.6796 - val_accuracy: 0.8092\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.8055 - accuracy: 0.7700 - val_loss: 1.7429 - val_accuracy: 0.4514\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 1.0351 - accuracy: 0.6826 - val_loss: 0.9870 - val_accuracy: 0.6928\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.9185 - accuracy: 0.7098 - val_loss: 0.8727 - val_accuracy: 0.6932\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.6905 - accuracy: 0.7481 - val_loss: 0.6694 - val_accuracy: 0.7696\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.6115 - accuracy: 0.7713 - val_loss: 0.6956 - val_accuracy: 0.7306\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5791 - accuracy: 0.7793 - val_loss: 0.6659 - val_accuracy: 0.7738\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5622 - accuracy: 0.7881 - val_loss: 0.7363 - val_accuracy: 0.7850\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.5253 - accuracy: 0.8470 - val_loss: 0.5484 - val_accuracy: 0.8578\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4401 - accuracy: 0.8694 - val_loss: 0.6724 - val_accuracy: 0.8602\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.4334 - accuracy: 0.8732 - val_loss: 0.5551 - val_accuracy: 0.8504\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.4179 - accuracy: 0.8771 - val_loss: 0.6685 - val_accuracy: 0.8554\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.4300 - accuracy: 0.8775 - val_loss: 0.5340 - val_accuracy: 0.8584\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.4069 - accuracy: 0.8777 - val_loss: 0.6519 - val_accuracy: 0.8478\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.3349 - accuracy: 0.8953 - val_loss: 0.4801 - val_accuracy: 0.8778\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2695 - accuracy: 0.9109 - val_loss: 0.4880 - val_accuracy: 0.8786\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2568 - accuracy: 0.9136 - val_loss: 0.4726 - val_accuracy: 0.8822\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2436 - accuracy: 0.9203 - val_loss: 0.4792 - val_accuracy: 0.8842\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2421 - accuracy: 0.9212 - val_loss: 0.5088 - val_accuracy: 0.8838\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2288 - accuracy: 0.9246 - val_loss: 0.5083 - val_accuracy: 0.8830\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2215 - accuracy: 0.9270 - val_loss: 0.5217 - val_accuracy: 0.8846\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2106 - accuracy: 0.9297 - val_loss: 0.5297 - val_accuracy: 0.8834\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2002 - accuracy: 0.9334 - val_loss: 0.5597 - val_accuracy: 0.8864\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.2005 - accuracy: 0.9350 - val_loss: 0.5533 - val_accuracy: 0.8868\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.LearningRateScheduler(piecewise_constant_fn)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAsqklEQVR4nO3de5xdVX338c8395nJ5DYJGQhmBjBMICCgKCpF0waJrW2lom0FKXhL1fpUsaLwFOTmrWhptSI1LYgo+iAaLoqiRTJSvHJTMJBEVAIkJCQhCZlkcv89f+x9ksOZc2b2TGafk5nzfb9e5zXn7L32OmuvOXN+s/Zaey1FBGZmZoNtRK0LYGZmw5MDjJmZ5cIBxszMcuEAY2ZmuXCAMTOzXDjAmJlZLhxgrCxJ50rqqnU5eiMpJL251uWwbCRdL+m7OeQ7Nf0szO3HMe3pMSeWe22DwwGmTqV/7JE+dkr6vaTPSmpKk9wEHF7LMmZwMPCdPN9AUrOkKyQ9Kqlb0hpJnZLeKqkqfz95fvn1J29Jr5X0I0nrJG2V9DtJN0qaMNjlqoGnSD5Pv6pxOYaVUbUugNXUXcDZwGjgFOC/gSbgvRHRDXTXsGx9iojVeeYvaRJwLzAZuAj4JbAD+CPgYuBnwBN5luFAIelo4E7gP4EPAluAFwOnA2NrVrBBEhG7gVw/T3UpIvyowwdwPfDdkm3/BTyTPj8X6CrZ/xfAA8A24A/AJ4AxRfvHAJ8EVgDbgd8D/1i0/2jgDmAz8CzwDaA13XcUEEWvG0m+zL9fdPy7gd8WvQ7gzUWvP1b03quBG4r2CfgI8DuSwPkI8LY+6uiLJF+kh5bZNw4Ylz6fDHwF2JDmfRcwpyjtuUAXMA/4TZrnYuCwojQvAm4DngO2AkuBvy06z+JHZ7r95cAPgXXA8yTB8FUl5QxgAXBz+r6/Lz7vSnmXOd8PAk9n+FzNBm4HNqXn/DPg2OLPHPABYGVaX18GGvvze0rPu/A5fAh4Q1r2uen+uenrqUXHtKfbTsz4upDHPOAX6e/kfuClJWV5B/Bkuv87wPuAqPXf94Hy8CUyK9ZN0prpQdJ84EbgC8Ackj+sN5MElIKvAH8HfIgkYLwT2JgefzBwD8kX7CuAU4HxwO2SRkTEY8Aakj9sgJNJvqT+SFKhpT0X6KxQvjOAD5P8gc8C/pykxVHw8bQ8/0AS6D4FfEnSGyrkNwL4W+DGiHi6dH9EbIuIbenL64GTgDem57YVuFNSQ9EhY4ELSertVcAkktZAwRdJguofk9TvB0nrLs0T4PUkl3HelL5uBr5K0vp8Bcnlne9JmlpS3I+RBK/jSC59XieprY+8S60Gpkn64wr7kXQISZAL4HXAS4GrgZFFyU4BjiH5/f8N8FckAaeg199Tegn3DpJAeSJwAfDZSmUaBJ9K3+OlwHrgRklKy/Iqklb/1cDxJIH1shzLMvTUOsL5UZsHJS0Yki+adcBN6etzKWrBkASHi0vyOJ3kv1SRfKkH8PoK73c58KOSbZPTY16Rvr4J+FL6/BPANSSXoF6VbnsaOKvo+L0tGJKgtgwYXea9m0iC5ykl2/8d+F6F8h6U5n9eH/VYOO/XFG2bSBIc31VUlwF0FKU5i6SFNiJ9/TBwSYX3aKfov+teyiLgGXq2UD5V9HoUSQB8Wz/zHknS2giSfwS+k9b5tKI0nyBpQY6pkMf1JH0do4q2/RdwV9bfE0lrbCMwvmj/28ivBTO/KI+T022Hpq+/AdxZUtaFuAWz9+EWTH17vaQuSdtILmXcA/yfCmlfBvxzmr4rHWH2dZIvhVbgBGAPyaWfSse/puT4p9J9R6Q/O9nXgpmb5vVjYK6kWcAMKrRgSC4BjQP+IOlaSW+RVOgbODrdd2fJ+7+36L1LqcL2UkeRnPfPChsiYhPJpZ2ji9Jtj4hlRa9XkbQWJ6WvPwdcJOlnkj4u6WV9vbGkgyR9SdJySZtILj0eBMwsSfpwUdl2AWvTdJlFxO6IeDtwKElL8UngfGCppDlpshOAeyNiRy9ZPZqWoWBVUVmy/J6OAh6OiOIRjj8jPw8XPV+V/iyUdzYvbCVDcjnNUu7kr2/3kPxHuBNYFRE7e0k7gqT5f3OZfWvp+wt5BMmljQ+X2bcm/dkJfDENJiemr5uAt5K0rh6PiJXlMo+IpyR1kFwzPxX4V+ASSSexb7TkX5B8MRardM5rSfoIjurjvHo77+KpyndV2DcCICKulfQD4M9Iyv9TSZ+KiEt7yf8rwHTgPJKW3nbgRyR9YcVKzzEY4AjStP6/CnxV0kXAcpJAcy7ZgnJvZcnye8ryHnvKpC176TeD4vK+4HeW5h9YRW7B1LetEfF4RKzoI7gAPAjMTtOXPnal+0eQ9CFUOn4OsKLM8ZsBYl8/zD+TBJNnSVoxJ5Nc0+/srYCR9IvcERHnkXQEz0mPfZTky7etzHuvqJDXHpJLdmdJOrR0v6RxksaleY8g6Vcp7JsAHJvuyywino6IhRHx1yT9JgvSXYUWwciSQ/4I+I/0nJeQtGAO7s979pJ3lvJuILkkNz7d9CBJn1lpgMsqy+/pUeDYouH0AK8syWdt+rO4Lo4fYJl68xj7+rAKSl/XNQcYy+py4ExJl0s6RtJsSW+WdCVARPwW+Cbw35LOkHSYpFMknZ0efzVJ38RNkk6SdLikUyUtlNRc9D4/JrmmvjjN9wmSL4w30UuASW8MfZekYyUdBryd5L/P36YB7LPAZyW9Q9KLJR0v6T2SFlTKE/i/JP9J/0LS2yXNSY89m2QUU2t63reRdESfIulY4Gsko7q+nrFukfQ5Sa9P6+V4kk73QoB6lqRvYr6k6ZImptuXA2+TdLSklwP/j30BI6tKeZeW7+8lXSPpNElHpHXxLySB9NY02RdJgs03Jb08rau3pufTp4y/p6+TtAavS8vwOpJ/SIo9TnL59VJJR0o6jWSY+WD7PHCapPMlzZL0TpJBC1ZQ604gP2rzoMww5ZL959JzmPJpwP+SdBI/TzJs8/1F+8cCV5IMQd1OMtS0eP8s4FvsG867DPgPXjjU+T30HH58fbptRkl5ijv5Tye5Fr+RZDjufcCfF6UVSf9S4b/ktcD/AK/ro54mknReLyUZFvssSaD7W/Z10GcaplyS71yKOqLTevht+h5rSYLFjKL07yIJdrvZN0z5OJJr/t1pXZ9NMkrv0nJ1VLTtCeDDveVdph5OSM+xMHx4PfBz4OySdHOA75EM/tgM/BQ4ptJnDrgU+E1/fk8kI/YeTPf/muSS2t5O/jTNq0lG1XWnn4vCUOb+dvJXHCiQbnsHSTDrJhn48E9Ad63/vg+Uh9JKMjOz/STp34BTI+LYWpflQOBOfjOzAZJ0PkkLq4tkcMZ7SC6tGrgFY2Y2UJJuIrmcNpFkdosvAZ8Lf7ECDjBmZpYTjyIzM7NcuA8mNWnSpHjxi19c62IccLZs2UJTU1PfCeuM66U810tPw71OHnjggXURMa3cPgeY1PTp07n//vtrXYwDTmdnJ3Pnzq11MQ44rpfyXC89Dfc6kVT2ZmXwJTIzM8uJA4yZmeXCAcbMzHLhAGNmZrlwgDEzs1w4wJiZWS4cYMzMLBcOMGZmlgsHGDMzy4UDjJmZ5cIBxszMcuEAY2ZmuXCAMTOzXDjAmJlZLhxgzMwsF1UNMJKmSLpF0hZJKySd2Uva8yStlrRJ0nWSxhbte7+k+yVtl3R9mWPnSVoqaaukxZLa+irbE8/v4eRP382tD63MdC63PrSSkz99N4ddcMewPM7MbH9VuwVzNbADmA6cBVwjaU5pIknzgQuAeUA7cDhwWVGSVcDHgevKHDsVWARcDEwB7gduylK4lRu7uXDRI31+Cd/60EouXPQIKzd2E8PwODOzwVC1FS0lNQFnAMdERBdwr6TbgbNJgkmxc4BrI2JJeuwVwI2FdBGxKN1+InBoybFvApZExM1pmkuBdZJmR8TSvsrZvXM3//eWR7j38XUV03zvkWfo3rl7yB73mR8s4/QTZlQ8zsxsMFRzyeQjgd0Rsbxo26+B15ZJOwe4rSTddEktEbG+j/eZk6YHICK2SPpduv0FAUbSAmABwJjWF+/dvnXHbhYvqfxf/tYdUWH70Dhu5cZuOjs7Kx5XrKurK3PaeuJ6Kc/10lM910k1A8x4YFPJtk1Ac4a0hefNQF8BZjywNsv7RMRCYCHA2INn7f02njGpgZ9c8CcV3+DkT9/Nyo3dPbYPpeOyrhE+3NcTHyjXS3mul57quU6q2QfTBUwo2TYB2JwhbeF5ubT78z49NIweyfnzO3pNc/78DhpGjxy2x5mZDYZqBpjlwChJs4q2HQcsKZN2SbqvON2aDJfHehyb9v0cUeF9XmDGpAY+9aZj++yfOP2EGXzqTccyY1IDGgLHHTxxHAATxo3KdJyZ2WCo2iWytC9kEXC5pHcBxwNvBF5dJvkNwPWSbgSeAS4Cri/slDSKpOwjgZGSxgG7ImIXcAvwGUlnAHcAHwMe7quDv33CiF4vN5U6/YQZA/qirtVxJ33yLk6ZNc3BxcyqptrDlN8HNADPAt8A3hsRSyTNlNQlaSZARNwJXAksBlakj0uK8rkI6CYZVfa29PlF6bFrSUarfQLYAJwE/G3+p3Zga2tpYsX6LbUuhpnVkWp28hMRzwGnl9n+JEnnfPG2q4CrKuRzKXBpL+9zFzB74CUdfg5raeJHS5+tdTHMrI54qpg60Ta1kXVd2+navqvWRTGzOuEAUyfaW5oAfJnMzKrGAaZOtLU0ArBi/dYal8TM6oUDTJ1oS1swT7gFY2ZV4gBTJ8aPHcXU8WNZsc4tGDOrDgeYOtLe0ugWjJlVjQNMHWlraXKAMbOqcYCpI4dNbWTN89vZusNDlc0sfw4wdaTQ0f/kc+6HMbP8OcDUkcK9ME+4o9/MqsABpo7M3HsvjPthzCx/DjB1ZGLDaKY0jeEJ32xpZlXgAFNn2loa3YIxs6pwgKkz7S1Nni7GzKrCAabOtLc0sWpTN9t27q51UcxsmHOAqTPtUxuJgKc8VNnMcuYAU2f2TXrpAGNm+XKAqTPtHqpsZlXiAFNnJjWOYWLDaM9JZma5c4CpQ+0tjR5JZma5c4CpQ55V2cyqwQGmDrW3NLJyQzc7du2pdVHMbBhzgKlD7VOb2BPw1AZfJjOz/DjA1KHCUGWPJDOzPDnA1KHCUGVP229meXKAqUNTmsbQPHaUWzBmlisHmDokibapjb6b38xy5QBTp9pamtyCMbNcOcDUqfaWRp7e0M3O3R6qbGb5cICpU+0tTezaE6zc0F3ropjZMFXVACNpiqRbJG2RtELSmb2kPU/SakmbJF0naWzWfCT9taTHJG2W9Kik03M8rSGpfWphVmVfJjOzfFS7BXM1sAOYDpwFXCNpTmkiSfOBC4B5QDtwOHBZlnwkzQC+BnwImACcD3xd0kH5nNLQ1LZ3VmV39JtZPqoWYCQ1AWcAF0dEV0TcC9wOnF0m+TnAtRGxJCI2AFcA52bM51BgY0R8PxJ3AFuAI3I8vSFn2vixNI4Z6RaMmeVmVBXf60hgd0QsL9r2a+C1ZdLOAW4rSTddUgsws4987gcek/SXwB3AXwDbgYdL30TSAmABwLRp0+js7BzAaQ1dLWODB5c/RWfn2oppurq66q5esnC9lOd66ame66SaAWY8sKlk2yagOUPawvPmvvKJiN2SbgC+DowjuZT2lojo8a96RCwEFgJ0dHTE3Llz+3E6Q98xTz/AsjWb6e28Ozs7e91fr1wv5bleeqrnOqlmH0wXSZ9IsQnA5gxpC88395WPpFOBK4G5wBiSls1/Szp+4EUfntpamnjqua3s3hO1LoqZDUOZA4ykP5X03XRU1ovSbe+SNC9jFsuBUZJmFW07DlhSJu2SdF9xujURsT5DPscD90TE/RGxJyLuA34BnJqxnHWjvaWRnbuDVRs9VNnMBl+mACPpLOCbwG+Bw4DR6a6RwEey5JFeoloEXC6pSdLJwBuBr5ZJfgPwTklHS5oMXARcnzGf+4BTCi0WSScAp1CmD6beeaiymeUpawvmI8C7I+I8YFfR9p+TtBiyeh/QADwLfAN4b0QskTRTUpekmQARcSfJZa7FwIr0cUlf+aTH/hi4FPiWpM3At4FPRsQP+1HOutDeUggwHqpsZoMvayf/LOBnZbaX6w+pKCKeA04vs/1Jks774m1XAVf1J5+i/V8AvpC1XPXqoOaxjBs9ghXr3IIxs8GXtQWzimSYcanXAL8bvOJYNY0YIdqmNLkFY2a5yBpgFgKfT/s7AF4k6RySy1jX5FIyq4q2lkbPqmxmuch0iSwirpQ0EfgfkntLFpPcvPjZiLg6x/JZztqnNtG5fC179gQjRqjWxTGzYSTzjZYR8c+SPgEcTdLyeTQiunIrmVVFW0sjO3btYfXz2zhkUkOti2Nmw0jWYcrXSWqOiK3p/SW/jIiudJjwdXkX0vJzWGEkmTv6zWyQZe2DOYdkWHCpBuDvBq84Vm1tUz1U2czy0eslMklTAKWPyZKK74EZCbwBWJNf8SxvB08Yx5hRI9zRb2aDrq8+mHVApI9Hy+wPXngDpA0xI0aImVMafTe/mQ26vgLMH5O0Xu4mWYPluaJ9O4AVEbEqp7JZlbS3NHrhMTMbdL0GmHTaFSQdBjwVEXuqUiqrqraWJu59fB0RgeShymY2OLLeB7MCQNIhJAt+jSnZf8/gF82qpb2lkW079/Ds5u1MnzCu1sUxs2EiU4BJA8vXSaaGCZLLZsWLiIwc/KJZtbSlQ5X/sG6LA4yZDZqsw5T/HdhNcpPlVpLp798CPAa8PpeSWdUclg5V9kgyMxtMWe/kfy3whohYKimAtRHxE0nbgStIppCxIergieMYPVK+F8bMBlXWFkwDyZBlSEaSHZQ+fxR4yWAXyqpr1MgRvGiyJ700s8GVNcAsBWanz38FvEdSG/APwMocymVV1tbSyBPr3IIxs8GT9RLZ54DW9PnlwJ3AW0lmVD4nh3JZlbW1NPHLPzznocpmNmiyDlO+sej5g5LaSVo0T0bEuooH2pDR3tLIlh27Wde1g2nNY2tdHDMbBrJeInuBdFblB4Etki4Y5DJZDbR5JJmZDbI+A4ykqZLeIOk0SSPTbaMlfRB4AvhwvkW0ajis6F4YM7PB0Ndsyq8G7gAmktxYeZ+kc4FbgNEkQ5S9HswwMGNyAyNHyHOSmdmg6asFcwXwA5KhyJ8DXgF8F/gUMCsivhAR/kYaBkaPHMGhkxs8q7KZDZq+AsxxwBUR8RvgIpJWzIURcUNERO+H2lDT1tLkFoyZDZq+AswUYC0kHfsk08Q8lHehrDbaW5J1Yfy/g5kNhizDlAsrWRYmuJyQrnS5V0Q8V/ZIG1LaWprYvG0XG7buZErTmL4PMDPrRZYAU7ySpYD7Sl4Hnk15WGhvaQTgifVbHGDMbL9lWdHS6kRh2v4n1m3hpTMn17g0ZjbUZVrR0urDi6Y0MEJ4VmUzGxQDupPfhqexo0ZyyKQG381vZoOiqgFG0hRJt0jaImmFpDN7SXuepNWSNkm6TtLYrPlIapT0RUnr0uO9pHNG7S1NbsGY2aCodgvmamAHMB04C7hG0pzSRJLmAxcA84B24HDgsn7ks5BkiPVR6c/zBvtEhqu2Fq8LY2aDo2oBRlITcAZwcUR0RcS9wO3A2WWSnwNcGxFLImIDyYwC52bJR1IH8JfAgohYGxG7I+KBnE9v2GhvaWLj1p1s3Lqj1kUxsyEu63owg+FIYHdELC/a9muS5ZhLzQFuK0k3XVILMLOPfE4CVgCXSTobeAa4NCK+XfomkhYACwCmTZtGZ2fnQM5rWNm8ZhcAi/7nfzl84ki6urpcL2W4XspzvfRUz3WSKcBIqjShZQDbgMeBmyJiVS/ZjAc2lWzbBDRnSFt43pwhn0OBY4BvA4cArwLukPRoRDz2gsJHLCS5nEZHR0fMnTu3l+LXh0PWbObzD91DS9ts5h4/g87OTlwvPbleynO99FTPdZK1BTMNOAXYA/wm3XYMyY2WDwBvAi6XdEpE/KpCHl3AhJJtE4DNGdIWnm/OkE83sBP4eETsAn4saTFwGvAY1quZUxqR8PLJZrbfsvbB/AT4PnBoRLwmIl5D0lL4HvBDoI1kWv9/7SWP5cAoSbOKth0HLCmTdkm6rzjdmohYnyGfhzOek5UxbvRIDp4wzh39ZrbfsgaYDwCXF0/Nnz7/BHBeROwA/gU4vlIGEbEFWETS0mmSdDLwRuCrZZLfALxT0tGSJpPM5Hx9xnzuAZ4ELpQ0Kt0/l2TZAcugraXJ0/ab2X7LGmDGAweX2d6a7gN4nr4vub0PaACeBb4BvDcilkiaKalL0kyAiLgTuBJYTNJhvwK4pK980mN3kgScPyPpm/kv4O8iYmnGc6177VMbPW2/me23rH0wtwDXSvoIyWSXQbL42JUkrQnS18vLH55IZ10+vcz2J9kXqArbrgKu6k8+RfuXkHTu2wC0tTSxfssOnt+2s9ZFMbMhLGuAeQ/Jl/3Xio7ZRbJc8ofT148B7x7U0llNFGZVftKtGDPbD5kCTNrf8h5J/wQcQTJ67PG0P6SQ5le5lNCqbu+syuu3vLBZaWbWD/260TINKB6lNcy1FdaFWbeFYzwdqpkNUNYbLceRjCSbBxxEyeCAiHjJ4BfNaqVxzCimTxjLE+u3csy0WpfGzIaqrC2YLwJ/BdwM/JSkk9+GsbaWpuReGAcYMxugrAHmdOAtEXFXjmWxA0h7SyOLl62lutPVmdlwkvUK+1bgqTwLYgeWtpYm1m7ezrZdbqya2cBkDTBXAh+S5C7fOtGejiR7duueGpfEzIaqrNc/Xkcy2eXrJT1KMpnkXhHxl4NdMKutwkiyNVvdgjGzgckaYNaR3M1vdaJ9qlswZrZ/st5o+fa8C2IHlrseXcMIwc3Ld/LTT9/N+fM7OP2EGX0ed+tDK/nMD5axamM3h0xqGLbHrdzYzYyfu17MeuMhQtbDrQ+t5MJFj7AnvTq2cmM3Fy56BKDXL5vCcd07d/u4OjrOrBJFlL/GLulh4LURsUHSI/Ry78twuNGyo6Mjli1bVutiHBBO/vTdrNzY3WP76JHi6EMmVjzu0VWb2Lm758fExw3t42ZMauAnF/xJxeOK1fPqjZUM9zqR9EBEnFhuX28tmG8D29Pn3xr0UtkBa1WZ4AKwc3cwqWF0xePKfTn5uKF/XKXPg1lfKgaYiLis3HMb/g6Z1FC2BTNjUgNfeccrKh5XqeXj44b2cYdMaqh4jFlvfF+L9XD+/A4aRo98wbaG0SM5f36Hj/NxZpllnexyCsnyyJUmu5ww+EWzWil06O4dLZVxNFHxcf0ZhTQUjxvO9XLp7UvY2L2T6RPGcuGfHuUOfhuwip38L0gk3QKcACwEVlHS4R8RX8mldFXkTv7yhnsH5UAN53p56MkN/NUXf8qXzn4Z8+e09uvY4VwvAzXc62SgnfzF5gGvi4hfDF6xzOxAdOT0ZgCWrd7c7wBjVixrH8yzQFeeBTGzA0PT2FHMnNLIstWba10UG+KyBph/Bi6X5BV0zerA7NZmlq5+vtbFsCEu6yWyi4B24FlJK+g52eWQv9HSzPaZ3drMXY+tYdvO3YwrGVlmllXWAOMbLc3qSEfrBPYEPP5sF8fMqHz3v1lv+gwwkkYDTcDVEbEi/yKZWa11tCYd/UtXb3aAsQHrsw8mInYC7wWUf3HM7EDQ3tLImFEjWOZ+GNsPWTv5fwhkm+3OzIa8USNHMOug8Sz1SDLbD1n7YH4EfFLSS4AHgC3FOyNi0WAXzMxqq6O1mXt/u67WxbAhLGuA+UL68x/L7AvAw0zMhpmjWiew6MGVPLdlB1OaxtS6ODYEZbpEFhEjenk4uJgNQ/s6+t0PYwPj2ZTNrKzZrfumjDEbiMwBRtIUSWdKukDSx4of/czjFklbJK2QdGYvac+TtFrSJknXSRrb33wkXSIpJJ2atYxmlpjWPJbJjaMdYGzAsk7X/0rgDpIVLqcBK4GD09dPAJdnfL+rgR3AdOB44A5Jv46IJSXvNx+4gGTk2irgFuCydFumfCQdAbwZeCZj2cysiCQ6Wps9kswGLGsL5jPAjcAMYBvJF/9M4H7gX7JkIKkJOAO4OCK6IuJe4Hbg7DLJzwGujYglEbEBuAI4t5/5fAH4KEkgMrMBmN06geVrNrNnT9/LepiVyjqK7CXAOyMiJO0GxkbE7yV9FPg6SfDpy5HA7ohYXrTt18Bry6SdA9xWkm66pBaSwNZrPpLeAuyIiO9Jle8PlbQAWAAwbdo0Ojs7M5xGfenq6nK9lFEv9aJNO9m6YzffunMxBzX2/f9ovdRLf9RznWQNMMWtgDVAG/AYyRT+h2TMYzywqWTbJqA5Q9rC8+a+8klnfP4kcFpfBYqIhSSLqNHR0RHDeVGggRruiyUNVL3Uy8QnN/DlJT9lYtvRzM2wNky91Et/1HOdZL1E9iDw8vR5J/BxSecAnwcezphHF1C6tPIEoNwF3tK0heebM+RzGfDViPhDxnKZWQXFi4+Z9Vd/1oNZlT6/CFgL/AcwmfQSUwbLgVGSZhVtOw5YUibtknRfcbo1EbE+Qz7zgH9MR6CtBl4EfDO9nGdm/eDFx2x/ZLpEFhH3Fz1fC/xpf98oIrZIWkSycNm7SEZ/vRF4dZnkNwDXS7qRZBTYRcD1GfOZB4wuyus+4EPA9/tbZjMjHUnmmy2t//p1o6WkEyX9TTqSC0lNkrL24wC8D2ggWYL5G8B7I2KJpJmSuiTNBIiIO4ErgcXAivRxSV/5pMeuj4jVhQewG9gQEV7y2WwAZrc288T6rWzbubvWRbEhJut9MNNJhgK/nGTusVnA74GrSIYtfyBLPhHxHHB6me1PknTeF2+7Ks0/cz4V0rZnSWdm5c1uncDuPeHFx6zfsrZg/g1YDbQAW4u230yG0VpmNnR1eMoYG6Csl7fmAfMiYkPJfSW/I7kvxcyGqcLiY+6Hsf7K2oJpoPwd8dNILpGZ2TDlxcdsoLIGmHtIp2pJhaSRJFOx/GiwC2VmB5aO1mZfIrN+y3qJ7CPAjyW9HBgL/CvJdC4TgZNzKpuZHSBmtzaz6MGVbNiyg8lefMwyyrrg2KPAscBPgR8C40g6+E+IiN/lVzwzOxDMbk0mz/BlMuuPzPewpPeUFN+LgqQ2Sd+MiL8e9JKZ2QFj3+Jjz/OqI1pqXBobKvZ3RctJJFPnm9kwVlh8zC0Y6w8vmWxmffLiYzYQDjBmlokXH7P+coAxs0w6WpvZumM3T2/ornVRbIjotZNf0u19HF+6LouZDVOFjv6lq59nZktjjUtjQ0Ffo8jWZ9jvhb3M6kDx4mOnZVjd0qzXABMRb69WQczswFZYfMwd/ZaV+2DMLDMvPmb94QBjZpl58THrDwcYM8uso7V57+JjZn1xgDGzzApzknlmZcvCAcbMMissPrZsjQOM9c0BxswyKyw+9tgz7ui3vjnAmFm/ePExy8oBxsz6ZXZrM89u3s6GLeVWUTfbxwHGzPqlw4uPWUYOMGbWL0cVLT5m1hsHGDPrl8LiYx5JZn1xgDGzfiksPvbYMw4w1jsHGDPrNy8+Zlk4wJhZv3nxMcvCAcbM+q2jaPExs0qqGmAkTZF0i6QtklZIOrOXtOdJWi1pk6TrJI3Nko+kV0r6H0nPSVor6WZJB+d9bmb1pKNo8TGzSqrdgrka2AFMB84CrpE0pzSRpPnABcA8oB04HLgsYz6TgYXpcW3AZuDLg38qZvVr7+JjHklmvahagJHUBJwBXBwRXRFxL3A7cHaZ5OcA10bEkojYAFwBnJsln4j4fkTcHBHPR8RW4AvAyTmfnlnd6WhtZqnnJLNe9Lpk8iA7EtgdEcuLtv0aeG2ZtHOA20rSTZfUAszsRz4ArwGWlNshaQGwAGDatGl0dnZmOI360tXV5Xopw/UC47bv4A/rdvLDHy1mzEgBrpdy6rlOqhlgxgObSrZtApozpC08b+5PPpJeAnwMeGO5AkXEQpLLaXR0dMTcuXN7PYF61NnZieulJ9cLdE1ZxXd+9xCHzH4px8yYCLheyqnnOqlmH0wXMKFk2wSSPpK+0haeb86aj6QXA98HPhAR/zvAMptZBbNb3dFvvatmgFkOjJI0q2jbcZS/fLUk3Vecbk1ErM+Sj6Q24C7gioj46iCV38yKtLc0efEx61XVAkxEbAEWAZdLapJ0Msmlq3IB4AbgnZKOljQZuAi4Pks+kmYAdwNXR8R/5nxaZnWrsPiYZ1W2Sqo9TPl9QAPwLPAN4L0RsUTSTEldkmYCRMSdwJXAYmBF+rikr3zSfe8iGdZ8SZpnl6SuKpybWd3xSDLrTTU7+YmI54DTy2x/kqTzvnjbVcBV/ckn3XcZL7xnxsxyMru1mUUPrmTDlh1MbhpT6+LYAcZTxZjZgHnxMeuNA4yZDdhsLz5mvXCAMbMBO6h5LJO8+JhV4ABjZgMmidmtzb5EZmU5wJjZfpndOoHlq734mPXkAGNm+6WjtZktXnzMynCAMbP94sXHrBIHGDPbL0d68TGrwAHGzPbL+LGjeNGUBi8+Zj04wJjZfpvdOsEtGOuhqlPFmNnwJILHn+3i3Dthxs/v5vz5HZx+wow+j7v1oZV85gfLWLWxm0MmNQyr4wrHrNzYPazrZEzri19WKY0DjJntl1sfWsniZWv3vl65sZsLFz0C0OsX1a0PreTCRY/QvXP3sDtuKJRxMI+rRBEeuw7JipbLli2rdTEOOPW8Gl9vXC/7nPzpu1m5secQ5bGjRnDS4S0Vj/vF79ezfdeeYXncUCjjYB33zFc+yPZnfqty6dyCMbP9sqpMcAHYvmsPz3fvrHhcuS+24XLcUChjHseVcoAxs/1yyKSGsi2YGZMauPUfTq54XKWWz3A4biiUMY/jSnkUmZntl/Pnd9AweuQLtjWMHsn58zvq9rihUMbBPq4ct2DMbL8UOoP3jpjKOBKp+Lj+jGAaCsfVU50800s6d/Kn3Mlfnjuzy3O9lOd66Wm414mkByLixHL7fInMzMxy4QBjZma5cIAxM7NcOMCYmVkuHGDMzCwXDjBmZpYLBxgzM8uFA4yZmeXCAcbMzHLhAGNmZrlwgDEzs1w4wJiZWS6qGmAkTZF0i6QtklZIOrOXtOdJWi1pk6TrJI3Nmo+keZKWStoqabGktjzPy8zMeqp2C+ZqYAcwHTgLuEbSnNJEkuYDFwDzgHbgcOCyLPlImgosAi4GpgD3AzflczpmZlZJ1QKMpCbgDODiiOiKiHuB24GzyyQ/B7g2IpZExAbgCuDcjPm8CVgSETdHxDbgUuA4SbPzOzszMytVzQXHjgR2R8Tyom2/Bl5bJu0c4LaSdNMltQAz+8hnTvoagIjYIul36falxW8iaQGwIH25XdJv+n1Ww99UYF2tC3EAcr2U53rpabjXScUuiGoGmPHAppJtm4DmDGkLz5sz5DMeWJvlfSJiIbAQQNL9lRbNqWeul/JcL+W5Xnqq5zqpZh9MFzChZNsEYHOGtIXnmzPk05/3MTOznFQzwCwHRkmaVbTtOGBJmbRL0n3F6dZExPoM+bzg2LTP5ogK72NmZjmpWoCJiC0ko7sul9Qk6WTgjcBXyyS/AXinpKMlTQYuAq7PmM8twDGSzpA0DvgY8HBELC19kxIL9+8Mhy3XS3mul/JcLz3VbZ0oIqr3ZtIU4DrgdcB64IKI+LqkmcCjwNER8WSa9kPAR4EG4NvAeyJie2/5FL3PqcAXSDqffgGcGxFPVOUkzcwMqHKAMTOz+uGpYszMLBcOMGZmlou6DzD9mR+tnkjqlLRNUlf6WFbrMtWCpPdLul/SdknXl+yryznvKtWJpHZJUfSZ6ZJ0cQ2LWlWSxkq6Nv0e2SzpIUl/WrS/7j4vdR9gyDg/Wp16f0SMTx8dtS5MjawCPk4yqGSvOp/zrmydFJlU9Lm5oorlqrVRwFMks4pMJPlsfDMNvHX5eanmnfwHnKJ5zY6JiC7gXkmFec0uqGnh7IAQEYsAJJ0IHFq0a++cd+n+S4F1kmZnGBI/pPVSJ3UtvYXi0qJN35X0B+BlQAt1+Hmp9xZMpfnR3IJJfErSOkk/kTS31oU5wPSY8w4ozHlX71ZIelrSl9P/3OuSpOkk3zFLqNPPS70HmP7Mj1ZvPkqyTMIMkhvFviPpiNoW6YDiz05P64CXk9x/9jKSurixpiWqEUmjSc79K2kLpS4/L/UeYDxvWQUR8YuI2BwR2yPiK8BPgD+rdbkOIP7slEiXz7g/InZFxBrg/cBpkkrraViTNIJkZpEdJHUAdfp5qfcA05/50epdAKp1IQ4gnvOub4W7uOvmcyNJwLUkg4bOiIid6a66/LzUdYDp5/xodUPSJEnzJY2TNErSWcBrgB/UumzVlp7/OGAkMLJQJwx8zrshr1KdSDpJUoekEenaTZ8HOiOi9NLQcHYNcBTwFxHRXbS9Pj8vEVHXD5Ihg7cCW4AngTNrXaZaP4BpwH0kzfeNwM+B19W6XDWqi0tJ/hMvflya7juVZBG7bqATaK91eWtZJ8BbgT+kf0vPkExa21rr8laxXtrSuthGckms8DirXj8vnovMzMxyUdeXyMzMLD8OMGZmlgsHGDMzy4UDjJmZ5cIBxszMcuEAY2ZmuXCAMRum0rVZ3lzrclj9coAxy4Gk69Mv+NLHz2tdNrNqqev1YMxydhfJ2kLFdtSiIGa14BaMWX62R8TqksdzsPfy1fsl3ZEuobtC0tuKD5Z0rKS7JHVLei5tFU0sSXOOpEfS5YvXlC7rDEyRdHO6JPjvS9/DLE8OMGa1cxlwO3A8yZo7N6SrRCKpEbiTZC6rVwB/BbyaomWKJf098CXgy8BLSJZTKJ2d92PAbSQz+d4EXFcPa8HbgcFzkZnlIG1JvI1k4sNiV0fERyUF8N8R8e6iY+4CVkfE2yS9G/gscGhEbE73zwUWA7Mi4nFJTwNfi4iyy3un7/HpiLgwfT0KeB5YEBFfG7yzNSvPfTBm+bkHWFCybWPR85+V7PsZ8Ib0+VEk07kXL0j1U2APcLSk50lWG/1RH2V4uPAkInZJWgsclKn0ZvvJAcYsP1sj4vEBHiv2LdhVqj+Lv+0seR340rhViT9oZrXzyjKvH0ufPwocJ6l4zfZXk/zNPhbJksQrgXm5l9JsgNyCMcvPWEmtJdt2R8Ta9PmbJN1HsvjUm0mCxUnpvhtJBgHcIOljwGSSDv1FRa2iTwD/JmkNcAfQCMyLiH/N64TM+sMBxiw/p5Ks7FhsJXBo+vxS4AySpYXXAm+PiPsAImKrpPnAvwO/JBkscBvwgUJGEXGNpB3APwH/AjwHfC+nczHrN48iM6uBdITXWyLiW7Uui1le3AdjZma5cIAxM7Nc+BKZmZnlwi0YMzPLhQOMmZnlwgHGzMxy4QBjZma5cIAxM7Nc/H/RZ10XkYhjxQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, [piecewise_constant_fn(epoch) for epoch in history.epoch], \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Tốc độ học\")\n",
    "plt.title(\"Hoạch định Hằng số Từng phần\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Định thời Năng lực"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.7116 - accuracy: 0.7769 - val_loss: 0.4869 - val_accuracy: 0.8478\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 2s 947us/step - loss: 0.4912 - accuracy: 0.8390 - val_loss: 0.5958 - val_accuracy: 0.8270\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 2s 987us/step - loss: 0.5222 - accuracy: 0.8379 - val_loss: 0.4869 - val_accuracy: 0.8584\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5061 - accuracy: 0.8467 - val_loss: 0.4588 - val_accuracy: 0.8548\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5216 - accuracy: 0.8469 - val_loss: 0.6096 - val_accuracy: 0.8300\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4984 - accuracy: 0.8546 - val_loss: 0.5359 - val_accuracy: 0.8498\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5104 - accuracy: 0.8579 - val_loss: 0.5457 - val_accuracy: 0.8522\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5375 - accuracy: 0.8538 - val_loss: 0.6445 - val_accuracy: 0.8218\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5333 - accuracy: 0.8522 - val_loss: 0.5472 - val_accuracy: 0.8560\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3280 - accuracy: 0.8902 - val_loss: 0.3826 - val_accuracy: 0.8876\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2410 - accuracy: 0.9135 - val_loss: 0.4025 - val_accuracy: 0.8876\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2266 - accuracy: 0.9180 - val_loss: 0.4540 - val_accuracy: 0.8694\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2129 - accuracy: 0.9221 - val_loss: 0.4310 - val_accuracy: 0.8866\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1959 - accuracy: 0.9270 - val_loss: 0.4406 - val_accuracy: 0.8814\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1975 - accuracy: 0.9277 - val_loss: 0.4341 - val_accuracy: 0.8840\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1409 - accuracy: 0.9464 - val_loss: 0.4220 - val_accuracy: 0.8932\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1181 - accuracy: 0.9542 - val_loss: 0.4409 - val_accuracy: 0.8948\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1124 - accuracy: 0.9560 - val_loss: 0.4480 - val_accuracy: 0.8898\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1070 - accuracy: 0.9579 - val_loss: 0.4610 - val_accuracy: 0.8932\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.1016 - accuracy: 0.9606 - val_loss: 0.4845 - val_accuracy: 0.8918\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0848 - accuracy: 0.9686 - val_loss: 0.4829 - val_accuracy: 0.8934\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0792 - accuracy: 0.9700 - val_loss: 0.4906 - val_accuracy: 0.8952\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0751 - accuracy: 0.9720 - val_loss: 0.4951 - val_accuracy: 0.8950\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0687 - accuracy: 0.9739 - val_loss: 0.5109 - val_accuracy: 0.8948\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.0683 - accuracy: 0.9752 - val_loss: 0.5241 - val_accuracy: 0.8936\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(learning_rate=0.02, momentum=0.9)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdMAAAEeCAYAAADRiP/HAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABeyElEQVR4nO2deXxU1fXAvycJhE1URFEgJKKCAlVwrxuobbWL1aptVWzVVqm4VG2rtT+1omhbrbW1dasWV3Cvu9W6Je67ghJUFCVUCbsiYSec3x/nDXmZzPIms2U538/nfTJz373n3fcY5sy59yyiqjiO4ziO03pKij0Bx3Ecx2nvuDJ1HMdxnCxxZeo4juM4WeLK1HEcx3GyxJWp4ziO42SJK1PHcRzHyRJXpk6nQ0QaROT4Ys+joyEis0XkN8Weh+MUA1emTptERG4REQ2OdSIyR0SuE5FNiz23XCAiY4J765vk/ITQ/a8XkbkiMkVEKgo912A+x4fmoyJSLyL3iMjWWcpsyOU8HadYuDJ12jJPA1sBVcCJwCHAtcWcUIH5ELv/gcCPga8B9xRxPiuC+fQHjgFGAg+LSGkR5+Q4bQJXpk5bZrWqzlPVz1T1SeBu4FvhDiJygojMEJFVIjJTRM4SkZLQ+W1FpCY4/6GIfC9ufFVgae0a164icmToff/AMlwsIitEZKqI7B86f4iIvBVc51MRuVREumZ5/+uC+5+rqi8ANwJ7ikjvVINE5HAReU9EVovI/0TkPBGR0PnZInK+iPxTRL4Skc9E5OwI89FgPvWqWg1cBIwAtk0yj1+JyLsislxEPheRf4nIJsG5McDNQM+QtTshONdVRC4L5rVcRN4QkYNCcktFZFLwnFeKyEcick7cv/stIvJo3HwmiMj0CPfpOBlTVuwJOE4URGQwcDCwNtR2EnAxcDrwFvbFfmPQ5+rgy/UB4Avg60AP4CqgPMNr9wSeAxYAPwA+B3YKnT8ImAKcATwPDAKuD66Tkz1EEdkSOBxoDI5k/XYB7gUuCea0G/BP4CvgH6GuZwEXAn8Gvg38XUReVNVXMpjWyuBvlyTn1wNnAp8AlcH1/wH8BHg5OPcHYJugf2zJ9+ag7RjgM+A7wCMispuqTsOMgM+BHwELgd2BG4DFwKQM5u84uUNV/fCjzR3ALcA67At2JaDBcVaozxzgJ3HjzgRmBK+/hSmeQaHz+wRyjg/eVwXvd42To8CRweuTgGVA3yRzfR64IK7tsGDukmTMmOAayWROCObegC2vxu7/qjTPbQrwbAJZn4XezwbujOvzEXB+CrnHAw2h9wOBV4D/AV1Dcn+TQsbBwGqgJJHMoG0bTAkPimt/ELg2hew/AU/HfX4eTfAcphf7s+1HxzzcMnXaMs8D44DumELbBvg7gIhsDlQA/xSR60JjyoDYkuYOwOeqOid0/jXsyzoTRgHvquqiJOd3AXYXkd+G2kqCeW8J1Gd4vRizMKusHDgUOAL4vzRjdgAei2t7EbhQRHqr6ldB27txfeYCW6SR3TNwGBLMyn8bOFxV1yTqLCIHAL8L5rQxUAp0xZ7J3CTX2DmQPyO0Mg32DJ4NyT4Z20evxJ5zF6AuzfwdJ2+4MnXaMitU9ePg9S9FpBq4ALMwYvtjJ2NLhomQJO1hYoo1vKcYv2yZTk4Jtn94b4JzCyPMIRlrQvdfKyLbAddgFl0yBLNgExFuX5vgXDofihWY09F6YL6qLk86CZFKTKnfCPweW4LdGbgTU6jJKAnmsluCOa4MZP8Y+Bu2hP4ytoR9KrYEH2M9Lf/dki1HO07WuDJ12hMXAY+LyA2qOldEPge2UdXbkvSfAQwQkQpV/V/QtjvNlUZM2W0VahsZJ+dt4FgR6ZvEOn0b2D6k+PLFROBDEfmHqr6VpM8MbCk7zD7YMu+yLK+vGdzjrpjSPEtVGwHinb+ANZi1GuYdTAluqebklIh9gNdU9epYg4hsE9dnIS3/HePfO07OcG9ep92gqjVALXB+0DQBOCfw4B0qIiNE5Kci8rvg/NPAB8BtIjJSRL4O/BXbi43JXAm8CvxWRIaLyF7AFXGXvgNzPnpQRPYVka1F5Pshb96LgWNE5OJgDtuLyJEicnmE2xoRzC18JPx/qaqfAA9jSjUZfwFGB56rQ0RkLPBrIMpccslH2PfLmcHzOhrbzw4zG+gmIt8Ukb4i0kNVZ2L7vrcEz3CwiOwqIr8RkcODcTOBnUXk2yKynYhcAIyOk/0sMEpEfibm0X0OsHee7tVx3AHJj7Z5kMCBJGg/BnNiqQzeH41Zhqswr90XgaNC/YdgnrirsS/472NOPceH+uwAvIQtY74H7EvIASnoMxALzfky6PcOMCZ0/lvAC8G5r4A3gdNS3N8YmpyK4o9eJHGWAfYK+uyVQvbhwX2swRyEziPkCEUCRyGgBrg6hczjiXMWStCnmVzgl5jX7UrgGcz7VoGqUJ/rgEVB+4SgrUtw/58E9zAP+xGxS3C+K+a1+0Xw7zEJW0qeHTefCdh+9VIsPvkPiZ6pH37k4hDVZNsrjuM4juNEwZd5HcdxHCdLXJk6juM4Tpa4MnUcx3GcLHFl6jiO4zhZ4nGmESkpKdHu3bsXexptjvXr11NS4r/J4vHn0hJ/Jonp6M9lxYoVqqod9wYDXJlGpGvXrixfnjThS6elpqaGMWPGFHsabQ5/Li3xZ5KYjv5cRGRl+l7tnw7/a8FxHMdx8o0rU8dxHKe4iPRB5AFEliNSh8gxKfoORuRRRJYhsohwpjGRGkRWIdIQHB8WYvrgytRxHMcpPtdg2a76AWOB6xAZ3qKXSFfgKSxd5JZYZrLJcb1OQ7VXcAzN66xDuDJ1HMdxiodIT6y84AWoNqD6IpY+8icJeh8PzEX1SlSXo7oK1fhygkXBlanjOI6TN/pCGSJvho5xcV2GAI1YkYMY04CWlinsCcxG5PFgibcGka/F9fljcO4lRMbk7EbSUFBlKkIfER4QYbkIdSIkXRcX4SwR5omwVISbRCgP2stFmBSMXybCOyJ8O27sgSJ8IMIKEapFqAydExEuE2FxcFwukr7u5erVpVRVwZQp0e51yhSoqoKSEjr0uAMOGF2w6+02sJ7nZDS7VczL+/Ucx8kNi2AdqruGjhviuvTCihGEWQpslEDcQOAo4O9Af6xm7kPB8i/Ab4HBwADgBuARWpbnyw+FzKoPeifo3aC9QPcBXQo6PEG/g0Dngw4H3RS0BvRPwbmeoBNAq0BLQL8Hugy0KjjfN5D7Q9BuoH8GfTUk+xegH4IOBB0AOgP05PRz76Gg2qOH6uTJmpLJk60fNB0+Lvtx1zBe11GiV3NKXq+XC6qrq/N/kXaGP5PEdPTnAizXVN+vMEphRVzbrxUeSdD3IYXq0HtRWKqwUxLZTyicnvL6OToKFmcqQmxdfIQqDcCLIhvWxc+N634cMEmV2mDsRKzG4bmqLMdKK8V4VIRPgV2wElCHA7Wq3BuMnQAsEmF7VT4IZP9Flc+C838BTgKuj3IfK1bAqafChyl8xP7+d+vn43I3rveKen7OvyhlPSdwMxNXXMCpp27Zquuddx6MHZt8nOM4BWUmthS8HaofBW07YbWL43mXzOrSKqRfecwFBSvBJsIo4GVVuofafgOMVuWQuL7TgD+ocnfwvi+wEOiryuK4vv2AOmCkKh+IcBXQVZXxoT7TgQtV+bcIS4FvqfJacG5XoFq15ZKCCOOAYH2/5y4QS9qgSIp/HnukiTr4uNaOu4mfcQK3ALCKrkziRE7j6lZdT0R59tnnkg/MAQ0NDfTq1Suv12hv+DNJTEd/Lvvvv/8KVe2ZspPIXZjiOxEYCfwH2AvV2rh+Q7Fawt8HqrGauadhNYl7AHtg9YvXAT/Glnp3RjX/ITKFMH9NYeu+oPPi2k4CrUnQdxbowaH3XYJluqq4fl1Anwb9Z6htUmxJONT2EujxwetG0O1D57YLZEvq+ffYsFRYWZlkPSOgslKbLS36uOzG7Tpgrq6ia7NBy+muuw6sz8v1ckFHX7prDf5MEtPRnwvplnlVUeij8KDCcoU5CscE7YMUGhQGhfoervCxwlcKNQrDg/bNFd5QWKbwpcKrCt9Me+0cHYV0QGoAese19QaWRegbe72hrwglwO1YbNJpGVwnkewG+7dIT48ecOmlqftceqn183G5GTdl+4mU0tisrYRGJg+dmJfrOY5TYFSXoHoYqj1RHYTqHUH7HCxedE6o7/2obotqb1THELNeVReiuhuqG6G6Cap7ovpUAe+hYJZpT9A1oNuF2m6LtyKD9jtALw29PyBs1YIK6M2g1aDd48aOA30p7rorYtYo6MugJ4XO/4yQg1Ly+ffQysroziuTJ5sFJKIdfNz6/F9v5MjEJubIkZGu1717k0VaCOcj1Y5vbbQGfyaJ6ejPhSiWaQc4Cnsx9C7Mo7cn6N4k9+Y9GHQe6DDMm/fZsNIFvR70VdBeCcZuHsg9AvPmvYzm3rwng76PefL2B60lgjdveXl58k9LJ6ZgXwQ77KDar599ZB95JKOhZ56p2rOn6vr1eZpbAjr6F2Rr8GeSmI7+XDqLMi100oZTgO7AAuBOYLwqtSIMEqFBhEEAqjwBXI5tMNcFx4UAQczoL7BN6nnBuAYRxgZjF2Jew5cCX2Ab0keF5vBP4BHgPWA6Fqf0z3zetJMlK1aYu+8xx4AITJ2a0fCqKli+HBYvTtvVcRynVRS0BJsqS4DDErTPwQJ3w21XAlcm6FtHGldnVZ4Gtk9yToFzgsNpD0yfDuvXw777wqOPwjvvZDS8stL+1tVB3755mJ/jOJ0eTyfotH1ilujIkXa0wjIFmD07ZzNyHMdphitTp+0zdSr07m1acdQo+OQTWBqffSw5rkwdx8k3rkydts/UqWaRitjfWFtENtnEdHFdXe6n5jiOA65MnbbO+vXw7rtNSnTUKPvbiqVet0wdx8kXrkydts2sWeaKG1OmW25pRyuckFyZOo6TL1yZOm2bsPNRjJEjM1amVVW2zKuFSUXtOE4nw5Wp07aZOhXKymDYsKa2UaNgxgxYvTqymKoq+Oor+PLLXE/QcRzHlanT1pk61RRpeXlT26hRsG4d1Caq0JSYcKyp4zhOrnFl6rRtpk6FnXZq3hZb8s1gqdfDYxzHySeuTJ22y4IFMHdu8/1SgG22gV69MvLodWXqOE4+cWXqtF2mTbO/8cq0pMSs1Qws0z59oGdPX+Z1HCc/uDJ12i4xyzN+mRds33TaNItDjYCIx5o6jpM/XJk6bZepU6GiAjbbrOW5UaOgocHiUCNSWemWqeM4+cGVqdN2iaURTEQrnZDcMnUcJx+4MnXaJitXwgcfJFemw4db/GmGTkhffGHxpo7jOLnElanTNqmttf3QZMq0vNwUagaWqceaOo6TL1yZOm2TRGkE48kwraCHxziOky9cmTpQXw+jR8O8ecWeSRPhGqbJGDUK5s+PPO+YZerK1HGcXFNQZSpCHxEeEGG5CHUiHJOi71kizBNhqQg3iVAeOneaCG+KsFqEW+LGjRWhIXSsEEFF2CU4P0GEtXF9BuftptsD//d/8OKLMHFisWfSRCzzUUmKj2isHFtE63SLLaBbN1/mdRwn9xTaMr0GWAP0A8YC14kwPL6TCAcB5wIHAlXAYOCiUJe5wCXATfFjVZmiSq/YAZwCfAK8Hep2d7iPKp/k5O7aI/X1cOuttj95881twzpdv95iSFMt8UJT/GlEZeqxpo7j5IuCKVMRegJHABeo0qDKi8DDwE8SdD8OmKRKrSpfABOB42MnVblflQeBxREufRxwmypefCsRF13UVJessbFtWKeffGIxpImSNYTZeGMYPDgjj16PNXUcJx+UFfBaQ4BGVWaG2qYBoxP0HQ48FNevnwibqUZSoACIUAnsB/ws7tQhIiwB6oGrVbkuyfhxwDiAsjKhpqYm6qXbBV0XL2bPm25q+kW1Zg2Nkybx2oEHsqZPn0gyGhoacv5cNn/uOYYDb65bR0Ma2cMHDKDnyy/zesQ5dO06hI8+6ktNzctZzzMV+Xgu7R1/Jonx59JBUNWCHKD7gs6LazsJtCZB31mgB4fedzHzSavi+l0CekuKa14QLx90GGh/0FLQvUDrQY9ON//y8nLtcIwfr1pWpho8XAXVrl1VTzklsojq6urcz+u881RLS1VXrkzfd+JEm/dXX0US/Yc/WPeGhiznmIa8PJd2jj+TxHT05wIs1wLpmWIehdwzbQB6x7X1BpZF6Bt7nahvKn4K3BpuUGWGKnNVaVTlZeAq4MgM5XYMXnnF6oKGWbMGXs6v1ZaWqVNhhx3MWygdMSekWFL8NMScg32p13GcXFJIZToTKBNhu1DbTkCiCs+1wblwv/kZLvHuDfQH7kvTVQGJKrdD8c47MGGCvf7BD6B/f3P+ySB2My+kSiMYT4ZpBT3W1HHaICJ9EHkAkeWI1CGSNNIDkcGIPIrIMkQWIXJ5q+TkmIIpU1WWA/cDF4vQM1B2hwK3J+h+G/BzEYaJsClwPjSFwIhQJkI3oBQoFaGbSIv93+OAf6s2t2ZFOFSETUUQEXYHfknz/dnORV0dbLUVHHSQ1Q79+OPizmfRIvj88+jKtH9/2HzzyE5IHmvqOG2SFpEeiLSI9ECkK/AU8CywJTAQmJyxnDxQ6NCYU4DuwALgTmC8KrUiDAriPQcBqPIEcDlQDdQFx4UhOecDK7HwmWOD1+fHTgaK9kfELfEGHAV8jC0Z3wZcppqwX+dg9mwz18aMsffFdoRIVsM0GSK21BvRMt1yS+ja1Zd5HafNILIh0gPVBlRTRXocD8xF9UpUl6O6CtV3WyEn5xRUmaqyRJXDVOmpyiBV7gja56jFe84J9b1SlX6q9FblBFVWh85NUEXijgmh86tU2USVZxLM4WhVNguut70qf8/bDbfFzELx1NWZuTZkiGmaYivTVDVMkzFyJEyfbvu9aSgpsdt1y9RxCkNfKEPkzdAxLq7LEKAR1fhIj0QW5Z7AbEQeD5Z4axD5Wivk5BxPJ5hPJk5se5mFwjQ2wv/+Z9pFxKzTmpqmuNNiMHUqDBwIfftGHzNqFKxdC++/H6m7x5o6TuFYBOtQ3TV03BDXpRewNK5tKbBRAnEDsdXFv2M+MY8BDwXLv5nIyTmuTPNFfT38619tK7NQPPX1poRiXjn771/8fdNMnI9iZJhW0LMgOU6bIpNIj5XAi6g+juoa4ApgM2CHDOXkHFem+WLiRFNU0HYyC8UTM89iXjnF3jddtcqsy0yV6bbbQo8ekZVpZaXlx1+5MvMpOo6Tc2ZiS8FRIj3ehaTZ7DKRk3NcmeaD+nqzRmOsWdM2rdN4ZbrddubZW11dnPnU1toPj0z2SwFKS21MRI/emCE+Z07Kbo7jFALVDZEeiPREJFWkx2RgT0S+gUgpcCawCHg/Qzk5x5VpPpg40ZRCmLZoncbWOmPKtNj7plFqmCZj5Egbv3592q4ea+o4bY4WkR6o1iIyCJEGRAYBoPohFsFxPfAFpiy/Hyz5JpdTAFyZ5oNXXmla4o3RFjILxVNXZ44+PXs2tY0ZY5b1Rx8Vfj5Tp0KvXpa8PlNGjYKvvoqkIT3W1HHaGKpLUD0M1Z6oDkL1jqB9Dqq9UJ0T6ns/qtui2hvVMc2UZTI5BcCVaT545x04/3xbfuzWDc480yy9YmcWiicWFhOmmPumUWqYJiMDJ6T+/aGszD16HcfJHa5M80VtrTnGDBtmr9sisYQNYWL7poVWplFrmCZjxAj78RJBmZaWQkWFW6aO4+QOV6b5orbWvuBHjGibylTVPHDiLdNi7ZvOng3LlrVemXbrZsnxM3BCcsvUcZxc4co0H6xaZbGaw4fbMXcufPFFsWfVnIULLTYk3jKF4uybZuN8FCODtIIea+o4Ti5xZZoPPvjAli1jlim0Pes03pM3zP77299CLvVOnWrrr8OzyPw1cqT9cFmwIG3Xykrrunp12q6O4zhpcWWaD6ZPt78xyzTc1laIjzENs+225qVTyHjTqVNh++2he/fWy4g5IUVY6o0Z5P/7X+sv5ziOE8OVaT6orYUuXcyZZ9AgC/doT5ZpMfZNW5NGMJ5YsocIS70eHuM4Ti5xZZoPpk+HoUNNoYqYddoWLdONN4ZNNkl8fswYy9g0c2bi87lk8WIzEbNVpn36mJaMoEw9cYPjOEkR6ZLpEFem+SDmyRujLXr0JooxDVPIeNNMa5imYtSoSMu8AwfaFq179DpOJ0fkl4gcEXo/CViJyIeIDI0qxpVprmlogE8/be5IM3y4ec9GcIwpGIliTMPE9k0LoUxbU8M0GSNHmjXd0JCyW1kZDBjglqnjOPwSWAiAyH7Aj4BjgKnAX6IKcWWaa2I1NeOVKbQd61Q1vWVayH3TqVNNcW++efayRo2y+b73XtquHmvqOA4wAJgdvD4EuBfVe4AJWDHySLgyzTWxvdH4Zd7wuWLz5ZeWICGVMoXC7ZvmwvkoRgZpBT3W1HEc4Csg9kv+m8Azweu1QLeoQiIrUxG+LcKjIswQoSJoO1GEA6PK6BTU1lo2nnCy9q22MkeftmKZxjRIqmVeKMy+6erVrathmoyBA80RKaJH7+eft6xJ4DhOp+JJ4MZgr3Rb4PGgfTjwaVQhkZSpCGOBe4CPgK2BmKdTKXBO1IuJ0EeEB0RYLkKdCMek6HuWCPNEWCrCTSKUh86dJsKbIqwW4Za4cVUiqAgNoeOC0HkR4TIRFgfH5SJI1HtIS22tpbUrLW02qTblhJQqxjTMttvaxmI+401nzIB163KnTEUiOyFVVVlujc8+y82lHcdpl5wKvAT0BY5EdUnQvjNWxi0SUS3Tc4CTVDkLWBdqfxUYGfViwDXAGqAfMBa4ToQWKW9EOAg4FzgQqAIGAxeFuswFLgFuSnGtTVTpFRzhQqLjgMOwCuw7At8DfpHBPaRm+vTEWXxi4THFqBMaT0yZprNMC7Fvmos0gvGMGmV7pmlMTo81dRwH1a9QPR3VQ1F9ItR+Iap/iComqjLdDnglQXsD0DuKABF6AkcAF6jSoMqLwMPATxJ0Pw6YpEqtKl8AE4HjYydVuV+VB4HFEecfL/svqnymyueYt9bxqYdEZOlSM3PC+6UxRoywvcr6+pxcKitmz4YePWCzzdL3HTMG5s+HDz/Mz1ymTrV6qttskzuZI0fa8vEHH6TsFvst4U5IjtOJERnWLARG5JuITEbkd4iUphjZjLKI/eYCQ4D4r539gFkRZQwBGlUJe7NMA0Yn6DsceCiuXz8RNlONrEDrRFDgKeBsVRaFZE+Lk50wIawI4zBLlrIyoSbN3mHv6dPZGXhv/XoWx/XdZO1aRgLTpkzhi912i3gL+WH4W2/RY/PNeeO559L27d69O3sAM2+4gbnf/36L8w0NDWmfSypG1tQgVVW88/zzrZYRT4+1a9kdeP/OO5m/OPnHZe1aQWQ/amrqqKqanbPrQ/bPpSPizyQx/lyKziTgKuBDRAZiuqcGW/7tDfwukhRVTXuAngP6PujeoMtAR4MeB7oQ9NSIMvYFnRfXdhJoTYK+s0APDr3vYuuMWhXX7xLQW+LaeoHuCloG2g/0PtD/hs43gm4fer9dIFtSzb+8vFzTcsMNqqD66actzy1YYOeuvDK9nHyz886q3/52tL7r16sOGKD64x8nPF1dXd36eaxfr9q7t+opp7ReRiLWrlXt1k31rLPSdh0wQPW443J7edUsn0sHxZ9JYjr6cwGWawQdUbQDvlQYErw+S6E6eL2/wuyociJZpqpcLsLGmJXXDagGVgNXqHJNNOWfcEm4N7AsQt/Y60R94+faALwZvJ0vwmlAvQi9VfkqiewGe3ZZMn26LVkOGtTy3Oab29EWwmNmz4bdd4/WN7Zv+vTT9ptDcuerxezZ8NVXud0vBcvIsOOOkcNjfJnXcTo1pZgvD5ifzn+C17Mw/55IRA6NUeU8zNtpdyyQdXPVJi/ZCMwEykTYLtS2E5DIxbU2OBfuNz+DJd4wMSUZ0wKJZOfGzba2FoYNg5Ikj7UtePQ2NMCSJek9ecPka980H85HMWIevWkcpyor3QHJcTo504HxiOyLKdOYE9IA2LA9mJaooTE3ibCRKitUeVOV11VpEKGnSEqP2g2oshy4H7g4GLc3cChwe4LutwE/F2GYCJsC50NTCIwIZSJ0w35RlIrQTcSsbBH2EGGoCCUibAb8HahRZWlI9q9EGCBCf+DXYdlZMX16YuejGMOHmzLNl2dsFKJ68obJV7zp1Kn2wyPVM2stI0eaw1cas7OqynLsr1uXspvjOB2X3wInYfukd6IaS5/2feD1qEKiWqbHAYkKTXYHfhr1YsApwZgFWPzOeFVqRRgUxIMOAlDlCeBybDm5LjguDMk5H1iJhc8cG7w+Pzg3GPtlsQz7xbEaODo09p/AI8B7wfnHgrbsWLzYrLdUxa2HDzfLcM6crC/XalKVXkvGNtvkJ9506lSrrpNNDdNkRKxtWlUFjY1WKNxxnE6I6vNYBqS+qP4sdOafwPioYlLumYrQB1seFWBTkWYxpqXAd4H50efMEizGM759DtArru1K4MokciZgeRMTnbuTFIG2wd7oOWSQbCISseXbVFZW7FxtbWbKLJdETdgQRgT23x+efDK3+6ZTp8Lee+dGVjxf+5pZve+8A4cdlrRbONY00Va34zidANVGRFYiMgLbGpyF6uxMRKSzTBdhVqQCM7DM+rFjHvAv4NoMp90xiTkWpbNMw32LwezZ0LUrbLllZuPGjLGqN2liNyOzZIlZ6PnYLwWLox06NK0TkseaOk4nR6QMkT8DX2Chku8BXyByeSZ1TdN58+6PWaXPYgkXloTOrQHqVPEFMjBrc+ONbTk0GZtuatVRiumEVFdnJlgyJ6lkhPdNd9gh+3m8+679zZcyBVvqfeGFlF1i1qg7ITlOp+VybCvwZODFoG1f4I+YwfmbKEJSKlNVngMQYWvgf6qsb+1sOzyxNILplkBjaQWLRbrSa8kYPNiSyNfUwPjI2wjJyWUN02SMGgV33GH72UmyPXXrZka6K1PH6bQcA/wM1f+E2mYhshBbfY2kTCOZJ6rUqbJehP4i7CnCfuEj87l3MFTN2ozilTpihFVJaWzM/7wSka4oeDJynad36lSrptMvchhX5sSs3ghLvb7M6zidlo1JnMlvFrBJVCFRQ2P6i1ADfIZl16/BPG1jR+dm/nyzflLtl8YYPhxWroRPI1f2yR2rVtlcW+v8lMt901zWME1GTH4aj16PNXWcIiPSB5EHEFmOSB0iiSuKiRyPSCMiDaFjTOh8DSKrQueiBMdPA36ZoP0MYGrUW4i6cfY3oBEYBqzA1pN/CLwPHBz1Yh2WKJ68McIevYUmFpLTGssUchdvumaNlV7LtzLt29eWpiNYpnPmWDk2x3GKQouKYogks05eQbVX6KiJO39a6NzQRALiOAc4DpGZiNyKyC2BEj4WODvqDURVpqOB36ryAebZu1CV+7Fg14kpR3YGonjyxhg2rPmYQtKaGNMwsX3TbONNZ8yw8mj5VqYQqbZpVZVNpy0U9HGcTofIhopiqDagmqqiWO6xONMhwL1YiGbv4PXQYC6RiKpMu9OUVmkJsEXwegZWE7RzU1trDi5bbJG+70YbmTIrhmXamhjTMLnaN81nGsF4Ro2yPep994V58xJ28bqmjlNUhgCNqMZXFEtmnYxCZFFgSV6ASLwj7R+D8y81WwJOhepcVM9D9QhUD0f1fKALIvdEvYmoyvQDYPvg9VTgZBEqsRI1n0e9WIcl5nwUNZlBLK1goamrg9LS1OE76dh/f1i40BRUa3npJQvN6dmz9TKiMnKkKf6XXoKJiRdRPNbUcfJHXyhD5M3QMS6uSy/YkO41xlJgowTingdGYAbdEVhIS3gp9rdYFrwBwA3AI4i0tljyJsE1IhFVmV4FxKL8Lwa+BXyCpQf8vwwm1/FQbQqLicrw4ebEU+iEsLNn2zJtWdQytgnIxb7pY4/ZBuUfIhexbz2xHw6qcPPNCa1TjzV1nPyxCNahumvouCGuS/SKYqqfoPopquuDHLoXA0eGzr+G6jJUV6N6K+Yw+51c3k8yoobGTFG1ZPCqvA1UAbsBg1S5N2+zaw98/rmVEcskWfuIEeaE8/HH+ZtXIlobYxpm662hoqJ1ynTdOrjiiqbNySTKLafcFKrD0NiY0Drt2dOq47kydZyiMBOzXqNUFItHaaoI1przOSPDNDhGUD3mbWC5COfmeE7ti0ycj2IUK61gXV3rPXljtGbfVBX+/W/7EXH22U3L4UmUW86or4dbb216v2ZNUgXusaaOUyRUN1QUQ6QnIskriol8G5F+wevtgQuAh4L3myByECLdghSBY4H9gP8W4jbSKlMR+orwXRG+JUJp0NZFhDOB2UTMDtFhie19ZqJMd9jBFEoh903XrjUrOhcJ9seMib5v+uyzsMcecOSRtrTbtWuTEk6h3HLCxIkt412SKHCPNXWcotKiohiqtYgMCuJFY2UoDgTeRWQ5VsT7fiC2X9QFuATLHb8IOB04DNXEsaYiD6c8bHszMimVqQh7AR9hJcseB14SYXvgXeA0LCymc9famD7d8tElSVeXkB49LMykkJbpZ5+ZYsmVMoXUS71vvQXf+hYceKApy5tvhgMOaNkvn9bpK6+Ywg6zZg28/HKLrjHL1GNNHacIqC5B9TBUe6I6CNU7gvY5QbzonOD9b1DtF/QbjOrvUV0bnFuI6m6oboTqJqjuiepTKa66OM3xKVb/OhLpPFEmYibyJcDPgDOBR7FN39uDcmadm6hpBOMZMaKwlmlrioInI7xvGoubjTFzJlxwAdxzj/3AuPJKy+XbrRtcdVVk5ZYTYskazj4b/vEPyzyVxOO6shJWr7YET5kW1HEcpx2iekIuxaVb5t0JmKjKdKz4tgK/U+U2V6SYGVNbm9kSb4zhw03xrF6d+3klItuEDWFi+6bPPMPIM84wy3PuXDj5ZFOujz1mCnXWLDjrLFOkYMpNteWRJkNR1lRU2HNetChpl9hvDF/qdRynNaRTpn2w9WdUWYGlEszzN187oq4OVqxonTIdMcKWOGfOTN83F9TVmRKsqMiNvDFjYMkSNn7vPfjOd2DbbWHSJFOos2bBxRdbSbq2QOye//e/pF081tRxnGyI4s27qQh9RNgMs0x7B+83HHmeY9sltufZmmXemAIu1FLv7NlWpaW8PDfygvlLzLI8+GD48EO4+ur8VoJpDRGUqWdBchwnG6Io0xmYdboAy1TxRvA+5jG1MOrFAuX7gAjLRagTIXFlAOt7lgjzRFgqwk0ilIfOnSbCmyKsFrH419C5PUV4SoQlIiwU4V4RtgqdnyDCWhEaQsfgqPfQjJgijN83jMLQoZaNqFBOSLmIMQ1zyy1N+49dupiiHty6x5h3Bg60vymU6UYbQZ8+rkwdx2kd6ZTp/sABoSPZ+6i0qAwg0jL/oggHAedibtBVWHqoi0Jd5mJOUTfFjwU2xdJIVQGVWBaNm+P63K1Kr9DxSQb30ERtrVk9rVnOLC+H7bYrnGWaixjTGPX1pkxjIS5r1xYmAUNr2WILU/iffZaym8eaOo7TWlJ686ryXK4uJEKsMsAIVRqAF0U2VAaIT/xwHDBJ1TJgiDARmBLrF1SsQYRdgYFxc3487rpXQ+7uoxmZphGMZ/hwmDYtd/NJRmOj1Rj70Y9yIy9V/OY11+TmGrmkpMSs0xSWKZjhnotSrY7jtDNEegAjsZy/zY1M1fujiMgiSWvGDAEaVYmvDDA6Qd/hxLJaNPXrJ8JmqizO8Lr70TIt1SEiLAHqgatVuS7RQBHGAeMAysqEmnBcZWMj+9XW8tnQoXzSyjy1Vb16UTlrFi/897+sz9VeZgLKFy7k6+vWMXP1auZmW4sU2OWpp9goQYjLsief5K0cyM8HIzfaCKZPZ2qK+ZWWbsMnn/SnuvqFyDULktHQ0ND88+L4M0mCP5ciI/INLFFEomQBCpasKC2qWpADdF/QeXFtJ4HWJOg7C/Tg0PsuQRxFVVy/S0BvSXHNHUGXgO4bahsG2h+0FHQv0HrQo9PNv7y8XJvx4YcW2HHzzdpq7rnHZLz1VutlROHFF+06jz+ec9HV1dU5l5kXjjlGtaoqZZe//c0e04IF2V+u3TyXAuLPJDEd/bkAy7VAeqZVB9Qq3KLQPxs5rcrN20qiVwZo2Tf2OlHfhIiwLZa16QxVXoi1qzJDlbmqNKryMpYy6shkcpIS2+tsjSdvjNjYfO+b5jLGtL1SUWHpFFOkOPJYU8fplFQBE1Gdm42QQirTmUCZCFEqA9QG58L95kdd4g1qrT6NJZxomSy5Oa2rKhDzwt1hh4yHbmDbbc0xJt8evTGvmkGdOPNjRYU5Si1YkLSLx5o6To6pr2codCv2NNLwEjA0WyEFU6aqbKgMIEJPEZJXBrB8iD8XYZgIm2LZl26JnRShTIRu2Fp2qQjdRGz/V4QBwLPANapcHy9YhENF2FQEEWF34Jc035+NRm2thYJkU+C6SxfYfvv8W6Z1dVZjrBDFuNsqHmvqOIVn4kR6FdZoaw3XA1cgciIieyCyc7MjIpEckEQShqCAWXWrgI+xcJN0ZvIpWDjLAiyR8HhVakUYhMWzDlNljipPiHA5UI1VEvg3cGFIzvlx74/FQmcmACdioTQXijT1UaVX8PKoYA7lwGfAZaqE6nRFJFtP3hjDh8Orr2YvJxWzZ3fuJV5oijX97DPYbbeEXTbZxKKcXJk6Tg6oq4Mbbyz2LKJwX/A3vmg5ZOCAFNWbd3NgX2A9EFuTHIEtj74FHI5ZnPuqMjWZEFWWAIclaJ8DG5RdrO1K4MokciZgijPRuYtoHpMaf/7oZOcis3atpQE85JCsRTFiBNx1FzQ0QK9e6fu3hro6+NrX8iO7vRDBMgX7zeHLvI6TJR9+CPvsA+vWFXsmUdg6F0Kimt8vYc48A1XZT5X9sPjO/wBPYskRHgP+kotJtXk++sgUajbORzFi1u2MGdnLSoRq7rMftUf69rWE+2mUaVWVW6aO02pUzRodNSplYYk2hWpdyiMiUZXpGcDFasnug+uzArgUOEuVNcBlWNBrxyfmMJSrZd6wzFyzYAGsWuXKVCRS4oZYFiT1mkiOkxmLF8MRR8C4cZabs2vXYs8oOiI7InIbIm8i8gYityKS0XJeVGXaC5ry24bYkqbl2a8obBKI4lFba1l1tt8+e1mDB5vFlC8npFzWMW3vDByYNqVgZSUsWwZffFGgOTlOR+CZZ2DHHeHRR+GKK2wlKD6xS1tF5PvA20AFtgL7BDAIeBuRyHt5UZXpA8AkEX4oQpUIlSL8EJiEeegC7A4UqJ5YkZk+3cJauuXA47u01MJr8mWZxpRpZ7dMwfZNI1im4Eu9jhOJNWvgnHPgm9+E3r3h9dfh17+GqVM31Ct+i6YVzTbKJcClqO6P6gXBsT/wx+BcJKIq05OB/wKTgVnAJ8HrJzAPXYD3gZOiXrhdU1ubm/3SGCNG5M8y9YQNTcQSNzQ2Ju0Se0zuhOQ4afjgA9hzT/jzn62O8VtvwciRxZ5VaxhC4hDN28kg/jSSMlVlhSonY8XCRwE7A31UGR/Ej6LK1FSevB2GVavMASkX+6Uxhg+3L/kvv8ydzBh1dU0xH52digpTpCmq27hl6jhpUIV//hN23tlWeh56CK69Fnr0KPbMWssCYJcE7bsA86MKySiYVpXlqryryrSYEu10fPihpaTLtWUK+bFO3ZO3iXCsaRL69LEIJVemjhOivh5Gj7bvqB/8wCzRffaBd9+F73+/2LPLlhuBfyJyHiL7IzIGkfOxZA6JYk8TEjVpQzfMo/dAEpSoUWXHqBds9+TSkzdG2KN3771zJxdMK7TVot2FJhxrusceCbuIeKyp47Rg4kR44QXYfXeLHb3ySjjjDHPEbP9cguWD/zUwMWibiyUG+ntUIVG9b68FfgDcC7yMZYXonNTWQlmZFfbOFYMGmTmUa8s0FmN6QCb12zswERM3eKyp44T47DOLHVWFlSvhySfhG98o9qxyh1W2+SvwV0Q2CtoiF1WJEVWZHgb8UJWnM71Ah6O2FoYOzW0MVUkJDBuWe2X6xRcW5+HLvMamm0L37pGU6UsvFWZKjtOmaWiAffdtymTUpQs88EDHUqZhWqFEY0S10VcAqb+BOgu5yskbz4gRuQ+P8bCY5oiYdRoh1vTLL/PjD+Y47YbPP4evf735Ms2aNXDzzSmd+NoFIu8ismnw+r3gfeIjIlGV6eXAr0TafPb//LJ8OXz6aW6dj2IMH27ZihYuzJ1MT9jQkgxiTX3f1Om0vPOO7Y9+8IFta4VpbLQ91PbNv4HVodepjkhEXeb9Jpbo/mARZgBrwydVaffuXJF4/33bN8iHZRqTWVsLY8bkRqbHmLakogKeeipll3Cs6U47pezqOB2PRx6Bo4821/ZttrEIhjBr1sDLLxdnbrlC9aLQ6wm5EBnV0lyEZUF6FpiHlU8LH52D2J5mPizTfITH1NVZDdPNNsudzPbOwIHm5p+imoXHmjqdElX429/g0EMtK9trr5llGmQyana8806xZ5s7RJ5FZJME7b0ReTaqmEiWqSonRJ9ZB2b6dCgvt19ruaZ/f0uskMt901iMqUjuZLZ3KiosTri+vsm7N47NNzc/JVemTqdh3ToLdbn2Wjj8cLj99sImYRDpg6Wn/RZmvP0O1TsS9Ds+6Lcy1Po9VGsyktOcMUAij9Ju2IpsJDpHYvpcUVtrv9hKI9WKzQyR3KcV9KLgLQmHxyRRph5r6nQqvvoKfvxjeOIJy7P7xz8WI370GmAN0A+rPvYYItNQTfSF+Aqq+2QtR2Tn0LsdEVkSel8KHAR8HvUGkipTEd4FRqvyhQjvkSK2tNMkbZg+HfbbL3/yhw+He++1ZZRcWJN1dZY702nCY00dp4m6Ovje92w598Yb4cQTCz8HkZ7AEcAIVBuAFxF5GPgJcG4e5byJ6TXF6nLHsxI4PerlU1mmYW+n+6IK7KiUgH0B58P5KMaIEXDDDeZ2vlWiincZsGwZLFnilmk8sZSCaZRpZSW88UYB5uM4xeL11y0V4KpVZpUeeGBeLtMXyhB5M9R0A6rhNH1DgEZUw1XHpgGjk4gchcgiYAmWjP6PqK5rhZytAcEKt+wOhEMp1gALUE1eFSOOpLa8KhfFioEHr5MeUS8mQh8RHhBhuQh1IhyTou9ZIswTYakIN4lQHjp3mghvirBahFsSjD1QhA9EWCFCtQiVoXMiwmUiLA6Oy0VIawZ2W7/eXuTD+ShG2KM3WzzGNDEbb2zZptLEmn7xhdU6LikxK3XKlGjip0yx/gccMLpV4zK5XmvGFGOc04aI5di98Ub726MHvPJK3hQpwCJYh+quoSM+320vYGlc21JgowTingdGYGltjwCOBs5uhRxQrUN1NqolqL4ZvI8d9Zko0kCeFuwAvRP0btBeoPuALgUdnqDfQaDzQYeDbgpaA/qn0PnDQQ8DvQ70lrixfQO5PwTtBvpn0FdD538B+iHoQNABoDNAT0439y3LysyPbdYszRvz59s1/vrX7GU9+qjJeuWV7GWloLq6Oq/y88IOO6gefnjS05Mnq3bt2tx9sUcPa0/F5MnWrxDjCnmtbMaFaZeflQJQ0Ody8smqIvYPuNdeqgsW5P2SwHJN9f0KoxRWxLX9WuGRlOOs31EKb+VATpnCXoG8nzY7Iuq3qInu+wCXkjzRfe8IMjasZ6vSALwoQrL17OOASarUBmMnAlNi/VStILkIuwID48YeDtSqcm/QZwKwSITtVfkgkP0XVT4Lzv8Fq8N6far5d1O1X3H5TICwxRZWod4t0/ySJnHDeedZKF2YFStg/Hh4++3kYm+80foVYlwhr5Vq3Hnnwdixycc5bYDGRivW/eijto2kak6UU6aY63rxmYktBW+H6kdB205AlC9ChQ0ri62TI7I98AhNy76N2BboWmyr87YoNxHVm3cSVsf0BiybfmsS3Q8BGlWJsp49HHgorl8/ETZTTRvXOjzoD1jZOBFmBe0fxJ8PXifcCBVhHDAOYIgoX1VU8Pbzz6e5fHbsNHAgJS+/zDs1NVnJGfz88wzs0oXn33+/ZdB1DmloaKAmy7kWmqGlpfSZNYtXksx7zpzRkGDlf9ky5brrkq/8rFxZWrBxhbxWqnFz5ig1Nc8lHRemPX5WCkFrnkvXxYsZdvHFzLjwQtb06dPsnKxbx0YzZ7LxtGlsMm0aG0+fTtlyq5gZ0zzrRag/80w+OvPMnNxDVqguR+R+4GJETsS8cA8F9mrRV+TbwNuozg+U4AVYAZbM5DTnb8BbQf95wd+NgeuA8zO4j0jLs1+B7hHV3E0iY1/QeXFtJ4HWJOg7C/Tg0PsuwdJSVVy/SxIs804KLwkHbS+BHh+8bgTdPnRuu0C2pJr/TqB6/PHR1jWy4dRTVTfaSHX9+uzk/OhHqtttl5s5paBdLt1deKEtda1enfB0ZWXz5czYUVmZWmwhx7WHOcbTLj8rBaBVz2X8eNWSEtVTTlFduVL1uedUJ05U/cY3mq/H77CDLe1ec41qeXnzf7zu3VXr63N+P/GQbpnXlln7KDyosFxhjsIxQfsghQaFQcH7KxTmB/0+UbhYoUtaOamvvVhhRPB6qcLQ4PVohXfTjg+OqMFEC7B6b9nQAC2Wg3sDibL0x/eNvY6S0T/ddRLJbrBnl5wyyK8nb4zhw80TN423aVq8KHhyKirs62Tu3ISnL720Zbx6jx7WnopCjmsPc3TyxNy5cNNNlnzk+uvNqW70aPj97y23989/DvfdB/Pnw4wZcN11FtancV9xbSnHruoSVA9DtSeqg4glWlCdg2ovVOcE73+Dar+g32BUf4/q2rRyUiNYMRcwj94BwevPgG2j3kJUZXoecLEIvaIKTsBMoEyEcCHQZOvZtcG5cL/5EZZ4W4wN9mq3CV0nkexom5T9+0fqlhUxb+Hvfje7ygyzZ3uC+2SkiTUdO9a2lmLJoyor7X26vcHm47SV46JdLzdzbN04sLHXX+/7pQVlxQpTojvtBKuDqEVVSyTz0EOwaJHtjf7973DEEeaDEeOVV1o6AnSEHLu5YTpNOuF14LeIjAYuAj6OLCWK+Qr6Hugy0BWg74O+Gz6imsGgd2EevT1B9ya5N+/BoPNAh2HevM/S3Ju3DPPU/SPo7cHrsuDc5oHcI4L2y2juzXtycA8DQPuD1hLBm3cXUP3JT7Jb74jC4sW2BCNiSzitYcUKkzFxYm7nloB2uXRXW2vPZ8qUvF2iXT6XiNx9tz2+11/PbFxHfibZkPa5zJypetZZqpts0vTdUITl2tZClGXeYh5wkMLhwevBCjMU1issUBgTVU5Uy/Q+4ArgMuAuWlmiBjgF6I4tG98JjFelVoRBIjSIMMgUPE9gZd+qgbrguDAk53wsO8W5wLHB6/ODsQsxr+FLgS+APYCjQmP/iXluvYf9InksaEvPffflv45f+Bdna+sGzrEVEV/mTULMMk0Ta+okJpYEzH2JQsTiN3P1/bBunVmbBx0EQ4bAP/5hrw891Ap0h2lLy7XtEdX/onp/8PoTVIcBfYF+xHL+RpOT1prsAno5aGXRf0EU8dgFLPiwtdZiVGKOBaDapUvrrvff/9r455/P/fziaLfWxsYbq552Wt7Et9vnEpGhQ1W/+93MxnToZxJ2CMqEuXP1ix13bLIs581TveQS1YoK+z88cKCtMMXOjxypCT3BRo7M7f3kENq6ZZqjI21ojCprRRgPXNsapd+hiFWZv+AC2HLL3Muvrzf5sWxLa9e27noeY5qeCEXCneSMHg133WVGUT7qPrQr6uvhX/9qcghavtz8FbbYwo5+/Zpeb7JJ87zbEyey8XvvWaBv9+62+rV2rWUkuuoqOOSQ5sW5O1Lps2IiUg0RQzxVD4jSLWqc6ZPAAcBNEft3XGJLKtdck3vZEyc2KdIYa9Zkfr3Zs+0brhAOU+2VgQNdmWbBmDHmkDRtGuy8c9ruHZtx40wBgv3/veuupu2aeLp0aVKsG28ML7yAqMKDD8JGG8Epp8DJJ8P22xds+p2UcK3LUmAsFmP6WtC2O7AVMDmqwKjK9BngDyLsiAW3Lg+f1CAjUacgnx5wiTzuGhsz35yqqzPLqyzqP28npKIidcofJyWjR9vfmppOrkzfeMMyC4UpKbEfamVlFp6yYEHzI9b2+uv2/xvsx+9RR1lxbif/qDZVgxH5K3ArcAa2LB1r/xuJMpUkIeq37dXB318mmham2Ts008vLrbpCPolfwqmrs1CZAQNsZyRqWTaPMU1PRYV9oa1ebQXfnYzo3x+23Raeew5+9atiz6ZIrFoF3/lOy/bGRqsJes01ybdn6uth8ODmYyZPhosvzs8WkpOKnwJfb6ZIjWuBV4EzogiJ5M2rSkmKo8Mr0qJRWQmXXw5PPWV7p1HxouDpcY/erBk9Gl54oeXORKdA1ZZkFy1qeS7K6lWiLR33yi0WAnwtQXuitqQUvJy6kyG/+IV9a/3qV/B5hKLva9dahhRP2JCaiHVNneSMHm2l6t57r9gzKQI33GA/cM8/P5FvbXpHIU+i0Ja4CfgXIuciMiY4zgVuBCJbMZE31YLKMQcDg4Cu4XOqXBxVjpMhJSXmKbjjjuaY8PDDqZd7P/vMfvG6ZZoat0yzJrZv+txzlpSn0/Daa3D66XDwwTBhQutkhJRtTU0NY8aMycnUnFZxDpb74AzgD0FbPfAn4C9RhUSyTEXYE/gIS9wwEfgZlmLwN8CRkafstI5tt7UkqI8+Cnfembrv7Nn215VpatKkFHTSM2gQbL11J0veMH++peobONBKmHX6uKAOgOp6VC9HdQCwCbAJqgOCtsgFwqMu8/4Zqyc6AFiFhckMAt7EsiI5+eaXv4Q997RfxPPnJ+8XizH1Zd7U9OgBffq4Ms2S0aPh+ec7yb7punXmcbt4Mfz73/b5cToWql+h+lVrhkZVpjsCV6uiWOHUclXmA78FJrTmwk6GlJZakuuGBlOoyairs2XgmOXlJGfgQF/mzZLRo023zJhR7JkUgHPPNTP8hhtg1Khiz8bJBpF3Edk0eP1e8D7xEZGoe6bhnfL5QCXwPlbOzDMDFIoddoALL4TzzrNfxkcc0bLP7NkWt9C1a8tzTnM8C1LWxLb6nnuuqeBRh+Tuu+Evf4FTT4Wf/KTYs3Gy599ALLPGfbkQGFWZvg3shpVRqwEuEaEflmQ+suZ2csDZZ1vKsVNPtW+yzTZrft5jTKNTUQGvvlrsWbRrqqps77Smxj6SHZLaWqsRutdecOWVxZ6NkwtUL0r4OgsyqWcaq6R8PlZA9R/ApsC4XEzEiUiXLrbcu3gxnHVWy/OuTKNTUWHPccWK9H2dpMT2TVuEvHcEli6FH/zAUv3de6+v+DhJiWSZqvJm6PVC4Nt5m5GTnpEj4Xe/swDvH//YComDBX3PmWNtTnpisaaffw7bbZe6r5OU0aPh9tvhgw9sJ6LDsH49/PSn8Omn8Oyznuu6IyHyHtET3e8YpVtGyVtF2BXYBnhUleUi9ARWq7IuEzlODjjvPLj/fkvqUFtrSbPr683j0C3TaITDY1yZtprwvmmHUqZ//KPFdV91Fey7b7Fn4+SWnOyThomkTIP90YexfVMFtgM+Aa7EQmUi5S50ckh5uS33fv3rto96ww0eY5opHmuaEwYPtvTRNTWWV6RD8N//WunDY45J7T3vtE9ytE8aJuqe6V+x8jSbAeENpnuBb+V6Uk5Edt8dfv1ruPFGePppjzHNlAED7K8r06wQsaXe557rAPum9fWwxx62VTJihP1IjVpgwunURFWmBwLnqfJFXPssLHmDUywuusiWKE86ydKcgTtJRKV7d+jb12NNc8Do0TBvHnz0UbFnkiW//72VRlu5Eh54AHr2LPaMnEIgcgIiTyLyASKfNDsiElWZdqd5rGmMzbFl3kiI0EeEB0RYLkKdCMek6HuWCPNEWCrCTSKUR5EjwlgRGkLHChFUhF2C8xNEWBvXZ3CiObQLune35d66uqYC4n+JnE7S8VjTnBDO09tueest+78UwxVp50DkbCwH71tAFfAgVjy8D5YEPxJRlenzwPGh9ypCKZYB6ZmoFwOuwZRyP6yy+XUiDI/vJMJBwLmYRVwFDAbCa9xJ5agyRZVesQM4BdvfDVeCvjvcR5XIvz7aJPvsAyec0JTT7eabzUxw0uPKNCcMGWJlONttnt5777U40nBeRC+H1lk4CRiH6u+AtcDVqH4fU7CRHVCiKtNzgJNEeAooDy4yA9gb+F0UAYHn7xHABao0qPIi5tSUKJ3IccAkVWqDpeWJBMo8QzkxWbcFqRA7LuF9Ha+LGJ2BA12Z5oB2u2/61Vdw/PHwox9Z+cIYa9b4j9LOw0Dg9eD1SqB38PpOTNdEImqc6QwRvgaMx1IwdcOcj65RpT7itYYAjarMDLVNA0Yn6DsceCiuXz8RNsP2aCPJEaES2A+rchPmEBGWYGV2rlblukQTFmEcQVKKsjKhpo3+7O66eDF73H57U5X2NWtonDSJ1w48kDV5Tsbd0NDQZp9LFAatXcvgL7/khccfp7F795zJbc1z6bp4McMuvpgZF16Y93+3fLDVVv35/PMh3HHHqwwY0HL3p619VnrX1rLDpZfSbf58lg0ZQq9PPqFkXVOU3/q1a6k/+WQ+OvPMvM6jrT2XTsg8oC8wB6gDvg5MBbYlaiwqgKq2+gCtBL0nYt99QefFtZ0EWpOg7yzQg0PvuwRVd6sylHNBfDvoMND+oKWge4HWgx6dbv7l5eXaZhk/XrVr1+blibt2VT3llLxfurq6Ou/XyCuTJ9vzev/9nIpt1XMZP161pKQg/275oLbWHuWkSYnPt5nPytq1qhdeqFpaqlpVpfrii6ojRyYq8W3teabNPJc8ASzXLPRM3g/4l8KE4PXJCisVqhWWKtwYVU7UZd5kbEJ0M7iBJvM5Rm9gWYS+sdfLMpTzU+DWcIMqM1SZq0qjKi8DV9Hea7K+8ootS4VZswZefrk482lPtJVY0/fftzCM9evb7fLiDjvA5pu38X3TWbMsAcNFF1kM6dSpsPfeVqw7kToNFfF2OhgiBwavxgGXAKB6Pbal+B6WRveUqOKyVaaZMBMoEyGcamYnoDZB39rgXLjffFUWR5Ujwt5YRZt0mS4UaN+BZP5F0HpiKQWLrUxPPNH2uqHd7nmH903bHKpwyy2WivODD+Cuu+C22yxzmFN8RPog8gAiyxGpQyRppEdozLOIKCJlobYaRFYh0hAcH6aQ8FQQ+vI7YIsNrap3o/pLVK9GdW3S0XEUTJmqshy4H7hYhJ6BsjsUuD1B99uAn4swTIRNseT6t2Qo5zjg36rNLVYRDhVhUxFEhN2BX9J8f9bpTMQSNxQz1rS+3lYXYrRj55fRoy09dCwZV5tgyRJzMDrhBNhlF3j3Xc9f3fZoEaGBSItIjw2IjCW5z89pqPYKjqEprjkc0yWnA3WIPIbIYYiUphiTlEJapmAmc3dgAeYpNV6VWhEGBfGegwBUeQK4HKjGNoTrgAvTyYmdFKEb8CPilngDjgI+xpaFbwMuU03Yz+kMlJdDv37FtUzPOqulC+yaNe3SOm0z8ab19TaZe++FHXeEhx6CP/0JnnmmaWnfaRuIbIjQQLUB1dQRGiIbY/rgnKyuq/o+qr/BvHl/jK1S3gt8jshliKRSxC1I6c0rwsNpxsfvXaZElSXAYQna5wC94tquxHL/RpYTOr8K289NdO7oqPN1OgnFjjWtrm7Z1tgITzxR+LlkyfDhVmK3pgaOO66IE7nwQqsL9/zzMHSoJazfeeciTqjz0hfKEHkz1HQDqjeE3g8BGlGNEukB8AfgOswLNxF/RORPwIfAeajWpJyg6jrMQr0fkf7YnukJwG8QeQnV/VKOD0gXGrM4wvlPo1zIcdosAwfCzJnp++WD9esti9W3vw3/+Y+1rVhhTjGffmr5+dpRRZuSEthvvyJaplOnwt//bsvkAKWl9lwHt98kZ+2dRbAO1V1TdOkFLI1rWwps1KKnyK5YfoMzMIsynt9iORDWYKuQjyAyEtVZkSarOheRa7GVywnBtSKRUpmqckJUQY7TbqmosHqVxeCllywV5B/+0NTWo4flhd11VzjsMHj1VStO3U4YPdqm/7//FWhFdckSuOMOSwX4zjum0UtK7IdKaaml14yl2nTaItEiNERKgGuBM1Bdl7AAgeproXe3InI08B3gH2lnIfINLCfBYVia3DuBf0W7hcLvmTpO26OiwjLhfPVV4a89ebLlgD300ObtVVVwzz3w4YdWoDqc5q6NU5B908ZGePJJOOoo2GorK5MmApdeaoUeYs+rHTtzdSJmYkvB6SI9egO7AncjMg94I2j/DJFkBWdTR2uIDELkQkQ+BZ7EIkDGAf1RPRXVyCERrkwdp1ixpqtXm8L8wQ8SJ1U/4AC44gp48EFTErkk5qCTByXzta/BJpvkKN40fp6ffGKVXbbeGg46yBTqL35hFulbb5lXdvwPj3YaatRpUN0QoYFIT0SSRWgsxZTdyOD4TtC+C/AaIpsgchAi3RApCzx+9wP+m/C6Ik9hedt/AdwFDEF1DKqTUY1cwCVGpHSCjtOhCceaDk/ujZ9z/vMf+PJLOPbY5H3OOMOUxIUXWozkIYfk5toTJ8KLL9rfHC+BlpbmcN80Ns9jjzWP52efNQv0W9+yHxrf/z5069bU3xOYtFdOwSq0LMB8ccajWovIIGwPdBiqcwg7HYnE/uHnB8u+G2PJF7YHGoEPgMNQTRZruhI4HHgM1cZsb8CVqePELNNCx5pOnmxhOQcemLyPiGVGmjHDFMrrr5t3ajbU18OkSWbB/etfcO65Od/cHD3aHGjnzoX+/bOY57/+ZfOMhbRMnGhuwsnm64lK2ieqiSM0TIH2atFu52YTXsJVXQjslsE1v5/JFNPhy7yO07+/Ka1CLvN+8QU8+igcfTSUpflN2727efSUl5tDUjZ7u2+/DXvs0WS9rVlj3sLjx1sYSY72ZnOyb3rssU2VXMrK4Hvfg/PP9zhRp03iytRxunQxJ5ZCKtN//9sUWaol3jCDBtn+6kcfwU9+krnSe/99+OEPLQNQ/H2uWwe33moasLISzj7blG58IokMGDkSevfOYt/02mube1ivW2fpAN2RyGmjuDJ1HCh8XdPJk2H77TNLJDBmDPz1r7Z+evHF0cZ8+qnV6xwxAv77Xwu36dq1eZ/SUlPqd9wBo0bB3/5mSneHHSwhfDgGN6LjUmmp5ZNvlWV6771w6qm0CH1wRyKnDePK1HHAlg4LtWc6Z45pmbFjWyqMdJx2mu0ZXnSRpchLxty5ppCGDoW774Zf/co8YdetS+yg88YbtuT88MMwf77t0261lV1n6FBTrn/5C/z2t02OS2kYPdoiezIyJh980Kq59OiROMWiOxI5bRRXpo4DTSkFs1jajMwdd9jfY9IXxmiBCFx/Pey2m1mTzz/f3FJcvBjOOQe22cYU4oknWtmxP/8Z+vaNVmGoTx846SRLc/i//5kSLSmB3/wGbr/dlpgnTUqrJWP7ps8/H/HeHn3UEtLvuqtZwF4JyWlHuDJ1HDBluny5harkE1VTSHvv3foUd926wf33m/V2yCFmKZ5/vlmRW29tyu9HPzKz8Nprs3Cnxarq/OpXTZZraVBQY/Vqu3a8lRti552hV6+I+6ZPPAFHHAE77WSve2eU9ttxio4rU8eBwtU1nTatKcwlGwYONAv1q6+aLMUJEyz+8r33zKEol/lo6+vNo7gxFI735ptWkeWNNxIOKSuDffaJsG/61FPmpTx8uCVh8BqjTjvElanjQOFiTSdPNu/hH/4we1lPPdVkKYrAkUfCfffBsGHZy45n4sSWHsRlZfa89tzTlpZXrmwxbPRo++2wcGESudXVlnhh6FC7n003zf3cHacAuDJ1HChMSsHGRrjzTvjOd6xOWTbU11vO2ZilqAqPPZa/0JFEmYXWrTPr9+c/tz3ZnXayJecQKfdNn3/eYke32Qaefjr7Z+I4RcSVqeOAea6WluZXmdbUmJft2LHZy0pkKeYzdCSZ49K775qj09NPW4KF/fazpPMNDYD5EvXokWDf9OWX7UfFoEGW3WjzzfMzb8cpEK5MHQdMkeY7ccPkyeZY873vZS+rreWgPfBA26s9/XTL9fu1r8HTT9OlC+y1V9y+6WuvwcEHm2PUs89aSkXHaee4MnWcGPmMNV2xwrIeHXmkpQfMlighLoWmVy+46ipbvu3aFb75TTjxRA7a40sWvlfPiFPPNE/dgw6CLbaw/dKttirefB0nh3iie8eJUVGRP2X0yCOwbFn2XrztgX32galTzbv4iis4vc/jDGMnNpvxrtVtjVmkAwYUe6aOkzMKapmK0EeEB0RYLkKdCEmj1kU4S4R5IiwV4SYRyqPIEaFKBBWhIXRcEDovIlwmwuLguFwkRfFYp/MQSymYj8QNkyeb/JhHTkene3e47DJ49VVWdtmI7/A4grJ+zVoeP+Ee2ytNw5QpViO9pMT+TpkS7dLtZZzTsSi0ZXoNsAbohxV3fUyEaarNK6qLcBBwLnAAMBd4ALgoaIsqZxNV1iWYwzis1M9OWBX2WIHY67O/PaddU1EBq1bBkiW59SxdtMiWN3/1K/vG7URMmbkbKxeM5ng+poxG1lLGnItv4Z/9duPII5OPu+8+OOuspmibujpLytTQQJsfN26cvc6Fn5nTjlDVghygPUHXgA4Jtd0O+qcEfe8A/UPo/YGg86LIAa0KNpDKkszjZdBxofc/B3013fzLy8vVaUl1dXWxp5A77rvPdh7feSdrUc2eyzXXmNx3381abntj1wFzdQXdmu3sLqe79qM+wYZvxzkqK6M/ow71fygBwHItkJ4p5lFIy3QI0KhKqAQF04BE617DgYfi+vUTYTNgUEQ5dSIbLM+zVVkUkj0tbuzwRBMWYRxmyVJWJtS0up5Ux6WhoaHDPJeNFixgF+C9//yHxVmmFQw/l1HXXkvp4MG8uXhxFjXJ2icnfH43QvMQnhIauYCL+fD0s5KO+8c/toWEuy/K6ad/3ObHzZmj1NREK5nTkf4PdWoKpbVB941Zl6G2k0BrEvSdBXpw6H2X4BdfVTo5oL1AdwUtA+0Heh/of0N9G0G3D73fLpAtqebvlmliOtSv6s8/N7PimmuyFrXhuXz8scm87LKsZbZHpncZmdB0m95lZMpxlZWts/jay7gwHer/UALoJJZpITdwGoD47NW9gWUR+sZeL0snR5UGVd5UZZ0q84HTgG+JbBiTSHaDKnnwOnHaFf36NaXIyxVTpliqv6OPzp3MdsTUm9+hZw9FaDp69lCm3pzaa/rSSy3ZQ5gePay9rY/r2jX9OKfjUUhlOhMoE2G7UNtO0Nz5KKA2OBfuN1+VxRnKATYoydhaTCLZycY6nYnSUgvXyFXiBlXz4h0zpildYSdj7FhLkFRZCSJKZaW9T+ec03wc7WZcly7mu3bUUanHOR2QQprBoHeB3ok5Ee0NuhR0eIJ+B4POAx0Guinos4QclVLJAd0DdChoCehmoHeDVofGngz6PugA0P6gtaAnp5u7L/MmpsMtUe2zj+ro0VmLqa6uVn3tNVvzmzQpa3kdgQ73WUnAPffYP/mdd0Yf09GfC77MmxdOAboDC4A7gfGq1IowKIgHHWQKnieAy4FqoC44LkwnJzg3GHgCW/adDqwGwmts/wQeAd4Lzj8WtDlOU6xpLpg8GcrLrU6n0yk44ggr2pModbLTsSlonKkqS7AYz/j2OUCvuLYrgSszkROcuxNTsMnmoMA5weE4zamosMLbqrZu10pk3Tq46y4roO31OTsNJSVwwQW2Rf7vf+em0p7TPuhcEeSOk46KCksYn7QAZzQ2festk9EZ0gc6zfjhD2H77d067Wy4MnWcMDmqa9rvqaegTx/49rdzMCmnPVFaCuefb0V0Hnyw2LNxCoUrU8cJM3Cg/c1GmS5bRt8XX4Qf/cjiJJxOx1FHwZAhcPHFtmPgdHxcmTpOmJhlmk2s6S23ULp6tVulnZjSUjjvPJg2DR5+uNizcQqBK1PHCbP55mZNZmOZXn65BTc/8USuZuW0Q445BrbZxq3TzoIrU8cJU1KSXXjMc8/BZ59ZhpBbboF583I4Oac9UVZm1unbb8NjjxV7Nm0ckT6IPIDIckTqEElanjM05llEFJGyUFvmcnKEK1PHiae1yrSuDr773ab3jY3m0ul0Wo49Frbe2q3TCITLao4FrkMkYQESAETGkji0MzM5OcSVqePEU1GR+Z7p/Pmw//6wfHlT25o1cPPNbp12Yrp0gf/7P3jjDV/1T4pIT+AI4AJUG1B9EXgY+EmS/htjSXzOiWvPTE6OcWXqOPFUVMDnn0cPEvzySzj4YJgzx749w7h12un56U9h0CC46KLOaZ32hTJE3gwd4+K6DAEaUY0vq5nMovwDcB0Q/ys1Uzk5xZWp48RTUQFr15q1mY4VKyzLUW2tZTtfu7b5+TVr4OWX8zNPp13QtatZp6+9Bk89VezZFJ5FsA7VXUPHDXFdegFL49qWAhu1ECayK7A38I8El4ouJw+4MnWceKLGmq5ZA0ceCS+9ZHl4Z83aUNKyprq6qbzlO6nLjTkdn+OPt49VZ7VO0xCtPKdICXAtcAaq61otJ0+4MnWceKLEmq5fb9+Qjz8O119vCRocJwnl5fC739kiRXV1sWfT5piJLQWnK6vZG9gVuBuRecAbQftniOybgZy84MrUceJJl1JQFU4/He68E/70JxgXvwXkOC352c+gf3+zTp0QqsuB+4GLEemJyN7AocDtcT2XAv2BkcHxnaB9F+C1DOTkBVemjhPPZptBt27Jlenvfw/XXgvnnAO//W1h5+a0W7p1s4/L889bOLLTjBZlNVGtRWQQIg2IDAoKh87bcECsGsV8VNeklFMAXJk6TjwiyWNNr7wSLrkETjzRrFLHyYCTToItt3TrtAWqS1A9DNWeqA5C9Y6gfQ6qvVCdk2DMbFSl2f5pMjkFwJWp4yQiUazpzTfDr39tTkfXX59VvVOnc9K9uy1oVFfDCy8UezZOLnFl6jiJqKhobpk+8IBZo9/8pnnulpYWb25Ou+YXv4AttrCsSE7HwZWp4ySiogLmzrWkC888YzW1dt8d7r/fXDMdp5X06AFnnw1PP+0hyB2JgipTEfqI8IAIy0WoEyFpEmIRzhJhnghLRbhJhPIockTYU4SnRFgiwkIR7hVhq9D5CSKsFaEhdAzO31077ZKBA02R7rijJWUYMsSylffqVeyZOR2A8ePto3TggXDAAaOpqoIpU6KNnTIFqqqsJkN7GGc/Hzo+hbZMWyQhFmmZ6kmEg4BzgQOBKmAwEN6yTyVnU+CGYFwlFrB7c9wl7lalV+j4JCd353QcYuExM2ZY+Y8nn4Q+fYo7J6fD8OCDsHo1rFoFqkJdnUVYpVNUU6ZYv7o6i9BqD+M6C6IFSschQk/gC2CEKjODttuBz1U5N67vHcBsVf4veH8gMEWVLTORE5zbGXhO1VJKiTAB2FaVYzOZf7du3XTVqlUZ3XNnoKamhjFjxhR7GrnnmWfgG9+w1926waefmhtmRDrsc8kCfyZNVFUlVjRlZbYIkoyZM2Fdgtw/bXtcT1SXd3hvvUQlbPLFEKAxpgADpgGjE/QdDjwU16+fCJsBgzKQA7AfLTNgHCLCEqAeuFqV6xINFGEcMA6grEyoqalJconOS0NDQ4d8LttddRX9S0qQ9etZv24d9SefzEdnnhl5fEd9Ltngz6SJOXNGAy31y7p1yuabL2w5IGDGjM3b9bgOjcXB5v8A3Rd0XlzbSaA1CfrOAj049L5LkOi0KkM5O4IuAd031DYMtD9oKeheoPWgR6ebf3l5uTotqa6uLvYUcs/cuardusUy69rRvbtqfX1kER3yuWSJP5MmKiubf7xiR2VlRxzXQ7VAeqaYRyH3TDNJQhzfN/Z6WVQ5ImwLPA6cocqGiC5VZqgyV5VGVV4GrgKOzPBenI7MxIkty695KTUnh1x6aUu3nB49rL2jjuvoFFKZzgTKRIiShLg2OBfuN1+VxVHkiFAJPA1MVE2bl1HpdOsRTkpeecUqwoTxUmpODhk7Fm64war2iSiVlfZ+7NhMxtEuxnUWCuaABCDCXZjyOhFLVPwfYC/V5gpVhIOBW4ADsH3NfwOva+BglEqOCAOA54HrVflzgjkcGpz/EtgNeAD4P1VuTTV3d0BKjDuVJMafS0v8mSSmoz8XEVmhqj2LPY98U+jQmBZJiAMFOCiI9xwEoMoTwOVANVAXHBemkxOcOxELpbkwHEsaGnsU8DG2LHwbcFk6Reo4juM4qSikNy+qLAEOS9A+B6uSHm67ErgyEznBuYtoHpMaf/7oyBN2HMdxnAh4OkHHcRzHyRJXpo7jOI6TJa5MHcdxHCdLCurN254RkfXAymLPow1SBiRIONbp8efSEn8mienoz6W7qnZ4w62gDkjtnLdVdddiT6KtISJv+nNpiT+XlvgzSYw/l45Bh/+14DiO4zj5xpWp4ziO42SJK9Po3FDsCbRR/Lkkxp9LS/yZJMafSwfAHZAcx3EcJ0vcMnUcx3GcLHFl6jiO4zhZ4srUcRzHcbLElWkaRKSPiDwgIstFpE5Ejin2nNoCIlIjIqtEpCE4Piz2nAqNiJwmIm+KyGoRuSXu3IEi8oGIrBCRahHpNJUdkz0XEakSEQ19ZhpE5IIiTrVgiEi5iEwKvkOWicg7IvLt0PlO+3npKLgyTc81wBqgHzAWuE5Ehhd3Sm2G01S1V3AMLfZkisBc4BLgpnCjiPQF7gcuAPoAbwJ3F3x2xSPhcwmxSehzM7GA8yomZcD/gNHAxthn457gB0Zn/7x0CDwDUgpEpCdwBDBCVRuAF0XkYeAnYIXKnc6Lqt4PICK7AgNDpw4HalX13uD8BGCRiGyvqh8UfKIFJsVz6bSo6nJgQqjpURH5FNgF2IxO/HnpKLhlmpohQKOqzgy1TQPcMjX+KCKLROQlERlT7Mm0IYZjnxNgwxfpLPxzE6NORD4TkZsDq6zTISL9sO+XWvzz0iFwZZqaXsDSuLalwEZFmEtb47fAYGAAFnT+iIhsU9wptRn8c5OYRcBuQCVmkW0ETCnqjIqAiHTB7vvWwPL0z0sHwJVpahqA3nFtvYFlRZhLm0JVX1PVZaq6WlVvBV4CvlPsebUR/HOTAFVtUNU3VXWdqs4HTgO+JSLxz6rDIiIlwO2YH8ZpQbN/XjoArkxTMxMoE5HtQm07YUszTnMUkGJPoo1Qi31OgA1779vgn5t4YunXOsXnRkQEmIQ5Mx6hqmuDU/556QC4Mk1BsHdxP3CxiPQUkb2BQ7Fflp0WEdlERA4SkW4iUiYiY4H9gP8We26FJLj3bkApUBp7HsADwAgROSI4/3vg3c7iTJLsuYjIHiIyVERKRGQz4O9AjarGL3F2VK4DdgAOUdVwbeRO/XnpKLgyTc8pQHdgAXAnMF5VO/svxi5Y6MNCbB/sdOAwVe1ssabnYwXjzwWODV6fr6oLMS/wS4EvgD2Ao4o1ySKQ8Llge+xPYMuX04HVwNFFmmNBCeJGfwGMBOaF4mzH+uelY+CJ7h3HcRwnS9wydRzHcZwscWXqOI7jOFniytRxHMdxssSVqeM4juNkiStTx3Ecx8kSV6aO4ziOkyWuTB2nkxLUFj2y2PNwnI6AK1PHKQIickugzOKPV4s9N8dxMsfrmTpO8Xgaq40bZk0xJuI4Tna4Zeo4xWO1qs6LO5bAhiXY00TkMRFZISJ1InJseLCIfE1EnhaRlSKyJLB2N47rc5yIvCciq0VkvojcEjeHPiJyr4gsF5FP4q/hOE40XJk6TtvlIuBhLJ/rDcBtIrIrgIj0wPLcNgC7Az8A9gJuig0WkV8A/wRuBnbESuTF55X+PfAQVrXkbuCmII+s4zgZ4Ll5HacIBBbiscCquFPXqOpvRUSBf6nqSaExTwPzVPVYETkJuAIYqKrLgvNjgGpgO1X9WEQ+Ayar6rlJ5qDAn1T1d8H7MuArYJyqTs7d3TpOx8f3TB2neDwPjItr+zL0+pW4c68A3w1e74CV6QoXkH4ZWA8ME5GvgAHAM2nm8G7shaquE5GFwBaRZu84zgZcmTpO8Vihqh+3cqzQVFw7nkwKta+Ne6/49o/jZIz/p3GctsueCd6/H7yeAewkIhuFzu+F/Z9+X1XnA58DB+Z9lo7juGXqOEWkXES2jGtrDIpFAxwuIm8ANcCRmGLcIzg3BXNQuk1Efg9sijkb3R+ydi8F/ioi84HHgB7Agar6l3zdkON0VlyZOk7x+AZQH9f2OTAweD0BOAL4O7AQOEFV3wBQ1RUichDwN+B1zJHpIeCMmCBVvU5E1gC/Bi4DlgD/ydO9OE6nxr15HacNEnja/lBV7yv2XBzHSY/vmTqO4zhOlrgydRzHcZws8WVex3Ecx8kSt0wdx3EcJ0tcmTqO4zhOlrgydRzHcZwscWXqOI7jOFniytRxHMdxsuT/Aaqq0VYt2x00AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, history.history[\"lr\"], \"bo-\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Tốc độ học\", color='b')\n",
    "plt.tick_params('y', colors='b')\n",
    "plt.gca().set_xlim(0, n_epochs - 1)\n",
    "plt.grid(True)\n",
    "\n",
    "ax2 = plt.gca().twinx()\n",
    "ax2.plot(history.epoch, history.history[\"val_loss\"], \"r^-\")\n",
    "ax2.set_ylabel('Mất mát Kiểm định', color='r')\n",
    "ax2.tick_params('y', colors='r')\n",
    "\n",
    "plt.title(\"Giảm LR trên Vùng bằng phẳng\", fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tf.keras schedulers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.5995 - accuracy: 0.7923 - val_loss: 0.4095 - val_accuracy: 0.8606\n",
      "Epoch 2/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3890 - accuracy: 0.8613 - val_loss: 0.3738 - val_accuracy: 0.8692\n",
      "Epoch 3/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3530 - accuracy: 0.8772 - val_loss: 0.3735 - val_accuracy: 0.8692\n",
      "Epoch 4/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3296 - accuracy: 0.8813 - val_loss: 0.3494 - val_accuracy: 0.8798\n",
      "Epoch 5/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.3178 - accuracy: 0.8867 - val_loss: 0.3430 - val_accuracy: 0.8794\n",
      "Epoch 6/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2930 - accuracy: 0.8951 - val_loss: 0.3414 - val_accuracy: 0.8826\n",
      "Epoch 7/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2854 - accuracy: 0.8985 - val_loss: 0.3354 - val_accuracy: 0.8810\n",
      "Epoch 8/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2714 - accuracy: 0.9039 - val_loss: 0.3364 - val_accuracy: 0.8824\n",
      "Epoch 9/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2714 - accuracy: 0.9047 - val_loss: 0.3265 - val_accuracy: 0.8846\n",
      "Epoch 10/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2570 - accuracy: 0.9084 - val_loss: 0.3238 - val_accuracy: 0.8854\n",
      "Epoch 11/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2502 - accuracy: 0.9117 - val_loss: 0.3250 - val_accuracy: 0.8862\n",
      "Epoch 12/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2453 - accuracy: 0.9145 - val_loss: 0.3299 - val_accuracy: 0.8830\n",
      "Epoch 13/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2408 - accuracy: 0.9154 - val_loss: 0.3219 - val_accuracy: 0.8870\n",
      "Epoch 14/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2380 - accuracy: 0.9154 - val_loss: 0.3221 - val_accuracy: 0.8860\n",
      "Epoch 15/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2378 - accuracy: 0.9166 - val_loss: 0.3208 - val_accuracy: 0.8864\n",
      "Epoch 16/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2318 - accuracy: 0.9191 - val_loss: 0.3184 - val_accuracy: 0.8892\n",
      "Epoch 17/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2266 - accuracy: 0.9212 - val_loss: 0.3197 - val_accuracy: 0.8906\n",
      "Epoch 18/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2284 - accuracy: 0.9185 - val_loss: 0.3169 - val_accuracy: 0.8906\n",
      "Epoch 19/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2286 - accuracy: 0.9205 - val_loss: 0.3197 - val_accuracy: 0.8884\n",
      "Epoch 20/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2288 - accuracy: 0.9211 - val_loss: 0.3169 - val_accuracy: 0.8906\n",
      "Epoch 21/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2265 - accuracy: 0.9212 - val_loss: 0.3179 - val_accuracy: 0.8904\n",
      "Epoch 22/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2258 - accuracy: 0.9205 - val_loss: 0.3163 - val_accuracy: 0.8914\n",
      "Epoch 23/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2224 - accuracy: 0.9226 - val_loss: 0.3170 - val_accuracy: 0.8904\n",
      "Epoch 24/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2182 - accuracy: 0.9244 - val_loss: 0.3165 - val_accuracy: 0.8898\n",
      "Epoch 25/25\n",
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.2224 - accuracy: 0.9229 - val_loss: 0.3164 - val_accuracy: 0.8904\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\n",
    "learning_rate = keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1)\n",
    "optimizer = keras.optimizers.SGD(learning_rate)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Đối với định thời từng phần cố định, hãy thử:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = keras.optimizers.schedules.PiecewiseConstantDecay(\n",
    "    boundaries=[5. * n_steps_per_epoch, 15. * n_steps_per_epoch],\n",
    "    values=[0.01, 0.005, 0.001])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Định thời 1Cycle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = keras.backend\n",
    "\n",
    "class ExponentialLearningRate(keras.callbacks.Callback):\n",
    "    def __init__(self, factor):\n",
    "        self.factor = factor\n",
    "        self.rates = []\n",
    "        self.losses = []\n",
    "    def on_batch_end(self, batch, logs):\n",
    "        self.rates.append(K.get_value(self.model.optimizer.learning_rate))\n",
    "        self.losses.append(logs[\"loss\"])\n",
    "        K.set_value(self.model.optimizer.learning_rate, self.model.optimizer.learning_rate * self.factor)\n",
    "\n",
    "def find_learning_rate(model, X, y, epochs=1, batch_size=32, min_rate=10**-5, max_rate=10):\n",
    "    init_weights = model.get_weights()\n",
    "    iterations = math.ceil(len(X) / batch_size) * epochs\n",
    "    factor = np.exp(np.log(max_rate / min_rate) / iterations)\n",
    "    init_lr = K.get_value(model.optimizer.learning_rate)\n",
    "    K.set_value(model.optimizer.learning_rate, min_rate)\n",
    "    exp_lr = ExponentialLearningRate(factor)\n",
    "    history = model.fit(X, y, epochs=epochs, batch_size=batch_size,\n",
    "                        callbacks=[exp_lr])\n",
    "    K.set_value(model.optimizer.learning_rate, init_lr)\n",
    "    model.set_weights(init_weights)\n",
    "    return exp_lr.rates, exp_lr.losses\n",
    "\n",
    "def plot_lr_vs_loss(rates, losses):\n",
    "    plt.plot(rates, losses)\n",
    "    plt.gca().set_xscale('log')\n",
    "    plt.hlines(min(losses), min(rates), max(rates))\n",
    "    plt.axis([min(rates), max(rates), min(losses), (losses[0] + min(losses)) / 2])\n",
    "    plt.xlabel(\"Learning rate\")\n",
    "    plt.ylabel(\"Loss\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cảnh báo**: Trong phương thức `on_batch_end()`, `logs[\"loss\"]` được sử dụng để chứa mất mát batch, nhưng trong TensorFlow 2.2.0, nó được thay thế bằng mất mát trung bình (kể từ khi bắt đầu epoch). Điều này giải thích tại sao biểu đồ bên dưới mượt hơn nhiều so với trong sách (nếu bạn đang sử dụng TF 2.2 trở lên). \n",
    "\n",
    "Điều đó cũng có nghĩa là có độ trễ giữa thời điểm mất batch bắt đầu bùng nổ và thời điểm bùng nổ trở nên rõ ràng trên biểu đồ. Vì vậy, bạn nên chọn tốc độ học nhỏ hơn một chút so với mức bạn đã chọn với biểu đồ \"nhiễu\". Ngoài ra, bạn có thể điều chỉnh lệnh gọi lại `ExponentialLearningRate` ở trên để tính toán mất mát batch (dựa trên mất mát trung bình hiện tại và mất mát trung bình trước đó):\n",
    "\n",
    "\n",
    "```python\n",
    "class ExponentialLearningRate(keras.callbacks.Callback):\n",
    "    def __init__(self, factor):\n",
    "        self.factor = factor\n",
    "        self.rates = []\n",
    "        self.losses = []\n",
    "    def on_epoch_begin(self, epoch, logs=None):\n",
    "        self.prev_loss = 0\n",
    "    def on_batch_end(self, batch, logs=None):\n",
    "        batch_loss = logs[\"loss\"] * (batch + 1) - self.prev_loss * batch\n",
    "        self.prev_loss = logs[\"loss\"]\n",
    "        self.rates.append(K.get_value(self.model.optimizer.learning_rate))\n",
    "        self.losses.append(batch_loss)\n",
    "        K.set_value(self.model.optimizer.learning_rate, self.model.optimizer.learning_rate * self.factor)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "430/430 [==============================] - 1s 2ms/step - loss: nan - accuracy: 0.3120\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAERCAYAAABcuFHLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAl3klEQVR4nO3deXhc9X3v8fdXiyVblndZEvIGBoM3MLYJEGMgCWAwISExhECBQLhA25Dk5rZPQ56QxmlI2iRNem/a3hZyoQYCFCgECDuBgKGAwcYGL9gE432Vd1u2tc33/jEjI4S2kc7M+Wn0eT3PeTxz5szo+/OR5+vfbu6OiIhId+XFHYCIiOQGJRQREYmEEoqIiERCCUVERCKhhCIiIpFQQhERkUgUxB1AlIYNG+ZjxoyJOwzJMSu27KO0uICRg/vFHYq0UN/orNy6j6pBfRlS0ifucHqsRYsW7XD3su5+Tk4llDFjxrBw4cK4w5Ac850HlvDy+9Us+P455OdZ3OFIM9v2HebUn77ArV+azBWnjoo7nB7LzNZF8Tlq8hLpwGdPGM6umjqWbNgddygiQVNCEenAWceXUZhvPLNsa9yhiARNCUWkAwOKC5l5XBlPLd2KlioKi25HWJRQRDrhgkkVbNpziGWb9sUdirTC1LUVBCUUkU74zAnDAXhp1faYIxEJlxKKSCcM61/ESSMG8qISikiblFBEOumzJ5SzZMMetu49HHcokuKoEyUkWUsoZlZkZneY2Toz229mi83sgjaunWRmz5rZDjPTb4wE4cITK3GHp5ZuiTsUaUFdKGHIZg2lANgAnAUMBH4APGhmY1q5th54ELgua9GJdODY4f05oaKUJ5VQRFqVtYTi7jXuPtfd17p7wt2fANYA01q5dpW73wEsz1Z8Ip1x0UlHsWjdbjbvORR3KCLBia0PxczKgXEoaUgPcuHkSkDNXqHQPJSwxJJQzKwQuBe4y91XdvOzbjCzhWa2sLq6OpoARdowZlgJ4ysH8OxyzZoPieahhCHrCcXM8oB7gDrgpu5+nrvf7u7T3X16WVm3F8sU6dB5E8pZuG43Ow7Uxh2KSFCymlDMzIA7gHJgjrvXZ/Pni0Rh1sQK3OEPK7bFHYpIULJdQ/k3YDxwkbu32atpScVAn9TzYjMrylKMIu0aX1nKiMF9eU4JJXbqQglLNuehjAZuBKYAW83sQOr4MzMblXrctKHBaOAQH3XYHwJWZStWkfaYGbMmVvDqn3ZwoLYh7nAEMM1ECUI2hw2vc3dz92J379/suNfd16cer09duzZ1bfNjTLZiFenIeRPKqWtM8PIqDQQRaaKlV0S6YPqYIQwp6aPRXiLNKKGIdEF+nnHO+OH8ceV26hoScYfTa2l/mrAooYh00ayJFeyvbeD1D3fGHYqoCyUISigiXTTj2GH065PPc2r2EgGUUES6rLgwn7OPL+P5FdtIJNT0IqKEItIN502oYPv+WpZs3BN3KL2SulDCooQi0g2fOWE4BXmm0V4xUxdKGJRQRLphYN9CTh87lOeWb9OII+n1lFBEuum8iRWs2VHDB9sPxB2KSKyUUES66dzx5QBa20t6PSUUkW6qGFjMlJGD1I8SI9OGKEFQQhGJwHkTy3l3415tDSy9mhKKSARmTawA4Hk1e0kvpoQiEoGxZf0ZW1aiZq8s08C6sCihiETkvIkVLFiziz0H6+IOpddRD0oYlFBEIjJrYgWNCeeF97bHHYpILJRQRCJyYtVAKgcW8/QyNXtJ76SEIhKRvDzj/EkVzP9TNfsP18cdjkjWKaGIRGj25ErqGhK8uFLNXtngqFc+JEooIhGaNmoww0uLeGrplrhD6RWadg3Iz1O3fAiUUEQilJdnXDCpgpdWVVNT2xB3ODkvkRo3rInyYVBCEYnY7MmV1KrZKyv8SEJRRgmBEopIxKaPGcKw/kU8vUzNXpnWNLFRLV5hUEIRiVh+nnH+pHJeXLmdg3Vq9sqkxJGEoowSAiUUkQyYPbmSw/UJXlpVHXcoOa2pD0U1lDAooYhkwKfGDGFoSR+N9sqwhPpQgqKEIpIBBfl5zJpUwYsrt3O4vjHucHJWUx+K0kkYlFBEMmT2pEoO1jWq2SuDXH0oQVFCEcmQ044ZwuB+hWr2yqAjfSj6JguCboNIhhTk5zFrYgUvvLdNzV4Zoj6UsCihiGTQBZMrqalr5JU/7Yg7lJyUUB9KULKWUMysyMzuMLN1ZrbfzBab2QXtXP8dM9tqZnvN7E4zK8pWrCJR+fTYoQzsq2avzGkaNqyUEoJs1lAKgA3AWcBA4AfAg2Y2puWFZjYLuBn4HDAGOAb4UbYCFYlKYX4e500o5w8rtlHboGavqGliY1iyllDcvcbd57r7WndPuPsTwBpgWiuXfw24w92Xu/tu4MfANdmKVSRKs0+sZH9tA6+q2StyiYQmNoYktj4UMysHxgHLW3l5IvBOs+fvAOVmNjQbsYlEacbYYZQWF/DUUu3kGLWmGoo6UcIQS0Ixs0LgXuAud1/ZyiX9gb3Nnjc9Lm3ls24ws4VmtrC6WuP9JTx9CvI4d0I5z6/YSl1DIu5wcoqrDyUoWU8oZpYH3APUATe1cdkBYECz502P97e80N1vd/fp7j69rKws0lhFonLh5Er2HW7gv1er2StKmtgYlqwmFEsOFr8DKAfmuHtbG28vB05q9vwkYJu778xwiCIZccZxwygtKuBpjfaKlBaHDEu2ayj/BowHLnL3Q+1cdzdwnZlNMLPBwC3AvCzEJ5IRRQX5nDOhnOdWbKO+Uc1eUTkyD0U1lCBkcx7KaOBGYAqw1cwOpI4/M7NRqcejANz9GeDnwB+Bdanjh9mKVSQTLphUwZ6D9by+WhXtqGgL4LAUZOsHufs62h+L0b/F9b8CfpXRoESy6MxxZZT0yefpZVs4c5z6+yKhPpSgaOkVkSwpLsznc+PLeXb5NhrU7BUJ9aGERQlFJItmT65gV00dC9bsijuUnKCZ8mFRQhHJorOPH06/Pvk8qdFekWiqoUgYlFBEsqi4MJ/PnDCcZ5dtpTGhL8Pu0jyUsCihiGTZhZMr2VlTx5tq9uo21wZbQdFtEMmys48vo7gwjyeXbo47lB5PfShhUUIRybJ+fQo4Z3w5T767RZMcu+nIPJSY45AkJRSRGHzp5Cp2H6xn/vta0LQ7jiw2rBpKEJRQRGJw5rgyBvcr5HeLN8UdSo/mmocSFCUUkRgU5udx0UlH8fyKbew/3NYaqdKRjyY2KqOEQAlFJCYXn1xFbUOCZ5Zp462uSqS6oJRQwqCEIhKTk0cOYvTQfjy6RM1eXaXFIcOihCISEzPj4ilVvLZ6J9v2HY47nB7po075WMOQFCUUkRhdfHIV7vD4Es1J6QpXH0pQlFBEYnT0sBJOGjmIx95Rs1dXaGJjWJRQRGJ24eQKlm3ax4ZdB+MOpcdRH0pYlFBEYjZrYgUAz63YFnMkPY8f2QI43jgkSQlFJGajh5ZwQkUpz2r4cNrUhxIWJRSRAJw/qYK31u2ien9t3KH0KOpDCYsSikgALphUiTs8vUwbb6VDWwCHRQlFJADHV5QyvnIAD7+t0V7paKqhmNYbDoISikgg5kyt4p0Ne/hg+4G4Q+kxmvpQTN9kQdBtEAnEF6YcRX6e8cjbG+MOpcfQFsBhUUIRCcTw0mLOPG4Yv1u8SfvNd5L6UMKihCISkC9PHcGWvYd5ffXOuEPpEdSHEhYlFJGAnDuhnNLiAh5Ws1enOJopHxIlFJGAFBfm8/kTj+KZZVs5UNsQdzjBUx9KWJRQRAJzybQqDtU38tRSzUnpSCKhPpSQKKGIBGbqqMEcPayEhxep2asjR/pQVEMJghKKSGDMjDlTq1iwZpdWIO5AUx+KaihhUEIRCdCXpo7ADB7RzPl2qYYSlqwmFDO7ycwWmlmtmc1r57oiM/snM9tsZrvN7P+aWWEWQxWJVdWgvpx+zFAefnvjkdng8knurtpJQLJdQ9kM3Arc2cF1NwPTgUnAOGAqcEtmQxMJy5ypI1i/6yBvrd0ddyjBSrhrhFdAsppQ3P0Rd38U6GjW1kXAr919l7tXA78Gvp7p+ERCcsHkCkr65Ktzvh0J1xyUkITah2Kpo/nzEWY2MKZ4RLKuX58CLphcyZNLt3CorjHucILkrv6TkISaUJ4Gvm1mZWZWAXwrdb5fywvN7IZUv8zC6urqrAYpkmlzpo7gQG0Dzy7Xbo6tUR9KWLqdUDLUWf4TYDGwBHgNeBSoB7a3vNDdb3f36e4+vaysLAOhiMTn1KOHUDWor5ZiaYP6UMKSVkIxs2+Z2Zxmz+8ADpnZKjM7Pqqg3P2Qu9/k7lXufgzJPpdF7q56v/QqeXnGnGkjePWDHWzZeyjucIKTcLQsZEDSraF8C6gGMLMzga8AV5CsSfyyozebWYGZFQP5QL6ZFZtZQSvXVZnZUZZ0GvAD4IdpxiqSE+ZMrcJdc1Ja4651vEKSbkKpAtamHl8EPOTuDwJzgdM68f5bgEMkhwVfmXp8i5mNMrMDZjYqdd1Ykk1dNcBdwM3u/lyasYrkhNFDSzhlzGDNSWlFwl2jvAKSbkLZBzR1VJwLvJB6XA8Ud/Rmd5/r7tbimOvu6929v7uvT103393HuHs/dz/e3e9NM06RnHLJtBF8WF3Dkg174g4lKO5Onnrlg5FuQnkO+E2q7+RYkqOxACYCa6IMTEQ+MntyJcWFeeqcb0F9KGFJN6F8A/hvYBhwibvvSp2fCtwfZWAi8pHS4kJmTazg8SWbOVyvsSlNNMorLJ/oEG+Pu+8DvtnKeXWYi2TYnKkjeGzJZl54bzsXnlgZdzhBcDSxMSTpDhue0Hx4sJmda2a/NbPvmVl+9OGJSJMZxw6jYkCxmr2a0cTGsKTb5HUHcDKAmY0AHgOGkGwKuzXa0ESkufw840tTq3j5/Wq27z8cdzhBSCQ0bDgk6SaU8cDbqceXAgvcfTZwFXB5lIGJyCfNmTqCxoTz2OLNcYcSBA0bDku6CSUfqEs9/hzwVOrxaqA8qqBEpHXHDu/PSSMHaU5KiqMaSkjSTSjLgL8ws5kkE8ozqfNVwI4oAxOR1l0ybQQrt+5n+eZ9cYcSO9VQwpJuQvkucD3wEnC/uy9Nnf8C8GaEcYlIGy46sZI++ZqTAlp6JTRpJRR3n09ypvwwd2++4dVtwF9EGZiItG5Qvz6cM2E4jy3ZTF1DIu5wYqUaSljSXr4+teLvITObZGYTzazY3de6+yeWlheRzLh0+kh21dTx3IrevU+KaihhSXceSoGZ/QLYDbwDLAV2m9nPM7Qvioi04szjyqga1Jf7FqyPO5RYqYYSlnRrKD8nuUrwnwPjgONINnVdBfx9tKGJSFvy84wrTh3Fa6t38mH1gbjDiY1qKGFJN6FcAVzn7ne5++rUMQ/4H8CfRR6diLTp0ukjKMgz7n+z99ZSEpopH5R0E8pAknNOWloNDOp2NCLSacNLizl3Qjn/tWhjr10wMuGOab3hYKSbUN4huWtjS99OvSYiWXTFqaPYfbCeZ5f3zs55d9SHEpC0VhsG/gZ4yszOBV4nOVH1dOAo4IKIYxORDswYO4zRQ/tx7xvr+eKUqrjDybqE+lCC0pV5KOOAh4D+wIDU41m0XnMRkQzKyzMu/9Qo3ly7i/e37Y87nKxL7tgYdxTSpCvzUDa7+/fdfY67f9ndbyG59/uc6MMTkY5cOm0EffLzuPeNdXGHknXqQwmLcrtIDze0fxGzJ1fwyNubqKltiDucrEouDhl3FNJECUUkB1x1+mj21zbw2JLetax9wrVjY0iUUERywNRRgzmhopTfvrGuVy1rrx0bw9KpUV5m9ngHlwyIIBYR6SIz46rTR/P93y3j7fV7mDZ6cNwhZUVy6RVllFB0toays4NjDXB3JgIUkc65eEoVJX3ye9XM+eQWwHFHIU06VUNx92szHYiIdE9JUQEXnXQUjy3ZzNwvTKR/UbrTzHoeRzWUkKgPRSSHfOWUkRyqb+TJd3tH53xyYmPcUUgTJRSRHHLyyEEcO7w/D7y1Ie5QsiLZKa+MEgolFJEcYmZcNn0kb6/fw8qtub/nfEJreQVFCUUkx1wybQRFBXnc9Vruz5xXDSUsSigiOWZwSR8unlLFo4s3sfdgfdzhZJQmNoZFCUUkB1396dEcqm/koUW53ZeiiY1hyWpCMbObzGyhmdWa2bx2rjMzu9XMNpnZXjN7ycwmZjFUkR5t4lEDmT56ML99Yx2JRO7OnE84WhoyINmuoWwGbgXu7OC6S4GvAzOBIST3Xrkns6GJ5JarTh/N2p0HeeWDHXGHkjGO+lBCktWE4u6PuPujJGfXt+do4FV3/9DdG4HfAhMyHZ9ILjl/UgXD+vfhntdzt3M+kVAfSkhC7UP5T+BYMxtnZoXA14BnYo5JpEcpKsjnslNG8uLKbWzcfTDucDIioT6UoISaULYArwCrgEMkm8C+09qFZnZDql9mYXV1dRZDFAnfFaeOBuC+Bbm5vpdrC+CghJpQfgicAowEioEfAS+aWb+WF7r77e4+3d2nl5WVZTlMkbBVDerL58aX88BbG6htaIw7nMglVxuOOwppEmpCOQl4wN03unuDu88DBqN+FJG0XXXaaHbW1PHMsq1xhxK55I6NyiihyPaw4QIzKwbygXwzKzaz1pZEfQu41MzKzSzPzK4CCoEPshmvSC4449hhHD2shHmvrY07lMiphhKWbNdQbiHZJ3IzcGXq8S1mNsrMDpjZqNR1PwPeAZYAe0j2n8xx9z1Zjlekx8vLM752+mgWr9/D4vW74w4nUupDCUu2hw3PdXdrccx19/Xu3t/d16euO+zu33D3Sncf4O5T3V2jvES66NLpIyktLuCOV9fEHUqkVEMJS6h9KCISoZKiAr56ykieXraVTXsOxR1OZFRDCYsSikgvcc2MozHgN/M/jDuUyKiGEhYlFJFeompQX748tYr731zPzgO1cYcTCdVQwqKEItKLXD/zGGobEjy0aGPcoUQi4a7FIQOihCLSixxXXsppxwzh3gXraMyBVYhVQwmLEopIL3PVaWPYsOsQ89/v+UsVJdzJ07dYMHQrRHqZ8yaWU1ZaxD1v9PxViLVjY1iUUER6mcL8PC4/ZSR/XLWdDbt69irErj6UoCihiPRCl586ijwz7u3hqxA3ulOg9euDoYQi0gtVDuzLOeOH8+DCDRyu77mrEDc0OvnqRAmG7oRIL3X16WPYVVPH40s2xx1Kl9U3JijMVw0lFEooIr3Up8cOZULlAG6bv5pEDx1C3JhwCpRQgqGEItJLmRk3nnUMq6treHHl9rjDSZu705BQk1dIdCdEerHZkyupGtSX2+avjjuUtDWkalWF6pQPhhKKSC9WmJ/HdWcczVtrd7NoXc/aK6WhMZlQCvL1NRYK3QmRXu6yU0YysG8ht/ewWkpDIgGgYcMBUUIR6eVKigq46rTRPLdiGx9WH4g7nE77qIaihBIKJRQR4WufHkNhfh6/eaXn7OhY31RDUZNXMHQnRISy0iLmTB3Bw29vpHp/z9grpWm1ZDV5hUMJRUQAuH7m0dQ3JrjrtbVxh9IpR5q8lFCCoYQiIgAcU9af8yaUc88b66ipbYg7nA7VNyabvArV5BUM3QkROeLGs8ay91A9D7y1Ie5QOnSkyUud8sFQQhGRI6aOGswpYwZzx6trjtQAQlWvJq/gKKGIyMfceOZYNu05xFNLt8QdSrs+moeir7FQ6E6IyMd89oThjC0r4baXP8Q93EUjG9TkFRwlFBH5mLw848Yzx7Jiyz5e/WBH3OG06aNRXvoaC4XuhIh8whdPPorhpUXc9vKHcYfSpobGpomNqqGEQglFRD6hqCCfa2aM4dUPdrBy6764w2nVkdWGlVCCoYQiIq264lOjKC7M4z9eXRt3KK1q6pTXfijh0J0QkVYN6teHOVNH8Lslm9h5ILzlWDRsODxKKCLSpmtnjKGuIcF9C9bHHconNHXKa6Z8OLJ6J8zsJjNbaGa1Zjavnev+3cwONDtqzWx/FkMVEeDY4aWcNa6Mu99YR11DWBMdP2ryUg0lFNlO7ZuBW4E727vI3f/c3fs3HcD9wEPZCFBEPu7aGWOo3l/LI29vjDuUj/mohqKEEoqsJhR3f8TdHwV2dvY9ZlYCzAHuylRcItK2s8aVcfKoQfzq+feDWjSyQfuhBKcn3Ik5QDUwP+5ARHojM+OWC8ezfX8tt88PZ15Kg/ZDCU5PSChfA+72NtaAMLMbUv0yC6urq7McmkjvMG30EC6cXMnt8z9k277DcYcDaD+UEAWdUMxsJHAWcHdb17j77e4+3d2nl5WVZS84kV7mu+efQGPC+fkzq+IOBfhoPxQ1eYUj9DtxNfCau4dTzxbppUYN7cd1M4/m4bc3smjdrrjD0RbAAcr2sOECMysG8oF8Mys2s4J23nI1MC8rwYlIh7752WM5amAxP3h0+ZG1tOKi1YbDk+0ayi3AIeBm4MrU41vMbFRqvsmopgvN7HRgBBouLBKMfn0K+MHnJ7Biyz5++8a6WGM5sgWwll4JRraHDc91d2txzHX39ak5J+ubXfu6u5e4uyY0igTk/EkVzDxuGL987n2q98e3JEtjwsmz5HL7EgaldhFJi5nxoy9M5HBDI3//9HuxxVHf6NoLJTC6GyKStmPK+nPdGcfwu8WbWLE5nuXtGxoT6j8JjBKKiHTJX5w9lgHFhfz82ZWx/PyGhGuEV2CUUESkSwb2LeQvzx7LS6uqeeG9bVn/+Q2JhOagBEZ3Q0S67GufHsP4ygF88/7FLN24N6s/u6FRNZTQKKGISJcVF+Yz79pTGNyvD9fOe4vtWVyWpb7RtRdKYHQ3RKRbygcU8x/XnkJNbQNX3/kmm/YcysrPbUwktBdKYJRQRKTbxpWXcvvV09i0+xB/+dtFRyYdZlJ9wjXKKzBKKCISiZnHlfGzS07knY17+fULf8r4z2toTGiWfGB0N0QkMrMnVzJn6gj+5Y8f8OS7WzL6sxoaXU1egVFCEZFI3XrxJKaNGsxfP/QO63bWZOznHKxrpKQoP2OfL+lTQhGRSPXtk88/X3EyBfnGJf/+Os8t35qRn3OwroF+fdpbrFyyTQlFRCJXObAv/3nDaZT1L+KGexbxzLLom78O1DbQv0gJJSRKKCKSEROPGsij35jB5KqBfO+RpeyuqYv08w/WNdKvj5q8QqKEIiIZ06cgj19ceiL7Djfwy+ej3Tr4QG0DJaqhBEUJRUQy6oSKAVx12mjuW7Ce5ZujWZ7F3dUpHyAlFBHJuO+cM45B/fow9/HluHu3P6+2IUFjwlVDCYwSiohk3MB+hfzNrON5a+1u7l2wvuM3dKCmtgGAEo3yCooSiohkxVemj+TMcWX8+IkVrNzavU25amobAVRDCYwSiohkRV6e8ctLT2JA30K+ff8SGrqx3ldNXVMNRX0oIVFCEZGsKSst4sdfnMSqbfu71fTV1OTVTzWUoCihiEhWzZpYzoxjh/Kr59/v8tyUmrpkk1d/jfIKihKKiGSVmfG3n59ITW0D19+9kAOp2kY6jtRQ1CkfFCUUEcm64ytK+fXlJ7Nkwx6uufNN6hrS609pSihaeiUsSigiEovZkyv5p8umsHDdbn7x7Mq03nsw1eSlpVfCooQiIrG56KSjuOLUUfy/V9ewZMOeTr9v76F6QMOGQ6OEIiKxuvmCExheWsT3Hlna6a2DV23dz4jBfSkuVA0lJEooIhKrAcWF/OgLk3hvyz7+4emVJBIdL83y7qY9nDhiYBaik3QooYhI7M6fVMHVp4/mjlfXcN7/ns/zK7a1ee2eg3Vs2HWIyVWDshegdIoSiogEYe5FE/k/X51CnsH1dy/kp0+9x56Dn5yn8trqnQCcpBpKcJRQRCQIeXnGF6dU8ftvnsGVp43i9vkfcvY/vsTaHR/fl37ef69lxOC+nHrM0JgilbYooYhIUIoK8rn14sk8+o0ZAFz+mzdY8GGyVrJh10HeXLuLK08bTX6exRmmtEIJRUSCNGXkIO75+qn0Lczn2nlvsXj9bl5atR2AcyeUxxydtCanBnF/WF3DZbe9HncYIhKhgX0L2bTnEJf8++u4O0UFeXzv4XcxUw0lNBbF7mmhMLP9QHc3rh4IdGaf0vaua+21js61fL3pefPzw4AdnYitPdkqX3vP23qcrfKlW7bWzsdRvkzdu9bOp1u+nvS72dq5XC5fZ75bjnf30k7E1j53z5kDWBjBZ9ze3etae62jcy1fb3re4poeU772nrfzOCvlS7dsoZQvU/cuivL1pN/N3la+bH23uLv6UFrx+wiua+21js61fP33bZzvrmyVr73n7ZW7uzrzeemWrbXzcZQvU/eutfO5VL50f19zrXzZ+m7JuSavhe4+Pe44MkXl69lyuXy5XDZQ+Tor12oot8cdQIapfD1bLpcvl8sGKl+n5FQNRURE4pNrNRQREYmJEoqIiESi1yUUMxtjZtVm9lLqKIs7pqiZ2eVmVh13HFEzs3Ize83MXjazF82sMu6YomRmp5vZ66ny3W9mhXHHFCUzG2hmb5rZATObFHc8UTCzn5jZK2b2X2bWL+54otSV+9XrEkrKy+5+durIqS9eM8sDLgE2xB1LBuwAznD3s4C7getijidq64DPpsr3IfDFmOOJ2kHgQuC/4g4kCqkv2bHuPhP4A/D1mEOKWtr3q7cmlBmp/1X81HJv/YYrSP4CdG7rux7E3RvdvalcpcDyOOOJmrtvdvdDqacN5Ng9dPf6HPsP3Ezg6dTjp4EzYowlcl25X0EnFDO7ycwWmlmtmc1r8doQM/udmdWY2Tozu6KTH7sFOBY4ExgOfDnaqDsnE2Uzs3zgK8ADGQg5LRm6d5jZFDNbANwEvB1x2J2WqfKl3n80cAHwRIQhpyWT5QtNN8o6mI+WLtkLDMlSyGnJ5r0MfXHIzcCtwCygb4vX/hWoA8qBKcCTZvaOuy83swpar6Zd4u5bgVoAM3sEOA14ODPhtyvysqU+60F3TwRQ8crIvXP3JcCpZvYV4HvAn2co/o5kpHxmNgC4C7jK3T+5u1T2ZOrfXoi6VFZgN8n1sEj9uSsr0aavq+VLXxTrt2T6SP1lzGv2vCT1lzCu2bl7gH/oxGcNaPb474Grc6hsPwOeA54h+T+mX+fYvStq9ngW8KscK18B8CTJfpRYy5WJ8jW7fh4wKe6ydbeswGTgvtTjG4Bvxl2GTNzLdO5X0E1e7RgHNLr7+83OvQNM7MR7zzKzRWb2ClAF3JeJALuhy2Vz9++6+3nufj7wJ3f/VqaC7Ibu3LupZjbfzP4I/E/gFxmIr7u6U77LgVOBv02NQLwsEwF2U3fKh5k9BZwH/MbMrok+vEi1W1Z3XwqsS32XzALuzH6I3dLhvUz3foXe5NWW/nxyuea9JDtq2+XuvycDi6JFqMtla87DXXeoO/fudZJ9XyHrTvnuIfk/xJB16/fT3WdHHlHmdFhWd/9eViOKVmfKl9b96qk1lAPAgBbnBgD7Y4glarlcNlD5erpcL19zuV7WyMvXUxPK+0CBmR3X7NxJ5MYw0lwuG6h8PV2ul6+5XC9r5OULOqGYWYGZFQP5QL6ZFZtZgbvXAI8Af2dmJWY2g+QksNCbC47I5bKByofK12PkelmzWr64Rx50MCphLuAtjrmp14YAjwI1wHrgirjjVdlUPpWv5x25XtZslk/L14uISCSCbvISEZGeQwlFREQioYQiIiKRUEIREZFIKKGIiEgklFBERCQSSigiIhIJJRSRCJnZXDNbFnccInHQxEbpcVK7zg1z98/HHUtLZtaf5L4tO+OOpS1m5sCl7p4Te7tLOFRDEekEM+vTmevc/UAcycTM8iy5BbRIbJRQJOeY2QQze9LM9pvZdjO7P7U1bdPrp5jZc2a2w8z2mdmrZnZ6i89wM/uGmT1iZjXAT5uas8zsq2a2OvX5j5rZsGbv+1iTl5nNM7MnzOzbZrbJzHab2X+YWb9m15SY2d1mdsDMtpnZ91LvmddOGa9JXT879fPqgPEdlc3M1qYePpQq49pmr12U2nzusJmtMbOfdDaRioASiuQYM6sE5gPLgE8B55DcSOhxM2v6fS8luaLqzNQ1S4CnmieGlB8CT5Hc6vVfU+fGAJcBXyK5k93JwE86CGsmMCkVS9N7v93s9V8CZ6XOf5bkEuIzO1HcYuAW4EZgArCuE2U7JfXn9UBl03MzmwXcC/wLyR37vg5cAvy0E3GIJMW9EqYOHekeJPe4fqKN1/4OeKHFucEkV1j9VBvvMWALcGWzcw78c4vr5gKHgYHNzn0f+KDFNctaxLoBKGh27jfAH1KP+5OsXXy12eslwG6a7f/dSszXpGKc1sHfVVtlu6TFdfOBH7Q4dzHJTZgs7nuuo2ccqqFIrpkGnJlqDjpgZgdIfqEDjAUws+FmdpuZvW9me0nuUDccGNXisxa28vnr3L35tqmbU+9tzwp3b2jjPWOBQuDNphc9uU9FZ0aKNZCsgRyRRtlamgZ8v8Xf230kk1tF+28VSeqpe8qLtCUPeBL461Ze25b68y6gHPgOsBaoBV4AWvYX1LTyGfUtnjsdNx239x5rdi5dte7e2OJcZ8vWUh7wI+ChVl6r7kJs0gspoUiueRv4CsmaRMsv8iZnAN9y9ycBzKycZH9CHD4gmXA+BaxJxdOPZJ/L6i58XmfKVk9y977m3gZOcPcPuvAzRQAlFOm5BpjZlBbn9pDsPL8eeMDMfkbyf9fHkEwyf+Xu+0nupX2lmS0g2aTzc5L9GFnn7gfM7E7gZ2a2g2R/xy0kawxdqbV0pmxrgc+Z2cskazm7SfY9PWFm64AHSTanTSLZ7/Q3XYhDeiH1oUhPNRNY3OL4R3ffDMwAEsAzwHKSSaY2dUByBFN/YBHwn8CdJL9k4/LXwCvA48AfgXdJ9t8c7sJndaZsfwV8hmTf0mIAd38WuDB1/s3UcTPJbWFFOkUz5UUCY2ZFJIcA/8Ldfxl3PCKdpSYvkZiZ2cnAeJK1glLgu6k/H4gzLpF0KaGIhOF/Acfz0VDgM919Y6wRiaRJTV4iIhIJdcqLiEgklFBERCQSSigiIhIJJRQREYmEEoqIiERCCUVERCLx/wGKUUuRqdIcAQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "rates, losses = find_learning_rate(model, X_train_scaled, y_train, epochs=1, batch_size=batch_size)\n",
    "plot_lr_vs_loss(rates, losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "class OneCycleScheduler(keras.callbacks.Callback):\n",
    "    def __init__(self, iterations, max_rate, start_rate=None,\n",
    "                 last_iterations=None, last_rate=None):\n",
    "        self.iterations = iterations\n",
    "        self.max_rate = max_rate\n",
    "        self.start_rate = start_rate or max_rate / 10\n",
    "        self.last_iterations = last_iterations or iterations // 10 + 1\n",
    "        self.half_iteration = (iterations - self.last_iterations) // 2\n",
    "        self.last_rate = last_rate or self.start_rate / 1000\n",
    "        self.iteration = 0\n",
    "    def _interpolate(self, iter1, iter2, rate1, rate2):\n",
    "        return ((rate2 - rate1) * (self.iteration - iter1)\n",
    "                / (iter2 - iter1) + rate1)\n",
    "    def on_batch_begin(self, batch, logs):\n",
    "        if self.iteration < self.half_iteration:\n",
    "            rate = self._interpolate(0, self.half_iteration, self.start_rate, self.max_rate)\n",
    "        elif self.iteration < 2 * self.half_iteration:\n",
    "            rate = self._interpolate(self.half_iteration, 2 * self.half_iteration,\n",
    "                                     self.max_rate, self.start_rate)\n",
    "        else:\n",
    "            rate = self._interpolate(2 * self.half_iteration, self.iterations,\n",
    "                                     self.start_rate, self.last_rate)\n",
    "        self.iteration += 1\n",
    "        K.set_value(self.model.optimizer.learning_rate, rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.6572 - accuracy: 0.7740 - val_loss: 0.4872 - val_accuracy: 0.8338\n",
      "Epoch 2/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.4580 - accuracy: 0.8397 - val_loss: 0.4274 - val_accuracy: 0.8520\n",
      "Epoch 3/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.4121 - accuracy: 0.8545 - val_loss: 0.4116 - val_accuracy: 0.8588\n",
      "Epoch 4/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3837 - accuracy: 0.8642 - val_loss: 0.3868 - val_accuracy: 0.8688\n",
      "Epoch 5/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3639 - accuracy: 0.8719 - val_loss: 0.3766 - val_accuracy: 0.8688\n",
      "Epoch 6/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3456 - accuracy: 0.8775 - val_loss: 0.3739 - val_accuracy: 0.8706\n",
      "Epoch 7/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3330 - accuracy: 0.8811 - val_loss: 0.3635 - val_accuracy: 0.8708\n",
      "Epoch 8/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3184 - accuracy: 0.8861 - val_loss: 0.3959 - val_accuracy: 0.8610\n",
      "Epoch 9/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.3065 - accuracy: 0.8890 - val_loss: 0.3475 - val_accuracy: 0.8770\n",
      "Epoch 10/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2943 - accuracy: 0.8927 - val_loss: 0.3392 - val_accuracy: 0.8806\n",
      "Epoch 11/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2838 - accuracy: 0.8963 - val_loss: 0.3467 - val_accuracy: 0.8800\n",
      "Epoch 12/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2707 - accuracy: 0.9024 - val_loss: 0.3646 - val_accuracy: 0.8696\n",
      "Epoch 13/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2536 - accuracy: 0.9079 - val_loss: 0.3350 - val_accuracy: 0.8842\n",
      "Epoch 14/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2405 - accuracy: 0.9135 - val_loss: 0.3465 - val_accuracy: 0.8794\n",
      "Epoch 15/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2279 - accuracy: 0.9185 - val_loss: 0.3257 - val_accuracy: 0.8830\n",
      "Epoch 16/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2159 - accuracy: 0.9232 - val_loss: 0.3294 - val_accuracy: 0.8824\n",
      "Epoch 17/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.2062 - accuracy: 0.9263 - val_loss: 0.3333 - val_accuracy: 0.8882\n",
      "Epoch 18/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1978 - accuracy: 0.9301 - val_loss: 0.3235 - val_accuracy: 0.8898\n",
      "Epoch 19/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1892 - accuracy: 0.9337 - val_loss: 0.3233 - val_accuracy: 0.8906\n",
      "Epoch 20/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1821 - accuracy: 0.9365 - val_loss: 0.3224 - val_accuracy: 0.8928\n",
      "Epoch 21/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1752 - accuracy: 0.9400 - val_loss: 0.3220 - val_accuracy: 0.8908\n",
      "Epoch 22/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1700 - accuracy: 0.9416 - val_loss: 0.3180 - val_accuracy: 0.8962\n",
      "Epoch 23/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1655 - accuracy: 0.9438 - val_loss: 0.3187 - val_accuracy: 0.8940\n",
      "Epoch 24/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1627 - accuracy: 0.9454 - val_loss: 0.3177 - val_accuracy: 0.8932\n",
      "Epoch 25/25\n",
      "430/430 [==============================] - 1s 2ms/step - loss: 0.1610 - accuracy: 0.9462 - val_loss: 0.3170 - val_accuracy: 0.8934\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 25\n",
    "onecycle = OneCycleScheduler(math.ceil(len(X_train) / batch_size) * n_epochs, max_rate=0.05)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs, batch_size=batch_size,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[onecycle])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tránh Quá khớp thông qua Điều chuẩn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Điều chuẩn $\\ell_1$ và $\\ell_2$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(100, activation=\"elu\",\n",
    "                           kernel_initializer=\"he_normal\",\n",
    "                           kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "# or l1(0.1) for ℓ1 regularization with a factor of 0.1\n",
    "# or l1_l2(0.1, 0.01) for both ℓ1 and ℓ2 regularization, with factors 0.1 and 0.01 respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 3.2189 - accuracy: 0.7967 - val_loss: 0.7169 - val_accuracy: 0.8340\n",
      "Epoch 2/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.7280 - accuracy: 0.8247 - val_loss: 0.6850 - val_accuracy: 0.8376\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"elu\",\n",
    "                       kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
    "    keras.layers.Dense(100, activation=\"elu\",\n",
    "                       kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
    "    keras.layers.Dense(10, activation=\"softmax\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 3.2911 - accuracy: 0.7924 - val_loss: 0.7218 - val_accuracy: 0.8310\n",
      "Epoch 2/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.7282 - accuracy: 0.8245 - val_loss: 0.6826 - val_accuracy: 0.8382\n"
     ]
    }
   ],
   "source": [
    "from functools import partial\n",
    "\n",
    "RegularizedDense = partial(keras.layers.Dense,\n",
    "                           activation=\"elu\",\n",
    "                           kernel_initializer=\"he_normal\",\n",
    "                           kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    RegularizedDense(300),\n",
    "    RegularizedDense(100),\n",
    "    RegularizedDense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.7611 - accuracy: 0.7576 - val_loss: 0.3730 - val_accuracy: 0.8644\n",
      "Epoch 2/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.4306 - accuracy: 0.8401 - val_loss: 0.3395 - val_accuracy: 0.8722\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(300, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(100, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alpha Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.8023 - accuracy: 0.7146 - val_loss: 0.5781 - val_accuracy: 0.8442\n",
      "Epoch 2/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.5663 - accuracy: 0.7905 - val_loss: 0.5182 - val_accuracy: 0.8520\n",
      "Epoch 3/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.5264 - accuracy: 0.8054 - val_loss: 0.4874 - val_accuracy: 0.8600\n",
      "Epoch 4/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.5126 - accuracy: 0.8092 - val_loss: 0.4890 - val_accuracy: 0.8598\n",
      "Epoch 5/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.5071 - accuracy: 0.8133 - val_loss: 0.4266 - val_accuracy: 0.8696\n",
      "Epoch 6/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.4793 - accuracy: 0.8198 - val_loss: 0.4585 - val_accuracy: 0.8640\n",
      "Epoch 7/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.4724 - accuracy: 0.8262 - val_loss: 0.4740 - val_accuracy: 0.8612\n",
      "Epoch 8/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4570 - accuracy: 0.8297 - val_loss: 0.4295 - val_accuracy: 0.8656\n",
      "Epoch 9/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4632 - accuracy: 0.8286 - val_loss: 0.4357 - val_accuracy: 0.8736\n",
      "Epoch 10/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4552 - accuracy: 0.8340 - val_loss: 0.4366 - val_accuracy: 0.8674\n",
      "Epoch 11/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.4461 - accuracy: 0.8346 - val_loss: 0.4278 - val_accuracy: 0.8684\n",
      "Epoch 12/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4419 - accuracy: 0.8351 - val_loss: 0.5086 - val_accuracy: 0.8558\n",
      "Epoch 13/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4329 - accuracy: 0.8385 - val_loss: 0.4280 - val_accuracy: 0.8728\n",
      "Epoch 14/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4305 - accuracy: 0.8399 - val_loss: 0.4460 - val_accuracy: 0.8628\n",
      "Epoch 15/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4315 - accuracy: 0.8397 - val_loss: 0.4361 - val_accuracy: 0.8706\n",
      "Epoch 16/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4251 - accuracy: 0.8403 - val_loss: 0.4280 - val_accuracy: 0.8758\n",
      "Epoch 17/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4207 - accuracy: 0.8427 - val_loss: 0.5336 - val_accuracy: 0.8584\n",
      "Epoch 18/20\n",
      "1719/1719 [==============================] - 3s 2ms/step - loss: 0.4365 - accuracy: 0.8387 - val_loss: 0.4769 - val_accuracy: 0.8736\n",
      "Epoch 19/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.4262 - accuracy: 0.8409 - val_loss: 0.4636 - val_accuracy: 0.8706\n",
      "Epoch 20/20\n",
      "1719/1719 [==============================] - 3s 1ms/step - loss: 0.4189 - accuracy: 0.8421 - val_loss: 0.4388 - val_accuracy: 0.8760\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9, nesterov=True)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 20\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 0s 834us/step - loss: 0.4723 - accuracy: 0.8639\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.47229740023612976, 0.8639000058174133]"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1719/1719 [==============================] - 1s 782us/step - loss: 0.3501 - accuracy: 0.8840\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.3501231074333191, 0.8840363621711731]"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1719/1719 [==============================] - 2s 1ms/step - loss: 0.4225 - accuracy: 0.8432\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MC Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_probas = np.stack([model(X_test_scaled, training=True)\n",
    "                     for sample in range(100)])\n",
    "y_proba = y_probas.mean(axis=0)\n",
    "y_std = y_probas.std(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.96]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(model.predict(X_test_scaled[:1]), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.57, 0.  , 0.42]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.93, 0.  , 0.05]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.  , 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.08, 0.  , 0.92]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.46, 0.  , 0.53]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.75, 0.  , 0.24]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.43, 0.  , 0.51]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.13, 0.  , 0.28, 0.  , 0.59]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.06, 0.  , 0.81]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.02, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.07, 0.  , 0.92]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.08, 0.  , 0.28, 0.  , 0.64]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.29, 0.  , 0.7 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.26, 0.  , 0.22, 0.  , 0.52]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.25, 0.  , 0.72]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.3 , 0.  , 0.06, 0.  , 0.65]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.  , 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.81, 0.  , 0.17]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.25, 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.03, 0.  , 0.93]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.84, 0.04, 0.01, 0.  , 0.1 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.2 , 0.  , 0.1 , 0.  , 0.7 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.28, 0.  , 0.72]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.12, 0.  , 0.76]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.75, 0.  , 0.21]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.11, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.28, 0.  , 0.39, 0.  , 0.34]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.04, 0.  , 0.95]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.18, 0.  , 0.51, 0.01, 0.3 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.59, 0.  , 0.01, 0.  , 0.39]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.8 , 0.  , 0.08]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.27, 0.  , 0.67]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.07, 0.  , 0.77, 0.  , 0.16]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.18, 0.  , 0.81]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.32, 0.  , 0.64]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.75, 0.  , 0.04, 0.  , 0.21]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.24, 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.44, 0.  , 0.38, 0.  , 0.17]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.02, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.12, 0.  , 0.82]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.97]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.28, 0.  , 0.71]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.17, 0.  , 0.82]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.23, 0.  , 0.27, 0.  , 0.5 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.78, 0.  , 0.16]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.03, 0.  , 0.94]],\n",
       "\n",
       "       [[0.  , 0.  , 0.01, 0.  , 0.02, 0.21, 0.01, 0.01, 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.06, 0.  , 0.93]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.4 , 0.  , 0.6 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.18, 0.01, 0.69]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.15, 0.  , 0.83]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.24, 0.  , 0.24, 0.  , 0.52]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.11, 0.  , 0.01, 0.  , 0.88]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.2 , 0.  , 0.8 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.04, 0.  , 0.91]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.21, 0.  , 0.77]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.26, 0.  , 0.67, 0.  , 0.07]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.27, 0.  , 0.55, 0.  , 0.19]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.09, 0.  , 0.47, 0.  , 0.43]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.07, 0.  , 0.71, 0.  , 0.22]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.1 , 0.  , 0.63, 0.01, 0.26]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.16, 0.  , 0.81]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.15, 0.  , 0.8 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.24, 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.13, 0.  , 0.81]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.42, 0.  , 0.01, 0.  , 0.58]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.15, 0.  , 0.83]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.79, 0.  , 0.19]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.21, 0.  , 0.79]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.93]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.12, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.06, 0.  , 0.9 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.01, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.08, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.51, 0.  , 0.46]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.96, 0.  , 0.02, 0.  , 0.02]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.38, 0.  , 0.05, 0.  , 0.57]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.07, 0.  , 0.92]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.06, 0.  , 0.89]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.05, 0.  , 0.94]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.37, 0.  , 0.37, 0.  , 0.26]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.37, 0.  , 0.19, 0.  , 0.44]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.11, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.01, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.41, 0.  , 0.59]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.08, 0.  , 0.21, 0.  , 0.71]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.13, 0.  , 0.75]]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(y_probas[:, :1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.1 , 0.  , 0.22, 0.  , 0.68]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(y_proba[:1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.18, 0.  , 0.24, 0.  , 0.29]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_std = y_probas.std(axis=0)\n",
    "np.round(y_std[:1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = np.argmax(y_proba, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8661"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy = np.sum(y_pred == y_test) / len(y_test)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MCDropout(keras.layers.Dropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)\n",
    "\n",
    "class MCAlphaDropout(keras.layers.AlphaDropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model = keras.models.Sequential([\n",
    "    MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout) else layer\n",
    "    for layer in model.layers\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_20\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_18 (Flatten)         (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout (MCAlphaDro (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_262 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout_1 (MCAlphaD (None, 300)               0         \n",
      "_________________________________________________________________\n",
      "dense_263 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout_2 (MCAlphaD (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "dense_264 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 266,610\n",
      "Trainable params: 266,610\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "mc_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9, nesterov=True)\n",
    "mc_model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model.set_weights(model.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bây giờ ta có thể sử dụng mô hình với MC Dropout:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.14, 0.  , 0.25, 0.01, 0.61]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(np.mean([mc_model.predict(X_test_scaled[:1]) for sample in range(100)], axis=0), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Max norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                           kernel_constraint=keras.constraints.max_norm(1.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.5763 - accuracy: 0.8020 - val_loss: 0.3674 - val_accuracy: 0.8674\n",
      "Epoch 2/2\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.3545 - accuracy: 0.8709 - val_loss: 0.3714 - val_accuracy: 0.8662\n"
     ]
    }
   ],
   "source": [
    "MaxNormDense = partial(keras.layers.Dense,\n",
    "                       activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       kernel_constraint=keras.constraints.max_norm(1.))\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    MaxNormDense(300),\n",
    "    MaxNormDense(100),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bài tập"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. đến 7."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tham khảo phụ lục A."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Học sâu với CIFAR10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a.\n",
    "*Bài tập: Xây dựng một DNN với 20 tầng ẩn, mỗi tầng 100 nơ-ron (khá là nhiều, nhưng đó là mục đích của bài tập này). Sử dụng khởi tạo He và hàm kích hoạt ELU.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 activation=\"elu\",\n",
    "                                 kernel_initializer=\"he_normal\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b.\n",
    "*Bài tập: Sử dụng tối ưu hóa Nadam và dừng sớm, huấn luyện mạng trên tập dữ liệu CIFAR10. Bạn có thể tải nó bằng `keras.datasets.cifar10.load_data()`. Bộ dữ liệu bao gồm 60.000 hình ảnh màu 32 × 32–pixel (50.000 để huấn luyện, 10.000 để thử nghiệm) với 10 lớp, vì vậy bạn sẽ cần một tầng đầu ra softmax với 10 nơ-ron. Hãy nhớ tìm kiếm tốc độ học phù hợp mỗi khi bạn thay đổi kiến trúc hoặc siêu tham số của mô hình.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hãy thêm tầng đầu ra vào mô hình:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hãy sử dụng trình tối ưu hóa Nadam với tỷ lệ học là 5e-5. Tôi đã thử tốc độ học 1e-5, 3e-5, 1e-4, 3e-4, 1e-3, 3e-3 và 1e-2, đồng thời so sánh các đường cong học tập của chúng trong 10 epochs cho mỗi tốc độc học (sử dụng lệnh gọi lại TensorBoard, bên dưới) . Tỷ lệ học tập 3e-5 và 1e-4 khá tốt, vì vậy tôi đã thử 5e-5 và có được kết quả tốt hơn một chút."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Nadam(learning_rate=5e-5)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hãy tải bộ dữ liệu CIFAR10. Chúng ta cũng muốn sử dụng tính năng dừng sớm, vì vậy ta cần một tập kiểm định. Hãy sử dụng 5,000 hình ảnh đầu tiên của tập huấn luyện ban đầu làm tập kiểm định:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.cifar10.load_data()\n",
    "\n",
    "X_train = X_train_full[5000:]\n",
    "y_train = y_train_full[5000:]\n",
    "X_valid = X_train_full[:5000]\n",
    "y_valid = y_train_full[:5000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bây giờ ta có thể tạo các callback mà chúng ta cần và huấn luyện mô hình:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ERROR: Failed to launch TensorBoard (exited with 255).\n",
       "Contents of stderr:\n",
       "E0213 19:12:55.493896 4621630912 program.py:311] TensorBoard could not bind to port 6006, it was already in use\n",
       "ERROR: TensorBoard could not bind to port 6006, it was already in use"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%tensorboard --logdir=./my_cifar10_logs --port=6006"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "1407/1407 [==============================] - 9s 5ms/step - loss: 9.4191 - accuracy: 0.1388 - val_loss: 2.2328 - val_accuracy: 0.2040\n",
      "Epoch 2/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 2.1097 - accuracy: 0.2317 - val_loss: 2.0485 - val_accuracy: 0.2402\n",
      "Epoch 3/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.9667 - accuracy: 0.2844 - val_loss: 1.9681 - val_accuracy: 0.2964\n",
      "Epoch 4/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.8740 - accuracy: 0.3149 - val_loss: 1.9178 - val_accuracy: 0.3254\n",
      "Epoch 5/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.8064 - accuracy: 0.3423 - val_loss: 1.8256 - val_accuracy: 0.3384\n",
      "Epoch 6/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.7525 - accuracy: 0.3595 - val_loss: 1.7430 - val_accuracy: 0.3692\n",
      "Epoch 7/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.7116 - accuracy: 0.3819 - val_loss: 1.7199 - val_accuracy: 0.3824\n",
      "Epoch 8/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.6782 - accuracy: 0.3935 - val_loss: 1.6746 - val_accuracy: 0.3972\n",
      "Epoch 9/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.6517 - accuracy: 0.4025 - val_loss: 1.6622 - val_accuracy: 0.4004\n",
      "Epoch 10/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.6140 - accuracy: 0.4194 - val_loss: 1.7065 - val_accuracy: 0.3840\n",
      "Epoch 11/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5884 - accuracy: 0.4301 - val_loss: 1.6736 - val_accuracy: 0.3914\n",
      "Epoch 12/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5640 - accuracy: 0.4378 - val_loss: 1.6220 - val_accuracy: 0.4224\n",
      "Epoch 13/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5437 - accuracy: 0.4448 - val_loss: 1.6332 - val_accuracy: 0.4144\n",
      "Epoch 14/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5214 - accuracy: 0.4555 - val_loss: 1.5785 - val_accuracy: 0.4326\n",
      "Epoch 15/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5117 - accuracy: 0.4564 - val_loss: 1.6267 - val_accuracy: 0.4164\n",
      "Epoch 16/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.4972 - accuracy: 0.4622 - val_loss: 1.5846 - val_accuracy: 0.4316\n",
      "Epoch 17/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.4888 - accuracy: 0.4661 - val_loss: 1.5549 - val_accuracy: 0.4420\n",
      "Epoch 18/100\n",
      "<<24 more lines>>\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.3362 - accuracy: 0.5212 - val_loss: 1.6025 - val_accuracy: 0.4500\n",
      "Epoch 31/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3360 - accuracy: 0.5207 - val_loss: 1.5175 - val_accuracy: 0.4602\n",
      "Epoch 32/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.3031 - accuracy: 0.5302 - val_loss: 1.5397 - val_accuracy: 0.4572\n",
      "Epoch 33/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.3082 - accuracy: 0.5308 - val_loss: 1.4997 - val_accuracy: 0.4776\n",
      "Epoch 34/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.2882 - accuracy: 0.5338 - val_loss: 1.5482 - val_accuracy: 0.4620\n",
      "Epoch 35/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.2889 - accuracy: 0.5355 - val_loss: 1.5474 - val_accuracy: 0.4604\n",
      "Epoch 36/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2761 - accuracy: 0.5410 - val_loss: 1.5434 - val_accuracy: 0.4658\n",
      "Epoch 37/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2658 - accuracy: 0.5481 - val_loss: 1.5502 - val_accuracy: 0.4706\n",
      "Epoch 38/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2554 - accuracy: 0.5489 - val_loss: 1.5527 - val_accuracy: 0.4624\n",
      "Epoch 39/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2504 - accuracy: 0.5471 - val_loss: 1.5482 - val_accuracy: 0.4602\n",
      "Epoch 40/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2516 - accuracy: 0.5545 - val_loss: 1.5881 - val_accuracy: 0.4574\n",
      "Epoch 41/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2401 - accuracy: 0.5566 - val_loss: 1.5403 - val_accuracy: 0.4670\n",
      "Epoch 42/100\n",
      "1407/1407 [==============================] - 8s 5ms/step - loss: 1.2305 - accuracy: 0.5570 - val_loss: 1.5343 - val_accuracy: 0.4790\n",
      "Epoch 43/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2228 - accuracy: 0.5615 - val_loss: 1.5344 - val_accuracy: 0.4708\n",
      "Epoch 44/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2161 - accuracy: 0.5619 - val_loss: 1.5782 - val_accuracy: 0.4526\n",
      "Epoch 45/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2124 - accuracy: 0.5641 - val_loss: 1.5182 - val_accuracy: 0.4794\n",
      "Epoch 46/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1870 - accuracy: 0.5766 - val_loss: 1.5435 - val_accuracy: 0.4650\n",
      "Epoch 47/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1925 - accuracy: 0.5701 - val_loss: 1.5532 - val_accuracy: 0.4686\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fb1d8ef8790>"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, epochs=100,\n",
    "          validation_data=(X_valid, y_valid),\n",
    "          callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "157/157 [==============================] - 0s 1ms/step - loss: 1.4960 - accuracy: 0.4762\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.4960416555404663, 0.47620001435279846]"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.models.load_model(\"my_cifar10_model.h5\")\n",
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mô hình có mất mát kiểm định thấp nhất có độ chính xác khoảng 47,6% trên tập kiểm định. Phải mất 27 epoch để đạt được mức mất mát kiểm định thấp nhất, với khoảng 8 giây mỗi epoch trên máy tính xách tay của tôi (không có GPU). Hãy xem liệu chúng ta có thể cải thiện hiệu suất bằng Chuẩn hóa theo Batch hay không."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### c.\n",
    "*Bài tập: Bây giờ hãy thử thêm Chuẩn hóa theo Batch và so sánh các đường cong học tập: Nó có hội tụ nhanh hơn trước không? Liệu nó tạo ra một mô hình tốt hơn? Nó ảnh hưởng như thế nào đến tốc độ huấn luyện?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Đoạn mã dưới đây rất giống với đoạn mã trên, với một vài thay đổi:\n",
    "\n",
    "* Ta thêm một tầng BN sau mỗi tầng Dày đặc (trước hàm kích hoạt), ngoại trừ tầng đầu ra. Ta cũng thêm một tầng BN trước tầng ẩn đầu tiên.\n",
    "* Ta thay đổi tỷ lệ học thành 5e-4. Ta thử nghiệm với 1e-5, 3e-5, 5e-5, 1e-4, 3e-4, 5e-4, 1e-3 và 3e-3 và chọn giá trị với hiệu suất kiểm định tốt nhất sau 20 epoch.\n",
    "* Ta đổi tên các thư mục chạy thành run_bn_* và tên tệp mô hình thành my_cifar10_bn_model.h5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "1407/1407 [==============================] - 19s 9ms/step - loss: 1.9765 - accuracy: 0.2968 - val_loss: 1.6602 - val_accuracy: 0.4042\n",
      "Epoch 2/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.6787 - accuracy: 0.4056 - val_loss: 1.5887 - val_accuracy: 0.4304\n",
      "Epoch 3/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.6097 - accuracy: 0.4274 - val_loss: 1.5781 - val_accuracy: 0.4326\n",
      "Epoch 4/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.5574 - accuracy: 0.4486 - val_loss: 1.5064 - val_accuracy: 0.4676\n",
      "Epoch 5/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.5075 - accuracy: 0.4642 - val_loss: 1.4412 - val_accuracy: 0.4844\n",
      "Epoch 6/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.4664 - accuracy: 0.4787 - val_loss: 1.4179 - val_accuracy: 0.4984\n",
      "Epoch 7/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.4334 - accuracy: 0.4932 - val_loss: 1.4277 - val_accuracy: 0.4906\n",
      "Epoch 8/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.4054 - accuracy: 0.5038 - val_loss: 1.3843 - val_accuracy: 0.5130\n",
      "Epoch 9/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.3816 - accuracy: 0.5106 - val_loss: 1.3691 - val_accuracy: 0.5108\n",
      "Epoch 10/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.3547 - accuracy: 0.5206 - val_loss: 1.3552 - val_accuracy: 0.5226\n",
      "Epoch 11/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 1.3244 - accuracy: 0.5371 - val_loss: 1.3678 - val_accuracy: 0.5142\n",
      "Epoch 12/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.3078 - accuracy: 0.5393 - val_loss: 1.3844 - val_accuracy: 0.5080\n",
      "Epoch 13/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 1.2889 - accuracy: 0.5431 - val_loss: 1.3566 - val_accuracy: 0.5164\n",
      "Epoch 14/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 1.2607 - accuracy: 0.5559 - val_loss: 1.3626 - val_accuracy: 0.5248\n",
      "Epoch 15/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.2580 - accuracy: 0.5587 - val_loss: 1.3616 - val_accuracy: 0.5276\n",
      "Epoch 16/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.2441 - accuracy: 0.5586 - val_loss: 1.3350 - val_accuracy: 0.5286\n",
      "Epoch 17/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.2241 - accuracy: 0.5676 - val_loss: 1.3370 - val_accuracy: 0.5408\n",
      "Epoch 18/100\n",
      "<<29 more lines>>\n",
      "Epoch 33/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.0336 - accuracy: 0.6369 - val_loss: 1.3682 - val_accuracy: 0.5450\n",
      "Epoch 34/100\n",
      "1407/1407 [==============================] - 11s 8ms/step - loss: 1.0228 - accuracy: 0.6388 - val_loss: 1.3348 - val_accuracy: 0.5458\n",
      "Epoch 35/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 1.0205 - accuracy: 0.6407 - val_loss: 1.3490 - val_accuracy: 0.5440\n",
      "Epoch 36/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 1.0008 - accuracy: 0.6489 - val_loss: 1.3568 - val_accuracy: 0.5408\n",
      "Epoch 37/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9785 - accuracy: 0.6543 - val_loss: 1.3628 - val_accuracy: 0.5396\n",
      "Epoch 38/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9832 - accuracy: 0.6592 - val_loss: 1.3617 - val_accuracy: 0.5482\n",
      "Epoch 39/100\n",
      "1407/1407 [==============================] - 12s 8ms/step - loss: 0.9707 - accuracy: 0.6581 - val_loss: 1.3767 - val_accuracy: 0.5446\n",
      "Epoch 40/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9590 - accuracy: 0.6651 - val_loss: 1.4200 - val_accuracy: 0.5314\n",
      "Epoch 41/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9548 - accuracy: 0.6668 - val_loss: 1.3692 - val_accuracy: 0.5450\n",
      "Epoch 42/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9480 - accuracy: 0.6667 - val_loss: 1.3841 - val_accuracy: 0.5310\n",
      "Epoch 43/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9411 - accuracy: 0.6716 - val_loss: 1.4036 - val_accuracy: 0.5382\n",
      "Epoch 44/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9383 - accuracy: 0.6708 - val_loss: 1.4114 - val_accuracy: 0.5236\n",
      "Epoch 45/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9258 - accuracy: 0.6769 - val_loss: 1.4224 - val_accuracy: 0.5324\n",
      "Epoch 46/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.9072 - accuracy: 0.6836 - val_loss: 1.3875 - val_accuracy: 0.5442\n",
      "Epoch 47/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.8996 - accuracy: 0.6850 - val_loss: 1.4449 - val_accuracy: 0.5280\n",
      "Epoch 48/100\n",
      "1407/1407 [==============================] - 13s 9ms/step - loss: 0.9050 - accuracy: 0.6835 - val_loss: 1.4167 - val_accuracy: 0.5338\n",
      "Epoch 49/100\n",
      "1407/1407 [==============================] - 12s 9ms/step - loss: 0.8934 - accuracy: 0.6880 - val_loss: 1.4260 - val_accuracy: 0.5294\n",
      "157/157 [==============================] - 1s 2ms/step - loss: 1.3344 - accuracy: 0.5398\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.3343921899795532, 0.5397999882698059]"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "model.add(keras.layers.BatchNormalization())\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100, kernel_initializer=\"he_normal\"))\n",
    "    model.add(keras.layers.BatchNormalization())\n",
    "    model.add(keras.layers.Activation(\"elu\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.Nadam(learning_rate=5e-4)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_bn_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_bn_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
    "\n",
    "model.fit(X_train, y_train, epochs=100,\n",
    "          validation_data=(X_valid, y_valid),\n",
    "          callbacks=callbacks)\n",
    "\n",
    "model = keras.models.load_model(\"my_cifar10_bn_model.h5\")\n",
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* *Mô hình có hội tụ nhanh hơn trước không?* Nhanh hơn nhiều! Mô hình trước đó mất 27 epoch để đạt được mức tổn thất xác thực thấp nhất, trong khi mô hình mới đạt được mức tổn thất tương tự chỉ trong 5 epoch và tiếp tục đạt được tiến bộ cho đến epoch thứ 16. Các tầng BN ổn định quá trình huấn luyện và cho phép chúng ta sử dụng tốc độ học lớn hơn nhiều, do đó, đạt được sự hội tụ nhanh hơn.\n",
    "* *BN có tạo ra một mô hình tốt hơn không?* Có! Mô hình cuối cùng cũng tốt hơn nhiều, với độ chính xác 54,0% thay vì 47,6%. Nó vẫn không phải là một mô hình quá tốt, nhưng ít nhất nó cũng tốt hơn nhiều so với trước đây (Mạng nơ-ron Tích chập sẽ hoạt động tốt hơn nhiều, nhưng đó là một chủ đề khác, xem chương 14).\n",
    "* *BN ảnh hưởng đến tốc độ huấn luyện như thế nào?* Mặc dù mô hình hội tụ nhanh hơn, nhưng mỗi epoch mất khoảng 12 giây thay vì 8 giây do các lớp BN yêu cầu tính toán bổ sung. Nhưng nhìn chung thời gian huấn luyện (wall time) đã được rút ngắn đáng kể!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### d.\n",
    "*Bài tập: Hãy thử thay thế Chuẩn hóa theo Batch bằng SELU và thực hiện các điều chỉnh cần thiết để đảm bảo mạng tự chuẩn hóa (nghĩa là chuẩn hóa các đặc trưng đầu vào, sử dụng khởi tạo thông thường LeCun, đảm bảo DNN chỉ chứa một chuỗi các tầng dày đặc, v.v.) .*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "1407/1407 [==============================] - 10s 5ms/step - loss: 2.0622 - accuracy: 0.2631 - val_loss: 1.7878 - val_accuracy: 0.3552\n",
      "Epoch 2/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.7328 - accuracy: 0.3830 - val_loss: 1.7028 - val_accuracy: 0.3828\n",
      "Epoch 3/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.6342 - accuracy: 0.4279 - val_loss: 1.6692 - val_accuracy: 0.4022\n",
      "Epoch 4/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5524 - accuracy: 0.4538 - val_loss: 1.6350 - val_accuracy: 0.4300\n",
      "Epoch 5/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.4979 - accuracy: 0.4756 - val_loss: 1.5773 - val_accuracy: 0.4356\n",
      "Epoch 6/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.4428 - accuracy: 0.4902 - val_loss: 1.5529 - val_accuracy: 0.4630\n",
      "Epoch 7/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3966 - accuracy: 0.5126 - val_loss: 1.5290 - val_accuracy: 0.4682\n",
      "Epoch 8/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3549 - accuracy: 0.5232 - val_loss: 1.4633 - val_accuracy: 0.4792\n",
      "Epoch 9/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3162 - accuracy: 0.5444 - val_loss: 1.4787 - val_accuracy: 0.4776\n",
      "Epoch 10/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2825 - accuracy: 0.5534 - val_loss: 1.4794 - val_accuracy: 0.4934\n",
      "Epoch 11/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2529 - accuracy: 0.5682 - val_loss: 1.5529 - val_accuracy: 0.4982\n",
      "Epoch 12/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2256 - accuracy: 0.5784 - val_loss: 1.4942 - val_accuracy: 0.4902\n",
      "Epoch 13/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2049 - accuracy: 0.5823 - val_loss: 1.4868 - val_accuracy: 0.5024\n",
      "Epoch 14/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1627 - accuracy: 0.6012 - val_loss: 1.4839 - val_accuracy: 0.5082\n",
      "Epoch 15/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1543 - accuracy: 0.6034 - val_loss: 1.5097 - val_accuracy: 0.4968\n",
      "Epoch 16/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1200 - accuracy: 0.6135 - val_loss: 1.5001 - val_accuracy: 0.5120\n",
      "Epoch 17/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1028 - accuracy: 0.6199 - val_loss: 1.4856 - val_accuracy: 0.5056\n",
      "Epoch 18/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0863 - accuracy: 0.6265 - val_loss: 1.5116 - val_accuracy: 0.4966\n",
      "Epoch 19/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0715 - accuracy: 0.6345 - val_loss: 1.5787 - val_accuracy: 0.5070\n",
      "Epoch 20/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0342 - accuracy: 0.6453 - val_loss: 1.4987 - val_accuracy: 0.5144\n",
      "Epoch 21/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0169 - accuracy: 0.6531 - val_loss: 1.6292 - val_accuracy: 0.4462\n",
      "Epoch 22/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1346 - accuracy: 0.6074 - val_loss: 1.5280 - val_accuracy: 0.5136\n",
      "Epoch 23/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9820 - accuracy: 0.6678 - val_loss: 1.5392 - val_accuracy: 0.5040\n",
      "Epoch 24/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9701 - accuracy: 0.6679 - val_loss: 1.5505 - val_accuracy: 0.5170\n",
      "Epoch 25/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3604 - accuracy: 0.6753 - val_loss: 1.5468 - val_accuracy: 0.4992\n",
      "Epoch 26/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0177 - accuracy: 0.6510 - val_loss: 1.5474 - val_accuracy: 0.5020\n",
      "Epoch 27/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9425 - accuracy: 0.6798 - val_loss: 1.5545 - val_accuracy: 0.5076\n",
      "Epoch 28/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9005 - accuracy: 0.6902 - val_loss: 1.5659 - val_accuracy: 0.5138\n",
      "157/157 [==============================] - 0s 1ms/step - loss: 1.4633 - accuracy: 0.4792\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.4633383750915527, 0.47920000553131104]"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.Nadam(learning_rate=7e-4)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_selu_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_selu_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
    "\n",
    "X_means = X_train.mean(axis=0)\n",
    "X_stds = X_train.std(axis=0)\n",
    "X_train_scaled = (X_train - X_means) / X_stds\n",
    "X_valid_scaled = (X_valid - X_means) / X_stds\n",
    "X_test_scaled = (X_test - X_means) / X_stds\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=100,\n",
    "          validation_data=(X_valid_scaled, y_valid),\n",
    "          callbacks=callbacks)\n",
    "\n",
    "model = keras.models.load_model(\"my_cifar10_selu_model.h5\")\n",
    "model.evaluate(X_valid_scaled, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "157/157 [==============================] - 0s 1ms/step - loss: 1.4633 - accuracy: 0.4792\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.4633383750915527, 0.47920000553131104]"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.models.load_model(\"my_cifar10_selu_model.h5\")\n",
    "model.evaluate(X_valid_scaled, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ta có được độ chính xác 47,9%, tốt hơn không nhiều so với mô hình ban đầu (47,6%) và không tốt bằng mô hình sử dụng Chuẩn hóa theo Batch (54,0%). Tuy nhiên, sự hội tụ gần như nhanh như với mô hình BN, cộng với việc mỗi epoch chỉ mất 7 giây. Vì vậy, nó là mô hình được huấn luyện nhanh nhất cho đến nay."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### e.\n",
    "*Bài tập: Hãy thử chuẩn hóa mô hình bằng cách loại bỏ alpha. Sau đó, không cần huấn luyện lại mô hình của bạn, hãy xem liệu bạn có thể đạt được độ chính xác cao hơn khi sử dụng MC Dropout không.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "1407/1407 [==============================] - 9s 5ms/step - loss: 2.0583 - accuracy: 0.2742 - val_loss: 1.7429 - val_accuracy: 0.3858\n",
      "Epoch 2/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.6852 - accuracy: 0.4008 - val_loss: 1.7055 - val_accuracy: 0.3792\n",
      "Epoch 3/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5963 - accuracy: 0.4413 - val_loss: 1.7401 - val_accuracy: 0.4072\n",
      "Epoch 4/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.5231 - accuracy: 0.4634 - val_loss: 1.5728 - val_accuracy: 0.4584\n",
      "Epoch 5/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.4619 - accuracy: 0.4887 - val_loss: 1.5448 - val_accuracy: 0.4702\n",
      "Epoch 6/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.4074 - accuracy: 0.5061 - val_loss: 1.5678 - val_accuracy: 0.4664\n",
      "Epoch 7/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.3718 - accuracy: 0.5222 - val_loss: 1.5764 - val_accuracy: 0.4824\n",
      "Epoch 8/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.3220 - accuracy: 0.5387 - val_loss: 1.4805 - val_accuracy: 0.4890\n",
      "Epoch 9/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.2908 - accuracy: 0.5487 - val_loss: 1.5521 - val_accuracy: 0.4638\n",
      "Epoch 10/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.2537 - accuracy: 0.5607 - val_loss: 1.5281 - val_accuracy: 0.4924\n",
      "Epoch 11/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.2215 - accuracy: 0.5782 - val_loss: 1.5147 - val_accuracy: 0.5046\n",
      "Epoch 12/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.1910 - accuracy: 0.5831 - val_loss: 1.5248 - val_accuracy: 0.5002\n",
      "Epoch 13/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.1659 - accuracy: 0.5982 - val_loss: 1.5620 - val_accuracy: 0.5066\n",
      "Epoch 14/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.1282 - accuracy: 0.6120 - val_loss: 1.5440 - val_accuracy: 0.5180\n",
      "Epoch 15/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.1127 - accuracy: 0.6133 - val_loss: 1.5782 - val_accuracy: 0.5146\n",
      "Epoch 16/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0917 - accuracy: 0.6266 - val_loss: 1.6182 - val_accuracy: 0.5182\n",
      "Epoch 17/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 1.0620 - accuracy: 0.6331 - val_loss: 1.6285 - val_accuracy: 0.5126\n",
      "Epoch 18/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0433 - accuracy: 0.6413 - val_loss: 1.6299 - val_accuracy: 0.5158\n",
      "Epoch 19/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 1.0087 - accuracy: 0.6549 - val_loss: 1.7172 - val_accuracy: 0.5062\n",
      "Epoch 20/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 0.9950 - accuracy: 0.6571 - val_loss: 1.6524 - val_accuracy: 0.5098\n",
      "Epoch 21/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9848 - accuracy: 0.6652 - val_loss: 1.7686 - val_accuracy: 0.5038\n",
      "Epoch 22/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9597 - accuracy: 0.6744 - val_loss: 1.6177 - val_accuracy: 0.5084\n",
      "Epoch 23/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9399 - accuracy: 0.6790 - val_loss: 1.7095 - val_accuracy: 0.5082\n",
      "Epoch 24/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.9148 - accuracy: 0.6884 - val_loss: 1.7160 - val_accuracy: 0.5150\n",
      "Epoch 25/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 0.9023 - accuracy: 0.6949 - val_loss: 1.7017 - val_accuracy: 0.5152\n",
      "Epoch 26/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.8732 - accuracy: 0.7031 - val_loss: 1.7274 - val_accuracy: 0.5088\n",
      "Epoch 27/100\n",
      "1407/1407 [==============================] - 6s 5ms/step - loss: 0.8542 - accuracy: 0.7091 - val_loss: 1.7648 - val_accuracy: 0.5166\n",
      "Epoch 28/100\n",
      "1407/1407 [==============================] - 7s 5ms/step - loss: 0.8499 - accuracy: 0.7118 - val_loss: 1.7973 - val_accuracy: 0.5000\n",
      "157/157 [==============================] - 0s 1ms/step - loss: 1.4805 - accuracy: 0.4890\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.4804893732070923, 0.48899999260902405]"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "\n",
    "model.add(keras.layers.AlphaDropout(rate=0.1))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.Nadam(learning_rate=5e-4)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_alpha_dropout_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_alpha_dropout_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
    "\n",
    "X_means = X_train.mean(axis=0)\n",
    "X_stds = X_train.std(axis=0)\n",
    "X_train_scaled = (X_train - X_means) / X_stds\n",
    "X_valid_scaled = (X_valid - X_means) / X_stds\n",
    "X_test_scaled = (X_test - X_means) / X_stds\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=100,\n",
    "          validation_data=(X_valid_scaled, y_valid),\n",
    "          callbacks=callbacks)\n",
    "\n",
    "model = keras.models.load_model(\"my_cifar10_alpha_dropout_model.h5\")\n",
    "model.evaluate(X_valid_scaled, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mô hình đạt độ chính xác 48,9% trên tập kiểm định. Con số này khá hơn một chút so với không có dropout (47,6%). Với tìm kiếm siêu tham số mở rộng, nó có thể làm tốt hơn (tôi đã thử tỷ lệ dropout là 5%, 10%, 20% và 40% và tỷ lệ học 1e-4, 3e-4, 5e-4 và 1e-3 ), nhưng có lẽ không tốt hơn nhiều trong trường hợp này."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hãy sử dụng MC Dropout ngay bây giờ. Ta sẽ cần tầng `MCAlphaDropout` mà chúng ta đã sử dụng trước đó, vì vậy hãy sao chép nó ở đây cho thuận tiện:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MCAlphaDropout(keras.layers.AlphaDropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bây giờ, hãy tạo một mô hình mới, giống với mô hình mà chúng ta vừa huấn luyện (có cùng trọng số), nhưng với các tầng dropout `MCAlphaDropout` thay vì các tầng `AlphaDropout`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model = keras.models.Sequential([\n",
    "    MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout) else layer\n",
    "    for layer in model.layers\n",
    "])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sau đó, hãy thêm một vài hàm tiện ích. Hàm đầu tiên sẽ chạy mô hình nhiều lần (10 theo mặc định) và nó sẽ trả về xác suất dự đoán lớp trung bình. Hàm thứ hai sẽ sử dụng các xác suất trung bình này để dự đoán lớp có khả năng nhất cho từng mẫu:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mc_dropout_predict_probas(mc_model, X, n_samples=10):\n",
    "    Y_probas = [mc_model.predict(X) for sample in range(n_samples)]\n",
    "    return np.mean(Y_probas, axis=0)\n",
    "\n",
    "def mc_dropout_predict_classes(mc_model, X, n_samples=10):\n",
    "    Y_probas = mc_dropout_predict_probas(mc_model, X, n_samples)\n",
    "    return np.argmax(Y_probas, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bây giờ, hãy đưa ra dự đoán cho tất cả các mẫu trong tập kiểm định và tính toán độ chính xác:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4892"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "y_pred = mc_dropout_predict_classes(mc_model, X_valid_scaled)\n",
    "accuracy = np.mean(y_pred == y_valid[:, 0])\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chúng ta không có được sự cải thiện về độ chính xác trong trường hợp này (vẫn ở mức chính xác 48,9%).\n",
    "\n",
    "Vì vậy, mô hình tốt nhất mà chúng ta có trong bài tập này là mô hình Chuẩn hóa theo Batch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### f.\n",
    "*Bài tập: Huấn luyện lại mô hình của bạn bằng cách sử dụng định thời 1cycle và xem liệu nó có cải thiện tốc độ huấn luyện và độ chính xác của mô hình hay không.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "\n",
    "model.add(keras.layers.AlphaDropout(rate=0.1))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.SGD(learning_rate=1e-3)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "352/352 [==============================] - 2s 6ms/step - loss: nan - accuracy: 0.1255\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(9.999999747378752e-06,\n",
       " 9.999868392944336,\n",
       " 2.6167492866516113,\n",
       " 3.9354368618556435)"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAERCAYAAACO6FuTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAng0lEQVR4nO3deXxV1bn/8c+TgYRMQEgCyBTCPAlIHJFRLc5jtdSrV+tVW/VqtXq1s8O1tWr1d1tbtVhaqqhVW1vnqcUJQSSoKCCiSJihYZ4DSZ7fH/sEY5oDCUnOPjn5vl+v/crZe69zzrM44TxZa+29lrk7IiIidUkKOwAREYlfShIiIhKVkoSIiESlJCEiIlEpSYiISFRKEiIiElVK2AE0pby8PC8sLAw7DBFpAht27GH15l0M7JJDSpKFHU5Cmzt37np3z6/rXEIlicLCQkpKSsIOQ0SawCPvLuMnf5/Paz86nvzstLDDSWhmtizaOXU3iYhIVEoSIiISlZKEiIhEpSQhIiJRKUmIiEhUShIiIhKVkoSIiESlJCEiIlEpSYiISFRKEiIiEpWShIiIRKUkISIiUSlJiIhIVEoSIiISlZKEiIhEpSQhIiJRKUmIiEhUMU0SZjbNzNaY2VYzW2xml0YpZ2Z2u5mtMrMtZvaGmQ2OZawiIhL7lsQdQKG75wCnA7eb2cg6yp0LXAKMBnKBWcAjMYtSRESAGCcJd1/g7uXVu5Gtdx1FewEz3P0Ld68EpgGDYhSmiIhExHxMwszuN7OdwCJgDfBiHcX+DPQxs35mlgpcBLwc5fUuN7MSMyspKytrtrhFRFqjmCcJd78SyCboSnoaKK+j2BrgbeBTYBdB99N1UV5vsrsXu3txfn5+8wQtItJKhXJ1k7tXuvsMoBtwRR1FbgYOB7oD6cCtwHQzy4hdlCIiEvYlsCnUPSYxDHjC3Ve6e4W7TwU6oHEJEZGYilmSMLMCM5tkZllmlmxmE4FvAtPrKD4HONfMOplZkpldCKQCn8cqXhERCf6SjxUn6Fp6kCA5LQOudfdnzKwHsBAY5O7LgTuBAuBDIJMgOZzj7ptjGK+ISKsXsyTh7mXA2CjnlgNZNfZ3A1dFNhERCUnYYxIiIhLHlCRERCQqJQkREYlKSUJERKJSkhARkaiUJEREJColCRERiUpJQkREolKSEBGRqJQkREQkKiUJERGJSklCRESiUpIQEZGolCRERCQqJQkREYlKSUJERKJSkhARkaiUJEREJColCRERiUpJQkREolKSEBGRqJQkREQkKiUJERGJKqZJwsymmdkaM9tqZovN7NL9lC0ys+fNbJuZrTezu2IZq4iIxL4lcQdQ6O45wOnA7WY2snYhM2sDvAZMBzoD3YBpsQxURERinCTcfYG7l1fvRrbedRS9GFjt7ve6+w533+3uH8UqThERCcR8TMLM7jezncAiYA3wYh3FjgJKzeylSFfTG2Y2NMrrXW5mJWZWUlZW1oyRi4i0PjFPEu5+JZANjAaeBsrrKNYNmAT8GjgEeAF4JtINVfv1Jrt7sbsX5+fnN1/gIiKtUChXN7l7pbvPIEgGV9RRZBcww91fcvc9wC+BjsDAGIYpItLqhX0JbAp1j0l8RDBeISIiIYpZkjCzAjObZGZZZpZsZhOBbxJcwVTbNOAoMzvezJKBa4H1wCexildERGLbknCCrqWVwCaCLqRr3f0ZM+thZtvNrAeAu38KXAA8GCl7BnB6pOtJRERiJCVWb+TuZcDYKOeWA1m1jj1NMLAtIiIhCXtMQkRE4piShIiIRKUkISIiUSlJiIhIVEoSIiISlZKEiIhEpSQhIiJRKUmIiEhUShIiIhKVkoSIiESlJCEiIlEpSYiISFRKEiIiEpWShIiIRKUkISIiUSlJiIhIVEoSIiISlZKEiIhEpSQhIiJRKUmIiEhUShIiIhKVkoSIiEQV0yRhZtPMbI2ZbTWzxWZ2aT2eM93M3MxSYhGjiIh8KdYtiTuAQnfPAU4HbjezkdEKm9l/AEoOIiIhiWmScPcF7l5evRvZetdV1szaATcDN8YoPBERqSXmYxJmdr+Z7QQWAWuAF6MU/TnwALA2VrGJiMhXxTxJuPuVQDYwGngaKK9dxsyKgVHAfQd6PTO73MxKzKykrKysqcMVEWnVQrm6yd0r3X0G0A24ouY5M0sC7ge+6+4V9Xitye5e7O7F+fn5zROwiEgrFfYlsCn8+5hEDlAMPGFma4E5keMrzWx0LIMTEWntYnblkJkVABOA54FdwPHAN4HzaxXdAhxSY7878B4wElB/kohIDMXy8lIn6Fp6kKAFswy41t2fMbMewEJgkLsvp8ZgtZmlRx6uq0/3k4iINJ2YJQl3LwPGRjm3HMiKcq4UsOaLTEREogl7TEJEROKYkoSIiETV6CRhZqlNEYiIiMSfBiUJM7vGzM6psT8F2GVmn5pZ/yaPTkREQtXQlsQ1RC5DNbMxwHkEl7B+CNzTpJGJiEjoGnp1U1egNPL4NOApd3/SzD4G3m7KwEREJHwNbUlsBarnvjgB+Gfk8V4gvc5niIhIi9XQlsSrwENm9gHQB3gpcnwwsLQpAxMRkfA1tCVxFfAOkAd83d03Ro4fBjzelIGJiEj4GtSScPetwNV1HL+5ySISEZG40dBLYAfVvNTVzE6IrFv9AzNLbvrwREQkTA3tbpoCjAAws27AM0AuQTfU7U0bmoiIhK2hSWIg8H7k8bnAbHc/GbiQYNpvERFJIA1NEsnAnsjj4/hyfeolQKemCkpEROJDQ5PEfOCKyApxxwEvR453BdY3ZWAiIhK+hiaJm4DLgDeAx93948jx0wlWjxMRkQTS0Etg3zKzfCDH3TfVOPU7YGeTRiYiIqFr8Mp07l5pZrvMbAjBkqRLIqvHiYhIgmnofRIpZnY3sAmYB3wMbDKzu7SuhIhI4mloS+IugktdvwPMiBwbDdxBkHBuaLrQREQkbA1NEucDl7j7izWOLTGzMuD3KEmIiCSUhl7d1I7gnojalgDtGx2NiIjElYYmiXkEq9PV9t3IORERSSANTRI3AheZ2WIz+5OZTTWzT4ELqEdXU2QywDVmtjXyGpdGKXeRmc2NlFsZGRhv8JVYIiLSOA1KEu7+FtAPeArIAnIijydSdwujtjuAQnfPIbgB73YzG1lHuQzgWoJ1K44kuLtb4x0iIjF2MPdJrAZ+VPOYmQ0DzqnHcxfU3I1svYG5tco9UGN3lZk9CoxvaKwiItI4De1uajQzu9/MdgKLgDV8OUng/owBFhywlIiINKmYJwl3vxLIJri/4mmgfH/lzexbQDHwyyjnLzezEjMrKSsra+pwRURatZgnCQim9nD3GUA34Ipo5czsTOAXwEnuXucss+4+2d2L3b04Pz+/WeIVEWmt6jUmYWbPHqBITiPev3eU9zwReAg4pcZssyIiEkP1HbjeUI/zS/dXwMwKgAnA88Au4HiCKT7Or6PsBOBR4Cx31xTkIiIhqVeScPdvNcF7OUHX0oME3VzLgGvd/Rkz6wEsBAa5+3LgJwR3d79oZtXPf9vdT2qCOEREpJ5idoOau5cBY6OcW05w30X1vi53FWnt3MOOQAhp4FpE5EAqq4IkkZxkBygpzUlJQkTiUmWkIZFsShJhUpIQkbhUFWlJJOlbKlT65xeRuFTp6m6KB0oSIhKXqiJJIkndTaFSkhCRuFSlgeu4oCQhInGpsir4qZZEuJQkRCQuVe7rbgo5kFZOSUJE4lJVlZNkYGpJhEpJQkTiUqW7xiPigJKEiMSloCWhJBE2JQkRiUuVVWpJxAMlCRGJS5XumpIjDihJiEhcqqpyktSSCJ2ShIjEJQ1cxwcliVqq7/IUkXBVVulGunigJFHDo7OXMeJ/X+OtxWVhhyLS6lVVOcn6hgpdwn8Ev/rHZzwxZ/lXjm3dvZen31/J5p172F5ewS9f+ZSfvbCQW59dyI7yCi57uITlG3aGFLGIgAau40XMli8Nw56KKh5483M6ZqZxXnF33OHCP8xm9hcbqahyDi/swOBD2jF1ZilJBqP65PHDkwdy9v0zueGpeXTLbUt5RRV3nnMotz+/kFWbd3HbGUPolZcZdtVEEl6Va+A6HiR0kpi/egu791axavMuFqzeSlpKEu98voFTD+3C0K7tuOOlRcwp3cQ5h3XjznOGkhJp2157fF/ueGkRn6xNYUd5BTM+W8+WXXtpm5rMhVNm89p1Y2nbJjnk2okktirdJxEXEjpJvLd0IwBm8OrCdXRtnw7A907oR1F+Fkf0yuWl+Wu5fEzRvgQB8O2xvfnPowtJS0ni1YXreG7eao7olUv/ztlMmvwud7z0CbeePnjfnDKVVc7spRuY/sm/aNsmmdTkJIZ2a8dn67axeN12ivIzuWRUL9JTlVhE6qvStXRpPEjoJFFSupGivEyy0lOYs3Qj63IzaJ+Ruq+7aESPDozo0aHO51a3FE4c0pkTh3Ted/xbowr54zulLNuwk6Fd25GRlszj7y1nxcZdtElJYm9lFV7jAqn87DT+Mnclr8xfy40nDmBUn7zmq7BIAtF9EvEhoZPE4nXbGda9Pe3bpvK3D1bxr227GdG9faNmlfzpqYPomZvBPa8u5u3Pyqhy6J2fyX3fHMHxAzvRJiWJHXsq+GD5ZgZ0zqZTTjovfbyGm59dwAVTZnP314fRrUNbjijM1X8Akf2ojMwCK+GKaZIws2nAcUAmsBa4y91/H6XsdcBNQFvgr8AV7l5e3/eqqKxi9eZdnD7sEArzMnnk3WVsL6vgvOLuja0DF4/qxaQjemAG67fvIT8rjTYpX3ZX5aSnMrZf/r79k4Z2YUy/fE69bwY3PDUPgEFdcvjdhSPpnpvRqHhEElWla4K/eBDrS2DvAArdPQc4HbjdzEbWLmRmE4HvEySUQqAIuLUhb7Rmy24qqpweuRkM795u3/GzDut68NHXkJ6aTFpKMl3bt/1KgogmMy2FKRcV879nDuGX5w5jxaadjLn7dc57cBYvz19LpW7iE/kKDVzHh5i2JNx9Qc3dyNYbmFur6EXAlOryZva/wKMEiaNeVmwM7nPoltuWorwsAHp2zKAgO/1gw2+0ovwsivKDWI4ozOVvH6ziqbkr+M60ubTPSOXIXrkcVdSRY/vksaRsO30KsuhTkB1avCJh0rQc8SHmYxJmdj9wMUE30gfAi3UUGww8U2N/HtDJzDq6+4b6vM/ySJLokZtBUpIx/fqxdMxKa0zoTapHxwy+e3xfrhrfm9cWrmP6on/x7tINvLJg3VfKnTK0Cz84eQDdOqhbSlqXSq0nERdiniTc/Uozuxo4GhgH1DXOkAVsqbFf/Tgb+EqSMLPLgcsBevTowfbyCm5/fiF/nrOC5CSjS7u2APv+go83KclJnDS0CycN7QIELaA3F5fRIzeDkmWbmPzWEl5ZsJY2KUnkZ6dRlJfJ2Yd1wwlaIx+v2kLJso1s311BZloKh3Zrx7Bu7enWoa2WfZQWrUotibgQytVN7l4JzDCzC4ArgF/XKrIdyKmxX/14Wx2vNRmYDFBcXOx3v7yIP89ZAbTMRUu652ZwwVE9ARjTL59vHN6dae8uo3xvFf/atpuS0k1c/fgHX3lOm+QkstJT2F5ewZ6KKgC6dWjLqN55nDqsC6P75v/b+4jEu8oqTcsRD8K+BDaFYEyitgXAMODJyP4wYF19upqWbtjJsG7tGNa9PdnpYVev8bq2b8tNJw7Yt79rTyXzVm4mJcn4eNUWeuVlMqpPHqnJSeypqOLTtdv4cMUm3vi0jFcWruWJkhUM7dqOY/vmcWyfPEb27KCb+qRFqKqCpISfXS7+xexb1MwKgAnA88Au4Hjgm8D5dRR/GJhqZo8Ca4AfA1Pr8z6bduyhY1YbbjtjSFOEHXfatknmqKKOABQX5n7lXJuU4E7vod3aceHRhZRXVPLIrGW8umAdD731BQ+8sYS0lCTOGH4Il40uoig/q8W1tKT1qHQnVVkidLH8U9sJupYeJLj0dhlwrbs/Y2Y9gIXAIHdf7u4vm9ldwOt8eZ/EzfV5k00799C3ID7HH2ItLSWZS0cXcenoInaUV/De0o289sk6/jp3JU+WrCQlyThhUCf+Z2L/uB2zkdZLA9fxIWZJwt3LgLFRzi0nGKyueexe4N6Gvs+mHXvokNnmoGJMZJlpKYwfUMD4AQVcf0I/nv9oDUvX7+Cv76/kH5+sY8KAAo7s1ZFh3dsz+JAcdUlJ6FwD13Gh5Xfa1+AOO/ZU0iEjNexQ4lrHrDQuOqYQgCvH9+bBN77gpflr9l1+m5JkjOjRnpOGdKFfp2yOKsr9ygSIIrGg9STiQ0IliYrIXctqSdRfQXY6Pz1tED89bRDrtu7mwxWb+XDFZl6Zv5bbnl8IBHNT3X3uMIZ3a6/5piRmKqvQ71scSKgkUVkVXP6Zm6EkcTA65aQzcXBnJg7uzI0T+7N++x5mL93Azc8s4Oz7Z9I9ty1Xj+/LucXddA+GNLsqXQIbFxIqSagl0XTMjPzsNE499BBG9c7jH5+s47H3lnPjXz9i2uxlXHBUT8b2y6cgO00JQ5qFpuWIDwmVJKonyeuglkST6pDZhnOLu3POYd14smQFv5+xlBv/8hEAmW2SKcrPYlz/fE4bdgj9OmmuKWkaWk8iPiRUkviyJaGB6+aQlGRMOqIH3zi8O+8v38TC1VtZUraDT9Zs5bevf8590z/n5KGduXJcH4Z0bXfgFxTZj2Cq8LCjkIRKEpWVThJqSTQ3M2Nkz1xG9vzyZr6ybeU8Nns5D765hBc/Xsvw7u254KienHpoF11OKwdF03LEh4S6rrGiqorstBRSdblmzOVnp/Hd4/vy7g+P4+bTBrF1915ueGoeR93xT+56eRFrt+wOO0RpYdTdFB8SqiWxa28lRbrbOlTt2qbyrVG9uPiYQmYt2cDUmaU88OYSJr/1Bacc2oXLRhepK0rqRfdJxIfEShJ7Kjmmd8ewwxCCLqlj+uRxTJ88lm/YyR9nLuXJOSt45sPVjO+fz4QBBZx3eHfSUtQVJXXTfRLxIaH6ZRw4ukhJIt706JjBzacNZuYPjuM7Y3tTumEnP3lmAcfcMZ0H3liipVulTsF6EmFHIQnVkjCguLBD2GFIFO3apvL9kwZw04n9mbVkAw+9/QV3vryIl+avYdLhPThuYAGdcsJbXlbiiwau40NCJYlD2rclo01CVSkhVXdFHd27I898uJq7X/mUH/7tY9o8m8S5xd24dHQRvfIyww5TQqaB6/iQUN+oubrTukUxM84c0ZUzhh/CkrLt/OGdUp4qWcmjs5dz7shuXDm+j5JFK1algeu4kFBJQlomM6NPQTY/P2so1x7fl9+/vZQpM5by1NyVDO3ajivH9ebEIZ01/Ucro2k54oOGhSSuFGSn88OTBzLjpvH8+JSB7NxTwRWPvs+Z989k5pL1YYcnMVSlq5vigpKExKUu7dpy6egiXr1uLHd9/VDKtu7m/Idmc/Ef32PNll1hhycxoPsk4oOShMS15CTjvOLuTL9hHD86eSBzlm7khHvf4mcvLGTr7r1hhyfNqFID13FBSUJahPTUZC4bU8QL14zmuIEFTJmxlOPueZNn563GXfdZJJqqyL0zakmET0lCWpTCvEx+NWkEz1x1LJ1z0rnm8Q+YNPldpi9at++LRVq+ykji18104dNHIC3S0G7t+PtVo7j19MGUbtjBJVNLOO7eN3n8veXsrawKOzxppOq78NXdFD4lCWmxkpOMi44pZMZNE/jVpOHkpKfwg6c/5oR7g24otSxaripXd1O8UJKQFi81OYkzhnfl71eNYspFxaSnJnPN4x9w6n0zeHn+Ws0N1QLta0koSYQuZknCzNLMbIqZLTOzbWb2gZmdFKWsmdntZrbKzLaY2RtmNjhWsUrLZGYcN7ATL14zmv/7xnB27KngO9PmMuGeN5j6zlK2l1eEHaLUU1Wkx1DdTeGLZUsiBVgBjAXaAT8BnjSzwjrKngtcAowGcoFZwCOxCVNauqSkYLqP6deP4/7/OIyOmW245bmFHP3zf3LbcwspXb8j7BDlAPYNXCtHhC5m03K4+w7glhqHnjezpcBIoLRW8V7ADHf/AsDMpgHXxSBMSSDJScbJQ7tw8tAufLhiM398ZykPzyrlD+8sZVz/fG74Wn8tgBSnqrubNC1H+EIbkzCzTkA/YEEdp/8M9DGzfmaWClwEvBzL+CSxDO/enl9NGsE735/Adcf346OVWzj1vhn8x+/fZfG6bWGHJ7VUD1yruyl8oSSJyBf/o8Cf3H1RHUXWAG8DnwK7CLqf6mxJmNnlZlZiZiVlZWXNFbIkiE456Xz3+L68fv04rj+hH5+s2cYpv36bH//9Y9Zt1Trc8UJXN8WPmCcJM0siGF/YA/x3lGI3A4cD3YF04FZgupll1C7o7pPdvdjdi/Pz85spakk07TJSufq4vrxy7RjOLe7On99bwZi7XudnLyxkw/bysMNr9XSfRPyIaZKwYK7nKUAn4Bx3jzb5zjDgCXdf6e4V7j4V6AAMik2k0lrkZ6fx87OGMv36cZxyaBemzFjKmLte555XP2XLLs0NFZbqq5vUkghfrFsSDwADgdPcfX9Tec4BzjWzTmaWZGYXAqnA57EIUlqfHh0zuPe84bx63RjGDSjgvumfM+oX07ntuYWs2Lgz7PBanS+n5VCSCFvMrm4ys57At4FyYG2NBWS+TTD+sBAY5O7LgTuBAuBDIJMgOZzj7ptjFa+0Tn0Ksvnt+Ydx1bitPPT2Fzw8q5SpM5dy3MBOXDmuNyN6aA31WFB3U/yI5SWwy4D9feJZNcruBq6KbCIxN+iQHP7fN4Zz04kDeHhWKU/MWcHZD8zkrBFduXpCXy2r2sw0cB0/NC2HyH50bpfOjScO4I3/Gcclo3rx8vy1TPy/t/j1Pz+jvKIy7PAS1pf3SYQciChJiNRHdnoqPzl1EG/cMI6vDerEva8t5uRfvc2z81Zr1tlmoLmb4oeShEgDFOSk85vzD2Pqtw7HHa55/ANG/WI6v/7nZ5Rt06WzTaVKA9dxI2ZjEiKJZFz/Asb0zefNxWVMnVnKva8t5jfTP+fUQ7tw9XEas2gsDVzHDyUJkYOUlGSMH1DA+AEFLCnbziOzlvFkyQqenbeaSUd055oJfSnISQ87zBZJA9fxQ0lCpAn0zs/iltMHc9X4Ptw3/TMem72cae8upyg/k7OGd+Xskd3o2r5t2GG2GNXDPBqTCJ/GJESaUH52GredMYR/fG8s/zOxPwXZadzz2mKOvXM6l0ydw0crN4cdYovwZXdTyIGIWhIizaEwL5OrxvfhqvF9WLFxJ0/NXcnDs0o5/TfvMGFAAZePKeLIXrmY/lKuk7qb4oeShEgz656bwfdO6Mdlo3sx9Z1S/jizlEmT36V/p2zOPqwrXx/ZjY5ZaWGHGVd0dVP8UGNOJEay04OZZ2d+fwK/OHsomWnJ3PHSIkbf9Tq3PreAL8q2hx1i3NDVTfFDLQmRGEtPTWbSET2YdEQPPlu3jd++/jnT3l3GH98p5ZShXbhhYv9Wfwmtupvih5KESIj6dsrm/yaN4EenDOKRWaX8fsZSXl6wlpE9O3Dm8K6cNqwL2empYYcZc9VXN6m7KXxKEiJxID87je99rT8XHl3In2aW8sqCtfzwbx9z63MLKC7sQN+CbE4a0pnDC3NbRReMpuWIH0oSInEkPzuNGyb25/qv9ePDFZt55sPVzF22iSfmrGDqzFK6tm/LqYd24ciiXEb2zKVd28RsZWjgOn4oSYjEITNjRI8O+9av2LmnglcXrOPpD1YxZcZSfvfWF5hB/07ZHF6YS3FhB8b2y6d9RpuQI28amgU2fihJiLQAGW1SOHNEV84c0ZVdeyr5cMVm5pRuZE7pRp5+fyWPvLuMjDbJnH1YV84Y3pWRPTq06G6p6paEupvCpyQh0sK0bZPM0b07cnTvjgBUVFbx8aotPDJrGX+Zu5Jp7y6nb0EWl40u4owRh5CWkhxyxA23pGwHoO6meGAeydiJoLi42EtKSsIOQyQ028sreHn+WqbMWMona7aSn53GxccUMnFwZ3rnZ7aIO7zfXFzGRX94j2N6d+SR/zpSiSIGzGyuuxfXeU5JQiTxuDszPl/P5Le+4O3P1gPQMbMNw7q3Z+LgThxdlEf33LZxmTR+9LeP+fsHq3j/pye0yFZQS7S/JKHuJpEEZGaM7pvP6L75LNuwg1lLNjCndBNzSjcyfdG/gOBKqqOKOvKN4u4c07tjXIxhuDtvfVbG0b3zlCDihJKESILr2TGTnh0zmXRED9ydRWu3UbJsE3NLN/Lm4jKem7eaXnmZnD2iKyN6dGBot3ahXVo764sNrNi4i8tHF4Xy/vLvlCREWhEzY2CXHAZ2yeHCo3qye28lL81fw7R3l3PPa4v3leuVl8mwbu0Y2bMDRxV1pE9BVrN2Tc1ftYXH3lvO4+8tp3NOOhOHdG6295KG0ZiEiACwZedePlq1mY9WbmHeis3MW7mZdVuDdbs7ZrYhIy2ZrLRUurRLp2fHDHrnZwVbQSb5WWkNSiJl28qZMmMpyzfuYNWmXcxbuYXkJOPCo3py/df6tcqpSMLUagauc3sO9BN++IewwxBJCO5OeUUVW3dXsG33XtyDm9z2VFaxe28lVTW+OpIsuKchKclom5pMkgWtFgPMiPwMksjOPZVsL6/AgLTUJFKSksjNbENeVhtSdfdcKJ78zjGtI0mY2Tbg00a+TDtgSyPL1XXuQMdqn6/er3k8D1hfj9j2J1b1299+tMexql9D61bX8TDq11yfXV3HG1q/lvS7WdexRK5ffb5berp7fp3v6O4JswElTfAakxtbrq5zBzpW+3z1fq0yLaZ++9vfz+OY1K+hdYuX+jXXZ9cU9WtJv5utrX71+W7Z36a23b97rgnK1XXuQMdqn38uyvHGilX99re/v3o3Vn1er6F1q+t4GPVrrs+uruOJVL+G/r4mWv0a9d2SaN1NJR6lXy0RqH4tWyLXL5HrBolfv/1JtJbE5LADaGaqX8uWyPVL5LpB4tcvqoRqSYiISNNKtJaEiIg0ISUJERGJqtUlCTMrNLMyM3sjstV9bXALZmbfNLOysONoambWycxmmtmbZjbdzLqEHVNTMrOjzWxWpH6Pm1lC3XZsZu3M7D0z225mQ8KOpymY2c/M7G0z+4uZZYQdT3NodUki4k13HxfZEurL1MySgK8DK8KOpRmsB45197HAw8B/hRxPU1sGTIjU7wvgjJDjaWo7gVOAv4QdSFOIJLre7j4a+AdwScghNYvWmiRGRbL/zy0eJ9RvnPMJ/hNWhR1IU3P3Snevrlc2sCDMeJqau692912R3QoS7DN0970J9kfZaOClyOOXgGNDjKXZxHWSMLP/NrMSMys3s6m1zuWa2d/MbIeZLTOz8+v5smuAPsAYoAA4u2mjrp/mqJuZJQPnAU80Q8gN0kyfHWY23MxmA/8NvN/EYddbc9Uv8vxewEnA800YcoM0Z/3iTSPq2oEvp7XYAuTGKOSYivepwlcDtwMTgba1zv0W2AN0AoYDL5jZPHdfYGadqbtJ+3V3XwuUA5jZ08BRwF+bJ/z9avK6RV7rSXevioMGUrN8du7+IXCkmZ0H/AD4TjPFfyDNUj8zywH+BFzo7nuaLfoDa67/e/HooOoKbCKY/4jIz40xiTbWGjsfSSw2gg9wao39TIIPrl+NY48Av6jHa+XUeHwH8J8JVLc7gVeBlwn+svl1gn12aTUeTwTuTbD6pQAvEIxLhFqv5qhfjfJTgSFh162xdQWGAo9FHl8OXB12HZpji+vupv3oB1S6++Iax+YBg+vx3LFmNtfM3ga6Ao81R4CNcNB1c/eb3P1r7n4i8Jm7X9NcQTZCYz67w8zsLTN7HbgWuLsZ4musxtTvm8CRwE8jV959ozkCbKTG1A8zexH4GvCQmV3c9OE1qf3W1d0/BpZFvksmAgm5TkG8dzdFk8W/T427hWAwc7/c/TmaflK5pnTQdavJ43eemcZ8drMIxpLiWWPq9wjBX6rxrFG/n+5+cpNH1HwOWFd3/0FMIwpBS21JbAdyah3LAbaFEEtTS+S6gerX0iV6/WpqTXWNqqUmicVAipn1rXFsGIlxSWQi1w1Uv5Yu0etXU2uqa1RxnSTMLMXM0oFkINnM0s0sxd13AE8Dt5lZppmNIrjxKN6b6vskct1A9UP1azFaU10PStgj5we42uAWwGttt0TO5QJ/B3YAy4Hzw45XdVP9VL+Wt7Wmuh7MpqnCRUQkqrjubhIRkXApSYiISFRKEiIiEpWShIiIRKUkISIiUSlJiIhIVEoSIiISlZKESBMys1vMbH7YcYg0Fd1MJy1OZPWwPHc/NexYajOzLIJ1LzaEHUs0ZubAue6eEGtNS/NSS0KkHsysTX3Kufv2MBKEmSVFlq8VaVJKEpJwzGyQmb1gZtvM7F9m9nhkWc3q84eb2atmtt7MtprZDDM7utZruJldZWZPm9kO4OfVXUlmNsnMlkRe/+9mllfjeV/pbjKzqWb2vJl918xWmdkmM/ujmWXUKJNpZg+b2XYzW2dmP4g8Z+p+6nhxpPzJkffbAww8UN3MrDTy8KlIHUtrnDstsiDXbjNbamY/q29ylMSlJCEJxcy6AG8B84EjgOMJFo951syqf9+zCWbyHB0p8yHwYs0v+4ibgRcJlqn8beRYIfAN4CyCFdZGAD87QFijgSGRWKqf+90a5+8BxkaOTyCYjnp0PaqbDvwY+DYwCFhWj7odHvl5GdClet/MJgKPAr8hWHntEoJ1039ejzgkkYU9w6A2bQ3dCNZIfj7KuduAf9Y61oFgZs8jojzHgDXABTWOOXBfrXK3ALuBdjWO/Qj4vFaZ+bViXQGk1Dj2EPCPyOMsglbApBrnM4FN1FhvuY6YL47EOPIA/1bR6vb1WuXeAn5S69iZBAvvWNifubbwNrUkJNGMBMZEumK2m9l2gi9pgN4AZlZgZr8zs8VmtoVgpbECoEet1yqp4/WXuXvNJS1XR567PwvdvSLKc3oDqcB71Sc9WMegPldIVRC0FPZpQN1qGwn8qNa/22MECavz/p8qiaylrnEtEk0S8AJwQx3n1kV+/gnoBFwHlALlwD+B2v3vO+p4jb219p0Dd9vu7zlW41hDlbt7Za1j9a1bbUnArcBTdZwrO4jYJEEoSUiieR84j+Av/tpfztWOBa5x9xcAzKwTQf98GD4nSCJHAEsj8WQQjGEsOYjXq0/d9hKswlbT+8AAd//8IN5TEpiShLRUOWY2vNaxzQQDzJcBT5jZnQR/BRcRJI7r3X0bwdrFF5jZbILulLsIxgVizt23m9kfgDvNbD3B+MGPCf6yP5jWRX3qVgocZ2ZvErRGNhGM5TxvZsuAJwm6soYQjOPceBBxSILQmIS0VKOBD2ptv3T31cAooAp4mWDR+t8SdLuUR557CcGA8Vzgz8AfCL44w3ID8DbwLPA68BHBeMjug3it+tTtemA8wVjNBwDu/gpwSuT4e5Ht+wRLdkorpjuuReKMmaURXM56t7vfE3Y80rqpu0kkZGY2AhhI8Nd7NnBT5OcTYcYlAkoSIvHie0B/vrysdYy7rww1IhHU3SQiIvuhgWsREYlKSUJERKJSkhARkaiUJEREJColCRERiUpJQkREovr/bDaS6ZZQmN4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "rates, losses = find_learning_rate(model, X_train_scaled, y_train, epochs=1, batch_size=batch_size)\n",
    "plot_lr_vs_loss(rates, losses)\n",
    "plt.axis([min(rates), max(rates), min(losses), (losses[0] + min(losses)) / 1.4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "\n",
    "model.add(keras.layers.AlphaDropout(rate=0.1))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.SGD(learning_rate=1e-2)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "352/352 [==============================] - 3s 6ms/step - loss: 2.2298 - accuracy: 0.2349 - val_loss: 1.7841 - val_accuracy: 0.3834\n",
      "Epoch 2/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.7928 - accuracy: 0.3689 - val_loss: 1.6806 - val_accuracy: 0.4086\n",
      "Epoch 3/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.6475 - accuracy: 0.4190 - val_loss: 1.6378 - val_accuracy: 0.4350\n",
      "Epoch 4/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.5428 - accuracy: 0.4543 - val_loss: 1.6266 - val_accuracy: 0.4390\n",
      "Epoch 5/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.4865 - accuracy: 0.4769 - val_loss: 1.6158 - val_accuracy: 0.4384\n",
      "Epoch 6/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.4339 - accuracy: 0.4866 - val_loss: 1.5850 - val_accuracy: 0.4412\n",
      "Epoch 7/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.4042 - accuracy: 0.5056 - val_loss: 1.6146 - val_accuracy: 0.4384\n",
      "Epoch 8/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.3437 - accuracy: 0.5229 - val_loss: 1.5299 - val_accuracy: 0.4846\n",
      "Epoch 9/15\n",
      "352/352 [==============================] - 2s 5ms/step - loss: 1.2721 - accuracy: 0.5459 - val_loss: 1.5145 - val_accuracy: 0.4874\n",
      "Epoch 10/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.1942 - accuracy: 0.5698 - val_loss: 1.4958 - val_accuracy: 0.5040\n",
      "Epoch 11/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.1211 - accuracy: 0.6033 - val_loss: 1.5406 - val_accuracy: 0.4984\n",
      "Epoch 12/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 1.0673 - accuracy: 0.6161 - val_loss: 1.5284 - val_accuracy: 0.5144\n",
      "Epoch 13/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 0.9927 - accuracy: 0.6435 - val_loss: 1.5449 - val_accuracy: 0.5140\n",
      "Epoch 14/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 0.9205 - accuracy: 0.6703 - val_loss: 1.5652 - val_accuracy: 0.5224\n",
      "Epoch 15/15\n",
      "352/352 [==============================] - 2s 6ms/step - loss: 0.8936 - accuracy: 0.6801 - val_loss: 1.5912 - val_accuracy: 0.5198\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 15\n",
    "onecycle = OneCycleScheduler(math.ceil(len(X_train_scaled) / batch_size) * n_epochs, max_rate=0.05)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs, batch_size=batch_size,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[onecycle])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1Cycle cho phép chúng ta huấn luyện mô hình chỉ trong 15 epoch, mỗi epoch chỉ mất 2 giây (nhờ kích thước Batch lớn hơn). Tốc độ này nhanh hơn nhiều lần so với mô hình nhanh nhất mà chúng ta đã huấn luyện cho đến nay. Hơn nữa, chúng ta đã cải thiện hiệu suất của mô hình (từ 47,6% lên 52,0%). Mô hình chuẩn hóa theo batch đạt hiệu suất tốt hơn một chút (54%), nhưng huấn luyện chậm hơn nhiều."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "nav_menu": {
   "height": "360px",
   "width": "416px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "c8d79a7fd254ccfb6bc705f40d8247ad32d2fecae19702dfa351d7ab4f046c4d"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}