lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training RL to do Cartpole Balancing\n",
    "\n",
    "This notebooks is part of [AI for Beginners Curriculum](http://aka.ms/ai-beginners). It has been inspired by [this blog post](https://medium.com/swlh/policy-gradient-reinforcement-learning-with-keras-57ca6ed32555), [official TensorFlow documentation](https://www.tensorflow.org/tutorials/reinforcement_learning/actor_critic) and [this Keras RL example](https://keras.io/examples/rl/actor_critic_cartpole/).\n",
    "\n",
    "In this example, we will use RL to train a model to balance a pole on a cart that can move left and right on horizontal scale. We will use [OpenAI Gym](https://www.gymlibrary.ml/) environment to simulate the pole.\n",
    "\n",
    "> **Note**: You can run this lesson's code locally (eg. from Visual Studio Code), in which case the simulation will open in a new window. When running the code online, you may need to make some tweaks to the code, as described [here](https://towardsdatascience.com/rendering-openai-gym-envs-on-binder-and-google-colab-536f99391cc7).\n",
    "\n",
    "We will start by making sure Gym is installed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Requirement already satisfied: gym in /home/leo/.local/lib/python3.10/site-packages (0.25.0)\n",
      "Requirement already satisfied: pygame in /home/leo/.local/lib/python3.10/site-packages (2.1.2)\n",
      "Requirement already satisfied: gym-notices>=0.0.4 in /home/leo/.local/lib/python3.10/site-packages (from gym) (0.0.7)\n",
      "Requirement already satisfied: cloudpickle>=1.2.0 in /home/leo/.local/lib/python3.10/site-packages (from gym) (2.1.0)\n",
      "Requirement already satisfied: numpy>=1.18.0 in /usr/lib/python3/dist-packages (from gym) (1.21.5)\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install gym pygame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create the CartPole environment and see how to operate on it. An environment has the following properties:\n",
    "\n",
    "* **Action space** is the set of possible actions that we can perform at each step of the simulation\n",
    "* **Observation space** is the space of observations that we can make"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Action space: Discrete(2)\n",
      "Observation space: Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/leo/.local/lib/python3.10/site-packages/gym/core.py:329: DeprecationWarning: \u001b[33mWARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
      "  deprecation(\n",
      "/home/leo/.local/lib/python3.10/site-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: \u001b[33mWARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
      "  deprecation(\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "import pygame\n",
    "import tqdm\n",
    "\n",
    "env = gym.make(\"CartPole-v1\")\n",
    "\n",
    "print(f\"Action space: {env.action_space}\")\n",
    "print(f\"Observation space: {env.observation_space}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how the simulation works. The following loop runs the simulation, until `env.step` does not return the termination flag `done`. We will randomly chose actions using `env.action_space.sample()`, which means the experiment will probably fail very fast (CartPole environment terminates when the speed of CartPole, its position or angle are outside certain limits).\n",
    "\n",
    "> Simulation will open in the new window. You can run the code several times and see how it behaves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/leo/.local/lib/python3.10/site-packages/gym/core.py:57: DeprecationWarning: \u001b[33mWARN: You are calling render method, but you didn't specified the argument render_mode at environment initialization. To maintain backward compatibility, the environment will render in human mode.\n",
      "If you want to render in human mode, initialize the environment in this way: gym.make('EnvName', render_mode='human') and don't call the render method.\n",
      "See here for more information: https://www.gymlibrary.ml/content/api/\u001b[0m\n",
      "  deprecation(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.00425272 -0.19994313  0.00917169  0.34113726] -> 1.0\n",
      "[ 0.00025386 -0.00495286  0.01599443  0.05136059] -> 1.0\n",
      "[ 1.5480528e-04  1.8993615e-01  1.7021643e-02 -2.3623335e-01] -> 1.0\n",
      "[ 0.00395353  0.38481084  0.01229698 -0.5234989 ] -> 1.0\n",
      "[ 0.01164974  0.18951797  0.001827   -0.22696657] -> 1.0\n",
      "[ 0.0154401   0.38461378 -0.00271233 -0.51907265] -> 1.0\n",
      "[ 0.02313238  0.5797738  -0.01309379 -0.812609  ] -> 1.0\n",
      "[ 0.03472786  0.38483363 -0.02934597 -0.5240733 ] -> 1.0\n",
      "[ 0.04242453  0.580356   -0.03982743 -0.8258571 ] -> 1.0\n",
      "[ 0.05403165  0.38580072 -0.05634458 -0.54596174] -> 1.0\n",
      "[ 0.06174766  0.19151384 -0.06726381 -0.27155042] -> 1.0\n",
      "[ 0.06557794 -0.00258703 -0.07269482 -0.00081817] -> 1.0\n",
      "[ 0.0655262  -0.19659522 -0.07271118  0.26807207] -> 1.0\n",
      "[ 0.0615943  -0.00051497 -0.06734974 -0.04662942] -> 1.0\n",
      "[ 0.061584    0.19550486 -0.06828233 -0.3597784 ] -> 1.0\n",
      "[ 0.06549409  0.00141663 -0.0754779  -0.08938391] -> 1.0\n",
      "[ 0.06552242 -0.19254686 -0.07726558  0.17856352] -> 1.0\n",
      "[ 0.06167149  0.00359088 -0.0736943  -0.1374588 ] -> 1.0\n",
      "[ 0.0617433   0.19968675 -0.07644348 -0.45245075] -> 1.0\n",
      "[ 0.06573704  0.3958018  -0.0854925  -0.7682167 ] -> 1.0\n",
      "[ 0.07365308  0.20195423 -0.10085683 -0.50361156] -> 1.0\n",
      "[ 0.07769216  0.0083876  -0.11092906 -0.24433874] -> 1.0\n",
      "[ 0.07785992 -0.18498953 -0.11581583  0.01139782] -> 1.0\n",
      "[ 0.07416012  0.01158649 -0.11558788 -0.31546465] -> 1.0\n",
      "[ 0.07439185  0.20814891 -0.12189718 -0.64224803] -> 1.0\n",
      "[ 0.07855483  0.01491799 -0.13474214 -0.3903015 ] -> 1.0\n",
      "[ 0.07885319 -0.17806001 -0.14254816 -0.14295265] -> 1.0\n",
      "[ 0.07529199  0.01878517 -0.14540721 -0.47699296] -> 1.0\n",
      "[ 0.07566769 -0.17401667 -0.15494707 -0.23344138] -> 1.0\n",
      "[ 0.07218736  0.0229406  -0.1596159  -0.57071024] -> 1.0\n",
      "[ 0.07264617  0.21989843 -0.1710301  -0.9091196 ] -> 1.0\n",
      "[ 0.07704414  0.02745241 -0.1892125  -0.6747003 ] -> 1.0\n",
      "[ 0.07759319 -0.16460665 -0.20270652 -0.4470505 ] -> 1.0\n",
      "[ 0.07430106 -0.35637102 -0.21164753 -0.22448184] -> 1.0\n",
      "Total reward: 34.0\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "\n",
    "done = False\n",
    "total_reward = 0\n",
    "while not done:\n",
    "   env.render()\n",
    "   obs, rew, done, info = env.step(env.action_space.sample())\n",
    "   total_reward += rew\n",
    "   print(f\"{obs} -> {rew}\")\n",
    "print(f\"Total reward: {total_reward}\")\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Youn can notice that observations contain 4 numbers. They are:\n",
    "- Position of cart\n",
    "- Velocity of cart\n",
    "- Angle of pole\n",
    "- Rotation rate of pole\n",
    "\n",
    "`rew` is the reward we receive at each step. You can see that in CartPole environment you are rewarded 1 point for each simulation step, and the goal is to maximize total reward, i.e. the time CartPole is able to balance without falling.\n",
    "\n",
    "During reinforcement learning, our goal is to train a **policy** $\\pi$, that for each state $s$ will tell us which action $a$ to take, so essentially $a = \\pi(s)$.\n",
    "\n",
    "If you want probabilistic solution, you can think of policy as returning a set of probabilities for each action, i.e. $\\pi(a|s)$ would mean a probability that we should take action $a$ at state $s$.\n",
    "\n",
    "## Policy Gradient Method\n",
    "\n",
    "In simplest RL algorithm, called **Policy Gradient**, we will train a neural network to predict the next action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/tensorflow/__init__.py:29: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives\n",
      "  import distutils as _distutils\n",
      "2022-07-24 16:50:47.597258: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:47.597280: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "/usr/local/lib/python3.10/dist-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  import imp\n",
      "2022-07-24 16:50:49.838826: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-07-24 16:50:49.839078: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839143: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839194: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839245: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839295: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839345: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839392: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839441: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory\n",
      "2022-07-24 16:50:49.839449: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.\n",
      "Skipping registering GPU devices...\n",
      "2022-07-24 16:50:49.839649: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "num_inputs = 4\n",
    "num_actions = 2\n",
    "\n",
    "model = keras.Sequential([\n",
    "    keras.layers.Dense(128, activation=\"relu\",input_shape=(num_inputs,)),\n",
    "    keras.layers.Dense(num_actions, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.Adam(learning_rate=0.01))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will train the network by running many experiments, and updating our network after each run. Let's define a function that will run the experiment and return the results (so-called **trace**) - all states, actions (and their recommended probabilities), and rewards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_episode(max_steps_per_episode = 10000,render=False):    \n",
    "    states, actions, probs, rewards = [],[],[],[]\n",
    "    state = env.reset()\n",
    "    for _ in range(max_steps_per_episode):\n",
    "        if render:\n",
    "            env.render()\n",
    "        action_probs = model(np.expand_dims(state,0))[0]\n",
    "        action = np.random.choice(num_actions, p=np.squeeze(action_probs))\n",
    "        nstate, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            break\n",
    "        states.append(state)\n",
    "        actions.append(action)\n",
    "        probs.append(action_probs)\n",
    "        rewards.append(reward)\n",
    "        state = nstate\n",
    "    return np.vstack(states), np.vstack(actions), np.vstack(probs), np.vstack(rewards)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can run one episode with untrained network and observe that total reward (AKA length of episode) is very low:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total reward: 27.0\n"
     ]
    }
   ],
   "source": [
    "s,a,p,r = run_episode()\n",
    "print(f\"Total reward: {np.sum(r)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the tricky aspects of policy gradient algorithm is to use **discounted rewards**. The idea is that we compute the vector of total rewards at each step of the game, and during this process we discount the early rewards using some coefficient $gamma$. We also normalize the resulting vector, because we will use it as weight to affect our training: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "eps = 0.0001\n",
    "\n",
    "def discounted_rewards(rewards,gamma=0.99,normalize=True):\n",
    "    ret = []\n",
    "    s = 0\n",
    "    for r in rewards[::-1]:\n",
    "        s = r + gamma * s\n",
    "        ret.insert(0, s)\n",
    "    if normalize:\n",
    "        ret = (ret-np.mean(ret))/(np.std(ret)+eps)\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's do the actual training! We will run 300 episodes, and at each episode we will do the following:\n",
    "\n",
    "1. Run the experiment and collect the trace\n",
    "1. Calculate the difference (`gradients`) between the actions taken, and by predicted probabilities. The less the difference is, the more we are sure that we have taken the right action.\n",
    "1. Calculate discounted rewards and multiply gradients by discounted rewards - that will make sure that steps with higher rewards will make more effect on the final result than lower-rewarded ones\n",
    "1. Expected target actions for our neural network would be partly taken from the predicted probabilities during the run, and partly from calculated gradients. We will use `alpha` parameter to determine to which extent gradients and rewards are taken into account - this is called *learning rate* of reinforcement algorithm.\n",
    "1. Finally, we train our network on states and expected actions, and repeat the process "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 -> 29.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-07-24 16:50:51.475024: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 -> 135.0\n",
      "200 -> 484.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-07-24 16:51:35.910774: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-07-24 16:51:37.151017: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-07-24 16:51:39.284311: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-07-24 16:51:42.235074: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-07-24 16:51:44.691458: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n",
      "2022-07-24 16:51:48.381946: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f40201e7b20>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABUUElEQVR4nO29d5gk1Xnv/z0VOk6OO7uzOcDuEpcVUWCiQEIGhYsuSsYWEvfK2ALLsi0s+1qyjS3bkvzzlSVb2JKMdLEAJYNkBCIIhERcwsImNofZmZ20k6dj1fn9UXWqT1VXh5npng77fp5nn+2uru4+1d3zrbe+73vewzjnIAiCIOoLpdIDIAiCIEoPiTtBEEQdQuJOEARRh5C4EwRB1CEk7gRBEHWIVukBAEBHRwdftWpVpYdBEARRU7zyyisjnPNOv8eqQtxXrVqFbdu2VXoYBEEQNQVj7Eiux8iWIQiCqENI3AmCIOoQEneCIIg6hMSdIAiiDiFxJwiCqEOKEnfG2GHG2JuMsdcZY9vsbW2MsccZY/vs/1ul/e9ijO1njL3FGLu2XIMnCIIg/JlL5H4F5/wczvlW+/5nATzJOV8P4En7PhhjmwDcDGAzgOsAfJ0xppZwzARBEEQBFlLnfiOAy+3b9wJ4GsCf2Nvv55wnABxijO0HcD6A5xfwXgRB5OC5AyN44cAorji9C+eusC6gXz82jqd2D+LCNe24eF0H9g9NYWgqgZ7mMPrGZrGqPYoDw9NY09GAH77aB845VnVE8b4tvRiaiuN7Lx6DYZq+79cU1vHB81fgO88fQSyZBgCEAxp+++JV+Okb/bjhnKUIaip+9Gofrt28BE/sHsQVp3ehKaRnvdaTuwexaWkTeprDzrZn9w1jaUsYazsb8Oy+Ybx86CSu2bQEZ/Y2Y/uxcTAGnNXbgl39k4ilDAQ1BT/feQJbVrbi8tO6Sva5pgwTP371OP7Heb1QFIa0YeJbvz6ElMFxy8Wr8P1txzA2k4SuKvjIhSvxyI4BDE7EneczxvDus3rw2tFx9I3H8P4ty7CyPVqy8RWiWHHnAH7OGOMAvsE5vwdAN+d8AAA45wOMMfGpLgPwgvTcPnubC8bYbQBuA4AVK1bMc/gEQdz937uxs38S2/smcO/HzgcA/N8n9+GpPUN4YvcQHrnjUlz9lV+6ntMc1jERS+HWt6/GN391yNl+3RlL8INX+vCPT+wFY9nvJZZ/mE0a+Mrje12PaQrD3Y/sRnNYx8aeJnz6we343LuSuPuR3fjb952JD56f/Xf+u/e9ik9cugafufY0Z9tHv/kSAODwF6/HX/90N94anMJbg1P4xke34sav/dp57Es/fwsj0wn0tobxyJsnsKYjWlJxf+7AKP74h29gXXcDtqxoxcuHx/A3j+wBACiM4e8e3ePsO51M4xvPHAQA53PjHPjqU/tg2p9ZLJnG567fVLLxFaJYcb+Ec95vC/jjjLE9efb1+Ukga0UQ+wRxDwBs3bqVVgwhiHkyk7Ci55SRibTF7Xja8H3ORCwFANg/NI3VHVHcefV63HH/6+gfj+H4WAytER2v/Z93ZD3v5cMncdO/Po/DIzMAgKc/czkm4ync8M+/xlTces1YysBs0nrf0ZkkACCZzr4K4JwjkTaRNPyvEDjnznGkjGyJmE6kEU8ZiKesfXK9znyZtD+juH0sh+xjBoDR6QQA4BsfPQ933P8anj8wCgD47q3n49L1VjeAExNx/MXDO3DJug58+ed7fT+DclKU584577f/HwLwY1g2yyBjrAcA7P+H7N37ACyXnt4LoL9UAyYIws2MLT5pMyOAhn1bCEpAc/+pa4oVg+3sn0RnQxBLWyxb5Ph4HAMTcZdNItMctqyVoydnnfuq/VoJ+70SadN5X3HikcfmHaPh8xhgianJc++TSBlISu+V9jkBLIRZ23ISJ41DI9POY+LkGNJV9LZGsOP4BACgtzXi7LOkOYRvfHQrfuuiVdBVhlSO4ywXBcWdMRZljDWK2wDeAWAHgIcB3GLvdguAh+zbDwO4mTEWZIytBrAewEulHjhBEBYxIe5S5CrEVAjuao/X2xIJAABGphPobApimRD3sRj6x2NY2hLyfS/hmwtxbwxp0BTF9V7JtImkYY1JiLuff58uIO6vHBmDYYu76bMcaDxlImXwjLiXWDxnEtYxiNc/NDKLzsYgAEncNQW9rWHHesn1uWmK4vp+FoNibJluAD9mlpGkAfhPzvmjjLGXATzIGLsVwFEANwEA53wnY+xBALsApAHczjn3vzYkCGJBcM6dCFMWSdMTubdG3cnMprCGEdta6GoMoqsxCFVh6B+3xP1tq9p8309E7kNTCTQENWiq4kTu8ZT1Z25F7tb7TzmWUbbw5hL3oKYgkTbx+rFxiHOC3wkgljJctk46RwJ4vsx4xn5oZBqnL2nE8FTCEfegrqK31ToxdjcFEdT8CwM1lZX8yqIQBcWdc34QwNk+20cBXJXjOXcDuHvBoyMIIi+JtOlEjXLkmvaIu1dYGoKZP/3OxiA0VcGSphD2Dk5hMp5GT44INKQrCKgKkobpCL3msWWSkuBOx7NPPM4Y7X0MT1QuovSZRNp5njcqT6QNxFMGUkb5bBlhd6UME4bJcfTkLK7a2I1n941ItoyCZS2WFSNbMl40hZX8yqIQVdHylyCIufP4rkGokrHqity5sGVsa8FjCShSKUxXoyXky1rDeOXIGABgaQ7PnTGGprCOkekEmmxxz47cjYznnsztuYuI2PQ85kT0PCP83n2m42lb3DNJ11JH7o7nnjZxfCyGlMGxrrMBusqcZGtQy0Tu4n8/NFUp+fgKQeJOEDXKJ77jXgNBrpYRQm9yK0JOpk1cvbEbl6xrxxd+sssRLsCyZQBgWUsYLx06CQDoafaP3AGg2bZ0msOWfOiqj+du357O47n7JVQNkzvllqbJHVH3RvdWpYwJk/OMLVPiyF2MPWmYGJyy6td7WkIIaqorci9K3BXma02VE+otQxB1gKqwLJEUJA0TKcNEUFccT1gkCwE4SUJZnET1jB/Cjmn2RO7iKiEhJVSFLeMnvOJkZLjsJPcJKlfkPhFLIWmYSJvcuWJImxzcJ/E6X2alhKpIWkcCKkK64lg2IU3Fmo4GNAQ1nLmsJedr6Wp1JlQJgqhymkKabykkYIlTyuAIqIpTEimiUiATuX/0opXYPTCFkzMJLMkbubvFXXNsmdyRe95SSO4/boNz330AYHQ66dyelU5UaZNDV/2m2swdYSmlDBMx+wQS0lVX0jSoK4gENGz7s6sR1HLHyppKnjtBEPOgMaS7JsnIIinqznWVOcInKkF0laHVLovsagzh32/ZikIIr12URaqqN3LPeO5iMpNvQtXMjtxl68KULBpv0DtsV/oA1uxQ+bj1EnWykieHiauDsK4iqGdEXAh9qMCb6oriss0WA7JlCKIOaAp7InfujdxN6KpV6QJYEe4Zy5rwpZvOhqLMLdLNFbknpMg9kXYLmV8y0UmoFhO5e54/PJURd84z1lApBVScmGRbJhzIRO66ypz3LUQlSiFJ3AmiDmgM6i4BNCR7QtSCBzTFNVP1tO4m3HhOVtungjjiHhHi7k6o+rUU8BM2v4Rq2vD33L2R/4gUuQNAxI6cSymgM84M1YyvH9Isz13cLha1AqWQJO4EUQc0+njuYVvwROQeUBWnsgUAAtr8vOncnrs8ickbuRebUHWXc4pEqtgsTlgjkucOWBE1AKRKWG4oz1CN2VclVuRufYayPVMIvQKlkCTuBFEHNIV1V9RqmByRgJVSEx64rrojd1no5/pe8v+KwsBYRtzFyUTGz3PPVQop3/ZG7mLMI1PuyD1qT8rK1cpgPsieu0ioBjXF8ddzzUb1Q1PIliEIYh40hfQsYYzY0WwsZcDkljC6Ivd5intHg5WAbY8GnG2awiRbxigycs9MVspsc9symYSqW+S9tky4xLZM2sjkDURCNayrYIw5dsxcI3dKqBIEMScUZtVfu2rEOXciTGEvBDTFJeh6ntK9fFy2vhP/8uEtOHNZs7NNlcQ96WPL+DcOs7aZOSJ3V+WMp4FYlucubJkSCaioYwcyCVVh/QTn4blXohSSxJ0gapxIQIOmMpg8I5Ry5D6dsGZT6ioriS2jqQreeWYPmNTCQCRVAf+EqizUYzNJpO0JSIC7kibtEvfs0k7x+NhsyvX6kRLbMvIM3qRty4irg5BT/lj852d1hSRxJwhiDoQDqpPUlDsthh1xz0Tu8gSffJNu5oomva5fKaQjzoaJy7/0NH7wSp8jdnJQLwugN4cg1717EdUypZriL8/gtRKqhhOxO//P1XOnhCpBEHMhGlCh2VG47E2HHVtGTFjyJlRLM5MTyFTMAPmrZRJpExOxFAYnE75dIYUAWr1Y3DaTd5aqjLhKKZWAzkgzeFOGiYQcuevziNyrseUvQRDVTTigSZG7CUB12zJxSdxLkFD1Qy0g7sJzFyKfkmwZv1LIoKa4rB3D5Hktl0iwxJF7UhZ3qzlZ2KmSmXvkTglVgiCKQm6QFQmojrimnQoUjrBdCin6u3gnMc03oeqH23M3cnruhkvcferc7f0CWkYMVYXBlMS9JeJeeASAU/ZZKs/dmZGqq1kJ1XlF7jSJiSCIYpB1IiLZMmnJmxaRpiPuKnMlUeebUPVDjtz9q2XcydOkYWZORD7VMpa4W7c1hblsmWU+HSszpZCliY7Fe0eDmp1QNaX69nl47iolVAmCKAJZEHtbw44tI0/8EbaM7LnLgl7ShGqRnrvhY8vIvWXEDNOgpiIlFvdWFZhmphLIT9yjwpYpUXQsTkKRgOqqcwespfWAuUXu1gLZ7s/k+QOj2D80neMZC4fEnSBqECGIn7pyHT5/w+aMLWOajoBmqmVkcc+IcCkjd82TnJXrxAHJc7ej11SaZxKqcuQu2TLC2glo1tR9cTJY5rMohrCgShW5i3FGApIt45RC2nXuc2g/qSoMnLuP9a4fvYGvP72/JOP1g8SdIGoQIRLRoIagJpVCGhlvWlOsFr9T8YznzhhzEqmlTai6X0vU1gu8FowroepTLROUPHddVWBKwugXuTuNw0oUuYv3diL3tOFE6kGPPVMMumObZU4+8VSmlXA5IHEniBpERO5iLVTZcxdiqSqWkMu2DAAnqVrahKo7chcnFIF3wlIuzz0tee5OszDN3c5XXtxb4JRClqr9gHTyFJF7KOCO3INziNw1T8Lbeg8TyXT5fHgSd4KoQUQAKHqxy5678KZVhSGoq1JC1RZ120IpZZ27t6/5tEfcvTNMc5VCOglVn8SvSHL69VAXM1RLVecu7J2wriJhT8ryeu5zidydk68k7kmfmbylhMSdyMlXHt+L2+97tdLDIHxwonNb5+TFKtKmO3J36tw1Ieqi2qN8kXva5JC6Ezii63juRsZzl3vLCAGXo2LdtnxEklZVGM5e3oLTlzQ67xstceSekjx3cRWyEM9dnEhTnlYLqTSJO1EB9p6Ywu6ByUoPg/DBMN1RrF/krjCrl4xYhi7gtWXKmFAFgGggY5/4ee4pH89dJF7dDc6s15bF/aHbL8Gjd17mHEu4xDNUM0lpzYmuM43D5hG5K9mRe8rIbo1cSmiGKpETs8CUb6JyOJ67ku25p6WEalBTnH4suieRGijTJCZBQ1DLWiBb/J9Mm46Q+3WClNvpivEmDSv5KNsy1jEZziSmUs1QTUmlkAIRqTfYZZdRH+8/F5p0ZfXxe1/G29d1ODNfywWJO5ETk/NFn3hBFIcTubPsyF08pijuLpDidjkidz8fvCGkAZPu8bqqZfJMYgr6eO4i+ahKfo84lkxCtbSlkFEfcV/b2YCvfOBsXHl6V9GvJ65sDJPjlSNjTlLYOx+glJC4Ezkp1M+DqByygAOQ2g+YTlSvKf4tfr0RfCnweu6Au6pFiK6wTVIG902oytUyAnE7KbUjcB5T3f53qUohxXhl71947owxvG9L75xeT5NKIRNp05kHUE5bhjx3IicmL90fC1FahNUioliRsJNtGZFQFYh9vBF8KfB6/4Bb3DMtf+XI3U6oSj8xR1R9TkqptI+42wuQyMdfClKmtcC4PA7ZopkrumPLcCTTptMvnqpliIpgcu67gg5ReQzHc7fui0lE3oSqHHlm7Bh31UwpELaD7EOLlgBApi2AEPmklFD1W6zDrzVxKkfkHtAUKWFZulJI1XPl4zd5qlg0x1qyqplEv/gU1bkTlcDknCL3KsWQBBxwJ+zkhKorclc8tkwZEqrruhqcbQ1BewFt5l/nLloNyPGD47lLTbkcYfQRd12zZuFqUmRcClIGh664e/H4tT0oFnHym7XtGBG5ky1DVATy3KsXU5qFKv/vTaiKRaw1hTn+fNATwZcCp948qOGs3mYAmaqSSECzF7vOXAmm0typSHG1HzBMMOYurRQnqISfLWNH7orCXCeRhWKYHJrqtbXmL5fixCpmC4vIvZwJ1aJHyxhTGWOvMcZ+at9vY4w9zhjbZ//fKu17F2NsP2PsLcbYteUYOFF+5H4eRHXhrZaRPWc5obq+u8HZLnASqz7li/NF9twv39AJADgxGQeQqQ83pHxAyjCzKmjEODWFuStiJEsDyK6WEVcgmqpkdV6cL2nThKYqTouGpc2hBb2e+HzEIiDV5rnfAWC3dP+zAJ7knK8H8KR9H4yxTQBuBrAZwHUAvs4Ym38mgqgYJkXuVUt2tUzGc09Lj63vbsx6rlhLVfGpcJkvItLWFIZbL12Dqzd24yMXrgQgL4HHXZ67XGZrSraNqrjH5u0t461zz5ysSreUnWXLZCL3pQvw24HMyVcsAlI11TKMsV4A1wP4d2nzjQDutW/fC+A90vb7OecJzvkhAPsBnF+S0RKLivDcOU1kqjocW8bHczelqH5Dd0PWc7193UuBE7mrDM1hHf9+y1a8fV0H/uja03Dd5iUA7EoeqVpGFjZnkpPtdWdPVHLPUBU0BDVnJqymKiULRtKGFbmLCLtngeIu8gZC1MWxmLx0SeCs9yxyv/8PwB8DkMOAbs75AABwzgcYY6KifxmAF6T9+uxtRI0hgiCTZ3qYENVBVvsBaZKMnFBd0pRtJ1iRe2nFXSRU5da/jDHcfsU6fOtXh6yxGdxV5y4LsThZGaYJVfW3Zfwi9z++7nRHgL2Lai+ElO25T8as1sUr2yILej1x8p1NpLMeSxkcc1jUqfj3LLQDY+zdAIY4568wxi4v4jX9ZCDrdMoYuw3AbQCwYsWKIl6WWGwyl8omVIWctWpC6KLQwMxiHVIppMLAWPaf403n9WJTT1NJx6Mp7isI12NOPsDdCdK7ADZgi6qiuG2ZPJH76o6o631K1vLXMKEpDB9423L0T8Txu1esXdDr6Z7IXSaZNp28RCkpJnK/BMANjLF3AQgBaGKM/T8Ag4yxHjtq7wEwZO/fB2C59PxeAP3eF+Wc3wPgHgDYunUrXfdXIZloir6easNbLaNJnrshJVQB4I+uPQ394zHnueeuaMW5K1pRSlTVXbXjekw68ci/pZgkdGLMhiESqpnnC2FM+ETuMppSuoSqYZ9kIgENf/qujQt+PXGCm/GJ3MuVVC14bcY5v4tz3ss5XwUrUfoU5/wjAB4GcIu92y0AHrJvPwzgZsZYkDG2GsB6AC+VfORE2fGrZiCqA2+1jF/LXxH93n7FOtz93jPLOh5xIvErr9QkcZcj61lZ3IUXb1qTh7y17EBmwo/f1QFgCWipfqspg5elVFRUy7jfq7Keux9fBPAgY+xWAEcB3AQAnPOdjLEHAewCkAZwO+e8fGtJEWVD5FFJ3KsP0yPguuS5mx7hXwxUx3P3E3f7qsLjs8dSPpG7Pe1ftmW8XSGVHMellbBaRpRClgrxWrMJf1umHMxJ3DnnTwN42r49CuCqHPvdDeDuBY6NqDDiD45mqVYfRo5JTLL1kcu+KAeaxx5yPebjuQNuW8ZbCimfmDK9ZfIfl64qpUuo2vZQqRC9ZWZ91kwtV+ROM1SJnJDnXr1ktx/ILAZREXEvwnO3Zs9mhGxWsiicQMIwoav+CVW/ahnv+5SyK2Rpe++IyL2KPHfi1EWOpojqwptQFXpnmGZWQnUxyFstI/V9kX9L8VR2tYzhG7lbtwsmVFWldOJuj6NUiJPftJ+4l8mWIXEnciJXMBDVhQiAhQgyZjXQkm2ZUs5ALYTw3P2W25MreVwLRPuVQopqGU9bX8C/FFLGmqFaOlumlAlV0eph1qcUslTNzryQuBM5EQJCS+1VH+I7kXOLqsJcjcMWM6GqOd5/tqSoOTx3IDN+OXLXctgyfr1lXGMoYZ27YZq++YP5Ik5Is4tYLUPiTuREnjVIVBemj6+uKYpr5udieu5+i3VkxuXvuQOZDpWmk7w3cydUbRHMJbqaopRsgey0wX2vQuaL7tS5L161DIk7kROTqmWqFm+1DCDqvM2KiLteREL1yT1DGJtNuR4TfdtF8Jq27RA5lyleW4hgroBaU0uXUE2ZpU2oMmZZTTGfaplyJVRpDVUiJ/IfHFFdeKtlALvO22eG6mLgeO4+7ylE8l+ePpD1mIjcRcSdNjkiiuI6Lm9vmXyRe6n863SJSyEB67PxqzyjyJ1YdDiVQlYt3moZcVsuhVzMhKpTLeMT7ea7ggjqti0j8jtmdkJVdxbIFseVewylTKiW0pYBci/2UY0zVIk6hyYxVS+Gp1oGsCLXB7YdA7ZlP1ZuivHc/RBRufitpeyGXf4JVcvSyJdQLVnL3xInVAFrTVm/UkhKqBKLDvWWqV4y7Qcy27yRplriyDMfmWqZ3J67HxnPXa6W8a9z91tDVSakq5iMp0uy/kCpE6oA0BoJ+G4nW4ZYdKi3TPUibBnZm055RKJaIvd8iUlhy8gLaKuexTrkZfYUBt82xgBw7ooWjEwncGB4en4HIZEucUIVANqiOcSd6tyJxYYi9+rFr1qmfyLu2mdxq2VyNw7zbgvrmd7lq9qtfuwZcTehK8x10sqUQvK8Vsll6621W5/ZOzKfQ3BRjoRqq0fcQ7o7UVxqSNyJnMi1x0R1YfpUy3ipRJ27X7TrFUl5YYqzepsBSHMqDO7T8leK3PMo1vK2CNZ0RPHLvcPzOwgbbi8vWcqukADQ5rFlxPKAZMsQiw41Dqteiqllr8wM1blF7qctsVbuNEyOlGFiKp5GUFf869yNwknO81a2YtfA5LyOQSAKCPQyRe7i8wjpKhRGkTtRAYSmU7VM9SFs2nwCvri9ZfJVy7hlRtgRQEboDc7x852DmEqkceXpXW5bRnF77vloieiYjmdXpMwFMa+j1AnptogOIHNi1lUGXVWoKySx+JDnXr34VctcvbErx97lJ1/LX2+wLdsyTjtgg+O7LxxGb2sYv7Ghy9XtUhbZQlZTY0hHLGUsKBoWS/XpJS6F9HrumqogoCpkyxCLi1xORuJeffglVP/9lrfhb99X3uX0ctEeDYIxoKMxmPVYV2MI//nxC3DB6jYAQEhTcdN5vbjv4xc4EbrBObYfm8A1m7qhSglVb58Zv8ZkMg1By8feOziFFw+OzutYRORe6lJIb7WMrioIaKVbYMQLiTvhiyzoJO7Vh18pJABEpKh4MVnVEcXzn70KW3IsvH3xug6nzltTGf7hprNxyboO5+QUSxqIpQx0NFgnB0VqZeyehZt/HI0hS9z/4bG38Mn7Xs277/HxGL765L6sunhRQFDqhKq3zl3YMmKFqVJD4k74Irf5Jc+9+shVLRMJVG7S+ZLmUN7HRU27XFEjPPqR6QQAyzMHpOUDGfP0z8kvWULcD43MYHw2mXdC06cfeB1ffnwv9pyYcm0XkXupE6p+kbuuMfLcicVF/puglr/Vh9N+wCNA0QpF7sUQsmejymMWSd/RmSSATHQr9N9bFlnIBm8MWSeH42MxmNx/cQyBuPqZiLk7VWZsmfJG7prCLM+dessQi4lsxVDkXn0Yji3j3h4JVu+ftKiSkStqhJ8+KiL3sCXOIlpXmPsYi43cxW92OpFGNMdn0mSfCCY94i4SqqWexBT2nHgDmoIfffIS6Fp5qpoocid8MSmhWtWYJvedil8pz70YQnp25C5uj05bkXtLxF0LrirM5bsX0tsGj5BP5SmLFCeC8ZyRe+lF98/fvQnf/98XQVMYNIWhOaKXzUqr3tM8UVFkJ4bEvfowuP8CztUs7kEfcRe2zIiwZaLuyF323g0UXrRa2DICvy6M3n1P2u8tKNQ3fiHc+vbVAGy/vcS2jxcSd8IXityrGyty9xP36v2TFraMvPiLprhtmVZP5O7YMwoAo3AppIjGBX4TmoYm43h234h01ZBwPZ6WJhmVi4CmOG0VykX1/hKIikLVMtWNYdZe5B7ytPcFMuI9Op1ESFeyrBsngnci+fzvEdQU6CpzVmSaiqey9rnl2y9j98Akrj+rx3lvGaNMpZAyuqqUvBrHC3nuhC8m1blXNSb3bxoWLHM0uBCEcKek35NT554yXNUkXltGcTz4/MfHGHNZM1M+tsxBuyXwrP3YsCdyT5WpFFJmWUsIS5rDZXt9gCJ3IgeyntMaqtWHyblvcjFXr/NqIOT0bs8kdOTZpy2SuGdE3X2/GKekIag5PrqfLZOwp/uLZKs3cnd6y5RR3O+/7aKyd+0kcSd8MVyeO9W5Vxu5bJlqxoncpWBB7hvTGslE3KrHjnHWaC0iySn77vkSqo64z3gi90WwZbxlkeWgeq/hiIpiUp17VZOrWqaaySRU/SN3ly2jiP99EqsFkMsh84u75cePTrtnsjozVBdxmcJyQOJO+OKqlinBmpREaclVLVPNiISqHCzIYt0sR+6KN5Gau+ukF+G5t0R034SqQETuaZO7Zqm+2TcOxoCeMnvi5YZsGcIXOVg3yHOvOmrRlgk6y8pJtox0glrVHnFuZyVU7fthvbBkNYU0RAMq2iKBvJOY5GTrRCyFWMrAd54/gp9s78clazvQ6dPhspYgcSeyeODlo3jlyJhzn2yZ6uJLj72Fh7f3Ox0UvfzeFevK1q9kIQSdUkjJlpFOUGf3tmRtZ053SGv7ms5owfe5/PQuRIMatveNZ9kyuRqJTcbS+PvH9uDZfdb6q3devaHg+1Q7BcWdMRYC8EsAQXv/H3DO/4Ix1gbgAQCrABwG8AHO+Zj9nLsA3ArAAPApzvljZRk9URYeefMEnjuQWWSYSiGri++/cgyJtJkzcv/Mtact8oiKQyRU5eorubrnjGXNzm1vXfsJe/HvtUWI+w1nL8UNZy/Fh//9haxqmRlPI7HGoIapRBqP7BjAs/tG8NsXr4KuMlx/Zs8cjqw6KSZyTwC4knM+zRjTAfyKMfYzAO8D8CTn/IuMsc8C+CyAP2GMbQJwM4DNAJYCeIIxtoFznrs9G1FVWOtZUkK1WomnrMi3xlwZJ0GZylF9JTf4Ujyeu/gNru1sKPr9GoIaRqdnXdvGZ91lj+0NAUwl0th+bBwA8PtXrkN7jiuiWqNgQpVbTNt3dfsfB3AjgHvt7fcCeI99+0YA93POE5zzQwD2Azi/lIMmyot3ZRgqhawu4ikrTlrMNVJLgSgtLHbehKqwrGOcm7jrWZ77+Kw7wSqE/NiYdRJoDrt709QyRXnujDEVwCsA1gH4Guf8RcZYN+d8AAA45wOMMbGA4zIAL0hP77O3eV/zNgC3AcCKFSvmfwREyfHaMFVo356ycM6dSTj5FseuRkSJ4paV7tWavvHR87Cppylrf5WxrGP0rkOaj8aQllUtkyXu9uv1j8fRGNTKWtu+2BQl7ralcg5jrAXAjxljZ+TZ3e8Xl3Wq5pzfA+AeANi6dStd91cRqSxxJ3WvFhJp/2RkLdAc1vGzOy7Fqna3b37t5iW++yvKwq5OQrrq+rwAYDyWbcsAVkDTEq2fqB2YY7UM53ycMfY0gOsADDLGeuyovQfAkL1bH4Dl0tN6AfSXYrDE4pD2hOrkuVcPwpIB/HvLVDsbfSL0XFhL7Fm37/v4BU5CtliCmoJE2gTn3EnceiP3kK4iGlAxkzTQEi7+qqAWKHgNwhjrtCN2MMbCAK4GsAfAwwBusXe7BcBD9u2HAdzMGAsyxlYDWA/gpRKPmygj2bYMiXu1IJKpQO1F7nNFkZbYu2RdB87z2DmFEHX1clmod0m9gKa4Jj3VE8VE7j0A7rV9dwXAg5zznzLGngfwIGPsVgBHAdwEAJzznYyxBwHsApAGcDtVytQW3oQqRe7VQyKd+VMy63zmsKqwBV2dBGz/PJE2nRr7eMqAwqzkbjJtIqgqaAxpODHpblxWDxQUd875GwDO9dk+CuCqHM+5G8DdCx4dURG8Yk6Re/UgR+6xVH3HTCpjC7o6ESs/JVImELK2xZIGQroKTWFIpk07crdksKWOKmUA6i1D+OAtVaPIvXqQPfd4sr7FXVGyq2XmguhtL1/txNMGwrrqCH9AU9AUPnVtGeIUI+2pjjFJ3KsGWdxPhch9IdUyGXGXrnaSJkK66jQsC6gZz72eatwBEnfCB+8al16xJypHPH0K2TILjtwlW8YmnjYQ1BXHy9clW6b1VPPciVMP2YbRVYU89yrCZcuk6vuka9W5z//5olomkTYwMZvCwZFpxJOWLSMIqAqa6rRahjx3Igu5zl1XGXnuVcB0Io2UYbrEvd4JqIpT8TIfZFvmP547jP95zwuYtROq4jFXQrXOxJ0idyILeYZqQFMpcq8CfvOrv8INZy/FspbaXkBiLvzNe89E2xzaDXhxbJm0iZMzCSTTJsZmk+hoCDplpEFNQZMj7vVly1Dkfgryy73D2Nk/kfNxWcwDKqMFsitMPGXg0MgMjozOuCo/6p0L1rRjfXfjvJ/vRO4pA9MJ63MbmU4ipCuuyP3y07rwWxetxMq2SM7XqkVI3E9BPv+TnfjXZw76PsY5d4u7ptT9ZJlqR/QyH4+l6t5nLyUhPWPLzNiLdpycSSCoq05UH1BVLG+L4C9vPKOumoYBZMuckiRSJpI5IsCUJ0oPaAp57hVmQIj7bOqU8twXimzLiBWZTA6EdRUpxTpJBrT6EnSZ+j0yIicpw8xptXj9dV1VshqJEYvLickYAKsvSjxt1H1PmVIhT2KSl9sL6YqzWDeJO1FXpAwzq62v85inpl1XFRhky1SUExMJALa4p0yE6liQSolc5z4jibs1Q9X6DMXqUPUI2TKnICmD54zGvRG9rjLQHKbKcmLCitzHZ5OIpQwEdRVfuulsLGkOVXhk1U1Q8tzdkXumzj1YxydKEvdTkHy2jHc2qqZQQrXSCM/d5MDIVAIhTcE762AB53KT6QrptWVUiJ90QJ1bj/haon5PW0ROLFvGLeJ/+uM38Wf/9WZ25K5ZM1RfOzqGH77St5jDJGxOTMad24OT8TkvWnGqoigMAVVB3GPLeCcx1SsUuZ9iGCaHybMTp7sHJqEr2a0GAiqDyTne+/XnAADvP6930cZKWAxMxNEWDeDkTBKDk4k5rSN6qhPUFEzEkpB/1iFdgWlSQpWoM8RCHN6Sx5RhIm2aWQt1aB7B55JF8+PX+vDer/+6jKMlOOcYn01ibae17ujgVNyp3yYKE9QVjE67100N6yresbkbd169Hq111nJAhn4lpxhiyTFvQjWZNpE2eVZNu7BlBLJ3uat/Eq8dHaf2BGUkkTaRMrjTdoBzOGV8RGGCmoqTM25xD+kqVrZHcefVG5y1VesREvdTjJTdMtYr4lYFDc/23BXmuqSVFxgWfbJpYk35ECfTZa2ZnjIUuRdPUFMwmiXup8bnd2ocJeEgRN1rvyTTJgyTZ1XLiJa/oh5YXmBY9Mmu977ilWQ6bot7S6bvCSVUiyegKRidTri2nSqfH4n7KUZSRO6eCD3peO7u7ZrKYHDurFbjEne7hUGszpd7qyQicu9oCDilffVcm11qgrqKSfsEKZKnJO5EXSIidm+EbiVUuW/7AdPkTs9rsmUWlylbmBpCGi5c2w4AOKu3pYIjqi3kE2F3UxAAXIt11DNUClmnfPSbL2LH8Ql8/obNuPGcZc52EZl7Pfdk2rQ9d7foBzSr/UBD0Bb3WMa/FKJOtkz5EJF7Y1DHdz52foVHU3u4xL0xhGMnYxS5E7UL5xzP7hvB2GwK2w6PuR5zInefUkjD5Fk9ZzSFgXNkxN03cqf+BOViOmF93g0hisPmQ1CqLOpusto1nCqRO4l7HZKrdBGQ69wzgsw5t6plTBOGT0IVgNOJ0O25U0K13IhFJsTJlZgbor8MY1beAjh1qmXoF1OHJCXhFp6tQLZlxmaS0FTmJJrSJs9KqIoqGZGIHZ/N2DKUUC0/olqmkSL3eSFsmdO6GxG1T5Cnii1Dv5g6RAgxAFdPDSATsRsmx//+f6+gtzWCL9y42drmU+eu2svPi+f5lUJSQrV8TCdS0BRGFTLzRJwcr928BBeuaUf/eOyU+SxJ3OsQWdy9towc1Q9MWE2oktLEJrmKRlUYxMpjIqL389zJlikf0/E0GkJaXc+kLCfbjlg5p3ds7sbmpc24yK44OhU4NU5hpxiJfJG79NhsMo1k2nSVR8qRu8IAxRYV38idbJmyM5VIk9++AP72fWfiytO7sKmnqdJDWXToV1OHiOg8ElAx5RF3uQRyNmkgaZg5I3eFMSeRKsT92MlZ7B6YxMaepky1TI71WImFMx0ncV8I125egms3L6n0MCoCRe51iBDi1kjA8Ry9jwGWuKcM0zkZcA4kpcjdsmWEuFvNq6JBDbd9dxuAjNcep8i9bExT5E7MExL3OkRE4u0NAcRShqs0UvbjxX1Z8BOSf64w5ni9ScPExp4mfOTClTh2MoZE2iDPfRGYTqSpxp2YFyTudYgQ8DZ7UQc5qeotdZRtGSDjn+sqg8IAVfLcVSUzhbt/PO4sVUbiXj4ocifmS0FxZ4wtZ4z9gjG2mzG2kzF2h729jTH2OGNsn/1/q/Scuxhj+xljbzHGri3nARDZOOIe8RP3ApG7/dyQpkKRqmWSaROqwtBlz/I7enLWeU4sSTNUy8EH73kBB4dnTpkZlURpKSZyTwP4Q875RgAXAridMbYJwGcBPMk5Xw/gSfs+7MduBrAZwHUAvs4Yo1/nIpIw3JH7TB5xTxkmkulMNC989FBAhcqYq1pGYQxdjVbkfkwSd6pzLz2JtIHnD44CAJrC9btaEFE+Coo753yAc/6qfXsKwG4AywDcCOBee7d7AbzHvn0jgPs55wnO+SEA+wFQx6NFRJQ7irU25VmqWbZM2nTVvovKl5CugDF3QlVTmNOfg8S9vIjv7PqzevDJy9dWeDRELTInz50xtgrAuQBeBNDNOR8ArBMAgC57t2UAjklP67O3eV/rNsbYNsbYtuHh4XkMnchF0sjnuXsjd+6qfY+nTCgMCKgKVCXTUwawVpNviwSgKQzHxiRbhsS95Eza8wnesakbHQ3BCo+GqEWKFnfGWAOAHwK4k3M+mW9Xn21Zi2xyzu/hnG/lnG/t7OwsdhhEEXgTqvlsmazIPWVAUxToqgJFsmUAK7mqKAydjUG3555H3GNJAz97c8C1sDZRGLHARFOILBlifhQl7owxHZaw38c5/5G9eZAx1mM/3gNgyN7eB2C59PReAP2lGS5RDE4ppIjcJVsm6RV3T7VMIm1CUxk0lbkmMQGZKL6rKYSjo5a4B1Ql7wzV/3r9OD5536t49ej4wg7qFENE7k1hqpQh5kcx1TIMwDcB7Oacf0V66GEAt9i3bwHwkLT9ZsZYkDG2GsB6AC+VbshEIYSAO567HLmnsyPoWUmc4ykDqsKgKQpUxR25K0LcG4NOZNkc0fN67vsGpwEAz+wl620uTMZtcafInZgnxUTulwD4KIArGWOv2//eBeCLAK5hjO0DcI19H5zznQAeBLALwKMAbueckym7iIhIvDWSbct4l9fzPh5PGdBVJVPnrrhtGSBT6w4ALWE9ry1zYNgS91+SuM+JyZhty1ClDDFPCl7zcc5/BX8fHQCuyvGcuwHcvYBxEQtA7i0T0hVMJ9L4u0f3YMuK1izPHXAnXBNpE5oduct17kBG6Je2hJ1tLREdg5PxnGMR4r69bxz3vXgEHzp/BXU4LAKK3ImFQoZeHSIi94CqoCGoYyqexvdeOgoAeP+W3qz9vZG7pmQ8d5ctY9/ubY0425rDAcTT/pOYYkkDx8dj+OD5y/H6sQl87sc7sGVFKzaegh365spkLAVdZafMqkFE6aFfTh2StKNvRWFoCKp5SyEBYCaZeTyWMqCpCjSFQc1KqFr/97ZmIveOhgCSadPXdz80MgPOgUvWdeCv33MGAOSN8okMk/EUmkI6XeUQ84bEvQ5Jpk1n6byGkIbR6YTzmDXT1L2/WKcTsFZX0lSGZa1hLGkOOT47kFmVSRZ3cXt4ynqPkekEfvBKH4CMJbO2s8GZ2Sr2I/IzGUuT304sCLJl6pCkIYl7UMMJKVpOGRyRgOaK5r22TEtEx+d/czNMDrx6dMx5TETundKkmuVtlkUzNJXA8rYIPvGdbXjt6Dgu29CBI6MzAIDVHVGnydjwNIl7MViRO/15EvOHIvcagnOOH73aV3Dlo2TahK5mxH1wQhZ3M2uBYFno42kTmqJAUxUENMW3Wka2CoT/PjxlvcfOfmt+20zCwOHRWXQ3BRHSVYQDKhqCWtVF7t/+9SH8xUM7Kj2MLCZjKYrciQVB4l5DHBqZwacf3I7Hdw/m3S9pmAhI4j4jnQxSholIwC3ucuRunRiyk6hAps5dxmvLiGTuZCyFI6MzWNkedfbtbAxiZDqZ/yAXmSd2D+LRnScqPYwsJuNpqpQhFgSJew0hJhsVWvkomTadFd6jnl7ghcQdgBP1A/517jIdDUEozLJlZKbiaRwZncXKtoi0b8CJ8KuFockETs4kq649ghW5ky1DzB/69dQQYkHqhE/Fi4w3oSpQmLWMXtgr7p6Thfy4HKzLkftjd16GV4+OQVUY2huCGJpMuCpxBifjGJpKYFWHO3J/68RUocNcVAYn40gZHJPxNJqryAYR1TIEMV8ocq8h4ilLPFM56soFckK1UYrcA5qCVLpw5B7UZHHP7i0DAKctacQHz18BwGpHMDydQP94zHl8R/8EAGCFFLl3NgTL5rn/xUM78OPX+ub0nHjKcNoonJypHrsokTYQT5nkuRMLgsS9hhC15N7mXynDxJVffhqP7rC842Q647nLtkxAVZA2TYR19wXbbNJwbBwArokzhWwZwIrIh6birk6RO45b4r5K8tw7GqyeNOXo/37v80fwBw9sn9NzhiYzJ5rRKqrimXI6QtKFNTF/SNxrCLEEnneR64lYCgeHZ/Dm8XHncblaRhDQVKR8bBnAbcXIy7p5+7n70dVoReTHTmYi9z0Dlv0i18R32rXuIyUWUtkvN8383vm3f30IX/zZHnDOMSj5/6NVFLlnOkJS5E7MHwoNaggR8XpnmYqWvmOzKedxEbE3StGfqgCJlIGQpkBhgKyDEV3FOKzny6WSsi2j5RT3EEamkzgwPI2ApkBXGKYSaQQ0BS2RjECJVZxOTMRdLQwWitzV8vDoDNZ0NuTc9ws/2QXASu72NGdOPH/937vwvZeO4j9+p/KLhlEvd6IUUOReZZycSeL3v/capuzGUTK5IndRpz5hi3tCSqjKtkzK4IinrTp30WJA4IrcA3OL3Fe2R2CYHM/sHcaq9oiTmOxqDLpq4kVy9fDorO/rzBe5fcIbfRN59z19SSMA4Aev9LlaIRw7GcPTbw1XRR0+9XInSgGJe5Xx+rEx/GR7vzMZSCaX5y7EfWw26Twuz1AVpOweMEE7upYraeT9QpL/rrLCnvuGbksw9w9NY3VH1LETRKQu6G0NQ1UYDo1YbQlSholv/uqQUwU0X2ak9gnyjFoA+MVbQ3jBXmgayHxW47MpDE7FoavMZUO9eGgUiwXnHF/82R5c/LdP4qdvZNazoY6QRCkgca8yRFTu1yNdVMt4I/eZhNuWSaZNBH0894RhibuI3OXHZCEOyaWQrpa//mNe15WxQVZ3NDhWkOgnI9BVBSvaIjg0YrUleP7AKP7qp7vwwsGT/i9cJHK1z6M7TsCQ/Kbf+fbLuPmeFzBk++ti35OzSQxNJtDVGHJ91vKJoJy8fmwcB0dm8K/PHMDITBJ//l87nIod6uVOlAIS9ypDWC9+LQZEhJvLlhkXkXuOOvdk2oTJrWoYXWUucZd7tIe0HLZMjsg9GtSwzH7+6o4IGkP+kbv1eBSHRixb5oTdFsHPgpoL4vg/fMEKDE0l8PwBS6Blkf/Hx/cBsKJ8hVmfxaGRGXQ1yX1ywnjuQPnF/eDwNN7ztV/jm786BAD442tPw9hsCv/12nEAFLkTpYHEvcrIJ+5OnbvHlpmKZ2yZvYNTmIqnfatlBCFdhaYornp3WYhdnnuOOncvG7qt6N0VuTcFs/Zb3RHF4ZEZcM4xYIu7vMbrfJi1Pfcbzl6KhqCGR3cOAICrYdobfeNIpA0kDdOpvd87OIXuxsxx33LRKhwcnnGuLMrFbruSSFwlXH5aJ8K6iuP2PAHq5U6UAvr1VBn5bBkncjf8bZl4ysQ7/vGXiKWMTEI1kC3uQbshmDxZqadZsmUkUVGUYsXd8t1XdUSciFMWTsGqjihiKQODkwmcmLTEbDqxMHEXLYvbGwJY0RbBiYk4Pv/wTtz1ozcBAGs6ougbiznevOhkOZs00N0UxD9/6FzcefV6XHfGEgDA47vK02tmZDqBG7/2a8dfPzhsnUS6m0LoaQ45VzLUy50oBZSOrzJE5O430Sfjubtruf3E8fiYJZyKwhANqK4WA0Fdha4y5wQAuKNsly3jsxKTHx++YCV6mkPoagzljdxFr5ljY7OZyH2B4i5ObtGghrZoAKMzSTyxe8h5/KK17bjvxaMYmLA+k+XSrNmuphDefdZS5/7mpU14bOcgbrts7YLG5MfP3hzA9mPj2H4ssy0aUNEY0rGkOeSMj3q5E6WAIvcqI5nPc89RLTPlsTVO627Ehy9c4dz/4PkrcNmGTud+SFeh2y19BXKPdndCtXCdOwCsaI/gty9ZDQB5PfcO+31GpxNOpLpQWyZL3KXOkwoDzl/dBgBOX5vlUo29N+l7xWldeP3YeFZLhlLw813Z3TzFZ7SkOYTj4zF8/N5teHTHCZqdSiwYEvcqI78tIyJ392NeIfrpp96OS9dnxPzP3r0J15+5xLkf0hTcefUGfMwWYyAze9R6fG517l429jRiaXPINTtV0NEQAACMTCcdT1yuU58PIvKP6CraogH0jWXq6Je2hJ0WCI64t2XG1eU5AZ23qhWGybG9b3zO43ijbxzn/uXPfZcSnIil8PyBUWcOgLgIEuLe0xzC4GQCT+weRNKgvjLEwiFxrzKEr+5fCmltm0kY+Pi927DTbs41nUi7ujfqPjWL8ragruL6s3pw0dp2Z5vcETFnQrVID/jy07rw3F1XIeLj97dGLXHvG4th3C7d9F55zJXZpIGQbi0w0hYNuGbermiLOCeZPba49zSHnZNWt8c62rK8FQDw6hF3vXwxbO+bwNhsCrsGrDkKw1MJ/Pa3X8LwVAL/9dpxpE2OP7t+IxgDLlxtffZL7FzHEs9JhipliIVC4l5l5LVl7Mf2nJjEE7sH8anvvQbAEnevOHiRLRh5kpJATt65E6qZffIlVItFV62WBOLEBMzdc+ec491ffRY/tNdqnU6knaqgNvvkAQDv39KLT1y6Bm3RAMK6ij0nLNFtDGlojVj7eZO+zREdG7ob8EoR4j48lcC/PnPA6WcjVrzqs/Mdzx8cxdNvDePlwyfx3ReO4OzeZty0dTme+PRv4GNvt66aRF5iSbP7KsdrvRHEXCFxXyTe7JvAn/zgjYKNrRJ5JzGJ3jLWa4iuhtOJNHrtJOE1m7p9X1eO3L3L7HlxNQ4rsBLTfGiPBpyukbrK5uxvT8RS2HF80pmNOpNIO20WZHH/5OVrcMXpXWCMobc1jEH782oIamiL6gio7t43gq2r2vDioZMY8rFXZB7e3o8v/mwP9g5ZVwTCjhG20GG7pPKJXYPYPzSND1+wEoC1YLiYF7BEsmUAOOM5MDQ9p8+EILyQuC8Sz+4fxgPbjjkTVHKRL3KPe7z2KVsUp+NptEcDePIPfwNf/eC5vq/ritwl8W6LBrC2M+raN5SjK2Sxtkwh2huCzmzajT1Nc7ZlRJXN8FQCH7znBTz0er9T8imLe4eUJF7ZnkmiRoNW5N7p6X0j+MSla5A2Of7yp7vyjkP0rxcljScm3ZG7qJd/dv8IAODcFS3Oczd0N+B3LlnlnIyFuP+PLb0AgPeeuyzvexNEISglv0ik7PLFmaSBljwNEcXluG9CNeV/qS5sibV5uiEGZM9dEvptn7sa3msJOaHKXJF77nHPBZFUjQZUrO1swMuH59Z+QFTZHD056/jowpZpt8VdV5krj7C2s8Epj4wGVFx+WlfOJmGrO6L4nYtX4Z5nD+LuWCrnCk2idFFE2YMecT9oi/vwVAKMuUswNVXBX/zmZud+e0MQ/3TzObhkXQf+5J2n561MIohiIHFfJMSs0tkCFoRIqPrWufs02BqfTWJasiVykcuW8bNaQgG3iqsKg2Hy0kXuUSuiXtvVYC3gPUdbRkTu+yTrImjnCUTCtj3qjsrXSv1vNFXBJy/PX8d+xeld+MYvD+LlQydxdQ6rq3/cGscD247hJ2/0Y++gNZ49A5O478Uj2DOQaf62pClU0A678RyK1onSQbbMIuGIexGLWwPFR+4HR2YwnUi7+rb7Ecix0pLvvp5qGyHqpUioAhm7ZG1nAxpCGqYT6TktUH3Cjpjl3jEiWm6NBMAY0NEYcD0n31WNH+csb0FAU/D4rkEcHPb3v0Xk3jcWc4S9MaghkTbxuR/vQCJtOiWPsi1EEIsBifsiIeyWQjXd+XvLZG/b1T8JzlFE5C5Xw+SPIL0+tLBjSiXu7bYts7YzioaghpTBneMuhhM+iU7hb6sKQ0tYd/nt4r3mQkhXsbGnCQ9sO4Yrv/xM1mefMkwM+dg6G5c2ue6fvsS6v7Jtbu9PEAuFxH2REBH5bCJ/5J5X3H0EUESshcRd9tm9kXkhSh+5C3FvcLzyuVgzwpYRtEUDuOudpzv3L17X4cxKFbRE3JF8MXz4/Mws3/2e6pXByTg4z65Pv+WiVfjMOzbgxT+9Cp95xwbc/LblAICVHRS5E4sLee6LhGPL5Fkc2jR5TluGc57V6heAkxSM+qyLKiM894CmzLmkUeyfr7fMXDhvZRuu2dSNC9e046k9VpJzOpFGe0N2Lxo/TkzE0RjUMJVIo6MhiG1/drXr8a99aEvO5/p1yczFB962HFtXteLKLz+DXQOTOGNZs/OYOMH85Y2bEQ6o2Ds4jb/66S5s7GnE9Wf1AAB+78r1eGbvMACK3InFp+AvnTH2LQDvBjDEOT/D3tYG4AEAqwAcBvABzvmY/dhdAG4FYAD4FOf8sbKMvMYQtem5EqoPvX4cd9z/ulPK5xV3EdELUQvrKtKm6Sw27TcbVEZ47n4TmAohIvZSRe6djUH8229tBZC54sg3kWnv4BTGZpI4d0Ur9g5O4fh4DGf3tuD5g6NY2pJ/8pbMji9cO+exrmyPIqyr2OVZGUuUQa7uiGJ9dyPevq4D15/Z48w4FVy4pg2fumo9rji9EwSxmBTzl/4fAK7zbPssgCc55+sBPGnfB2NsE4CbAWy2n/N1xlj+kPIUIZknoco5xx33vw4Azmo88ZTpmvAkkqkicRoJqIgEtEzkHiwuci/kt/tRaltGRhxPruZhQ1NxvPOfnsX/vOcF/M0ju/H+f3kODMD7z7PqwXuaixf3hqA2p8gdsI759J5G7B5wi/tDr/ejJaI75Y2MsSxhB4CgpuLT12woePIliFJTUNw5578E4C1EvhHAvfbtewG8R9p+P+c8wTk/BGA/gMovJ28zOp3IWuhisUgJz90nofp8jqXd5CSjKIMUHRcjQRXRgCpF7sWJe3AeC0CU2paRKRS5v3pkDIbJ0RYN4D+eO4xE2sQPf/divO/cZQhoCnpby+9lb+xpwq6BSaei542+cTy1ZwifuHTNvE6WBLEYzDeh2s05HwAA+/8ue/syAFK3avTZ2yqOYXJc+eVn8L2Xjlbk/VNOtUx25O7XRRBwWzOiWsOJ3HUNkaCGUTvSLxQZBh1bJrcYPXrnpXjkU5dmbS9n5C5yBblKRLcdHkNAU/AHV68HYAnt6UuaoCgM37rlbbjtsjUlH5OXTfYsWrFS0ov2mq8iWUoQ1Uipq2X8/vp9C5gZY7cxxrYxxrYNDw+XeBjZJNIGJmIpZ3bjYiM8d78qmFjSfTUhervIUb6I4kUr2HDAitxFrbffiksyxdgypy9pwiZPKR8gee5ljNz9rmgA4JWjYzi7txk3nLMMrREdH7ogU8Hy9vUdvj3jS83GHuszEb67KGedTwUOQSwW8xX3QcZYDwDY/4tlb/oAyOFML4B+vxfgnN/DOd/KOd/a2Vn+ZJNYxchvctBi4NS5+9gP3jGJ6e7xPJF7NKi6ovVIAc9dVRhUZX7rcoo691K1H5ARJ6UZnxLReMrAjuMT2LKyFc1hHS997mp8RBL3xeL0JY1gLLP2qWgxXI4rGYIoFfP9c30YwC327VsAPCRtv5kxFmSMrQawHsBLCxtiacg3rX8xcOrc80xOEmIhOgPKEb1TLWOLe1jXXEnUQpE7YE1kCuaxZXIhInatDOouesf7nfSGpxJIGRxrO6zZpbqqVGRd0WhQw6r2KP7xib145z89i8lYas6JWYJYbAr+tTLGvgfgeQCnMcb6GGO3AvgigGsYY/sAXGPfB+d8J4AHAewC8CiA2znnlVFTDyJyj+dovlVuMu0HskUsnjLAGNBqi7qI3OXZrJnI3XpMjtwZK9xSALAmL80vcrcTqmWI3AOagoCq+OYiRAfNpnDlhVSUqO4emET/RJyqX4iqp+AvlHP+wRwPXZVj/7sB3L2QQZUDZ4WjAr1dykW+hGo8ZSCsq05ULZa8m4hl2gP7lUIKogGtqIg2oCkILqQUskxRcySo+p70RCvgxipYleidZyxxFvAYnIgXrE4iiEpzyoQflfLcd/VP4tEdA3n7tMdSBkK66pQpiiTh+GxmoWdRCimWX4sENIheW8UKTWNIz9m+Nh+lnsTkJRrQfD33jLhX/mf6sUtWozUSwB9+fztOTMaxvmtujcgIYrGp/F/NIiFsjcUW95/tGMBXn9rvNLLyaxwWT5muyF30KxELWoh9AP/IvVhx/8ZHz/NdeagQor69VCsxeYkEsiN3zjkm7SuXalhPVFEYltqrJ03EUoiQ505UOafML1QkJBOLLO4iIp2y/ePZhIG0YaJvLIZVHVa/kVjKQFBXnFr05oi1BNy4JO4Jn8hdUKz/u6G7cV7HUM5SSACIBDWXXXVyJonL/v4XzgLe1RC5A+5xFOrlQxCV5pTpClmpyF1EpAlphur/fWo/Lv/S0zgyarWpTdieu0h2BjVrbU+XLeMTuYtqmUKtBxaKUnZbRnX13NkzMInpRBov2jN3q8FzB9xXEIW6cBJEpTllxD3fwtPlxJtAnU0a2H5sHACcBR5inoRqUFPQGglgTBJ3EbmLqo2msOZE7OWu3BCaXo72A4A1fvlzEsvTTcbTCOmKa6GRSkKRO1FLVMdfzSIgIvfFLoX0doFMm9zxvcWiE/GUaSVUbRHj3LJm/Dz31R1R/NtvbcU7z+hxBKbckXumzr084t7gqZYRC28A1RO1A0BDSJ40RpE7Ud2cMr9QEbnHF7kUUk6gNoY0TMXTTgR8xBaxWNJAayTglCkm0iZaI7pL5BIpA0HNmsRzjb2mpxCYskfuSpkTqkGrWubHr/Xhbx7Z46ooqha/HbAmUYV0BfGUSZOYiKrnlIncExXz3DPv125bKqK/jRDveMqazi4i90TasG0ZOaFqulZTAjLWQLktApWxsk61F90t/+CB7RieSrg6RFZT5A5IXTnJliGqnJoX95HpBH7jH36BvYNTefcTkXva5Atq+/v3j+7BH31/e9H7y9PqxUpDYmFlWdzDuorrz7RW8DmrtwUtkQDGZ5NOm9m4XQsv43juZY4iVYWVrVIGcF95nNXb7HqsqYoid0Dq7UMzVIkqp+bFff/QNI6MzjpJylz4NeHKx8HhaUeEZZ7dN4L/fnPA6cZYCDlyF8lQsUTb0ZOzSBumM4npitO7cOBv3oWNPU1ojehIGdx5fiJtZvVid6plyhxFKgorS+sBgZwz+N3L1wLIdMashhp3mUz7BxJ3orqpeXEXU/TFCka5kBe+KGTNJNMmrvzyM7jhn3+d9djARAyzSSNrweRcyOIuFoaWryL2nJiyJjHZAi3sj1a7nayomImnjKxe7EJgyi00KitfjTvgjtyvO6MHD91+CT59zQYA1eW5A5kriUJdOAmi0tS+uM8WJ+6uyN3TP/1fnzmAX+0bce4/tvMEgMzi04JE2sDItPU+ha4UBHIViIjcAeDcFS1gDPj5rkErcvf46aKiRkxk8rNlOhqC+Pv3n4XfPHtpUWOZL6rCypZMBTKRu/h8zl7egm57ybpqE3eyZYhaoebFfTxmia0Q3Vy4xD3tjty/9ov9+OGrfc79+1/OrNYknzTkhT629427XuPg8LSz0r0gmTadRToAoC0adG6vbItg68pW/PQNq919yGOttNtR/tCU9Z5+CVUA+MDbljutDcqFUuaEqojcxZWNfLvqbJlgpisnQVQztS/uTuSeyLufy5axrZJ/++VBvHz4JKbiaZeIHxqeQXeTJZh7TmQWRu4ft4Q2pCvY0e9eMPlrvziA3//PV13bvP1S2qXIPRrUcM2mbhwctpKqYU9UvsbuYX5gSK6oqYyglDuhqqvWa4tumADQaZ+wKHIniPlR++JepOcuR+6xlAHOOf7hsbfwrV8dAgDXbNCx2RQuWmP1NXnrRKYKRyRYN/U0YcRj2fSPxzAZTzs9yIHs2akhXXVK6BqCGs5Y1ux6TKY1GkBHQ9CpAsoVuS8GilLeyF10zFzZHnW2rWyP4jfPXopL1nWU7X3ng7xAOUFUMzUv7iKhWsiW8SZU4ykTScN0yhHFySGeMhBLGVjf3Yj2aADfef4IfvbmAABLwAFrTU251zqQmW0q9vnuC0dwyRefcu0T0Ji0TJ7mEjNv5A4AG7obsNdO3FY0ci+zLXPVxm7cefV6/Om7NjrbApqCr37wXKyfZ7OzcnHGsias6YyiJUzrpxLVTe2L+xwSqkJYEynDEefDdvOuMfv5wuZpiei4+fzlODQyg2//+jAAoH8ijrZoAEuaQphOpJ2Ik3PuRPXHx6z/Xzs6ljUGXVVcpXQ90uLOfiskbehuxP7BKXDOEU9ll0IuFprCytZXBrA+lzuv3lATsz6v2tiNp/7w8qrpd0MQuaj+v6YCiIRqLGVgNpl2ldWdnEliy189jq99aIs9pT+AqXgaMUncRc+WmaSBB14+6iRAWyMBfPiClTgwNIMDw1b0fGIijiVNIaeSZSKWQmdjEJOxtPM6InIXIi+jq4pTStcQVF0VKH5R+bquBswkDfRPxG1bpjKR+4cuWIG3r68ue4QgiPzUvrjPpqAwwOTA6HQSkbbMIT1i2ynff+UY4ikTLREdR09aC097bRUAuOtHbzrRoxDwloju+Poj0wl0NgbRbNegT8SS6GwMYmAyI+R9trj3SxOgxPjkyL0hmJnGPpv0t1xE//V9g1NI2C0KKsHWVW3YuqqtIu9NEMT8qPlry4nZFHpbIwCyrZmfbLfKDHuaw0ikDWeJubgUucuY3GozC2QmEcltAEank2hvCKDFfh3R+2VAKpH8xjMHceM//woD45lton47oCqS526JeZddIeIn7ms6LU/+0MhMRSN3giBqj5oW97RhYiqRdkRwZNqqYImnDHzo317Ai4dOOtsTKRMttmDHUoazhFsuhLiLNgAzSQOjMwm0RwPOY8KfF/XvwnLZ3jeBtCnXt9virsmRu7WvODH5tURojwbQGNRwYHgaScOsWOROEETtUdNqIaLs81a0IqAq+NV+a5bp47sG8dyBUdxx1XpcsLrNEve0lVAN6QqGpxK+kbuMsGWEkPePxxBPmWiLBqXZo9aVwsBEHIwBS5pDvq/Vbk9e0lXmnABEy4Av3XQ2PnLhCpy3sjXreYwxrO6MYs+AVQ5ZqWoZgiBqj5oWdyGuy9siuPL0Lvxkez/ShomHXj+OJU0h3HHVevQ0hzAynbAWxNBUnLeyFS8dOukSd2/lQ0hXHCEVQn7ALklsbwigWUqoApZt0t0Ywtc/vAX//KFzsxp5tdmzLXVVQVPYHbkvaQ7hr99zJnTV/6tY1R7FHrvWvlJ17gRB1B41rRYi0dkc0fGec5dhZDqJR3eewNNvDeOGc5ZCURg6GoIYmUo6PdMvXN2O3ScmcezkrPM6y1vDrtcV0ToAx8oRFTPCKlEVhrHZJGaTaTy1exCXn9aJdV2NePdZS3HuCisK/+6t5+N/XbbGidYDmoJV7VE0BDVXn5l8rO6IOv3NKXInCKJYarpaZlNPE5749G9gSXMIKmPQFIav/eIA0ibHFad1AbB6qIsukEFNxYVr28EfB556awiqwmCYHO0NQYxMJ9HTHMKeE1OOoAOW5w4AB+w2AW3RABhjaAnrGJ9N4dEdJzCTNHDjOcuc59xw9lIYJsel6ztx6fpOfOEnOwFYkfu7zlyCy0/rLLqTo8gnWOOv6XMxQRCLSE2Le0hXsa6rwbm/aWkT3uibgKownLO8BYC7GVVPSwhn9TYjpCsYn01hVXsEh0dn0RzW8dsXr8KG7kZ86v7XHEEHsiN30aSrOaLjvheP4r4Xj2JlewQXrM6UCn7gbcvxgbctd+6L2ae6ysAYm1OL3s1LMy0K/GaxEgRB+FFXoeAW2w7ZvLTJ6Y/eITWjumhNO4K27w5YJZIBTUFzWMcfXLMB15/Vg97WsKuBlddzF3ZKwp60dPXGLnz/f12UtyVuc1iHprB5zWpc19WAH/3uxfjUVetpIhFBEEVT05G7l62rWvEfzx12VZ50Su1we21v/cLV7fj1/lE0h3V89MKVuNBuEgYA//Lh89AUznwsuqqgMahhKpFGUFOcxl+il8z/efdmdDX5V8kIbj5/BbasbJ13nfqWFa3OiYsgCKIY6krcL1jdjvZoANds7Ha2yb3Omd0f5cK17cDjQFNYw5+/e5PrNTYtbcp63eaIjqlEGh0NQec1vvux89E/EceK9kjBcTWHdbyNZngSBLGI1JW4dzYG8cqfX5O17R2buvE7l6x2tp3V24ymkIalLWHvS/jSGgmgbyzmqnC5uMpa0RIEQcjUlbj7oSoM9/zWVte2oKbisT+4rOi2rR+/dDUe23kC125eUo4hEgRBlJy6F/dc9DQXF7UDwI3nLHOVOhIEQVQ7ZauWYYxdxxh7izG2nzH22XK9D0EQBJFNWcSdMaYC+BqAdwLYBOCDjLFN+Z9FEARBlIpyRe7nA9jPOT/IOU8CuB/AjWV6L4IgCMJDucR9GYBj0v0+e5sDY+w2xtg2xti24eHhMg2DIAji1KRc4u43XZO77nB+D+d8K+d8a2dnZ5mGQRAEcWpSLnHvA7Bcut8LoL9M70UQBEF4KJe4vwxgPWNsNWMsAOBmAA+X6b0IgiAID2Wpc+ecpxljvwfgMQAqgG9xzneW470IgiCIbBjnvPBe5R4EY8MAjizgJToAjJRoOJWkXo4DoGOpVuhYqpP5HstKzrlv0rIqxH2hMMa2cc63Ft6zuqmX4wDoWKoVOpbqpBzHUlf93AmCIAgLEneCIIg6pF7E/Z5KD6BE1MtxAHQs1QodS3VS8mOpC8+dIAiCcFMvkTtBEAQhQeJOEARRh9S0uNd6z3jG2GHG2JuMsdcZY9vsbW2MsccZY/vs/6tyZWzG2LcYY0OMsR3StpxjZ4zdZX9PbzHGrq3MqP3JcSyfZ4wdt7+b1xlj75Ieq8pjYYwtZ4z9gjG2mzG2kzF2h7295r6XPMdSi99LiDH2EmNsu30sX7C3l/d74ZzX5D9YM18PAFgDIABgO4BNlR7XHI/hMIAOz7a/B/BZ+/ZnAfxdpceZY+yXAdgCYEehscPq6b8dQBDAavt7Uyt9DAWO5fMAPuOzb9UeC4AeAFvs240A9trjrbnvJc+x1OL3wgA02Ld1AC8CuLDc30stR+712jP+RgD32rfvBfCeyg0lN5zzXwI46dmca+w3Arifc57gnB8CsB/W91cV5DiWXFTtsXDOBzjnr9q3pwDshtVqu+a+lzzHkotqPhbOOZ+27+r2P44yfy+1LO4Fe8bXABzAzxljrzDGbrO3dXPOBwDrBw6gq2Kjmzu5xl6r39XvMcbesG0bcclcE8fCGFsF4FxYUWJNfy+eYwFq8HthjKmMsdcBDAF4nHNe9u+llsW9YM/4GuASzvkWWMsR3s4Yu6zSAyoTtfhd/QuAtQDOATAA4Mv29qo/FsZYA4AfAriTcz6Zb1efbdV+LDX5vXDODc75ObDan5/PGDsjz+4lOZZaFvea7xnPOe+3/x8C8GNYl16DjLEeALD/H6rcCOdMrrHX3HfFOR+0/yBNAP+GzGVxVR8LY0yHJYb3cc5/ZG+uye/F71hq9XsRcM7HATwN4DqU+XupZXGv6Z7xjLEoY6xR3AbwDgA7YB3DLfZutwB4qDIjnBe5xv4wgJsZY0HG2GoA6wG8VIHxFY34o7N5L6zvBqjiY2GMMQDfBLCbc/4V6aGa+15yHUuNfi+djLEW+3YYwNUA9qDc30ulM8kLzEK/C1YW/QCAz1V6PHMc+xpYGfHtAHaK8QNoB/AkgH32/22VHmuO8X8P1mVxClakcWu+sQP4nP09vQXgnZUefxHH8l0AbwJ4w/5j66n2YwHwdliX728AeN3+965a/F7yHEstfi9nAXjNHvMOAP/H3l7W74XaDxAEQdQhtWzLEARBEDkgcScIgqhDSNwJgiDqEBJ3giCIOoTEnSAIog4hcScIgqhDSNwJgiDqkP8fSGijqAmoc1sAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "alpha = 1e-4\n",
    "\n",
    "history = []\n",
    "for epoch in range(300):\n",
    "    states, actions, probs, rewards = run_episode()\n",
    "    one_hot_actions = np.eye(2)[actions.T][0]\n",
    "    gradients = one_hot_actions-probs\n",
    "    dr = discounted_rewards(rewards)\n",
    "    gradients *= dr\n",
    "    target = alpha*np.vstack([gradients])+probs\n",
    "    model.train_on_batch(states,target)\n",
    "    history.append(np.sum(rewards))\n",
    "    if epoch%100==0:\n",
    "        print(f\"{epoch} -> {np.sum(rewards)}\")\n",
    "\n",
    "plt.plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run the episode with rendering to see the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "error",
     "evalue": "display Surface quit",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31merror\u001b[0m                                     Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_44248/1459719159.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrun_episode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/tmp/ipykernel_44248/3855001447.py\u001b[0m in \u001b[0;36mrun_episode\u001b[0;34m(max_steps_per_episode, render)\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmax_steps_per_episode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m             \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m         \u001b[0maction_probs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexpand_dims\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m         \u001b[0maction\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchoice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_actions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction_probs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m                 )\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    429\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    430\u001b[0m         \u001b[0;34m\"\"\"Renders the environment.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 431\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    432\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    433\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m                 )\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/wrappers/order_enforcing.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     49\u001b[0m                 \u001b[0;34m\"set `disable_render_order_enforcing=True` on the OrderEnforcer wrapper.\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     50\u001b[0m             )\n\u001b[0;32m---> 51\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     52\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m                 )\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    429\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    430\u001b[0m         \u001b[0;34m\"\"\"Renders the environment.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 431\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    432\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    433\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m                 )\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/wrappers/env_checker.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     53\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0menv_render_passive_checker\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 55\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m                 )\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/envs/classic_control/cartpole.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    215\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_renders\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    216\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 217\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_render\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    218\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    219\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_render\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"human\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.10/site-packages/gym/envs/classic_control/cartpole.py\u001b[0m in \u001b[0;36m_render\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    296\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    297\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msurf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpygame\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msurf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 298\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscreen\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mblit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msurf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    299\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"human\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    300\u001b[0m             \u001b[0mpygame\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpump\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31merror\u001b[0m: display Surface quit"
     ]
    }
   ],
   "source": [
    "_ = run_episode(render=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully, you can see that pole can now balance pretty well!\n",
    "\n",
    "## Actor-Critic Model\n",
    "\n",
    "Actor-Critic model is the further development of policy gradients, in which we build a neural network to learn both the policy and estimated rewards. The network will have two outputs (or you can view it as two separate networks):\n",
    "* **Actor** will recommend the action to take by giving us the state probability distribution, as in policy gradient model\n",
    "* **Critic** would estimate what the reward would be from those actions. It returns total estimated rewards in the future at the given state.\n",
    "\n",
    "Let's define such a model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_inputs = 4\n",
    "num_actions = 2\n",
    "num_hidden = 128\n",
    "\n",
    "inputs = keras.layers.Input(shape=(num_inputs,))\n",
    "common = keras.layers.Dense(num_hidden, activation=\"relu\")(inputs)\n",
    "action = keras.layers.Dense(num_actions, activation=\"softmax\")(common)\n",
    "critic = keras.layers.Dense(1)(common)\n",
    "\n",
    "model = keras.Model(inputs=inputs, outputs=[action, critic])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would need to slightly modify our `run_episode` function to return also critic results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_episode(max_steps_per_episode = 10000,render=False):    \n",
    "    states, actions, probs, rewards, critic = [],[],[],[],[]\n",
    "    state = env.reset()\n",
    "    for _ in range(max_steps_per_episode):\n",
    "        if render:\n",
    "            env.render()\n",
    "        action_probs, est_rew = model(np.expand_dims(state,0))\n",
    "        action = np.random.choice(num_actions, p=np.squeeze(action_probs[0]))\n",
    "        nstate, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            break\n",
    "        states.append(state)\n",
    "        actions.append(action)\n",
    "        probs.append(tf.math.log(action_probs[0,action]))\n",
    "        rewards.append(reward)\n",
    "        critic.append(est_rew[0,0])\n",
    "        state = nstate\n",
    "    return states, actions, probs, rewards, critic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will run the main training loop. We will use manual network training process by computing proper loss functions and updating network parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running reward: 5.82 at episode 10\n",
      "running reward: 9.43 at episode 20\n",
      "running reward: 10.30 at episode 30\n",
      "running reward: 10.28 at episode 40\n",
      "running reward: 11.00 at episode 50\n",
      "running reward: 13.01 at episode 60\n",
      "running reward: 21.78 at episode 70\n",
      "running reward: 40.54 at episode 80\n",
      "running reward: 73.70 at episode 90\n",
      "running reward: 100.19 at episode 100\n",
      "running reward: 159.20 at episode 110\n",
      "Solved at episode 114!\n"
     ]
    }
   ],
   "source": [
    "optimizer = keras.optimizers.Adam(learning_rate=0.01)\n",
    "huber_loss = keras.losses.Huber()\n",
    "episode_count = 0\n",
    "running_reward = 0\n",
    "\n",
    "while True:  # Run until solved\n",
    "    state = env.reset()\n",
    "    episode_reward = 0\n",
    "    with tf.GradientTape() as tape:\n",
    "        _,_,action_probs, rewards, critic_values = run_episode()\n",
    "        episode_reward = np.sum(rewards)\n",
    "        \n",
    "        # Update running reward to check condition for solving\n",
    "        running_reward = 0.05 * episode_reward + (1 - 0.05) * running_reward\n",
    "\n",
    "        # Calculate discounted rewards that will be labels for our critic\n",
    "        dr = discounted_rewards(rewards)\n",
    "\n",
    "        # Calculating loss values to update our network\n",
    "        actor_losses = []\n",
    "        critic_losses = []\n",
    "        for log_prob, value, rew in zip(action_probs, critic_values, dr):\n",
    "            # When we took the action with probability `log_prob`, we received discounted reward of `rew`,\n",
    "            # while critic predicted it to be `value` \n",
    "            # First we calculate actor loss, to make actor predict actions that lead to higher rewards\n",
    "            diff = rew - value\n",
    "            actor_losses.append(-log_prob * diff)\n",
    "\n",
    "            # The critic loss is to minimize the difference between predicted reward `value` and actual\n",
    "            # discounted reward `rew`\n",
    "            critic_losses.append(\n",
    "                huber_loss(tf.expand_dims(value, 0), tf.expand_dims(rew, 0))\n",
    "            )\n",
    "\n",
    "        # Backpropagation\n",
    "        loss_value = sum(actor_losses) + sum(critic_losses)\n",
    "        grads = tape.gradient(loss_value, model.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "\n",
    "    # Log details\n",
    "    episode_count += 1\n",
    "    if episode_count % 10 == 0:\n",
    "        template = \"running reward: {:.2f} at episode {}\"\n",
    "        print(template.format(running_reward, episode_count))\n",
    "\n",
    "    if running_reward > 195:  # Condition to consider the task solved\n",
    "        print(\"Solved at episode {}!\".format(episode_count))\n",
    "        break\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run the episode and see how good our model is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = run_episode(render=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Takeaway\n",
    "\n",
    "We have seen two RL algorithms in this demo: simple policy gradient, and more sophisticated actor-critic. You can see that those algorithms operate with abstract notions of state, action and reward - thus they can be applied to very different environments.\n",
    "\n",
    "Reinforcement learning allows us to learn the best strategy to solve the problem just by looking at the final reward. The fact that we do not need labelled datasets allows us to repeat simulations many times to optimize our models. However, there are still many challenges in RL, which you may learn if you decide to focus more on this interesting area of AI.   "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.4 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}