t81_558_class_06_4_embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hRuM9Pg2oBZQ"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_06_4_embedding.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nPIDsF57oBZR"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 11: Natural Language Processing and Speech Recognition**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4iEXcrLBoBZS"
   },
   "source": [
    "# Module 6 Material\n",
    "\n",
    "* **Part 6.1: Introduction to Transformers** [[Video]](https://www.youtube.com/watch?v=mn6r5PYJcu0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_1_transformers.ipynb)\n",
    "* Part 6.2: Accessing the ChatGPT API [[Video]](https://www.youtube.com/watch?v=tcdscXl4o5w&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_2_chat_gpt.ipynb)\n",
    "* Part 6.3: Llama, Alpaca, and LORA [[Video]](https://www.youtube.com/watch?v=oGQ3TQx1Qs8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_3_alpaca_lora.ipynb)\n",
    "* **Part 6.4: Introduction to Embeddings** [[Video]](https://www.youtube.com/watch?v=e6kcs9Uj_ps&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_4_embedding.ipynb)\n",
    "* Part 6.5: Prompt Engineering [[Video]](https://www.youtube.com/watch?v=miTpIDR7k6c&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_5_prompt_engineering.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "92qJtJOToBZS"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LvTVJyBMoBZS",
    "outputId": "13a4c525-a091-4ede-98dc-b814c34e0f44"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    from google.colab import drive\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wTUM5nqjoBZT"
   },
   "source": [
    "# Part 6.4: What are Embedding Layers in PyTorch\n",
    "\n",
    "[Embedding Layers](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html) are a handy feature of PyTorch that allows the program to automatically insert additional information into the data flow of your neural network. An embedding layer would automatically allow you to insert vectors in the place of word indexes.  \n",
    "\n",
    "Programmers often use embedding layers with Natural Language Processing (NLP); however, you can use these layers when you wish to insert a lengthier vector in an index value place. In some ways, you can think of an embedding layer as dimension expansion. However, the hope is that these additional dimensions provide more information to the model and provide a better score.\n",
    "\n",
    "## Simple Embedding Layer Example\n",
    "\n",
    "* **num_embeddings** = How large is the vocabulary?  How many categories are you encoding? This parameter is the number of items in your \"lookup table.\"\n",
    "* **embedding_dim** = How many numbers in the vector you wish to return. \n",
    "\n",
    "Now we create a neural network with a vocabulary size of 10, which will reduce those values between 0-9 to 4 number vectors. This neural network does nothing more than passing the embedding on to the output. But it does let us see what the embedding is doing. Each feature vector coming in will have two such features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "CZk4DnzDoBZU"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "embedding_layer = nn.Embedding(num_embeddings=10, embedding_dim=4)\n",
    "optimizer = torch.optim.Adam(embedding_layer.parameters(), lr=0.001)\n",
    "loss_function = nn.MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eXHSrr3poBZU"
   },
   "source": [
    "Let's take a look at the structure of this neural network to see what is happening inside it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kqawbb_PoBZU",
    "outputId": "72fc2a6d-0656-455a-be6a-4e235e00d35d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embedding(10, 4)\n"
     ]
    }
   ],
   "source": [
    "print(embedding_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6kh6wKszoBZV"
   },
   "source": [
    "For this neural network, which is just an embedding layer, the input is a vector of size 2. These two inputs are integer numbers from 0 to 9 (corresponding to the requested input_dim quantity of 10 values). Looking at the summary above, we see that the embedding layer has 40 parameters. This value comes from the embedded lookup table that contains four amounts (output_dim) for each of the 10 (input_dim) possible integer values for the two inputs. The output is 2 (input_length) length 4 (output_dim) vectors, resulting in a total output size of 8, which corresponds to the Output Shape given in the summary above.\n",
    "\n",
    "Now, let us query the neural network with two rows. The input is two integer values, as was specified when we created the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iRBKtw-_oBZV",
    "outputId": "d316ef2f-dbb6-46a5-fa18-d145d9003021"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 2])\n",
      "tensor([[[ 0.1541,  0.5140,  1.0844, -0.4065],\n",
      "         [ 1.3428, -2.2125, -0.8158, -1.7568]]], grad_fn=<EmbeddingBackward0>)\n"
     ]
    }
   ],
   "source": [
    "input_tensor = torch.tensor([[1, 2]], dtype=torch.long)\n",
    "pred = embedding_layer(input_tensor)\n",
    "\n",
    "print(input_tensor.shape)\n",
    "print(pred)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DRfFids1oBZV"
   },
   "source": [
    "Here we see two length-4 vectors that PyTorch looked up for each input integer. Recall that Python arrays are zero-based. PyTorch replaced the value of 1 with the second row of the 10 x 4 lookup matrix. Similarly, PyTorch returned the value of 2 by the third row of the lookup matrix. The following code displays the lookup matrix in its entirety. The embedding layer performs no mathematical operations other than inserting the correct row from the lookup table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YStGrTSKoBZW",
    "outputId": "9c28d583-fa9e-472b-dd2a-e3325348b56e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.6017e-03,  4.9949e-01, -6.1325e-01,  5.3117e-01],\n",
       "        [ 1.5415e-01,  5.1404e-01,  1.0844e+00, -4.0654e-01],\n",
       "        [ 1.3428e+00, -2.2125e+00, -8.1585e-01, -1.7568e+00],\n",
       "        [-1.3535e+00,  9.9905e-01,  3.0198e-01,  2.0349e+00],\n",
       "        [ 4.8691e-01, -1.2690e+00, -1.2903e-01, -7.8702e-01],\n",
       "        [-2.1178e+00, -1.4623e+00,  1.0718e-01, -5.1232e-01],\n",
       "        [-1.3009e+00,  7.6286e-02, -3.6805e+00,  1.0944e+00],\n",
       "        [-1.2939e+00, -2.3363e-01,  8.1395e-01,  9.0485e-01],\n",
       "        [ 8.2642e-01, -5.8817e-01,  9.4454e-01, -1.7300e+00],\n",
       "        [-6.7173e-01,  1.1404e+00, -1.1704e-01,  2.9404e-02]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_layer.weight.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y0QLcjnwoBZW"
   },
   "source": [
    "The values above are random parameters that PyTorch generated as starting points.  Generally, we will transfer an embedding or train these random values into something useful.  The following section demonstrates how to embed a hand-coded embedding. \n",
    "\n",
    "## Transferring An Embedding\n",
    "\n",
    "Now, we see how to hard-code an embedding lookup that performs a simple one-hot encoding.  One-hot encoding would transform the input integer values of 0, 1, and 2 to the vectors $[1,0,0]$, $[0,1,0]$, and $[0,0,1]$ respectively. The following code replaced the random lookup values in the embedding layer with this one-hot coding-inspired lookup table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "AIt-SWK6oBZW"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# Define the embedding lookup matrix\n",
    "embedding_lookup = torch.tensor([\n",
    "    [1, 0, 0],\n",
    "    [0, 1, 0],\n",
    "    [0, 0, 1]\n",
    "], dtype=torch.float32)  # Make sure to use float32 for weight matrices\n",
    "\n",
    "# Create the embedding layer\n",
    "embedding_layer = nn.Embedding(num_embeddings=3, embedding_dim=3)\n",
    "\n",
    "# Set the weights of the embedding layer\n",
    "embedding_layer.weight.data = embedding_lookup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oQ_bVdtkoBZW"
   },
   "source": [
    "We have the following parameters for the Embedding layer:\n",
    "    \n",
    "* input_dim=3 - There are three different integer categorical values allowed.\n",
    "* output_dim=3 - Three columns represent a categorical value with three possible values per one-hot encoding.\n",
    "* input_length=2 - The input vector has two of these categorical values.\n",
    "\n",
    "We query the neural network with two categorical values to see the lookup performed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ufBqgk6joBZW",
    "outputId": "3e54206d-3c56-47b8-ab17-98b1c1111ec3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 2])\n",
      "tensor([[[1., 0., 0.],\n",
      "         [0., 1., 0.]]], grad_fn=<EmbeddingBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Create the input tensor directly in PyTorch\n",
    "input_tensor = torch.tensor([[0, 1]], dtype=torch.long)\n",
    "\n",
    "# Forward pass to get the predictions\n",
    "pred = embedding_layer(input_tensor)\n",
    "\n",
    "print(input_tensor.shape)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pX76cE9HoBZW"
   },
   "source": [
    "The given output shows that we provided the program with two rows from the one-hot encoding table. This encoding is a correct one-hot encoding for the values 0 and 1, where there are up to 3 unique values possible. \n",
    "\n",
    "The following section demonstrates how to train this embedding lookup table.\n",
    "\n",
    "## Training an Embedding\n",
    "\n",
    "First, we make use of the following imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "mb-nwgKhoBZX"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from torch.nn.utils.rnn import pad_sequence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dJ7UlNZfoBZX"
   },
   "source": [
    "We create a neural network that classifies restaurant reviews according to positive or negative.  This neural network can accept strings as input, such as given here.  This code also includes positive or negative labels for each review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "vr20wr0ZoBZX"
   },
   "outputs": [],
   "source": [
    "# Define 10 resturant reviews.\n",
    "reviews = [\n",
    "    'Never coming back!',\n",
    "    'Horrible service',\n",
    "    'Rude waitress',\n",
    "    'Cold food.',\n",
    "    'Horrible food!',\n",
    "    'Awesome',\n",
    "    'Awesome service!',\n",
    "    'Rocks!',\n",
    "    'poor work',\n",
    "    'Couldn\\'t have done better']\n",
    "\n",
    "# Define labels (1=negative, 0=positive)\n",
    "labels = [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ID5bpFo2oBZX"
   },
   "source": [
    "Notice that the second to the last label is incorrect.  Errors such as this are not too out of the ordinary, as most training data could have some noise.\n",
    "\n",
    "We define a vocabulary size of 50 words.  Though we do not have 50 words, it is okay to use a value larger than needed.  If there are more than 50 words, the least frequently used words in the training set are automatically dropped by the embedding layer during training.  For input, we one-hot encode the strings.  We use the TensorFlow one-hot encoding method here rather than Scikit-Learn. Scikit-learn would expand these strings to the 0's and 1's as we would typically see for dummy variables.  TensorFlow translates all words to index values and replaces each word with that index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aq0ph8hYoBZX",
    "outputId": "886877b4-2ce1-48d3-bb68-61bbd5785b42"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded reviews: [tensor([16, 18, 25]), tensor([37,  7]), tensor([24, 28]), tensor([42,  5]), tensor([37, 21]), tensor([36]), tensor([36, 42]), tensor([2]), tensor([42, 38]), tensor([19,  4, 15,  6])]\n"
     ]
    }
   ],
   "source": [
    "# One-hot encode reviews\n",
    "VOCAB_SIZE = 50\n",
    "encoded_reviews = [torch.tensor([hash(word) % VOCAB_SIZE for word in review.split()]) for review in reviews]\n",
    "\n",
    "print(f\"Encoded reviews: {encoded_reviews}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XQ2VeMHGoBZX"
   },
   "source": [
    "The program one-hot encodes these reviews to word indexes; however, their lengths are different.  We pad these reviews to 4 words and truncate any words beyond the fourth word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tfrvYtaBoBZX",
    "outputId": "877076a6-c379-4b4f-c94b-0db19ae4d61b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[16, 18, 25,  0],\n",
      "        [37,  7,  0,  0],\n",
      "        [24, 28,  0,  0],\n",
      "        [42,  5,  0,  0],\n",
      "        [37, 21,  0,  0],\n",
      "        [36,  0,  0,  0],\n",
      "        [36, 42,  0,  0],\n",
      "        [ 2,  0,  0,  0],\n",
      "        [42, 38,  0,  0],\n",
      "        [19,  4, 15,  6]])\n"
     ]
    }
   ],
   "source": [
    "MAX_LENGTH = 4\n",
    "padded_reviews = pad_sequence(encoded_reviews, batch_first=True, padding_value=0).narrow(1, 0, MAX_LENGTH)\n",
    "print(padded_reviews)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nwMJm4ztoBZX"
   },
   "source": [
    "As specified by the **padding=post** setting, each review is padded by appending zeros at the end, as specified by the **padding=post** setting.\n",
    "\n",
    "Next, we create a neural network to learn to classify these reviews. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GK2yiTFRoBZX",
    "outputId": "c23f9d36-ae89-4ef6-9e1a-788f1977b918"
   },
   "outputs": [],
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Embedding(VOCAB_SIZE, 8),\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(8 * MAX_LENGTH, 1),\n",
    "    nn.Sigmoid()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jvpQk1GXoBZX"
   },
   "source": [
    "This network accepts four integer inputs that specify the indexes of a padded movie review. The first embedding layer converts these four indexes into four length vectors 8. These vectors come from the lookup table that contains 50 (VOCAB_SIZE) rows of vectors of length 8. This encoding is evident by the 400 (8 times 50) parameters in the embedding layer. The output size from the embedding layer is 32 (4 words expressed as 8-number embedded vectors). A single output neuron is connected to the embedding layer by 33 weights (32 from the embedding layer and a single bias neuron). Because this is a single-class classification network, we use the sigmoid activation function and binary_crossentropy.\n",
    "\n",
    "The program now trains the neural network. The embedding lookup and dense 33 weights are updated to produce a better score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BkRJvhz7oBZY",
    "outputId": "49c3fb2f-ae02-4627-dee1-896e516eb446"
   },
   "outputs": [],
   "source": [
    "criterion = nn.BCELoss()  # Binary Cross Entropy\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "# Training the model\n",
    "epochs = 100\n",
    "for epoch in range(epochs):\n",
    "    optimizer.zero_grad()\n",
    "    outputs = model(padded_reviews.long())\n",
    "    loss = criterion(outputs.squeeze(), torch.tensor(labels, dtype=torch.float))\n",
    "    loss.backward()\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m8306knWoBZY"
   },
   "source": [
    "We can see the learned embeddings.  Think of each word's vector as a location in the 8 dimension space where words associated with positive reviews are close to other words.  Similarly, training places negative reviews close to each other.  In addition to the training setting these embeddings, the 33 weights between the embedding layer and output neuron similarly learn to transform these embeddings into an actual prediction.  You can see these embeddings here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bt_4XhUooBZY",
    "outputId": "a1e0e5f5-ca83-45dc-ae17-9277b46ccf34"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([50, 8])\n",
      "Parameter containing:\n",
      "tensor([[-6.6702e-01,  7.9332e-01,  1.4016e+00, -4.5947e-01, -6.4406e-01,\n",
      "         -4.1454e-01, -1.0128e+00,  6.8736e-01],\n",
      "        [ 5.0813e-01,  1.5599e+00, -1.1821e-01,  4.0191e-01, -1.4427e-01,\n",
      "          4.3327e-01,  7.3892e-01,  7.8627e-01],\n",
      "        [-1.4003e+00,  1.1563e+00, -6.7240e-02, -1.9336e+00,  2.6581e-01,\n",
      "         -1.0005e+00,  1.7278e+00,  7.4226e-01],\n",
      "        [ 1.5777e+00, -5.7344e-01, -6.9826e-01, -4.1497e-01,  3.2424e-01,\n",
      "          9.6881e-01,  4.2793e-01,  1.2810e+00],\n",
      "        [ 1.9169e-01,  2.5659e-01, -1.0394e+00, -4.2579e-01, -4.9159e-01,\n",
      "          2.6051e-01,  1.2269e-01,  3.4496e-02],\n",
      "        [ 1.1741e+00,  6.4354e-01, -8.8106e-01, -1.8514e+00, -8.2117e-01,\n",
      "          1.3525e-01,  5.6363e-01,  3.2708e+00],\n",
      "        [ 8.8823e-01, -7.7813e-01,  1.1075e+00, -2.7059e-01,  1.2459e-01,\n",
      "          6.8583e-01, -1.4626e+00, -5.6774e-01],\n",
      "        [-1.0156e+00, -8.9942e-01, -7.3420e-01,  1.8556e+00, -2.2097e+00,\n",
      "         -2.2803e-01,  1.0830e+00, -1.4904e+00],\n",
      "        [-1.0521e-01, -3.8405e-01, -5.3452e-01,  6.8482e-01,  1.7273e-01,\n",
      "         -7.9672e-01,  1.9246e+00, -7.4732e-01],\n",
      "        [ 4.3600e-01,  1.3698e+00, -1.2133e-01, -2.2614e-01,  1.8918e+00,\n",
      "          1.3579e+00,  9.1514e-01,  9.2330e-01],\n",
      "        [-1.6008e+00,  4.0155e-01, -1.3538e-02, -1.9124e+00,  4.1689e-01,\n",
      "         -5.6815e-01,  6.2543e-01, -4.6442e-01],\n",
      "        [ 1.7123e+00,  2.9109e-01, -1.7047e+00,  8.8100e-02,  1.0011e+00,\n",
      "          1.7266e+00,  1.3843e+00, -8.1988e-02],\n",
      "        [-3.7716e-01,  4.3761e-01, -7.8938e-02, -9.2456e-01,  1.4867e-01,\n",
      "          1.0113e+00,  4.7175e-02,  1.7339e+00],\n",
      "        [-8.5024e-01, -6.7561e-01, -1.4018e+00, -2.2150e-01,  1.3553e+00,\n",
      "          1.5159e-01,  6.7301e-01,  3.9054e-01],\n",
      "        [ 6.3771e-01, -1.2871e+00,  2.7904e+00,  9.0033e-01, -2.3311e-01,\n",
      "          6.1338e-02,  1.0261e+00,  8.8827e-01],\n",
      "        [ 1.9374e+00, -7.8647e-01,  4.7993e-02,  1.0075e+00,  2.7917e+00,\n",
      "          4.9331e-01, -5.1282e-01, -1.5207e-01],\n",
      "        [ 1.8961e+00, -5.6269e-01,  1.2642e-01, -4.7384e-01, -1.3468e+00,\n",
      "         -1.9411e+00, -8.3413e-01, -2.0996e+00],\n",
      "        [ 1.5152e-01, -1.1077e+00, -1.3681e+00, -1.9511e+00, -7.4823e-01,\n",
      "          9.3593e-01,  7.9611e-01, -1.5027e+00],\n",
      "        [-7.9450e-01, -3.8764e-01, -6.0665e-01,  7.4525e-01,  2.0712e-01,\n",
      "          2.1966e+00,  1.7216e+00, -1.5077e-01],\n",
      "        [-7.0564e-01, -5.0741e-01, -7.7887e-01, -1.9045e-01, -2.6549e+00,\n",
      "          9.3792e-01, -1.0562e+00,  1.5715e-01],\n",
      "        [ 1.8277e-01,  3.9739e-01,  1.1895e-01,  5.6093e-01, -2.0330e-01,\n",
      "          2.5441e-01, -3.8130e-01, -2.2621e+00],\n",
      "        [ 3.7933e-01, -7.2717e-01,  1.3971e+00, -1.5541e+00, -5.1660e-01,\n",
      "         -1.4806e-01, -5.4782e-01,  2.6070e-01],\n",
      "        [-5.4112e-01, -7.5402e-01, -3.1927e-01, -5.3326e-01,  3.0431e-01,\n",
      "          5.7911e-01,  1.0965e+00, -4.0861e-01],\n",
      "        [ 4.7599e-01,  7.1882e-01, -2.2762e+00,  8.9179e-01,  1.6638e-01,\n",
      "         -4.4428e-02,  7.8429e-01, -2.5079e-01],\n",
      "        [ 5.5856e-01, -1.9664e-02, -1.7946e+00,  4.6495e-01, -1.4472e+00,\n",
      "         -2.5688e-01,  8.4403e-01, -2.6500e-01],\n",
      "        [ 7.7538e-01,  3.6602e-01,  7.5876e-01,  5.6585e-01, -3.5565e-01,\n",
      "          1.4483e+00, -2.1903e-01, -1.5637e+00],\n",
      "        [-3.0146e-01, -1.5962e-01, -7.3492e-03, -9.4069e-02, -6.8921e-01,\n",
      "         -2.1429e-01, -5.9890e-01,  1.3138e+00],\n",
      "        [ 1.1876e-01, -1.5687e+00, -6.8103e-02, -1.7152e+00,  9.5150e-01,\n",
      "          7.8378e-01,  1.9705e-01,  3.5594e-01],\n",
      "        [ 1.8070e+00, -1.5244e+00,  2.6259e-01,  1.4963e+00,  1.1206e+00,\n",
      "         -6.3902e-01,  1.5162e+00,  1.6231e+00],\n",
      "        [ 6.7195e-01, -5.9119e-01,  5.0575e-01,  3.2938e-01,  3.7329e-01,\n",
      "         -2.6238e-01,  3.1243e-01, -1.3216e+00],\n",
      "        [-1.2375e-01,  2.1892e+00, -7.8851e-01,  5.7490e-02,  1.2776e-01,\n",
      "         -1.1435e+00,  1.5062e-01, -1.6976e-02],\n",
      "        [-1.6501e+00,  1.6578e-01,  1.7706e-01,  9.8052e-01, -9.5547e-01,\n",
      "         -1.0818e+00,  1.0567e+00, -1.0973e+00],\n",
      "        [ 6.6054e-01,  5.7431e-01,  6.8836e-01,  2.6019e-01,  5.6870e-01,\n",
      "          5.1099e-02,  3.5230e-01,  2.2851e-01],\n",
      "        [ 1.2754e+00,  9.2134e-01, -1.2935e-01,  2.7038e-02,  9.0194e-01,\n",
      "          1.0162e+00, -5.0666e-01,  1.3711e+00],\n",
      "        [-3.9027e-01,  6.6352e-01, -1.2921e-02, -1.0271e+00, -6.5763e-01,\n",
      "          8.6408e-01, -1.1467e+00,  8.6446e-01],\n",
      "        [-1.8190e+00,  4.7725e-01,  5.4321e-02,  1.7555e-01, -8.2536e-01,\n",
      "         -6.2547e-01,  5.0091e-01, -3.8092e-01],\n",
      "        [ 1.2992e+00, -1.4612e+00,  2.6087e-01,  1.1669e-01,  1.4442e+00,\n",
      "         -1.4732e+00,  1.9373e-01,  8.9084e-01],\n",
      "        [ 5.8313e-02, -4.1466e-01, -3.3735e-02,  9.0895e-01,  7.3321e-02,\n",
      "          5.1023e-01,  1.2149e+00,  2.5479e+00],\n",
      "        [-8.5097e-03, -2.5438e-01, -1.1571e+00,  1.9788e-01,  2.4154e-01,\n",
      "         -3.0632e-01, -1.6056e+00,  1.5104e-01],\n",
      "        [-1.1181e+00,  8.7565e-01,  1.5133e+00, -1.2652e+00, -7.1202e-01,\n",
      "         -4.6201e-01, -2.0504e-01,  1.5024e+00],\n",
      "        [ 1.1408e+00,  1.9478e+00,  9.2127e-01,  1.5719e+00, -2.1558e+00,\n",
      "          7.1428e-01, -1.2282e+00,  1.1108e+00],\n",
      "        [-2.8547e-01, -5.4418e-01,  5.3058e-01, -9.3545e-01,  1.5527e-01,\n",
      "          6.2774e-01, -2.9297e-01,  1.0583e-03],\n",
      "        [-3.0681e-02,  2.3530e+00, -1.2075e+00,  1.5494e+00, -1.1206e+00,\n",
      "          6.0487e-01,  6.4096e-02,  3.2888e-01],\n",
      "        [-1.3945e-01,  3.0064e-01, -3.8944e-01, -7.9335e-02,  5.5942e-01,\n",
      "          2.1926e-01, -7.1905e-02, -1.5605e+00],\n",
      "        [-9.5522e-01,  9.8357e-01,  3.7635e-01,  2.9963e-02,  7.6076e-01,\n",
      "          8.8953e-01,  1.4897e-01,  8.8721e-01],\n",
      "        [ 2.1536e-01, -1.0406e-01, -6.0218e-01,  3.0429e-01, -1.7406e+00,\n",
      "         -1.9792e+00,  2.5653e+00,  2.4415e-01],\n",
      "        [ 4.1522e-01,  5.4550e-01,  2.3991e-01,  8.2152e-01,  3.0893e-01,\n",
      "          7.5649e-01, -7.9431e-01,  8.3140e-01],\n",
      "        [ 9.7768e-02, -6.1646e-01, -1.3989e+00, -8.0681e-02,  1.0449e+00,\n",
      "         -4.7097e-01,  1.9099e-01,  1.8727e+00],\n",
      "        [-1.2126e+00,  2.4311e-01, -1.5341e-01, -1.1258e-01, -8.9832e-03,\n",
      "         -9.9758e-01, -3.7556e-01, -2.8560e-01],\n",
      "        [-6.0017e-01, -8.3009e-01,  1.2779e+00, -1.4838e+00, -1.3227e+00,\n",
      "         -5.0677e-01,  1.6180e+00, -1.6004e+00]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "embedding_weights = list(model[0].parameters())[0]\n",
    "print(embedding_weights.shape)\n",
    "print(embedding_weights)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X7sIWHLvoBZY"
   },
   "source": [
    "We can now evaluate this neural network's accuracy, including the embeddings and the learned dense layer.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "of3oQrPxoBZY",
    "outputId": "d33fc5a1-97bc-4b55-ed99-f2ed4a7e3999"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.8999999761581421\n",
      "Log-loss: 0.4487539827823639\n"
     ]
    }
   ],
   "source": [
    "# Evaluation\n",
    "with torch.no_grad():\n",
    "    outputs = model(padded_reviews.long())\n",
    "    predictions = (outputs > 0.5).float().squeeze()\n",
    "    accuracy = (predictions == torch.tensor(labels)).float().mean().item()\n",
    "    loss_value = criterion(outputs.squeeze(), torch.tensor(labels, dtype=torch.float)).item()\n",
    "\n",
    "print(f'Accuracy: {accuracy}')\n",
    "print(f'Log-loss: {loss_value}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PMmvVz92oBZY"
   },
   "source": [
    "The accuracy is great, but there could be overfitting. It would be good to use early stopping to not overfit for a more complex data set. However, the loss is not perfect. Even though the predicted probabilities indicated a correct prediction in every case, the program did not achieve absolute confidence in each correct answer. The lack of confidence was likely due to the small amount of noise (previously discussed) in the data set. Some words that appeared in both positive and negative reviews contributed to this lack of absolute certainty. "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "t81_558_class_11_05_embedding.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}