Neural_Language_Model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Neural Language Model",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9kTKaVK47knP"
      },
      "source": [
        "# Neural Language Model\n",
        "A language model predicts the next word in the sequence based on the specific words that have come before it in the sequence.\n",
        "\n",
        "It is also possible to develop language models at the character level using neural networks. The benefit of character-based language models is their small vocabulary and flexibility in handling any words, punctuation, and other document structure. This comes at the cost of requiring larger models that are slower to train.\n",
        "\n",
        "Nevertheless, in the field of neural language models, character-based models offer a lot of promise for a general, flexible and powerful approach to language modeling."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QK2Ew8_C7knQ"
      },
      "source": [
        "As a prerequisite for the lab, make sure to pip install:\n",
        "- keras\n",
        "- tensorflow\n",
        "- h5py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xIs5eFI97knR"
      },
      "source": [
        "# Source Text Creation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zQku8-L97knT"
      },
      "source": [
        "To start out with, we'll be using a simple nursery rhyme. It's quite short so we can actually train something on your CPU and see relatively interesting results. Please copy and paste the following text in a text file and save it as \"rhyme.txt\". Place this in the same directory as this jupyter notebook:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6wl1ZLGG7wii",
        "outputId": "42e46f30-f9f6-489e-b219-58a10fb441d7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 605
        }
      },
      "source": [
        "!pip install tensorflow\n",
        "!pip install keras\n",
        "!pip install h5py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: tensorflow in /usr/local/lib/python3.6/dist-packages (1.15.0)\n",
            "Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.15.0)\n",
            "Requirement already satisfied: numpy<2.0,>=1.16.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.17.4)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.1.0)\n",
            "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.15.1)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.11.2)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.12.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (0.1.8)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (0.8.1)\n",
            "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (0.2.2)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (0.8.1)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (3.1.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.1.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.0.8)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (0.33.6)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.15.0)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (3.10.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow) (3.1.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow) (0.16.0)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow) (42.0.2)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow) (2.8.0)\n",
            "Requirement already satisfied: keras in /usr/local/lib/python3.6/dist-packages (2.2.5)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from keras) (1.1.0)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras) (3.13)\n",
            "Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.6/dist-packages (from keras) (1.17.4)\n",
            "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from keras) (1.3.3)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras) (2.8.0)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from keras) (1.12.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from keras) (1.0.8)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (2.8.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from h5py) (1.12.0)\n",
            "Requirement already satisfied: numpy>=1.7 in /usr/local/lib/python3.6/dist-packages (from h5py) (1.17.4)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-iZwM8Ew8Weg",
        "outputId": "69c98975-1ade-492b-b00f-c03984e850ea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        }
      },
      "source": [
        "from keras.layers import Dropout"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iqQdG7M68HVt"
      },
      "source": [
        "s='Sing a song of sixpence,\\\n",
        "A pocket full of rye.\\\n",
        "Four and twenty blackbirds,\\\n",
        "Baked in a pie.\\\n",
        "When the pie was opened\\\n",
        "The birds began to sing;\\\n",
        "Wasn’t that a dainty dish,\\\n",
        "To set before the king.\\\n",
        "The king was in his counting house,\\\n",
        "Counting out his money;\\\n",
        "The queen was in the parlour,\\\n",
        "Eating bread and honey.\\\n",
        "The maid was in the garden,\\\n",
        "Hanging out the clothes,\\\n",
        "When down came a blackbird\\\n",
        "And pecked off her nose.'\n",
        "\n",
        "with open('rhymes.txt','w') as f:\n",
        "  f.write(s)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gq7wq3fF7knV"
      },
      "source": [
        "    Sing a song of sixpence,\n",
        "    A pocket full of rye.\n",
        "    Four and twenty blackbirds,\n",
        "    Baked in a pie.\n",
        "\n",
        "    When the pie was opened\n",
        "    The birds began to sing;\n",
        "    Wasn’t that a dainty dish,\n",
        "    To set before the king.\n",
        "\n",
        "    The king was in his counting house,\n",
        "    Counting out his money;\n",
        "    The queen was in the parlour,\n",
        "    Eating bread and honey.\n",
        "\n",
        "    The maid was in the garden,\n",
        "    Hanging out the clothes,\n",
        "    When down came a blackbird\n",
        "    And pecked off her nose."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tdOJmdjU7knW"
      },
      "source": [
        "# Sequence Generation\n",
        "\n",
        "A language model must be trained on the text, and in the case of a character-based language model, the input and output sequences must be characters.\n",
        "\n",
        "The number of characters used as input will also define the number of characters that will need to be provided to the model in order to elicit the first predicted character.\n",
        "\n",
        "After the first character has been generated, it can be appended to the input sequence and used as input for the model to generate the next character.\n",
        "\n",
        "Longer sequences offer more context for the model to learn what character to output next but take longer to train and impose more burden on seeding the model when generating text.\n",
        "\n",
        "We will use an arbitrary length of 10 characters for this model.\n",
        "\n",
        "There is not a lot of text, and 10 characters is a few words.\n",
        "\n",
        "We can now transform the raw text into a form that our model can learn; specifically, input and output sequences of characters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JjJfvv1U7knY"
      },
      "source": [
        "#load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# save tokens to file, one dialog per line\n",
        "def save_doc(lines, filename):\n",
        "    data = '\\n'.join(lines)\n",
        "    file = open(filename, 'w')\n",
        "    file.write(data)\n",
        "    file.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F6GbMmMs7knb",
        "outputId": "86eee3c0-ef5b-4c6a-924a-ae8d66c23e6d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "#load text\n",
        "raw_text = load_doc('rhymes.txt')\n",
        "print(raw_text)\n",
        "\n",
        "# clean\n",
        "tokens = raw_text.split()\n",
        "raw_text = ' '.join(tokens)\n",
        "\n",
        "# organize into sequences of characters\n",
        "length = 10\n",
        "sequences = list()\n",
        "for i in range(length, len(raw_text)):\n",
        "    # select sequence of tokens\n",
        "    seq = raw_text[i-length:i+1]\n",
        "    # store\n",
        "    sequences.append(seq)\n",
        "print('Total Sequences: %d' % len(sequences))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sing a song of sixpence,A pocket full of rye.Four and twenty blackbirds,Baked in a pie.When the pie was openedThe birds began to sing;Wasn’t that a dainty dish,To set before the king.The king was in his counting house,Counting out his money;The queen was in the parlour,Eating bread and honey.The maid was in the garden,Hanging out the clothes,When down came a blackbirdAnd pecked off her nose.\n",
            "Total Sequences: 384\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iBepEyDn7kne"
      },
      "source": [
        "# save sequences to file\n",
        "out_filename = 'char_sequences.txt'\n",
        "save_doc(sequences, out_filename)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ohjVbv1l7kng"
      },
      "source": [
        "# Train a Model\n",
        "In this section, we will develop a neural language model for the prepared sequence data.\n",
        "\n",
        "The model will read encoded characters and predict the next character in the sequence. A Long Short-Term Memory recurrent neural network hidden layer will be used to learn the context from the input sequence in order to make the predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7HBgtQvY7knh"
      },
      "source": [
        "from numpy import array\n",
        "from pickle import dump\n",
        "from keras.utils import to_categorical\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import LSTM\n",
        "\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r8ruXoPS7knl"
      },
      "source": [
        "# load\n",
        "\n",
        "in_filename = 'char_sequences.txt'\n",
        "raw_text = load_doc(in_filename)\n",
        "lines = raw_text.split('\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uTnvoY-f7kno"
      },
      "source": [
        "The sequences of characters must be encoded as integers.This means that each unique character will be assigned a specific integer value and each sequence of characters will be encoded as a sequence of integers. We can create the mapping given a sorted set of unique characters in the raw input data. The mapping is a dictionary of character values to integer values.\n",
        "\n",
        "Next, we can process each sequence of characters one at a time and use the dictionary mapping to look up the integer value for each character. The result is a list of integer lists.\n",
        "\n",
        "We need to know the size of the vocabulary later. We can retrieve this as the size of the dictionary mapping."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kMndzt5v7kno",
        "outputId": "d886844f-650b-4319-c0f6-33e4baa0291e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# integer encode sequences of characters\n",
        "chars = sorted(list(set(raw_text)))\n",
        "mapping = dict((c, i) for i, c in enumerate(chars))\n",
        "sequences = list()\n",
        "for line in lines:\n",
        "    # integer encode line\n",
        "    encoded_seq = [mapping[char] for char in line]\n",
        "    # store\n",
        "    sequences.append(encoded_seq)\n",
        "\n",
        "# vocabulary size\n",
        "vocab_size = len(mapping)\n",
        "print('Vocabulary Size: %d' % vocab_size)\n",
        "\n",
        "# separate into input and output\n",
        "sequences = array(sequences)\n",
        "X, y = sequences[:,:-1], sequences[:,-1]\n",
        "sequences = [to_categorical(x, num_classes=vocab_size) for x in X]\n",
        "X = array(sequences)\n",
        "y = to_categorical(y, num_classes=vocab_size)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulary Size: 38\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0qyCICfV7knx"
      },
      "source": [
        "The model is defined with an input layer that takes sequences that have 10 time steps and 38 features for the one hot encoded input sequences. Rather than specify these numbers, we use the second and third dimensions on the X input data. This is so that if we change the length of the sequences or size of the vocabulary, we do not need to change the model definition.\n",
        "\n",
        "The model has a single LSTM hidden layer with 75 memory cells. The model has a fully connected output layer that outputs one vector with a probability distribution across all characters in the vocabulary. A softmax activation function is used on the output layer to ensure the output has the properties of a probability distribution.\n",
        "\n",
        "The model is learning a multi-class classification problem, therefore we use the categorical log loss intended for this type of problem. The efficient Adam implementation of gradient descent is used to optimize the model and accuracy is reported at the end of each batch update. The model is fit for 50 training epochs.\n",
        "\n",
        "# To Do:\n",
        "- Try different numbers of memory cells\n",
        "- Try different types and amounts of recurrent and fully connected layers\n",
        "- Try different lengths of training epochs\n",
        "- Try different sequence lengths and pre-processing of data\n",
        "- Try regularization techniques such as Dropout"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tn8p4Bvw-uLq"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IpmBIRIq-9CH"
      },
      "source": [
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": false,
        "id": "K1jO9lYX7kny",
        "outputId": "bb6bf5c6-5c6c-4469-c50d-d216b4eeabf1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# define model\n",
        "model = Sequential()\n",
        "model.add(LSTM(200, input_shape=(X.shape[1], X.shape[2])))\n",
        "model.add(Dropout(0.2))\n",
        "model.add(Dense(vocab_size, activation='softmax'))\n",
        "print(model.summary())\n",
        "# compile model\n",
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "# fit model\n",
        "history=model.fit(X, y, validation_split = 0.3, epochs=80)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_5\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "lstm_5 (LSTM)                (None, 200)               191200    \n",
            "_________________________________________________________________\n",
            "dropout_5 (Dropout)          (None, 200)               0         \n",
            "_________________________________________________________________\n",
            "dense_7 (Dense)              (None, 38)                7638      \n",
            "=================================================================\n",
            "Total params: 198,838\n",
            "Trainable params: 198,838\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n",
            "Train on 268 samples, validate on 116 samples\n",
            "Epoch 1/80\n",
            "268/268 [==============================] - 2s 8ms/step - loss: 3.6143 - acc: 0.0821 - val_loss: 3.5595 - val_acc: 0.1552\n",
            "Epoch 2/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 3.4493 - acc: 0.1604 - val_loss: 3.2009 - val_acc: 0.1552\n",
            "Epoch 3/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 3.1507 - acc: 0.1604 - val_loss: 3.1222 - val_acc: 0.1552\n",
            "Epoch 4/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 3.0854 - acc: 0.1754 - val_loss: 3.1149 - val_acc: 0.1552\n",
            "Epoch 5/80\n",
            "268/268 [==============================] - 0s 986us/step - loss: 3.0417 - acc: 0.1604 - val_loss: 3.0836 - val_acc: 0.1552\n",
            "Epoch 6/80\n",
            "268/268 [==============================] - 0s 970us/step - loss: 3.0150 - acc: 0.1642 - val_loss: 3.0808 - val_acc: 0.1552\n",
            "Epoch 7/80\n",
            "268/268 [==============================] - 0s 974us/step - loss: 3.0194 - acc: 0.1642 - val_loss: 3.0708 - val_acc: 0.1552\n",
            "Epoch 8/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.9939 - acc: 0.1642 - val_loss: 3.0860 - val_acc: 0.1552\n",
            "Epoch 9/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.9767 - acc: 0.1604 - val_loss: 3.0817 - val_acc: 0.1552\n",
            "Epoch 10/80\n",
            "268/268 [==============================] - 0s 968us/step - loss: 2.9699 - acc: 0.1754 - val_loss: 3.0931 - val_acc: 0.1552\n",
            "Epoch 11/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.9379 - acc: 0.1903 - val_loss: 3.0600 - val_acc: 0.1897\n",
            "Epoch 12/80\n",
            "268/268 [==============================] - 0s 980us/step - loss: 2.8999 - acc: 0.2015 - val_loss: 3.0815 - val_acc: 0.1897\n",
            "Epoch 13/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.8672 - acc: 0.1679 - val_loss: 3.0941 - val_acc: 0.1724\n",
            "Epoch 14/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.8471 - acc: 0.1716 - val_loss: 3.1022 - val_acc: 0.1983\n",
            "Epoch 15/80\n",
            "268/268 [==============================] - 0s 977us/step - loss: 2.8249 - acc: 0.1791 - val_loss: 3.0495 - val_acc: 0.1983\n",
            "Epoch 16/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.7973 - acc: 0.1716 - val_loss: 3.1012 - val_acc: 0.1810\n",
            "Epoch 17/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.7302 - acc: 0.1978 - val_loss: 3.0805 - val_acc: 0.1810\n",
            "Epoch 18/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.7023 - acc: 0.2313 - val_loss: 3.0989 - val_acc: 0.1897\n",
            "Epoch 19/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.6708 - acc: 0.2388 - val_loss: 3.1297 - val_acc: 0.1724\n",
            "Epoch 20/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.5843 - acc: 0.2537 - val_loss: 3.1052 - val_acc: 0.1552\n",
            "Epoch 21/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.5379 - acc: 0.2537 - val_loss: 3.1069 - val_acc: 0.1983\n",
            "Epoch 22/80\n",
            "268/268 [==============================] - 0s 961us/step - loss: 2.5067 - acc: 0.2799 - val_loss: 3.1832 - val_acc: 0.2241\n",
            "Epoch 23/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.4232 - acc: 0.2873 - val_loss: 3.0932 - val_acc: 0.1724\n",
            "Epoch 24/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.4187 - acc: 0.2724 - val_loss: 3.1353 - val_acc: 0.1810\n",
            "Epoch 25/80\n",
            "268/268 [==============================] - 0s 975us/step - loss: 2.3228 - acc: 0.3470 - val_loss: 3.1697 - val_acc: 0.1724\n",
            "Epoch 26/80\n",
            "268/268 [==============================] - 0s 958us/step - loss: 2.2528 - acc: 0.3321 - val_loss: 3.1081 - val_acc: 0.1897\n",
            "Epoch 27/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.1676 - acc: 0.3582 - val_loss: 3.0956 - val_acc: 0.1810\n",
            "Epoch 28/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.1463 - acc: 0.3545 - val_loss: 3.1653 - val_acc: 0.1897\n",
            "Epoch 29/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 2.1032 - acc: 0.3246 - val_loss: 3.0847 - val_acc: 0.1810\n",
            "Epoch 30/80\n",
            "268/268 [==============================] - 0s 999us/step - loss: 2.0195 - acc: 0.3881 - val_loss: 3.1464 - val_acc: 0.2155\n",
            "Epoch 31/80\n",
            "268/268 [==============================] - 0s 976us/step - loss: 1.9472 - acc: 0.3582 - val_loss: 3.1923 - val_acc: 0.1638\n",
            "Epoch 32/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 1.8619 - acc: 0.4366 - val_loss: 3.2121 - val_acc: 0.1810\n",
            "Epoch 33/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 1.8430 - acc: 0.4590 - val_loss: 3.2010 - val_acc: 0.2069\n",
            "Epoch 34/80\n",
            "268/268 [==============================] - 0s 959us/step - loss: 1.7805 - acc: 0.4664 - val_loss: 3.2195 - val_acc: 0.1638\n",
            "Epoch 35/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 1.7013 - acc: 0.4925 - val_loss: 3.3580 - val_acc: 0.1897\n",
            "Epoch 36/80\n",
            "268/268 [==============================] - 0s 982us/step - loss: 1.6375 - acc: 0.5112 - val_loss: 3.1911 - val_acc: 0.1810\n",
            "Epoch 37/80\n",
            "268/268 [==============================] - 0s 979us/step - loss: 1.5249 - acc: 0.5336 - val_loss: 3.3681 - val_acc: 0.1724\n",
            "Epoch 38/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 1.4880 - acc: 0.5448 - val_loss: 3.2681 - val_acc: 0.1810\n",
            "Epoch 39/80\n",
            "268/268 [==============================] - 0s 958us/step - loss: 1.3684 - acc: 0.6493 - val_loss: 3.4410 - val_acc: 0.1897\n",
            "Epoch 40/80\n",
            "268/268 [==============================] - 0s 941us/step - loss: 1.3317 - acc: 0.6045 - val_loss: 3.5119 - val_acc: 0.1466\n",
            "Epoch 41/80\n",
            "268/268 [==============================] - 0s 978us/step - loss: 1.1917 - acc: 0.6381 - val_loss: 3.5061 - val_acc: 0.1724\n",
            "Epoch 42/80\n",
            "268/268 [==============================] - 0s 982us/step - loss: 1.1859 - acc: 0.6754 - val_loss: 3.4413 - val_acc: 0.2069\n",
            "Epoch 43/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 1.0836 - acc: 0.7127 - val_loss: 3.5027 - val_acc: 0.1724\n",
            "Epoch 44/80\n",
            "268/268 [==============================] - 0s 965us/step - loss: 1.0681 - acc: 0.7090 - val_loss: 3.4813 - val_acc: 0.1983\n",
            "Epoch 45/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.9854 - acc: 0.7649 - val_loss: 3.6690 - val_acc: 0.1638\n",
            "Epoch 46/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.8932 - acc: 0.7910 - val_loss: 3.6530 - val_acc: 0.1810\n",
            "Epoch 47/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.8954 - acc: 0.7649 - val_loss: 3.7244 - val_acc: 0.1552\n",
            "Epoch 48/80\n",
            "268/268 [==============================] - 0s 943us/step - loss: 0.7755 - acc: 0.8358 - val_loss: 3.8517 - val_acc: 0.1897\n",
            "Epoch 49/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.7118 - acc: 0.8582 - val_loss: 3.8169 - val_acc: 0.1552\n",
            "Epoch 50/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.6416 - acc: 0.8881 - val_loss: 3.9020 - val_acc: 0.1638\n",
            "Epoch 51/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.6181 - acc: 0.8619 - val_loss: 4.0898 - val_acc: 0.1897\n",
            "Epoch 52/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.5076 - acc: 0.9216 - val_loss: 3.9487 - val_acc: 0.1724\n",
            "Epoch 53/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.5220 - acc: 0.8918 - val_loss: 4.1488 - val_acc: 0.1810\n",
            "Epoch 54/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.5157 - acc: 0.8843 - val_loss: 4.2351 - val_acc: 0.1552\n",
            "Epoch 55/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.4815 - acc: 0.9142 - val_loss: 4.1312 - val_acc: 0.1983\n",
            "Epoch 56/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.3892 - acc: 0.9403 - val_loss: 4.1477 - val_acc: 0.1983\n",
            "Epoch 57/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.3679 - acc: 0.9478 - val_loss: 4.3125 - val_acc: 0.1897\n",
            "Epoch 58/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.3137 - acc: 0.9590 - val_loss: 4.3787 - val_acc: 0.1724\n",
            "Epoch 59/80\n",
            "268/268 [==============================] - 0s 991us/step - loss: 0.3108 - acc: 0.9627 - val_loss: 4.4546 - val_acc: 0.1724\n",
            "Epoch 60/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.2814 - acc: 0.9701 - val_loss: 4.4507 - val_acc: 0.1810\n",
            "Epoch 61/80\n",
            "268/268 [==============================] - 0s 978us/step - loss: 0.2513 - acc: 0.9776 - val_loss: 4.4457 - val_acc: 0.1724\n",
            "Epoch 62/80\n",
            "268/268 [==============================] - 0s 965us/step - loss: 0.2244 - acc: 0.9776 - val_loss: 4.5889 - val_acc: 0.1897\n",
            "Epoch 63/80\n",
            "268/268 [==============================] - 0s 948us/step - loss: 0.2099 - acc: 0.9776 - val_loss: 4.5749 - val_acc: 0.1810\n",
            "Epoch 64/80\n",
            "268/268 [==============================] - 0s 960us/step - loss: 0.1928 - acc: 0.9963 - val_loss: 4.6518 - val_acc: 0.1897\n",
            "Epoch 65/80\n",
            "268/268 [==============================] - 0s 968us/step - loss: 0.1917 - acc: 0.9813 - val_loss: 4.6645 - val_acc: 0.1638\n",
            "Epoch 66/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.1982 - acc: 0.9701 - val_loss: 4.7755 - val_acc: 0.1552\n",
            "Epoch 67/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.1601 - acc: 0.9925 - val_loss: 4.6985 - val_acc: 0.1638\n",
            "Epoch 68/80\n",
            "268/268 [==============================] - 0s 969us/step - loss: 0.1549 - acc: 0.9888 - val_loss: 4.7533 - val_acc: 0.1810\n",
            "Epoch 69/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.1472 - acc: 1.0000 - val_loss: 4.8317 - val_acc: 0.1810\n",
            "Epoch 70/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.1339 - acc: 0.9925 - val_loss: 4.8429 - val_acc: 0.1293\n",
            "Epoch 71/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.1318 - acc: 0.9963 - val_loss: 4.9198 - val_acc: 0.1638\n",
            "Epoch 72/80\n",
            "268/268 [==============================] - 0s 988us/step - loss: 0.1076 - acc: 1.0000 - val_loss: 4.9435 - val_acc: 0.1810\n",
            "Epoch 73/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.1189 - acc: 0.9888 - val_loss: 4.9159 - val_acc: 0.1466\n",
            "Epoch 74/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.1248 - acc: 0.9925 - val_loss: 4.9523 - val_acc: 0.1897\n",
            "Epoch 75/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.1130 - acc: 0.9963 - val_loss: 4.9331 - val_acc: 0.1552\n",
            "Epoch 76/80\n",
            "268/268 [==============================] - 0s 1ms/step - loss: 0.0960 - acc: 0.9963 - val_loss: 5.0186 - val_acc: 0.1810\n",
            "Epoch 77/80\n",
            "268/268 [==============================] - 0s 997us/step - loss: 0.0969 - acc: 0.9925 - val_loss: 4.9849 - val_acc: 0.1466\n",
            "Epoch 78/80\n",
            "268/268 [==============================] - 0s 951us/step - loss: 0.0865 - acc: 0.9963 - val_loss: 5.1230 - val_acc: 0.1983\n",
            "Epoch 79/80\n",
            "268/268 [==============================] - 0s 981us/step - loss: 0.0761 - acc: 0.9963 - val_loss: 5.0345 - val_acc: 0.1552\n",
            "Epoch 80/80\n",
            "268/268 [==============================] - 0s 969us/step - loss: 0.0784 - acc: 0.9963 - val_loss: 5.0217 - val_acc: 0.1724\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pl6XD3QB-795",
        "outputId": "17680c98-80bd-4a19-cc7b-ae80136c24cb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "results = model.evaluate(X_test, y_test)\n",
        "print(results)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "77/77 [==============================] - 0s 312us/step\n",
            "[1.430651663185714, 0.7922077937559648]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "raAPR9Qp7kn1"
      },
      "source": [
        "# save the model to file\n",
        "model.save('model.h5')\n",
        "# save the mapping\n",
        "dump(mapping, open('mapping.pkl', 'wb'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vv4Q4Rkf7kn4"
      },
      "source": [
        "# Generating Text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EigunCf57kn6"
      },
      "source": [
        "We must provide sequences of 10 characters as input to the model in order to start the generation process. We will pick these manually. A given input sequence will need to be prepared in the same way as preparing the training data for the model. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6UCWctow7kn-"
      },
      "source": [
        "from pickle import load\n",
        "from keras.models import load_model\n",
        "from keras.utils import to_categorical\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "# generate a sequence of characters with a language model\n",
        "def generate_seq(model, mapping, seq_length, seed_text, n_chars):\n",
        "    in_text = seed_text\n",
        "    # generate a fixed number of characters\n",
        "    for _ in range(n_chars):\n",
        "        # encode the characters as integers\n",
        "        encoded = [mapping[char] for char in in_text]\n",
        "        # truncate sequences to a fixed length\n",
        "        encoded = pad_sequences([encoded], maxlen=seq_length, truncating='pre')\n",
        "        # one hot encode\n",
        "        encoded = to_categorical(encoded, num_classes=len(mapping))\n",
        "        #encoded = encoded.reshape(1, encoded.shape[0], encoded.shape[1])\n",
        "        # predict character\n",
        "        yhat = model.predict_classes(encoded, verbose=0)\n",
        "        # reverse map integer to character\n",
        "        out_char = ''\n",
        "        for char, index in mapping.items():\n",
        "            if index == yhat:\n",
        "                out_char = char\n",
        "                break\n",
        "        # append to input\n",
        "        in_text += char\n",
        "    return in_text\n",
        "\n",
        "# load the model\n",
        "model = load_model('model.h5')\n",
        "# load the mapping\n",
        "mapping = load(open('mapping.pkl', 'rb'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PXn18EY57koA"
      },
      "source": [
        "Running the example generates three sequences of text.\n",
        "\n",
        "The first is a test to see how the model does at starting from the beginning of the rhyme. The second is a test to see how well it does at beginning in the middle of a line. The final example is a test to see how well it does with a sequence of characters never seen before."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5UhgrJYh7koB",
        "outputId": "1e1f765a-f805-4c92-f8d6-e39542a26d84",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "# test start of rhyme\n",
        "print(generate_seq(model, mapping, 10, 'Sing a son', 20))\n",
        "# test mid-line\n",
        "print(generate_seq(model, mapping, 10, 'king was i', 20))\n",
        "# test not in original\n",
        "print(generate_seq(model, mapping, 10, 'hello worl', 20))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sing a song of sixpence,A pock\n",
            "king was in his counting house\n",
            "hello worl..FFurttnradddaaidtt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UXhN-xhG7koE"
      },
      "source": [
        "If the results aren't satisfactory, try out the suggestions above or these below:\n",
        "- Padding. Update the example to provides sequences line by line only and use padding to fill out each sequence to the maximum line length.\n",
        "- Sequence Length. Experiment with different sequence lengths and see how they impact the behavior of the model.\n",
        "- Tune Model. Experiment with different model configurations, such as the number of memory cells and epochs, and try to develop a better model for fewer resources.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7kyXdm-f7koF"
      },
      "source": [
        "# Deliverables to receive credit\n",
        "\n",
        "1. (3 points) Optimize the cells above to get the model to work reasonably well on the generated sentences. Again, this is a toy problem as language models require a lot of computation... so this toy problem is great for rapid experimentation to explore different aspects of deep learning language models.\n",
        "2. (2 points) Write a function to split the text corpus file into training and validation and pipe the validation data into the model.fit() function to be able to track validation error per epoch. Lookup Keras documentation to see how this is handled.\n",
        "3. (2 points) Write a summary (methods and results) in the cells below of the different things you applied. You must include your intuitions behind what did work and what did not work well.\n",
        "4. (3 points) Do something even more interesting. Try a different source text. Train a word-level model. We'll leave it up to your creativity to explore and write a summary of your methods and results.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "njvIURzq7koG",
        "outputId": "a53ceb48-315f-42d6-e9df-42feb4b126fd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "\"summary below\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'summary below'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zBBGxt3yeB7J"
      },
      "source": [
        "I tried different combinations of hidden layers (Depth) and epochs and using Dropout at different locations. For some reason, adding another layer makes the accuracy of the test incredibly low, so I did not include it. Increasing the epochs allowed the model to train more, making the prediction better. Having it at 100, however, felt like it was too much and overtraining. Including the Dropout was in hopes of decreasing that effect."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bh9kQZnr7koK"
      },
      "source": [
        "a=\"To be, or not to be, that is the question:\\\n",
        "Whether tis nobler in the mind to suffer\\\n",
        "The slings and arrows of outrageous fortune,\\\n",
        "Or to take arms against a sea of troubles\\\n",
        "And by opposing end them. To die—to sleep,\\\n",
        "No more; and by a sleep to say we end\\\n",
        "The heart-ache and the thousand natural shocks\\\n",
        "That flesh is heir to: 'tis a consummation\\\n",
        "Devoutly to be wish'd. To die, to sleep;\\\n",
        "To sleep, perchance to dream—ay, there's the rub:\\\n",
        "For in that sleep of death what dreams may come,\\\n",
        "When we have shuffled off this mortal coil,\\\n",
        "Must give us pause—there's the respect\\\n",
        "That makes calamity of so long life.\\\n",
        "For who would bear the whips and scorns of time,\\\n",
        "Th'oppressor's wrong, the proud man's contumely,\\\n",
        "The pangs of dispriz'd love, the law's delay,\\\n",
        "The insolence of office, and the spurns\\\n",
        "That patient merit of th'unworthy takes,\\\n",
        "When he himself might his quietus make\\\n",
        "With a bare bodkin? Who would fardels bear,\\\n",
        "To grunt and sweat under a weary life,\\\n",
        "But that the dread of something after death,\\\n",
        "The undiscovere'd country, from whose bourn\\\n",
        "No traveller returns, puzzles the will,\\\n",
        "And makes us rather bear those ills we have\\\n",
        "Than fly to others that we know not of?\\\n",
        "Thus conscience does make cowards of us all,\\\n",
        "And thus the native hue of resolution\\\n",
        "Is sicklied o'er with the pale cast of thought,\\\n",
        "And enterprises of great pitch and moment\\\n",
        "With this regard their currents turn awry\\\n",
        "And lose the name of action.\"\n",
        "\n",
        "with open('poem.txt','w') as b:\n",
        "  b.write(a)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4oZaQgqliu9Z",
        "outputId": "98c12ae7-9cc0-492c-d2df-dfc7abab381d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "#load text\n",
        "raw_text1 = load_doc('poem.txt')\n",
        "print(raw_text1)\n",
        "\n",
        "# clean\n",
        "tokens1 = raw_text1.split()\n",
        "raw_text1 = ' '.join(tokens1)\n",
        "\n",
        "# organize into sequences of characters\n",
        "length = 10\n",
        "sequences = list()\n",
        "for i in range(length, len(raw_text1)):\n",
        "    # select sequence of tokens\n",
        "    seq = raw_text1[i-length:i+1]\n",
        "    # store\n",
        "    sequences.append(seq)\n",
        "print('Total Sequences: %d' % len(sequences))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "To be, or not to be, that is the question:Whether tis nobler in the mind to sufferThe slings and arrows of outrageous fortune,Or to take arms against a sea of troublesAnd by opposing end them. To die—to sleep,No more; and by a sleep to say we endThe heart-ache and the thousand natural shocksThat flesh is heir to: 'tis a consummationDevoutly to be wish'd. To die, to sleep;To sleep, perchance to dream—ay, there's the rub:For in that sleep of death what dreams may come,When we have shuffled off this mortal coil,Must give us pause—there's the respectThat makes calamity of so long life.For who would bear the whips and scorns of time,Th'oppressor's wrong, the proud man's contumely,The pangs of dispriz'd love, the law's delay,The insolence of office, and the spurnsThat patient merit of th'unworthy takes,When he himself might his quietus makeWith a bare bodkin? Who would fardels bear,To grunt and sweat under a weary life,But that the dread of something after death,The undiscovere'd country, from whose bournNo traveller returns, puzzles the will,And makes us rather bear those ills we haveThan fly to others that we know not of?Thus conscience does make cowards of us all,And thus the native hue of resolutionIs sicklied o'er with the pale cast of thought,And enterprises of great pitch and momentWith this regard their currents turn awryAnd lose the name of action.\n",
            "Total Sequences: 1363\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9vOpRsIkjK4d"
      },
      "source": [
        "# save sequences to file\n",
        "out_filename1 = 'char_sequences1.txt'\n",
        "save_doc(sequences, out_filename1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cbmrcVxljTVJ"
      },
      "source": [
        "in_filename = 'char_sequences1.txt'\n",
        "raw_text1 = load_doc(in_filename)\n",
        "lines = raw_text1.split('\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "En6IXZ_RjcPn",
        "outputId": "1326c14c-0ee3-4ac6-f6aa-80fa4c710302",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# integer encode sequences of characters\n",
        "chars = sorted(list(set(raw_text1)))\n",
        "mapping = dict((c, i) for i, c in enumerate(chars))\n",
        "sequences = list()\n",
        "for line in lines:\n",
        "    # integer encode line\n",
        "    encoded_seq = [mapping[char] for char in line]\n",
        "    # store\n",
        "    sequences.append(encoded_seq)\n",
        "\n",
        "# vocabulary size\n",
        "vocab_size = len(mapping)\n",
        "print('Vocabulary Size: %d' % vocab_size)\n",
        "\n",
        "# separate into input and output\n",
        "sequences = array(sequences)\n",
        "X, y = sequences[:,:-1], sequences[:,-1]\n",
        "sequences = [to_categorical(x, num_classes=vocab_size) for x in X]\n",
        "X = array(sequences)\n",
        "y = to_categorical(y, num_classes=vocab_size)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulary Size: 44\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vaw3JUhHjnan",
        "outputId": "33064804-0550-4c7f-d01c-b3f664086f33",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "model = Sequential()\n",
        "model.add(LSTM(200, input_shape=(X.shape[1], X.shape[2])))\n",
        "model.add(Dropout(0.2))\n",
        "model.add(Dense(vocab_size, activation='softmax'))\n",
        "print(model.summary())\n",
        "# compile model\n",
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "# fit model\n",
        "history=model.fit(X, y, validation_split = 0.3, epochs=50)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_7\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "lstm_7 (LSTM)                (None, 200)               196000    \n",
            "_________________________________________________________________\n",
            "dropout_7 (Dropout)          (None, 200)               0         \n",
            "_________________________________________________________________\n",
            "dense_9 (Dense)              (None, 44)                8844      \n",
            "=================================================================\n",
            "Total params: 204,844\n",
            "Trainable params: 204,844\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n",
            "Train on 954 samples, validate on 409 samples\n",
            "Epoch 1/50\n",
            "954/954 [==============================] - 3s 3ms/step - loss: 3.4536 - acc: 0.1300 - val_loss: 3.0893 - val_acc: 0.1002\n",
            "Epoch 2/50\n",
            "954/954 [==============================] - 1s 902us/step - loss: 3.1040 - acc: 0.1300 - val_loss: 3.0352 - val_acc: 0.1589\n",
            "Epoch 3/50\n",
            "954/954 [==============================] - 1s 953us/step - loss: 3.0536 - acc: 0.1709 - val_loss: 3.0300 - val_acc: 0.1589\n",
            "Epoch 4/50\n",
            "954/954 [==============================] - 1s 969us/step - loss: 3.0334 - acc: 0.1625 - val_loss: 3.0144 - val_acc: 0.1589\n",
            "Epoch 5/50\n",
            "954/954 [==============================] - 1s 953us/step - loss: 3.0104 - acc: 0.1604 - val_loss: 2.9973 - val_acc: 0.1589\n",
            "Epoch 6/50\n",
            "954/954 [==============================] - 1s 962us/step - loss: 2.9964 - acc: 0.1782 - val_loss: 2.9824 - val_acc: 0.1760\n",
            "Epoch 7/50\n",
            "954/954 [==============================] - 1s 940us/step - loss: 2.9569 - acc: 0.1771 - val_loss: 2.9770 - val_acc: 0.1589\n",
            "Epoch 8/50\n",
            "954/954 [==============================] - 1s 968us/step - loss: 2.9416 - acc: 0.1813 - val_loss: 2.9703 - val_acc: 0.1687\n",
            "Epoch 9/50\n",
            "954/954 [==============================] - 1s 933us/step - loss: 2.8930 - acc: 0.2159 - val_loss: 2.9061 - val_acc: 0.1932\n",
            "Epoch 10/50\n",
            "954/954 [==============================] - 1s 933us/step - loss: 2.8252 - acc: 0.2338 - val_loss: 2.8791 - val_acc: 0.2347\n",
            "Epoch 11/50\n",
            "954/954 [==============================] - 1s 983us/step - loss: 2.7394 - acc: 0.2579 - val_loss: 2.8452 - val_acc: 0.2543\n",
            "Epoch 12/50\n",
            "954/954 [==============================] - 1s 941us/step - loss: 2.6512 - acc: 0.2725 - val_loss: 2.8490 - val_acc: 0.2347\n",
            "Epoch 13/50\n",
            "954/954 [==============================] - 1s 940us/step - loss: 2.5889 - acc: 0.2841 - val_loss: 2.8119 - val_acc: 0.2641\n",
            "Epoch 14/50\n",
            "954/954 [==============================] - 1s 967us/step - loss: 2.5245 - acc: 0.2987 - val_loss: 2.7962 - val_acc: 0.2885\n",
            "Epoch 15/50\n",
            "954/954 [==============================] - 1s 909us/step - loss: 2.4400 - acc: 0.3302 - val_loss: 2.7958 - val_acc: 0.2763\n",
            "Epoch 16/50\n",
            "954/954 [==============================] - 1s 958us/step - loss: 2.3749 - acc: 0.3218 - val_loss: 2.8149 - val_acc: 0.2738\n",
            "Epoch 17/50\n",
            "954/954 [==============================] - 1s 934us/step - loss: 2.2960 - acc: 0.3470 - val_loss: 2.8028 - val_acc: 0.2616\n",
            "Epoch 18/50\n",
            "954/954 [==============================] - 1s 966us/step - loss: 2.2344 - acc: 0.3753 - val_loss: 2.7942 - val_acc: 0.2689\n",
            "Epoch 19/50\n",
            "954/954 [==============================] - 1s 921us/step - loss: 2.1485 - acc: 0.3711 - val_loss: 2.8890 - val_acc: 0.2567\n",
            "Epoch 20/50\n",
            "954/954 [==============================] - 1s 937us/step - loss: 2.0890 - acc: 0.4004 - val_loss: 2.8511 - val_acc: 0.2641\n",
            "Epoch 21/50\n",
            "954/954 [==============================] - 1s 952us/step - loss: 2.0230 - acc: 0.4099 - val_loss: 2.9097 - val_acc: 0.2543\n",
            "Epoch 22/50\n",
            "954/954 [==============================] - 1s 966us/step - loss: 1.9543 - acc: 0.4235 - val_loss: 2.8673 - val_acc: 0.2641\n",
            "Epoch 23/50\n",
            "954/954 [==============================] - 1s 987us/step - loss: 1.8853 - acc: 0.4570 - val_loss: 2.9561 - val_acc: 0.2592\n",
            "Epoch 24/50\n",
            "954/954 [==============================] - 1s 926us/step - loss: 1.8015 - acc: 0.4570 - val_loss: 2.9461 - val_acc: 0.2689\n",
            "Epoch 25/50\n",
            "954/954 [==============================] - 1s 962us/step - loss: 1.7219 - acc: 0.4822 - val_loss: 2.9764 - val_acc: 0.2469\n",
            "Epoch 26/50\n",
            "954/954 [==============================] - 1s 955us/step - loss: 1.6403 - acc: 0.5273 - val_loss: 2.9853 - val_acc: 0.2518\n",
            "Epoch 27/50\n",
            "954/954 [==============================] - 1s 980us/step - loss: 1.5680 - acc: 0.5325 - val_loss: 2.9956 - val_acc: 0.2567\n",
            "Epoch 28/50\n",
            "954/954 [==============================] - 1s 887us/step - loss: 1.5133 - acc: 0.5535 - val_loss: 3.0432 - val_acc: 0.2543\n",
            "Epoch 29/50\n",
            "954/954 [==============================] - 1s 987us/step - loss: 1.3907 - acc: 0.6174 - val_loss: 3.1795 - val_acc: 0.2616\n",
            "Epoch 30/50\n",
            "954/954 [==============================] - 1s 890us/step - loss: 1.3057 - acc: 0.6237 - val_loss: 3.2019 - val_acc: 0.2469\n",
            "Epoch 31/50\n",
            "954/954 [==============================] - 1s 920us/step - loss: 1.2233 - acc: 0.6499 - val_loss: 3.1838 - val_acc: 0.2347\n",
            "Epoch 32/50\n",
            "954/954 [==============================] - 1s 954us/step - loss: 1.1350 - acc: 0.6782 - val_loss: 3.3244 - val_acc: 0.2249\n",
            "Epoch 33/50\n",
            "954/954 [==============================] - 1s 988us/step - loss: 1.0659 - acc: 0.7117 - val_loss: 3.3456 - val_acc: 0.2127\n",
            "Epoch 34/50\n",
            "954/954 [==============================] - 1s 970us/step - loss: 0.9703 - acc: 0.7463 - val_loss: 3.4438 - val_acc: 0.2641\n",
            "Epoch 35/50\n",
            "954/954 [==============================] - 1s 979us/step - loss: 0.9122 - acc: 0.7642 - val_loss: 3.4299 - val_acc: 0.2323\n",
            "Epoch 36/50\n",
            "954/954 [==============================] - 1s 921us/step - loss: 0.8114 - acc: 0.8082 - val_loss: 3.5612 - val_acc: 0.2176\n",
            "Epoch 37/50\n",
            "954/954 [==============================] - 1s 1ms/step - loss: 0.7226 - acc: 0.8208 - val_loss: 3.6069 - val_acc: 0.2274\n",
            "Epoch 38/50\n",
            "954/954 [==============================] - 1s 974us/step - loss: 0.6595 - acc: 0.8407 - val_loss: 3.6544 - val_acc: 0.2274\n",
            "Epoch 39/50\n",
            "954/954 [==============================] - 1s 909us/step - loss: 0.5948 - acc: 0.8679 - val_loss: 3.7283 - val_acc: 0.2054\n",
            "Epoch 40/50\n",
            "954/954 [==============================] - 1s 924us/step - loss: 0.5371 - acc: 0.8784 - val_loss: 3.8036 - val_acc: 0.2200\n",
            "Epoch 41/50\n",
            "954/954 [==============================] - 1s 938us/step - loss: 0.4589 - acc: 0.9099 - val_loss: 3.9075 - val_acc: 0.2029\n",
            "Epoch 42/50\n",
            "954/954 [==============================] - 1s 932us/step - loss: 0.4337 - acc: 0.9256 - val_loss: 3.9260 - val_acc: 0.2127\n",
            "Epoch 43/50\n",
            "954/954 [==============================] - 1s 920us/step - loss: 0.3993 - acc: 0.9350 - val_loss: 4.0171 - val_acc: 0.2005\n",
            "Epoch 44/50\n",
            "954/954 [==============================] - 1s 903us/step - loss: 0.3169 - acc: 0.9581 - val_loss: 4.0734 - val_acc: 0.1932\n",
            "Epoch 45/50\n",
            "954/954 [==============================] - 1s 942us/step - loss: 0.2795 - acc: 0.9612 - val_loss: 4.0986 - val_acc: 0.1907\n",
            "Epoch 46/50\n",
            "954/954 [==============================] - 1s 890us/step - loss: 0.2626 - acc: 0.9759 - val_loss: 4.1426 - val_acc: 0.1932\n",
            "Epoch 47/50\n",
            "954/954 [==============================] - 1s 926us/step - loss: 0.2112 - acc: 0.9822 - val_loss: 4.2196 - val_acc: 0.2029\n",
            "Epoch 48/50\n",
            "954/954 [==============================] - 1s 891us/step - loss: 0.1933 - acc: 0.9790 - val_loss: 4.3293 - val_acc: 0.1736\n",
            "Epoch 49/50\n",
            "954/954 [==============================] - 1s 887us/step - loss: 0.1711 - acc: 0.9885 - val_loss: 4.3555 - val_acc: 0.1980\n",
            "Epoch 50/50\n",
            "954/954 [==============================] - 1s 954us/step - loss: 0.1493 - acc: 0.9895 - val_loss: 4.3749 - val_acc: 0.1907\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "grC8iHpCjs6B"
      },
      "source": [
        "# save the model to file\n",
        "model.save('model1.h5')\n",
        "# save the mapping\n",
        "dump(mapping, open('mapping1.pkl', 'wb'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rmNXrWyhkCII"
      },
      "source": [
        "# load the model\n",
        "model = load_model('model1.h5')\n",
        "# load the mapping\n",
        "mapping = load(open('mapping1.pkl', 'rb'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NKiSxIRdkQnU",
        "outputId": "4b36d4e6-4d52-445b-c307-6b9d5d965aec",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "# test start of rhyme\n",
        "print(generate_seq(model, mapping, 10, 'To be or', 20))\n",
        "# test mid-line\n",
        "print(generate_seq(model, mapping, 10, 'Must give us', 20))\n",
        "# test not in original\n",
        "print(generate_seq(model, mapping, 10, 'hello worl', 20))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "To be or nnd the s ordsThang\n",
            "Must give us pause—there's the r\n",
            "hello worllfffi,, therhe's to \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "75oPCi5dkd5G",
        "outputId": "d166b60b-a011-4265-dcf9-7a36c8e4eb50",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "\"summary below\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'summary below'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_KgeWFn6k_bG"
      },
      "source": [
        "I tried using the famous Hamlet poem for this. It is much longer so took longer to train. So I decreased the number of epochs. That also increased the accuracy of the validation test and was also much faster to complete. The original 80 epochs made the train accuracy all the way to 1 so was not practical. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wQEGYy_Ak-ev"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7v84IUr1k_GR"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}