LSTM Language Generator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelo generativo de texto para discursos de CFK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Las redes neuronales recurrentes (conocidas com *RNN* por sus siglas en inglés) son una arquitectura especial de redes neuronales que implementan la capacidad de poseer memoria. Este tipo de arquitectura resulta ideal para implementar sistemas generativos, es decir, modelos capaces de generar contenido.\n",
    "\n",
    "En este *notebook* en particular, vamos a realizar un modelo generativo de texto entrenado sobre los discursos de CFK desde 2007 a 2015. La red va a ser capaz de generar texto como si fuese un discurso de CFK. La generacion de texto se va a realizar **caracter por caracter**, es decir, la red produce un caracter tras de otro. La red no conoce las palabras ni las estructuras semánticas, sino que simplemente produce caracteres.\n",
    "\n",
    "La arquitectura tiene la siguiente topología (aproximada):\n",
    "\n",
    "![img](http://karpathy.github.io/assets/rnn/charseq.jpeg)\n",
    "\n",
    "La `hidden layer` es la que persiste la *memoria* de la red. Notar que para generar cada caracter siguiente, la red toma el caracter actual y la memoria.\n",
    "\n",
    "Para más información acerca de las redes neuronales recurrentes y su utilidad, proponemos leer el [siguiente excelent blog](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) de A. Karpathy, el actual director de AI de Tesla."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importando librerías"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "El siguiente comando importa las librerias requeridas por el resto del programa. Detallamos las mas importantes:\n",
    "\n",
    "* **numpy**: para el manejo en CPU de los tensores (vectores multidimensionales)\n",
    "* **tensorflow**: para construir y entrenar la red neuronal\n",
    "\n",
    "En este *notebook* no hay un modulo `utils` (donde se almacenan funciones auxiliares). Todas las funciones adicionales se van a declarar en el mismo *notebook*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import os\n",
    "from tqdm import tqdm\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cargando la data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A continuacion, vamos a cargar todos los datos (que son los discursos). Los discursos se encuentran en la carpeta `./speeches`, que fueron extraidos de la [siguiente pagina](https://es.wikisource.org/wiki/Autor:Cristina_Fern%C3%A1ndez_de_Kirchner).\n",
    "\n",
    "La unidad de dato será el párrafo. Esto es lo que la red va a aprender a generar, **un párrafo**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Definimos la minima longitud de un párrafo para ser considerado:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MIN_PARAGRAPH_LEN = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La siguiente función carga la data. Retorna una lista con todos los párrafos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data(_dir):\n",
    "    ret = []\n",
    "    for each in os.listdir(_dir):\n",
    "        full_path = os.path.join(_dir, each)\n",
    "        if each.endswith(\"txt\"):\n",
    "            with open(full_path, \"rb\") as f:\n",
    "                aux = f.read().decode(\"utf-8\").split('\\n\\n')\n",
    "                for paragraph in aux:\n",
    "                    paragraph = paragraph.strip('\\n')\n",
    "                    paragraph += '\\n'\n",
    "                    if len(paragraph) < MIN_PARAGRAPH_LEN:\n",
    "                        continue\n",
    "                    ret.append(paragraph)\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Carguemos los datos de la carpeta `./speeches`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps = load_data(\"./speeches/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A continuación, algunas métricas de la data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of Paragraphs: {}\".format(len(ps)))\n",
    "\n",
    "arr = np.asarray([len(x) for x in ps])\n",
    "\n",
    "print(\"Mean {}\".format(np.mean(arr)))\n",
    "print(\"Median {}\".format(np.median(arr)))\n",
    "print(\"Std {}\".format(np.std(arr)))\n",
    "print(\"Max {}\".format(np.max(arr)))\n",
    "print(\"Min {}\".format(np.min(arr)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Veamos algunos ejemplos de párrafos presentes en el *dataset*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(3):\n",
    "    idx = random.choice(range(len(ps)))\n",
    "    print(ps[idx])\n",
    "    print(\"----------------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Definimos la siguiente función que, dado una lista de párrafos, retorna:\n",
    "* *ret*: un *numpy array* 3-dimensional de NxLxV (N: cantidad de parrafos, L:longitud del maximo parrafo, V: tamaño del vocabulario). Cada parrafo es \"alargado\" (agregando caracteres `\\n`) para tener la longitud del parrafo mas largo. Ademas, cada caracter se *encodea* con one-hot encoding teniendo en cuenta el identificador numerico de *char_to_ix*.\n",
    "* *lens*: un *numpy array* 1-dimensional con las longitudes de cada parrafo\n",
    "* *char_to_ix*: un mapa para acceder con un caracter a un identificador asignado\n",
    "* *ix_to_char*: un mapa para acceder con un identificador al caracter correspondiente"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(paragraphs):\n",
    "    chars = set()\n",
    "    \n",
    "    for each in paragraphs:\n",
    "        chars.update(set(each))\n",
    "    \n",
    "    char_to_ix = { ch:i for i,ch in enumerate(chars) }\n",
    "    ix_to_char = { i:ch for i,ch in enumerate(chars) }\n",
    "    \n",
    "    aux = len(char_to_ix)\n",
    "    char_to_ix[\"<START>\"] = aux\n",
    "    ix_to_char[aux] = \"<START>\"\n",
    "    \n",
    "    vocab_size = len(char_to_ix)\n",
    "\n",
    "    max_p = max([len(i) for i in paragraphs]) + 1 # Plus one because of the START token\n",
    "    \n",
    "    ret = np.zeros(shape=(len(paragraphs), max_p, vocab_size), dtype=np.uint8)\n",
    "    lens = np.zeros(shape=len(paragraphs), dtype=np.uint8)\n",
    "\n",
    "    for idx, each in enumerate(paragraphs):\n",
    "        lens[idx] = len(each) + 1\n",
    "        for i in range(max_p - len(each) - 1):\n",
    "            each += '\\n'\n",
    "\n",
    "        aux = np.zeros(shape=(max_p, vocab_size))\n",
    "        aux[0][char_to_ix[\"<START>\"]] = 1\n",
    "        for i, c in enumerate(each):\n",
    "            aux[i+1][char_to_ix[c]] = 1\n",
    "        ret[idx] = aux\n",
    "        \n",
    "    return ret, lens, char_to_ix, ix_to_char"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ejecutemos la función sobre la *data*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, lens, char_to_ix, ix_to_char = preprocess(ps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definiendo el modelo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos a utilizar una implementacion de red neuronal recurrente llamada LSTM. En términos de utilidad, es similar a una red neuronal recurrente tradicional. Su arquitectura unicamente propone mejoras para prevenir problemas y tiempos de convergencia. Para más información de estas redes, leer el [siguiente post](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 64\n",
    "INPUT_SIZE = len(ix_to_char)\n",
    "\n",
    "TIMES = 32\n",
    "N_HIDDEN = 512"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A diferencia de los otros *notebooks* (el de [Simpsons](./Simpsons Classification.ipynb) y el [MNIST](MNIST Classification.ipynb)), no van a haber funciones para definir el grafo, sino que todos los nodos se definen como variables globales en las siguientes celdas.\n",
    "\n",
    "No vamos a entrar en profundidad en los detalles de implementacion de Tensorflow. Para aquello, puede recurrir a los tutoriales oficiales [aqui](https://www.tensorflow.org/tutorials/), muy simples de seguir y entender."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Los siguientes nodos definen los `placeholders` para introducir los datos en la red."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "init = tf.contrib.layers.xavier_initializer()\n",
    "x = tf.placeholder(tf.float32, shape=(None, TIMES, INPUT_SIZE), name=\"x\")\n",
    "y = tf.placeholder(tf.float32, shape=(None, TIMES, INPUT_SIZE))\n",
    "seq_len = tf.placeholder(tf.int64, shape=(None), name=\"seq_len\")\n",
    "\n",
    "x_2 = tf.unstack(x, axis=1)\n",
    "\n",
    "init_state_c_1 = tf.placeholder(tf.float32, shape=[None, N_HIDDEN], name=\"init_state_c_1\")\n",
    "init_state_h_1 = tf.placeholder(tf.float32, shape=[None, N_HIDDEN], name=\"init_state_h_1\")\n",
    "\n",
    "init_state_c_2 = tf.placeholder(tf.float32, shape=[None, N_HIDDEN], name=\"init_state_c_2\")\n",
    "init_state_h_2 = tf.placeholder(tf.float32, shape=[None, N_HIDDEN], name=\"init_state_h_2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A continuación definimos la red neuronal recurrente. En este caso, la red esta compuesta por 2 capas de LSTMs con `N_HIDDEN` unidades de neuronas cada una. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cell_1 = tf.contrib.rnn.BasicLSTMCell(N_HIDDEN)\n",
    "cell_2 = tf.contrib.rnn.BasicLSTMCell(N_HIDDEN)\n",
    "\n",
    "cell = tf.contrib.rnn.MultiRNNCell([cell_1, cell_2])\n",
    "    \n",
    "t_1 = tf.contrib.rnn.LSTMStateTuple(init_state_c_1, init_state_h_1)\n",
    "t_2 = tf.contrib.rnn.LSTMStateTuple(init_state_c_2, init_state_h_2)\n",
    "\n",
    "outputs, states = tf.contrib.rnn.static_rnn(cell, x_2, dtype=tf.float32, sequence_length=seq_len, initial_state=(t_1, t_2))\n",
    "\n",
    "states_0 = tf.nn.rnn_cell.LSTMStateTuple(tf.identity(states[0][0], name=\"states_0_c\"), tf.identity(states[0][1], name=\"states_0_h\"))\n",
    "states_1 = tf.nn.rnn_cell.LSTMStateTuple(tf.identity(states[1][0], name=\"states_1_c\"), tf.identity(states[1][1], name=\"states_1_h\"))\n",
    "\n",
    "states = (states_0, states_1)\n",
    "\n",
    "outputs_2 = tf.stack(outputs, axis=1)\n",
    "\n",
    "out = tf.layers.dense(outputs_2, units=INPUT_SIZE, kernel_initializer=init, name=\"out\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finalmente, definimos nuestro costo y las operaciones de minimización."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_softmax = tf.nn.softmax(out, name=\"out_softmax\")\n",
    "\n",
    "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=out, labels=y))\n",
    "\n",
    "tf.summary.scalar('loss', loss)\n",
    "\n",
    "merge = tf.summary.merge_all()\n",
    "\n",
    "upd = tf.train.AdamOptimizer().minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Corriendo la red"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Definimos la siguiente funcion que es capaz de correr una generación."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(max_=1000, T=None):\n",
    "\n",
    "    pred = \"<START>\"\n",
    "    \n",
    "    c_1 = np.zeros((1, N_HIDDEN))\n",
    "    h_1 = np.zeros((1, N_HIDDEN))\n",
    "    \n",
    "    c_2 = np.zeros((1, N_HIDDEN))\n",
    "    h_2 = np.zeros((1, N_HIDDEN))\n",
    "    \n",
    "    ret = []\n",
    "        \n",
    "    while True:\n",
    "        \n",
    "        in_ = np.zeros(shape=(1, TIMES, INPUT_SIZE), dtype=np.uint)\n",
    "        in_[0, 0, char_to_ix[pred]] = 1\n",
    "\n",
    "        if T is None:\n",
    "            net_out, net_states = sess.run([out_softmax, states], feed_dict={x: in_, init_state_c_1: c_1, init_state_h_1: h_1, init_state_c_2: c_2, init_state_h_2: h_2, seq_len: np.ones(shape=(1,))})\n",
    "            c_1, h_1 = net_states[0].c, net_states[0].h\n",
    "            c_2, h_2 = net_states[1].c, net_states[1].h\n",
    "            p = np.squeeze(net_out)[0]\n",
    "        else:\n",
    "            net_out, net_states = sess.run([out, states], feed_dict={x: in_, init_state_c_1: c_1, init_state_h_1: h_1, init_state_c_2: c_2, init_state_h_2: h_2, seq_len: np.ones(shape=(1,))})\n",
    "            c_1, h_1 = net_states[0].c, net_states[0].h\n",
    "            c_2, h_2 = net_states[1].c, net_states[1].h\n",
    "            p = np.squeeze(net_out)[0]\n",
    "            p = np.exp(p/T) / np.sum(np.exp(p/T))\n",
    "            \n",
    "        char_out = ix_to_char[int(np.random.choice(np.arange(INPUT_SIZE), p=p))]\n",
    "        ret.append(char_out)\n",
    "\n",
    "        pred = char_out\n",
    "                                                                         \n",
    "        if char_out == '\\n' or len(ret) > max_:\n",
    "            break\n",
    "        \n",
    "    return ret                                                                       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La red va a entrenarse con el siguiente *script*. Cada 10 epocas, el sistema va a generar un párrafo con el entrenamiento que tiene hasta ese momento."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "EPOCHS = 1000\n",
    "\n",
    "N, M, V = data.shape\n",
    "\n",
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "zeros = np.zeros(shape=(BATCH_SIZE))\n",
    "times_minus_one = (TIMES - 1) * np.ones(shape=(BATCH_SIZE))\n",
    "\n",
    "train_writer = tf.summary.FileWriter('./logs/train', sess.graph)\n",
    "\n",
    "counter = 0\n",
    "for e in tqdm(range(EPOCHS)):\n",
    "    \n",
    "    idxs = np.random.choice(N, BATCH_SIZE, replace=False)\n",
    "    batch = data[idxs]\n",
    "    batch_lens = lens[idxs].astype(np.int32)\n",
    "    \n",
    "    ts = (M-1) // TIMES # + 1\n",
    "    \n",
    "    # Initial state\n",
    "    c_1 = np.zeros((BATCH_SIZE, N_HIDDEN))\n",
    "    h_1 = np.zeros((BATCH_SIZE, N_HIDDEN))\n",
    "\n",
    "    c_2 = np.zeros((BATCH_SIZE, N_HIDDEN))\n",
    "    h_2 = np.zeros((BATCH_SIZE, N_HIDDEN))\n",
    "    \n",
    "    if e % 10 == 0:\n",
    "        print(\"\".join(test(max_=100)))\n",
    "    \n",
    "    for t in range(ts):\n",
    "        batch_x = batch[:, t*TIMES:TIMES*(t+1), :]\n",
    "        batch_y = batch[:, t*TIMES+1:TIMES*(t+1)+1, :]\n",
    "        \n",
    "        batch_lens_aux = batch_lens -  (TIMES * t)\n",
    "        \n",
    "        batch_lens_aux = np.maximum(zeros, batch_lens_aux)\n",
    "        batch_lens_aux = np.minimum(times_minus_one, batch_lens_aux)\n",
    "        \n",
    "        batch_lens_aux = batch_lens_aux.astype(np.uint8)\n",
    "        \n",
    "        non_zero_idxs = batch_lens_aux > 0\n",
    "        batch_lens_aux = batch_lens_aux[non_zero_idxs]\n",
    "\n",
    "        batch_x = batch_x[non_zero_idxs, :, :]\n",
    "        batch_y = batch_y[non_zero_idxs, :, :]\n",
    "        c_l_1 = c_1[non_zero_idxs]\n",
    "        h_l_1 = h_1[non_zero_idxs]\n",
    "        \n",
    "        c_l_2 = c_2[non_zero_idxs]\n",
    "        h_l_2 = h_2[non_zero_idxs]\n",
    "        \n",
    "        if np.all(batch_lens_aux == 0):\n",
    "            break\n",
    "    \n",
    "           \n",
    "        m, states_, _ = sess.run([merge, states, upd], feed_dict={x: batch_x, y: batch_y, init_state_c_1: c_l_1, init_state_h_1: h_l_1, init_state_c_2: c_l_2, init_state_h_2: h_l_2, seq_len: batch_lens_aux})\n",
    "        train_writer.add_summary(m, counter)\n",
    "        \n",
    "        counter += 1\n",
    "        \n",
    "        c_1[non_zero_idxs] = states_[0].c\n",
    "        h_1[non_zero_idxs] = states_[0].h\n",
    "\n",
    "        c_2[non_zero_idxs] = states_[1].c\n",
    "        h_2[non_zero_idxs] = states_[1].h\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generando texto!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Una vez entrenada la red, generemos párrafos!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 3\n",
    "for _ in range(N):\n",
    "    print(\"\".join(test(max_=1000, T=None)))\n",
    "    print()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}