Panóptico_de_Twitch_0_1_0.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wzmprV7zNNtO"
      },
      "source": [
        "#  Panóptico de Twitch v0.1.0\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IYtt5eWXVZlX"
      },
      "source": [
        "Aplicación de *chat-scrapping* o *minería de texto* basada en API."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AGcBrBG4NaYJ"
      },
      "source": [
        "## Preparación del entorno de desarrollo\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0AoXdxEnNjQh"
      },
      "source": [
        "En esta sección se va a inicializar la **máquina** en los **servidores de Google** Colab para realizar nuestra **monitorización**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lvC8H-4fI6vS"
      },
      "outputs": [],
      "source": [
        "import socket\n",
        "import time\n",
        "import re\n",
        "import datetime\n",
        "import pytz\n",
        "import pandas as pd\n",
        "from os import mkdir \n",
        "import os.path\n",
        "from os import path"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WrNZdtnENwCs"
      },
      "source": [
        "## Variables globales del proyecto"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eg272dJDWezv"
      },
      "source": [
        "En esta función definimos las **variables globales** de nuestro proyecto. Se recomienda modificar los datos en los cuadros de *UI* a la derecha del código.\n",
        "\n",
        "Se recomienda un **tamaño pequeño** (entre 5.000 y 10.000 mensajes) para el tamaño máximo de los archivos .csv para **minimizar** las posibilidades de **pérdida de la información**.\n",
        "\n",
        "Se presentan dos opciones para el guardado de los datos: **local** (en la máquina virtual de Google Colab); y en **Google Drive** (recomendado).\n",
        "\n",
        "Para obtener el token **OAuth** para acceder a la información de la API sigue las siguientes instrucciones:\n",
        "1. **Creamos** una cuenta en *Twitch.tv*: https://www.twitch.tv/signup\n",
        "\n",
        "2. **Registramos** nuestra cuenta como desarrollador: https://dev.twitch.tv/console/apps/create y establecemos *'OAuth Redirect URL'*: 'http://localhost'\n",
        "\n",
        "3. **Adquirimos** el *OAuth* en:\n",
        "https://twitchapps.com/tmi/ "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VCpC2QriJrhm",
        "outputId": "be88e2e2-cdc1-444f-a9d4-19a968c3ec36",
        "cellView": "form"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "tz = pytz.timezone('Europe/Paris')\n",
        "\n",
        "#@markdown Servidor IRC al que queremos conectarnos.\n",
        "server = 'irc.chat.twitch.tv' #@param {type:\"string\"}\n",
        "port = 6667\n",
        "\n",
        "#@markdown Nombre de la cuenta con la que queremos conectarnos.\n",
        "nickname = \"repurposing_twitch\" #@param {type:\"string\"}\n",
        "\n",
        "#@markdown Token OAuth de la cuenta.\n",
        "token = \"oauth:\" #@param {type:\"string\"}\n",
        "\n",
        "#@markdown Canal que queremos monitorizar (¡en minúsculas!).\n",
        "channel = \"lvpes\" #@param {type:\"string\"}\n",
        "\n",
        "#@markdown Duración de la monitorización en segundos.\n",
        "duration = 60 #@param {type:\"integer\"}\n",
        "\n",
        "#@markdown Tamaño máximo de los CSV\n",
        "tam_csv = 5000 #@param {type:\"integer\"}\n",
        "\n",
        "#@markdown ¿Guardar datos en el Drive personal?\n",
        "save_to_gdrive = True #@param {type:\"boolean\"}\n",
        "\n",
        "#@markdown Modo debug\n",
        "debug_mode = True #@param {type:\"boolean\"}\n",
        "\n",
        "if save_to_gdrive:\n",
        "  from google.colab import drive\n",
        "  drive.mount('/content/drive')\n",
        "  output_dir = \"drive/MyDrive/Panoptico_Twitch/\" + channel + \"/\" + datetime.datetime.now().strftime(\"%Y_%m_%d\") + \"/\"\n",
        "\n",
        "  if not os.path.isdir(output_dir):\n",
        "    os.makedirs(output_dir)\n",
        "\n",
        "    \n",
        "else:\n",
        "    output_dir = \"/content/\" + channel + \"/\"\n",
        "    mkdir(output_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "amMB6WeiOHg-"
      },
      "source": [
        "## Funciones auxiliares"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fA2JWp2IWaNf"
      },
      "source": [
        "Las siguientes funciones están dedicadas a la **segmentación** del texto enviado por la API a través de **expresiones regulares** (regex)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Dl8sfoYxLUoM"
      },
      "outputs": [],
      "source": [
        "def parse_msg(msg, debug=True):\n",
        "    try:\n",
        "        pat_message = re.compile(\n",
        "            fr\"@badge-info=(?P<badge_info>[^;]*).*badges=(?P<badges>[^;]*).*first-msg=(?P<first_msg>[^;]*).*mod=(?P<mod>[^;]*).*subscriber=(?P<subscriber>[^;]*).+:(?P<username>[\\d\\w]+)![^:]+:(?P<text>.*)\", \n",
        "            flags=re.IGNORECASE\n",
        "        )\n",
        "\n",
        "        msg_data = pat_message.search(msg).groupdict()\n",
        "        \n",
        "        msg_data[\"months_as_sub\"] = 0\n",
        "        msg_data[\"medal\"] = 0\n",
        "        msg_data[\"bits\"] = 0\n",
        "        msg_data[\"sub-gifter\"] = 0\n",
        "        msg_data[\"mentions\"] = \"\"\n",
        "        msg_data[\"predictions\"] = \"\"\n",
        "        msg_data[\"amazon_prime\"] = 0\n",
        "        msg_data[\"vip\"] = 0\n",
        "\n",
        "        badge_info_list = msg_data[\"badge_info\"].split(\",\")\n",
        "        for badge_info in badge_info_list:\n",
        "          if \"subscriber\" in badge_info:\n",
        "            msg_data[\"months_as_sub\"] = int(badge_info.split('/')[-1])\n",
        "          if \"predictions\" in badge_info:\n",
        "            msg_data[\"predictions\"] = badge_info.split('/')[-1]\n",
        "\n",
        "        badge_list = msg_data[\"badges\"].split(\",\")\n",
        "        interest_list = [\"subscriber\", \"bits\", \"sub-gifter\", \"premium\"]\n",
        "        for badge in badge_list:\n",
        "          if \"subscriber\" in badge:\n",
        "            msg_data[\"medal\"] = int(badge.split('/')[-1])\n",
        "          if \"bits\" in badge:\n",
        "            msg_data[\"bits\"] = int(badge.split('/')[-1])\n",
        "          if \"sub-gifter\" in badge:\n",
        "            msg_data[\"sub-gifter\"] = int(badge.split('/')[-1])\n",
        "          if \"premium\" in badge:\n",
        "            msg_data[\"amazon_prime\"] = int(badge.split('/')[-1])\n",
        "          if \"vip\" in badge:\n",
        "            msg_data[\"vip\"] = int(badge.split('/')[-1])\n",
        "\n",
        "        text_list = msg_data[\"text\"].split(\" \")\n",
        "        for text_info in text_list:\n",
        "          if \"@\" in text_info:\n",
        "            msg_data[\"mentions\"] = str(text_info.split('/')[-1])            \n",
        "\n",
        "#@badge-info=predictions/VISITANTE,subscriber/4;badges=predictions/blue-3,subscriber/3,moments/2;color=#1E90FF;display-name=DidacticDani;emotes=;first-msg=0;flags=;id=bb3eb7c0-6322-4297-8370-97d35a4b20c9;mod=0;returning-chatter=0;room-id=48878319;subscriber=1;tmi-sent-ts=1669839182999;turbo=0;user-id=255326102;user-type= :didacticdani!didacticdani@didacticdani.tmi.twitch.tv PRIVMSG #thegrefg :lo peor es que son 2\n",
        "\n",
        "        date = datetime.datetime.now(tz).strftime(\"%m/%d/%Y\")\n",
        "        time = datetime.datetime.now(tz).strftime(\"%H:%M:%S\")\n",
        "        msg_data[\"date\"] = str(date)\n",
        "        msg_data[\"time\"] = str(time)\n",
        "\n",
        "        if debug:\n",
        "            print(msg_data)\n",
        "        return msg_data\n",
        "    except Exception as e:\n",
        "        return None"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u8WwdBA3OTab"
      },
      "source": [
        "## Función principal de monitorización\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kSJpsUpqWWQH"
      },
      "source": [
        "La siguiente función es nuestro **main**. Realiza, en primer lugar, la comunicación mediante envío de sockets a la API enviando nuestras credenciales y entra en el bucle descifrando los mensajes recibidos y tabulándolos a través de las funciones auxiliares hasta que finalice el tiempo indicado."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pGquUUCJLWJu",
        "outputId": "95c05de2-345f-4065-b5e4-de468ba087c7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<socket.socket fd=46, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.28.0.12', 38276), raddr=('44.226.36.141', 6667)>\n",
            ":tmi.twitch.tv 001 repurposing_twitch :Welcome, GLHF!\n",
            ":tmi.twitch.tv 002 repurposing_twitch :Your host is tmi.twitch.tv\n",
            ":tmi.twitch.tv 003 repurposing_twitch :This server is rather new\n",
            ":tmi.twitch.tv 004 repurposing_twitch :-\n",
            ":tmi.twitch.tv 375 repurposing_twitch :-\n",
            ":tmi.twitch.tv 372 repurposing_twitch :You are in a maze of twisty passages, all alike.\n",
            ":tmi.twitch.tv 376 repurposing_twitch :>\n",
            "\n",
            ":repurposing_twitch!repurposing_twitch@repurposing_twitch.tmi.twitch.tv JOIN #lvpes\n",
            ":tmi.twitch.tv CAP * ACK :twitch.tv/tags\n",
            ":repurposing_twitch.tmi.twitch.tv 353 repurposing_twitch = #lvpes :repurposing_twitch\n",
            ":repurposing_twitch.tmi.twitch.tv 366 repurposing_twitch #lvpes :End of /NAMES list\n",
            "\n",
            "{'badge_info': 'subscriber/5', 'badges': 'subscriber/3', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'zumodebrocoli', 'text': '@agrishad literalmente el chaval tilteao en susurros diciendome que neon y kaiser habian gapeado a toda la lec', 'months_as_sub': 5, 'medal': 3, 'bits': 0, 'sub-gifter': 0, 'mentions': '@agrishad', 'predictions': '', 'amazon_prime': 0, 'vip': 0, 'date': '02/27/2023', 'time': '19:29:57'}\n",
            "1 / 5000\n",
            "{'badge_info': 'subscriber/8', 'badges': 'subscriber/6,premium/1', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'geisan', 'text': '@benigiku no quiero mas botlines iguales', 'months_as_sub': 8, 'medal': 6, 'bits': 0, 'sub-gifter': 0, 'mentions': '@benigiku', 'predictions': '', 'amazon_prime': 1, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:24'}\n",
            "2 / 5000\n",
            "{'badge_info': 'subscriber/8', 'badges': 'subscriber/6,premium/1', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'awitadecoco', 'text': 'diossssssssssssssss', 'months_as_sub': 8, 'medal': 6, 'bits': 0, 'sub-gifter': 0, 'mentions': '', 'predictions': '', 'amazon_prime': 1, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:24'}\n",
            "3 / 5000\n",
            "{'badge_info': 'subscriber/6', 'badges': 'subscriber/6,sub-gifter/1', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'notaxlife', 'text': 'Perder contra guasones es la verguenza en la SL', 'months_as_sub': 6, 'medal': 6, 'bits': 0, 'sub-gifter': 1, 'mentions': '', 'predictions': '', 'amazon_prime': 0, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:25'}\n",
            "4 / 5000\n",
            "{'badge_info': 'predictions/GUASONES,subscriber/5', 'badges': 'predictions/pink-2,subscriber/3', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'elmotraceurr', 'text': 'jajaja XD', 'months_as_sub': 5, 'medal': 3, 'bits': 0, 'sub-gifter': 0, 'mentions': '', 'predictions': 'GUASONES', 'amazon_prime': 0, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:27'}\n",
            "5 / 5000\n",
            "{'badge_info': 'subscriber/2', 'badges': 'subscriber/2,premium/1', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'mrvila93', 'text': '@ZumoDeBrocoli jajajajjajajajajajjajajja', 'months_as_sub': 2, 'medal': 2, 'bits': 0, 'sub-gifter': 0, 'mentions': '@ZumoDeBrocoli', 'predictions': '', 'amazon_prime': 1, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:27'}\n",
            "6 / 5000\n",
            "{'badge_info': 'subscriber/7', 'badges': 'subscriber/6', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'jaaxsito', 'text': 'BOMBA', 'months_as_sub': 7, 'medal': 6, 'bits': 0, 'sub-gifter': 0, 'mentions': '', 'predictions': '', 'amazon_prime': 0, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:27'}\n",
            "7 / 5000\n",
            "{'badge_info': 'predictions/BISONS\\\\sECLUB,subscriber/7', 'badges': 'predictions/blue-1,subscriber/6,premium/1', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'aaleex_21', 'text': 'Q???', 'months_as_sub': 7, 'medal': 6, 'bits': 0, 'sub-gifter': 0, 'mentions': '', 'predictions': 'BISONS\\\\sECLUB', 'amazon_prime': 1, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:29'}\n",
            "8 / 5000\n",
            "{'badge_info': 'subscriber/3', 'badges': 'subscriber/3,bits/1', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'guilleyx', 'text': 'legolas eres un tinter', 'months_as_sub': 3, 'medal': 3, 'bits': 1, 'sub-gifter': 0, 'mentions': '', 'predictions': '', 'amazon_prime': 0, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:40'}\n",
            "9 / 5000\n",
            "{'badge_info': 'subscriber/3', 'badges': 'subscriber/3,bits/1', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'guilleyx', 'text': 'inter*', 'months_as_sub': 3, 'medal': 3, 'bits': 1, 'sub-gifter': 0, 'mentions': '', 'predictions': '', 'amazon_prime': 0, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:43'}\n",
            "10 / 5000\n",
            "{'badge_info': 'subscriber/5', 'badges': 'subscriber/3', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'zumodebrocoli', 'text': 'No se yo si los jugadores de bisons prefieren ganar o perder en este punto, da igual cual de las dos les van a caer una bronca letal', 'months_as_sub': 5, 'medal': 3, 'bits': 0, 'sub-gifter': 0, 'mentions': '', 'predictions': '', 'amazon_prime': 0, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:53'}\n",
            "11 / 5000\n",
            "{'badge_info': 'subscriber/4', 'badges': 'subscriber/3,premium/1', 'first_msg': '0', 'mod': '0', 'subscriber': '1', 'username': 'el_goncen', 'text': 'guasones ganaría la lec', 'months_as_sub': 4, 'medal': 3, 'bits': 0, 'sub-gifter': 0, 'mentions': '', 'predictions': '', 'amazon_prime': 1, 'vip': 0, 'date': '02/27/2023', 'time': '19:30:54'}\n",
            "12 / 5000\n"
          ]
        }
      ],
      "source": [
        "server_socket = socket.socket()\n",
        "server_socket.connect((server, port))\n",
        "\n",
        "if (debug_mode):\n",
        "  print(server_socket)\n",
        "\n",
        "server_socket.send(f\"PASS {token}\\n\".encode('utf-8'))\n",
        "server_socket.send(f\"NICK {nickname}\\n\".encode('utf-8'))\n",
        "server_socket.send(f\"JOIN #{channel}\\n\".encode('utf-8'))\n",
        "server_socket.send(f\"CAP REQ :twitch.tv/tags\\n\".encode('utf-8'))\n",
        "\n",
        "response = server_socket.recv(16384).decode('utf-8')\n",
        "\n",
        "if (debug_mode):\n",
        "  print(response)\n",
        "\n",
        "response = server_socket.recv(16384).decode('utf-8')\n",
        "\n",
        "if (debug_mode):\n",
        "  print(response)\n",
        "\n",
        "msg_list = []\n",
        "start_time = time.time()\n",
        "fragment_counter = 0\n",
        "bucle_time = time.time()\n",
        "\n",
        "while (time.time() - start_time) < duration:\n",
        "  try:\n",
        "    response = server_socket.recv(1024).decode('utf-8')\n",
        "    if response.startswith(\"PING\"):\n",
        "      server_socket.send(f\"PONG\\n\".encode('utf-8'))\n",
        "    else:\n",
        "      for m in response.split(\"\\r\\n\"):\n",
        "        msg_data = parse_msg(m, debug=debug_mode)\n",
        "        if msg_data is not None:\n",
        "          msg_list.append(msg_data)\n",
        "\n",
        "          if (debug_mode):\n",
        "            print(len(msg_list), \"/\", tam_csv)\n",
        "\n",
        "          if len(msg_list) >= tam_csv:\n",
        "\n",
        "            if debug_mode:\n",
        "              print(f\"Volcando el fragmento {fragment_counter}\")\n",
        "            \n",
        "            df_msg = pd.DataFrame(msg_list)\n",
        "            df_msg.to_csv(output_dir + channel + datetime.datetime.now().strftime(\"_%Y_%m_%d_%Hh%M\") + \"_\" + str(fragment_counter) + '.csv', index=False)\n",
        "            fragment_counter += 1\n",
        "            msg_list = []    \n",
        "\n",
        "          if (time.time() - bucle_time) > 3600: \n",
        "            if debug_mode:\n",
        "              print(f\"Ha pasado una hora. Volcando el fragmento {fragment_counter}\")\n",
        "            \n",
        "            df_msg = pd.DataFrame(msg_list)\n",
        "            df_msg.to_csv(output_dir + channel + datetime.datetime.now().strftime(\"_%Y_%m_%d_%Hh%M\") + \"_\" + str(fragment_counter) + '.csv', index=False)\n",
        "            fragment_counter += 1\n",
        "            msg_list = []\n",
        "            bucle_time = time.time()\n",
        "\n",
        "  except Exception as e:\n",
        "    print(\"Ha ocurrido una excepción, guardamos la información\")\n",
        "    df_msg = pd.DataFrame(msg_list)\n",
        "    df_msg.to_csv(output_dir + channel + datetime.datetime.now().strftime(\"_%Y_%m_%d_%Hh%M\") + \"_\" + str(fragment_counter) + '.csv', index=False)\n",
        "\n",
        "    if debug_mode:\n",
        "      print(f\"Volcando el fragmento {fragment_counter}\")\n",
        "      \n",
        "    fragment_counter += 1\n",
        "    msg_list =[]\n",
        "    bucle_time = time.time()\n",
        "\n",
        "server_socket.close()\n",
        "\n",
        "df_msg = pd.DataFrame(msg_list)\n",
        "df_msg.to_csv(output_dir + channel + datetime.datetime.now().strftime(\"_%Y_%m_%d_%Hh%M\") + \"_\" + str(fragment_counter) + '.csv', index=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MPB5w5c5-rVL"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qZVqmnoIBb2b"
      },
      "source": [
        "Para la limpieza y unificación de la base de datos: https://drive.google.com/file/d/1tXV2XKBkFMQDIJT6BHqn6OvT6DtA1Nmv/view?usp=sharing\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}