LSTM_Next_Word_Prediction.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNn5tfgrbno/t375JERx7uz",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/singhsourav0/Deep-Learning-Odyssey/blob/main/LSTM_Next_Word_Prediction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<p><font color=\\\"blueviolet\\\">In this notebook i cover the basic to advance concept of LSTM (long short term memory). This architecture is one of the complex one in the deep learning. So, i just try to understand it in very simple way.</br> And why we use LSTM instead of RNN. Why RNN is not GOOD for long textual data to make sense on that. </font></p>\n",
        "\n",
        "\n",
        "1. **Long Short-Term Memory (LSTM)**:\n",
        "   - <font color='lime'>LSTM is a type of recurrent neural network (RNN) architecture. RNNs are great for processing sequences of data because they can remember past information while processing new inputs. However, traditional RNNs sometimes struggle to capture long-term dependencies in data.</font>\n",
        "   \n",
        "2. **Key Concepts of LSTM**:\n",
        "\n",
        "a. **Short Memory and Long Memory**: <font color='lime'>LSTM (Long Short-Term Memory) is a special type of neural network designed to remember and use information over long sequences, like in text or time series data.</font>\n",
        "\n",
        "b. **Short-Term Memory (Cell State)**:<font color='lime'> In LSTM, there's a short-term memory called the \"cell state.\" It flows through the whole sequence of LSTM units, almost unchanged, acting like a conveyor belt.</font>\n",
        "\n",
        "c. **Storing Important Information**:<font color='lime'> LSTM decides what information to store in its memory using gates.</font>\n",
        "   - **Forget Gate**: It decides what to forget from the cell state.\n",
        "   - **Input Gate**: It decides what new information to store in the cell state.\n",
        "   - **Output Gate**: It decides what information to use for predictions.\n",
        "\n",
        "d. **Storing Keywords**:<font color='lime'> When processing text, important words or features are stored in the cell state using the input gate. If a word is important, LSTM updates its memory to remember it.</font>\n",
        "\n",
        "e. **Predicting with Memory**: <font color='lime'>As LSTM processes the text, it learns to remember important information in its memory. By the end of the text, its memory contains crucial details about the entire sequence.</font>\n",
        "\n",
        "f. **Using Memory for Predictions**: <font color='lime'>The output gate decides which parts of the memory are useful for making predictions. It chooses what to output and how to use it for tasks like text generation or sentiment analysis.</font>\n",
        "\n",
        "3. **Advantages of LSTM over RNN**:\n",
        "   - <font color='lime'>LSTM is better at capturing long-term dependencies in data because it can learn when to forget or remember information over long sequences.</font>\n",
        "   - <font color='lime'>Traditional RNNs often suffer from the vanishing gradient problem, where gradients diminish as they are backpropagated through time. LSTM's architecture helps alleviate this issue, allowing for better training on longer sequences.</font>\n",
        "\n",
        "4. **Applications**:\n",
        "   - <font color='lime'>LSTMs are widely used in natural language processing (NLP) tasks such as language translation, text generation, and sentiment analysis.</font>\n",
        "\n",
        "   - <font color='lime'>They're also used in speech recognition, time series prediction, and many other sequential data tasks where understanding context over long sequences is important.</font>\n",
        "\n",
        "\n",
        "<font color='teal'>In essence, LSTM is like a smart storyteller. It reads through a long story, remembers key details, and uses them to predict what happens next.</font>"
      ],
      "metadata": {
        "id": "zJ3uKXVGJW8r"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<h3><font color='teal'>In this script i take text of data mining pdf of NPTEl courses and decide to generate future word of that pdf according to present word context.</h3></font>\n",
        "\n",
        "<h3><font color='teal'>Step 1: Preparing the Text Data</font></h3>\n",
        "<font color='yellow'>1. <b>Data Collection:</b></font> I obtained text data from a PDF about data mining from NPTEL courses.\n",
        "\n",
        "<font color='yellow'>2. <b>Data Cleaning:</b></font> Removed unnecessary data from the text to make it more manageable.\n",
        "\n",
        "<font color='yellow'>3. <b>Filtering Sentences:</b></font> Selected only those sentences that have at least 5 words, ensuring better prediction quality.\n",
        "\n",
        "\n",
        "<h3><font color='teal'>Step 2: Tokenization</font></h3>\n",
        "\n",
        "<font color='yellow'>4. <b>Tokenization:</b></font> Utilized TensorFlow's Keras API to tokenize the text data. Each word is assigned a unique numerical identifier (token).\n",
        "\n",
        "<h3><font color='teal'>Step 3: Data Preparation</font></h3>\n",
        "\n",
        "<font color='yellow'>5. <b>Input Sequences:</b></font> Created input sequences by sliding a window over the tokenized text. These sequences serve as inputs for the model.\n",
        "\n",
        "<font color='yellow'>6. <b>Padding:</b></font> Ensured that all input sequences have the same length by padding them with zeros.\n",
        "\n",
        "<h3><font color='teal'>Step 4: Model Building</font></h3>\n",
        "\n",
        "<font color='yellow'>7. <b>Model Architecture:</b></font>\n",
        "   - Initialized a Sequential model.\n",
        "   - Added an Embedding layer to map the tokens to dense vectors of fixed size.\n",
        "   - Added an LSTM layer with 150 units to learn patterns in the sequences.\n",
        "   - Added a Dense layer with softmax activation to predict the next word in the sequence.\n",
        "\n",
        "<font color='yellow'>8. <b>Model Compilation:</b></font> Compiled the model, specifying optimizer, loss function, and evaluation metrics.\n",
        "\n",
        "<h3><font color='teal'>Step 5: Prediction</font></h3>\n",
        "<font color='yellow'>9. <b>Generating Text:</b></font>\n",
        "   - Defined a starting text sequence.\n",
        "   - Iterated over a loop to predict the next word for a fixed number of iterations (10 in this case).\n",
        "   - Tokenized the current text sequence and padded it.\n",
        "   - Predicted the next word using the trained model.\n",
        "   - Appended the predicted word to the text sequence and printed the updated sequence.\n",
        "   - Added a delay (1.5 seconds) between each iteration to observe the output sequentially.\n",
        "\n",
        "<h3><font color='teal'>Conclusion</font></h3>\n",
        "By following these steps, you've successfully built a text generation model using LSTM. This model can take a starting text sequence and predict the next word based on the context provided by the input sequence.\n"
      ],
      "metadata": {
        "id": "wPITeKQ_OwRW"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z_5MwS6DIoSk"
      },
      "outputs": [],
      "source": [
        "Text = '''\n",
        "Data Mining\n",
        "Week 1: Introduction, Association Rules\n",
        "Pabitra Mitra\n",
        "Computer Science and Engineering, IIT Kharagpur\n",
        "Email: pabitra@gmail.com\n",
        "NPTEL\n",
        "2\n",
        "Course Outline:\n",
        "• Introduction: KDD Process\n",
        "• Data Preprocessing\n",
        "• Association Rule Mining\n",
        "• Classification\n",
        "• Clustering and Anomaly Detection\n",
        "• Regression\n",
        "• Case Studies\n",
        "NPTEL\n",
        "3\n",
        "Data Mining\n",
        "Introduction\n",
        "Pabitra Mitra\n",
        "Computer Science and Engineering, IIT Kharagpur\n",
        "NPTEL\n",
        "Why Data Mining?\n",
        "• The Explosive Growth of Data: from terabytes to petabytes\n",
        "– Data collection and data availability\n",
        "• Automated data collection tools, database systems, Web, computerized society\n",
        "– Major sources of abundant data\n",
        "• Business: Web, e-commerce, transactions, stocks, …\n",
        "• Science: Remote sensing, bioinformatics, scientific simulation, …\n",
        "• Society and everyone: news, digital cameras, YouTube\n",
        "• We are drowning in data, but starving for knowledge!\n",
        "• “Necessity is the mother of invention”—Data mining—Automated analysis of massive data\n",
        "NPTEL\n",
        "5\n",
        "What Is Data Mining?\n",
        "• Data mining (knowledge discovery from data)\n",
        "– Extraction of interesting (non-trivial, implicit, previously unknown and\n",
        "potentially useful) patterns or knowledge from huge amount of data\n",
        "• Alternative names\n",
        "– Knowledge discovery (mining) in databases (KDD), knowledge extraction,\n",
        "data/pattern analysis, data archeology, data dredging, information\n",
        "harvesting, business intelligence, etc.\n",
        "• Watch out: Is everything “data mining”?\n",
        "– Simple search and query processing\n",
        "– (Deductive) expert systems\n",
        "NPTEL\n",
        "6\n",
        "NPTEL\n",
        "7\n",
        "Data Mining: Confluence of Multiple Disciplines\n",
        "Data Mining\n",
        "Database\n",
        "Technology Statistics\n",
        "Machine\n",
        "Learning\n",
        "Pattern\n",
        "Recognition Algorithm\n",
        "Other\n",
        "Disciplines\n",
        "Visualization\n",
        "NPTEL\n",
        "8\n",
        "Why Not Traditional Data Analysis?\n",
        "• Tremendous amount of data\n",
        "– Algorithms must be highly scalable to handle such as tera-bytes of data\n",
        "• High-dimensionality of data\n",
        "– Micro-array may have tens of thousands of dimensions\n",
        "• High complexity of data\n",
        "– Data streams and sensor data\n",
        "– Time-series data, temporal data, sequence data\n",
        "– Structure data, graphs, social networks and multi-linked data\n",
        "– Heterogeneous databases and legacy databases\n",
        "– Spatial, spatiotemporal, multimedia, text and Web data\n",
        "NPTEL\n",
        "9\n",
        "Data Mining: On What Kinds of Data?\n",
        "• Database-oriented data sets and applications\n",
        "– Relational database, data warehouse, transactional database\n",
        "• Advanced data sets and advanced applications\n",
        "– Data streams and sensor data\n",
        "– Time-series data, temporal data, sequence data (incl. bio-sequences)\n",
        "– Structure data, graphs, social networks and multi-linked data\n",
        "– Object-relational databases\n",
        "– Heterogeneous databases and legacy databases\n",
        "– Spatial data and spatiotemporal data\n",
        "– Multimedia database\n",
        "– Text databases\n",
        "– The World-Wide Web\n",
        "NPTEL\n",
        "10\n",
        "Data Mining Functionalities\n",
        "• Multidimensional concept description: Characterization and discrimination\n",
        "– Generalize, summarize, and contrast data characteristics, e.g., dry vs. wet\n",
        "regions\n",
        "• Frequent patterns, association, correlation vs. causality\n",
        "– Tea  Sugar [0.5%, 75%] (Correlation or causality?)\n",
        "• Classification and prediction\n",
        "– Construct models (functions) that describe and distinguish classes or\n",
        "concepts for future prediction\n",
        "• E.g., classify countries based on (climate), or classify cars based on (gas\n",
        "mileage)\n",
        "– Predict some unknown or missing numerical values\n",
        "NPTEL\n",
        "11\n",
        "Data Mining Functionalities\n",
        "• Cluster analysis\n",
        "– Class label is unknown: Group data to form new classes, e.g., cluster houses to find\n",
        "distribution patterns\n",
        "– Maximizing intra-class similarity & minimizing interclass similarity\n",
        "• Outlier analysis\n",
        "– Outlier: Data object that does not comply with the general behavior of the data\n",
        "– Noise or exception? Useful in fraud detection, rare events analysis\n",
        "• Trend and evolution analysis\n",
        "– Trend and deviation: e.g., regression analysis\n",
        "– Sequential pattern mining: e.g., digital camera  large SD memory\n",
        "– Periodicity analysis\n",
        "– Similarity-based analysis\n",
        "• Other pattern-directed or statistical analyses\n",
        "NPTEL\n",
        "12\n",
        "Major Issues in Data Mining\n",
        "• Mining methodology\n",
        "– Mining different kinds of knowledge from diverse data types, e.g., bio, stream, Web\n",
        "– Performance: efficiency, effectiveness, and scalability\n",
        "– Pattern evaluation: the interestingness problem\n",
        "– Incorporation of background knowledge\n",
        "– Handling noise and incomplete data\n",
        "– Parallel, distributed and incremental mining methods\n",
        "– Integration of the discovered knowledge with existing one: knowledge fusion\n",
        "• User interaction\n",
        "– Data mining query languages and ad-hoc mining\n",
        "– Expression and visualization of data mining results\n",
        "– Interactive mining of knowledge at multiple levels of abstraction\n",
        "• Applications and social impacts\n",
        "– Domain-specific data mining & invisible data mining\n",
        "– Protection of data security, integrity, and privacy\n",
        "NPTEL\n",
        "13\n",
        "Architecture: Typical Data Mining System\n",
        "data cleaning, integration, and selection\n",
        "Database or Data Warehouse\n",
        "Server\n",
        "Data Mining Engine\n",
        "Pattern Evaluation\n",
        "Graphical User Interface\n",
        "Knowledge\n",
        "-Base\n",
        "Database Data\n",
        "Warehouse\n",
        "World-Wide\n",
        "Web\n",
        "Other Info\n",
        "Repositories\n",
        "NPTEL\n",
        "14\n",
        "KDD Process: Summary\n",
        "• Learning the application domain\n",
        "– relevant prior knowledge and goals of application\n",
        "• Creating a target data set: data selection\n",
        "• Data cleaning and preprocessing: (may take 60% of effort!)\n",
        "• Data reduction and transformation\n",
        "– Find useful features, dimensionality/variable reduction, invariant representation\n",
        "• Choosing functions of data mining\n",
        "– summarization, classification, regression, association, clustering\n",
        "• Choosing the mining algorithm(s)\n",
        "• Data mining: search for patterns of interest\n",
        "• Pattern evaluation and knowledge presentation\n",
        "– visualization, transformation, removing redundant patterns, etc.\n",
        "• Use of discovered knowledge\n",
        "NPTEL\n",
        "End of Introduction\n",
        "15\n",
        "NPTEL\n",
        "16\n",
        "Data Mining\n",
        "Data Preprocessing\n",
        "Pabitra Mitra\n",
        "Computer Science and Engineering, IIT Kharagpur\n",
        "NPTEL\n",
        "What is Data?\n",
        "• Collection of data objects and their\n",
        "attributes\n",
        "• An attribute is a property or\n",
        "characteristic of an object\n",
        "– Examples: eye color of a person,\n",
        "temperature, etc.\n",
        "– Attribute is also known as variable,\n",
        "field, characteristic, or feature\n",
        "• A collection of attributes describe\n",
        "an object\n",
        "– Object is also known as record,\n",
        "point, case, sample, entity, or\n",
        "instance Tid Refund Marital\n",
        "Status\n",
        "Taxable\n",
        "10\n",
        "Attributes\n",
        "NPTEL\n",
        "Objects\n",
        "Types of Attributes\n",
        "• There are different types of attributes\n",
        "– Nominal\n",
        "• Examples: ID numbers, eye color, zip codes\n",
        "– Ordinal\n",
        "• Examples: rankings (e.g., taste of potato chips on a scale from 1-\n",
        "10), grades, height in {tall, medium, short}\n",
        "– Interval\n",
        "• Examples: calendar dates, temperatures in Celsius or Fahrenheit.\n",
        "– Ratio\n",
        "• Examples: temperature in Kelvin, length, time, counts\n",
        "NPTEL\n",
        "Properties of Attribute Values\n",
        "• The type of an attribute depends on which of the following\n",
        "\n",
        "Attribute\n",
        "Type\n",
        "Description Examples Operations\n",
        "Nominal The values of a nominal attribute are just\n",
        "different names, i.e., nominal attributes\n",
        "provide only enough information to\n",
        "distinguish one object from another. (=, )\n",
        "zip codes, employee\n",
        "ID numbers, eye color,\n",
        "sex: {male, female}\n",
        "mode, entropy,\n",
        "contingency\n",
        "correlation, \n",
        "2\n",
        "test\n",
        "Ordinal The values of an ordinal attribute\n",
        "provide enough information to order\n",
        "objects (< >).\n",
        "hardness of minerals,\n",
        "{good, better, best},\n",
        "grades, street numbers\n",
        "median, percentiles,\n",
        "rank correlation,\n",
        "run tests, sign tests\n",
        "Interval For interval attributes, the differences\n",
        "between values are meaningful, i.e., a unit\n",
        "of measurement exists.\n",
        "(+, - )\n",
        "calendar dates,\n",
        "temperature in Celsius\n",
        "or Fahrenheit\n",
        "mean, standard\n",
        "deviation, Pearson's\n",
        "correlation, t and F\n",
        "tests\n",
        "Ratio For ratio variables, both differences\n",
        "and ratios are meaningful. (*, /)\n",
        "temperature in Kelvin,\n",
        "monetary quantities,\n",
        "counts, age, mass, length,\n",
        "electrical current\n",
        "geometric mean,\n",
        "harmonic mean,\n",
        "percent variation\n",
        "NPTEL\n",
        "Discrete and Continuous Attributes\n",
        "• Discrete Attribute\n",
        "– Has only a finite or countably infinite set of values\n",
        "– Examples: zip codes, counts, or the set of words in a collection of\n",
        "documents\n",
        "– Often represented as integer variables.\n",
        "– Note: binary attributes are a special case of discrete attributes\n",
        "• Continuous Attribute\n",
        "– Has real numbers as attribute values\n",
        "– Examples: temperature, height, or weight.\n",
        "– Practically, real values can only be measured and represented using a\n",
        "finite number of digits.\n",
        "– Continuous attributes are typically represented as floating-point variables.\n",
        "NPTEL\n",
        "Types of data sets\n",
        "• Record\n",
        "– Data Matrix\n",
        "– Document Data\n",
        "– Transaction Data\n",
        "• Graph\n",
        "– World Wide Web\n",
        "– Molecular Structures\n",
        "• Ordered\n",
        "– Spatial Data\n",
        "– Temporal Data\n",
        "– Sequential Data\n",
        "– Genetic Sequence Data\n",
        "NPTEL\n",
        "Record Data\n",
        "• Data that consists of a collection of records, each of which consists of a\n",
        "fixed set of attributes Tid Refund Marital\n",
        "Status\n",
        "\n",
        "10\n",
        "NPTEL\n",
        "Data Matrix\n",
        "• If data objects have the same fixed set of numeric attributes, then\n",
        "the data objects can be thought of as points in a multi-dimensional\n",
        "space, where each dimension represents a distinct attribute\n",
        "• Such data set can be represented by an m by n matrix, where there\n",
        "are m rows, one for each object, and n columns, one for each\n",
        "\n",
        "Projection Distance Load Thickness\n",
        "of y load\n",
        "Projection\n",
        "of x Load\n",
        "NPTEL\n",
        "Text Data\n",
        "• Each document becomes a `term' vector,\n",
        "each term is a component (attribute) of the vector,\n",
        "– the value of each component is the number of times the corresponding term\n",
        "occurs in the document.\n",
        "Document 1\n",
        "timeout\n",
        "season\n",
        "wi\n",
        "lost\n",
        "n\n",
        "score\n",
        "game\n",
        "pla\n",
        "ball\n",
        "y\n",
        "NPTEL\n",
        "Transaction Data\n",
        "• A special type of record data, where\n",
        "– each record (transaction) involves a set of items.\n",
        "– For example, consider a grocery store. The set of products purchased\n",
        "by a customer during one shopping trip constitute a transaction, while\n",
        "the individual products that were purchased are the items.\n",
        "\n",
        "NPTEL\n",
        "Graph Data\n",
        "• Examples: Facebook graph and HTML Links\n",
        "5\n",
        "2\n",
        "1\n",
        " 2\n",
        "5\n",
        "NPTEL\n",
        "Ordered Data\n",
        "Data Quality\n",
        "• What kinds of data quality problems?\n",
        "• How can we detect problems with the data?\n",
        "• What can we do about these problems?\n",
        "• Examples of data quality problems:\n",
        "– Noise and outliers\n",
        "– missing values\n",
        "– duplicate data\n",
        "NPTEL\n",
        "Noise\n",
        "• Noise refers to modification of original values\n",
        "– Examples: distortion of a person’s voice when talking on a poor phone\n",
        "and “snow” on television screen\n",
        "Two Sine Waves Two Sine Waves + Noise\n",
        "NPTEL\n",
        "Outliers\n",
        "• Outliers are data objects with characteristics that are\n",
        "considerably different than most of the other data objects in\n",
        "the data set\n",
        "NPTEL\n",
        "Missing Values\n",
        "• Reasons for missing values\n",
        "– Information is not collected\n",
        "(e.g., people decline to give their age and weight)\n",
        "– Attributes may not be applicable to all cases\n",
        "(e.g., annual income is not applicable to children)\n",
        "• Handling missing values\n",
        "– Eliminate Data Objects\n",
        "– Estimate Missing Values\n",
        "– Ignore the Missing Value During Analysis\n",
        "– Replace with all possible values (weighted by their probabilities)\n",
        "NPTEL\n",
        "Duplicate Data\n",
        "• Data set may include data objects that are duplicates, or\n",
        "almost duplicates of one another\n",
        "– Major issue when merging data from heterogenous sources\n",
        "• Examples:\n",
        "– Same person with multiple email addresses\n",
        "• Data cleaning\n",
        "– Process of dealing with duplicate data issues\n",
        "NPTEL\n",
        "Data Preprocessing\n",
        "• Aggregation\n",
        "• Sampling\n",
        "• Dimensionality Reduction\n",
        "• Feature subset selection\n",
        "• Feature creation\n",
        "• Discretization and Binarization\n",
        "• Attribute Transformation\n",
        "NPTEL\n",
        "Aggregation\n",
        "• Combining two or more attributes (or objects) into a\n",
        "single attribute (or object)\n",
        "• Purpose\n",
        "– Data reduction\n",
        "• Reduce the number of attributes or objects\n",
        "– Change of scale\n",
        "• Cities aggregated into regions, states, countries, etc\n",
        "– More “stable” data\n",
        "• Aggregated data tends to have less variability\n",
        "NPTEL\n",
        "Sampling\n",
        "• Sampling is the main technique employed for data selection.\n",
        "– It is often used for both the preliminary investigation of the data and the final\n",
        "data analysis.\n",
        "• Statisticians sample because obtaining the entire set of data of interest\n",
        "is too expensive or time consuming.\n",
        "• Sampling is used in data mining because processing the entire set of\n",
        "data of interest is too expensive or time consuming.\n",
        "NPTEL\n",
        "Sample Size\n",
        "8000 points\n",
        "NPTEL\n",
        "2000 Points 500 Points\n",
        "Sampling …\n",
        "• The key principle for effective sampling is the\n",
        "following:\n",
        "– using a sample will work almost as well as using the\n",
        "entire data sets, if the sample is representative\n",
        "– A sample is representative if it has approximately the\n",
        "same property (of interest) as the original set of data\n",
        "NPTEL\n",
        "Types of Sampling\n",
        "• Simple Random Sampling\n",
        "– There is an equal probability of selecting any particular item\n",
        "• Sampling without replacement\n",
        "– As each item is selected, it is removed from the population\n",
        "• Sampling with replacement\n",
        "– Objects are not removed from the population as they are selected for the sample.\n",
        "• In sampling with replacement, the same object can be picked up more than once\n",
        "• Stratified sampling\n",
        "– Split the data into several partitions; then draw random samples from each\n",
        "partition\n",
        "NPTEL\n",
        "Curse of Dimensionality\n",
        "• When dimensionality increases, data becomes increasingly sparse\n",
        "in the space that it occupies\n",
        "• Definitions of density and distance between points, which is\n",
        "critical for clustering and outlier detection, become less\n",
        "meaningful\n",
        "NPTEL\n",
        "Dimensionality Reduction\n",
        "• Purpose:\n",
        "– Avoid curse of dimensionality\n",
        "– Reduce amount of time and memory required by data mining\n",
        "algorithms\n",
        "– Allow data to be more easily visualized\n",
        "– May help to eliminate irrelevant features or reduce noise\n",
        "• Techniques\n",
        "– Principle Component Analysis\n",
        "– Singular Value Decomposition\n",
        "– Others: supervised and non-linear techniques\n",
        "NPTEL\n",
        "Discretization\n",
        "Data Equal interval width\n",
        "Equal frequency K-means\n",
        "NPTEL\n",
        "Attribute Transformation\n",
        "• A function that maps the entire set of values of\n",
        "a given attribute to a new set of replacement\n",
        "values such that each old value can be\n",
        "identified with one of the new values\n",
        "– Simple functions: x\n",
        "k\n",
        ", log(x), ex\n",
        ", |x|\n",
        "– Standardization and Normalization\n",
        "NPTEL\n",
        "Similarity and Dissimilarity\n",
        "• Similarity\n",
        "– Numerical measure of how alike two data objects are.\n",
        "– Is higher when objects are more alike.\n",
        "– Often falls in the range [0,1]\n",
        "• Dissimilarity\n",
        "– Numerical measure of how different are two data objects\n",
        "– Lower when objects are more alike\n",
        "– Minimum dissimilarity is often 0\n",
        "– Upper limit varies\n",
        "• Proximity refers to a similarity or dissimilarity\n",
        "NPTEL\n",
        "Similarity/Dissimilarity for Simple Attributes\n",
        "p and q are the attribute values for two data objects.\n",
        "NPTEL\n",
        "Euclidean Distance\n",
        "• Euclidean Distance\n",
        "Where n is the number of dimensions (attributes) and pk\n",
        "and qk\n",
        "are,\n",
        "respectively, the k\n",
        "th attributes (components) or data objects p and q.\n",
        "End of Data Preprocessing\n",
        "NPTEL\n",
        "Data Mining\n",
        "Association Rules\n",
        "Pabitra Mitra\n",
        "Computer Science and Engineering\n",
        "NPTEL\n",
        "Association Rule Mining\n",
        "• Given a set of transactions, find rules that will predict the\n",
        "occurrence of an item based on the occurrences of other items in\n",
        "the transaction\n",
        "Market-Basket transactions\n",
        "\n",
        "NPTEL\n",
        "Definition: Frequent Itemset\n",
        "• Itemset\n",
        "– A collection of one or more items\n",
        "• Example: {Milk, Bread, Diaper}\n",
        "– k-itemset\n",
        "• An itemset that contains k items\n",
        "• Support count ()\n",
        "– Frequency of occurrence of an itemset\n",
        "– E.g. ({Milk, Bread,Diaper}) = 2\n",
        "• Support\n",
        "– Fraction of transactions that contain an itemset\n",
        "– E.g. s({Milk, Bread, Diaper}) = 2/5\n",
        "• Frequent Itemset\n",
        "– An itemset whose support is greater than or equal to a\n",
        "minsup threshold TID Items\n",
        " Beer}Diaper,Milk{\n",
        "4.0\n",
        "5\n",
        "2\n",
        "|T|\n",
        ")BeerDiaper,,Milk(\n",
        " \n",
        "\n",
        "s\n",
        ".0 67\n",
        "3\n",
        "2\n",
        ")Diaper,Milk(\n",
        "Milk,( )BeerDiaper,\n",
        " \n",
        "\n",
        "\n",
        "c\n",
        " Association Rule\n",
        "– An implication expression of the form X  Y, where X\n",
        "and Y are itemsets\n",
        "– Example:\n",
        "{Milk, Diaper}  {Beer}\n",
        " Rule Evaluation Metrics\n",
        "– Support (s)\n",
        " Fraction of transactions that contain both X and\n",
        "Y\n",
        "– Confidence (c)\n",
        " Measures how often items in Y\n",
        "appear in transactions that\n",
        "contain X\n",
        "TID Items\n",
        "Association Rule Mining Task\n",
        "• Given a set of transactions T, the goal of association rule mining is\n",
        "to find all rules having\n",
        "– support ≥ minsup threshold\n",
        "– confidence ≥ minconf threshold\n",
        "• Brute-force approach:\n",
        "– List all possible association rules\n",
        "– Compute the support and confidence for each rule\n",
        "– Prune rules that fail the minsup and minconf thresholds\n",
        " Computationally prohibitive!\n",
        "NPTEL\n",
        "Mining Association Rules\n",
        "• Two-step approach:\n",
        "1. Frequent Itemset Generation\n",
        "– Generate all itemsets whose support  minsup\n",
        "2. Rule Generation\n",
        "– Generate high confidence rules from each frequent\n",
        "itemset, where each rule is a binary partitioning of a\n",
        "frequent itemset\n",
        "NPTEL\n",
        "Frequent Itemset Generation\n",
        "• Brute-force approach:\n",
        "– Each itemset in the lattice is a candidate frequent itemset\n",
        "– Count the support of each candidate by scanning the database\n",
        "– Match each transaction against every candidate\n",
        "N\n",
        "Transactions\n",
        "List of\n",
        "Candidates\n",
        "M\n",
        "w\n",
        "NPTEL\n",
        "Frequent Itemset Generation Strategies\n",
        "• Reduce the number of candidates (M)\n",
        "– Complete search: M=2d\n",
        "– Use pruning techniques to reduce M\n",
        "• Reduce the number of transactions (N)\n",
        "– Reduce size of N as the size of itemset increases\n",
        "– Used by DHP and vertical-based mining algorithms\n",
        "• Reduce the number of comparisons (NM)\n",
        "– Use efficient data structures to store the candidates or transactions\n",
        "– No need to match every candidate against every transaction\n",
        "NPTEL\n",
        "Reducing Number of Candidates\n",
        "• Apriori principle:\n",
        "– If an itemset is frequent, then all of its subsets must also be\n",
        "frequent\n",
        "• Apriori principle holds due to the following property of the\n",
        "support measure:\n",
        "– Support of an itemset never exceeds the support of its subsets\n",
        "– This is known as the anti-monotone property of support\n",
        "NPTEL\n",
        "Pruned\n",
        "supersets\n",
        "NPTEL\n",
        "Illustrating Apriori Principle\n",
        "Item Count\n",
        "Bread 4\n",
        "Coke 2\n",
        "Milk 4\n",
        "Beer 3\n",
        "Diaper 4\n",
        "Eggs 1\n",
        "Itemset Count\n",
        "{Bread,Milk} 3\n",
        "{Bread,Beer} 2\n",
        "{Bread,Diaper} 3\n",
        "{Milk,Beer} 2\n",
        "{Milk,Diaper} 3\n",
        "{Beer,Diaper} 3\n",
        "Itemset Count\n",
        "{Bread,Milk,Diaper} 3\n",
        "Items (1-itemsets)\n",
        "Pairs (2-itemsets)\n",
        "(No need to generate\n",
        "candidates involving Coke\n",
        "or Eggs)\n",
        "Triplets (3-itemsets) Minimum Support = 3\n",
        "If every subset is considered,\n",
        "\n",
        "With support-based pruning,\n",
        "\n",
        "NPTEL\n",
        "Apriori Algorithm\n",
        "• Method:\n",
        "– Let k=1\n",
        "– Generate frequent itemsets of length 1\n",
        "– Repeat until no new frequent itemsets are identified\n",
        "• Generate length (k+1) candidate itemsets from length k frequent\n",
        "itemsets\n",
        "• Prune candidate itemsets containing subsets of length k that are\n",
        "infrequent\n",
        "• Count the support of each candidate by scanning the DB\n",
        "• Eliminate candidates that are infrequent, leaving only those that\n",
        "are frequent\n",
        "NPTEL\n",
        "Factors Affecting Complexity\n",
        "• Choice of minimum support threshold\n",
        "– lowering support threshold results in more frequent itemsets\n",
        "– this may increase number of candidates and max length of frequent itemsets\n",
        "• Dimensionality (number of items) of the data set\n",
        "– more space is needed to store support count of each item\n",
        "– if number of frequent items also increases, both computation and I/O costs may\n",
        "also increase\n",
        "• Size of database\n",
        "– Apriori makes multiple passes, run time of algorithm increase with number of\n",
        "transactions\n",
        "• Average transaction width\n",
        "– This may increase max length of frequent itemsets and traversals of hash tree\n",
        "(number of subsets in a transaction increases with its width)\n",
        "NPTEL\n",
        "Rule Generation\n",
        "• How to efficiently generate rules from frequent itemsets?\n",
        "– In general, confidence does not have an anti-monotone property\n",
        "c(ABC D) can be larger or smaller than c(AB D)\n",
        "– But confidence of rules generated from the same itemset has property\n",
        "– e.g., L = {A,B,C,D}:\n",
        "c(ABC  D)  c(AB  CD)  c(A  BCD)\n",
        "• Confidence is anti-monotone w.r.t. number of items on the RHS of the rule\n",
        "\n",
        "Low\n",
        "Confidence\n",
        "Rule\n",
        "NPTEL\n",
        "Rule Generation for Apriori Algorithm\n",
        "• Candidate rule is generated by merging two rules that share the same prefix\n",
        "in the rule consequent\n",
        "• join(CD=>AB,BD=>AC)\n",
        "would produce the candidate\n",
        "rule D ABC\n",
        "• Prune rule D=>ABC if its\n",
        "subset AD=>BC does not have\n",
        "high confidence\n",
        "CD=>AB BD=>AC\n",
        "D=>ABC\n",
        "NPTEL\n",
        "Pattern Evaluation\n",
        "• Association rule algorithms tend to produce too many rules\n",
        "– many of them are uninteresting or redundant\n",
        "– Redundant if {A,B,C}  {D} and {A,B}  {D}\n",
        "have same support & confidence\n",
        "• Interestingness measures can be used to prune/rank the derived\n",
        "patterns\n",
        "• In the original formulation of association rules, support &\n",
        "confidence are the only measures used\n",
        "NPTEL\n",
        "Application of Interestingness Measure\n",
        "Interestingness\n",
        "Measures\n",
        "NPTEL\n",
        "Computing Interestingness Measure\n",
        "• Given a rule X  Y, information needed to compute rule interestingness\n",
        "can be obtained from a contingency table\n",
        "\n",
        "Association Rule: Tea  Coffee\n",
        "Confidence= P(Coffee|Tea) = 0.75\n",
        "but P(Coffee) = 0.9\n",
        " Lift = 0.75/0.9= 0.8333 (< 1, therefore is negatively associated)\n",
        "NPTEL\n",
        "There are lots of\n",
        "measures proposed in\n",
        "the literature\n",
        "Some measures are\n",
        "good for certain\n",
        "applications, but not for\n",
        "others\n",
        "What criteria should we\n",
        "use to determine\n",
        "whether a measure is\n",
        "good or bad?\n",
        "What about Apriori\u0002style support based\n",
        "pruning? How does it\n",
        "NPTEL\n",
        "Subjective Interestingness Measure\n",
        "• Objective measure:\n",
        "– Rank patterns based on statistics computed from data\n",
        "– e.g., 21 measures of association (support, confidence, Laplace, Gini,\n",
        "mutual information, Jaccard, etc).\n",
        "• Subjective measure:\n",
        "– Rank patterns according to user’s interpretation\n",
        "• A pattern is subjectively interesting if it contradicts the\n",
        "expectation of a user (Silberschatz & Tuzhilin)\n",
        "• A pattern is subjectively interesting if it is actionable\n",
        "(Silberschatz & Tuzhilin)\n",
        "NPTEL\n",
        "Interestingness via Unexpectedness\n",
        "• Need to model expectation of users (domain knowledge)\n",
        "• Need to combine expectation of users with evidence from data (i.e., extracted patterns)\n",
        "+ Pattern expected to be frequent\n",
        "-\n",
        "Pattern expected to be infrequent\n",
        "Pattern found to be frequent\n",
        "Pattern found to be infrequent\n",
        "\n",
        "Expected Patterns - NPTEL\n",
        " Unexpected Patterns\n",
        "End of Association Rule\n",
        "NPTEL\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import re\n",
        "\n",
        "\n",
        "sentences = Text.split('\\n')\n",
        "\n",
        "# Remove sentences with less than 5 words\n",
        "filtered_sentences = [sentence.strip() for sentence in sentences if len(sentence.split()) >= 5]\n",
        "\n",
        "# Join the filtered sentences back into a single text\n",
        "filtered_text = '\\n'.join(filtered_sentences)\n",
        "\n",
        "print(filtered_text)"
      ],
      "metadata": {
        "id": "kwuW3mLGK2Yi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "96bd063f-9f99-4e25-e829-fbe4932236cc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Week 1: Introduction, Association Rules\n",
            "Computer Science and Engineering, IIT Kharagpur\n",
            "• Clustering and Anomaly Detection\n",
            "Computer Science and Engineering, IIT Kharagpur\n",
            "• The Explosive Growth of Data: from terabytes to petabytes\n",
            "– Data collection and data availability\n",
            "• Automated data collection tools, database systems, Web, computerized society\n",
            "– Major sources of abundant data\n",
            "• Business: Web, e-commerce, transactions, stocks, …\n",
            "• Science: Remote sensing, bioinformatics, scientific simulation, …\n",
            "• Society and everyone: news, digital cameras, YouTube\n",
            "• We are drowning in data, but starving for knowledge!\n",
            "• “Necessity is the mother of invention”—Data mining—Automated analysis of massive data\n",
            "• Data mining (knowledge discovery from data)\n",
            "– Extraction of interesting (non-trivial, implicit, previously unknown and\n",
            "potentially useful) patterns or knowledge from huge amount of data\n",
            "– Knowledge discovery (mining) in databases (KDD), knowledge extraction,\n",
            "data/pattern analysis, data archeology, data dredging, information\n",
            "• Watch out: Is everything “data mining”?\n",
            "– Simple search and query processing\n",
            "Data Mining: Confluence of Multiple Disciplines\n",
            "Why Not Traditional Data Analysis?\n",
            "• Tremendous amount of data\n",
            "– Algorithms must be highly scalable to handle such as tera-bytes of data\n",
            "– Micro-array may have tens of thousands of dimensions\n",
            "• High complexity of data\n",
            "– Data streams and sensor data\n",
            "– Time-series data, temporal data, sequence data\n",
            "– Structure data, graphs, social networks and multi-linked data\n",
            "– Heterogeneous databases and legacy databases\n",
            "– Spatial, spatiotemporal, multimedia, text and Web data\n",
            "Data Mining: On What Kinds of Data?\n",
            "• Database-oriented data sets and applications\n",
            "– Relational database, data warehouse, transactional database\n",
            "• Advanced data sets and advanced applications\n",
            "– Data streams and sensor data\n",
            "– Time-series data, temporal data, sequence data (incl. bio-sequences)\n",
            "– Structure data, graphs, social networks and multi-linked data\n",
            "– Heterogeneous databases and legacy databases\n",
            "– Spatial data and spatiotemporal data\n",
            "• Multidimensional concept description: Characterization and discrimination\n",
            "– Generalize, summarize, and contrast data characteristics, e.g., dry vs. wet\n",
            "• Frequent patterns, association, correlation vs. causality\n",
            "– Tea  Sugar [0.5%, 75%] (Correlation or causality?)\n",
            "– Construct models (functions) that describe and distinguish classes or\n",
            "• E.g., classify countries based on (climate), or classify cars based on (gas\n",
            "– Predict some unknown or missing numerical values\n",
            "– Class label is unknown: Group data to form new classes, e.g., cluster houses to find\n",
            "– Maximizing intra-class similarity & minimizing interclass similarity\n",
            "– Outlier: Data object that does not comply with the general behavior of the data\n",
            "– Noise or exception? Useful in fraud detection, rare events analysis\n",
            "• Trend and evolution analysis\n",
            "– Trend and deviation: e.g., regression analysis\n",
            "– Sequential pattern mining: e.g., digital camera  large SD memory\n",
            "• Other pattern-directed or statistical analyses\n",
            "Major Issues in Data Mining\n",
            "– Mining different kinds of knowledge from diverse data types, e.g., bio, stream, Web\n",
            "– Performance: efficiency, effectiveness, and scalability\n",
            "– Pattern evaluation: the interestingness problem\n",
            "– Incorporation of background knowledge\n",
            "– Handling noise and incomplete data\n",
            "– Parallel, distributed and incremental mining methods\n",
            "– Integration of the discovered knowledge with existing one: knowledge fusion\n",
            "– Data mining query languages and ad-hoc mining\n",
            "– Expression and visualization of data mining results\n",
            "– Interactive mining of knowledge at multiple levels of abstraction\n",
            "• Applications and social impacts\n",
            "– Domain-specific data mining & invisible data mining\n",
            "– Protection of data security, integrity, and privacy\n",
            "Architecture: Typical Data Mining System\n",
            "data cleaning, integration, and selection\n",
            "• Learning the application domain\n",
            "– relevant prior knowledge and goals of application\n",
            "• Creating a target data set: data selection\n",
            "• Data cleaning and preprocessing: (may take 60% of effort!)\n",
            "• Data reduction and transformation\n",
            "– Find useful features, dimensionality/variable reduction, invariant representation\n",
            "• Choosing functions of data mining\n",
            "– summarization, classification, regression, association, clustering\n",
            "• Choosing the mining algorithm(s)\n",
            "• Data mining: search for patterns of interest\n",
            "• Pattern evaluation and knowledge presentation\n",
            "– visualization, transformation, removing redundant patterns, etc.\n",
            "• Use of discovered knowledge\n",
            "Computer Science and Engineering, IIT Kharagpur\n",
            "• Collection of data objects and their\n",
            "• An attribute is a property or\n",
            "– Examples: eye color of a person,\n",
            "– Attribute is also known as variable,\n",
            "• A collection of attributes describe\n",
            "– Object is also known as record,\n",
            "point, case, sample, entity, or\n",
            "• There are different types of attributes\n",
            "• Examples: ID numbers, eye color, zip codes\n",
            "• Examples: rankings (e.g., taste of potato chips on a scale from 1-\n",
            "10), grades, height in {tall, medium, short}\n",
            "• Examples: calendar dates, temperatures in Celsius or Fahrenheit.\n",
            "• Examples: temperature in Kelvin, length, time, counts\n",
            "• The type of an attribute depends on which of the following\n",
            "Nominal The values of a nominal attribute are just\n",
            "different names, i.e., nominal attributes\n",
            "provide only enough information to\n",
            "distinguish one object from another. (=, )\n",
            "Ordinal The values of an ordinal attribute\n",
            "provide enough information to order\n",
            "Interval For interval attributes, the differences\n",
            "between values are meaningful, i.e., a unit\n",
            "Ratio For ratio variables, both differences\n",
            "and ratios are meaningful. (*, /)\n",
            "– Has only a finite or countably infinite set of values\n",
            "– Examples: zip codes, counts, or the set of words in a collection of\n",
            "– Often represented as integer variables.\n",
            "– Note: binary attributes are a special case of discrete attributes\n",
            "– Has real numbers as attribute values\n",
            "– Examples: temperature, height, or weight.\n",
            "– Practically, real values can only be measured and represented using a\n",
            "– Continuous attributes are typically represented as floating-point variables.\n",
            "• Data that consists of a collection of records, each of which consists of a\n",
            "fixed set of attributes Tid Refund Marital\n",
            "• If data objects have the same fixed set of numeric attributes, then\n",
            "the data objects can be thought of as points in a multi-dimensional\n",
            "space, where each dimension represents a distinct attribute\n",
            "• Such data set can be represented by an m by n matrix, where there\n",
            "are m rows, one for each object, and n columns, one for each\n",
            "• Each document becomes a `term' vector,\n",
            "each term is a component (attribute) of the vector,\n",
            "– the value of each component is the number of times the corresponding term\n",
            "• A special type of record data, where\n",
            "– each record (transaction) involves a set of items.\n",
            "– For example, consider a grocery store. The set of products purchased\n",
            "by a customer during one shopping trip constitute a transaction, while\n",
            "the individual products that were purchased are the items.\n",
            "• Examples: Facebook graph and HTML Links\n",
            "• What kinds of data quality problems?\n",
            "• How can we detect problems with the data?\n",
            "• What can we do about these problems?\n",
            "• Examples of data quality problems:\n",
            "• Noise refers to modification of original values\n",
            "– Examples: distortion of a person’s voice when talking on a poor phone\n",
            "and “snow” on television screen\n",
            "Two Sine Waves Two Sine Waves + Noise\n",
            "• Outliers are data objects with characteristics that are\n",
            "considerably different than most of the other data objects in\n",
            "• Reasons for missing values\n",
            "– Information is not collected\n",
            "(e.g., people decline to give their age and weight)\n",
            "– Attributes may not be applicable to all cases\n",
            "(e.g., annual income is not applicable to children)\n",
            "– Ignore the Missing Value During Analysis\n",
            "– Replace with all possible values (weighted by their probabilities)\n",
            "• Data set may include data objects that are duplicates, or\n",
            "almost duplicates of one another\n",
            "– Major issue when merging data from heterogenous sources\n",
            "– Same person with multiple email addresses\n",
            "– Process of dealing with duplicate data issues\n",
            "• Combining two or more attributes (or objects) into a\n",
            "• Reduce the number of attributes or objects\n",
            "• Cities aggregated into regions, states, countries, etc\n",
            "• Aggregated data tends to have less variability\n",
            "• Sampling is the main technique employed for data selection.\n",
            "– It is often used for both the preliminary investigation of the data and the final\n",
            "• Statisticians sample because obtaining the entire set of data of interest\n",
            "is too expensive or time consuming.\n",
            "• Sampling is used in data mining because processing the entire set of\n",
            "data of interest is too expensive or time consuming.\n",
            "• The key principle for effective sampling is the\n",
            "– using a sample will work almost as well as using the\n",
            "entire data sets, if the sample is representative\n",
            "– A sample is representative if it has approximately the\n",
            "same property (of interest) as the original set of data\n",
            "– There is an equal probability of selecting any particular item\n",
            "– As each item is selected, it is removed from the population\n",
            "– Objects are not removed from the population as they are selected for the sample.\n",
            "• In sampling with replacement, the same object can be picked up more than once\n",
            "– Split the data into several partitions; then draw random samples from each\n",
            "• When dimensionality increases, data becomes increasingly sparse\n",
            "in the space that it occupies\n",
            "• Definitions of density and distance between points, which is\n",
            "critical for clustering and outlier detection, become less\n",
            "– Avoid curse of dimensionality\n",
            "– Reduce amount of time and memory required by data mining\n",
            "– Allow data to be more easily visualized\n",
            "– May help to eliminate irrelevant features or reduce noise\n",
            "– Others: supervised and non-linear techniques\n",
            "• A function that maps the entire set of values of\n",
            "a given attribute to a new set of replacement\n",
            "values such that each old value can be\n",
            "identified with one of the new values\n",
            "– Numerical measure of how alike two data objects are.\n",
            "– Is higher when objects are more alike.\n",
            "– Often falls in the range [0,1]\n",
            "– Numerical measure of how different are two data objects\n",
            "– Lower when objects are more alike\n",
            "– Minimum dissimilarity is often 0\n",
            "• Proximity refers to a similarity or dissimilarity\n",
            "p and q are the attribute values for two data objects.\n",
            "Where n is the number of dimensions (attributes) and pk\n",
            "th attributes (components) or data objects p and q.\n",
            "• Given a set of transactions, find rules that will predict the\n",
            "occurrence of an item based on the occurrences of other items in\n",
            "– A collection of one or more items\n",
            "• Example: {Milk, Bread, Diaper}\n",
            "• An itemset that contains k items\n",
            "– Frequency of occurrence of an itemset\n",
            "– E.g. ({Milk, Bread,Diaper}) = 2\n",
            "– Fraction of transactions that contain an itemset\n",
            "– E.g. s({Milk, Bread, Diaper}) = 2/5\n",
            "– An itemset whose support is greater than or equal to a\n",
            "– An implication expression of the form X  Y, where X\n",
            " Fraction of transactions that contain both X and\n",
            " Measures how often items in Y\n",
            "• Given a set of transactions T, the goal of association rule mining is\n",
            "to find all rules having\n",
            "– support ≥ minsup threshold\n",
            "– confidence ≥ minconf threshold\n",
            "– List all possible association rules\n",
            "– Compute the support and confidence for each rule\n",
            "– Prune rules that fail the minsup and minconf thresholds\n",
            "– Generate all itemsets whose support  minsup\n",
            "– Generate high confidence rules from each frequent\n",
            "itemset, where each rule is a binary partitioning of a\n",
            "– Each itemset in the lattice is a candidate frequent itemset\n",
            "– Count the support of each candidate by scanning the database\n",
            "– Match each transaction against every candidate\n",
            "• Reduce the number of candidates (M)\n",
            "– Use pruning techniques to reduce M\n",
            "• Reduce the number of transactions (N)\n",
            "– Reduce size of N as the size of itemset increases\n",
            "– Used by DHP and vertical-based mining algorithms\n",
            "• Reduce the number of comparisons (NM)\n",
            "– Use efficient data structures to store the candidates or transactions\n",
            "– No need to match every candidate against every transaction\n",
            "– If an itemset is frequent, then all of its subsets must also be\n",
            "• Apriori principle holds due to the following property of the\n",
            "– Support of an itemset never exceeds the support of its subsets\n",
            "– This is known as the anti-monotone property of support\n",
            "Triplets (3-itemsets) Minimum Support = 3\n",
            "If every subset is considered,\n",
            "– Generate frequent itemsets of length 1\n",
            "– Repeat until no new frequent itemsets are identified\n",
            "• Generate length (k+1) candidate itemsets from length k frequent\n",
            "• Prune candidate itemsets containing subsets of length k that are\n",
            "• Count the support of each candidate by scanning the DB\n",
            "• Eliminate candidates that are infrequent, leaving only those that\n",
            "• Choice of minimum support threshold\n",
            "– lowering support threshold results in more frequent itemsets\n",
            "– this may increase number of candidates and max length of frequent itemsets\n",
            "• Dimensionality (number of items) of the data set\n",
            "– more space is needed to store support count of each item\n",
            "– if number of frequent items also increases, both computation and I/O costs may\n",
            "– Apriori makes multiple passes, run time of algorithm increase with number of\n",
            "– This may increase max length of frequent itemsets and traversals of hash tree\n",
            "(number of subsets in a transaction increases with its width)\n",
            "• How to efficiently generate rules from frequent itemsets?\n",
            "– In general, confidence does not have an anti-monotone property\n",
            "c(ABC D) can be larger or smaller than c(AB D)\n",
            "– But confidence of rules generated from the same itemset has property\n",
            "– e.g., L = {A,B,C,D}:\n",
            "c(ABC  D)  c(AB  CD)  c(A  BCD)\n",
            "• Confidence is anti-monotone w.r.t. number of items on the RHS of the rule\n",
            "Rule Generation for Apriori Algorithm\n",
            "• Candidate rule is generated by merging two rules that share the same prefix\n",
            "• Prune rule D=>ABC if its\n",
            "subset AD=>BC does not have\n",
            "• Association rule algorithms tend to produce too many rules\n",
            "– many of them are uninteresting or redundant\n",
            "– Redundant if {A,B,C}  {D} and {A,B}  {D}\n",
            "have same support & confidence\n",
            "• Interestingness measures can be used to prune/rank the derived\n",
            "• In the original formulation of association rules, support &\n",
            "confidence are the only measures used\n",
            "• Given a rule X  Y, information needed to compute rule interestingness\n",
            "can be obtained from a contingency table\n",
            "Association Rule: Tea  Coffee\n",
            " Lift = 0.75/0.9= 0.8333 (< 1, therefore is negatively associated)\n",
            "What about Apriori\u0002style support based\n",
            "– Rank patterns based on statistics computed from data\n",
            "– e.g., 21 measures of association (support, confidence, Laplace, Gini,\n",
            "– Rank patterns according to user’s interpretation\n",
            "• A pattern is subjectively interesting if it contradicts the\n",
            "expectation of a user (Silberschatz & Tuzhilin)\n",
            "• A pattern is subjectively interesting if it is actionable\n",
            "• Need to model expectation of users (domain knowledge)\n",
            "• Need to combine expectation of users with evidence from data (i.e., extracted patterns)\n",
            "+ Pattern expected to be frequent\n",
            "Pattern expected to be infrequent\n",
            "Pattern found to be frequent\n",
            "Pattern found to be infrequent\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer"
      ],
      "metadata": {
        "id": "5pBm2LQPSMZX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = Tokenizer()"
      ],
      "metadata": {
        "id": "UGrrlGNeTQd4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.fit_on_texts([filtered_text])"
      ],
      "metadata": {
        "id": "yrMH_EwXTVxr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.word_index"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_Y_SLA5KTsB_",
        "outputId": "c85f9efa-7c74-48dd-d02b-36a86b90a1df"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'of': 1,\n",
              " '–': 2,\n",
              " 'data': 3,\n",
              " '•': 4,\n",
              " 'the': 5,\n",
              " 'and': 6,\n",
              " 'a': 7,\n",
              " 'is': 8,\n",
              " 'to': 9,\n",
              " 'or': 10,\n",
              " 'are': 11,\n",
              " 'mining': 12,\n",
              " 'in': 13,\n",
              " 'each': 14,\n",
              " 'that': 15,\n",
              " 'from': 16,\n",
              " 'e': 17,\n",
              " 'set': 18,\n",
              " 'support': 19,\n",
              " 'for': 20,\n",
              " 'be': 21,\n",
              " 'objects': 22,\n",
              " 'knowledge': 23,\n",
              " 'as': 24,\n",
              " 'frequent': 25,\n",
              " 'values': 26,\n",
              " 'an': 27,\n",
              " 'attributes': 28,\n",
              " 'g': 29,\n",
              " 'with': 30,\n",
              " 'number': 31,\n",
              " 'rules': 32,\n",
              " 'pattern': 33,\n",
              " 'itemset': 34,\n",
              " 'rule': 35,\n",
              " 'on': 36,\n",
              " 'attribute': 37,\n",
              " 'examples': 38,\n",
              " 'can': 39,\n",
              " 'if': 40,\n",
              " 'itemsets': 41,\n",
              " 'association': 42,\n",
              " 'by': 43,\n",
              " 'items': 44,\n",
              " 'confidence': 45,\n",
              " 'not': 46,\n",
              " 'may': 47,\n",
              " 'one': 48,\n",
              " 'more': 49,\n",
              " 'reduce': 50,\n",
              " '\\uf0ae': 51,\n",
              " 'candidate': 52,\n",
              " 'collection': 53,\n",
              " 'transactions': 54,\n",
              " 'analysis': 55,\n",
              " 'patterns': 56,\n",
              " 'time': 57,\n",
              " 'length': 58,\n",
              " 'same': 59,\n",
              " 'two': 60,\n",
              " 'c': 61,\n",
              " '1': 62,\n",
              " 'have': 63,\n",
              " '0': 64,\n",
              " 'based': 65,\n",
              " 'property': 66,\n",
              " 'sample': 67,\n",
              " 'where': 68,\n",
              " 'all': 69,\n",
              " 'it': 70,\n",
              " 'database': 71,\n",
              " 'databases': 72,\n",
              " 'information': 73,\n",
              " 'object': 74,\n",
              " 'noise': 75,\n",
              " 'different': 76,\n",
              " 'only': 77,\n",
              " 'often': 78,\n",
              " 'n': 79,\n",
              " 'transaction': 80,\n",
              " 'how': 81,\n",
              " 'when': 82,\n",
              " 'used': 83,\n",
              " 'generate': 84,\n",
              " 'd': 85,\n",
              " 'science': 86,\n",
              " 'web': 87,\n",
              " 'multiple': 88,\n",
              " 'what': 89,\n",
              " 'new': 90,\n",
              " 'find': 91,\n",
              " 'dimensionality': 92,\n",
              " 'interest': 93,\n",
              " 'also': 94,\n",
              " 'i': 95,\n",
              " 'both': 96,\n",
              " 'has': 97,\n",
              " 'represented': 98,\n",
              " 'm': 99,\n",
              " 'problems': 100,\n",
              " 'than': 101,\n",
              " 'sampling': 102,\n",
              " 'entire': 103,\n",
              " 'item': 104,\n",
              " 'increases': 105,\n",
              " 'given': 106,\n",
              " 'k': 107,\n",
              " 'x': 108,\n",
              " 'measures': 109,\n",
              " 'threshold': 110,\n",
              " 'prune': 111,\n",
              " 'every': 112,\n",
              " 'candidates': 113,\n",
              " 'its': 114,\n",
              " 'subsets': 115,\n",
              " 'computer': 116,\n",
              " 'engineering': 117,\n",
              " 'iit': 118,\n",
              " 'kharagpur': 119,\n",
              " 'clustering': 120,\n",
              " 'detection': 121,\n",
              " 'major': 122,\n",
              " 'we': 123,\n",
              " 'interesting': 124,\n",
              " 'unknown': 125,\n",
              " 'useful': 126,\n",
              " 'amount': 127,\n",
              " 'algorithms': 128,\n",
              " 'such': 129,\n",
              " 'social': 130,\n",
              " 'multi': 131,\n",
              " 'kinds': 132,\n",
              " 'sets': 133,\n",
              " 'applications': 134,\n",
              " 'missing': 135,\n",
              " 'numerical': 136,\n",
              " 'similarity': 137,\n",
              " 'does': 138,\n",
              " 'other': 139,\n",
              " 'interestingness': 140,\n",
              " 'domain': 141,\n",
              " 'selection': 142,\n",
              " 'algorithm': 143,\n",
              " 'redundant': 144,\n",
              " 'use': 145,\n",
              " 'their': 146,\n",
              " 'known': 147,\n",
              " 'record': 148,\n",
              " 'there': 149,\n",
              " 'which': 150,\n",
              " 'nominal': 151,\n",
              " 'variables': 152,\n",
              " 'using': 153,\n",
              " 'then': 154,\n",
              " 'space': 155,\n",
              " 'value': 156,\n",
              " 'store': 157,\n",
              " 'original': 158,\n",
              " 'into': 159,\n",
              " 'too': 160,\n",
              " 'alike': 161,\n",
              " 'minimum': 162,\n",
              " 'milk': 163,\n",
              " 'bread': 164,\n",
              " 'diaper': 165,\n",
              " 'y': 166,\n",
              " 'minsup': 167,\n",
              " '\\uf0b3': 168,\n",
              " 'count': 169,\n",
              " 'need': 170,\n",
              " 'apriori': 171,\n",
              " 'this': 172,\n",
              " 'anti': 173,\n",
              " 'monotone': 174,\n",
              " 'infrequent': 175,\n",
              " 'increase': 176,\n",
              " 'abc': 177,\n",
              " 'b': 178,\n",
              " 'rank': 179,\n",
              " 'expectation': 180,\n",
              " 'society': 181,\n",
              " 'sources': 182,\n",
              " '…': 183,\n",
              " 'digital': 184,\n",
              " 'but': 185,\n",
              " 'discovery': 186,\n",
              " 'extraction': 187,\n",
              " 'non': 188,\n",
              " 'search': 189,\n",
              " 'query': 190,\n",
              " 'processing': 191,\n",
              " 'must': 192,\n",
              " 'dimensions': 193,\n",
              " 'high': 194,\n",
              " 'streams': 195,\n",
              " 'sensor': 196,\n",
              " 'series': 197,\n",
              " 'temporal': 198,\n",
              " 'sequence': 199,\n",
              " 'structure': 200,\n",
              " 'graphs': 201,\n",
              " 'networks': 202,\n",
              " 'linked': 203,\n",
              " 'heterogeneous': 204,\n",
              " 'legacy': 205,\n",
              " 'spatial': 206,\n",
              " 'spatiotemporal': 207,\n",
              " 'advanced': 208,\n",
              " 'bio': 209,\n",
              " 'characteristics': 210,\n",
              " 'vs': 211,\n",
              " 'correlation': 212,\n",
              " 'causality': 213,\n",
              " 'tea': 214,\n",
              " '\\uf0e0': 215,\n",
              " '5': 216,\n",
              " '75': 217,\n",
              " 'functions': 218,\n",
              " 'describe': 219,\n",
              " 'distinguish': 220,\n",
              " 'classes': 221,\n",
              " 'classify': 222,\n",
              " 'countries': 223,\n",
              " 'predict': 224,\n",
              " 'class': 225,\n",
              " 'form': 226,\n",
              " 'outlier': 227,\n",
              " 'general': 228,\n",
              " 'trend': 229,\n",
              " 'regression': 230,\n",
              " 'memory': 231,\n",
              " 'issues': 232,\n",
              " 'types': 233,\n",
              " 'evaluation': 234,\n",
              " 'integration': 235,\n",
              " 'discovered': 236,\n",
              " 'ad': 237,\n",
              " 'expression': 238,\n",
              " 'visualization': 239,\n",
              " 'results': 240,\n",
              " 'cleaning': 241,\n",
              " 'application': 242,\n",
              " 'reduction': 243,\n",
              " 'transformation': 244,\n",
              " 'features': 245,\n",
              " 'variable': 246,\n",
              " 'choosing': 247,\n",
              " 's': 248,\n",
              " 'etc': 249,\n",
              " 'eye': 250,\n",
              " 'color': 251,\n",
              " 'person': 252,\n",
              " 'point': 253,\n",
              " 'case': 254,\n",
              " 'numbers': 255,\n",
              " 'zip': 256,\n",
              " 'codes': 257,\n",
              " 'height': 258,\n",
              " 'temperature': 259,\n",
              " 'counts': 260,\n",
              " 'type': 261,\n",
              " 'following': 262,\n",
              " 'provide': 263,\n",
              " 'enough': 264,\n",
              " 'another': 265,\n",
              " 'ordinal': 266,\n",
              " 'interval': 267,\n",
              " 'differences': 268,\n",
              " 'between': 269,\n",
              " 'meaningful': 270,\n",
              " 'ratio': 271,\n",
              " 'binary': 272,\n",
              " 'special': 273,\n",
              " 'real': 274,\n",
              " 'weight': 275,\n",
              " 'consists': 276,\n",
              " 'fixed': 277,\n",
              " 'points': 278,\n",
              " 'becomes': 279,\n",
              " 'vector': 280,\n",
              " 'term': 281,\n",
              " 'component': 282,\n",
              " 'example': 283,\n",
              " 'products': 284,\n",
              " 'purchased': 285,\n",
              " 'during': 286,\n",
              " 'quality': 287,\n",
              " 'about': 288,\n",
              " 'refers': 289,\n",
              " 'sine': 290,\n",
              " 'waves': 291,\n",
              " 'applicable': 292,\n",
              " 'possible': 293,\n",
              " 'duplicates': 294,\n",
              " 'almost': 295,\n",
              " 'merging': 296,\n",
              " 'aggregated': 297,\n",
              " 'less': 298,\n",
              " 'because': 299,\n",
              " 'expensive': 300,\n",
              " 'consuming': 301,\n",
              " 'principle': 302,\n",
              " 'will': 303,\n",
              " 'representative': 304,\n",
              " 'equal': 305,\n",
              " 'selected': 306,\n",
              " 'removed': 307,\n",
              " 'population': 308,\n",
              " 'replacement': 309,\n",
              " 'eliminate': 310,\n",
              " 'techniques': 311,\n",
              " 'identified': 312,\n",
              " 'measure': 313,\n",
              " 'dissimilarity': 314,\n",
              " 'p': 315,\n",
              " 'q': 316,\n",
              " 'occurrence': 317,\n",
              " '2': 318,\n",
              " 'fraction': 319,\n",
              " 'contain': 320,\n",
              " 'whose': 321,\n",
              " '\\uf075': 322,\n",
              " 't': 323,\n",
              " '≥': 324,\n",
              " 'minconf': 325,\n",
              " 'compute': 326,\n",
              " 'scanning': 327,\n",
              " 'match': 328,\n",
              " 'against': 329,\n",
              " 'size': 330,\n",
              " 'no': 331,\n",
              " '3': 332,\n",
              " 'subset': 333,\n",
              " 'max': 334,\n",
              " 'needed': 335,\n",
              " '\\uf0aed': 336,\n",
              " 'ab': 337,\n",
              " 'generated': 338,\n",
              " 'many': 339,\n",
              " 'subjectively': 340,\n",
              " 'users': 341,\n",
              " 'expected': 342,\n",
              " 'found': 343,\n",
              " 'week': 344,\n",
              " 'introduction': 345,\n",
              " 'anomaly': 346,\n",
              " 'explosive': 347,\n",
              " 'growth': 348,\n",
              " 'terabytes': 349,\n",
              " 'petabytes': 350,\n",
              " 'availability': 351,\n",
              " 'automated': 352,\n",
              " 'tools': 353,\n",
              " 'systems': 354,\n",
              " 'computerized': 355,\n",
              " 'abundant': 356,\n",
              " 'business': 357,\n",
              " 'commerce': 358,\n",
              " 'stocks': 359,\n",
              " 'remote': 360,\n",
              " 'sensing': 361,\n",
              " 'bioinformatics': 362,\n",
              " 'scientific': 363,\n",
              " 'simulation': 364,\n",
              " 'everyone': 365,\n",
              " 'news': 366,\n",
              " 'cameras': 367,\n",
              " 'youtube': 368,\n",
              " 'drowning': 369,\n",
              " 'starving': 370,\n",
              " '“necessity': 371,\n",
              " 'mother': 372,\n",
              " 'invention”—data': 373,\n",
              " 'mining—automated': 374,\n",
              " 'massive': 375,\n",
              " 'trivial': 376,\n",
              " 'implicit': 377,\n",
              " 'previously': 378,\n",
              " 'potentially': 379,\n",
              " 'huge': 380,\n",
              " 'kdd': 381,\n",
              " 'archeology': 382,\n",
              " 'dredging': 383,\n",
              " 'watch': 384,\n",
              " 'out': 385,\n",
              " 'everything': 386,\n",
              " '“data': 387,\n",
              " 'mining”': 388,\n",
              " 'simple': 389,\n",
              " 'confluence': 390,\n",
              " 'disciplines': 391,\n",
              " 'why': 392,\n",
              " 'traditional': 393,\n",
              " 'tremendous': 394,\n",
              " 'highly': 395,\n",
              " 'scalable': 396,\n",
              " 'handle': 397,\n",
              " 'tera': 398,\n",
              " 'bytes': 399,\n",
              " 'micro': 400,\n",
              " 'array': 401,\n",
              " 'tens': 402,\n",
              " 'thousands': 403,\n",
              " 'complexity': 404,\n",
              " 'multimedia': 405,\n",
              " 'text': 406,\n",
              " 'oriented': 407,\n",
              " 'relational': 408,\n",
              " 'warehouse': 409,\n",
              " 'transactional': 410,\n",
              " 'incl': 411,\n",
              " 'sequences': 412,\n",
              " 'multidimensional': 413,\n",
              " 'concept': 414,\n",
              " 'description': 415,\n",
              " 'characterization': 416,\n",
              " 'discrimination': 417,\n",
              " 'generalize': 418,\n",
              " 'summarize': 419,\n",
              " 'contrast': 420,\n",
              " 'dry': 421,\n",
              " 'wet': 422,\n",
              " 'sugar': 423,\n",
              " 'construct': 424,\n",
              " 'models': 425,\n",
              " 'climate': 426,\n",
              " 'cars': 427,\n",
              " 'gas': 428,\n",
              " 'some': 429,\n",
              " 'label': 430,\n",
              " 'group': 431,\n",
              " 'cluster': 432,\n",
              " 'houses': 433,\n",
              " 'maximizing': 434,\n",
              " 'intra': 435,\n",
              " 'minimizing': 436,\n",
              " 'interclass': 437,\n",
              " 'comply': 438,\n",
              " 'behavior': 439,\n",
              " 'exception': 440,\n",
              " 'fraud': 441,\n",
              " 'rare': 442,\n",
              " 'events': 443,\n",
              " 'evolution': 444,\n",
              " 'deviation': 445,\n",
              " 'sequential': 446,\n",
              " 'camera': 447,\n",
              " 'large': 448,\n",
              " 'sd': 449,\n",
              " 'directed': 450,\n",
              " 'statistical': 451,\n",
              " 'analyses': 452,\n",
              " 'diverse': 453,\n",
              " 'stream': 454,\n",
              " 'performance': 455,\n",
              " 'efficiency': 456,\n",
              " 'effectiveness': 457,\n",
              " 'scalability': 458,\n",
              " 'problem': 459,\n",
              " 'incorporation': 460,\n",
              " 'background': 461,\n",
              " 'handling': 462,\n",
              " 'incomplete': 463,\n",
              " 'parallel': 464,\n",
              " 'distributed': 465,\n",
              " 'incremental': 466,\n",
              " 'methods': 467,\n",
              " 'existing': 468,\n",
              " 'fusion': 469,\n",
              " 'languages': 470,\n",
              " 'hoc': 471,\n",
              " 'interactive': 472,\n",
              " 'at': 473,\n",
              " 'levels': 474,\n",
              " 'abstraction': 475,\n",
              " 'impacts': 476,\n",
              " 'specific': 477,\n",
              " 'invisible': 478,\n",
              " 'protection': 479,\n",
              " 'security': 480,\n",
              " 'integrity': 481,\n",
              " 'privacy': 482,\n",
              " 'architecture': 483,\n",
              " 'typical': 484,\n",
              " 'system': 485,\n",
              " 'learning': 486,\n",
              " 'relevant': 487,\n",
              " 'prior': 488,\n",
              " 'goals': 489,\n",
              " 'creating': 490,\n",
              " 'target': 491,\n",
              " 'preprocessing': 492,\n",
              " 'take': 493,\n",
              " '60': 494,\n",
              " 'effort': 495,\n",
              " 'invariant': 496,\n",
              " 'representation': 497,\n",
              " 'summarization': 498,\n",
              " 'classification': 499,\n",
              " 'presentation': 500,\n",
              " 'removing': 501,\n",
              " 'entity': 502,\n",
              " 'id': 503,\n",
              " 'rankings': 504,\n",
              " 'taste': 505,\n",
              " 'potato': 506,\n",
              " 'chips': 507,\n",
              " 'scale': 508,\n",
              " '10': 509,\n",
              " 'grades': 510,\n",
              " 'tall': 511,\n",
              " 'medium': 512,\n",
              " 'short': 513,\n",
              " 'calendar': 514,\n",
              " 'dates': 515,\n",
              " 'temperatures': 516,\n",
              " 'celsius': 517,\n",
              " 'fahrenheit': 518,\n",
              " 'kelvin': 519,\n",
              " 'depends': 520,\n",
              " 'just': 521,\n",
              " 'names': 522,\n",
              " '\\uf0b9': 523,\n",
              " 'order': 524,\n",
              " 'unit': 525,\n",
              " 'ratios': 526,\n",
              " 'finite': 527,\n",
              " 'countably': 528,\n",
              " 'infinite': 529,\n",
              " 'words': 530,\n",
              " 'integer': 531,\n",
              " 'note': 532,\n",
              " 'discrete': 533,\n",
              " 'practically': 534,\n",
              " 'measured': 535,\n",
              " 'continuous': 536,\n",
              " 'typically': 537,\n",
              " 'floating': 538,\n",
              " 'records': 539,\n",
              " 'tid': 540,\n",
              " 'refund': 541,\n",
              " 'marital': 542,\n",
              " 'numeric': 543,\n",
              " 'thought': 544,\n",
              " 'dimensional': 545,\n",
              " 'dimension': 546,\n",
              " 'represents': 547,\n",
              " 'distinct': 548,\n",
              " 'matrix': 549,\n",
              " 'rows': 550,\n",
              " 'columns': 551,\n",
              " 'document': 552,\n",
              " \"term'\": 553,\n",
              " 'times': 554,\n",
              " 'corresponding': 555,\n",
              " 'involves': 556,\n",
              " 'consider': 557,\n",
              " 'grocery': 558,\n",
              " 'customer': 559,\n",
              " 'shopping': 560,\n",
              " 'trip': 561,\n",
              " 'constitute': 562,\n",
              " 'while': 563,\n",
              " 'individual': 564,\n",
              " 'were': 565,\n",
              " 'facebook': 566,\n",
              " 'graph': 567,\n",
              " 'html': 568,\n",
              " 'links': 569,\n",
              " 'detect': 570,\n",
              " 'do': 571,\n",
              " 'these': 572,\n",
              " 'modification': 573,\n",
              " 'distortion': 574,\n",
              " 'person’s': 575,\n",
              " 'voice': 576,\n",
              " 'talking': 577,\n",
              " 'poor': 578,\n",
              " 'phone': 579,\n",
              " '“snow”': 580,\n",
              " 'television': 581,\n",
              " 'screen': 582,\n",
              " 'outliers': 583,\n",
              " 'considerably': 584,\n",
              " 'most': 585,\n",
              " 'reasons': 586,\n",
              " 'collected': 587,\n",
              " 'people': 588,\n",
              " 'decline': 589,\n",
              " 'give': 590,\n",
              " 'age': 591,\n",
              " 'cases': 592,\n",
              " 'annual': 593,\n",
              " 'income': 594,\n",
              " 'children': 595,\n",
              " 'ignore': 596,\n",
              " 'replace': 597,\n",
              " 'weighted': 598,\n",
              " 'probabilities': 599,\n",
              " 'include': 600,\n",
              " 'issue': 601,\n",
              " 'heterogenous': 602,\n",
              " 'email': 603,\n",
              " 'addresses': 604,\n",
              " 'process': 605,\n",
              " 'dealing': 606,\n",
              " 'duplicate': 607,\n",
              " 'combining': 608,\n",
              " 'cities': 609,\n",
              " 'regions': 610,\n",
              " 'states': 611,\n",
              " 'tends': 612,\n",
              " 'variability': 613,\n",
              " 'main': 614,\n",
              " 'technique': 615,\n",
              " 'employed': 616,\n",
              " 'preliminary': 617,\n",
              " 'investigation': 618,\n",
              " 'final': 619,\n",
              " 'statisticians': 620,\n",
              " 'obtaining': 621,\n",
              " 'key': 622,\n",
              " 'effective': 623,\n",
              " 'work': 624,\n",
              " 'well': 625,\n",
              " 'approximately': 626,\n",
              " 'probability': 627,\n",
              " 'selecting': 628,\n",
              " 'any': 629,\n",
              " 'particular': 630,\n",
              " 'they': 631,\n",
              " 'picked': 632,\n",
              " 'up': 633,\n",
              " 'once': 634,\n",
              " 'split': 635,\n",
              " 'several': 636,\n",
              " 'partitions': 637,\n",
              " 'draw': 638,\n",
              " 'random': 639,\n",
              " 'samples': 640,\n",
              " 'increasingly': 641,\n",
              " 'sparse': 642,\n",
              " 'occupies': 643,\n",
              " 'definitions': 644,\n",
              " 'density': 645,\n",
              " 'distance': 646,\n",
              " 'critical': 647,\n",
              " 'become': 648,\n",
              " 'avoid': 649,\n",
              " 'curse': 650,\n",
              " 'required': 651,\n",
              " 'allow': 652,\n",
              " 'easily': 653,\n",
              " 'visualized': 654,\n",
              " 'help': 655,\n",
              " 'irrelevant': 656,\n",
              " 'others': 657,\n",
              " 'supervised': 658,\n",
              " 'linear': 659,\n",
              " 'function': 660,\n",
              " 'maps': 661,\n",
              " 'old': 662,\n",
              " 'higher': 663,\n",
              " 'falls': 664,\n",
              " 'range': 665,\n",
              " 'lower': 666,\n",
              " 'proximity': 667,\n",
              " 'pk': 668,\n",
              " 'th': 669,\n",
              " 'components': 670,\n",
              " 'occurrences': 671,\n",
              " 'contains': 672,\n",
              " 'frequency': 673,\n",
              " '\\uf073': 674,\n",
              " 'greater': 675,\n",
              " 'implication': 676,\n",
              " 'goal': 677,\n",
              " 'having': 678,\n",
              " 'list': 679,\n",
              " 'fail': 680,\n",
              " 'thresholds': 681,\n",
              " 'partitioning': 682,\n",
              " 'lattice': 683,\n",
              " 'pruning': 684,\n",
              " 'dhp': 685,\n",
              " 'vertical': 686,\n",
              " 'comparisons': 687,\n",
              " 'nm': 688,\n",
              " 'efficient': 689,\n",
              " 'structures': 690,\n",
              " 'holds': 691,\n",
              " 'due': 692,\n",
              " 'never': 693,\n",
              " 'exceeds': 694,\n",
              " 'triplets': 695,\n",
              " 'considered': 696,\n",
              " 'repeat': 697,\n",
              " 'until': 698,\n",
              " 'containing': 699,\n",
              " 'db': 700,\n",
              " 'leaving': 701,\n",
              " 'those': 702,\n",
              " 'choice': 703,\n",
              " 'lowering': 704,\n",
              " 'computation': 705,\n",
              " 'o': 706,\n",
              " 'costs': 707,\n",
              " 'makes': 708,\n",
              " 'passes': 709,\n",
              " 'run': 710,\n",
              " 'traversals': 711,\n",
              " 'hash': 712,\n",
              " 'tree': 713,\n",
              " 'width': 714,\n",
              " 'efficiently': 715,\n",
              " 'larger': 716,\n",
              " 'smaller': 717,\n",
              " 'l': 718,\n",
              " 'cd': 719,\n",
              " 'bcd': 720,\n",
              " 'w': 721,\n",
              " 'r': 722,\n",
              " 'rhs': 723,\n",
              " 'generation': 724,\n",
              " 'share': 725,\n",
              " 'prefix': 726,\n",
              " 'bc': 727,\n",
              " 'tend': 728,\n",
              " 'produce': 729,\n",
              " 'them': 730,\n",
              " 'uninteresting': 731,\n",
              " 'derived': 732,\n",
              " 'formulation': 733,\n",
              " 'obtained': 734,\n",
              " 'contingency': 735,\n",
              " 'table': 736,\n",
              " 'coffee': 737,\n",
              " '\\uf0de': 738,\n",
              " 'lift': 739,\n",
              " '9': 740,\n",
              " '8333': 741,\n",
              " 'therefore': 742,\n",
              " 'negatively': 743,\n",
              " 'associated': 744,\n",
              " 'apriori\\x02style': 745,\n",
              " 'statistics': 746,\n",
              " 'computed': 747,\n",
              " '21': 748,\n",
              " 'laplace': 749,\n",
              " 'gini': 750,\n",
              " 'according': 751,\n",
              " 'user’s': 752,\n",
              " 'interpretation': 753,\n",
              " 'contradicts': 754,\n",
              " 'user': 755,\n",
              " 'silberschatz': 756,\n",
              " 'tuzhilin': 757,\n",
              " 'actionable': 758,\n",
              " 'model': 759,\n",
              " 'combine': 760,\n",
              " 'evidence': 761,\n",
              " 'extracted': 762}"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(tokenizer.word_index)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Wn9T49h1oy1N",
        "outputId": "42185068-2836-45f5-c780-96a403cc4b47"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "762"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "input_sequences = []\n",
        "for sentence in filtered_text.split('\\n'):\n",
        "  tokenized_sentence = tokenizer.texts_to_sequences([sentence])[0]\n",
        "\n",
        "  for i in range(1,len(tokenized_sentence)):\n",
        "    input_sequences.append(tokenized_sentence[:i+1])"
      ],
      "metadata": {
        "id": "XLFMqmPZTxX3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "input_sequences"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WvsSv-6kT0h2",
        "outputId": "efe2a40a-5d25-4b1c-f9ee-a6eeeba15934"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[344, 62],\n",
              " [344, 62, 345],\n",
              " [344, 62, 345, 42],\n",
              " [344, 62, 345, 42, 32],\n",
              " [116, 86],\n",
              " [116, 86, 6],\n",
              " [116, 86, 6, 117],\n",
              " [116, 86, 6, 117, 118],\n",
              " [116, 86, 6, 117, 118, 119],\n",
              " [4, 120],\n",
              " [4, 120, 6],\n",
              " [4, 120, 6, 346],\n",
              " [4, 120, 6, 346, 121],\n",
              " [116, 86],\n",
              " [116, 86, 6],\n",
              " [116, 86, 6, 117],\n",
              " [116, 86, 6, 117, 118],\n",
              " [116, 86, 6, 117, 118, 119],\n",
              " [4, 5],\n",
              " [4, 5, 347],\n",
              " [4, 5, 347, 348],\n",
              " [4, 5, 347, 348, 1],\n",
              " [4, 5, 347, 348, 1, 3],\n",
              " [4, 5, 347, 348, 1, 3, 16],\n",
              " [4, 5, 347, 348, 1, 3, 16, 349],\n",
              " [4, 5, 347, 348, 1, 3, 16, 349, 9],\n",
              " [4, 5, 347, 348, 1, 3, 16, 349, 9, 350],\n",
              " [2, 3],\n",
              " [2, 3, 53],\n",
              " [2, 3, 53, 6],\n",
              " [2, 3, 53, 6, 3],\n",
              " [2, 3, 53, 6, 3, 351],\n",
              " [4, 352],\n",
              " [4, 352, 3],\n",
              " [4, 352, 3, 53],\n",
              " [4, 352, 3, 53, 353],\n",
              " [4, 352, 3, 53, 353, 71],\n",
              " [4, 352, 3, 53, 353, 71, 354],\n",
              " [4, 352, 3, 53, 353, 71, 354, 87],\n",
              " [4, 352, 3, 53, 353, 71, 354, 87, 355],\n",
              " [4, 352, 3, 53, 353, 71, 354, 87, 355, 181],\n",
              " [2, 122],\n",
              " [2, 122, 182],\n",
              " [2, 122, 182, 1],\n",
              " [2, 122, 182, 1, 356],\n",
              " [2, 122, 182, 1, 356, 3],\n",
              " [4, 357],\n",
              " [4, 357, 87],\n",
              " [4, 357, 87, 17],\n",
              " [4, 357, 87, 17, 358],\n",
              " [4, 357, 87, 17, 358, 54],\n",
              " [4, 357, 87, 17, 358, 54, 359],\n",
              " [4, 357, 87, 17, 358, 54, 359, 183],\n",
              " [4, 86],\n",
              " [4, 86, 360],\n",
              " [4, 86, 360, 361],\n",
              " [4, 86, 360, 361, 362],\n",
              " [4, 86, 360, 361, 362, 363],\n",
              " [4, 86, 360, 361, 362, 363, 364],\n",
              " [4, 86, 360, 361, 362, 363, 364, 183],\n",
              " [4, 181],\n",
              " [4, 181, 6],\n",
              " [4, 181, 6, 365],\n",
              " [4, 181, 6, 365, 366],\n",
              " [4, 181, 6, 365, 366, 184],\n",
              " [4, 181, 6, 365, 366, 184, 367],\n",
              " [4, 181, 6, 365, 366, 184, 367, 368],\n",
              " [4, 123],\n",
              " [4, 123, 11],\n",
              " [4, 123, 11, 369],\n",
              " [4, 123, 11, 369, 13],\n",
              " [4, 123, 11, 369, 13, 3],\n",
              " [4, 123, 11, 369, 13, 3, 185],\n",
              " [4, 123, 11, 369, 13, 3, 185, 370],\n",
              " [4, 123, 11, 369, 13, 3, 185, 370, 20],\n",
              " [4, 123, 11, 369, 13, 3, 185, 370, 20, 23],\n",
              " [4, 371],\n",
              " [4, 371, 8],\n",
              " [4, 371, 8, 5],\n",
              " [4, 371, 8, 5, 372],\n",
              " [4, 371, 8, 5, 372, 1],\n",
              " [4, 371, 8, 5, 372, 1, 373],\n",
              " [4, 371, 8, 5, 372, 1, 373, 374],\n",
              " [4, 371, 8, 5, 372, 1, 373, 374, 55],\n",
              " [4, 371, 8, 5, 372, 1, 373, 374, 55, 1],\n",
              " [4, 371, 8, 5, 372, 1, 373, 374, 55, 1, 375],\n",
              " [4, 371, 8, 5, 372, 1, 373, 374, 55, 1, 375, 3],\n",
              " [4, 3],\n",
              " [4, 3, 12],\n",
              " [4, 3, 12, 23],\n",
              " [4, 3, 12, 23, 186],\n",
              " [4, 3, 12, 23, 186, 16],\n",
              " [4, 3, 12, 23, 186, 16, 3],\n",
              " [2, 187],\n",
              " [2, 187, 1],\n",
              " [2, 187, 1, 124],\n",
              " [2, 187, 1, 124, 188],\n",
              " [2, 187, 1, 124, 188, 376],\n",
              " [2, 187, 1, 124, 188, 376, 377],\n",
              " [2, 187, 1, 124, 188, 376, 377, 378],\n",
              " [2, 187, 1, 124, 188, 376, 377, 378, 125],\n",
              " [2, 187, 1, 124, 188, 376, 377, 378, 125, 6],\n",
              " [379, 126],\n",
              " [379, 126, 56],\n",
              " [379, 126, 56, 10],\n",
              " [379, 126, 56, 10, 23],\n",
              " [379, 126, 56, 10, 23, 16],\n",
              " [379, 126, 56, 10, 23, 16, 380],\n",
              " [379, 126, 56, 10, 23, 16, 380, 127],\n",
              " [379, 126, 56, 10, 23, 16, 380, 127, 1],\n",
              " [379, 126, 56, 10, 23, 16, 380, 127, 1, 3],\n",
              " [2, 23],\n",
              " [2, 23, 186],\n",
              " [2, 23, 186, 12],\n",
              " [2, 23, 186, 12, 13],\n",
              " [2, 23, 186, 12, 13, 72],\n",
              " [2, 23, 186, 12, 13, 72, 381],\n",
              " [2, 23, 186, 12, 13, 72, 381, 23],\n",
              " [2, 23, 186, 12, 13, 72, 381, 23, 187],\n",
              " [3, 33],\n",
              " [3, 33, 55],\n",
              " [3, 33, 55, 3],\n",
              " [3, 33, 55, 3, 382],\n",
              " [3, 33, 55, 3, 382, 3],\n",
              " [3, 33, 55, 3, 382, 3, 383],\n",
              " [3, 33, 55, 3, 382, 3, 383, 73],\n",
              " [4, 384],\n",
              " [4, 384, 385],\n",
              " [4, 384, 385, 8],\n",
              " [4, 384, 385, 8, 386],\n",
              " [4, 384, 385, 8, 386, 387],\n",
              " [4, 384, 385, 8, 386, 387, 388],\n",
              " [2, 389],\n",
              " [2, 389, 189],\n",
              " [2, 389, 189, 6],\n",
              " [2, 389, 189, 6, 190],\n",
              " [2, 389, 189, 6, 190, 191],\n",
              " [3, 12],\n",
              " [3, 12, 390],\n",
              " [3, 12, 390, 1],\n",
              " [3, 12, 390, 1, 88],\n",
              " [3, 12, 390, 1, 88, 391],\n",
              " [392, 46],\n",
              " [392, 46, 393],\n",
              " [392, 46, 393, 3],\n",
              " [392, 46, 393, 3, 55],\n",
              " [4, 394],\n",
              " [4, 394, 127],\n",
              " [4, 394, 127, 1],\n",
              " [4, 394, 127, 1, 3],\n",
              " [2, 128],\n",
              " [2, 128, 192],\n",
              " [2, 128, 192, 21],\n",
              " [2, 128, 192, 21, 395],\n",
              " [2, 128, 192, 21, 395, 396],\n",
              " [2, 128, 192, 21, 395, 396, 9],\n",
              " [2, 128, 192, 21, 395, 396, 9, 397],\n",
              " [2, 128, 192, 21, 395, 396, 9, 397, 129],\n",
              " [2, 128, 192, 21, 395, 396, 9, 397, 129, 24],\n",
              " [2, 128, 192, 21, 395, 396, 9, 397, 129, 24, 398],\n",
              " [2, 128, 192, 21, 395, 396, 9, 397, 129, 24, 398, 399],\n",
              " [2, 128, 192, 21, 395, 396, 9, 397, 129, 24, 398, 399, 1],\n",
              " [2, 128, 192, 21, 395, 396, 9, 397, 129, 24, 398, 399, 1, 3],\n",
              " [2, 400],\n",
              " [2, 400, 401],\n",
              " [2, 400, 401, 47],\n",
              " [2, 400, 401, 47, 63],\n",
              " [2, 400, 401, 47, 63, 402],\n",
              " [2, 400, 401, 47, 63, 402, 1],\n",
              " [2, 400, 401, 47, 63, 402, 1, 403],\n",
              " [2, 400, 401, 47, 63, 402, 1, 403, 1],\n",
              " [2, 400, 401, 47, 63, 402, 1, 403, 1, 193],\n",
              " [4, 194],\n",
              " [4, 194, 404],\n",
              " [4, 194, 404, 1],\n",
              " [4, 194, 404, 1, 3],\n",
              " [2, 3],\n",
              " [2, 3, 195],\n",
              " [2, 3, 195, 6],\n",
              " [2, 3, 195, 6, 196],\n",
              " [2, 3, 195, 6, 196, 3],\n",
              " [2, 57],\n",
              " [2, 57, 197],\n",
              " [2, 57, 197, 3],\n",
              " [2, 57, 197, 3, 198],\n",
              " [2, 57, 197, 3, 198, 3],\n",
              " [2, 57, 197, 3, 198, 3, 199],\n",
              " [2, 57, 197, 3, 198, 3, 199, 3],\n",
              " [2, 200],\n",
              " [2, 200, 3],\n",
              " [2, 200, 3, 201],\n",
              " [2, 200, 3, 201, 130],\n",
              " [2, 200, 3, 201, 130, 202],\n",
              " [2, 200, 3, 201, 130, 202, 6],\n",
              " [2, 200, 3, 201, 130, 202, 6, 131],\n",
              " [2, 200, 3, 201, 130, 202, 6, 131, 203],\n",
              " [2, 200, 3, 201, 130, 202, 6, 131, 203, 3],\n",
              " [2, 204],\n",
              " [2, 204, 72],\n",
              " [2, 204, 72, 6],\n",
              " [2, 204, 72, 6, 205],\n",
              " [2, 204, 72, 6, 205, 72],\n",
              " [2, 206],\n",
              " [2, 206, 207],\n",
              " [2, 206, 207, 405],\n",
              " [2, 206, 207, 405, 406],\n",
              " [2, 206, 207, 405, 406, 6],\n",
              " [2, 206, 207, 405, 406, 6, 87],\n",
              " [2, 206, 207, 405, 406, 6, 87, 3],\n",
              " [3, 12],\n",
              " [3, 12, 36],\n",
              " [3, 12, 36, 89],\n",
              " [3, 12, 36, 89, 132],\n",
              " [3, 12, 36, 89, 132, 1],\n",
              " [3, 12, 36, 89, 132, 1, 3],\n",
              " [4, 71],\n",
              " [4, 71, 407],\n",
              " [4, 71, 407, 3],\n",
              " [4, 71, 407, 3, 133],\n",
              " [4, 71, 407, 3, 133, 6],\n",
              " [4, 71, 407, 3, 133, 6, 134],\n",
              " [2, 408],\n",
              " [2, 408, 71],\n",
              " [2, 408, 71, 3],\n",
              " [2, 408, 71, 3, 409],\n",
              " [2, 408, 71, 3, 409, 410],\n",
              " [2, 408, 71, 3, 409, 410, 71],\n",
              " [4, 208],\n",
              " [4, 208, 3],\n",
              " [4, 208, 3, 133],\n",
              " [4, 208, 3, 133, 6],\n",
              " [4, 208, 3, 133, 6, 208],\n",
              " [4, 208, 3, 133, 6, 208, 134],\n",
              " [2, 3],\n",
              " [2, 3, 195],\n",
              " [2, 3, 195, 6],\n",
              " [2, 3, 195, 6, 196],\n",
              " [2, 3, 195, 6, 196, 3],\n",
              " [2, 57],\n",
              " [2, 57, 197],\n",
              " [2, 57, 197, 3],\n",
              " [2, 57, 197, 3, 198],\n",
              " [2, 57, 197, 3, 198, 3],\n",
              " [2, 57, 197, 3, 198, 3, 199],\n",
              " [2, 57, 197, 3, 198, 3, 199, 3],\n",
              " [2, 57, 197, 3, 198, 3, 199, 3, 411],\n",
              " [2, 57, 197, 3, 198, 3, 199, 3, 411, 209],\n",
              " [2, 57, 197, 3, 198, 3, 199, 3, 411, 209, 412],\n",
              " [2, 200],\n",
              " [2, 200, 3],\n",
              " [2, 200, 3, 201],\n",
              " [2, 200, 3, 201, 130],\n",
              " [2, 200, 3, 201, 130, 202],\n",
              " [2, 200, 3, 201, 130, 202, 6],\n",
              " [2, 200, 3, 201, 130, 202, 6, 131],\n",
              " [2, 200, 3, 201, 130, 202, 6, 131, 203],\n",
              " [2, 200, 3, 201, 130, 202, 6, 131, 203, 3],\n",
              " [2, 204],\n",
              " [2, 204, 72],\n",
              " [2, 204, 72, 6],\n",
              " [2, 204, 72, 6, 205],\n",
              " [2, 204, 72, 6, 205, 72],\n",
              " [2, 206],\n",
              " [2, 206, 3],\n",
              " [2, 206, 3, 6],\n",
              " [2, 206, 3, 6, 207],\n",
              " [2, 206, 3, 6, 207, 3],\n",
              " [4, 413],\n",
              " [4, 413, 414],\n",
              " [4, 413, 414, 415],\n",
              " [4, 413, 414, 415, 416],\n",
              " [4, 413, 414, 415, 416, 6],\n",
              " [4, 413, 414, 415, 416, 6, 417],\n",
              " [2, 418],\n",
              " [2, 418, 419],\n",
              " [2, 418, 419, 6],\n",
              " [2, 418, 419, 6, 420],\n",
              " [2, 418, 419, 6, 420, 3],\n",
              " [2, 418, 419, 6, 420, 3, 210],\n",
              " [2, 418, 419, 6, 420, 3, 210, 17],\n",
              " [2, 418, 419, 6, 420, 3, 210, 17, 29],\n",
              " [2, 418, 419, 6, 420, 3, 210, 17, 29, 421],\n",
              " [2, 418, 419, 6, 420, 3, 210, 17, 29, 421, 211],\n",
              " [2, 418, 419, 6, 420, 3, 210, 17, 29, 421, 211, 422],\n",
              " [4, 25],\n",
              " [4, 25, 56],\n",
              " [4, 25, 56, 42],\n",
              " [4, 25, 56, 42, 212],\n",
              " [4, 25, 56, 42, 212, 211],\n",
              " [4, 25, 56, 42, 212, 211, 213],\n",
              " [2, 214],\n",
              " [2, 214, 215],\n",
              " [2, 214, 215, 423],\n",
              " [2, 214, 215, 423, 64],\n",
              " [2, 214, 215, 423, 64, 216],\n",
              " [2, 214, 215, 423, 64, 216, 217],\n",
              " [2, 214, 215, 423, 64, 216, 217, 212],\n",
              " [2, 214, 215, 423, 64, 216, 217, 212, 10],\n",
              " [2, 214, 215, 423, 64, 216, 217, 212, 10, 213],\n",
              " [2, 424],\n",
              " [2, 424, 425],\n",
              " [2, 424, 425, 218],\n",
              " [2, 424, 425, 218, 15],\n",
              " [2, 424, 425, 218, 15, 219],\n",
              " [2, 424, 425, 218, 15, 219, 6],\n",
              " [2, 424, 425, 218, 15, 219, 6, 220],\n",
              " [2, 424, 425, 218, 15, 219, 6, 220, 221],\n",
              " [2, 424, 425, 218, 15, 219, 6, 220, 221, 10],\n",
              " [4, 17],\n",
              " [4, 17, 29],\n",
              " [4, 17, 29, 222],\n",
              " [4, 17, 29, 222, 223],\n",
              " [4, 17, 29, 222, 223, 65],\n",
              " [4, 17, 29, 222, 223, 65, 36],\n",
              " [4, 17, 29, 222, 223, 65, 36, 426],\n",
              " [4, 17, 29, 222, 223, 65, 36, 426, 10],\n",
              " [4, 17, 29, 222, 223, 65, 36, 426, 10, 222],\n",
              " [4, 17, 29, 222, 223, 65, 36, 426, 10, 222, 427],\n",
              " [4, 17, 29, 222, 223, 65, 36, 426, 10, 222, 427, 65],\n",
              " [4, 17, 29, 222, 223, 65, 36, 426, 10, 222, 427, 65, 36],\n",
              " [4, 17, 29, 222, 223, 65, 36, 426, 10, 222, 427, 65, 36, 428],\n",
              " [2, 224],\n",
              " [2, 224, 429],\n",
              " [2, 224, 429, 125],\n",
              " [2, 224, 429, 125, 10],\n",
              " [2, 224, 429, 125, 10, 135],\n",
              " [2, 224, 429, 125, 10, 135, 136],\n",
              " [2, 224, 429, 125, 10, 135, 136, 26],\n",
              " [2, 225],\n",
              " [2, 225, 430],\n",
              " [2, 225, 430, 8],\n",
              " [2, 225, 430, 8, 125],\n",
              " [2, 225, 430, 8, 125, 431],\n",
              " [2, 225, 430, 8, 125, 431, 3],\n",
              " [2, 225, 430, 8, 125, 431, 3, 9],\n",
              " [2, 225, 430, 8, 125, 431, 3, 9, 226],\n",
              " [2, 225, 430, 8, 125, 431, 3, 9, 226, 90],\n",
              " [2, 225, 430, 8, 125, 431, 3, 9, 226, 90, 221],\n",
              " [2, 225, 430, 8, 125, 431, 3, 9, 226, 90, 221, 17],\n",
              " [2, 225, 430, 8, 125, 431, 3, 9, 226, 90, 221, 17, 29],\n",
              " [2, 225, 430, 8, 125, 431, 3, 9, 226, 90, 221, 17, 29, 432],\n",
              " [2, 225, 430, 8, 125, 431, 3, 9, 226, 90, 221, 17, 29, 432, 433],\n",
              " [2, 225, 430, 8, 125, 431, 3, 9, 226, 90, 221, 17, 29, 432, 433, 9],\n",
              " [2, 225, 430, 8, 125, 431, 3, 9, 226, 90, 221, 17, 29, 432, 433, 9, 91],\n",
              " [2, 434],\n",
              " [2, 434, 435],\n",
              " [2, 434, 435, 225],\n",
              " [2, 434, 435, 225, 137],\n",
              " [2, 434, 435, 225, 137, 436],\n",
              " [2, 434, 435, 225, 137, 436, 437],\n",
              " [2, 434, 435, 225, 137, 436, 437, 137],\n",
              " [2, 227],\n",
              " [2, 227, 3],\n",
              " [2, 227, 3, 74],\n",
              " [2, 227, 3, 74, 15],\n",
              " [2, 227, 3, 74, 15, 138],\n",
              " [2, 227, 3, 74, 15, 138, 46],\n",
              " [2, 227, 3, 74, 15, 138, 46, 438],\n",
              " [2, 227, 3, 74, 15, 138, 46, 438, 30],\n",
              " [2, 227, 3, 74, 15, 138, 46, 438, 30, 5],\n",
              " [2, 227, 3, 74, 15, 138, 46, 438, 30, 5, 228],\n",
              " [2, 227, 3, 74, 15, 138, 46, 438, 30, 5, 228, 439],\n",
              " [2, 227, 3, 74, 15, 138, 46, 438, 30, 5, 228, 439, 1],\n",
              " [2, 227, 3, 74, 15, 138, 46, 438, 30, 5, 228, 439, 1, 5],\n",
              " [2, 227, 3, 74, 15, 138, 46, 438, 30, 5, 228, 439, 1, 5, 3],\n",
              " [2, 75],\n",
              " [2, 75, 10],\n",
              " [2, 75, 10, 440],\n",
              " [2, 75, 10, 440, 126],\n",
              " [2, 75, 10, 440, 126, 13],\n",
              " [2, 75, 10, 440, 126, 13, 441],\n",
              " [2, 75, 10, 440, 126, 13, 441, 121],\n",
              " [2, 75, 10, 440, 126, 13, 441, 121, 442],\n",
              " [2, 75, 10, 440, 126, 13, 441, 121, 442, 443],\n",
              " [2, 75, 10, 440, 126, 13, 441, 121, 442, 443, 55],\n",
              " [4, 229],\n",
              " [4, 229, 6],\n",
              " [4, 229, 6, 444],\n",
              " [4, 229, 6, 444, 55],\n",
              " [2, 229],\n",
              " [2, 229, 6],\n",
              " [2, 229, 6, 445],\n",
              " [2, 229, 6, 445, 17],\n",
              " [2, 229, 6, 445, 17, 29],\n",
              " [2, 229, 6, 445, 17, 29, 230],\n",
              " [2, 229, 6, 445, 17, 29, 230, 55],\n",
              " [2, 446],\n",
              " [2, 446, 33],\n",
              " [2, 446, 33, 12],\n",
              " [2, 446, 33, 12, 17],\n",
              " [2, 446, 33, 12, 17, 29],\n",
              " [2, 446, 33, 12, 17, 29, 184],\n",
              " [2, 446, 33, 12, 17, 29, 184, 447],\n",
              " [2, 446, 33, 12, 17, 29, 184, 447, 215],\n",
              " [2, 446, 33, 12, 17, 29, 184, 447, 215, 448],\n",
              " [2, 446, 33, 12, 17, 29, 184, 447, 215, 448, 449],\n",
              " [2, 446, 33, 12, 17, 29, 184, 447, 215, 448, 449, 231],\n",
              " [4, 139],\n",
              " [4, 139, 33],\n",
              " [4, 139, 33, 450],\n",
              " [4, 139, 33, 450, 10],\n",
              " [4, 139, 33, 450, 10, 451],\n",
              " [4, 139, 33, 450, 10, 451, 452],\n",
              " [122, 232],\n",
              " [122, 232, 13],\n",
              " [122, 232, 13, 3],\n",
              " [122, 232, 13, 3, 12],\n",
              " [2, 12],\n",
              " [2, 12, 76],\n",
              " [2, 12, 76, 132],\n",
              " [2, 12, 76, 132, 1],\n",
              " [2, 12, 76, 132, 1, 23],\n",
              " [2, 12, 76, 132, 1, 23, 16],\n",
              " [2, 12, 76, 132, 1, 23, 16, 453],\n",
              " [2, 12, 76, 132, 1, 23, 16, 453, 3],\n",
              " [2, 12, 76, 132, 1, 23, 16, 453, 3, 233],\n",
              " [2, 12, 76, 132, 1, 23, 16, 453, 3, 233, 17],\n",
              " [2, 12, 76, 132, 1, 23, 16, 453, 3, 233, 17, 29],\n",
              " [2, 12, 76, 132, 1, 23, 16, 453, 3, 233, 17, 29, 209],\n",
              " [2, 12, 76, 132, 1, 23, 16, 453, 3, 233, 17, 29, 209, 454],\n",
              " [2, 12, 76, 132, 1, 23, 16, 453, 3, 233, 17, 29, 209, 454, 87],\n",
              " [2, 455],\n",
              " [2, 455, 456],\n",
              " [2, 455, 456, 457],\n",
              " [2, 455, 456, 457, 6],\n",
              " [2, 455, 456, 457, 6, 458],\n",
              " [2, 33],\n",
              " [2, 33, 234],\n",
              " [2, 33, 234, 5],\n",
              " [2, 33, 234, 5, 140],\n",
              " [2, 33, 234, 5, 140, 459],\n",
              " [2, 460],\n",
              " [2, 460, 1],\n",
              " [2, 460, 1, 461],\n",
              " [2, 460, 1, 461, 23],\n",
              " [2, 462],\n",
              " [2, 462, 75],\n",
              " [2, 462, 75, 6],\n",
              " [2, 462, 75, 6, 463],\n",
              " [2, 462, 75, 6, 463, 3],\n",
              " [2, 464],\n",
              " [2, 464, 465],\n",
              " [2, 464, 465, 6],\n",
              " [2, 464, 465, 6, 466],\n",
              " [2, 464, 465, 6, 466, 12],\n",
              " [2, 464, 465, 6, 466, 12, 467],\n",
              " [2, 235],\n",
              " [2, 235, 1],\n",
              " [2, 235, 1, 5],\n",
              " [2, 235, 1, 5, 236],\n",
              " [2, 235, 1, 5, 236, 23],\n",
              " [2, 235, 1, 5, 236, 23, 30],\n",
              " [2, 235, 1, 5, 236, 23, 30, 468],\n",
              " [2, 235, 1, 5, 236, 23, 30, 468, 48],\n",
              " [2, 235, 1, 5, 236, 23, 30, 468, 48, 23],\n",
              " [2, 235, 1, 5, 236, 23, 30, 468, 48, 23, 469],\n",
              " [2, 3],\n",
              " [2, 3, 12],\n",
              " [2, 3, 12, 190],\n",
              " [2, 3, 12, 190, 470],\n",
              " [2, 3, 12, 190, 470, 6],\n",
              " [2, 3, 12, 190, 470, 6, 237],\n",
              " [2, 3, 12, 190, 470, 6, 237, 471],\n",
              " [2, 3, 12, 190, 470, 6, 237, 471, 12],\n",
              " [2, 238],\n",
              " [2, 238, 6],\n",
              " [2, 238, 6, 239],\n",
              " [2, 238, 6, 239, 1],\n",
              " [2, 238, 6, 239, 1, 3],\n",
              " [2, 238, 6, 239, 1, 3, 12],\n",
              " [2, 238, 6, 239, 1, 3, 12, 240],\n",
              " [2, 472],\n",
              " [2, 472, 12],\n",
              " [2, 472, 12, 1],\n",
              " [2, 472, 12, 1, 23],\n",
              " [2, 472, 12, 1, 23, 473],\n",
              " [2, 472, 12, 1, 23, 473, 88],\n",
              " [2, 472, 12, 1, 23, 473, 88, 474],\n",
              " [2, 472, 12, 1, 23, 473, 88, 474, 1],\n",
              " [2, 472, 12, 1, 23, 473, 88, 474, 1, 475],\n",
              " [4, 134],\n",
              " [4, 134, 6],\n",
              " [4, 134, 6, 130],\n",
              " [4, 134, 6, 130, 476],\n",
              " [2, 141],\n",
              " [2, 141, 477],\n",
              " [2, 141, 477, 3],\n",
              " [2, 141, 477, 3, 12],\n",
              " [2, 141, 477, 3, 12, 478],\n",
              " [2, 141, 477, 3, 12, 478, 3],\n",
              " [2, 141, 477, 3, 12, 478, 3, 12],\n",
              " [2, 479],\n",
              " [2, 479, 1],\n",
              " [2, 479, 1, 3],\n",
              " [2, 479, 1, 3, 480],\n",
              " [2, 479, 1, 3, 480, 481],\n",
              " [2, 479, 1, 3, 480, 481, 6],\n",
              " [2, 479, 1, 3, 480, 481, 6, 482],\n",
              " [483, 484],\n",
              " [483, 484, 3],\n",
              " [483, 484, 3, 12],\n",
              " [483, 484, 3, 12, 485],\n",
              " [3, 241],\n",
              " [3, 241, 235],\n",
              " [3, 241, 235, 6],\n",
              " [3, 241, 235, 6, 142],\n",
              " [4, 486],\n",
              " [4, 486, 5],\n",
              " [4, 486, 5, 242],\n",
              " [4, 486, 5, 242, 141],\n",
              " [2, 487],\n",
              " [2, 487, 488],\n",
              " [2, 487, 488, 23],\n",
              " [2, 487, 488, 23, 6],\n",
              " [2, 487, 488, 23, 6, 489],\n",
              " [2, 487, 488, 23, 6, 489, 1],\n",
              " [2, 487, 488, 23, 6, 489, 1, 242],\n",
              " [4, 490],\n",
              " [4, 490, 7],\n",
              " [4, 490, 7, 491],\n",
              " [4, 490, 7, 491, 3],\n",
              " [4, 490, 7, 491, 3, 18],\n",
              " [4, 490, 7, 491, 3, 18, 3],\n",
              " [4, 490, 7, 491, 3, 18, 3, 142],\n",
              " [4, 3],\n",
              " [4, 3, 241],\n",
              " [4, 3, 241, 6],\n",
              " [4, 3, 241, 6, 492],\n",
              " [4, 3, 241, 6, 492, 47],\n",
              " [4, 3, 241, 6, 492, 47, 493],\n",
              " [4, 3, 241, 6, 492, 47, 493, 494],\n",
              " [4, 3, 241, 6, 492, 47, 493, 494, 1],\n",
              " [4, 3, 241, 6, 492, 47, 493, 494, 1, 495],\n",
              " [4, 3],\n",
              " [4, 3, 243],\n",
              " [4, 3, 243, 6],\n",
              " [4, 3, 243, 6, 244],\n",
              " [2, 91],\n",
              " [2, 91, 126],\n",
              " [2, 91, 126, 245],\n",
              " [2, 91, 126, 245, 92],\n",
              " [2, 91, 126, 245, 92, 246],\n",
              " [2, 91, 126, 245, 92, 246, 243],\n",
              " [2, 91, 126, 245, 92, 246, 243, 496],\n",
              " [2, 91, 126, 245, 92, 246, 243, 496, 497],\n",
              " [4, 247],\n",
              " [4, 247, 218],\n",
              " [4, 247, 218, 1],\n",
              " [4, 247, 218, 1, 3],\n",
              " [4, 247, 218, 1, 3, 12],\n",
              " [2, 498],\n",
              " [2, 498, 499],\n",
              " [2, 498, 499, 230],\n",
              " [2, 498, 499, 230, 42],\n",
              " [2, 498, 499, 230, 42, 120],\n",
              " [4, 247],\n",
              " [4, 247, 5],\n",
              " [4, 247, 5, 12],\n",
              " [4, 247, 5, 12, 143],\n",
              " [4, 247, 5, 12, 143, 248],\n",
              " [4, 3],\n",
              " [4, 3, 12],\n",
              " [4, 3, 12, 189],\n",
              " [4, 3, 12, 189, 20],\n",
              " [4, 3, 12, 189, 20, 56],\n",
              " [4, 3, 12, 189, 20, 56, 1],\n",
              " [4, 3, 12, 189, 20, 56, 1, 93],\n",
              " [4, 33],\n",
              " [4, 33, 234],\n",
              " [4, 33, 234, 6],\n",
              " [4, 33, 234, 6, 23],\n",
              " [4, 33, 234, 6, 23, 500],\n",
              " [2, 239],\n",
              " [2, 239, 244],\n",
              " [2, 239, 244, 501],\n",
              " [2, 239, 244, 501, 144],\n",
              " [2, 239, 244, 501, 144, 56],\n",
              " [2, 239, 244, 501, 144, 56, 249],\n",
              " [4, 145],\n",
              " [4, 145, 1],\n",
              " [4, 145, 1, 236],\n",
              " [4, 145, 1, 236, 23],\n",
              " [116, 86],\n",
              " [116, 86, 6],\n",
              " [116, 86, 6, 117],\n",
              " [116, 86, 6, 117, 118],\n",
              " [116, 86, 6, 117, 118, 119],\n",
              " [4, 53],\n",
              " [4, 53, 1],\n",
              " [4, 53, 1, 3],\n",
              " [4, 53, 1, 3, 22],\n",
              " [4, 53, 1, 3, 22, 6],\n",
              " [4, 53, 1, 3, 22, 6, 146],\n",
              " [4, 27],\n",
              " [4, 27, 37],\n",
              " [4, 27, 37, 8],\n",
              " [4, 27, 37, 8, 7],\n",
              " [4, 27, 37, 8, 7, 66],\n",
              " [4, 27, 37, 8, 7, 66, 10],\n",
              " [2, 38],\n",
              " [2, 38, 250],\n",
              " [2, 38, 250, 251],\n",
              " [2, 38, 250, 251, 1],\n",
              " [2, 38, 250, 251, 1, 7],\n",
              " [2, 38, 250, 251, 1, 7, 252],\n",
              " [2, 37],\n",
              " [2, 37, 8],\n",
              " [2, 37, 8, 94],\n",
              " [2, 37, 8, 94, 147],\n",
              " [2, 37, 8, 94, 147, 24],\n",
              " [2, 37, 8, 94, 147, 24, 246],\n",
              " [4, 7],\n",
              " [4, 7, 53],\n",
              " [4, 7, 53, 1],\n",
              " [4, 7, 53, 1, 28],\n",
              " [4, 7, 53, 1, 28, 219],\n",
              " [2, 74],\n",
              " [2, 74, 8],\n",
              " [2, 74, 8, 94],\n",
              " [2, 74, 8, 94, 147],\n",
              " [2, 74, 8, 94, 147, 24],\n",
              " [2, 74, 8, 94, 147, 24, 148],\n",
              " [253, 254],\n",
              " [253, 254, 67],\n",
              " [253, 254, 67, 502],\n",
              " [253, 254, 67, 502, 10],\n",
              " [4, 149],\n",
              " [4, 149, 11],\n",
              " [4, 149, 11, 76],\n",
              " [4, 149, 11, 76, 233],\n",
              " [4, 149, 11, 76, 233, 1],\n",
              " [4, 149, 11, 76, 233, 1, 28],\n",
              " [4, 38],\n",
              " [4, 38, 503],\n",
              " [4, 38, 503, 255],\n",
              " [4, 38, 503, 255, 250],\n",
              " [4, 38, 503, 255, 250, 251],\n",
              " [4, 38, 503, 255, 250, 251, 256],\n",
              " [4, 38, 503, 255, 250, 251, 256, 257],\n",
              " [4, 38],\n",
              " [4, 38, 504],\n",
              " [4, 38, 504, 17],\n",
              " [4, 38, 504, 17, 29],\n",
              " [4, 38, 504, 17, 29, 505],\n",
              " [4, 38, 504, 17, 29, 505, 1],\n",
              " [4, 38, 504, 17, 29, 505, 1, 506],\n",
              " [4, 38, 504, 17, 29, 505, 1, 506, 507],\n",
              " [4, 38, 504, 17, 29, 505, 1, 506, 507, 36],\n",
              " [4, 38, 504, 17, 29, 505, 1, 506, 507, 36, 7],\n",
              " [4, 38, 504, 17, 29, 505, 1, 506, 507, 36, 7, 508],\n",
              " [4, 38, 504, 17, 29, 505, 1, 506, 507, 36, 7, 508, 16],\n",
              " [4, 38, 504, 17, 29, 505, 1, 506, 507, 36, 7, 508, 16, 62],\n",
              " [509, 510],\n",
              " [509, 510, 258],\n",
              " [509, 510, 258, 13],\n",
              " [509, 510, 258, 13, 511],\n",
              " [509, 510, 258, 13, 511, 512],\n",
              " [509, 510, 258, 13, 511, 512, 513],\n",
              " [4, 38],\n",
              " [4, 38, 514],\n",
              " [4, 38, 514, 515],\n",
              " [4, 38, 514, 515, 516],\n",
              " [4, 38, 514, 515, 516, 13],\n",
              " [4, 38, 514, 515, 516, 13, 517],\n",
              " [4, 38, 514, 515, 516, 13, 517, 10],\n",
              " [4, 38, 514, 515, 516, 13, 517, 10, 518],\n",
              " [4, 38],\n",
              " [4, 38, 259],\n",
              " [4, 38, 259, 13],\n",
              " [4, 38, 259, 13, 519],\n",
              " [4, 38, 259, 13, 519, 58],\n",
              " [4, 38, 259, 13, 519, 58, 57],\n",
              " [4, 38, 259, 13, 519, 58, 57, 260],\n",
              " [4, 5],\n",
              " [4, 5, 261],\n",
              " [4, 5, 261, 1],\n",
              " [4, 5, 261, 1, 27],\n",
              " [4, 5, 261, 1, 27, 37],\n",
              " [4, 5, 261, 1, 27, 37, 520],\n",
              " [4, 5, 261, 1, 27, 37, 520, 36],\n",
              " [4, 5, 261, 1, 27, 37, 520, 36, 150],\n",
              " [4, 5, 261, 1, 27, 37, 520, 36, 150, 1],\n",
              " [4, 5, 261, 1, 27, 37, 520, 36, 150, 1, 5],\n",
              " [4, 5, 261, 1, 27, 37, 520, 36, 150, 1, 5, 262],\n",
              " [151, 5],\n",
              " [151, 5, 26],\n",
              " [151, 5, 26, 1],\n",
              " [151, 5, 26, 1, 7],\n",
              " [151, 5, 26, 1, 7, 151],\n",
              " [151, 5, 26, 1, 7, 151, 37],\n",
              " [151, 5, 26, 1, 7, 151, 37, 11],\n",
              " [151, 5, 26, 1, 7, 151, 37, 11, 521],\n",
              " [76, 522],\n",
              " [76, 522, 95],\n",
              " [76, 522, 95, 17],\n",
              " [76, 522, 95, 17, 151],\n",
              " [76, 522, 95, 17, 151, 28],\n",
              " [263, 77],\n",
              " [263, 77, 264],\n",
              " [263, 77, 264, 73],\n",
              " [263, 77, 264, 73, 9],\n",
              " [220, 48],\n",
              " [220, 48, 74],\n",
              " [220, 48, 74, 16],\n",
              " [220, 48, 74, 16, 265],\n",
              " [220, 48, 74, 16, 265, 523],\n",
              " [266, 5],\n",
              " [266, 5, 26],\n",
              " [266, 5, 26, 1],\n",
              " [266, 5, 26, 1, 27],\n",
              " [266, 5, 26, 1, 27, 266],\n",
              " [266, 5, 26, 1, 27, 266, 37],\n",
              " [263, 264],\n",
              " [263, 264, 73],\n",
              " [263, 264, 73, 9],\n",
              " [263, 264, 73, 9, 524],\n",
              " [267, 20],\n",
              " [267, 20, 267],\n",
              " [267, 20, 267, 28],\n",
              " [267, 20, 267, 28, 5],\n",
              " [267, 20, 267, 28, 5, 268],\n",
              " [269, 26],\n",
              " [269, 26, 11],\n",
              " [269, 26, 11, 270],\n",
              " [269, 26, 11, 270, 95],\n",
              " [269, 26, 11, 270, 95, 17],\n",
              " [269, 26, 11, 270, 95, 17, 7],\n",
              " [269, 26, 11, 270, 95, 17, 7, 525],\n",
              " [271, 20],\n",
              " [271, 20, 271],\n",
              " [271, 20, 271, 152],\n",
              " [271, 20, 271, 152, 96],\n",
              " [271, 20, 271, 152, 96, 268],\n",
              " [6, 526],\n",
              " [6, 526, 11],\n",
              " [6, 526, 11, 270],\n",
              " [2, 97],\n",
              " [2, 97, 77],\n",
              " [2, 97, 77, 7],\n",
              " [2, 97, 77, 7, 527],\n",
              " [2, 97, 77, 7, 527, 10],\n",
              " [2, 97, 77, 7, 527, 10, 528],\n",
              " [2, 97, 77, 7, 527, 10, 528, 529],\n",
              " [2, 97, 77, 7, 527, 10, 528, 529, 18],\n",
              " [2, 97, 77, 7, 527, 10, 528, 529, 18, 1],\n",
              " [2, 97, 77, 7, 527, 10, 528, 529, 18, 1, 26],\n",
              " [2, 38],\n",
              " [2, 38, 256],\n",
              " [2, 38, 256, 257],\n",
              " [2, 38, 256, 257, 260],\n",
              " [2, 38, 256, 257, 260, 10],\n",
              " [2, 38, 256, 257, 260, 10, 5],\n",
              " [2, 38, 256, 257, 260, 10, 5, 18],\n",
              " [2, 38, 256, 257, 260, 10, 5, 18, 1],\n",
              " [2, 38, 256, 257, 260, 10, 5, 18, 1, 530],\n",
              " [2, 38, 256, 257, 260, 10, 5, 18, 1, 530, 13],\n",
              " [2, 38, 256, 257, 260, 10, 5, 18, 1, 530, 13, 7],\n",
              " [2, 38, 256, 257, 260, 10, 5, 18, 1, 530, 13, 7, 53],\n",
              " [2, 38, 256, 257, 260, 10, 5, 18, 1, 530, 13, 7, 53, 1],\n",
              " [2, 78],\n",
              " [2, 78, 98],\n",
              " [2, 78, 98, 24],\n",
              " [2, 78, 98, 24, 531],\n",
              " [2, 78, 98, 24, 531, 152],\n",
              " [2, 532],\n",
              " [2, 532, 272],\n",
              " [2, 532, 272, 28],\n",
              " [2, 532, 272, 28, 11],\n",
              " [2, 532, 272, 28, 11, 7],\n",
              " [2, 532, 272, 28, 11, 7, 273],\n",
              " [2, 532, 272, 28, 11, 7, 273, 254],\n",
              " [2, 532, 272, 28, 11, 7, 273, 254, 1],\n",
              " [2, 532, 272, 28, 11, 7, 273, 254, 1, 533],\n",
              " [2, 532, 272, 28, 11, 7, 273, 254, 1, 533, 28],\n",
              " [2, 97],\n",
              " [2, 97, 274],\n",
              " [2, 97, 274, 255],\n",
              " [2, 97, 274, 255, 24],\n",
              " [2, 97, 274, 255, 24, 37],\n",
              " [2, 97, 274, 255, 24, 37, 26],\n",
              " [2, 38],\n",
              " [2, 38, 259],\n",
              " [2, 38, 259, 258],\n",
              " [2, 38, 259, 258, 10],\n",
              " [2, 38, 259, 258, 10, 275],\n",
              " [2, 534],\n",
              " [2, 534, 274],\n",
              " [2, 534, 274, 26],\n",
              " [2, 534, 274, 26, 39],\n",
              " [2, 534, 274, 26, 39, 77],\n",
              " [2, 534, 274, 26, 39, 77, 21],\n",
              " [2, 534, 274, 26, 39, 77, 21, 535],\n",
              " [2, 534, 274, 26, 39, 77, 21, 535, 6],\n",
              " [2, 534, 274, 26, 39, 77, 21, 535, 6, 98],\n",
              " [2, 534, 274, 26, 39, 77, 21, 535, 6, 98, 153],\n",
              " [2, 534, 274, 26, 39, 77, 21, 535, 6, 98, 153, 7],\n",
              " [2, 536],\n",
              " [2, 536, 28],\n",
              " [2, 536, 28, 11],\n",
              " [2, 536, 28, 11, 537],\n",
              " [2, 536, 28, 11, 537, 98],\n",
              " [2, 536, 28, 11, 537, 98, 24],\n",
              " [2, 536, 28, 11, 537, 98, 24, 538],\n",
              " [2, 536, 28, 11, 537, 98, 24, 538, 253],\n",
              " [2, 536, 28, 11, 537, 98, 24, 538, 253, 152],\n",
              " [4, 3],\n",
              " [4, 3, 15],\n",
              " [4, 3, 15, 276],\n",
              " [4, 3, 15, 276, 1],\n",
              " [4, 3, 15, 276, 1, 7],\n",
              " [4, 3, 15, 276, 1, 7, 53],\n",
              " [4, 3, 15, 276, 1, 7, 53, 1],\n",
              " [4, 3, 15, 276, 1, 7, 53, 1, 539],\n",
              " [4, 3, 15, 276, 1, 7, 53, 1, 539, 14],\n",
              " [4, 3, 15, 276, 1, 7, 53, 1, 539, 14, 1],\n",
              " [4, 3, 15, 276, 1, 7, 53, 1, 539, 14, 1, 150],\n",
              " [4, 3, 15, 276, 1, 7, 53, 1, 539, 14, 1, 150, 276],\n",
              " [4, 3, 15, 276, 1, 7, 53, 1, 539, 14, 1, 150, 276, 1],\n",
              " [4, 3, 15, 276, 1, 7, 53, 1, 539, 14, 1, 150, 276, 1, 7],\n",
              " [277, 18],\n",
              " [277, 18, 1],\n",
              " [277, 18, 1, 28],\n",
              " [277, 18, 1, 28, 540],\n",
              " [277, 18, 1, 28, 540, 541],\n",
              " [277, 18, 1, 28, 540, 541, 542],\n",
              " [4, 40],\n",
              " [4, 40, 3],\n",
              " [4, 40, 3, 22],\n",
              " [4, 40, 3, 22, 63],\n",
              " [4, 40, 3, 22, 63, 5],\n",
              " [4, 40, 3, 22, 63, 5, 59],\n",
              " [4, 40, 3, 22, 63, 5, 59, 277],\n",
              " [4, 40, 3, 22, 63, 5, 59, 277, 18],\n",
              " [4, 40, 3, 22, 63, 5, 59, 277, 18, 1],\n",
              " [4, 40, 3, 22, 63, 5, 59, 277, 18, 1, 543],\n",
              " [4, 40, 3, 22, 63, 5, 59, 277, 18, 1, 543, 28],\n",
              " [4, 40, 3, 22, 63, 5, 59, 277, 18, 1, 543, 28, 154],\n",
              " [5, 3],\n",
              " [5, 3, 22],\n",
              " [5, 3, 22, 39],\n",
              " [5, 3, 22, 39, 21],\n",
              " [5, 3, 22, 39, 21, 544],\n",
              " [5, 3, 22, 39, 21, 544, 1],\n",
              " [5, 3, 22, 39, 21, 544, 1, 24],\n",
              " [5, 3, 22, 39, 21, 544, 1, 24, 278],\n",
              " [5, 3, 22, 39, 21, 544, 1, 24, 278, 13],\n",
              " [5, 3, 22, 39, 21, 544, 1, 24, 278, 13, 7],\n",
              " [5, 3, 22, 39, 21, 544, 1, 24, 278, 13, 7, 131],\n",
              " [5, 3, 22, 39, 21, 544, 1, 24, 278, 13, 7, 131, 545],\n",
              " [155, 68],\n",
              " [155, 68, 14],\n",
              " [155, 68, 14, 546],\n",
              " [155, 68, 14, 546, 547],\n",
              " [155, 68, 14, 546, 547, 7],\n",
              " [155, 68, 14, 546, 547, 7, 548],\n",
              " [155, 68, 14, 546, 547, 7, 548, 37],\n",
              " [4, 129],\n",
              " [4, 129, 3],\n",
              " [4, 129, 3, 18],\n",
              " [4, 129, 3, 18, 39],\n",
              " [4, 129, 3, 18, 39, 21],\n",
              " [4, 129, 3, 18, 39, 21, 98],\n",
              " [4, 129, 3, 18, 39, 21, 98, 43],\n",
              " [4, 129, 3, 18, 39, 21, 98, 43, 27],\n",
              " [4, 129, 3, 18, 39, 21, 98, 43, 27, 99],\n",
              " [4, 129, 3, 18, 39, 21, 98, 43, 27, 99, 43],\n",
              " [4, 129, 3, 18, 39, 21, 98, 43, 27, 99, 43, 79],\n",
              " [4, 129, 3, 18, 39, 21, 98, 43, 27, 99, 43, 79, 549],\n",
              " [4, 129, 3, 18, 39, 21, 98, 43, 27, 99, 43, 79, 549, 68],\n",
              " [4, 129, 3, 18, 39, 21, 98, 43, 27, 99, 43, 79, 549, 68, 149],\n",
              " [11, 99],\n",
              " [11, 99, 550],\n",
              " [11, 99, 550, 48],\n",
              " [11, 99, 550, 48, 20],\n",
              " [11, 99, 550, 48, 20, 14],\n",
              " [11, 99, 550, 48, 20, 14, 74],\n",
              " [11, 99, 550, 48, 20, 14, 74, 6],\n",
              " [11, 99, 550, 48, 20, 14, 74, 6, 79],\n",
              " [11, 99, 550, 48, 20, 14, 74, 6, 79, 551],\n",
              " [11, 99, 550, 48, 20, 14, 74, 6, 79, 551, 48],\n",
              " [11, 99, 550, 48, 20, 14, 74, 6, 79, 551, 48, 20],\n",
              " [11, 99, 550, 48, 20, 14, 74, 6, 79, 551, 48, 20, 14],\n",
              " [4, 14],\n",
              " [4, 14, 552],\n",
              " [4, 14, 552, 279],\n",
              " [4, 14, 552, 279, 7],\n",
              " [4, 14, 552, 279, 7, 553],\n",
              " [4, 14, 552, 279, 7, 553, 280],\n",
              " [14, 281],\n",
              " [14, 281, 8],\n",
              " [14, 281, 8, 7],\n",
              " [14, 281, 8, 7, 282],\n",
              " [14, 281, 8, 7, 282, 37],\n",
              " [14, 281, 8, 7, 282, 37, 1],\n",
              " [14, 281, 8, 7, 282, 37, 1, 5],\n",
              " [14, 281, 8, 7, 282, 37, 1, 5, 280],\n",
              " [2, 5],\n",
              " [2, 5, 156],\n",
              " [2, 5, 156, 1],\n",
              " [2, 5, 156, 1, 14],\n",
              " [2, 5, 156, 1, 14, 282],\n",
              " [2, 5, 156, 1, 14, 282, 8],\n",
              " [2, 5, 156, 1, 14, 282, 8, 5],\n",
              " [2, 5, 156, 1, 14, 282, 8, 5, 31],\n",
              " [2, 5, 156, 1, 14, 282, 8, 5, 31, 1],\n",
              " [2, 5, 156, 1, 14, 282, 8, 5, 31, 1, 554],\n",
              " [2, 5, 156, 1, 14, 282, 8, 5, 31, 1, 554, 5],\n",
              " [2, 5, 156, 1, 14, 282, 8, 5, 31, 1, 554, 5, 555],\n",
              " [2, 5, 156, 1, 14, 282, 8, 5, 31, 1, 554, 5, 555, 281],\n",
              " [4, 7],\n",
              " [4, 7, 273],\n",
              " [4, 7, 273, 261],\n",
              " [4, 7, 273, 261, 1],\n",
              " [4, 7, 273, 261, 1, 148],\n",
              " [4, 7, 273, 261, 1, 148, 3],\n",
              " [4, 7, 273, 261, 1, 148, 3, 68],\n",
              " [2, 14],\n",
              " [2, 14, 148],\n",
              " [2, 14, 148, 80],\n",
              " [2, 14, 148, 80, 556],\n",
              " [2, 14, 148, 80, 556, 7],\n",
              " [2, 14, 148, 80, 556, 7, 18],\n",
              " [2, 14, 148, 80, 556, 7, 18, 1],\n",
              " [2, 14, 148, 80, 556, 7, 18, 1, 44],\n",
              " [2, 20],\n",
              " [2, 20, 283],\n",
              " [2, 20, 283, 557],\n",
              " [2, 20, 283, 557, 7],\n",
              " [2, 20, 283, 557, 7, 558],\n",
              " [2, 20, 283, 557, 7, 558, 157],\n",
              " [2, 20, 283, 557, 7, 558, 157, 5],\n",
              " [2, 20, 283, 557, 7, 558, 157, 5, 18],\n",
              " [2, 20, 283, 557, 7, 558, 157, 5, 18, 1],\n",
              " [2, 20, 283, 557, 7, 558, 157, 5, 18, 1, 284],\n",
              " [2, 20, 283, 557, 7, 558, 157, 5, 18, 1, 284, 285],\n",
              " [43, 7],\n",
              " [43, 7, 559],\n",
              " [43, 7, 559, 286],\n",
              " [43, 7, 559, 286, 48],\n",
              " [43, 7, 559, 286, 48, 560],\n",
              " [43, 7, 559, 286, 48, 560, 561],\n",
              " [43, 7, 559, 286, 48, 560, 561, 562],\n",
              " [43, 7, 559, 286, 48, 560, 561, 562, 7],\n",
              " [43, 7, 559, 286, 48, 560, 561, 562, 7, 80],\n",
              " [43, 7, 559, 286, 48, 560, 561, 562, 7, 80, 563],\n",
              " [5, 564],\n",
              " [5, 564, 284],\n",
              " [5, 564, 284, 15],\n",
              " [5, 564, 284, 15, 565],\n",
              " [5, 564, 284, 15, 565, 285],\n",
              " [5, 564, 284, 15, 565, 285, 11],\n",
              " [5, 564, 284, 15, 565, 285, 11, 5],\n",
              " [5, 564, 284, 15, 565, 285, 11, 5, 44],\n",
              " [4, 38],\n",
              " [4, 38, 566],\n",
              " [4, 38, 566, 567],\n",
              " [4, 38, 566, 567, 6],\n",
              " [4, 38, 566, 567, 6, 568],\n",
              " [4, 38, 566, 567, 6, 568, 569],\n",
              " [4, 89],\n",
              " [4, 89, 132],\n",
              " [4, 89, 132, 1],\n",
              " [4, 89, 132, 1, 3],\n",
              " [4, 89, 132, 1, 3, 287],\n",
              " [4, 89, 132, 1, 3, 287, 100],\n",
              " [4, 81],\n",
              " [4, 81, 39],\n",
              " [4, 81, 39, 123],\n",
              " [4, 81, 39, 123, 570],\n",
              " [4, 81, 39, 123, 570, 100],\n",
              " [4, 81, 39, 123, 570, 100, 30],\n",
              " [4, 81, 39, 123, 570, 100, 30, 5],\n",
              " [4, 81, 39, 123, 570, 100, 30, 5, 3],\n",
              " [4, 89],\n",
              " [4, 89, 39],\n",
              " [4, 89, 39, 123],\n",
              " [4, 89, 39, 123, 571],\n",
              " [4, 89, 39, 123, 571, 288],\n",
              " [4, 89, 39, 123, 571, 288, 572],\n",
              " [4, 89, 39, 123, 571, 288, 572, 100],\n",
              " [4, 38],\n",
              " [4, 38, 1],\n",
              " [4, 38, 1, 3],\n",
              " [4, 38, 1, 3, 287],\n",
              " [4, 38, 1, 3, 287, 100],\n",
              " [4, 75],\n",
              " [4, 75, 289],\n",
              " [4, 75, 289, 9],\n",
              " [4, 75, 289, 9, 573],\n",
              " [4, 75, 289, 9, 573, 1],\n",
              " [4, 75, 289, 9, 573, 1, 158],\n",
              " [4, 75, 289, 9, 573, 1, 158, 26],\n",
              " [2, 38],\n",
              " [2, 38, 574],\n",
              " [2, 38, 574, 1],\n",
              " [2, 38, 574, 1, 7],\n",
              " [2, 38, 574, 1, 7, 575],\n",
              " [2, 38, 574, 1, 7, 575, 576],\n",
              " [2, 38, 574, 1, 7, 575, 576, 82],\n",
              " [2, 38, 574, 1, 7, 575, 576, 82, 577],\n",
              " ...]"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "max_len = max([len(x) for x in input_sequences])"
      ],
      "metadata": {
        "id": "iNZgLbe4T9v2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "padded_input_sequences = pad_sequences(input_sequences, maxlen = max_len, padding='pre')"
      ],
      "metadata": {
        "id": "IfFae9ebULbN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "padded_input_sequences"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AbGnNNK4UlTc",
        "outputId": "9f7c72d5-b0a2-426e-f4c6-3edb85f45e11"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[  0,   0,   0, ...,   0, 344,  62],\n",
              "       [  0,   0,   0, ..., 344,  62, 345],\n",
              "       [  0,   0,   0, ...,  62, 345,  42],\n",
              "       ...,\n",
              "       [  0,   0,   0, ...,  33, 343,   9],\n",
              "       [  0,   0,   0, ..., 343,   9,  21],\n",
              "       [  0,   0,   0, ...,   9,  21, 175]], dtype=int32)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "padded_input_sequences.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "32CrlMUJUpBO",
        "outputId": "1a4724dc-1ec5-469a-f081-c2081433ae01"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2220, 17)"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = padded_input_sequences[:,:-1]"
      ],
      "metadata": {
        "id": "n9xwKuPMVqY6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y = padded_input_sequences[:,-1]"
      ],
      "metadata": {
        "id": "fOe0bC7WqVcE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ugYsYnfIqZ-k",
        "outputId": "d716d540-c2a4-4b58-b436-0c5bbf1f4d9f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2220,)"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        " x.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N9p1cYteqlcj",
        "outputId": "f96a5bb6-66d6-4644-dbb2-186359c64330"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2220, 16)"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.keras.utils import to_categorical\n",
        "y = to_categorical(y ,num_classes= 783)"
      ],
      "metadata": {
        "id": "PWslecLfrv_7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        " from tensorflow.keras.models import Sequential\n",
        " from tensorflow.keras.layers import Embedding, LSTM, Dense"
      ],
      "metadata": {
        "id": "cknZ_buNskf8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model = Sequential()\n",
        "model.add(Embedding(783, 100, input_length=16))\n",
        "model.add(LSTM(150))\n",
        "model.add(Dense(783,activation= 'softmax'))\n"
      ],
      "metadata": {
        "id": "-rVGmeCTtgB7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model.compile(loss = 'categorical_crossentropy', optimizer= 'adam', metrics= ['accuracy'])"
      ],
      "metadata": {
        "id": "wdqsdOnWxujo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4ixWkl38yMDS",
        "outputId": "72e2b343-34b0-4699-a8d5-f64caa26377c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding_2 (Embedding)     (None, 16, 100)           78300     \n",
            "                                                                 \n",
            " lstm_2 (LSTM)               (None, 150)               150600    \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 783)               118233    \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 347133 (1.32 MB)\n",
            "Trainable params: 347133 (1.32 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model.fit(x,y, epochs=100)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NIGoC6OcyO46",
        "outputId": "6740f69f-aa91-4fd4-bb70-67a1718033bc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/100\n",
            "70/70 [==============================] - 5s 33ms/step - loss: 6.2857 - accuracy: 0.0455\n",
            "Epoch 2/100\n",
            "70/70 [==============================] - 2s 34ms/step - loss: 5.8724 - accuracy: 0.0563\n",
            "Epoch 3/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 5.7884 - accuracy: 0.0563\n",
            "Epoch 4/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 5.6781 - accuracy: 0.0577\n",
            "Epoch 5/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 5.5811 - accuracy: 0.0703\n",
            "Epoch 6/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 5.4765 - accuracy: 0.0770\n",
            "Epoch 7/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 5.3395 - accuracy: 0.0811\n",
            "Epoch 8/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 5.1541 - accuracy: 0.0883\n",
            "Epoch 9/100\n",
            "70/70 [==============================] - 4s 51ms/step - loss: 4.9339 - accuracy: 0.1216\n",
            "Epoch 10/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 4.7027 - accuracy: 0.1419\n",
            "Epoch 11/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 4.4636 - accuracy: 0.1635\n",
            "Epoch 12/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 4.2349 - accuracy: 0.1860\n",
            "Epoch 13/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 4.0165 - accuracy: 0.2131\n",
            "Epoch 14/100\n",
            "70/70 [==============================] - 4s 53ms/step - loss: 3.8038 - accuracy: 0.2419\n",
            "Epoch 15/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 3.6066 - accuracy: 0.2685\n",
            "Epoch 16/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 3.4162 - accuracy: 0.2932\n",
            "Epoch 17/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 3.2225 - accuracy: 0.3410\n",
            "Epoch 18/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 3.0489 - accuracy: 0.3739\n",
            "Epoch 19/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 2.8800 - accuracy: 0.4131\n",
            "Epoch 20/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 2.7148 - accuracy: 0.4464\n",
            "Epoch 21/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 2.5493 - accuracy: 0.4838\n",
            "Epoch 22/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 2.4048 - accuracy: 0.5279\n",
            "Epoch 23/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 2.2641 - accuracy: 0.5599\n",
            "Epoch 24/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 2.1251 - accuracy: 0.6018\n",
            "Epoch 25/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 1.9985 - accuracy: 0.6185\n",
            "Epoch 26/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 1.8783 - accuracy: 0.6532\n",
            "Epoch 27/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 1.7704 - accuracy: 0.6833\n",
            "Epoch 28/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 1.6658 - accuracy: 0.6968\n",
            "Epoch 29/100\n",
            "70/70 [==============================] - 4s 53ms/step - loss: 1.5695 - accuracy: 0.7194\n",
            "Epoch 30/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 1.4842 - accuracy: 0.7338\n",
            "Epoch 31/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 1.4036 - accuracy: 0.7586\n",
            "Epoch 32/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 1.3256 - accuracy: 0.7721\n",
            "Epoch 33/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 1.2611 - accuracy: 0.7833\n",
            "Epoch 34/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 1.1975 - accuracy: 0.7955\n",
            "Epoch 35/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 1.1400 - accuracy: 0.8099\n",
            "Epoch 36/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 1.0864 - accuracy: 0.8189\n",
            "Epoch 37/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 1.0390 - accuracy: 0.8221\n",
            "Epoch 38/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.9930 - accuracy: 0.8311\n",
            "Epoch 39/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 0.9553 - accuracy: 0.8320\n",
            "Epoch 40/100\n",
            "70/70 [==============================] - 2s 34ms/step - loss: 0.9161 - accuracy: 0.8428\n",
            "Epoch 41/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.8820 - accuracy: 0.8500\n",
            "Epoch 42/100\n",
            "70/70 [==============================] - 2s 34ms/step - loss: 0.8513 - accuracy: 0.8536\n",
            "Epoch 43/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.8223 - accuracy: 0.8541\n",
            "Epoch 44/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 0.7979 - accuracy: 0.8563\n",
            "Epoch 45/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.7740 - accuracy: 0.8608\n",
            "Epoch 46/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.7533 - accuracy: 0.8622\n",
            "Epoch 47/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.7320 - accuracy: 0.8626\n",
            "Epoch 48/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 0.7156 - accuracy: 0.8608\n",
            "Epoch 49/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 0.6980 - accuracy: 0.8649\n",
            "Epoch 50/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 0.6838 - accuracy: 0.8635\n",
            "Epoch 51/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.6697 - accuracy: 0.8631\n",
            "Epoch 52/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.6572 - accuracy: 0.8658\n",
            "Epoch 53/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 0.6471 - accuracy: 0.8649\n",
            "Epoch 54/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 0.6357 - accuracy: 0.8640\n",
            "Epoch 55/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.6268 - accuracy: 0.8631\n",
            "Epoch 56/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 0.6170 - accuracy: 0.8635\n",
            "Epoch 57/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 0.6096 - accuracy: 0.8653\n",
            "Epoch 58/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.6017 - accuracy: 0.8640\n",
            "Epoch 59/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 0.5946 - accuracy: 0.8658\n",
            "Epoch 60/100\n",
            "70/70 [==============================] - 2s 34ms/step - loss: 0.5884 - accuracy: 0.8649\n",
            "Epoch 61/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5829 - accuracy: 0.8640\n",
            "Epoch 62/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5769 - accuracy: 0.8631\n",
            "Epoch 63/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5722 - accuracy: 0.8671\n",
            "Epoch 64/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 0.5676 - accuracy: 0.8676\n",
            "Epoch 65/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 0.5647 - accuracy: 0.8658\n",
            "Epoch 66/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 0.5592 - accuracy: 0.8658\n",
            "Epoch 67/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5569 - accuracy: 0.8631\n",
            "Epoch 68/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 0.5547 - accuracy: 0.8667\n",
            "Epoch 69/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 0.5494 - accuracy: 0.8635\n",
            "Epoch 70/100\n",
            "70/70 [==============================] - 2s 34ms/step - loss: 0.5479 - accuracy: 0.8635\n",
            "Epoch 71/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5445 - accuracy: 0.8653\n",
            "Epoch 72/100\n",
            "70/70 [==============================] - 2s 34ms/step - loss: 0.5426 - accuracy: 0.8635\n",
            "Epoch 73/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5396 - accuracy: 0.8676\n",
            "Epoch 74/100\n",
            "70/70 [==============================] - 4s 52ms/step - loss: 0.5373 - accuracy: 0.8667\n",
            "Epoch 75/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5352 - accuracy: 0.8658\n",
            "Epoch 76/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5323 - accuracy: 0.8658\n",
            "Epoch 77/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5323 - accuracy: 0.8653\n",
            "Epoch 78/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5303 - accuracy: 0.8644\n",
            "Epoch 79/100\n",
            "70/70 [==============================] - 4s 51ms/step - loss: 0.5273 - accuracy: 0.8658\n",
            "Epoch 80/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5268 - accuracy: 0.8671\n",
            "Epoch 81/100\n",
            "70/70 [==============================] - 2s 34ms/step - loss: 0.5249 - accuracy: 0.8622\n",
            "Epoch 82/100\n",
            "70/70 [==============================] - 2s 34ms/step - loss: 0.5235 - accuracy: 0.8658\n",
            "Epoch 83/100\n",
            "70/70 [==============================] - 3s 37ms/step - loss: 0.5226 - accuracy: 0.8662\n",
            "Epoch 84/100\n",
            "70/70 [==============================] - 3s 49ms/step - loss: 0.5206 - accuracy: 0.8671\n",
            "Epoch 85/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5203 - accuracy: 0.8640\n",
            "Epoch 86/100\n",
            "70/70 [==============================] - 2s 34ms/step - loss: 0.5186 - accuracy: 0.8658\n",
            "Epoch 87/100\n",
            "70/70 [==============================] - 2s 32ms/step - loss: 0.5168 - accuracy: 0.8689\n",
            "Epoch 88/100\n",
            "70/70 [==============================] - 3s 39ms/step - loss: 0.5163 - accuracy: 0.8676\n",
            "Epoch 89/100\n",
            "70/70 [==============================] - 3s 45ms/step - loss: 0.5156 - accuracy: 0.8617\n",
            "Epoch 90/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5143 - accuracy: 0.8667\n",
            "Epoch 91/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5140 - accuracy: 0.8644\n",
            "Epoch 92/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5129 - accuracy: 0.8658\n",
            "Epoch 93/100\n",
            "70/70 [==============================] - 3s 41ms/step - loss: 0.5112 - accuracy: 0.8658\n",
            "Epoch 94/100\n",
            "70/70 [==============================] - 3s 44ms/step - loss: 0.5105 - accuracy: 0.8694\n",
            "Epoch 95/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5086 - accuracy: 0.8671\n",
            "Epoch 96/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5100 - accuracy: 0.8667\n",
            "Epoch 97/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5083 - accuracy: 0.8644\n",
            "Epoch 98/100\n",
            "70/70 [==============================] - 3s 43ms/step - loss: 0.5075 - accuracy: 0.8676\n",
            "Epoch 99/100\n",
            "70/70 [==============================] - 3s 41ms/step - loss: 0.5075 - accuracy: 0.8622\n",
            "Epoch 100/100\n",
            "70/70 [==============================] - 2s 33ms/step - loss: 0.5057 - accuracy: 0.8658\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.src.callbacks.History at 0x791b7c612b30>"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        " import time\n",
        "text = \"Such data set can be\"\n",
        "\n",
        "for i in range (10):\n",
        "  token_text = tokenizer.texts_to_sequences([text])[0]\n",
        "  padded_token_text = pad_sequences([token_text], maxlen = 16 , padding = 'pre')\n",
        "  pos =np.argmax(model.predict(padded_token_text))\n",
        "\n",
        "  for word,index in tokenizer.word_index.items():\n",
        "    if index ==pos:\n",
        "      text = text+ \" \" +word\n",
        "      print(text)\n",
        "      time.sleep(1.5)"
      ],
      "metadata": {
        "id": "nEBYvBx20zcp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b51816de-27b8-471c-9527-2aefcd70207c"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1/1 [==============================] - 0s 29ms/step\n",
            "Such data set can be represented\n",
            "1/1 [==============================] - 0s 26ms/step\n",
            "Such data set can be represented by\n",
            "1/1 [==============================] - 0s 23ms/step\n",
            "Such data set can be represented by an\n",
            "1/1 [==============================] - 0s 22ms/step\n",
            "Such data set can be represented by an m\n",
            "1/1 [==============================] - 0s 43ms/step\n",
            "Such data set can be represented by an m by\n",
            "1/1 [==============================] - 0s 23ms/step\n",
            "Such data set can be represented by an m by n\n",
            "1/1 [==============================] - 0s 22ms/step\n",
            "Such data set can be represented by an m by n matrix\n",
            "1/1 [==============================] - 0s 25ms/step\n",
            "Such data set can be represented by an m by n matrix where\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "Such data set can be represented by an m by n matrix where there\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "Such data set can be represented by an m by n matrix where there where\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n"
      ],
      "metadata": {
        "id": "dILzfIASHv6T"
      },
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "JAW2398nIElo"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}