Hw12.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ec121eb8-92fd-47a6-a5f0-c3bbc6e8ec8a",
   "metadata": {},
   "source": [
    "# Text summary using libraries for NLP: nltk and SpaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d28375ac-6981-4df0-91dc-293f8ffd6563",
   "metadata": {},
   "source": [
    "### Import of necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "55731a17-644c-4a72-be23-3c493cb9b481",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "from nltk.probability import FreqDist\n",
    "import string\n",
    "from heapq import nlargest\n",
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0db02fb9-5cef-40b6-96c0-eb11131bfc33",
   "metadata": {},
   "source": [
    "### Preparation of the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "14400ccb-52b0-4a76-bf71-576c1ff81233",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text to process\n",
    "text = 'The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering. Because it has achieved significance within the past fifty years, Criteria Consideration G applies. Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA. Unlike the Mercury, Gemini, and Apollo programs, the SSP’s emphasis was on cost effectiveness and reusability, and eventually the construction of a space station. Including her maiden voyage (launched August 30, 1984), Discovery flew to space thirty-nine times, more than any of the other four orbiters; she was also the first orbiter to fly twenty missions. She had the honor of being chosen as the Return to Flight vehicle after both the Challenger and Columbia accidents. Discovery was the first shuttle to fly with the redesigned SRBs, a result of the Challenger accident, and the first shuttle to fly with the Phase II and Block I SSME. Discovery also carried the Hubble Space Telescope to orbit and performed two of the five servicing missions to the observatory. She flew the first and last dedicated Department of Defense (DoD) missions, as well as the first unclassified defense-related mission. In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle. She was the first orbiter to dock to the ISS, and the first to perform an exchange of a resident crew. Under Criterion C, Discovery is significant as a feat of engineering. According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.” Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials. The Space Shuttle orbiter also featured the first reusable TPS; all previous spaceflight vehicles had a single-use, ablative heat shield. Other notable engineering achievements of the orbiter included the first reusable orbital propulsion system, and the first two-fault-tolerant Integrated Avionics System. As Hale stated, the Space Shuttle remains “the largest, fastest, winged hypersonic aircraft in history,” having regularly flown at twenty-five times the speed of sound.'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f76556b0-2018-4d12-8201-fca1a4f97572",
   "metadata": {},
   "source": [
    "### Text summary using nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ca070b6b-6873-4ad4-89a0-d99ff02bcd92",
   "metadata": {},
   "outputs": [],
   "source": [
    "class NLTKProcessor:\n",
    "    def __init__(self, text):\n",
    "        self.text = text\n",
    "        nltk.download('punkt')\n",
    "        nltk.download('stopwords')\n",
    "        self.stop_words = set(stopwords.words('english'))\n",
    "        self.punctuation = string.punctuation + '\\n'\n",
    "\n",
    "    def prepare_text(self):\n",
    "        words = word_tokenize(self.text)\n",
    "        words = [word.lower() for word in words if word.isalnum()]\n",
    "        words = [word for word in words if word not in self.stop_words]\n",
    "        return words\n",
    "\n",
    "    def generate_summary(self, num_sentences=3):\n",
    "        prepared_text = self.prepare_text()\n",
    "        freq_dist = FreqDist(prepared_text)\n",
    "        ranking = {}\n",
    "        for i, sentence in enumerate(sent_tokenize(self.text)):\n",
    "            for word in word_tokenize(sentence.lower()):\n",
    "                if word in freq_dist:\n",
    "                    if i not in ranking:\n",
    "                        ranking[i] = freq_dist[word]\n",
    "                    else:\n",
    "                        ranking[i] += freq_dist[word]\n",
    "\n",
    "        top_sentences = nlargest(num_sentences, ranking, key=ranking.get)\n",
    "        summary = [sent_tokenize(self.text)[i] for i in sorted(top_sentences)]\n",
    "        return ' '.join(summary)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6ef3e906-c3fe-462f-86c3-82c46ba23cb7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens:\n",
      "['orbiter', 'discovery', 'considered', 'eligible', 'listing', 'national', 'register', 'historic', 'places', 'nrhp', 'context', 'space', 'shuttle', 'program', 'criterion', 'areas', 'space', 'exploration', 'transportation', 'criterion', 'c', 'area', 'engineering', 'achieved', 'significance', 'within', 'past', 'fifty', 'years', 'criteria', 'consideration', 'g', 'applies', 'criterion', 'discovery', 'significant', 'oldest', 'three', 'extant', 'orbiter', 'vehicles', 'constructed', 'space', 'shuttle', 'program', 'ssp', 'longest', 'running', 'american', 'space', 'program', 'date', 'third', 'five', 'orbiters', 'built', 'nasa', 'unlike', 'mercury', 'gemini', 'apollo', 'programs', 'ssp', 'emphasis', 'cost', 'effectiveness', 'reusability', 'eventually', 'construction', 'space', 'station', 'including', 'maiden', 'voyage', 'launched', 'august', '30', '1984', 'discovery', 'flew', 'space', 'times', 'four', 'orbiters', 'also', 'first', 'orbiter', 'fly', 'twenty', 'missions', 'honor', 'chosen', 'return', 'flight', 'vehicle', 'challenger', 'columbia', 'accidents', 'discovery', 'first', 'shuttle', 'fly', 'redesigned', 'srbs', 'result', 'challenger', 'accident', 'first', 'shuttle', 'fly', 'phase', 'ii', 'block', 'ssme', 'discovery', 'also', 'carried', 'hubble', 'space', 'telescope', 'orbit', 'performed', 'two', 'five', 'servicing', 'missions', 'observatory', 'flew', 'first', 'last', 'dedicated', 'department', 'defense', 'dod', 'missions', 'well', 'first', 'unclassified', 'mission', 'addition', 'discovery', 'vital', 'construction', 'international', 'space', 'station', 'iss', 'flew', 'thirteen', 'total', 'missions', 'flown', 'station', 'space', 'shuttle', 'first', 'orbiter', 'dock', 'iss', 'first', 'perform', 'exchange', 'resident', 'crew', 'criterion', 'c', 'discovery', 'significant', 'feat', 'engineering', 'according', 'wayne', 'hale', 'flight', 'director', 'johnson', 'space', 'center', 'space', 'shuttle', 'orbiter', 'represents', 'huge', 'technological', 'leap', 'expendable', 'rockets', 'capsules', 'reusable', 'winged', 'hypersonic', 'although', 'base', 'structure', 'followed', 'conventional', 'aircraft', 'design', 'used', 'advanced', 'materials', 'minimized', 'weight', 'purposes', 'featured', 'low', 'thermal', 'expansion', 'ratios', 'provided', 'stable', 'base', 'thermal', 'protection', 'system', 'tps', 'materials', 'space', 'shuttle', 'orbiter', 'also', 'featured', 'first', 'reusable', 'tps', 'previous', 'spaceflight', 'vehicles', 'ablative', 'heat', 'shield', 'notable', 'engineering', 'achievements', 'orbiter', 'included', 'first', 'reusable', 'orbital', 'propulsion', 'system', 'first', 'integrated', 'avionics', 'system', 'hale', 'stated', 'space', 'shuttle', 'remains', 'largest', 'fastest', 'winged', 'hypersonic', 'aircraft', 'history', 'regularly', 'flown', 'times', 'speed', 'sound']\n",
      "\n",
      "NLTK Summary:\n",
      "The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering. Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA. According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.” Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\user\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\user\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "nltk_processor = NLTKProcessor(text)\n",
    "prepared_text = nltk_processor.prepare_text()\n",
    "print(f'Tokens:\\n{prepared_text}\\n')\n",
    "nltk_summary = nltk_processor.generate_summary()\n",
    "print(f'NLTK Summary:\\n{nltk_summary}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e1aa8ab-79ad-4f8e-93c0-225c5d02e800",
   "metadata": {},
   "source": [
    "### Text summary using SpaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bd59cf65-1b45-42f6-9b40-1a8bb25db7f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SpaCyProcessor:\n",
    "    def __init__(self, text):\n",
    "        self.text = text\n",
    "        self.nlp = spacy.load('en_core_web_sm')\n",
    "        nltk.download('punkt')\n",
    "        nltk.download('stopwords')\n",
    "        self.stop_words = set(stopwords.words('english'))\n",
    "        self.punctuation = string.punctuation + '\\n'\n",
    "\n",
    "    def prepare_text(self):\n",
    "        doc = self.nlp(self.text)\n",
    "        tokens = [token.text for token in doc]\n",
    "        tokens = [token.lower() for token in tokens if token.isalnum()]\n",
    "        tokens = [token for token in tokens if token not in self.stop_words]\n",
    "        return tokens\n",
    "\n",
    "    def generate_summary(self, prepared_text, num_sentences=3):\n",
    "        freq_dist = FreqDist(prepared_text)\n",
    "        ranking = {}\n",
    "        for i, sentence in enumerate(sent_tokenize(self.text)):\n",
    "            doc = self.nlp(sentence)\n",
    "            for token in doc:\n",
    "                if token.text.lower() in freq_dist:\n",
    "                    if i not in ranking:\n",
    "                        ranking[i] = freq_dist[token.text.lower()]\n",
    "                    else:\n",
    "                        ranking[i] += freq_dist[token.text.lower()]\n",
    "\n",
    "        top_sentences = nlargest(num_sentences, ranking, key=ranking.get)\n",
    "        summary = [sent_tokenize(self.text)[i] for i in sorted(top_sentences)]\n",
    "        return ' '.join(summary)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "544f338d-91f9-42a2-b69a-8a5e230d4d77",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SpaCy Summary:\n",
      "The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering. Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA. According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.” Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\user\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\user\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "spacy_processor = SpaCyProcessor(text)\n",
    "spacy_summary = spacy_processor.generate_summary(prepared_text)\n",
    "print(f'SpaCy Summary:\\n{spacy_summary}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be595fa1-76d1-42a0-8045-841fd7e8f198",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}