cashtag_scraper_Github_version.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "original code from: https://www.data-blogger.com/2017/02/24/gathering-tweets-with-python/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Cashtag Scraper \n",
    "https://stackoverflow.com/questions/14156625/fetching-tweets-with-hashtag-from-twitter-using-python\n",
    "\n",
    "http://followthehashtag.com/datasets/nasdaq-100-companies-free-twitter-dataset/\n",
    "\n",
    "Fetching tweets with a certain hashtag\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Another version, saving tweets in a dataframe\n",
    "last of the replies in https://stackoverflow.com/questions/44948628/how-to-take-all-tweets-in-a-hashtag-with-tweepy\n",
    "\n",
    "Time zone seems to be GMT!! Two hours behind EET.\n",
    "\n",
    "Scrape cashtags: ** TMUS, WDC, EXPE, MDLX, INCT, ADP, TXN, CERN, AAL, FISV ** <br>\n",
    "GILD\tSBUX\t   (excluded: CSCO MSFT\tGOOG NFLX\tGOOGL NVDA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import tweepy\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "consumer_key= [\"insert your own here\"]\n",
    "consumer_secret= [\"insert your own here\"]\n",
    "access_key= [\"insert your own here\"]\n",
    "access_secret= [\"insert your own here\"]\n",
    "\n",
    "#Twitter Access\n",
    "auth = tweepy.OAuthHandler( consumer_key,consumer_secret)\n",
    "auth.set_access_token(access_key,access_secret)\n",
    "api = tweepy.API(auth,wait_on_rate_limit = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To get the full lenghts of the tweet texts, make sure to add **tweet_mode=\"extended\"** and **tweet.full_text** in the code below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame()\n",
    "msgs = []\n",
    "msg =[]\n",
    "\n",
    "for tweet in tweepy.Cursor(api.search, q='$AAL', rpp=100, tweet_mode=\"extended\").items(3000): \n",
    "    #msg = [tweet.created_at, tweet.text, tweet.retweet_count, tweet.favorite_count] \n",
    "    msg = [tweet.created_at, tweet.full_text, tweet.user.followers_count] #choose the twitter data for your df\n",
    "    msg = tuple(msg)                    \n",
    "    msgs.append(msg)\n",
    "\n",
    "df = pd.DataFrame(msgs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created at</th>\n",
       "      <th>text</th>\n",
       "      <th>follower count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>441</th>\n",
       "      <td>2019-03-21 23:25:21</td>\n",
       "      <td>$10.71 Billion in Sales Expected for American ...</td>\n",
       "      <td>259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>442</th>\n",
       "      <td>2019-03-21 23:23:45</td>\n",
       "      <td>American Airlines Group Inc $AAL Expected to A...</td>\n",
       "      <td>506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>443</th>\n",
       "      <td>2019-03-21 23:11:27</td>\n",
       "      <td>$AAL Learn more about the zigzag #Elliottwave ...</td>\n",
       "      <td>18648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>444</th>\n",
       "      <td>2019-03-21 23:06:44</td>\n",
       "      <td>$AAL Correction from the highs seen as a zigza...</td>\n",
       "      <td>18648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>445</th>\n",
       "      <td>2019-03-21 22:05:04</td>\n",
       "      <td>#ThursdayThoughts $AAL Bullish Tweezer Bottom ...</td>\n",
       "      <td>4202</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             created at                                               text  \\\n",
       "441 2019-03-21 23:25:21  $10.71 Billion in Sales Expected for American ...   \n",
       "442 2019-03-21 23:23:45  American Airlines Group Inc $AAL Expected to A...   \n",
       "443 2019-03-21 23:11:27  $AAL Learn more about the zigzag #Elliottwave ...   \n",
       "444 2019-03-21 23:06:44  $AAL Correction from the highs seen as a zigza...   \n",
       "445 2019-03-21 22:05:04  #ThursdayThoughts $AAL Bullish Tweezer Bottom ...   \n",
       "\n",
       "     follower count  \n",
       "441             259  \n",
       "442             506  \n",
       "443           18648  \n",
       "444           18648  \n",
       "445            4202  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns = ['created at', 'text', 'follower count']\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Make dataframe into excel file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import xlsxwriter\n",
    "import os\n",
    "\n",
    "writer_df = pd.ExcelWriter('df.xlsx', engine='xlsxwriter')\n",
    "df.to_excel(writer_df)\n",
    "writer_df.save()\n",
    "\n",
    "os.rename('df.xlsx', '$AAL_tweets.xlsx') # Update cashtag!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}