1_Web_Scraping_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web Scraping ##\n",
    "\n",
    "This tutorial shows how to \"scrape\" data online off of a basic HTML website.\n",
    "\n",
    "[Just give me the code!](#codecell)\n",
    "\n",
    "Suppose we wanted to get a list of the top movie quotes from the American Film Institute. You can see the table here: <a>http://www.afi.com/100Years/quotes.aspx</a>. \n",
    "\n",
    "We'll use BeautifulSoup as the main workhorse to work around the HTML. \n",
    "\n",
    "### HTML Summary ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Text based websites are usually written using just HTML. It is a front-end language which uses tags to separate different styles. For example, to make something appear in bold, you would write:\n",
    "\n",
    "<b>BOLD TEXT WOULD BE WRITTEN HERE</b>\n",
    "\n",
    "Similarly, a link, like the site http://www.afi.com/100Years/quotes.aspx will appear as blue and underlined only if you write it as:\n",
    "\n",
    "<a> http://www.afi.com/100Years/quotes.aspx</a>\n",
    "\n",
    "Tables are typically written as the following:\n",
    "<TABLE>\n",
    "  <tr>\n",
    "  <td> First Row, First Column Text <\\td>\n",
    "  <td> First Row, Second Column Text <\\td>\n",
    "  <td> First Row, Third Column Text <\\td>\n",
    "  <tr>\n",
    "  <tr>\n",
    "  <td> Second Row, First Column Text <\\td>\n",
    "  <td> Second Row, Second Column Text <\\td>\n",
    "  <td> Second Row, Third Column Text <\\td>\n",
    "  <tr>\n",
    "<\\TABLE>\n",
    "\n",
    "You can probably guess that <tr> means a row is about to start, and </tr> indicates the end of the row.\n",
    "Likewise, <td> indicates the beginning of an element of the row, and </td> indicates the end of that element."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Go ahead, make my day. ###\n",
    "\n",
    "So now, let's get the HTML of the AFI website above.\n",
    "\n",
    "We'll start by importing the Beautiful Soup module to parse the HTML, urllib2 to connect to the internet, and pandas to save our data neatly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import urllib2\n",
    "import pandas as pd\n",
    "\n",
    "def get_HTML(url):\n",
    "\treq = urllib2.Request(url, headers={ 'User-Agent': 'Mozilla/5.0' })\n",
    "\thtml = urllib2.urlopen(req).read()\n",
    "\tsoup = BeautifulSoup(html, 'html.parser')\n",
    "\treturn soup\n",
    "\n",
    "\"\"\"\n",
    "If you are using Python 3, do the following\n",
    "\n",
    "1) \"import urllib\" instead of \"import urllib2\"\n",
    "2) To open a webpage, \n",
    "\n",
    "def get_HTML(url):\n",
    "\treq = urllib.request.urlopen(url)\n",
    "\thtml = req.read() \n",
    "\tsoup = BeautifulSoup(html, 'html.parser')\n",
    "\treturn soup\n",
    "\"\"\"\n",
    "\n",
    "afi_soup = get_HTML(\"http://www.afi.com/100Years/quotes.aspx\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>get_HTML</b> is a function which will make a request to our url (<b>req</b>), retrieve the html by reading all the text (<b>html</b>, and make it a BeautifulSoup instance to make it easy to parse the HTML for what we want.\n",
    "\n",
    "<b>afi_soup</b> is a bunch of html text. \n",
    "\n",
    "A nice way to find our target HTML tags (that is, either \"b\" for bold text, \"a\" for website links, \"table\" for tables, etc.) is to go to the website on Chrome or Firefox, right-click and view the page source. Then CTRL+F to search for the table, (like search for \"Frankly, my dear, I don't give a damn\"). You will see the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "OTHER STUFF...\n",
    "\n",
    "\n",
    "<TABLE width=\"530\" border=1 cellspacing=0 cellpadding=2 style=\"border-collapse:collapse;border:solid black .5pt;\">\n",
    "                        <tr style='height:13.0pt'>\n",
    "  <td   valign=top class=\"lttext\" >\n",
    "  <p><span style='font-size:9.0pt;font-family:Verdana; color:black'><b>#</b></span></p>\n",
    "  </td>\n",
    "    <td   valign=top  class=\"lttext\">\n",
    "  <p><span style='font-size:9.0pt;font-family:Verdana; color:black'><b>QUOTE</b></span></p>\n",
    "  </td>\n",
    "  <td   valign=top  class=\"lttext\">\n",
    " <p><span style='font-size:9.0pt;font-family:Verdana; color:black'><b>MOVIE</b></span></p>\n",
    "  </td>\n",
    "    <td   valign=top  class=\"lttext\">\n",
    " <p><span style='font-size:9.0pt;font-family:Verdana; color:black'><b>YEAR</b></span></p>\n",
    "  </td>\n",
    " </tr>\n",
    "  <tr style='height:13.0pt'>\n",
    "  <td   valign=top sclass=\"lttext\" >\n",
    "  <p><span style='font-size:9.0pt;font-family:Verdana; color:black'><b>1</b></span></p>\n",
    "  </td>\n",
    "  <td   valign=top class=\"lttext\" >\n",
    "  <p><span style='font-size:9.0pt;font-family:Verdana; color:black'>Frankly, my dear, I don't give a damn.</span></p>\n",
    "  </td>\n",
    "  <td   valign=top class=\"lttext\" >\n",
    " <p><span style='font-size:9.0pt;font-family:Verdana; color:black'><a href=\"http://www.afi.com/members/catalog/DetailView.aspx?s=&Movie=1181\">GONE WITH THE WIND</a></span></p>\n",
    "  </td>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we see the quote, which corresponds to the table, is wrapped in a paragraph tag (this is the \"p\" tag), which is within an element tag (\"td\"), which is within a row tag (\"tr\") which is in the \"TABLE\". \n",
    "\n",
    "So, let's use BeautifulSoup to retrieve only the table elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "tables = afi_soup.find_all(\"table\")\n",
    "\n",
    "print(len(tables))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Oh no, many tables ####\n",
    "Within our HTML page, we found 3 tables. So now we need to see if we should only analyze the first, second or third. We can do this by finding the first row of each table, and printing out the first 100 characters of the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table 0\n",
      "NEWSLETTER \n",
      "LOGIN\n",
      "SEARCH\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "        $(document).ready(function($){\n",
      "            $('.megamenu').megaMenuCompleteSet({\n",
      "                menu_speed_show : 400, // Time (in milliseconds) to show a \n",
      "-----------------------------\n",
      "Table 1\n",
      "#\n",
      "\n",
      "\n",
      "QUOTE\n",
      "\n",
      "\n",
      "MOVIE\n",
      "\n",
      "\n",
      "YEAR\n",
      "-----------------------------\n",
      "Table 2\n",
      "\"Great movie quotes become part of our cultural vocabulary. When you consider that any phrase from American film is eligible, you realize this is our most subjective topic to date. We expect nothing l\n",
      "-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"Table 0\")\n",
    "print(tables[0].find(\"tr\").text.strip()[0:200])\n",
    "print(\"-----------------------------\")\n",
    "print(\"Table 1\")\n",
    "print(tables[1].find(\"tr\").text.strip()[0:200])\n",
    "print(\"-----------------------------\")\n",
    "print(\"Table 2\")\n",
    "print(tables[2].find(\"tr\").text.strip()[0:200])\n",
    "print(\"-----------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can see that we need to use the 2nd table, <b>tables[1]</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "quote_table = tables[1]\n",
    "\n",
    "rows = quote_table.find_all(\"tr\")\n",
    "\n",
    "sample_row = rows[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect some characteristics of a sample row, <b>sample_row</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tr style=\"height:13.0pt\">\n",
      "<td sclass=\"lttext\" valign=\"top\">\n",
      "<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\"><b>3</b></span></p>\n",
      "</td>\n",
      "<td class=\"lttext\" valign=\"top\">\n",
      "<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\">You don't understand!  I coulda had class. I coulda been a contender. I could've been somebody, instead of a bum, which is what I am.</span></p>\n",
      "</td>\n",
      "<td class=\"lttext\" valign=\"top\">\n",
      "<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\"><a href=\"http://www.afi.com/members/catalog/DetailView.aspx?s=&amp;Movie=51286\">ON THE WATERFRONT</a></span></p>\n",
      "</td>\n",
      "<td class=\"lttext\" valign=\"top\">\n",
      "<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\">1954</span></p>\n",
      "</td>\n",
      "</tr>\n"
     ]
    }
   ],
   "source": [
    "print(sample_row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that it still contains a lot of tags. The meat of each row, as mentioned above, is in the \"td\" tag. So let's extract all the td tags of this row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<td sclass=\"lttext\" valign=\"top\">\\n<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\"><b>3</b></span></p>\\n</td>, <td class=\"lttext\" valign=\"top\">\\n<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\">You don't understand!  I coulda had class. I coulda been a contender. I could've been somebody, instead of a bum, which is what I am.</span></p>\\n</td>, <td class=\"lttext\" valign=\"top\">\\n<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\"><a href=\"http://www.afi.com/members/catalog/DetailView.aspx?s=&amp;Movie=51286\">ON THE WATERFRONT</a></span></p>\\n</td>, <td class=\"lttext\" valign=\"top\">\\n<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\">1954</span></p>\\n</td>]\n"
     ]
    }
   ],
   "source": [
    "print(sample_row.find_all(\"td\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only want the text though. For example, within the row, and for one element we can print out the text by using the <i>text</i> method provided by BeautifulSoup:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "3\n",
      "\n",
      "\n",
      "You don't understand!  I coulda had class. I coulda been a contender. I could've been somebody, instead of a bum, which is what I am.\n",
      "\n",
      "\n",
      "ON THE WATERFRONT\n",
      "\n",
      "\n",
      "1954\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(sample_row.find_all(\"td\")[0].text)\n",
    "print(sample_row.find_all(\"td\")[1].text)\n",
    "print(sample_row.find_all(\"td\")[2].text)\n",
    "print(sample_row.find_all(\"td\")[3].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect! Looks like almost everything we want. It would be nice though to put this in a list of 4 elements: the quote number , the quote, the movie, and the year. I'll use the <b>map</b> function from Python to keep only the text of each \"td\" tag in the row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'3', u\"You don't understand!  I coulda had class. I coulda been a contender. I could've been somebody, instead of a bum, which is what I am.\", u'ON THE WATERFRONT', u'1954']\n"
     ]
    }
   ],
   "source": [
    "row_elements = sample_row.find_all(\"td\")\n",
    "print(map(lambda x: x.text.strip(),row_elements))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to cycle through all the rows, and then loop through all the elements of each row to extract only the quote number, quote, movie and year. I'll use <b>pandas</b> to store them neatly in a dataframe. Note, the first row of the table is just the column headers, which can be fed perfectly into the columns argument of the pandas dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "first = True\n",
    "for row in quote_table.find_all(\"tr\"):\n",
    "\trow_elements = row.find_all('td')\n",
    "\trow_txt = map(lambda x: x.text.strip(),row_elements)\n",
    "\tif first:\n",
    "\t\tfinal_df = pd.DataFrame(columns=row_txt)\n",
    "\t\tfirst = False\n",
    "\t\tcontinue\n",
    "\tfinal_df.loc[len(final_df)+1] = row_txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check out our database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   #                                              QUOTE               MOVIE  \\\n",
      "1  1             Frankly, my dear, I don't give a damn.  GONE WITH THE WIND   \n",
      "2  2       I'm gonna make him an offer he can't refuse.       THE GODFATHER   \n",
      "3  3  You don't understand!  I coulda had class. I c...   ON THE WATERFRONT   \n",
      "4  4  Toto, I've a feeling we're not in Kansas anymore.    THE WIZARD OF OZ   \n",
      "5  5                        Here's looking at you, kid.          CASABLANCA   \n",
      "\n",
      "   YEAR  \n",
      "1  1939  \n",
      "2  1972  \n",
      "3  1954  \n",
      "4  1939  \n",
      "5  1942  \n"
     ]
    }
   ],
   "source": [
    "print(final_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More Dynamic Web-Scraping ###\n",
    "\n",
    "Suppose we also wanted to get more information of each quote, but this required going to the actual link of the movie. For example, we want to see who directed Gone With the Wind (who is Victor Fleming, Alex) which we can see by clicking on the link to the movie. <a> http://www.afi.com/members/catalog/DetailView.aspx?s=&Movie=1181</a>. \n",
    "\n",
    "Note that this link shows up in the HTML of the row where Gone with the Wind is located"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tr style=\"height:13.0pt\">\n",
      "<td sclass=\"lttext\" valign=\"top\">\n",
      "<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\"><b>1</b></span></p>\n",
      "</td>\n",
      "<td class=\"lttext\" valign=\"top\">\n",
      "<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\">Frankly, my dear, I don't give a damn.</span></p>\n",
      "</td>\n",
      "<td class=\"lttext\" valign=\"top\">\n",
      "<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\"><a href=\"http://www.afi.com/members/catalog/DetailView.aspx?s=&amp;Movie=1181\">GONE WITH THE WIND</a></span></p>\n",
      "</td>\n",
      "<td class=\"lttext\" valign=\"top\">\n",
      "<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\">1939</span></p>\n",
      "</td>\n",
      "</tr>\n"
     ]
    }
   ],
   "source": [
    "print(rows[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<td class=\"lttext\" valign=\"top\">\n",
      "<p><span style=\"font-size:9.0pt;font-family:Verdana; color:black\"><a href=\"http://www.afi.com/members/catalog/DetailView.aspx?s=&amp;Movie=1181\">GONE WITH THE WIND</a></span></p>\n",
      "</td>\n"
     ]
    }
   ],
   "source": [
    "print(rows[1].find_all(\"td\")[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access specific attributes within an HTML tag by calling it as if it were a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'http://www.afi.com/members/catalog/DetailView.aspx?s=&Movie=1181'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rows[1].find_all(\"td\")[2].find(\"a\")[\"href\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, to find the link we just need to find the \"a\" element of the cell and call \"href\". \n",
    "\n",
    "Next, looking at the Gone with the Wind page source, the word <b>Director:</b> is in bold, followed by Victor Flemming. Fortunately, we can use BeautifulSoup to find the tag with <b>Director:</b> and then retrieve the next element by simply using the <b>find_next()</b> method of BeautifulSoup. \n",
    "\n",
    "We will do this in two steps.\n",
    "    1. Run <b>get_HTML</b> on the link to the movie which we retrieve with .find(\"a\")[\"href\"]\n",
    "    2. Find <b>Director:</b>, then get the next element which will presumably be the name.\n",
    "    \n",
    "### NOTE:  I will only do this for the first 10 movies on the link. Data scraping can cause a lot of traffic to the site and they can tell you are a web-scraper by making hundreds of requests rapidly from one IP. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "final_df['Director'] = \"\"\n",
    "\n",
    "max_iter = 10\n",
    "index = 0\n",
    "for row in quote_table.find_all(\"tr\"):\n",
    "\tif index>max_iter:\n",
    "\t\tbreak\n",
    "\tif row.find(\"a\"):\n",
    "\t\tmovie_soup = get_HTML(row.find(\"a\")['href'])\n",
    "\t\tdirector = movie_soup.find(text=\"Director:\").find_next().text\n",
    "\t\tfinal_df.loc[index,\"Director\"] = director\n",
    "\tindex+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   #                                              QUOTE               MOVIE  \\\n",
      "1  1             Frankly, my dear, I don't give a damn.  GONE WITH THE WIND   \n",
      "2  2       I'm gonna make him an offer he can't refuse.       THE GODFATHER   \n",
      "3  3  You don't understand!  I coulda had class. I c...   ON THE WATERFRONT   \n",
      "4  4  Toto, I've a feeling we're not in Kansas anymore.    THE WIZARD OF OZ   \n",
      "5  5                        Here's looking at you, kid.          CASABLANCA   \n",
      "\n",
      "   YEAR                     Director  \n",
      "1  1939          Victor Fleming       \n",
      "2  1972    Francis Ford Coppola       \n",
      "3  1954              Elia Kazan       \n",
      "4  1939          Victor Fleming       \n",
      "5  1942          Michael Curtiz       \n"
     ]
    }
   ],
   "source": [
    "print(final_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can save our database nicely with Pandas as a Stata, Excel, CSV or R file for text analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "final_df.to_csv(\"movie_quotes.csv\",index=False, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want more information, check out the BeautifulSoup docs <a>https://www.crummy.com/software/BeautifulSoup/bs4/doc</a>.\n",
    "\n",
    "If you need to datascrape java websites or non-html sites, I recommend Selenium <a>selenium-python.readthedocs.io/</a>.\n",
    "\n",
    "<a id='codecell'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#################################\n",
    "### Author: Paul Soto \t\t  ###\n",
    "### \t\tpaul.soto@upf.edu ###\n",
    "#\t\t\t\t\t\t\t\t#\n",
    "# This file shows the basic of ##\n",
    "# BeautifulSoup to datascrape ###\n",
    "# an HTML based website and save#\n",
    "# the data as a csv. \t\t\t#\n",
    "#################################\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "import urllib2\n",
    "import pandas as pd\n",
    "\n",
    "def get_HTML(url):\n",
    "\t\"\"\"\n",
    "\tThis file creates a HTML soup from a given url\n",
    "\t\"\"\"\n",
    "\treq = urllib2.Request(url, headers={ 'User-Agent': 'Mozilla/5.0' })\n",
    "\thtml = urllib2.urlopen(req).read()\n",
    "\tsoup = BeautifulSoup(html, 'html.parser')\n",
    "\treturn soup\n",
    "\n",
    "\n",
    "# Get HTML Soup\n",
    "afi_soup = get_HTML(\"http://www.afi.com/100Years/quotes.aspx\")\n",
    "\n",
    "# Isolate the table with the  \n",
    "tables = afi_soup.find_all(\"table\")\n",
    "quote_table = tables[1]\n",
    "\n",
    "# Loop through each row, retrieving the \n",
    "# quote ID number, quote, movie and year\n",
    "first = True\n",
    "for row in quote_table.find_all(\"tr\"):\n",
    "\t# Get row elements\n",
    "\trow_elements = row.find_all('td')\n",
    "\t# Strip unnecessary text\n",
    "\trow_txt = map(lambda x: x.text.strip(),row_elements)\n",
    "\tif first:\n",
    "\t\tfinal_df = pd.DataFrame(columns=row_txt)\n",
    "\t\tfirst = False\n",
    "\t\tcontinue\n",
    "\t# Add to dataset\n",
    "\tfinal_df.loc[len(final_df)+1] = row_txt\n",
    "\n",
    "# Clicking movie link to get the director\n",
    "final_df['Director'] = \"\"\n",
    "max_iter = 10\n",
    "index = 0\n",
    "for row in quote_table.find_all(\"tr\"):\n",
    "\tif index>max_iter:\n",
    "\t\tbreak\n",
    "\tif row.find(\"a\"):\n",
    "\t\tmovie_soup = get_HTML(row.find(\"a\")['href'])\n",
    "\t\tdirector = movie_soup.find(text=\"Director:\").find_next().text\n",
    "\t\tfinal_df.loc[index,\"Director\"] = director\n",
    "\tindex+=1\n",
    "\n",
    "# Export to CSV\n",
    "final_df.to_csv(\"movie_quotes.csv\",index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}