TripAdvisor Crawling Suite User Guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TripAdvisor Crawling Suite User Guide\n",
    "\n",
    "Kevin Jiahua Du <br />\n",
    "14 July, 2017"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last Updated: 2017-07-14 04:30:01.661954\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "print('Last Updated: ' + str(datetime.datetime.now()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This guide will walk you through the steps of using the crawling suite to effortlessly collect hotel data from TripAdvisor, including basic hotel information, reviews, and user profiles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "\n",
    "First of all, we need to configure the suite to specify locations where we want to collect hotel data.\n",
    "Open `config.ini` with a text editor, and you will see something like this:\n",
    "\n",
    "> **[THREAD]** <br />\n",
    "SleepTime = 2 <br />\n",
    "SnippetThread = 3 <br />\n",
    "HotelThread = 3 <br />\n",
    "ReviewThread = 3 <br />\n",
    "UserThread = 3 <br />\n",
    "**[LOCATION]** <br />\n",
    "List = placeholder\n",
    "\n",
    "Calm down. If you have no idea what a thread is, just leave the THREAD section as it is. \n",
    "Otherwise edit accordingly `SLEEP_TIME` seconds to wait between each connection to TripAdvisor; `SNIPPET_THREAD_NUM`, `DETAIL_THREAD_NUM`, `REVIEW_THREAD_NUM`, and `USER_THREAD_NUM` the number of threads in each crawling module. \n",
    "In practice, a minimum pause of two seconds and at most three threads for each crawler are recommended.\n",
    "\n",
    "Telling the suite where to collect data is easy. The most simple way is to replace the placeholder with an URL describing a location.\n",
    "You can find the URL after you submit a query for a location on TripAdvisor. Take Melbourne, it should be: \n",
    "\n",
    "> https://www.tripadvisor.com.au/Hotels-g255100-Melbourne_Victoria-Hotels.html\n",
    "\n",
    "The placeholder is also available for multiple locations by concatenating all URLs with semicolon and no space. For example, $URL_1;URL_2;URL_3;...;URL_n$. \n",
    "\n",
    "We offer a crawling module to automatically gather all location URLs within a given area, say all cities within Victoria and even Australia. \n",
    "Once the gathering finished, the module will store the URL string in the same directory.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "locations in 10 pages.\n",
      "[page 1] https://www.tripadvisor.com.au/Hotels-g255098-Victoria-Hotels.html\n",
      "[page 2] https://www.tripadvisor.com.au/Hotels-g255098-oa20-Victoria-Hotels.html\n",
      "[page 3] https://www.tripadvisor.com.au/Hotels-g255098-oa40-Victoria-Hotels.html\n",
      "[page 4] https://www.tripadvisor.com.au/Hotels-g255098-oa60-Victoria-Hotels.html\n",
      "[page 5] https://www.tripadvisor.com.au/Hotels-g255098-oa80-Victoria-Hotels.html\n",
      "[page 6] https://www.tripadvisor.com.au/Hotels-g255098-oa100-Victoria-Hotels.html\n",
      "[page 7] https://www.tripadvisor.com.au/Hotels-g255098-oa120-Victoria-Hotels.html\n",
      "[page 8] https://www.tripadvisor.com.au/Hotels-g255098-oa140-Victoria-Hotels.html\n",
      "[page 9] https://www.tripadvisor.com.au/Hotels-g255098-oa160-Victoria-Hotels.html\n",
      "[page 10] https://www.tripadvisor.com.au/Hotels-g255098-oa180-Victoria-Hotels.html\n",
      "195 locations found.\n"
     ]
    }
   ],
   "source": [
    "from crawlers import crawlLocations\n",
    "\n",
    "area_url = 'https://www.tripadvisor.com.au/Hotels-g255098-Victoria-Hotels.html'\n",
    "crawlLocations.start(area_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to set up a database called ta.db, and create four tables for storing locations, hotels, reviews, and reviewers with a few lines of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "database ta.db created.\n"
     ]
    }
   ],
   "source": [
    "from os.path import isfile\n",
    "from tadb import taDB\n",
    "\n",
    "fn = 'ta.db'\n",
    "if not isfile(fn):\n",
    "    with taDB(fn) as db:\n",
    "        db.create_tables()\n",
    "        print('database {} created.'.format(fn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we get started, we also need to turn on the event logger to record the crawling process and apply the configuration we just made to the suite."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO - parameters: [0; 3; 3; 3; 5]\n",
      "INFO - 2 locations found\n"
     ]
    }
   ],
   "source": [
    "from start import *\n",
    "\n",
    "# Trigger the event logger.\n",
    "init_logger()\n",
    "# Load and set crawler parameters.\n",
    "urls = load_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, so far we have done all the preparation.\n",
    "\n",
    "## Data Acquisition\n",
    "\n",
    "Collecting data from TripAdvisor using is straightforward: The suite traverses each URL in the location list, invoking different modules to collect and dump the correponding information in the form of raw HTML. \n",
    "Note that those four modules must be run in the shown order.\n",
    "Nevertheless, depending on the type of data required, you can always stop the procedure at any step as long as the preceding steps are completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO - [location 552264] Trawool Victoria\n",
      "INFO - 1 hotels in 1 pages\n",
      "INFO - 0 hotels in local cache\n",
      "INFO - [page 1] https://www.tripadvisor.com.au/Hotels-g552264-Trawool_Victoria-Hotels.html?seen=0&sequence=1&geo=552264&requestingServlet=Hotels&refineForm=true&hs=&adults=2&rooms=1&o=a0&pageSize=&rad=0&dateBumped=NONE&displayedSortOrder=popularity\n",
      "INFO - \t#0, totaling 1\n",
      "INFO - [worker 2] shutting down\n",
      "INFO - [worker 3] shutting down\n",
      "INFO - [worker 1] shutting down\n",
      "INFO - all hotel ids are ready\n",
      "INFO - [hotel 660346] https://www.tripadvisor.com.au/Hotel_Review-g552264-d660346-Reviews-Comfort_Inn_Trawool-Trawool_Victoria.html\n",
      "INFO - \t[hotel 660346] 72 reviews in 15 pages\n",
      "INFO - \t[hotel 660346] [page 1] 5 reviews, totaling 5\n",
      "INFO - \t[hotel 660346] [page 2] 5 reviews, totaling 10\n",
      "INFO - \t[hotel 660346] [page 3] 5 reviews, totaling 15\n",
      "INFO - \t[hotel 660346] [page 4] 5 reviews, totaling 20\n",
      "INFO - \t[hotel 660346] [page 5] 5 reviews, totaling 25\n",
      "INFO - \t[hotel 660346] [page 6] 5 reviews, totaling 30\n",
      "INFO - \t[hotel 660346] [page 7] 5 reviews, totaling 35\n",
      "INFO - \t[hotel 660346] [page 8] 5 reviews, totaling 40\n",
      "INFO - \t[hotel 660346] [page 9] 5 reviews, totaling 45\n",
      "INFO - \t[hotel 660346] [page 10] 5 reviews, totaling 50\n",
      "INFO - \t[hotel 660346] [page 11] 5 reviews, totaling 55\n",
      "INFO - \t[hotel 660346] [page 12] 5 reviews, totaling 60\n",
      "INFO - \t[hotel 660346] [page 13] 5 reviews, totaling 65\n",
      "INFO - \t[hotel 660346] [page 14] 5 reviews, totaling 70\n",
      "INFO - \t[hotel 660346] [page 15] 2 reviews, totaling 72\n",
      "INFO - \t[hotel 660346] 72 reviews retrieved\n",
      "INFO - [worker 2] shutting down\n",
      "INFO - [worker 3] shutting down\n",
      "INFO - [worker 1] shutting down\n",
      "INFO - all review ids are ready\n",
      "INFO - \t[hotel 660346] from 1 to 72\n",
      "INFO - [worker 3] shutting down\n",
      "INFO - [worker 1] shutting down\n",
      "INFO - [worker 2] shutting down\n",
      "INFO - all reviews are ready\n",
      "INFO - retrieving users...\n",
      "INFO - 72 users found\n",
      "INFO - [user 00488B15DF5F89809CE9B96D77FA8B42]\n",
      "INFO - [user 0493E9D5D9FAD24D83730C8ACA63E706]\n",
      "INFO - [user 0A5CE434B4E1E6B3D8040C616F9D38CD]\n",
      "INFO - [user 11B8A797BB374E34A08F14AE238AD48B]\n",
      "INFO - [user 141C3D69FEB9FDC3B1A957AA70F7FC7D]\n",
      "INFO - [user 1AB8CA0E7DAD2309F20FFE8E5955A6C4]\n",
      "INFO - [user 1B9B2B1A2949FBD460DE043F6F3B1BEF]\n",
      "INFO - [user 1D0BB50A8EC974891F971B3931EB5C36]\n",
      "INFO - [user 1E08569ED485AAC973871B5BB28F1D1B]\n",
      "INFO - [user 24E99EDCF5B0466B3EE706CB6E3B47FB]\n",
      "INFO - [user 2640635A1E66DB34622FB46B19D26E11]\n",
      "INFO - [user 26D703A30F2C59107C5D9E5FB045774A]\n",
      "INFO - [user 274E7836B2F8E34D99938414D6C8A449]\n",
      "INFO - [user 27C70FEA98C66EFB286993B18F19E1CB]\n",
      "INFO - [user 2F5E043307ECD5A3879A19863C26AE6F]\n",
      "INFO - [user 31AC551DC2E3853CB4BC2A950F99A750]\n",
      "INFO - [user 32C4CE108C3D9580062C30F4F5CADEE4]\n",
      "INFO - [user 369F522943CAC780A1EDB1095B2A2C51]\n",
      "INFO - [user 3AEDC7ED7A750B51B4230BAFDE04104F]\n",
      "INFO - [user 3D1DA5DB66CA62F380DEEA1F2E42EF29]\n",
      "INFO - [user 3F71BCA9F88C5BAFEAE279BB33907E92]\n",
      "INFO - [user 4091DE0A1D050C60FF7D3E31FCA265CE]\n",
      "INFO - [user 4216644B2B3E5333E35A30565290DF74]\n",
      "INFO - [user 44C072E5D7CB641B6C74618532EE0C67]\n",
      "INFO - [user 471F5F614835BDAE44624ADCC5BC3179]\n",
      "INFO - [user 4F346C4C664117DE8D840F498F38806E]\n",
      "INFO - [user 546ACCC7FA3B724786513F49043ABE4F]\n",
      "INFO - [user 54F0A20ADAB6F11948BCE047DFB9F605]\n",
      "INFO - [user 5A434ACB51E8CE5FF2B4E68972C20F51]\n",
      "INFO - [user 5B09E3237D7B24CB450A534DBFBA02AA]\n",
      "INFO - [user 5D311C1A5C54D251F2EEDC8E79273CC3]\n",
      "INFO - [user 6BA070D60358CD1DA4CAD9D29BFF7CD1]\n",
      "INFO - [user 72C227ACF2158E2AFC461D91BA7B86ED]\n",
      "INFO - [user 75105453B9261F313D437E154646367C]\n",
      "INFO - [user 79E95D034C1E6EC625D350191185D00D]\n",
      "INFO - [user 7D5B2A809A82FAAFDDFECE414E2C13B5]\n",
      "INFO - [user 7FB95052A8FDC1458FA16F4063C3D4FF]\n",
      "INFO - [user 812AF6FA6D84E4A6EB7B6822E7943310]\n",
      "INFO - [user 83991B6C21929E9B62398A46DB9E8BF4]\n",
      "INFO - [user 83EAE255A3CB3B7261923F2AB68DB911]\n",
      "INFO - [user 841AC2C00AF29D506951CB9678D43474]\n",
      "INFO - [user 86B42A2102F20203772D0DD0EE0F5858]\n",
      "INFO - [user 871E2B6D2863474AD1406839E906E19F]\n",
      "INFO - [user 8C2AE9C2EBDF526C013F3C4B46A9D080]\n",
      "INFO - [user 8C2E786919CBDAED72EF110B38DAA178]\n",
      "INFO - [user 90F99D6B1FC3A1FD2E1AC52A97C0509E]\n",
      "INFO - [user 961813041BFA0419D5880E73EAEEF9DC]\n",
      "INFO - [user 9D2082EDCF3DC92961AECB77759150D2]\n",
      "INFO - [user B1A5B7467678224CF6FFBD87DA7D5E66]\n",
      "INFO - [user B79A8D783CEAC3DBE4FC7AC6618DC554]\n",
      "INFO - [user BA0B5797D5A342537EB070B8EF99A793]\n",
      "INFO - [user BB274CFD1EDFBD03176807FA6CF542BA]\n",
      "INFO - [user C122BF073E12846D0372A76B7E5CF43F]\n",
      "INFO - [user C34780BD8C16A26BB79BD04EFF6637F4]\n",
      "INFO - [user C45A2291956717B567989754E111C089]\n",
      "INFO - [user C4E4F4D3630AA153F0F0D38B8DF9DF7F]\n",
      "INFO - [user C5D7BCD24820A6877AC9CF929F49482F]\n",
      "INFO - [user C7F9CCF1CADBDDE2570AED7973927704]\n",
      "INFO - [user C943E71137CCFE899F068309DF9A2DCB]\n",
      "INFO - [user CD24804E5992917D21E46C80B62A41EA]\n",
      "INFO - [user CF5BDC137D67502744A58D0A18105E57]\n",
      "INFO - [user D4003973FB5F2F4175FAB8BCAD23CD7A]\n",
      "INFO - [user DC9CA1AC54BEE97BC4F2AA051EB6CF02]\n",
      "INFO - [user DCDB40BBB398F92DBD9F6ABFE420C4B6]\n",
      "INFO - [user DF1712CA573E96625477EF4A969BB819]\n",
      "INFO - [user E0E6E07A87FEE810709097734D63CDC8]\n",
      "INFO - [user E903C4F5B4A581405AE124E12CDA28E0]\n",
      "INFO - [user E98BD2A2AFF876C71F2BE150B4E4B5F2]\n",
      "INFO - [user EDAB30A6D7E94AFBF76495113B4A839D]\n",
      "INFO - [user F26B9B8B90F2ECFE57C8AADBDA9B5D37]\n",
      "INFO - [user F423E9B7B61EA54F6D9379FD83E0606E]\n",
      "INFO - [user F9A55264CB8FFA786ADD87178D1DAD2F]\n",
      "INFO - [worker 2] shutting down\n",
      "INFO - [worker 4] shutting down\n",
      "INFO - [worker 3] shutting down\n",
      "INFO - [worker 1] shutting down\n",
      "INFO - [worker 5] shutting down\n",
      "INFO - all user ids are ready\n",
      "INFO - [location 644412] Mount Waverley Greater Melbourne Victoria\n",
      "INFO - 1 hotels in 1 pages\n",
      "INFO - 0 hotels in local cache\n",
      "INFO - [page 1] https://www.tripadvisor.com.au/Hotels-g644412-Mount_Waverley_Greater_Melbourne_Victoria-Hotels.html?seen=0&sequence=1&geo=644412&requestingServlet=Hotels&refineForm=true&hs=&adults=2&rooms=1&o=a0&pageSize=&rad=0&dateBumped=NONE&displayedSortOrder=popularity\n",
      "INFO - \t#0, totaling 1\n",
      "INFO - [worker 3] shutting down\n",
      "INFO - [worker 1] shutting down\n",
      "INFO - [worker 2] shutting down\n",
      "INFO - all hotel ids are ready\n",
      "INFO - [hotel 566124] https://www.tripadvisor.com.au/Hotel_Review-g644412-d566124-Reviews-Lang_Mews-Mount_Waverley_Greater_Melbourne_Victoria.html\n",
      "INFO - \t[hotel 566124] 1 reviews in 1 pages\n",
      "INFO - \t[hotel 566124] [page 1] 1 reviews, totaling 1\n",
      "INFO - \t[hotel 566124] 1 reviews retrieved\n",
      "INFO - [worker 2] shutting down\n",
      "INFO - [worker 1] shutting down\n",
      "INFO - [worker 3] shutting down\n",
      "INFO - all review ids are ready\n",
      "INFO - \t[hotel 566124] from 1 to 1\n",
      "INFO - [worker 2] shutting down\n",
      "INFO - [worker 3] shutting down\n",
      "INFO - [worker 1] shutting down\n",
      "INFO - all reviews are ready\n",
      "INFO - retrieving users...\n",
      "INFO - 73 users found\n",
      "INFO - [user 00488B15DF5F89809CE9B96D77FA8B42] PASSED: verified\n",
      "INFO - [user 0493E9D5D9FAD24D83730C8ACA63E706] PASSED: verified\n",
      "INFO - [user 0A5CE434B4E1E6B3D8040C616F9D38CD] PASSED: verified\n",
      "INFO - [user 11B8A797BB374E34A08F14AE238AD48B] PASSED: verified\n",
      "INFO - [user 141C3D69FEB9FDC3B1A957AA70F7FC7D] PASSED: verified\n",
      "INFO - [user 18FBF1599D461C99958B1DB4F6CA4D55]\n",
      "INFO - [user 1AB8CA0E7DAD2309F20FFE8E5955A6C4] PASSED: verified\n",
      "INFO - [user 1B9B2B1A2949FBD460DE043F6F3B1BEF] PASSED: verified\n",
      "INFO - [user 1D0BB50A8EC974891F971B3931EB5C36] PASSED: verified\n",
      "INFO - [user 1E08569ED485AAC973871B5BB28F1D1B] PASSED: verified\n",
      "INFO - [user 24E99EDCF5B0466B3EE706CB6E3B47FB] PASSED: verified\n",
      "INFO - [user 2640635A1E66DB34622FB46B19D26E11] PASSED: verified\n",
      "INFO - [user 26D703A30F2C59107C5D9E5FB045774A] PASSED: verified\n",
      "INFO - [user 274E7836B2F8E34D99938414D6C8A449] PASSED: verified\n",
      "INFO - [user 27C70FEA98C66EFB286993B18F19E1CB] PASSED: verified\n",
      "INFO - [user 2F5E043307ECD5A3879A19863C26AE6F] PASSED: verified\n",
      "INFO - [user 31AC551DC2E3853CB4BC2A950F99A750] PASSED: verified\n",
      "INFO - [user 32C4CE108C3D9580062C30F4F5CADEE4] PASSED: verified\n",
      "INFO - [user 369F522943CAC780A1EDB1095B2A2C51] PASSED: verified\n",
      "INFO - [user 3AEDC7ED7A750B51B4230BAFDE04104F] PASSED: verified\n",
      "INFO - [user 3D1DA5DB66CA62F380DEEA1F2E42EF29] PASSED: verified\n",
      "INFO - [user 3F71BCA9F88C5BAFEAE279BB33907E92] PASSED: verified\n",
      "INFO - [user 4091DE0A1D050C60FF7D3E31FCA265CE] PASSED: verified\n",
      "INFO - [user 4216644B2B3E5333E35A30565290DF74] PASSED: verified\n",
      "INFO - [user 44C072E5D7CB641B6C74618532EE0C67] PASSED: verified\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO - [user 471F5F614835BDAE44624ADCC5BC3179] PASSED: verified\n",
      "INFO - [user 4F346C4C664117DE8D840F498F38806E] PASSED: verified\n",
      "INFO - [user 546ACCC7FA3B724786513F49043ABE4F] PASSED: verified\n",
      "INFO - [user 54F0A20ADAB6F11948BCE047DFB9F605] PASSED: verified\n",
      "INFO - [user 5A434ACB51E8CE5FF2B4E68972C20F51] PASSED: verified\n",
      "INFO - [user 5B09E3237D7B24CB450A534DBFBA02AA] PASSED: verified\n",
      "INFO - [user 5D311C1A5C54D251F2EEDC8E79273CC3] PASSED: verified\n",
      "INFO - [user 6BA070D60358CD1DA4CAD9D29BFF7CD1] PASSED: verified\n",
      "INFO - [user 72C227ACF2158E2AFC461D91BA7B86ED] PASSED: verified\n",
      "INFO - [user 75105453B9261F313D437E154646367C] PASSED: verified\n",
      "INFO - [user 79E95D034C1E6EC625D350191185D00D] PASSED: verified\n",
      "INFO - [user 7D5B2A809A82FAAFDDFECE414E2C13B5] PASSED: verified\n",
      "INFO - [user 7FB95052A8FDC1458FA16F4063C3D4FF] PASSED: verified\n",
      "INFO - [user 812AF6FA6D84E4A6EB7B6822E7943310] PASSED: verified\n",
      "INFO - [user 83991B6C21929E9B62398A46DB9E8BF4] PASSED: verified\n",
      "INFO - [user 83EAE255A3CB3B7261923F2AB68DB911] PASSED: verified\n",
      "INFO - [user 841AC2C00AF29D506951CB9678D43474] PASSED: verified\n",
      "INFO - [user 86B42A2102F20203772D0DD0EE0F5858] PASSED: verified\n",
      "INFO - [user 871E2B6D2863474AD1406839E906E19F] PASSED: verified\n",
      "INFO - [user 8C2AE9C2EBDF526C013F3C4B46A9D080] PASSED: verified\n",
      "INFO - [user 8C2E786919CBDAED72EF110B38DAA178] PASSED: verified\n",
      "INFO - [user 90F99D6B1FC3A1FD2E1AC52A97C0509E] PASSED: verified\n",
      "INFO - [user 961813041BFA0419D5880E73EAEEF9DC] PASSED: verified\n",
      "INFO - [user 9D2082EDCF3DC92961AECB77759150D2] PASSED: verified\n",
      "INFO - [user B1A5B7467678224CF6FFBD87DA7D5E66] PASSED: verified\n",
      "INFO - [user B79A8D783CEAC3DBE4FC7AC6618DC554] PASSED: verified\n",
      "INFO - [user BA0B5797D5A342537EB070B8EF99A793] PASSED: verified\n",
      "INFO - [user BB274CFD1EDFBD03176807FA6CF542BA] PASSED: verified\n",
      "INFO - [user C122BF073E12846D0372A76B7E5CF43F] PASSED: verified\n",
      "INFO - [user C34780BD8C16A26BB79BD04EFF6637F4] PASSED: verified\n",
      "INFO - [user C45A2291956717B567989754E111C089] PASSED: verified\n",
      "INFO - [user C4E4F4D3630AA153F0F0D38B8DF9DF7F] PASSED: verified\n",
      "INFO - [user C5D7BCD24820A6877AC9CF929F49482F] PASSED: verified\n",
      "INFO - [user C7F9CCF1CADBDDE2570AED7973927704] PASSED: verified\n",
      "INFO - [user C943E71137CCFE899F068309DF9A2DCB] PASSED: verified\n",
      "INFO - [user CD24804E5992917D21E46C80B62A41EA] PASSED: verified\n",
      "INFO - [user CF5BDC137D67502744A58D0A18105E57] PASSED: verified\n",
      "INFO - [user D4003973FB5F2F4175FAB8BCAD23CD7A] PASSED: verified\n",
      "INFO - [user DC9CA1AC54BEE97BC4F2AA051EB6CF02] PASSED: verified\n",
      "INFO - [user DCDB40BBB398F92DBD9F6ABFE420C4B6] PASSED: verified\n",
      "INFO - [user DF1712CA573E96625477EF4A969BB819] PASSED: verified\n",
      "INFO - [user E0E6E07A87FEE810709097734D63CDC8] PASSED: verified\n",
      "INFO - [user E903C4F5B4A581405AE124E12CDA28E0] PASSED: verified\n",
      "INFO - [user E98BD2A2AFF876C71F2BE150B4E4B5F2] PASSED: verified\n",
      "INFO - [user EDAB30A6D7E94AFBF76495113B4A839D] PASSED: verified\n",
      "INFO - [user F26B9B8B90F2ECFE57C8AADBDA9B5D37] PASSED: verified\n",
      "INFO - [user F423E9B7B61EA54F6D9379FD83E0606E] PASSED: verified\n",
      "INFO - [user F9A55264CB8FFA786ADD87178D1DAD2F] PASSED: verified\n",
      "INFO - [worker 2] shutting down\n",
      "INFO - [worker 1] shutting down\n",
      "INFO - [worker 5] shutting down\n",
      "INFO - [worker 3] shutting down\n",
      "INFO - [worker 4] shutting down\n",
      "INFO - all user ids are ready\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "from crawlers import crawlSnippets\n",
    "from crawlers import crawlHotels\n",
    "from crawlers import crawlReviews\n",
    "from crawlers import crawlUsers\n",
    "\n",
    "# As an example, we only collect data \n",
    "# from Trawool and Mount Waverley.\n",
    "for url in urls:\n",
    "    gid = re.sub('\\D', '', url)\n",
    "    crawlSnippets.start(gid, url.strip())\n",
    "    crawlHotels.start(gid)\n",
    "    crawlReviews.start(gid)\n",
    "    crawlUsers.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Extraction\n",
    "\n",
    "The suite then extracts disired information by matching several patterns from the raw HTML strings in an offline manner.\n",
    "Unlike the crawling modules, the implementation of extration components is order-insensitive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "extracting hotel 566124...\n",
      "extracting hotel 660346...\n",
      "hotel extraction done.\n",
      "extracting review 104230596...\n",
      "extracting review 108754073...\n",
      "extracting review 111906174...\n",
      "extracting review 112695566...\n",
      "extracting review 116100228...\n",
      "extracting review 120515398...\n",
      "extracting review 121842783...\n",
      "extracting review 122253992...\n",
      "extracting review 122793559...\n",
      "extracting review 124190506...\n",
      "extracting review 129439069...\n",
      "extracting review 129840731...\n",
      "extracting review 132278498...\n",
      "extracting review 134164136...\n",
      "extracting review 136358640...\n",
      "extracting review 142183627...\n",
      "extracting review 143872401...\n",
      "extracting review 144521437...\n",
      "extracting review 146191725...\n",
      "extracting review 157990498...\n",
      "extracting review 161113564...\n",
      "extracting review 165109868...\n",
      "extracting review 165745798...\n",
      "extracting review 170788761...\n",
      "extracting review 174595897...\n",
      "extracting review 175185987...\n",
      "extracting review 175374291...\n",
      "extracting review 178915904...\n",
      "extracting review 179569259...\n",
      "extracting review 184434640...\n",
      "extracting review 184813329...\n",
      "extracting review 184943983...\n",
      "extracting review 186121899...\n",
      "extracting review 186408981...\n",
      "extracting review 186618540...\n",
      "extracting review 188537325...\n",
      "extracting review 190082399...\n",
      "extracting review 191281504...\n",
      "extracting review 196724141...\n",
      "extracting review 205144888...\n",
      "extracting review 205994270...\n",
      "extracting review 208305330...\n",
      "extracting review 214421391...\n",
      "extracting review 226191013...\n",
      "extracting review 237625449...\n",
      "extracting review 238839119...\n",
      "extracting review 257001852...\n",
      "extracting review 257733449...\n",
      "extracting review 258645093...\n",
      "extracting review 259716035...\n",
      "extracting review 264974356...\n",
      "extracting review 275453852...\n",
      "extracting review 281506791...\n",
      "extracting review 311948949...\n",
      "extracting review 345070062...\n",
      "extracting review 369093531...\n",
      "extracting review 462399174...\n",
      "extracting review 488069727...\n",
      "extracting review 49681569...\n",
      "extracting review 501400746...\n",
      "extracting review 54718591...\n",
      "extracting review 58266016...\n",
      "extracting review 59606946...\n",
      "extracting review 61803895...\n",
      "extracting review 65548988...\n",
      "extracting review 67031830...\n",
      "extracting review 74229867...\n",
      "extracting review 74555440...\n",
      "extracting review 77756173...\n",
      "extracting review 81982519...\n",
      "extracting review 83364012...\n",
      "extracting review 84181175...\n",
      "extracting review 86346545...\n",
      "review extraction done.\n",
      "extracting user 00488B15DF5F89809CE9B96D77FA8B42...\n",
      "extracting user 0493E9D5D9FAD24D83730C8ACA63E706...\n",
      "extracting user 0A5CE434B4E1E6B3D8040C616F9D38CD...\n",
      "extracting user 11B8A797BB374E34A08F14AE238AD48B...\n",
      "extracting user 141C3D69FEB9FDC3B1A957AA70F7FC7D...\n",
      "extracting user 18FBF1599D461C99958B1DB4F6CA4D55...\n",
      "extracting user 1AB8CA0E7DAD2309F20FFE8E5955A6C4...\n",
      "extracting user 1B9B2B1A2949FBD460DE043F6F3B1BEF...\n",
      "extracting user 1D0BB50A8EC974891F971B3931EB5C36...\n",
      "extracting user 1E08569ED485AAC973871B5BB28F1D1B...\n",
      "extracting user 24E99EDCF5B0466B3EE706CB6E3B47FB...\n",
      "extracting user 2640635A1E66DB34622FB46B19D26E11...\n",
      "extracting user 26D703A30F2C59107C5D9E5FB045774A...\n",
      "extracting user 274E7836B2F8E34D99938414D6C8A449...\n",
      "extracting user 27C70FEA98C66EFB286993B18F19E1CB...\n",
      "extracting user 2F5E043307ECD5A3879A19863C26AE6F...\n",
      "extracting user 31AC551DC2E3853CB4BC2A950F99A750...\n",
      "extracting user 32C4CE108C3D9580062C30F4F5CADEE4...\n",
      "extracting user 369F522943CAC780A1EDB1095B2A2C51...\n",
      "extracting user 3AEDC7ED7A750B51B4230BAFDE04104F...\n",
      "extracting user 3D1DA5DB66CA62F380DEEA1F2E42EF29...\n",
      "extracting user 3F71BCA9F88C5BAFEAE279BB33907E92...\n",
      "extracting user 4091DE0A1D050C60FF7D3E31FCA265CE...\n",
      "extracting user 4216644B2B3E5333E35A30565290DF74...\n",
      "extracting user 44C072E5D7CB641B6C74618532EE0C67...\n",
      "extracting user 471F5F614835BDAE44624ADCC5BC3179...\n",
      "extracting user 4F346C4C664117DE8D840F498F38806E...\n",
      "extracting user 546ACCC7FA3B724786513F49043ABE4F...\n",
      "extracting user 54F0A20ADAB6F11948BCE047DFB9F605...\n",
      "extracting user 5A434ACB51E8CE5FF2B4E68972C20F51...\n",
      "extracting user 5B09E3237D7B24CB450A534DBFBA02AA...\n",
      "extracting user 5D311C1A5C54D251F2EEDC8E79273CC3...\n",
      "extracting user 6BA070D60358CD1DA4CAD9D29BFF7CD1...\n",
      "extracting user 72C227ACF2158E2AFC461D91BA7B86ED...\n",
      "extracting user 75105453B9261F313D437E154646367C...\n",
      "extracting user 79E95D034C1E6EC625D350191185D00D...\n",
      "extracting user 7D5B2A809A82FAAFDDFECE414E2C13B5...\n",
      "extracting user 7FB95052A8FDC1458FA16F4063C3D4FF...\n",
      "extracting user 812AF6FA6D84E4A6EB7B6822E7943310...\n",
      "extracting user 83991B6C21929E9B62398A46DB9E8BF4...\n",
      "extracting user 83EAE255A3CB3B7261923F2AB68DB911...\n",
      "extracting user 841AC2C00AF29D506951CB9678D43474...\n",
      "extracting user 86B42A2102F20203772D0DD0EE0F5858...\n",
      "extracting user 871E2B6D2863474AD1406839E906E19F...\n",
      "extracting user 8C2AE9C2EBDF526C013F3C4B46A9D080...\n",
      "extracting user 8C2E786919CBDAED72EF110B38DAA178...\n",
      "extracting user 90F99D6B1FC3A1FD2E1AC52A97C0509E...\n",
      "extracting user 961813041BFA0419D5880E73EAEEF9DC...\n",
      "extracting user 9D2082EDCF3DC92961AECB77759150D2...\n",
      "extracting user B1A5B7467678224CF6FFBD87DA7D5E66...\n",
      "extracting user B79A8D783CEAC3DBE4FC7AC6618DC554...\n",
      "extracting user BA0B5797D5A342537EB070B8EF99A793...\n",
      "extracting user BB274CFD1EDFBD03176807FA6CF542BA...\n",
      "extracting user C122BF073E12846D0372A76B7E5CF43F...\n",
      "extracting user C34780BD8C16A26BB79BD04EFF6637F4...\n",
      "extracting user C45A2291956717B567989754E111C089...\n",
      "extracting user C4E4F4D3630AA153F0F0D38B8DF9DF7F...\n",
      "extracting user C5D7BCD24820A6877AC9CF929F49482F...\n",
      "extracting user C7F9CCF1CADBDDE2570AED7973927704...\n",
      "extracting user C943E71137CCFE899F068309DF9A2DCB...\n",
      "extracting user CD24804E5992917D21E46C80B62A41EA...\n",
      "extracting user CF5BDC137D67502744A58D0A18105E57...\n",
      "extracting user D4003973FB5F2F4175FAB8BCAD23CD7A...\n",
      "extracting user DC9CA1AC54BEE97BC4F2AA051EB6CF02...\n",
      "extracting user DCDB40BBB398F92DBD9F6ABFE420C4B6...\n",
      "extracting user DF1712CA573E96625477EF4A969BB819...\n",
      "extracting user E0E6E07A87FEE810709097734D63CDC8...\n",
      "extracting user E903C4F5B4A581405AE124E12CDA28E0...\n",
      "extracting user E98BD2A2AFF876C71F2BE150B4E4B5F2...\n",
      "extracting user EDAB30A6D7E94AFBF76495113B4A839D...\n",
      "extracting user F26B9B8B90F2ECFE57C8AADBDA9B5D37...\n",
      "extracting user F423E9B7B61EA54F6D9379FD83E0606E...\n",
      "extracting user F9A55264CB8FFA786ADD87178D1DAD2F...\n",
      "user extraction done.\n"
     ]
    }
   ],
   "source": [
    "with taDB(fn) as db:\n",
    "    db.extract_hotel_info()\n",
    "    db.extract_review_info()\n",
    "    db.extract_user_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "That's it. Now you have your data ready in the local database. Enjoy!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "84px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": true,
   "threshold": "2",
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}