sentiment_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Sentiment Analysis\n",
    "\n",
    "_Artificial Intelligence Nanodegree Program | Natural Language Processing_\n",
    "\n",
    "---\n",
    "\n",
    "With the rise of online social media platforms like Twitter, Facebook and Reddit, and the proliferation of customer reviews on sites like Amazon and Yelp, we now have access, more than ever before, to massive text-based data sets! They can be analyzed in order to determine how large portions of the population feel about certain products, events, etc. This sort of analysis is called _sentiment analysis_. In this notebook you will build an end-to-end sentiment classification system from scratch.\n",
    "\n",
    "## Instructions\n",
    "\n",
    "Some template code has already been provided for you, and you will need to implement additional functionality to successfully complete this notebook. You will not need to modify the included code beyond what is requested. Sections that begin with '**TODO**' in the header indicate that you need to complete or implement some portion within them. Instructions will be provided for each section and the specifics of the implementation are marked in the code block with a `# TODO: ...` comment. Please be sure to read the instructions carefully!\n",
    "\n",
    "In addition to implementing code, there will be questions for you to answer which relate to the task and your implementation. Each section where you will answer a question is preceded by a '**Question:**' header. Carefully read each question and provide your answer below the '**Answer:**' header by editing the Markdown cell.\n",
    "\n",
    "> **Note**: Code and Markdown cells can be executed using the **Shift+Enter** keyboard shortcut. In addition, a cell can be edited by typically clicking it (double-click for Markdown cells) or by pressing **Enter** while it is highlighted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Step 1: Exploring the data!\n",
    "\n",
    "The dataset we are going to use is very popular among researchers in Natural Language Processing, usually referred to as the [IMDb dataset](http://ai.stanford.edu/~amaas/data/sentiment/). It consists of movie reviews from the website [imdb.com](http://www.imdb.com/), each labeled as either '**pos**itive', if the reviewer enjoyed the film, or '**neg**ative' otherwise.\n",
    "\n",
    "> Maas, Andrew L., et al. [Learning Word Vectors for Sentiment Analysis](http://ai.stanford.edu/~amaas/data/sentiment/). In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, 2011.\n",
    "\n",
    "We have provided the dataset for you. You can load it in by executing the Python cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "\n",
    "def read_imdb_data(data_dir='data/imdb-reviews'):\n",
    "    \"\"\"Read IMDb movie reviews from given directory.\n",
    "    \n",
    "    Directory structure expected:\n",
    "    - data/\n",
    "        - train/\n",
    "            - pos/\n",
    "            - neg/\n",
    "        - test/\n",
    "            - pos/\n",
    "            - neg/\n",
    "    \n",
    "    \"\"\"\n",
    "\n",
    "    # Data, labels to be returned in nested dicts matching the dir. structure\n",
    "    data = {}\n",
    "    labels = {}\n",
    "\n",
    "    # Assume 2 sub-directories: train, test\n",
    "    for data_type in ['train', 'test']:\n",
    "        data[data_type] = {}\n",
    "        labels[data_type] = {}\n",
    "\n",
    "        # Assume 2 sub-directories for sentiment (label): pos, neg\n",
    "        for sentiment in ['pos', 'neg']:\n",
    "            data[data_type][sentiment] = []\n",
    "            labels[data_type][sentiment] = []\n",
    "            \n",
    "            # Fetch list of files for this sentiment\n",
    "            path = os.path.join(data_dir, data_type, sentiment, '*.txt')\n",
    "            files = glob.glob(path)\n",
    "            \n",
    "            # Read reviews data and assign labels\n",
    "            for f in files:\n",
    "                with open(f) as review:\n",
    "                    data[data_type][sentiment].append(review.read())\n",
    "                    labels[data_type][sentiment].append(sentiment)\n",
    "            \n",
    "            assert len(data[data_type][sentiment]) == len(labels[data_type][sentiment]), \\\n",
    "                    \"{}/{} data size does not match labels size\".format(data_type, sentiment)\n",
    "    \n",
    "    # Return data, labels as nested dicts\n",
    "    return data, labels\n",
    "\n",
    "\n",
    "data, labels = read_imdb_data()\n",
    "print(\"IMDb reviews: train = {} pos / {} neg, test = {} pos / {} neg\".format(\n",
    "        len(data['train']['pos']), len(data['train']['neg']),\n",
    "        len(data['test']['pos']), len(data['test']['neg'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now that the data is loaded in, let's take a quick look at one of the positive reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "print(data['train']['pos'][2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "And one with a negative sentiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "print(data['train']['neg'][2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can also make a wordcloud visualization of the reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from wordcloud import WordCloud, STOPWORDS\n",
    "\n",
    "sentiment = 'pos'\n",
    "\n",
    "# Combine all reviews for the desired sentiment\n",
    "combined_text = \" \".join([review for review in data['train'][sentiment]])\n",
    "\n",
    "# Initialize wordcloud object\n",
    "wc = WordCloud(background_color='white', max_words=50,\n",
    "        # update stopwords to include common words like film and movie\n",
    "        stopwords = STOPWORDS.update(['br','film','movie']))\n",
    "\n",
    "# Generate and plot wordcloud\n",
    "plt.imshow(wc.generate(combined_text))\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Try changing the sentiment to `'neg'` and see if you can spot any obvious differences between the wordclouds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### TODO: Form training and test sets\n",
    "\n",
    "Now that you've seen what the raw data looks like, combine the positive and negative documents to get one unified training set and one unified test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "def prepare_imdb_data(data):\n",
    "    \"\"\"Prepare training and test sets from IMDb movie reviews.\"\"\"\n",
    "    \n",
    "    # TODO: Combine positive and negative reviews and labels\n",
    "    \n",
    "    # TODO: Shuffle reviews and corresponding labels within training and test sets\n",
    "    \n",
    "    # Return a unified training data, test data, training labels, test labets\n",
    "    return data_train, data_test, labels_train, labels_test\n",
    "\n",
    "\n",
    "data_train, data_test, labels_train, labels_test = prepare_imdb_data(data)\n",
    "print(\"IMDb reviews (combined): train = {}, test = {}\".format(len(data_train), len(data_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Step 2. Preprocessing\n",
    "\n",
    "As you might have noticed in the sample reviews, our raw data includes HTML. Therefore there are HTML tags that need to be removed. We also need to remove non-letter characters, normalize uppercase letters by converting them to lowercase, tokenize, remove stop words, and stem the remaining words in each document.\n",
    "\n",
    "### TODO: Convert each review to words\n",
    "\n",
    "As your next task, you should complete the function `review_to_words()` that performs all these steps. For your convenience, in the Python cell below we provide you with all the libraries that you may need in order to accomplish these preprocessing steps. Make sure you can import all of them! (If not, pip install from a terminal and run/import again.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# BeautifulSoup to easily remove HTML tags\n",
    "from bs4 import BeautifulSoup \n",
    "\n",
    "# RegEx for removing non-letter characters\n",
    "import re\n",
    "\n",
    "# NLTK library for the remaining steps\n",
    "import nltk\n",
    "nltk.download(\"stopwords\")   # download list of stopwords (only once; need not run it again)\n",
    "from nltk.corpus import stopwords # import stopwords\n",
    "\n",
    "from nltk.stem.porter import *\n",
    "stemmer = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def review_to_words(review):\n",
    "    \"\"\"Convert a raw review string into a sequence of words.\"\"\"\n",
    "    \n",
    "    # TODO: Remove HTML tags and non-letters,\n",
    "    #       convert to lowercase, tokenize,\n",
    "    #       remove stopwords and stem\n",
    "\n",
    "    # Return final list of words\n",
    "    return words\n",
    "\n",
    "\n",
    "review_to_words(\"\"\"This is just a <em>test</em>.<br/><br />\n",
    "But if it wasn't a test, it would make for a <b>Great</b> movie review!\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "With the function `review_to_words()` fully implemeneted, we can apply it to all reviews in both training and test datasets. This may take a while, so let's build in a mechanism to write to a cache file and retrieve from it later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "cache_dir = os.path.join(\"cache\", \"sentiment_analysis\")  # where to store cache files\n",
    "os.makedirs(cache_dir, exist_ok=True)  # ensure cache directory exists\n",
    "\n",
    "def preprocess_data(data_train, data_test, labels_train, labels_test,\n",
    "                    cache_dir=cache_dir, cache_file=\"preprocessed_data.pkl\"):\n",
    "    \"\"\"Convert each review to words; read from cache if available.\"\"\"\n",
    "\n",
    "    # If cache_file is not None, try to read from it first\n",
    "    cache_data = None\n",
    "    if cache_file is not None:\n",
    "        try:\n",
    "            with open(os.path.join(cache_dir, cache_file), \"rb\") as f:\n",
    "                cache_data = pickle.load(f)\n",
    "            print(\"Read preprocessed data from cache file:\", cache_file)\n",
    "        except:\n",
    "            pass  # unable to read from cache, but that's okay\n",
    "    \n",
    "    # If cache is missing, then do the heavy lifting\n",
    "    if cache_data is None:\n",
    "        # Preprocess training and test data to obtain words for each review\n",
    "        words_train = list(map(review_to_words, data_train))\n",
    "        words_test = list(map(review_to_words, data_test))\n",
    "        \n",
    "        # Write to cache file for future runs\n",
    "        if cache_file is not None:\n",
    "            cache_data = dict(words_train=words_train, words_test=words_test,\n",
    "                              labels_train=labels_train, labels_test=labels_test)\n",
    "            with open(os.path.join(cache_dir, cache_file), \"wb\") as f:\n",
    "                pickle.dump(cache_data, f)\n",
    "            print(\"Wrote preprocessed data to cache file:\", cache_file)\n",
    "    else:\n",
    "        # Unpack data loaded from cache file\n",
    "        words_train, words_test, labels_train, labels_test = (cache_data['words_train'],\n",
    "                cache_data['words_test'], cache_data['labels_train'], cache_data['labels_test'])\n",
    "    \n",
    "    return words_train, words_test, labels_train, labels_test\n",
    "\n",
    "\n",
    "# Preprocess data\n",
    "words_train, words_test, labels_train, labels_test = preprocess_data(\n",
    "        data_train, data_test, labels_train, labels_test)\n",
    "\n",
    "# Take a look at a sample\n",
    "print(\"\\n--- Raw review ---\")\n",
    "print(data_train[1])\n",
    "print(\"\\n--- Preprocessed words ---\")\n",
    "print(words_train[1])\n",
    "print(\"\\n--- Label ---\")\n",
    "print(labels_train[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Step 3: Extracting Bag-of-Words features\n",
    "\n",
    "Now that each document has been preprocessed, we can transform each into a Bag-of-Words feature representation. Note that we need to create this transformation based on the training data alone, as we are not allowed to peek at the testing data at all!\n",
    "\n",
    "The dictionary or _vocabulary_ $V$ (set of words shared by documents in the training set) used here will be the one on which we train our supervised learning algorithm. Any future test data must be transformed in the same way for us to be able to apply the learned model for prediction. Hence, it is important to store the transformation / vocabulary as well.\n",
    "\n",
    "> **Note**: The set of words in the training set may not be exactly the same as the test set. What do you do if you encounter a word during testing that you haven't seen before? Unfortunately, we'll have to ignore it, or replace it with a special `<UNK>` token.\n",
    "\n",
    "### TODO: Compute Bag-of-Words features\n",
    "\n",
    "Implement the `extract_BoW_features()` function, apply it to both training and test datasets, and store the results in `features_train` and `features_test` NumPy arrays, respectively. Choose a reasonable vocabulary size, say $|V| = 5000$, and keep only the top $|V|$ occuring words and discard the rest. This number will also serve as the number of columns in the BoW matrices.\n",
    "\n",
    "> **Hint**: You may find it useful to take advantage of `CountVectorizer` from scikit-learn. Also make sure to pickle your Bag-of-Words transformation so that you can use it in future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.externals import joblib\n",
    "# joblib is an enhanced version of pickle that is more efficient for storing NumPy arrays\n",
    "\n",
    "def extract_BoW_features(words_train, words_test, vocabulary_size=5000,\n",
    "                         cache_dir=cache_dir, cache_file=\"bow_features.pkl\"):\n",
    "    \"\"\"Extract Bag-of-Words for a given set of documents, already preprocessed into words.\"\"\"\n",
    "    \n",
    "    # If cache_file is not None, try to read from it first\n",
    "    cache_data = None\n",
    "    if cache_file is not None:\n",
    "        try:\n",
    "            with open(os.path.join(cache_dir, cache_file), \"rb\") as f:\n",
    "                cache_data = joblib.load(f)\n",
    "            print(\"Read features from cache file:\", cache_file)\n",
    "        except:\n",
    "            pass  # unable to read from cache, but that's okay\n",
    "    \n",
    "    # If cache is missing, then do the heavy lifting\n",
    "    if cache_data is None:\n",
    "        # TODO: Fit a vectorizer to training documents and use it to transform them\n",
    "        # NOTE: Training documents have already been preprocessed and tokenized into words;\n",
    "        #       pass in dummy functions to skip those steps, e.g. preprocessor=lambda x: x\n",
    "        vectorizer = ?\n",
    "        features_train = ?\n",
    "\n",
    "        # TODO: Apply the same vectorizer to transform the test documents (ignore unknown words)\n",
    "        features_test = ?\n",
    "        \n",
    "        # NOTE: Remember to convert the features using .toarray() for a compact representation\n",
    "        \n",
    "        # Write to cache file for future runs (store vocabulary as well)\n",
    "        if cache_file is not None:\n",
    "            vocabulary = vectorizer.vocabulary_\n",
    "            cache_data = dict(features_train=features_train, features_test=features_test,\n",
    "                             vocabulary=vocabulary)\n",
    "            with open(os.path.join(cache_dir, cache_file), \"wb\") as f:\n",
    "                joblib.dump(cache_data, f)\n",
    "            print(\"Wrote features to cache file:\", cache_file)\n",
    "    else:\n",
    "        # Unpack data loaded from cache file\n",
    "        features_train, features_test, vocabulary = (cache_data['features_train'],\n",
    "                cache_data['features_test'], cache_data['vocabulary'])\n",
    "    \n",
    "    # Return both the extracted features as well as the vocabulary\n",
    "    return features_train, features_test, vocabulary\n",
    "\n",
    "\n",
    "# Extract Bag of Words features for both training and test datasets\n",
    "features_train, features_test, vocabulary = extract_BoW_features(words_train, words_test)\n",
    "\n",
    "# Inspect the vocabulary that was computed\n",
    "print(\"Vocabulary: {} words\".format(len(vocabulary)))\n",
    "\n",
    "import random\n",
    "print(\"Sample words: {}\".format(random.sample(list(vocabulary.keys()), 8)))\n",
    "\n",
    "# Sample\n",
    "print(\"\\n--- Preprocessed words ---\")\n",
    "print(words_train[5])\n",
    "print(\"\\n--- Bag-of-Words features ---\")\n",
    "print(features_train[5])\n",
    "print(\"\\n--- Label ---\")\n",
    "print(labels_train[5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's try to visualize the Bag-of-Words feature vector for one of our training documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Plot the BoW feature vector for a training document\n",
    "plt.plot(features_train[5,:])\n",
    "plt.xlabel('Word')\n",
    "plt.ylabel('Count')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Question: Reflecting on Bag-of-Words feature representation\n",
    "\n",
    "What is the average sparsity level of BoW vectors in our training set? In other words, on average what percentage of entries in a BoW feature vector are zero?\n",
    "\n",
    "#### Answer:\n",
    "\n",
    "...\n",
    "\n",
    "### Zipf's law\n",
    "\n",
    "[Zipf's law](https://en.wikipedia.org/wiki/Zipf%27s_law), named after the famous American linguist George Zipf, is an empirical law stating that given a large collection of documents, the frequency of any word is inversely proportional to its rank in the frequency table. So the most frequent word will occur about twice as often as the second most frequent word, three times as often as the third most frequent word, and so on. In the figure below we plot number of appearances of each word in our training set against its rank."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Find number of occurrences for each word in the training set\n",
    "word_freq = features_train.sum(axis=0)\n",
    "\n",
    "# Sort it in descending order\n",
    "sorted_word_freq = np.sort(word_freq)[::-1]\n",
    "\n",
    "# Plot \n",
    "plt.plot(sorted_word_freq)\n",
    "plt.gca().set_xscale('log')\n",
    "plt.gca().set_yscale('log')\n",
    "plt.xlabel('Rank')\n",
    "plt.ylabel('Number of occurrences')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Question: Zipf's law\n",
    "\n",
    "What is the total number of occurrences of the most frequent word? What is the the total number of occurrences of the second most frequent word? Do your numbers follow  Zipf's law? If not, why?\n",
    "\n",
    "#### Answer:\n",
    "\n",
    "...\n",
    "\n",
    "### TODO: Normalize feature vectors\n",
    "\n",
    "Bag-of-Words features are intuitive to understand as they are simply word counts. But counts can vary a lot, and potentially throw off learning algorithms later in the pipeline. So, before we proceed further, let's normalize the BoW feature vectors to have unit length.\n",
    "\n",
    "This makes sure that each document's representation retains the unique mixture of feature components, but prevents documents with large word counts from dominating those with fewer words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import sklearn.preprocessing as pr\n",
    "\n",
    "# TODO: Normalize BoW features in training and test set\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Step 4: Classification using BoW features\n",
    "\n",
    "Now that the data has all been properly transformed, we can feed it into a classifier. To get a baseline model, we train a Naive Bayes classifier from scikit-learn (specifically, [`GaussianNB`](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html)), and evaluate its accuracy on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "# TODO: Train a Guassian Naive Bayes classifier\n",
    "clf1 = ?\n",
    "\n",
    "# Calculate the mean accuracy score on training and test sets\n",
    "print(\"[{}] Accuracy: train = {}, test = {}\".format(\n",
    "        clf1.__class__.__name__,\n",
    "        clf1.score(features_train, labels_train),\n",
    "        clf1.score(features_test, labels_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Tree-based algorithms often work quite well on Bag-of-Words as their highly discontinuous and sparse nature is nicely matched by the structure of trees. As your next task, you will try to improve on the Naive Bayes classifier's performance by using scikit-learn's Gradient-Boosted Decision Tree classifer.\n",
    "\n",
    "### TODO: Gradient-Boosted Decision Tree classifier\n",
    "\n",
    "Use [`GradientBoostingClassifier`](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) from scikit-learn to classify the BoW data. This model has a number of parameters. We use default parameters for some of them and pre-set the rest for you, except one: `n_estimators`. Find a proper value for this hyperparameter, use it to classify the data, and report how much improvement you get over Naive Bayes in terms of accuracy.\n",
    "\n",
    "> **Tip**: Use a model selection technique such as cross-validation, grid-search, or an information criterion method, to find an optimal value for the hyperparameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "n_estimators = ?\n",
    "\n",
    "def classify_gboost(X_train, X_test, y_train, y_test):        \n",
    "    # Initialize classifier\n",
    "    clf = GradientBoostingClassifier(n_estimators=n_estimators, learning_rate=1.0, max_depth=1, random_state=0)\n",
    "\n",
    "    # TODO: Classify the data using GradientBoostingClassifier\n",
    "    \n",
    "    # TODO(optional): Perform hyperparameter tuning / model selection\n",
    "    \n",
    "    # TODO: Print final training & test accuracy\n",
    "    \n",
    "    # Return best classifier model\n",
    "    return clf\n",
    "\n",
    "\n",
    "clf2 = classify_gboost(features_train, features_test, labels_train, labels_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### TODO: Adverserial testing\n",
    "\n",
    "Write a short movie review to trick your machine learning model! That is, a movie review with a clear positive or negative sentiment that your model will classify incorrectly.\n",
    "\n",
    "> **Hint**: You might want to take advantage of the biggest weakness of the Bag-of-Words scheme!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# TODO: Write a sample review and set its true sentiment\n",
    "my_review = \"...\"\n",
    "true_sentiment = ''  # sentiment must be 'pos' or 'neg'\n",
    "\n",
    "# TODO: Apply the same preprocessing and vectorizing steps as you did for your training data\n",
    "\n",
    "# TODO: Then call your classifier to label it\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Step 5: Switching gears - RNNs\n",
    "\n",
    "We just saw how the task of sentiment analysis can be solved via a traditional machine learning approach: BoW + a nonlinear classifier. We now switch gears and use Recurrent Neural Networks, and in particular LSTMs, to perform sentiment analysis in Keras. Conveniently, Keras has a built-in [IMDb movie reviews dataset](https://keras.io/datasets/#imdb-movie-reviews-sentiment-classification) that we can use, with the same vocabulary size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from keras.datasets import imdb  # import the built-in imdb dataset in Keras\n",
    "\n",
    "# Set the vocabulary size\n",
    "vocabulary_size = 5000\n",
    "\n",
    "# Load in training and test data (note the difference in convention compared to scikit-learn)\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=vocabulary_size)\n",
    "print(\"Loaded dataset with {} training samples, {} test samples\".format(len(X_train), len(X_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Inspect a sample review and its label\n",
    "print(\"--- Review ---\")\n",
    "print(X_train[7])\n",
    "print(\"--- Label ---\")\n",
    "print(y_train[7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Notice that the label is an integer (0 for negative, 1 for positive), and the review itself is stored as a sequence of integers. These are word IDs that have been preassigned to individual words. To map them back to the original words, you can use the dictionary returned by `imdb.get_word_index()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Map word IDs back to words\n",
    "word2id = imdb.get_word_index()\n",
    "id2word = {i: word for word, i in word2id.items()}\n",
    "print(\"--- Review (with words) ---\")\n",
    "print([id2word.get(i, \" \") for i in X_train[7]])\n",
    "print(\"--- Label ---\")\n",
    "print(y_train[7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Unlike our Bag-of-Words approach, where we simply summarized the counts of each word in a document, this representation essentially retains the entire sequence of words (minus punctuation, stopwords, etc.). This is critical for RNNs to function. But it also means that now the features can be of different lengths!\n",
    "\n",
    "#### Question: Variable length reviews\n",
    "\n",
    "What is the maximum review length (in terms of number of words) in the training set? What is the minimum?\n",
    "\n",
    "#### Answer:\n",
    "\n",
    "...\n",
    "\n",
    "\n",
    "### TODO: Pad sequences\n",
    "\n",
    "In order to feed this data into your RNN, all input documents must have the same length. Let's limit the maximum review length to `max_words` by truncating longer reviews and padding shorter reviews with a null value (0). You can accomplish this easily using the [`pad_sequences()`](https://keras.io/preprocessing/sequence/#pad_sequences) function in Keras. For now, set `max_words` to 500."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from keras.preprocessing import sequence\n",
    "\n",
    "# Set the maximum number of words per document (for both training and testing)\n",
    "max_words = 500\n",
    "\n",
    "# TODO: Pad sequences in X_train and X_test\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### TODO: Design an RNN model for sentiment analysis\n",
    "\n",
    "Build your model architecture in the code cell below. We have imported some layers from Keras that you might need but feel free to use any other layers / transformations you like.\n",
    "\n",
    "Remember that your input is a sequence of words (technically, integer word IDs) of maximum length = `max_words`, and your output is a binary sentiment label (0 or 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, LSTM, Dense, Dropout\n",
    "\n",
    "# TODO: Design your model\n",
    "model = ?\n",
    "\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Question: Architecture and parameters\n",
    "\n",
    "Briefly describe your neural net architecture. How many model parameters does it have that need to be trained?\n",
    "\n",
    "#### Answer:\n",
    "\n",
    "...\n",
    "\n",
    "### TODO: Train and evaluate your model\n",
    "\n",
    "Now you are ready to train your model. In Keras world, you first need to _compile_ your model by specifying the loss function and optimizer you want to use while training, as well as any evaluation metrics you'd like to measure. Specify the approprate parameters, including at least one metric `'accuracy'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# TODO: Compile your model, specifying a loss function, optimizer, and metrics\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Once compiled, you can kick off the training process. There are two important training parameters that you have to specify - **batch size** and **number of training epochs**, which together with your model architecture determine the total training time.\n",
    "\n",
    "Training may take a while, so grab a cup of coffee, or better, go for a hike! If possible, consider using a GPU, as a single training run can take several hours on a CPU.\n",
    "\n",
    "> **Tip**: You can split off a small portion of the training set to be used for validation during training. This will help monitor the training process and identify potential overfitting. You can supply a validation set to `model.fit()` using its `validation_data` parameter, or just specify `validation_split` - a fraction of the training data for Keras to set aside for this purpose (typically 5-10%). Validation metrics are evaluated once at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# TODO: Specify training parameters: batch size and number of epochs\n",
    "batch_size = ?\n",
    "num_epochs = ?\n",
    "\n",
    "# TODO(optional): Reserve/specify some training data for validation (not to be used for training)\n",
    "\n",
    "# TODO: Train your model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Save your model, so that you can quickly load it in future (and perhaps resume training)\n",
    "model_file = \"rnn_model.h5\"  # HDF5 file\n",
    "model.save(os.path.join(cache_dir, model_file))\n",
    "\n",
    "# Later you can load it using keras.models.load_model()\n",
    "#from keras.models import load_model\n",
    "#model = load_model(os.path.join(cache_dir, model_file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Once you have trained your model, it's time to see how well it performs on unseen test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Evaluate your model on the test set\n",
    "scores = model.evaluate(X_test, y_test, verbose=0)  # returns loss and other metrics specified in model.compile()\n",
    "print(\"Test accuracy:\", scores[1])  # scores[1] should correspond to accuracy if you passed in metrics=['accuracy']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Question: Comparing RNNs and Traditional Methods\n",
    "\n",
    "How well does your RNN model perform compared to the BoW + Gradient-Boosted Decision Trees?\n",
    "\n",
    "#### Answer:\n",
    "\n",
    "...\n",
    "\n",
    "## Extensions\n",
    "\n",
    "There are several ways in which you can build upon this notebook. Each comes with its set of challenges, but can be a rewarding experience.\n",
    "\n",
    "- The first thing is to try and improve the accuracy of your model by experimenting with different architectures, layers and parameters. How good can you get without taking prohibitively long to train? How do you prevent overfitting?\n",
    "\n",
    "- Then, you may want to deploy your model as a mobile app or web service. What do you need to do in order to package your model for such deployment? How would you accept a new review, convert it into a form suitable for your model, and perform the actual prediction? (Note that the same environment you used during training may not be available.)\n",
    "\n",
    "- One simplification we made in this notebook is to limit the task to binary classification. The dataset actually includes a more fine-grained review rating that is indicated in each review's filename (which is of the form `<[id]_[rating].txt>` where `[id]` is a unique identifier and `[rating]` is on a scale of 1-10; note that neutral reviews > 4 or < 7 have been excluded). How would you modify the notebook to perform regression on the review ratings? In what situations is regression more useful than classification, and vice-versa?\n",
    "\n",
    "Whatever direction you take, make sure to share your results and learnings with your peers, through blogs, discussions and participating in online competitions. This is also a great way to become more visible to potential employers!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}