2_Text_Preprocessing_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Preprocessing ##\n",
    "\n",
    "This tutorial walks through a basic class for preprocessing text. \n",
    "\n",
    "Suppose we have a list of texts, for example a list of the top movie quotes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   #                                              QUOTE               MOVIE  \\\n",
      "0  1             Frankly, my dear, I don't give a damn.  GONE WITH THE WIND   \n",
      "1  2       I'm gonna make him an offer he can't refuse.       THE GODFATHER   \n",
      "2  3  You don't understand!  I coulda had class. I c...   ON THE WATERFRONT   \n",
      "3  4  Toto, I've a feeling we're not in Kansas anymore.    THE WIZARD OF OZ   \n",
      "4  5                        Here's looking at you, kid.          CASABLANCA   \n",
      "\n",
      "   YEAR                     Director  \n",
      "0  1939          Victor Fleming       \n",
      "1  1972    Francis Ford Coppola       \n",
      "2  1954              Elia Kazan       \n",
      "3  1939          Victor Fleming       \n",
      "4  1942          Michael Curtiz       \n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"movie_quotes.csv\", encoding='utf-8')\n",
    "\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will build a class which cleans up the quote and generates a one-hot encoding or a TF-IDF matrix. \n",
    "\n",
    "Typically, text data is cleaned by removing unnecessary elements. For example, words such as <i>i, me, myself, you, the</i> show up in many texts and are hardly informative. These are called <b>stopwords</b>. We should remove these. Secondly, to create a more homogeneous corpus of text, we could lower case all characters so that the feature <i>Frankly, my dear</i> is identical to <i>frankly, my dear</i>. Lastly, in this tutorial we will <b>stem</b> the words to their root form. This means we will transform words like <i>fishing, fishes, fish</i> to the root word <i>fish</i>.\n",
    "\n",
    "A class in Python is similar to making a cookie cutter. We want to make a prototype so that every time we use the cookie cutter on a text dataset, the object will have the same shape, characteristics and functions. \n",
    "\n",
    "Let's start by initializing our class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk import PorterStemmer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk.corpus import stopwords\n",
    "from nltk import download\n",
    "\n",
    "class Preprocess():\n",
    "\tdef __init__(self, text, sw=stopwords.words('english'), lower=True, stem = True):\n",
    "\n",
    "\t\tif not (type(text)==pd.core.series.Series):\n",
    "\t\t\ttext = pd.Series(text)\n",
    "\n",
    "\t\tself.text = text\n",
    "\t\tself.sw = sw\n",
    "\t\tself.lower = lower\n",
    "\t\tself.stem = stem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every class in Python should be <i>initialized</i>. Basically, we want to use the function <b>$__init__$</b> (that is two underscrolls preceding and following the word init) to give the object some basic characteristics. In our case, these will be the characteristics for how we want to clean the data.\n",
    "\n",
    "<b>self</b> is an important element of object oriented program. This is the class itself and will be a variable which we can append attributes to. \n",
    "\n",
    "<b>text</b> will be the list of texts we want to clean up. This variable will be mandatory for the user to input when calling the Preprocess class we are creating. \n",
    "\n",
    "<b>sw</b> is a list of stopwords. You'll notice this is equal to a value (stopwords.words('english')). This means by default the stopwords will be imported from the nltk module (a nice natural language tool created by the kind people at UPenn <a>http://www.nltk.org</a>). This means sw is NOT mandatory as it will take a value by default if the user doesn't feed it a value.\n",
    "\n",
    "<b>lower</b> is a Boolean variable by default equal to True. We will use this later on to know whether or not to lowercase the texts.\n",
    "\n",
    "<b>stem</b> is a Boolean variable by default equal to True. We will use this later on to know whether or not to stem the words in the texts.\n",
    "\n",
    "The last lines like <i>self.text = text</i> are pegging attributes to our object. That is to say, in other places within the class, we can refer to these attributes by running <i>class_instance</i>.text. \n",
    "\n",
    "(type(text)==pd.core.series.Series) checks to see if the text is a Pandas series.If it is not, then we'll convert it to one. Pandas is a module which makes it easy to manipulate a list, for example lower casing or returning a dataframe. \n",
    "\n",
    "Let's show this by example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0               Frankly, my dear, I don't give a damn.\n",
      "1         I'm gonna make him an offer he can't refuse.\n",
      "2    You don't understand!  I coulda had class. I c...\n",
      "3    Toto, I've a feeling we're not in Kansas anymore.\n",
      "4                          Here's looking at you, kid.\n",
      "Name: QUOTE, dtype: object\n",
      "True\n",
      "[u'i', u'me', u'my', u'myself', u'we', u'our', u'ours', u'ourselves', u'you', u'your', u'yours', u'yourself', u'yourselves', u'he', u'him', u'his', u'himself', u'she', u'her', u'hers', u'herself', u'it', u'its', u'itself', u'they', u'them', u'their', u'theirs', u'themselves', u'what', u'which', u'who', u'whom', u'this', u'that', u'these', u'those', u'am', u'is', u'are', u'was', u'were', u'be', u'been', u'being', u'have', u'has', u'had', u'having', u'do', u'does', u'did', u'doing', u'a', u'an', u'the', u'and', u'but', u'if', u'or', u'because', u'as', u'until', u'while', u'of', u'at', u'by', u'for', u'with', u'about', u'against', u'between', u'into', u'through', u'during', u'before', u'after', u'above', u'below', u'to', u'from', u'up', u'down', u'in', u'out', u'on', u'off', u'over', u'under', u'again', u'further', u'then', u'once', u'here', u'there', u'when', u'where', u'why', u'how', u'all', u'any', u'both', u'each', u'few', u'more', u'most', u'other', u'some', u'such', u'no', u'nor', u'not', u'only', u'own', u'same', u'so', u'than', u'too', u'very', u's', u't', u'can', u'will', u'just', u'don', u'should', u'now', u'd', u'll', u'm', u'o', u're', u've', u'y', u'ain', u'aren', u'couldn', u'didn', u'doesn', u'hadn', u'hasn', u'haven', u'isn', u'ma', u'mightn', u'mustn', u'needn', u'shan', u'shouldn', u'wasn', u'weren', u'won', u'wouldn']\n"
     ]
    }
   ],
   "source": [
    "docs = Preprocess(df.QUOTE)\n",
    "\n",
    "print(docs.text.head())\n",
    "print(docs.lower)\n",
    "print(docs.sw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we created an <i>instance</i> of the class, then we called their attributes (text I printed out only the top part, or the head, of the dataframe to save space). \n",
    "\n",
    "Now that we have the $__init__$ function, let's define our first <b>method</b>. A method is a function within a class. The first argument must be self, as this variable contains the attributes we defined in the $__init__$ function. First, we'll check if the <i>lower</i> attribute is True; if so, let's lowercase the text. Then, let's split each text into a list of words. This way we can cycle through each word and either i) stem it if self.stem is True ii) remove it if it is a stop word. Similar to any other Python function, we can define functions within functions to make cleaner code. Lastly, we will join each list of words to create a string, and then initialized a TfidfVectorizer object. TfidfVectorizer will allow us to create a dense matrix where each column is a word in our vocabulary, and each row corresponds to a document.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Preprocess():\n",
    "\tdef __init__(self, text, sw=stopwords.words('english'), lower=True, stem = True):\n",
    "\n",
    "\t\tif not (type(text)==pd.core.series.Series):\n",
    "\t\t\ttext = pd.Series(text)\n",
    "\n",
    "\t\tself.text = text\n",
    "\t\tself.sw = sw\n",
    "\t\tself.lower = lower\n",
    "\t\tself.stem = stem\n",
    "\n",
    "\n",
    "\tdef clean_text(self):\n",
    "\t\tdef stem(word_list):\n",
    "\t\t\treturn map(lambda x: PorterStemmer().stem(x), word_list)\n",
    "\n",
    "\t\tdef remove_sw(word_list):\n",
    "\t\t\tkeep = []\n",
    "\t\t\tfor word in word_list:\n",
    "\t\t\t\tif not word in self.sw:\n",
    "\t\t\t\t\tkeep.append(word)\n",
    "\t\t\treturn keep\n",
    "\n",
    "\t\tif self.lower:\n",
    "\t\t\tself.text = self.text.str.lower()\n",
    "\n",
    "\t\tself.text = self.text.apply(lambda x: x.split())\n",
    "\t\t\n",
    "\t\tif self.stem: self.text = self.text.apply(stem)\n",
    "\t\tif self.sw: self.text = self.text.apply(remove_sw)\n",
    "\n",
    "\t\tself.text = self.text.apply(lambda x: ' '.join(x))\n",
    "\t\tself.vectorizer = TfidfVectorizer()\n",
    "\t\tself.df_dense = self.vectorizer.fit_transform(self.text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test it out again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0                      frankly, dear, don't give damn.\n",
      "1                   i'm gonna make offer can't refuse.\n",
      "2    don't understand! coulda class. coulda contend...\n",
      "3                   toto, i'v feel we'r kansa anymore.\n",
      "4                                 here' look you, kid.\n",
      "Name: QUOTE, dtype: object\n",
      "TfidfVectorizer(analyzer=u'word', binary=False, decode_error=u'strict',\n",
      "        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',\n",
      "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
      "        ngram_range=(1, 1), norm=u'l2', preprocessor=None, smooth_idf=True,\n",
      "        stop_words=None, strip_accents=None, sublinear_tf=False,\n",
      "        token_pattern=u'(?u)\\\\b\\\\w\\\\w+\\\\b', tokenizer=None, use_idf=True,\n",
      "        vocabulary=None)\n",
      "  (0, 71)\t0.451149812367\n",
      "  (0, 113)\t0.491651345454\n",
      "  (0, 83)\t0.330886087089\n",
      "  (0, 74)\t0.451149812367\n",
      "  (0, 106)\t0.491651345454\n",
      "  (1, 237)\t0.500815076903\n",
      "  (1, 50)\t0.407581696307\n",
      "  (1, 211)\t0.500815076903\n",
      "  (1, 177)\t0.407581696307\n",
      "  (1, 117)\t0.407581696307\n"
     ]
    }
   ],
   "source": [
    "docs = Preprocess(df.QUOTE)\n",
    "docs.clean_text()\n",
    "\n",
    "print(docs.text.head())\n",
    "print(docs.vectorizer)\n",
    "print(docs.df_dense[0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The <b>df_dense</b> attribute is a dense representation of the TF-IDF matrix (<a>https://en.wikipedia.org/wiki/Tf%E2%80%93idf).</a>. (0, 71) corresponds to the first document,\"Frankly, my dear, I don't give a damn.\", and the 71st word in our vocabulary which is contained in that quote. 0.451149... is the TF-IDF score.\n",
    "\n",
    "We may want to return this in the form of a more readable dataset. So let's create a Pandas dataframe where the columns are the words in our vocabulary after cleaning the text, self.vectorizer.get_feature_names(), and then the values we can get from self.df_dense.toarray(). \n",
    "\n",
    "<b>onehot</b> will be a variable which we can use so the user can either return a matrix with values of 1 if the document contains the word in the vocabulary in the ith position, or the TF-IDF score. I do this in another function to make it easy for the user to return just the array or the dataframe if they choose. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                Text  adrian  again  ahead  \\\n",
      "0                    frankly, dear, don't give damn.       0      0      0   \n",
      "1                 i'm gonna make offer can't refuse.       0      0      0   \n",
      "2  don't understand! coulda class. coulda contend...       0      0      0   \n",
      "3                 toto, i'v feel we'r kansa anymore.       0      0      0   \n",
      "4                               here' look you, kid.       0      0      0   \n",
      "\n",
      "   ain  airplanes  alive  all  alone  alway    ...      win  wire  witness  \\\n",
      "0    0          0      0    0      0      0    ...        0     0        0   \n",
      "1    0          0      0    0      0      0    ...        0     0        0   \n",
      "2    0          0      0    0      0      0    ...        0     0        0   \n",
      "3    0          0      0    0      0      0    ...        0     0        0   \n",
      "4    0          0      0    0      0      0    ...        0     0        0   \n",
      "\n",
      "   word  world  ya  yet  yo  you  youngster  \n",
      "0     0      0   0    0   0    0          0  \n",
      "1     0      0   0    0   0    0          0  \n",
      "2     0      0   0    0   0    0          0  \n",
      "3     0      0   0    0   0    0          0  \n",
      "4     0      0   0    0   0    1          0  \n",
      "\n",
      "[5 rows x 337 columns]\n"
     ]
    }
   ],
   "source": [
    "#################################\n",
    "### Author: Paul Soto \t\t  ###\n",
    "### \t\tpaul.soto@upf.edu ###\n",
    "#\t\t\t\t\t\t\t\t#\n",
    "# This file is a class to #######\n",
    "# clean a series of text   ######\n",
    "# with basic preprocessing     ##\n",
    "#################################\n",
    "import pandas as pd\n",
    "from nltk import PorterStemmer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk.corpus import stopwords\n",
    "from nltk import download\n",
    "\n",
    "# download('stopwords')\n",
    "\n",
    "df = pd.read_csv(\"final_text_db.csv\", encoding='utf-8')\n",
    "\n",
    "class Preprocess():\n",
    "\tdef __init__(self, text, sw=stopwords.words('english'), lower=True, stem = True):\n",
    "\n",
    "\t\tif not (type(text)==pd.core.series.Series):\n",
    "\t\t\ttext = pd.Series(text)\n",
    "\n",
    "\t\tself.text = text\n",
    "\t\tself.sw = sw\n",
    "\t\tself.lower = lower\n",
    "\t\tself.stem = stem\n",
    "\n",
    "\n",
    "\tdef clean_text(self):\n",
    "\t\tdef stem(word_list):\n",
    "\t\t\treturn map(lambda x: PorterStemmer().stem(x), word_list)\n",
    "\n",
    "\t\tdef remove_sw(word_list):\n",
    "\t\t\tkeep = []\n",
    "\t\t\tfor word in word_list:\n",
    "\t\t\t\tif not word in self.sw:\n",
    "\t\t\t\t\tkeep.append(word)\n",
    "\t\t\treturn keep\n",
    "\n",
    "\t\tif self.lower:\n",
    "\t\t\tself.text = self.text.str.lower()\n",
    "\n",
    "\t\tself.text = self.text.apply(lambda x: x.split())\n",
    "\t\t\n",
    "\t\tif self.stem: self.text = self.text.apply(stem)\n",
    "\t\tif self.sw: self.text = self.text.apply(remove_sw)\n",
    "\n",
    "\t\tself.text = self.text.apply(lambda x: ' '.join(x))\n",
    "\t\tself.vectorizer = TfidfVectorizer()\n",
    "\t\tself.df_dense = self.vectorizer.fit_transform(self.text)\n",
    "\n",
    "\tdef array(self, onehot=1):\n",
    "\t\tarray = self.df_dense.toarray().copy()\n",
    "\t\tif onehot:\n",
    "\t\t\tarray[array>0] = 1\n",
    "\t\treturn array\n",
    "\n",
    "\tdef make_df(self,onehot=1):\n",
    "\t\tdf = pd.DataFrame(columns=self.vectorizer.get_feature_names(),\n",
    "\t\t\t\t\t\t\tdata = self.array(onehot))\n",
    "\t\tdf['Text'] = self.text\n",
    "\t\tdf = df[['Text']+list(df.columns[:-1])]\n",
    "\t\treturn df\n",
    "\n",
    "docs = Preprocess(df.QUOTE)\n",
    "docs.clean_text()\n",
    "text_df = docs.make_df(onehot=1)\n",
    "\n",
    "print(text_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many other methods you could add, such as one which lemmatizes the word, gives the part of speech, etc. I'll leave it here for now."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}