Saving and reloading sentence tokenized corpora .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a jupyter notebook which details how one can tokenize a file from COCA and then store it in an HDF5 file to call up later.  This addresses the issue Graeme had where he thought that each time he would search for a word, he would have to tokenize the whole COCA corpus anew."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pandas import Series, DataFrame\n",
    "import nltk\n",
    "from nltk.tokenize import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from pandas import HDFStore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.tokenize import sent_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f = open('COCA/text_newspaper_lsp/w_news_1990.txt')\n",
    "text = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "##3000001 <p> He is trying to make the best of it , but the days have seemed to pass like months f\n"
     ]
    }
   ],
   "source": [
    "print text[0:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "207190"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent_tokenize_list = sent_tokenize(text)\n",
    "len(sent_tokenize_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\\r\\n##3000001 <p> He is trying to make the best of it , but the days have seemed to pass like months for Lou Piniella .', 'Some days , he visits relatives .', 'Others , he spends the afternoon fishing .', 'Mostly , he waits .', 'It has been like this since the first week of February , when Piniella and his family arrived at their beach home in Reddington Shores , Fla. , and waited for baseball to begin .']\n"
     ]
    }
   ],
   "source": [
    "print sent_tokenize_list[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sents = pd.Series(sent_tokenize_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    \\r\\n##3000001 <p> He is trying to make the bes...\n",
       "1                    Some days , he visits relatives .\n",
       "dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents[0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Need to open a \"store\" so that you can save it into the HDF5 format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "store = HDFStore('COCA_store.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "store['w_news_1990'] = sents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that one HDF5 container can hold many files, so we can also store the next chunk of COCA there too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f2 = open('COCA/text_newspaper_lsp/w_news_1991.txt')\n",
    "text2 = f2.read()\n",
    "sent_tokenize_list2 = sent_tokenize(text2)\n",
    "sents2 = pd.Series(sent_tokenize_list2)\n",
    "store['w_news_1991'] = sents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see what is in the store, just use 'store'.  When done, it is good practice to close the store.  Note you can see this file in whatever directory you are in, here it is called \"COCA_store.h5\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<class 'pandas.io.pytables.HDFStore'>\n",
       "File path: COCA_store.h5\n",
       "/w_news_1990            series       (shape->[1])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to open the file, use the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "news1990_sents = pd.read_hdf('store.h5', 'w_news_1990')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    \\r\\n##3000001 <p> He is trying to make the bes...\n",
       "1                    Some days , he visits relatives .\n",
       "2           Others , he spends the afternoon fishing .\n",
       "3                                  Mostly , he waits .\n",
       "4    It has been like this since the first week of ...\n",
       "dtype: object"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "news1990_sents[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now everytime you need to access the tokenized sentences from COCA, you just have to load them from the HDF5 file---much much quicker!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}