Part-Of-Speech Tagging.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stanford Part-Of-Speech tagger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load our dataset and take a peek at some of the values in it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load dataset:\n",
    "vuelos = pd.read_csv('data/vuelos.csv', index_col=0)\n",
    "with pd.option_context('max_colwidth', 800):\n",
    "    print(vuelos.loc[:40:5][['label']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To interface with the Stanford tagger we could use the `StanforPOSTagger` inside the `nltk.tag.stanford` module, then we create an object passing in both our language-specific model as well as the tagger `.jar` we previously downloaded from Stanford's website.  \n",
    "\n",
    "Then, as a quick test we tag a spanish sentence to see what is it that we get back from the tagger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tag.stanford import StanfordPOSTagger\n",
    "\n",
    "spanish_postagger = StanfordPOSTagger('stanford-models/spanish.tagger', \n",
    "                                      'stanford-models/stanford-postagger.jar')\n",
    "\n",
    "tags = spanish_postagger.tag('Amo el canto del cenzontle, pájaro de cuatrocientas voces.'.split()) \n",
    "print(tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing to note is the fact that the tagger takes in lists of strings, not a full sentence, that is why we need to split our sentence before passing it in. A second thing to note is that we get back of tuples; where the first element of each tuple is the token and the second is the POS tag assigned to said token. The POS tags are [explained here](https://nlp.stanford.edu/software/spanish-faq.html) and I have made a dictionary for easy lookups.\n",
    "\n",
    "We can inspect the tokens a bit more:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"aux/spanish-tags.json\", \"r\") as r:\n",
    "    spanish_tags = json.load(r)\n",
    "    \n",
    "for token, tag in tags[:10]:\n",
    "    print(f\"{token:15} -> {spanish_tags[tag]['description']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specific tokenisation  \n",
    "\n",
    "As you may imagine, using `split` to tokenise our text is not the best idea; it is almost certainly better to create our own function, taking into consideration the kind of text that we are going to process. The function aboce uses the `TweetTokenizer`, and takes flag emojis into consideration. As a final touch, it also returns the position of each one of the returned tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import TweetTokenizer\n",
    "\n",
    "TWEET_TOKENIZER = TweetTokenizer()\n",
    "\n",
    "# This function exists in vuelax.tokenisation in this same repository\n",
    "def index_emoji_tokenize(string, return_flags=False):\n",
    "    flag = ''\n",
    "    ix = 0\n",
    "    tokens, positions = [], []\n",
    "    for t in TWEET_TOKENIZER.tokenize(string):\n",
    "        ix = string.find(t, ix)\n",
    "        if len(t) == 1 and ord(t) >= 127462:  # this is the code for 🇦\n",
    "            if not return_flags: continue\n",
    "            if flag:\n",
    "                tokens.append(flag + t)\n",
    "                positions.append(ix - 1)\n",
    "                flag = ''\n",
    "            else:\n",
    "                flag = t\n",
    "        else:\n",
    "            tokens.append(t)\n",
    "            positions.append(ix)\n",
    "        ix = +1\n",
    "    return tokens, positions\n",
    "\n",
    "\n",
    "        \n",
    "\n",
    "label = vuelos.iloc[75]['label']\n",
    "print(label)\n",
    "print()\n",
    "tokens, positions = index_emoji_tokenize(label, return_flags=True)\n",
    "print(tokens)\n",
    "print(positions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}