06_shakespeare_simulator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's try a more interesting Markov model, in which we simulate the linguistic styles of various authors with recognizable styles.  To begin with, we'll train a Markov model with the complete works of William Shakespeare.  Much as in the weather example, training will consist of finding the transition probabilities from one word to the next.  As such, we'll compute prior probabilities representing the relative frequency of words in the body of work, and we'll also compute a gigantic set of transition probabilities from each word to all the other words.  First, we'll read in the Shakespeare text,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import division,print_function\n",
    "\n",
    "import numpy as np\n",
    "import string\n",
    "from collections import Counter\n",
    "import re\n",
    "import json\n",
    "import unicodedata\n",
    "\n",
    "sequence_shakespeare = []\n",
    "file = open('datasets/t8.shakespeare.txt','r')\n",
    "for line in file:\n",
    "    line.strip('\\n')\n",
    "    if line[:2] == '  ':\n",
    "        line_words = re.findall(r\"[\\w']+|[.,!?;]\",line)\n",
    "        line_words = [str(w).lower() for w in line_words if not w.isupper() and not w.isdigit()] \n",
    "\n",
    "        sequence_shakespeare.extend(line_words)\n",
    "        \n",
    "print (' '.join(sequence_shakespeare[:100]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "He goes on an on.  Now, we'll import a model that I made for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0,'./libraries/')\n",
    "from markov_models import FirstOrderMarkovModel\n",
    "\n",
    "mm_shakespeare = FirstOrderMarkovModel(sequence_shakespeare)\n",
    "mm_shakespeare.build_transition_matrices()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last line has built the transition matrices, which are actually not matrices in this implementation, but instead dictionaries that store only entries for which there appears a transition.  For example, 'the' is never followed by another 'the', so it would be a waste to explicitly keep track of a zero probability case.  This is actually true for the vast majority of word pairs, so not keeping a 30000 by 30000 matrix is advantageous.  \n",
    "\n",
    "With this model in hand, we can do interesting things like generate synthetic data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#q = mm_shakespeare.transitions[0]\n",
    "mm_shakespeare.generate_phrase()\n",
    "#mm_shakespeare.transitions[0].values()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feel is right, if not exactly sensible!  These models are great at capturing tone and style, but not so much the meaning.  Another thing that we can do is to use our statistical model to evaluate the probability of new examples.  For example, if I wanted to evaluate how probable it was that Shakespeare generate the phrase "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_string = 'to be or not to be'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I would just evaluate the prior probability on 'to' then multiply that by $P(be|to)$ then the probability of $P(or|be)$ and so on.  In practice we'll use log probabilities to avoid underflow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_like_shakespeare = mm_shakespeare.compute_log_likelihood(test_string,lamda=0.01,unknown_probability=1e-10)\n",
    "print (log_like_shakespeare)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These aren't that interesting on their own.  A better use for these log-likelihoods is as a classification scheme.  If I had another statistical model built upon a corpus of text, I could compute the likelihood for both and decide which writer produced the text.  \n",
    "\n",
    "One contemporary goldmine of idiosyncratic text is the twitter account of Donald J. Trump.  Let us create a model for his tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequence_trump = []\n",
    "file = open('datasets/trump_tweets.json')\n",
    "tweet_list = json.loads(file.read())\n",
    "for t in tweet_list:\n",
    "    tweet = unicodedata.normalize('NFKD',t['text'])\n",
    "    tweet = re.sub(r'\\w+:\\/{2}[\\d\\w-]+(\\.[\\d\\w-]+)*(?:(?:\\/[^\\s/]*))*', '', tweet)\n",
    "    line_words = re.findall(r\"[\\w']+|[.,!?;]\",tweet)\n",
    "    line_words = [str(w).lower() for w in line_words if not w.isdigit()] \n",
    "\n",
    "    sequence_trump.extend(line_words)\n",
    "\n",
    "print (' '.join(sequence_trump[:100]))\n",
    "    \n",
    "mm_trump = FirstOrderMarkovModel(sequence_trump)\n",
    "mm_trump.build_transition_matrices()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can, again, generate data using this model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mm_trump.generate_phrase()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can evaluate the likelihood of to be or not to be for Trump"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_like_trump = mm_trump.compute_log_likelihood(test_string,lamda=0.01,unknown_probability=1e-10)\n",
    "\n",
    "trump_factor = np.exp(log_like_trump - log_like_shakespeare)\n",
    "print(trump_factor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'to be or not to be' is 4 times more likely to be Shakespeare, which is not very strong evidence.  Let's try again with a longer phrase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_string = 'to be or not to be , that is the question'\n",
    "log_like_shakespeare = mm_shakespeare.compute_log_likelihood(test_string,lamda=0.01,unknown_probability=1e-10)\n",
    "log_like_trump = mm_trump.compute_log_likelihood(test_string,lamda=0.01,unknown_probability=1e-10)\n",
    "\n",
    "trump_factor = np.exp(log_like_trump - log_like_shakespeare)\n",
    "print(trump_factor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is much more likely to be Shakespeare, now that we have more data.  Conversely, let's try something that Trump actually said, but was certainly not in the tweet corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_string = \"i moved on her very heavily\"\n",
    "log_like_shakespeare = mm_shakespeare.compute_log_likelihood(test_string,lamda=0.01,unknown_probability=1e-10)\n",
    "log_like_trump = mm_trump.compute_log_likelihood(test_string,lamda=0.01,unknown_probability=1e-10)\n",
    "\n",
    "trump_factor = np.exp(log_like_trump - log_like_shakespeare)\n",
    "print(trump_factor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is much more likely to have been written by Trump.\n",
    "\n",
    "Finally, these models in which words are only dependent on their immediate predecessor are called 'bigram' models.  They aren't particularly good at generating realistic text.  Better results can be had by considering the previous two or more words, albeit with a commensurate increase in cost and tendency to overfit.  Let's see what kind of data a trigram model generates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from markov_models import SecondOrderMarkovModel\n",
    "\n",
    "mm_shakespeare = SecondOrderMarkovModel(sequence_shakespeare)\n",
    "mm_shakespeare.build_transition_matrices()\n",
    "mm_trump = SecondOrderMarkovModel(sequence_trump)\n",
    "mm_trump.build_transition_matrices()\n",
    "\n",
    "shakespeare_or_trump = ['Shakespeare','Trump']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use these to play a game: we'll randomly select from the two models, generate a phrase and try to decide who said it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = [mm_shakespeare,mm_trump]\n",
    "index = np.random.randint(2)\n",
    "models[index].generate_phrase()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(shakespeare_or_trump[index])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}