Lecture 04 - Character ngram model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    },
    "colab": {
      "name": "Lecture 04 - Character ngram model.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fB7n742QgJCJ"
      },
      "source": [
        "# Generating text with character n-gram models\n",
        "\n",
        "First, we need some training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q1oWxKzggJCL"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sTRYyRPCgfrj"
      },
      "source": [
        "We will use the `gutenberg` module for downloading some training data. You can see a list of open books below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eWSJA3BngJCM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "045529e4-2fe9-45b0-e3d2-2a0c86ff7f4a"
      },
      "source": [
        "import nltk\n",
        "nltk.download('gutenberg')\n",
        "\n",
        "from nltk.corpus import gutenberg\n",
        "print(\"Available books:\", gutenberg.fileids())"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package gutenberg to /root/nltk_data...\n",
            "[nltk_data]   Package gutenberg is already up-to-date!\n",
            "Available books: ['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt', 'bible-kjv.txt', 'blake-poems.txt', 'bryant-stories.txt', 'burgess-busterbrown.txt', 'carroll-alice.txt', 'chesterton-ball.txt', 'chesterton-brown.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'shakespeare-caesar.txt', 'shakespeare-hamlet.txt', 'shakespeare-macbeth.txt', 'whitman-leaves.txt']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T0bZsx2CgJCO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "69ac10a6-3532-4fb1-ea06-aa5f93db0ae0"
      },
      "source": [
        "fileids = gutenberg.fileids()[:3]\n",
        "print(\"Using:\", fileids)\n",
        "\n",
        "raw_text = gutenberg.raw(fileids)\n",
        "raw_text = raw_text.replace('\\n', ' ')\n",
        "data = list(raw_text)\n",
        "print(\"Training data consists of %i characters\" % len(data))"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using: ['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt']\n",
            "Training data consists of 2026385 characters\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2DcuI7i4gsDP"
      },
      "source": [
        "The model is the same as with word n-grams. The difference comes from the training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_HMRNR_bgZf0",
        "outputId": "bf1226ea-a54d-4362-96c7-82cc4ef7935f"
      },
      "source": [
        "!wget -N https://raw.githubusercontent.com/fredrikwahlberg/5LN445/master/ngram.py\n",
        "\n",
        "from ngram import NGramModel"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-09-06 14:34:10--  https://raw.githubusercontent.com/fredrikwahlberg/5LN445/master/ngram.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 8687 (8.5K) [text/plain]\n",
            "Saving to: ‘ngram.py’\n",
            "\n",
            "ngram.py            100%[===================>]   8.48K  --.-KB/s    in 0s      \n",
            "\n",
            "Last-modified header missing -- time-stamps turned off.\n",
            "2021-09-06 14:34:11 (65.0 MB/s) - ‘ngram.py’ saved [8687/8687]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bLOOsaBHgJCO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "baca0972-58df-49fd-8488-380b898d0bbf"
      },
      "source": [
        "model1 = NGramModel(data, 1)\n",
        "print(model1)\n",
        "\n",
        "model2 = NGramModel(data, 2)\n",
        "print(model2)\n",
        "\n",
        "model3 = NGramModel(data, 3)\n",
        "print(model3)\n",
        "\n",
        "model4 = NGramModel(data, 4)\n",
        "print(model4)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1-gram model with 81 unique keys\n",
            "2-gram model with 1542 unique keys\n",
            "3-gram model with 10212 unique keys\n",
            "4-gram model with 40030 unique keys\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wqeOBeH3hfdn",
        "outputId": "49cae387-9d01-4e4d-de29-cd6de9f89238"
      },
      "source": [
        "print(model1.predict_sequence(20))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['v', ' ', ' ', 'e', 'a', 't', 'a', 'e', 'n', 'r', ' ', 's', 'y', 'a', 's', 't', ' ', 'h', 's', ' ']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w0OxXPbjgJCP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aeb2c9ee-a12d-454d-93aa-ab77bb0189dc"
      },
      "source": [
        "print(\"unigram:\", \"\".join(model1.predict_sequence(200)))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "unigram: ddoanut otaate?gtaayweh  im   ea trrtaorh  ef nidnc nhccwet rtwbxocnaiAsgihtn rn,ooa rlen g,ar.hera ntere oe,timeaio n  .v onnet  nosnltsp.,eihcnaiayeea myeiwm Morl  e TlHbet u i  \"odaooh eeytat sM -g\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kv9Hwsn0hU58",
        "outputId": "a0bf38b0-21e2-470e-d6e1-23e31301d714"
      },
      "source": [
        "print(\"bigram:\", \"\".join(model2.predict_sequence(200)))"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "bigram: e oben man bin con d wn fr me trichear t, nold he wherreeystincond m Mre, d tomast iawe or. ise hishatoned ortiod t s dittticest wknckn, wireximpesnil ar. Win f s- tes tourdsero me avareshale whin ted\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OdZy0cX1hRxw",
        "outputId": "a487da5f-eeb3-403e-e660-ec7aa11feecf"
      },
      "source": [
        "print(\"trigram:\", \"\".join(model3.predict_sequence(200)))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "trigram:  he send that give he guance to post donind much washe by to-daught a boaccought, whery uncomanningand nortabsed nothing whind ther so but thave wrompostaill ch theanch as hild to Emmost befordly re t\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lqurP-xrhRz5",
        "outputId": "3ff0b47c-91e1-4ef9-912a-c1af4da406f0"
      },
      "source": [
        "print(\"quadgram:\", \"\".join(model4.predict_sequence(200)))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "quadgram:  a made excuse's right was the Elinor's regants, but or Sir fromorning than howed rathe mome the spothem as to the poings ing, and may the Mary.  Trouse would not_ is it sits of cotted fortain fillevo\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tWPtPeuahYKd"
      },
      "source": [
        ""
      ],
      "execution_count": 10,
      "outputs": []
    }
  ]
}