CS246_Colab_1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CS246_Colab_1.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "p0-YhEpP_Ds-"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kPt5q27L5557"
      },
      "source": [
        "# CS246 - Colab 1\n",
        "## Wordcount in Spark"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p0-YhEpP_Ds-"
      },
      "source": [
        "### Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zsj5WYpR9QId"
      },
      "source": [
        "Let's setup Spark on your Colab environment.  Run the cell below!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "k-qHai2252mI"
      },
      "source": [
        "!pip install pyspark\n",
        "!pip install -U -q PyDrive\n",
        "!apt install openjdk-8-jdk-headless -qq\n",
        "import os\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-CJ71AKe91eh"
      },
      "source": [
        "Now we authenticate a Google Drive client to download the file we will be processing in our Spark job.\n",
        "\n",
        "**Make sure to follow the interactive instructions.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "5K93ABEy9Zlo"
      },
      "source": [
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        "\n",
        "# Authenticate and create the PyDrive client\n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "0orRvrc1-545"
      },
      "source": [
        "id='1SE6k_0YukzGd5wK-E4i6mG83nydlfvSa'\n",
        "downloaded = drive.CreateFile({'id': id})\n",
        "downloaded.GetContentFile('pg100.txt')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qwtlO4_m_LbQ"
      },
      "source": [
        "If you executed the cells above, you should be able to see the file *pg100.txt* under the \"Files\" tab on the left panel."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CRaF2A_j_nC7"
      },
      "source": [
        "### Your task"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ebLNUxP0_8x3"
      },
      "source": [
        "If you run successfully the setup stage, you are ready to work on the *pg100.txt* file which contains a copy of the complete works of Shakespeare.\n",
        "\n",
        "Write a Spark application which outputs the number of words that start with each letter. This means that for every letter we want to count the total number of (non-unique) words that start with a specific letter. In your implementation **ignore the letter case**, i.e., consider all words as lower case. Also, you can ignore all the words **starting** with a non-alphabetic character."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xu-e7Ph2_ruG"
      },
      "source": [
        "from pyspark.sql import *\n",
        "from pyspark.sql.functions import *\n",
        "from pyspark import SparkContext\n",
        "import pandas as pd\n",
        "import regex as re\n",
        "# create the Spark Session\n",
        "spark = SparkSession.builder.getOrCreate()\n",
        "\n",
        "# create the Spark Context\n",
        "sc = spark.sparkContext"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2oHkCpth1HxO"
      },
      "source": [
        "sc.textFile(\"testo.txt\") => RDD[String]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AuAxGFPFB43Y"
      },
      "source": [
        "shakespeare = sc.textFile(\"pg100.txt\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A-SSxDs9FdJa"
      },
      "source": [
        "def clean_line(str):\n",
        "  lowercase = str.lower()\n",
        "  lowercase = re.sub(\"[^0-9a-zA-Z]+\", \" \", lowercase)\n",
        "  return lowercase"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x9P69W0mjucG",
        "outputId": "5653e9ba-9a45-485b-ec0c-539f686914ab"
      },
      "source": [
        "shakespeare_txt = shakespeare.map(clean_line)\\\n",
        ".flatMap(lambda st: st.split(\" \"))\\\n",
        ".filter(lambda x: x != '')\\\n",
        ".map(lambda word: (word[0], 1))\n",
        "\n",
        "count_words = shakespeare_txt.reduceByKey(lambda a,b: (a+b))\n",
        "count_words = count_words.sortBy(lambda x: -x[1])\n",
        "\n",
        "count_words.take(36)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('t', 127781),\n",
              " ('a', 86000),\n",
              " ('s', 75226),\n",
              " ('i', 62420),\n",
              " ('h', 61029),\n",
              " ('w', 60097),\n",
              " ('m', 56252),\n",
              " ('b', 46001),\n",
              " ('o', 43712),\n",
              " ('d', 39173),\n",
              " ('f', 37186),\n",
              " ('c', 34983),\n",
              " ('l', 32389),\n",
              " ('p', 28059),\n",
              " ('n', 27313),\n",
              " ('y', 25926),\n",
              " ('g', 21167),\n",
              " ('e', 20409),\n",
              " ('r', 15234),\n",
              " ('k', 9535),\n",
              " ('u', 9230),\n",
              " ('v', 5802),\n",
              " ('j', 3372),\n",
              " ('q', 2388),\n",
              " ('1', 932),\n",
              " ('2', 330),\n",
              " ('3', 84),\n",
              " ('z', 79),\n",
              " ('4', 67),\n",
              " ('5', 43),\n",
              " ('9', 34),\n",
              " ('6', 28),\n",
              " ('7', 24),\n",
              " ('8', 23),\n",
              " ('x', 14),\n",
              " ('0', 10)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    }
  ]
}