FP_growth_Market_basket_analysis_(eng_newspapers).ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/LorenzoPolli/market-basket-analysis/blob/main/FP_growth_Market_basket_analysis_(eng_newspapers).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dzPd2cVi4h0p"
      },
      "source": [
        "# Market-basket analysis\n",
        "\n",
        "The task is to implement a system finding frequent itemsets (aka market-basket analysis), analyzing the «Old Newspapers» dataset published on Kaggle and released under the public domain license (CC0)."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "> Project authors: Mathias Cardarello Fierro & Lorenzo Polli\n",
        "\n",
        "Algorithms for Massive Data\n",
        "\n",
        "\n",
        "\n",
        "*Università degli Studi di Milano*\n",
        "\n",
        "\n",
        "15-Dec-2022\n"
      ],
      "metadata": {
        "id": "dFn-9Uw78wzk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **FP growth Algorithm**"
      ],
      "metadata": {
        "id": "IJ15K8NWC6-M"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NTzg-Ca1_v6x"
      },
      "source": [
        "### **1. Setup and data import**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BjsA1sbWTf2e"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "# Download the dataset containing old newspapers\n",
        "import os\n",
        "\n",
        "os.environ[\"KAGGLE_USERNAME\"] = \"mathiascardarello\"\n",
        "os.environ[\"KAGGLE_KEY\"] = \"89f16dcdf267d017756e3a2e5cece19a\"\n",
        "!pip install kaggle --upgrade\n",
        "!kaggle datasets download alvations/old-newspapers --unzip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EPU0IrIShQOW"
      },
      "source": [
        "#### **1.1 Setting up PySpark and Spark NLP**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QGghMt_OjppQ"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!wget http://setup.johnsnowlabs.com/colab.sh -O - | bash"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LzR3wOaKn6Fh",
        "outputId": "0b1ee6a7-f05e-4275-8a8f-b1eb830ecb85"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Spark NLP version 4.2.4\n",
            "Apache Spark version: 3.2.1\n"
          ]
        }
      ],
      "source": [
        "import sparknlp\n",
        "\n",
        "spark = sparknlp.start()\n",
        "\n",
        "print(\"Spark NLP version\", sparknlp.version())\n",
        "print(\"Apache Spark version:\", spark.version)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "id": "RZ8MxGvNo-7k",
        "outputId": "3c916ae3-4803-4e52-cf0b-fbe5eda05ea9"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pyspark.sql.session.SparkSession at 0x7f48b4ce0df0>"
            ],
            "text/html": [
              "\n",
              "            <div>\n",
              "                <p><b>SparkSession - in-memory</b></p>\n",
              "                \n",
              "        <div>\n",
              "            <p><b>SparkContext</b></p>\n",
              "\n",
              "            <p><a href=\"http://f42ab2e8299e:4040\">Spark UI</a></p>\n",
              "\n",
              "            <dl>\n",
              "              <dt>Version</dt>\n",
              "                <dd><code>v3.2.1</code></dd>\n",
              "              <dt>Master</dt>\n",
              "                <dd><code>local[*]</code></dd>\n",
              "              <dt>AppName</dt>\n",
              "                <dd><code>Spark NLP</code></dd>\n",
              "            </dl>\n",
              "        </div>\n",
              "        \n",
              "            </div>\n",
              "        "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "spark"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t8aKsVAreVlH"
      },
      "source": [
        "#### **1.2 Import the dataset**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U7gV2qM6rco4",
        "outputId": "039b0f63-a028-4b5c-bdb1-99a0ad56eb4c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------+------------+----------+--------------------+\n",
            "|Language|      Source|      Date|                Text|\n",
            "+--------+------------+----------+--------------------+\n",
            "| English| latimes.com|2012/04/29|He wasn't home al...|\n",
            "| English|stltoday.com|2011/07/10|The St. Louis pla...|\n",
            "| English|   freep.com|2012/05/07|WSU's plans quick...|\n",
            "| English|      nj.com|2011/02/05|The Alaimo Group ...|\n",
            "| English|  sacbee.com|2011/10/02|And when it's oft...|\n",
            "+--------+------------+----------+--------------------+\n",
            "only showing top 5 rows\n",
            "\n",
            "CPU times: user 337 ms, sys: 34.6 ms, total: 371 ms\n",
            "Wall time: 59 s\n"
          ]
        }
      ],
      "source": [
        "# Import the dataset and display only rows where the language is English\n",
        "\n",
        "%%time\n",
        "\n",
        "df = spark.read.csv('old-newspaper.tsv', sep='\\t', header=True)\n",
        "df = df.filter(\"Language == 'English'\")\n",
        "df.show(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E1zwWG0qlP70",
        "outputId": "5ca36578-2b43-4a28-e837-67a12cf33810"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------------+------+\n",
            "|          Source| count|\n",
            "+----------------+------+\n",
            "|   cleveland.com|152716|\n",
            "|          nj.com|125230|\n",
            "|    stltoday.com|120632|\n",
            "|  oregonlive.com|103496|\n",
            "|     latimes.com| 60637|\n",
            "|   azcentral.com| 42693|\n",
            "|      sfgate.com| 42121|\n",
            "|baltimoresun.com| 39994|\n",
            "|       freep.com| 36469|\n",
            "| startribune.com| 31821|\n",
            "+----------------+------+\n",
            "only showing top 10 rows\n",
            "\n",
            "CPU times: user 373 ms, sys: 39.4 ms, total: 412 ms\n",
            "Wall time: 1min 10s\n"
          ]
        }
      ],
      "source": [
        "# Group by source\n",
        "\n",
        "%%time\n",
        "\n",
        "from pyspark.sql.functions import desc\n",
        "df.groupby(\"Source\").count().sort(desc(\"count\")).show(10)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Check whether there are NULL values or not\n",
        "\n",
        "%%time\n",
        "import pyspark.sql.functions as F\n",
        "\n",
        "df.select([F.count(F.when(F.isnull(c), c)).alias(c) for c in df.columns]).show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EK7Few75C__u",
        "outputId": "2e1b6728-903f-4c16-e7d8-e5efb615380f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------+------+----+----+\n",
            "|Language|Source|Date|Text|\n",
            "+--------+------+----+----+\n",
            "|       0|     0|   0|   0|\n",
            "+--------+------+----+----+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HfDgujX_eN_A"
      },
      "source": [
        "### **2. Pre-processing**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AkTW64M4ldrG"
      },
      "outputs": [],
      "source": [
        "from pyspark.ml import Pipeline\n",
        "import pyspark.sql.types as T\n",
        "from typing import List\n",
        "from sparknlp.annotator import *\n",
        "from sparknlp.base import *\n",
        "from sparknlp.common import *\n",
        "from sparknlp.pretrained import PretrainedPipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MpIZhnA1us_J",
        "outputId": "34e27aef-db2a-4295-9aa3-3e0e0e165ab2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "stopwords_en download started this may take some time.\n",
            "Approximate size to download 2.9 KB\n",
            "[OK!]\n",
            "lemma_antbnc download started this may take some time.\n",
            "Approximate size to download 907.6 KB\n",
            "[OK!]\n"
          ]
        }
      ],
      "source": [
        "# Initialize the annotators\n",
        "document = DocumentAssembler()\\\n",
        "    .setInputCol(\"Text\")\\\n",
        "    .setOutputCol(\"document\")\\\n",
        "    .setCleanupMode(\"shrink\") # remove new lines and tabs, plus merging multiple spaces and blank lines to a single space\n",
        "\n",
        "sentence = SentenceDetector()\\\n",
        "    .setInputCols(['document'])\\\n",
        "    .setOutputCol('sentence')\n",
        "\n",
        "token = Tokenizer()\\\n",
        "    .setInputCols(['sentence'])\\\n",
        "    .setOutputCol('token')\n",
        "\n",
        "normalizer = Normalizer()\\\n",
        "    .setInputCols([\"token\"])\\\n",
        "    .setOutputCol(\"normalized\")\\\n",
        "    .setLowercase(True)\\\n",
        "    .setCleanupPatterns([\"\"\"[^\\w\\d\\s]\"\"\"]) # remove punctuations (keep alphanumeric chars)\n",
        "\n",
        "stop_words = StopWordsCleaner.pretrained('stopwords_en', 'en')\\\n",
        "    .setInputCols([\"normalized\"])\\\n",
        "    .setOutputCol(\"cleanTokens\")\\\n",
        "    .setCaseSensitive(False)\n",
        "\n",
        "lemmatizer = LemmatizerModel.pretrained()\\\n",
        "    .setInputCols([\"cleanTokens\"])\\\n",
        "    .setOutputCol(\"lemma\")\n",
        "\n",
        "prediction_pipeline = Pipeline(stages = [document, sentence, token, normalizer, stop_words, lemmatizer])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GF5kjlwSvBuj"
      },
      "outputs": [],
      "source": [
        "clean_df = prediction_pipeline.fit(df).transform(df)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ePOvOAw2Gn03",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e0aa8940-3bf3-4cab-a920-20b5e20e6d09"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------------------+--------------------+\n",
            "|                Text|              Basket|\n",
            "+--------------------+--------------------+\n",
            "|He wasn't home al...|[wasnt, home, app...|\n",
            "|The St. Louis pla...|[st, louis, plant...|\n",
            "|WSU's plans quick...|[wsus, plan, quic...|\n",
            "|The Alaimo Group ...|[alaimo, group, m...|\n",
            "|And when it's oft...|[difficult, predi...|\n",
            "|There was a certa...|[amount, scoff, y...|\n",
            "|14915 Charlevoix,...|[14915, charlevoi...|\n",
            "|\"\"\"It’s just anot...|[long, line, fail...|\n",
            "|But time and agai...|[time, report, su...|\n",
            "|I was just trying...|[hit, hard, somep...|\n",
            "|MHTA President an...|[mhta, president,...|\n",
            "|\"\"\"The absurdity ...|[absurdity, attem...|\n",
            "|\"GM labor relatio...|[gm, labor, relat...|\n",
            "|Here is why Wandr...|[wandry, matter, ...|\n",
            "|\"\"\"Cheap,\"\" he sa...|[cheap, hit, hard...|\n",
            "|Andrade's childre...|[andrades, child,...|\n",
            "|\"\"\"Let your hair ...|              [hair]|\n",
            "|Born on April 15,...|[bear, april, 15,...|\n",
            "|House Minority Le...|[house, minority,...|\n",
            "|The first is the ...|[love, love, mone...|\n",
            "+--------------------+--------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Add a column where each row corresponds to a different basket of items \n",
        "clean_df = clean_df.withColumn(\"Basket\", clean_df.lemma.result) \n",
        "clean_df.select(['Text','Basket']).show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2eWYkD_J75wJ"
      },
      "source": [
        "Note: in the last row, item \"love\" is repeated twice, and this is not good for further algorithms application."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "urP8MzWUGNOK"
      },
      "outputs": [],
      "source": [
        "# Remove duplicates inside baskets\n",
        "remove_duplicates = F.udf(lambda x: list(set(x)))\n",
        "clean_df = clean_df.withColumn(\"BasketNoDup\", remove_duplicates(F.col(\"Basket\")))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bxopvkmDvMlf"
      },
      "source": [
        "### **3. Exploratory Data Analysis**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "malWuym2AswU"
      },
      "outputs": [],
      "source": [
        "clean_df.createOrReplaceTempView(\"df_view\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BtmsomlG_Ty3",
        "outputId": "2731996b-1ddf-42f8-ca33-7e54a795b3b5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+\n",
            "|total_rows|\n",
            "+----------+\n",
            "|   1010092|\n",
            "+----------+\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Total number of articles\n",
        "total_rows = spark.sql(\"\"\"SELECT COUNT(DISTINCT Text) AS total_rows \n",
        "                        FROM df_view\"\"\")\n",
        "total_rows.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **4. Market-Basket Analysis**"
      ],
      "metadata": {
        "id": "YTtE4SU8ELw6"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zQtBumzsvMwI"
      },
      "source": [
        "#### **4.1 FPGrowth Algorithm**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yzsQAUFp8Rc-"
      },
      "outputs": [],
      "source": [
        "#Frequent Pattern Growth – FP Growth is a method of mining frequent itemsets using support, lift, and confidence.\n",
        "from pyspark.ml.fpm import FPGrowth"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F1NUv_r3DyLG"
      },
      "outputs": [],
      "source": [
        "# Create a new object. It contains just text and baskets with no duplications inside\n",
        "baskets = spark.sql(\"SELECT Text, SPLIT(BasketNoDup,',') AS BasketNoDup FROM df_view\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B81QI5bqrRl_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e080e11c-f675-47f1-bdf7-09b98981eaa8"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "50342"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "# Obtain a sample that is 5% the size of the original dataset\n",
        "sample = baskets.sample(False, 0.05, 10)\n",
        "sample.count()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DkBjH1vkY_66"
      },
      "outputs": [],
      "source": [
        "fp_growth = FPGrowth(itemsCol=\"BasketNoDup\", minSupport=0.006, minConfidence=0.006)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HLSjsZhGbC-g"
      },
      "outputs": [],
      "source": [
        "# Apply Frequent Pattern Growth algorithm to the sample\n",
        "model = fp_growth.fit(sample)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **4.2 Results**"
      ],
      "metadata": {
        "id": "HTAH-UQ0LCaM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### **4.2.1 Experiment #1**"
      ],
      "metadata": {
        "id": "OJi9r9gfMGMJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "FREQUENT ITEMSETS"
      ],
      "metadata": {
        "id": "d02uQdGOP57D"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display frequent itemsets, with respect to parameters minSupport and minConfidence\n",
        "freq_itemsets = model.freqItemsets\n",
        "freq_itemsets.sort(freq_itemsets.freq.desc()).show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kqDHllnw73kM",
        "outputId": "bd7340c3-f326-4218-dee2-f77e85e4d881"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+----+\n",
            "|     items|freq|\n",
            "+----------+----+\n",
            "|   [ year]|3884|\n",
            "|   [ make]|3655|\n",
            "|  [ state]|2760|\n",
            "|   [ time]|2340|\n",
            "| [ people]|2010|\n",
            "|   [ game]|1930|\n",
            "|[ include]|1837|\n",
            "|   [ city]|1774|\n",
            "|    [ day]|1768|\n",
            "|   [ back]|1731|\n",
            "|   [ play]|1730|\n",
            "|   [ home]|1513|\n",
            "|   [ team]|1497|\n",
            "|    [ run]|1441|\n",
            "|   [ week]|1430|\n",
            "|   [ work]|1406|\n",
            "| [ county]|1378|\n",
            "|   [ find]|1370|\n",
            "|   [ good]|1343|\n",
            "|   [ show]|1322|\n",
            "+----------+----+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Display frequent itemsets, given the configured minSupport and minConfidence parameters\n",
        "freq_itemsets = model.freqItemsets\n",
        "freq_itemsets.sort(freq_itemsets.freq.asc()).show(5)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F-7rX6bLOX1L",
        "outputId": "597ddb1e-c7f2-4bb5-dc69-ec8c34eb1ab8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-----------+----+\n",
            "|      items|freq|\n",
            "+-----------+----+\n",
            "|   [ mayor]| 304|\n",
            "|[ hospital]| 304|\n",
            "| [ current]| 306|\n",
            "| [[company]| 306|\n",
            "|[ election]| 306|\n",
            "+-----------+----+\n",
            "only showing top 5 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Total number of \"frequent\" items returned\n",
        "freq_itemsets.count()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o61VTT9zOJVA",
        "outputId": "4853c878-a85f-4fa8-d0c0-0a1ce05ce1aa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "436"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: among words with less frequency, there are some not-so-general words (i.e. mayor, hospital, election)."
      ],
      "metadata": {
        "id": "6xRtziF0Ottu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display frequent pairs of items\n",
        "freq_itemsets.where(F.size(F.col(\"items\"))>1).sort(freq_itemsets.freq.desc()).show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Fq6HKRI19y0x",
        "outputId": "42868436-6d4b-463e-a669-2848a0672591"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-----------------+----+\n",
            "|            items|freq|\n",
            "+-----------------+----+\n",
            "|   [ play,  game]| 397|\n",
            "|    [ louis,  st]| 383|\n",
            "|   [ make,  year]| 381|\n",
            "|    [ ago,  year]| 377|\n",
            "|  [ state,  year]| 369|\n",
            "|   [ time,  year]| 338|\n",
            "|[ million,  year]| 315|\n",
            "| [ season,  game]| 308|\n",
            "+-----------------+----+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "ASSOCIATION RULES"
      ],
      "metadata": {
        "id": "RtCTXkmXP-eU"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wQgAfhsld53K"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "# Display generated association rules\n",
        "association_rules = model.associationRules "
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Order by CONFIDENCE\n",
        "association_rules.sort(association_rules.confidence.desc()).show(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mUL8nE_IId12",
        "outputId": "443e2909-a4ce-42b8-a61c-16ff23c61029"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|antecedent|consequent|         confidence|              lift|             support|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|  [ louis]|     [ st]| 0.8362445414847162| 46.67208725878445|0.007607961543045...|\n",
            "|    [ ago]|   [ year]| 0.5974643423137876|7.7439623894852465|0.007488776766914306|\n",
            "|     [ st]|  [ louis]| 0.4246119733924612| 46.67208725878446|0.007607961543045...|\n",
            "|[ million]|   [ year]| 0.2830188679245283|3.6683151001690533|0.006257200746891263|\n",
            "| [ season]|   [ game]|0.23692307692307693| 6.179886807493025|0.006118151841404791|\n",
            "|   [ play]|   [ game]|0.22947976878612716| 5.985736020845188|0.007886059354018513|\n",
            "|   [ game]|   [ play]|0.20569948186528497| 5.985736020845188|0.007886059354018513|\n",
            "|   [ game]| [ season]|0.15958549222797927| 6.179886807493025|0.006118151841404791|\n",
            "|   [ time]|   [ year]|0.14444444444444443|1.8721993363085019|0.006714075722061102|\n",
            "|  [ state]|   [ year]|0.13369565217391305|1.7328801549276855|0.007329863732072623|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Order by LIFT (desc) - Look for highly dependent items (LIFT >> 1)\n",
        "association_rules.sort(association_rules.lift.desc()).show(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E0uUnxIKJ3ib",
        "outputId": "bef18b51-e1d6-44f0-d30d-406ef6a46d19"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|antecedent|consequent|         confidence|              lift|             support|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|     [ st]|  [ louis]| 0.4246119733924612| 46.67208725878446|0.007607961543045...|\n",
            "|  [ louis]|     [ st]| 0.8362445414847162| 46.67208725878445|0.007607961543045...|\n",
            "|    [ ago]|   [ year]| 0.5974643423137876|7.7439623894852465|0.007488776766914306|\n",
            "|   [ year]|    [ ago]| 0.0970648815653965| 7.743962389485246|0.007488776766914306|\n",
            "|   [ game]| [ season]|0.15958549222797927| 6.179886807493025|0.006118151841404791|\n",
            "| [ season]|   [ game]|0.23692307692307693| 6.179886807493025|0.006118151841404791|\n",
            "|   [ play]|   [ game]|0.22947976878612716| 5.985736020845188|0.007886059354018513|\n",
            "|   [ game]|   [ play]|0.20569948186528497| 5.985736020845188|0.007886059354018513|\n",
            "|[ million]|   [ year]| 0.2830188679245283|3.6683151001690533|0.006257200746891263|\n",
            "|   [ year]|[ million]|0.08110195674562307|3.6683151001690533|0.006257200746891263|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: pairs under the lift columns have the same value."
      ],
      "metadata": {
        "id": "1JhpLZiNKjlw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Order by LIFT (asc) - Look for perfect substitutes items (LIFT << 1)\n",
        "association_rules.sort(association_rules.lift.asc()).show(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cU8Sf4ZlJgdS",
        "outputId": "c24bc8ea-4712-4750-942d-99c03ffd8113"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|antecedent|consequent|         confidence|              lift|             support|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|   [ make]|   [ year]|0.10424076607387141|1.3511041827216361|0.007568233284335147|\n",
            "|   [ year]|   [ make]|0.09809474768280124|1.3511041827216363|0.007568233284335147|\n",
            "|  [ state]|   [ year]|0.13369565217391305|1.7328801549276855|0.007329863732072623|\n",
            "|   [ year]|  [ state]|0.09500514933058703|1.7328801549276855|0.007329863732072623|\n",
            "|   [ time]|   [ year]|0.14444444444444443|1.8721993363085019|0.006714075722061102|\n",
            "|   [ year]|   [ time]|0.08702368692070031| 1.872199336308502|0.006714075722061102|\n",
            "|   [ year]|[ million]|0.08110195674562307|3.6683151001690533|0.006257200746891263|\n",
            "|[ million]|   [ year]| 0.2830188679245283|3.6683151001690533|0.006257200746891263|\n",
            "|   [ game]|   [ play]|0.20569948186528497| 5.985736020845188|0.007886059354018513|\n",
            "|   [ play]|   [ game]|0.22947976878612716| 5.985736020845188|0.007886059354018513|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: since the minimum value under the lift column is greater than 1, no perfect substitutes items have been returned."
      ],
      "metadata": {
        "id": "SNsc2a3FKxAC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Here, the most frequent pairs of items are also displayed \n",
        "association_rules.sort(association_rules.support.desc()).show(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j99YtAv9Le5E",
        "outputId": "efeb887f-8dc3-4404-e51c-73cc82eed056"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|antecedent|consequent|         confidence|              lift|             support|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|   [ game]|   [ play]|0.20569948186528497| 5.985736020845188|0.007886059354018513|\n",
            "|   [ play]|   [ game]|0.22947976878612716| 5.985736020845188|0.007886059354018513|\n",
            "|     [ st]|  [ louis]| 0.4246119733924612| 46.67208725878446|0.007607961543045...|\n",
            "|  [ louis]|     [ st]| 0.8362445414847162| 46.67208725878445|0.007607961543045...|\n",
            "|   [ year]|   [ make]|0.09809474768280124|1.3511041827216363|0.007568233284335147|\n",
            "|   [ make]|   [ year]|0.10424076607387141|1.3511041827216361|0.007568233284335147|\n",
            "|   [ year]|    [ ago]| 0.0970648815653965| 7.743962389485246|0.007488776766914306|\n",
            "|    [ ago]|   [ year]| 0.5974643423137876|7.7439623894852465|0.007488776766914306|\n",
            "|  [ state]|   [ year]|0.13369565217391305|1.7328801549276855|0.007329863732072623|\n",
            "|   [ year]|  [ state]|0.09500514933058703|1.7328801549276855|0.007329863732072623|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "PREDICTIONS"
      ],
      "metadata": {
        "id": "wrQrqjAHQDYk"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z3AVUvptd8KO",
        "outputId": "ff9d9549-d970-49f6-deaa-0bb2a33f99ea"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+\n",
            "|Text                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |BasketNoDup                                                                                                                                                                                                                                                                                                    |prediction                            |\n",
            "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+\n",
            "|But time and again in the report, Sullivan called on CPS to correct problems to improve employee accountability, saying, for example, that measures to keep employees from submitting fraudulent invoices or to block employees from accessing inappropriate websites were not in place.                                                                                                                                                                                                          |[[employee,  improve,  block,  place,  sullivan,  correct,  report,  accountability,  websites,  cps,  inappropriate,  time,  call,  problem,  submit,  fraudulent,  measure,  invoice,  access]]                                                                                                              |[ year]                               |\n",
            "|I was just trying to hit it hard someplace, said Rizzo, who hit the pitch to the opposite field in left-center. Im just up there trying to make good contact.                                                                                                                                                                                                                                                                                                                                     |[[rizzo,  im,  field,  opposite,  pitch,  make,  good,  hit,  someplace,  leftcenter,  contact,  hard]]                                                                                                                                                                                                        |[ year]                               |\n",
            "|Andrade's children, Erin and son Patrick, are adults, and he said he was intrigued by the opportunity to play a mentoring role in the life of a high school student.                                                                                                                                                                                                                                                                                                                              |[[school,  son,  student,  mentore,  child,  andrades,  opportunity,  adult,  play,  role,  intrigue,  patrick,  life,  high,  erin]]                                                                                                                                                                          |[ game]                               |\n",
            "|The new law, which takes effect immediately, requires bareheaded riders 21 and older to carry at least $20,000 in additional medical insurance. Riders also must have passed a motorcycle safety course or have had their motorcycle endorsement for at least two years. These safeguards are woefully inadequate, given the increased risks and medical costs that come with riding without a helmet.                                                                                            |[[additional,  bareheaded,  require,  endorsement,  insurance,  pass,  effect,  carry,  immediately,  woefully,  21,  cost,  law,  old,  inadequate,  safety,  20000,  year,  helmet,  safeguard,  risk,  rider,  medical,  ride,  motorcycle,  increase,  take]]                                              |[ make,  state,  time,  million,  ago]|\n",
            "|Laura Taiby says she last saw her daughter, Laudan Taiby, as she was leaving their home in the 6700 block of Southwest 14th Avenue in the Hillsdale area about 5 p.m. Monday.                                                                                                                                                                                                                                                                                                                     |[[hillsdale,  5,  laura,  block,  14th,  6700,  home,  taiby,  leave,  laudan,  southwest,  area,  monday,  pm,  daughter,  avenue]]                                                                                                                                                                           |[]                                    |\n",
            "|\"Context: Last summer, when a panel of growers and winemakers at Foothill Grape Day in Plymouth got around to talking of barbera's role at the table, Justin Boeger remarked: \"\"Barbera is what I bring to dinner when I don't know the menu. It's versatile.\"\" Other than that, any Italian dish enhanced with tomatoes is traditionally paired with barbera.\"                                                                                                                                   |[[panel,  plymouth,  barbera,  justin,  foothill,  pair,  boeger,  dont,  tomato,  grape,  dinner,  menu,  italian,  bring,  day,  summer,  remark,  barberas,  versatile,  enhance,  traditionally,  talk,  winemaker,  grower,  context,  role,  table,  dish]]                                              |[]                                    |\n",
            "|The U.S.-led coalition routinely reports each time an American or other foreign soldier is killed by an Afghan in uniform, but the military is under-reporting the number of overall attacks. The Associated Press reported earlier this month that the International Security Assistance Force, or ISAF, does not report attacks in which the Afghan wounds or misses his U.S. or allied target. It also doesn't report the wounding of troops who were attacked alongside those who were killed.|[[afghan,  wounding,  international,  isaf,  foreign,  month,  target,  routinely,  underreporting,  report,  assistance,  force,  usled,  wounds,  attack,  miss,  ally,  troop,  number,  military,  press,  kill,  uniform,  time,  american,  alongside,  security,  soldier,  early,  coalition,  doesnt]]|[ year]                               |\n",
            "|\"\"\"They're supposed to be all checked out,\"\" said the father of the teenage girl involved in the incident. \"\"Something is not right.\"\"\"                                                                                                                                                                                                                                                                                                                                                           |[[suppose,  girl,  involve,  theyre,  teenage,  incident,  check,  father]]                                                                                                                                                                                                                                    |[]                                    |\n",
            "|While most of the wines are from Napa Valley and elsewhere in California, the international selections round out the list nicely. On any given night, 26 wines are offered by the glass (plus 17 dessert offerings), with a good selection of half bottles.                                                                                                                                                                                                                                       |[[17,  bottle,  international,  half,  selection,  round,  napa,  dessert,  offerings,  good,  list,  26,  nicely,  glass,  wine,  valley,  night,  offer,  california]]                                                                                                                                       |[]                                    |\n",
            "|As the officers and the St. Andres sat in a row behind eight defense lawyers this morning, about a dozen family and friends of the officers watched from the audience.                                                                                                                                                                                                                                                                                                                            |[[family,  andres,  row,  officer,  sit,  dozen,  defense,  friend,  morning,  lawyer,  audience,  st,  watch]]                                                                                                                                                                                                |[ louis]                              |\n",
            "|Economists expect the economy added 155,000 net jobs, the seventh straight month of solid hiring. The unemployment rate is expected stay for the second month at 8.5 percent, near a three-year low.                                                                                                                                                                                                                                                                                              |[[seventh,  155000,  economy,  month,  net,  unemployment,  threeyear,  economist,  expect,  rate,  low,  percent,  straight,  add,  job,  hire,  stay,  85,  solid]]                                                                                                                                          |[]                                    |\n",
            "|Hey, Tony: Are you pleased with the current regime? Based on what you've seen, would it have been better to start a third year in the same system vs. what the Browns currently have? -- Mike B., Dover, Ohio                                                                                                                                                                                                                                                                                     |[[start,  regime,  year,  mike,  system,  ohio,  brown,  current,  tony,  please,  hey,  dover,  youve,  base]]                                                                                                                                                                                                |[ make,  state,  time,  million,  ago]|\n",
            "|The question wasn't rocket science, but it momentarily stumped a Minneapolis Institute of Arts tour guide.                                                                                                                                                                                                                                                                                                                                                                                        |[[guide,  minneapolis,  tour,  question,  momentarily,  science,  art,  institute,  stump,  wasnt,  rocket]]                                                                                                                                                                                                   |[]                                    |\n",
            "|\"\"\"First-class travel is excessive,\"\" said William Sannwald, a professor in the School of Business at San Diego State University. \"\"You have to show a business case as to why it's valuable to the port and I don't know if they can do that.\"\"\"                                                                                                                                                                                                                                                 |[[school,  diego,  university,  san,  case,  william,  valuable,  port,  dont,  show,  state,  sannwald,  professor,  business,  firstclass,  travel,  excessive]]                                                                                                                                             |[ year]                               |\n",
            "|Rather than enter camp intent on imposing his will, Matheny solicited input from veterans about their individual programs, when they preferred down time and what they believed warranted particular emphasis.                                                                                                                                                                                                                                                                                    |[[impose,  solicit,  warrant,  emphasis,  program,  camp,  enter,  input,  believe,  intent,  veteran,  prefer,  individual,  time,  matheny]]                                                                                                                                                                 |[ year]                               |\n",
            "|It concluded that Zidell's plan is not likely to jeopardize threatened wild fish or adversely modify their habitat.                                                                                                                                                                                                                                                                                                                                                                               |[[fish,  jeopardize,  threaten,  adversely,  habitat,  zidells,  plan,  conclude,  wild,  modify]]                                                                                                                                                                                                             |[]                                    |\n",
            "|An eviction notice from the 36th District Court fluttered in the breeze on the front window of the modest colonial on Detroit's west side Tuesday.                                                                                                                                                                                                                                                                                                                                                |[[breeze,  side,  west,  notice,  tuesday,  modest,  window,  colonial,  flutter,  front,  district,  eviction,  court,  36th,  detroits]]                                                                                                                                                                     |[]                                    |\n",
            "|\"And on the non-buying front, Baltimore neighborhoods can enter the city's \"\"Clean Community Competition\"\" for a chance at winning prizes of $1,000 or $5,000. You've got to apply by April 9, either by calling 311 or downloading the application here. The contest itself begins later that month and runs through October.\"                                                                                                                                                                   |[[city,  1000,  month,  contest,  311,  community,  april,  enter,  front,  baltimore,  prize,  download,  application,  clean,  neighborhood,  win,  october,  call,  youve,  nonbuying,  run,  begin,  apply,  9,  5000,  competition,  chance]]                                                             |[]                                    |\n",
            "|Second baseman Freddy Sanchez (right shoulder) threw for the first time since his rehab was temporarily shut down last week.                                                                                                                                                                                                                                                                                                                                                                      |[[shut,  shoulder,  freddy,  week,  baseman,  sanchez,  temporarily,  throw,  rehab,  time]]                                                                                                                                                                                                                   |[]                                    |\n",
            "|\"\"\"Her parents and our family prayed long and hard for her recovery,\"\" her father-in-law, George Shipp, wrote in an email. \"\"St. Jude is the patron saint not only of things almost despaired of, but of actors as well. Broadway may be a reality.\"\"\"                                                                                                                                                                                                                                            |[[recovery,  pray,  thing,  st,  shipp,  hard,  write,  actor,  fatherinlaw,  despair,  family,  parent,  email,  saint,  jude,  broadway,  patron,  reality,  george,  long]]                                                                                                                                 |[ louis]                              |\n",
            "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "model.transform(sample).show(truncate=False) # transform examines the input items against all the association rules and summarize the consequents as prediction"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "----------------------"
      ],
      "metadata": {
        "id": "QTdPO8y-SCD4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### **4.2.1 Experiment #2**"
      ],
      "metadata": {
        "id": "4ublIIWPV6fN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fp_growth = FPGrowth(itemsCol=\"BasketNoDup\", minSupport=0.003, minConfidence=0.003)"
      ],
      "metadata": {
        "id": "TvwQ61uyW9Oo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "POMZVRExW9YO"
      },
      "outputs": [],
      "source": [
        "# Apply Frequent Pattern Growth algorithm to the sample\n",
        "model = fp_growth.fit(sample)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "FREQUENT ITEMSETS"
      ],
      "metadata": {
        "id": "TepRWMCkV8kg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display frequent itemsets, with respect to parameters minSupport and minConfidence\n",
        "freq_itemsets = model.freqItemsets\n",
        "freq_itemsets.sort(freq_itemsets.freq.desc()).show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "45eacbcc-23ea-4788-a9b0-125deceda7e5",
        "id": "-Vol-UhXW2sS"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/pyspark/sql/context.py:125: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+----+\n",
            "|     items|freq|\n",
            "+----------+----+\n",
            "|   [ year]|3884|\n",
            "|   [ make]|3655|\n",
            "|  [ state]|2760|\n",
            "|   [ time]|2340|\n",
            "| [ people]|2010|\n",
            "|   [ game]|1930|\n",
            "|[ include]|1837|\n",
            "|   [ city]|1774|\n",
            "|    [ day]|1768|\n",
            "|   [ back]|1731|\n",
            "|   [ play]|1730|\n",
            "|   [ home]|1513|\n",
            "|   [ team]|1497|\n",
            "|    [ run]|1441|\n",
            "|   [ week]|1430|\n",
            "|   [ work]|1406|\n",
            "| [ county]|1378|\n",
            "|   [ find]|1370|\n",
            "|   [ good]|1343|\n",
            "|   [ show]|1322|\n",
            "+----------+----+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: the most frequent items are the same obtained in experiment #1. The main difference can be observed by looking at the tail of this dataset where less frequent items appear, indeed."
      ],
      "metadata": {
        "id": "nhN1WGwKGmes"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display frequent itemsets, given the configured minSupport and minConfidence parameters\n",
        "freq_itemsets = model.freqItemsets\n",
        "freq_itemsets.sort(freq_itemsets.freq.asc()).show(5)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fa1975a0-773a-4773-e241-f9dd35eb1c2e",
        "id": "N0sVRwwcW2sT"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-----------+----+\n",
            "|      items|freq|\n",
            "+-----------+----+\n",
            "| [ classic]| 152|\n",
            "| [ compete]| 152|\n",
            "|  [ crisis]| 152|\n",
            "|[ audience]| 152|\n",
            "|   [ human]| 152|\n",
            "+-----------+----+\n",
            "only showing top 5 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Total number of \"frequent\" items returned\n",
        "freq_itemsets.count()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "be12c62d-38ce-4804-b570-830162282a42",
        "id": "kF2JmHFDW2sT"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1055"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Display frequent pairs of items\n",
        "freq_itemsets.where(F.size(F.col(\"items\"))>1).sort(freq_itemsets.freq.desc()).show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5b2c5d51-2c78-454f-90dd-9a4330544bce",
        "id": "D_ey0tj5W2sU"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-----------------+----+\n",
            "|            items|freq|\n",
            "+-----------------+----+\n",
            "|   [ play,  game]| 397|\n",
            "|    [ louis,  st]| 383|\n",
            "|   [ make,  year]| 381|\n",
            "|    [ ago,  year]| 377|\n",
            "|  [ state,  year]| 369|\n",
            "|   [ time,  year]| 338|\n",
            "|[ million,  year]| 315|\n",
            "| [ season,  game]| 308|\n",
            "|   [ time,  make]| 294|\n",
            "|   [ team,  game]| 266|\n",
            "|    [ win,  game]| 254|\n",
            "|[ percent,  year]| 253|\n",
            "| [ people,  make]| 241|\n",
            "| [ school,  year]| 221|\n",
            "|   [ good,  make]| 220|\n",
            "|   [ team,  play]| 216|\n",
            "|   [ work,  year]| 216|\n",
            "|   [ play,  make]| 211|\n",
            "| [ high,  school]| 208|\n",
            "|[ include,  year]| 202|\n",
            "+-----------------+----+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: \"year\", \"make\", \"play\" are easily found in pairs."
      ],
      "metadata": {
        "id": "RliSQp9PHsmf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "ASSOCIATION RULES"
      ],
      "metadata": {
        "id": "-Iu94omjW2sU"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kzTZoMt8W2sU"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "# Display generated association rules\n",
        "association_rules = model.associationRules "
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Order by CONFIDENCE\n",
        "association_rules.sort(association_rules.confidence.desc()).show(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "de0a764e-cb2b-4411-c61a-81366f8b8354",
        "id": "BZzwbLJjW2sU"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-----------+------------+-------------------+------------------+--------------------+\n",
            "| antecedent|  consequent|         confidence|              lift|             support|\n",
            "+-----------+------------+-------------------+------------------+--------------------+\n",
            "|   [ louis]|       [ st]| 0.8362445414847162| 46.67208725878445|0.007607961543045...|\n",
            "|    [ vice]|[ president]|           0.796875|39.291166748285995|0.003039211791347...|\n",
            "|  [ united]|    [ state]| 0.6599326599326599|12.037076074757234|0.003893369353621...|\n",
            "|     [ ago]|     [ year]| 0.5974643423137876|7.7439623894852465|0.007488776766914306|\n",
            "|      [ st]|    [ louis]| 0.4246119733924612| 46.67208725878446|0.007607961543045...|\n",
            "|    [ ohio]|    [ state]| 0.3163841807909605|5.7708016048472945|0.003337173731675...|\n",
            "|    [ past]|     [ year]|0.30615640599001664| 3.968209523776884|0.003654999801358...|\n",
            "| [ million]|     [ year]| 0.2830188679245283|3.6683151001690533|0.006257200746891263|\n",
            "| [ student]|   [ school]|               0.25|11.462204007285974|0.003972825871042072|\n",
            "|[ district]|   [ school]| 0.2465564738292011|11.304322409389473|0.003555679154582655|\n",
            "+-----------+------------+-------------------+------------------+--------------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Order by LIFT (desc) - Look for highly dependent items (LIFT >> 1)\n",
        "association_rules.sort(association_rules.lift.desc()).show(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "966fe9bf-c17b-45cc-d153-515e42408d4f",
        "id": "Q41AztlUW2sU"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+------------+------------+-------------------+------------------+--------------------+\n",
            "|  antecedent|  consequent|         confidence|              lift|             support|\n",
            "+------------+------------+-------------------+------------------+--------------------+\n",
            "|       [ st]|    [ louis]| 0.4246119733924612| 46.67208725878446|0.007607961543045...|\n",
            "|    [ louis]|       [ st]| 0.8362445414847162| 46.67208725878445|0.007607961543045...|\n",
            "|[ president]|     [ vice]|0.14985308521057786|39.291166748285995|0.003039211791347...|\n",
            "|     [ vice]|[ president]|           0.796875|39.291166748285995|0.003039211791347...|\n",
            "|    [ state]|   [ united]|0.07101449275362319|12.037076074757236|0.003893369353621...|\n",
            "|   [ united]|    [ state]| 0.6599326599326599|12.037076074757234|0.003893369353621...|\n",
            "|   [ school]|  [ student]|0.18214936247723132|11.462204007285974|0.003972825871042072|\n",
            "|  [ student]|   [ school]|               0.25|11.462204007285974|0.003972825871042072|\n",
            "|   [ school]| [ district]|0.16302367941712204|11.304322409389474|0.003555679154582655|\n",
            "| [ district]|   [ school]| 0.2465564738292011|11.304322409389473|0.003555679154582655|\n",
            "+------------+------------+-------------------+------------------+--------------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Order by LIFT (asc) - Look for perfect substitutes items (LIFT << 1)\n",
        "association_rules.sort(association_rules.lift.asc()).show(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1bb4660f-1dfe-4e61-92b3-987153ca53d1",
        "id": "9J8R6UNGW2sV"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|antecedent|consequent|         confidence|              lift|             support|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|  [ state]|   [ make]|0.06847826086956521|0.9431826562778802|0.003754320448134...|\n",
            "|   [ make]|  [ state]|0.05170998632010944|0.9431826562778802|0.003754320448134...|\n",
            "|   [ year]| [ people]|0.04582904222451081|1.1478237033165788|0.003535815025227444|\n",
            "| [ people]|   [ year]|0.08855721393034825|1.1478237033165788|0.003535815025227444|\n",
            "|   [ time]|  [ state]|0.06752136752136752|1.2315799578843056|0.003138532438123237|\n",
            "|  [ state]|   [ time]| 0.0572463768115942|1.2315799578843056|0.003138532438123237|\n",
            "|   [ year]|   [ play]|0.04402677651905252|  1.28115374770066|0.003396766119740...|\n",
            "|   [ play]|   [ year]|0.09884393063583816|1.2811537477006603|0.003396766119740...|\n",
            "|   [ make]|    [ day]| 0.0454172366621067|1.2932095746853933| 0.00329744547296492|\n",
            "|    [ day]|   [ make]|0.09389140271493213|1.2932095746853935| 0.00329744547296492|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: \"state\" and \"make\" are independent. \"year\" and \"people\", \"time\" and \"state\" are mostly likewise independent from each other."
      ],
      "metadata": {
        "id": "oxcZFYm4W2sV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Here, the most frequent pairs of items are also displayed \n",
        "association_rules.sort(association_rules.support.desc()).show(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f544fa15-f339-4d76-d9ad-cd9e459a85ee",
        "id": "de-SeE_WW2sV"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|antecedent|consequent|         confidence|              lift|             support|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "|   [ game]|   [ play]|0.20569948186528497| 5.985736020845188|0.007886059354018513|\n",
            "|   [ play]|   [ game]|0.22947976878612716| 5.985736020845188|0.007886059354018513|\n",
            "|     [ st]|  [ louis]| 0.4246119733924612| 46.67208725878446|0.007607961543045...|\n",
            "|  [ louis]|     [ st]| 0.8362445414847162| 46.67208725878445|0.007607961543045...|\n",
            "|   [ make]|   [ year]|0.10424076607387141|1.3511041827216361|0.007568233284335147|\n",
            "|   [ year]|   [ make]|0.09809474768280124|1.3511041827216363|0.007568233284335147|\n",
            "|   [ year]|    [ ago]| 0.0970648815653965| 7.743962389485246|0.007488776766914306|\n",
            "|    [ ago]|   [ year]| 0.5974643423137876|7.7439623894852465|0.007488776766914306|\n",
            "|  [ state]|   [ year]|0.13369565217391305|1.7328801549276855|0.007329863732072623|\n",
            "|   [ year]|  [ state]|0.09500514933058703|1.7328801549276855|0.007329863732072623|\n",
            "+----------+----------+-------------------+------------------+--------------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "PREDICTIONS"
      ],
      "metadata": {
        "id": "QUw0FNE9W2sV"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "821f1412-cb5e-4cce-b608-5e5eb2178476",
        "id": "vU5NuOpCW2sV"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
            "|Text                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |BasketNoDup                                                                                                                                                                                                                                                                                                    |prediction                                                                                                                                                                             |\n",
            "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
            "|But time and again in the report, Sullivan called on CPS to correct problems to improve employee accountability, saying, for example, that measures to keep employees from submitting fraudulent invoices or to block employees from accessing inappropriate websites were not in place.                                                                                                                                                                                                          |[[employee,  improve,  block,  place,  sullivan,  correct,  report,  accountability,  websites,  cps,  inappropriate,  time,  call,  problem,  submit,  fraudulent,  measure,  invoice,  access]]                                                                                                              |[ state,  make,  year,  game,  day,  play]                                                                                                                                             |\n",
            "|I was just trying to hit it hard someplace, said Rizzo, who hit the pitch to the opposite field in left-center. Im just up there trying to make good contact.                                                                                                                                                                                                                                                                                                                                     |[[rizzo,  im,  field,  opposite,  pitch,  make,  good,  hit,  someplace,  leftcenter,  contact,  hard]]                                                                                                                                                                                                        |[ year,  state,  time,  people,  game,  day,  back,  play,  team,  work,  dont]                                                                                                        |\n",
            "|Andrade's children, Erin and son Patrick, are adults, and he said he was intrigued by the opportunity to play a mentoring role in the life of a high school student.                                                                                                                                                                                                                                                                                                                              |[[school,  son,  student,  mentore,  child,  andrades,  opportunity,  adult,  play,  role,  intrigue,  patrick,  life,  high,  erin]]                                                                                                                                                                          |[ school,  game,  make,  time,  year,  team,  season]                                                                                                                                  |\n",
            "|The new law, which takes effect immediately, requires bareheaded riders 21 and older to carry at least $20,000 in additional medical insurance. Riders also must have passed a motorcycle safety course or have had their motorcycle endorsement for at least two years. These safeguards are woefully inadequate, given the increased risks and medical costs that come with riding without a helmet.                                                                                            |[[additional,  bareheaded,  require,  endorsement,  insurance,  pass,  effect,  carry,  immediately,  woefully,  21,  cost,  law,  old,  inadequate,  safety,  20000,  year,  helmet,  safeguard,  risk,  rider,  medical,  ride,  motorcycle,  increase,  take]]                                              |[ make,  past,  start,  state,  time,  people,  include,  city,  day,  back,  play,  team,  10,  work,  county,  plan,  big,  percent,  month,  million,  ago,  school,  spend,  early]|\n",
            "|Laura Taiby says she last saw her daughter, Laudan Taiby, as she was leaving their home in the 6700 block of Southwest 14th Avenue in the Hillsdale area about 5 p.m. Monday.                                                                                                                                                                                                                                                                                                                     |[[hillsdale,  5,  laura,  block,  14th,  6700,  home,  taiby,  leave,  laudan,  southwest,  area,  monday,  pm,  daughter,  avenue]]                                                                                                                                                                           |[]                                                                                                                                                                                     |\n",
            "|\"Context: Last summer, when a panel of growers and winemakers at Foothill Grape Day in Plymouth got around to talking of barbera's role at the table, Justin Boeger remarked: \"\"Barbera is what I bring to dinner when I don't know the menu. It's versatile.\"\" Other than that, any Italian dish enhanced with tomatoes is traditionally paired with barbera.\"                                                                                                                                   |[[panel,  plymouth,  barbera,  justin,  foothill,  pair,  boeger,  dont,  tomato,  grape,  dinner,  menu,  italian,  bring,  day,  summer,  remark,  barberas,  versatile,  enhance,  traditionally,  talk,  winemaker,  grower,  context,  role,  table,  dish]]                                              |[ make,  time,  year]                                                                                                                                                                  |\n",
            "|The U.S.-led coalition routinely reports each time an American or other foreign soldier is killed by an Afghan in uniform, but the military is under-reporting the number of overall attacks. The Associated Press reported earlier this month that the International Security Assistance Force, or ISAF, does not report attacks in which the Afghan wounds or misses his U.S. or allied target. It also doesn't report the wounding of troops who were attacked alongside those who were killed.|[[afghan,  wounding,  international,  isaf,  foreign,  month,  target,  routinely,  underreporting,  report,  assistance,  force,  usled,  wounds,  attack,  miss,  ally,  troop,  number,  military,  press,  kill,  uniform,  time,  american,  alongside,  security,  soldier,  early,  coalition,  doesnt]]|[ state,  make,  year,  game,  day,  play]                                                                                                                                             |\n",
            "|\"\"\"They're supposed to be all checked out,\"\" said the father of the teenage girl involved in the incident. \"\"Something is not right.\"\"\"                                                                                                                                                                                                                                                                                                                                                           |[[suppose,  girl,  involve,  theyre,  teenage,  incident,  check,  father]]                                                                                                                                                                                                                                    |[]                                                                                                                                                                                     |\n",
            "|While most of the wines are from Napa Valley and elsewhere in California, the international selections round out the list nicely. On any given night, 26 wines are offered by the glass (plus 17 dessert offerings), with a good selection of half bottles.                                                                                                                                                                                                                                       |[[17,  bottle,  international,  half,  selection,  round,  napa,  dessert,  offerings,  good,  list,  26,  nicely,  glass,  wine,  valley,  night,  offer,  california]]                                                                                                                                       |[ make]                                                                                                                                                                                |\n",
            "|As the officers and the St. Andres sat in a row behind eight defense lawyers this morning, about a dozen family and friends of the officers watched from the audience.                                                                                                                                                                                                                                                                                                                            |[[family,  andres,  row,  officer,  sit,  dozen,  defense,  friend,  morning,  lawyer,  audience,  st,  watch]]                                                                                                                                                                                                |[ louis]                                                                                                                                                                               |\n",
            "|Economists expect the economy added 155,000 net jobs, the seventh straight month of solid hiring. The unemployment rate is expected stay for the second month at 8.5 percent, near a three-year low.                                                                                                                                                                                                                                                                                              |[[seventh,  155000,  economy,  month,  net,  unemployment,  threeyear,  economist,  expect,  rate,  low,  percent,  straight,  add,  job,  hire,  stay,  85,  solid]]                                                                                                                                          |[ year]                                                                                                                                                                                |\n",
            "|Hey, Tony: Are you pleased with the current regime? Based on what you've seen, would it have been better to start a third year in the same system vs. what the Browns currently have? -- Mike B., Dover, Ohio                                                                                                                                                                                                                                                                                     |[[start,  regime,  year,  mike,  system,  ohio,  brown,  current,  tony,  please,  hey,  dover,  youve,  base]]                                                                                                                                                                                                |[ make,  past,  start,  state,  time,  people,  include,  city,  day,  back,  play,  team,  10,  work,  county,  plan,  big,  percent,  month,  million,  ago,  school,  spend,  early]|\n",
            "|The question wasn't rocket science, but it momentarily stumped a Minneapolis Institute of Arts tour guide.                                                                                                                                                                                                                                                                                                                                                                                        |[[guide,  minneapolis,  tour,  question,  momentarily,  science,  art,  institute,  stump,  wasnt,  rocket]]                                                                                                                                                                                                   |[]                                                                                                                                                                                     |\n",
            "|\"\"\"First-class travel is excessive,\"\" said William Sannwald, a professor in the School of Business at San Diego State University. \"\"You have to show a business case as to why it's valuable to the port and I don't know if they can do that.\"\"\"                                                                                                                                                                                                                                                 |[[school,  diego,  university,  san,  case,  william,  valuable,  port,  dont,  show,  state,  sannwald,  professor,  business,  firstclass,  travel,  excessive]]                                                                                                                                             |[ make,  year,  time,  law,  include,  county,  school,  united,  ohio]                                                                                                                |\n",
            "|Rather than enter camp intent on imposing his will, Matheny solicited input from veterans about their individual programs, when they preferred down time and what they believed warranted particular emphasis.                                                                                                                                                                                                                                                                                    |[[impose,  solicit,  warrant,  emphasis,  program,  camp,  enter,  input,  believe,  intent,  veteran,  prefer,  individual,  time,  matheny]]                                                                                                                                                                 |[ state,  make,  year,  game,  day,  play]                                                                                                                                             |\n",
            "|It concluded that Zidell's plan is not likely to jeopardize threatened wild fish or adversely modify their habitat.                                                                                                                                                                                                                                                                                                                                                                               |[[fish,  jeopardize,  threaten,  adversely,  habitat,  zidells,  plan,  conclude,  wild,  modify]]                                                                                                                                                                                                             |[ year]                                                                                                                                                                                |\n",
            "|An eviction notice from the 36th District Court fluttered in the breeze on the front window of the modest colonial on Detroit's west side Tuesday.                                                                                                                                                                                                                                                                                                                                                |[[breeze,  side,  west,  notice,  tuesday,  modest,  window,  colonial,  flutter,  front,  district,  eviction,  court,  36th,  detroits]]                                                                                                                                                                     |[ school]                                                                                                                                                                              |\n",
            "|\"And on the non-buying front, Baltimore neighborhoods can enter the city's \"\"Clean Community Competition\"\" for a chance at winning prizes of $1,000 or $5,000. You've got to apply by April 9, either by calling 311 or downloading the application here. The contest itself begins later that month and runs through October.\"                                                                                                                                                                   |[[city,  1000,  month,  contest,  311,  community,  april,  enter,  front,  baltimore,  prize,  download,  application,  clean,  neighborhood,  win,  october,  call,  youve,  nonbuying,  run,  begin,  apply,  9,  5000,  competition,  chance]]                                                             |[ game,  team,  year]                                                                                                                                                                  |\n",
            "|Second baseman Freddy Sanchez (right shoulder) threw for the first time since his rehab was temporarily shut down last week.                                                                                                                                                                                                                                                                                                                                                                      |[[shut,  shoulder,  freddy,  week,  baseman,  sanchez,  temporarily,  throw,  rehab,  time]]                                                                                                                                                                                                                   |[]                                                                                                                                                                                     |\n",
            "|\"\"\"Her parents and our family prayed long and hard for her recovery,\"\" her father-in-law, George Shipp, wrote in an email. \"\"St. Jude is the patron saint not only of things almost despaired of, but of actors as well. Broadway may be a reality.\"\"\"                                                                                                                                                                                                                                            |[[recovery,  pray,  thing,  st,  shipp,  hard,  write,  actor,  fatherinlaw,  despair,  family,  parent,  email,  saint,  jude,  broadway,  patron,  reality,  george,  long]]                                                                                                                                 |[ louis]                                                                                                                                                                               |\n",
            "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "model.transform(sample).show(truncate=False) # transform examines the input items against all the association rules and summarize the consequents as prediction"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}