01-eda.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory analysis\n",
    "\n",
    "We aren't focusing on defining metrics or data cleaning in this workshop, so our first step will be exploratory analysis on a prepared dataset.  This will involve looking at the data and then examining each variable to see if we can identify any interesting patterns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use a range of Python libraries to help us explore, visualise and interact with the data. We could install these into the running environment, one by one. For example: \n",
    "\n",
    "`!pip install numpy`\n",
    "\n",
    "would install `numpy`. However, it is good practice to keep track of all your requirements and specific versions in one location. The explicit versions of the libraries we will use are listed in the `requirements.txt` file, which you can open from the file viewer on the left hand side of the screen. We install all these requirements into our environment below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then import the installed libraries into our notebook environment like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we get started with exploring our data, we first need to load the data. The data is stored in an S3 bucket, and we connect to it using the `boto3` library. The `boto3` library was built into the `Standard Data Science` notebook image which we selected from the spawner page. As such, it is already installed in our environment. (However, it's also good practice to keep track of every library you use in your `requirements.txt` file - go ahead and add `boto3` to the `requirements.txt`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "boto3 expects all requests to be signed, meaning you have to authenticate and provide credentials (keys, ids) to connect to data. However we are connecting to public data which is in a open s3 bucket, so we use the `UNSIGNED` signature from botocore to allow us to access the data without providing credentials. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from botocore import UNSIGNED\n",
    "from botocore.client import Config\n",
    "\n",
    "s3 = boto3.client('s3', config = Config(signature_version=UNSIGNED))\n",
    "s3.download_file('fraud-detection-dataset', 'fraud-cleaned-sample.parquet', 'fraud-cleaned-sample.parquet')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll start by reading in our fraud dataset and looking at the column names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_parquet(\"fraud-cleaned-sample.parquet\")\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When dealing with an unfamiliar dataset, we first need to understand what data we've collected.  If we're lucky, our organization will publish a data catalog or data glossary that explains what each dataset contains.  If we don't have this extra metadata, we'll need to explore the dataset manually and document our observations.  Sometimes the column names will be self-explanatory, but if they aren't, we'll have to dig deeper to make sense of them.  A good second step is examining the types of each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, `string`-valued attributes in structured data are free-form text, but some string-valued variables are [categorical variables](https://en.wikipedia.org/wiki/Categorical_variable).  We can usually identify these by looking at unique values to identify attributes with a small number of possible values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(df[\"label\"].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(df[\"trans_type\"].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we're dealing with messy data, categorical variables may have a large number of unique values, some of which are spurious and may occur only once.  In this case, we don't want to examine every unique value but we can look at the top few values to get a sense for what the range of values _should_ be.  In this next cell, we're looking at the three most common transaction types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[[\"timestamp\", \"trans_type\"]].groupby([\"trans_type\"]).count().nlargest(3, \"timestamp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the most common transaction types for each class of transaction, as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df[\"label\"] == \"fraud\"][[\"timestamp\", \"trans_type\"]].groupby(\n",
    "    [\"trans_type\"]\n",
    ").count(\n",
    ").nlargest(3, \"timestamp\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df[\"label\"] == \"legitimate\"][[\"timestamp\", \"trans_type\"]].groupby(\n",
    "    [\"trans_type\"]\n",
    ").count(\n",
    ").nlargest(3, \"timestamp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transaction type distribution\n",
    "\n",
    "Let's look at the transaction type first to see what the distribution looks like for legitimate and fraud transactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pt = pd.pivot_table(df[[\"label\", \"trans_type\", \"timestamp\"]], \n",
    "                    index=[\"label\", \"trans_type\"], aggfunc=len)\n",
    "\n",
    "pt.columns = ['count']\n",
    "gdf = pd.DataFrame(pt.to_records())\n",
    "\n",
    "gdf['total'] = gdf.groupby('label')['count'].transform(np.sum)\n",
    "gdf['percentage'] = gdf['count'] / gdf['total']\n",
    "\n",
    "gdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import altair as alt\n",
    "\n",
    "alt.Chart(gdf).mark_bar().encode(\n",
    "    alt.Y('percentage:Q', axis=alt.Axis(format='.0%')), column='trans_type', x=\"label\", color='label'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Foreign transaction distribution\n",
    "\n",
    "We can repeat this process for percentage of foreign vs domestic transactions for each label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pt = pd.pivot_table(df[[\"label\", \"foreign\", \"timestamp\"]], \n",
    "                    index=[\"label\", \"foreign\"], aggfunc=len)\n",
    "\n",
    "pt.columns = ['count']\n",
    "gdf = pd.DataFrame(pt.to_records())\n",
    "\n",
    "gdf['total'] = gdf.groupby('label')['count'].transform(np.sum)\n",
    "gdf['pctage'] = gdf['count'] / gdf['total']\n",
    "\n",
    "gdf\n",
    "\n",
    "alt.Chart(gdf).mark_bar().encode(\n",
    "    alt.Y('pctage:Q', axis=alt.Axis(format='.0%')), column='foreign', x=\"label\", color='label'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transaction amount distribution\n",
    "\n",
    "When we're dealing with continuous variables, rather than categorical ones, we'll need to do something other than simply counting values to make sense of their distributions (we aren't interested, for example, in knowing that there were exactly 24 transactions for \\$17.35).  There are a few options for how to proceed here:\n",
    "\n",
    "1.  we could _quantize_ values by rounding to the nearest dollar (or ten dollars, or to the nearest power of _k_ dollars),\n",
    "1.  we could _discretize_ values by grouping nearby values into histogram buckets, or\n",
    "1.  we could take the _cumulative distribution_ of values, which gives us a curve for the whole distribution.\n",
    "\n",
    "Let's take a look at the latter approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qs = df[['label','amount']].groupby('label').quantile(q=[0.01,0.05,0.1,0.25,0.5,0.75,0.9,0.95,0.99])\n",
    "qs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does the above data mean?  For each label, we have a set of rows, corresponding to the first, fifth, tenth, ..., ninety-fifth, and ninety-ninth percentiles.  The value at each row indicates the transaction amount at each percentile.  As an example, this means that 25% of legitimate transaction amounts are less than or equal to $12.43.\n",
    "\n",
    "We can plot the cumulative distribution of each transaction type to compare these distributions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qdf = pd.DataFrame(qs.to_records())\n",
    "\n",
    "alt.Chart(qdf).mark_line(interpolate=\"monotone\").encode(\n",
    "    alt.Y(\"amount\", axis=alt.Axis(title='transaction amounts (log scale)'), scale=alt.Scale(type='log')), \n",
    "    alt.X(\"level_0\", axis=alt.Axis(title='cumulative distribution'), scale=alt.Scale(type='linear')), \n",
    "    color=\"label\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, relatively more of the fraudulent transactions are for low dollar amounts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interarrival times\n",
    "\n",
    "Another interesting feature to look at is the distribution of interarrival times, or the time gaps between transactions.  Our intuition is that legitimate transactions will be spread apart further (and will have some large gaps), while fraudulent transactions will cluster together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fraudsamp = df[df[\"label\"] == \"fraud\"].copy()\n",
    "legitsamp = df[df[\"label\"] == \"legitimate\"].sample(len(fraudsamp)).copy()\n",
    "\n",
    "fraudsamp['irank'] = fraudsamp['interarrival'].rank(pct=True, method=\"dense\")\n",
    "legitsamp['irank'] = legitsamp['interarrival'].rank(pct=True, method=\"dense\")\n",
    "qdf = pd.concat([fraudsamp.groupby(['label', 'interarrival', 'irank']).size(), legitsamp.groupby(['label', 'interarrival', 'irank']).size()])\n",
    "qdf = pd.DataFrame(pd.DataFrame(qdf).to_records())\n",
    "qdf = qdf[qdf['interarrival'] > 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(qdf.sample(5000)).mark_line().encode(\n",
    "    alt.Y(\"interarrival\", axis=alt.Axis(title='interarrival time'), scale=alt.Scale(type='log')), \n",
    "    alt.X(\"irank\", axis=alt.Axis(title='cumulative distribution'), scale=alt.Scale(type='linear')), \n",
    "    color=\"label\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activity by time of day\n",
    "\n",
    "We've generated users to act during typical sleep-wake cycles at different times of day.  (To simplify things, the data generator assumes that a user with a user ID _k_ is in time zone _t_ if _k_ mod 24 == _t_.)  As we'll see, legitimate transactions are centered around waking hours, while fraudulent transactions are roughly equally likely to happen at any time of day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"hour\"] = df[\"timestamp\"] % 86400 // 3600\n",
    "df[\"tz\"] = df[\"user_id\"] % 24\n",
    "\n",
    "tzpivot = pd.pivot_table(df[[\"label\", \"hour\", \"tz\", \"timestamp\"]], \n",
    "                         index=[\"label\", \"hour\", \"tz\"], aggfunc=len)\n",
    "\n",
    "tzdf = pd.DataFrame(tzpivot.to_records()).set_axis([\"label\", \"tz\", \"hour\", \"count\"], axis=\"columns\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(tzdf).mark_area().encode(\n",
    "    x=\"hour\",\n",
    "    y=alt.Y(\"count\", stack=\"normalize\", axis=alt.Axis(title=\"percentage of total\")), \n",
    "    color=\"tz\",\n",
    "    column=\"label\",\n",
    "    tooltip=['label', 'tz', 'hour', 'count']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've explored our data and visualised the distribution of values, let's move on to the next notebook, [02-feature-engineering.ipynb](02-feature-engineering.ipynb) where we transform the data into feature vectors. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}