DeepView Demo0_basics.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepview import DeepView\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import time\n",
    "# ---------------------------\n",
    "import demo_utils as demo\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib qt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# matplotlib qt seems to be a bit buggy with notebooks, so we execute it multiple times\n",
    "%matplotlib qt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of contents\n",
    "\n",
    "<br>\n",
    "\n",
    "<font size=\"+1\"><b>\n",
    "    \n",
    " 0. [Usage Instructions](#DeepView-Usage-Instructions)\n",
    " 0. [Setting up DeepView](#Demo-with-Torch-model)\n",
    " 0. [Tuning $\\lambda$ Hyperparameter](#Tuning-the-$\\lambda$-Hyperparameter)\n",
    " \n",
    "</b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DeepView Usage Instructions\n",
    "\n",
    " 1. Create a wrapper funktion like ```pred_wrapper``` which receives a numpy array of samples and returns according class probabilities from the classifier as numpy arrays\n",
    " 2. Initialize DeepView-object and pass the created method to the constructor\n",
    " 3. Run your code and call ```add_samples(samples, labels)``` at any time to add samples to the visualization together with the ground truth labels.\n",
    "    * The ground truth labels will be visualized along with the predicted labels\n",
    "    * The object will keep track of a maximum number of samples specified by ```max_samples``` and it will throw away the oldest samples first\n",
    " 4. Call the ```show``` method to render the plot\n",
    "\n",
    "The following parameters with \"(!)\" must be specified on initialization:\n",
    "\n",
    "| <p align=\"left\">Variable               | <p align=\"left\">Meaning           |\n",
    "|------------------------|-------------------|\n",
    "| <p align=\"left\">(!)```pred_wrapper```     | <p align=\"left\">Wrapper function allowing DeepView to use your model. Expects a single argument, which should be a batch of samples to classify. Returns (valid / softmaxed) prediction probabilities for this batch of samples. |\n",
    "| <p align=\"left\">(!)```classes```          | <p align=\"left\">Names of all different classes in the data. |\n",
    "| <p align=\"left\">(!)```max_samples```      | <p align=\"left\">The maximum amount of samples that DeepView will keep track of. When more samples are added, the oldest samples are removed from DeepView. |\n",
    "| <p align=\"left\">(!)```batch_size```       | <p align=\"left\">The batch size used for classification |\n",
    "| <p align=\"left\">(!)```data_shape```       | <p align=\"left\">Shape of the input data (complete shape; excluding the batch dimension) |\n",
    "| <p align=\"left\">```resolution```       | <p align=\"left\">x- and y- Resolution of the decision boundary plot. A high resolution will compute significantly longer than a lower resolution, as every point must be classified, default 100. |\n",
    "| <p align=\"left\">```cmap```             | <p align=\"left\">Name of the colormap that should be used in the plots, default 'tab10'. |\n",
    "| <p align=\"left\">```interactive```      | <p align=\"left\">When ```interactive``` is True, this method is non-blocking to allow plot updates. When ```interactive``` is False, this method is blocking to prevent termination of python scripts, default True. |\n",
    "| <p align=\"left\">```title```            | <p align=\"left\">Title of the deepview-plot. |\n",
    "| <p align=\"left\">```data_viz```         | <p align=\"left\">DeepView has a reactive plot, that responds to mouse clicks and shows the according data sample, when it is clicked. You can pass a custom visualization function, if ```data_viz``` is None, DeepView will try to show each sample as an image, if possible. (optional, default None)  |\n",
    "| <p align=\"left\">```mapper```           | <p align=\"left\">An object that maps samples from the data space to 2D space. Normally UMAP is used for this, but you can pass a custom mapper as well. (optional)  |\n",
    "| <p align=\"left\">```inv_mapper```       | <p align=\"left\">An object that maps samples from the 2D space to the data space. Normally ```deepview.embeddings.InvMapper``` is used for this, but you can pass a custom inverse mapper as well. (optional)  |\n",
    "| <p align=\"left\">```kwargs```       | <p align=\"left\">Configuration for the embeddings in case they are not specifically given in mapper and inv_mapper. Defaults to ```deepview.config.py```.  (optional)  |\n",
    "    \n",
    "    \n",
    "The hyperparameters that influence the visualization are\n",
    "\n",
    "    \n",
    "|Variable            |Meaning          |\n",
    "|--------------------|------------------|\n",
    "| ```lam```          | (Fisher metric parameter) Controls the amount of euclidean regularization of the Fisher metric, the larger the more. Between 0 and 1, default 0.65. |\n",
    "| ```n_neighbors```  | (UMAP parameter) Number of neighbors used in UMAP. Determines how many points are considered to be close for each data point, roughly speaking. Default is 30. |\n",
    "| ```a```            | (inverse mapping parameter) Determines the nonlinearity of the inverse mapping. Large values correspond to nonlinear functions. Compared to the definition in the paper, we additionally devide by the range of the data, such that the letter can be ignored in setting this parameter. Default is 500. \n",
    "| ```min_dist```     | (UMAP parameter) The minimum distance between embedded points. Smaller values cause more clustered or clumped visualizations. Interdepends with ```spread```. Default is 0.1. |\n",
    "| ```spread```       | (UMAP parameter) The scale of embedded points. Together with ```min_dist``` causes more/less clustering. Default is 1.0.|\n",
    "|```n```             | (Fisher metric parameter) Number of interpolation steps for distance calculation between two points. In the paper, this is also called n, default 5. |\n",
    "| ```random_state``` | (UMAP parameter) Seed used by UMAP. |\n",
    "    \n",
    "  \n",
    "In this notebook, we will focus on the most important paramter ```lam``` (corresponds to $\\lambda$ in the paper)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demo with Torch model\n",
    "\n",
    "1. Initialize a pretrained torch model\n",
    "2. Create CIFAR10 dataset\n",
    "3. Write the wrapper function ```pred_wrapper```\n",
    "    1. (optional) Create a visualization function if you want to visualize single examples by clicking in the DeepView plot.\n",
    "4. Initialize a ```DeepView``` object\n",
    "5. Add samples to DeepView and call ```deepview.show```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# device will be detected automatically\n",
    "# Set to 'cpu' or 'cuda:0' to set the device manually\n",
    "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
    "# get torch model\n",
    "torch_model = demo.create_torch_model(device)\n",
    "# get CIFAR-10 data\n",
    "testset = demo.make_cifar_dataset()\n",
    "\n",
    "print('\\nUsing device:', device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# softmax operation to use in pred_wrapper\n",
    "softmax = torch.nn.Softmax(dim=-1)\n",
    "\n",
    "# this is the prediction wrapper, it encapsulates the call to the model\n",
    "# and does all the casting to the appropriate datatypes\n",
    "def pred_wrapper(x):\n",
    "    with torch.no_grad():\n",
    "        x = np.array(x, dtype=np.float32)\n",
    "        tensor = torch.from_numpy(x).to(device)\n",
    "        logits = torch_model(tensor)\n",
    "        probabilities = softmax(logits).cpu().numpy()\n",
    "    return probabilities\n",
    "\n",
    "def visualization(image, point2d, pred, label=None, title=None):\n",
    "    f, a = plt.subplots()\n",
    "    a.set_title(title)\n",
    "    a.imshow(image.transpose([1, 2, 0]))\n",
    "\n",
    "# the classes in the dataset to be used as labels in the plots\n",
    "classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')\n",
    "\n",
    "# --- Deep View Parameters ----\n",
    "batch_size = 512\n",
    "max_samples = 500\n",
    "data_shape = (3, 32, 32)\n",
    "lam = .65 # default parameter\n",
    "title = 'ResNet-20 - CIFAR10'\n",
    "\n",
    "deepview = DeepView(pred_wrapper, classes, max_samples, batch_size, \n",
    "                    data_shape, lam=lam, title=title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "n_samples = 200\n",
    "sample_ids = np.random.choice(len(testset), n_samples)\n",
    "X = np.array([ testset[i][0].numpy() for i in sample_ids ])\n",
    "Y = np.array([ testset[i][1] for i in sample_ids ])\n",
    "\n",
    "t0 = time.time()\n",
    "deepview.add_samples(X, Y)\n",
    "deepview.show()\n",
    "\n",
    "print('Time to calculate visualization for %d samples: %.2f sec' % (n_samples, time.time() - t0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The visualization seems reasonable with the default $\\lambda$ = 0.65. But how can we evaluate which value is good for our data set?\n",
    "\n",
    "\n",
    "## Tuning the $\\lambda$ Hyperparameter\n",
    "\n",
    "For $\\lambda = 0$, the DeepView plot will be organized according to the fisher metric, induced by the certainty of the model. This tends to create collapsed clusters, since variation inside a region of one class (as viewed by the classifier) is strongly reduced.\n",
    "\n",
    "<img alt=\"img_collapsed_clusters\" width=400px src=\"https://user-images.githubusercontent.com/30961397/90417945-be51fa00-e0b4-11ea-8951-d0183432c90b.png\">\n",
    "\n",
    "For $\\lambda = 1$, the DeepView plot will be organized according to structural properties of the datapoints as measured by the euclidean metric, ignoring the predictions of the model. So for more complex datasets, the points will be scattered all over the place, as the euclidean metric is an insufficient indicator for sample class.\n",
    "\n",
    "<img alt=\"img_diffused_clusters\" width=400px src=\"https://user-images.githubusercontent.com/30961397/90418110-fc4f1e00-e0b4-11ea-8c85-fa3b53a8e531.png\">\n",
    "\n",
    "A balance must be found here in order to obtain a discriminative visualization (a visualization that is dominated by the class structure as induced by the classifier) that also locally incorporates as much structure of the original data as possible.\n",
    "\n",
    "**The [DeepView-paper](https://www.ijcai.org/Proceedings/2020/0319.pdf) suggests the following tuning method for $\\lambda$:**\n",
    "\n",
    "As a metric for how well the embedding $\\pi$ is representing the view of the model, the consistency of the point clusters in the embedding with the models predictions can be used:\n",
    "> [...] points that are close to each other should be points that are classified similarity by the classification model.\n",
    "\n",
    "To evaluate on this, a KNN model is trained on the embedded samples with the model predictions as labels.\n",
    "Thus, the leave-one-out error $Q_{kNN}$ of this classifier is the according metric. In order to tune $\\lambda$,\n",
    "\n",
    "> [...] we evaluate $\\pi$ for $Q_{kNN}$ with $\\lambda \\in [0.2;0.8]$ and choose the largest one that does not degrade $Q_{kNN}$ significantly.[...]\n",
    "\n",
    "This is applied here for 6 values linearly distributed accross $[0; 1]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepview.evaluate import evaluate_umap\n",
    "\n",
    "print('Evaluation of DeepView: %s\\n' % deepview.title)\n",
    "\n",
    "for l in np.linspace(0., 1., 6):\n",
    "    deepview.verbose = False\n",
    "    deepview.set_lambda(l)\n",
    "    q_knn = evaluate_umap(deepview, True)\n",
    "    print('Lambda: %.2f - Q_kNN: %.3f' % (l, q_knn))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for l in np.linspace(0.6, 0.8, 6):\n",
    "    deepview.verbose = False\n",
    "    deepview.set_lambda(l)\n",
    "    q_knn = evaluate_umap(deepview, True)\n",
    "    print('Lambda: %.2f - Q_kNN: %.3f' % (l, q_knn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results**\n",
    "\n",
    "These values show, that around  $\\lambda = 0.64$ a jump seems to occur (this varies a bit due to the randomly selected data subset). Hence, a value around this is reasonable here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Plot after $\\lambda$-tuning**\n",
    "\n",
    "<img alt=\"img_good_clusters\" width=500px src=\"https://user-images.githubusercontent.com/30961397/90418547-8a2b0900-e0b5-11ea-8ae2-2b4a76c2e6ce.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deepview.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}