S02b_Numpy.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute before using this notebook if using google colab\n",
    "\n",
    "kernel = str(get_ipython())\n",
    "\n",
    "if 'google.colab' in kernel:    \n",
    "    !wget https://raw.githubusercontent.com/fredzett/rmqa/master/utils.py -P local_modules -nc \n",
    "    !npx degit fredzett/rmqa/data data\n",
    "    import sys\n",
    "    sys.path.append('local_modules')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy is one of the most important libraries / modules in python. It is especially useful when working with numerical arrays (which researchers usually do). \n",
    "\n",
    "**Why is `numpy` so important:** \n",
    "\n",
    "- useful when working with numerical arrays (which researchers usually do)\n",
    "- enables store and manipulate in-memory data in Python\n",
    "- very fast (roughly $~50x$ faster than python's lists (which makes a hugh difference when working with large datasets)\n",
    "\n",
    "Numpy's functionality is very broad and we will not be able to cover all of it in this course. We will, however, focus on some basic functionality that will make your life easier when working with (research) data. \n",
    "\n",
    "Also don't forget that you can always use `np.<TAB>` to display contents of functionality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scalar = np.array(12)\n",
    "scalar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not very useful, however. Let's create some arrays from lists (or lists of lists) and from scratch using numpy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## from lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Arrays (such as vectors or matrices) can be created from Python's lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector = np.array([1,2,3,4,5])\n",
    "vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike lists `np.array` must only contain one data type (such as int, float, str). So the following is possible, but it converts all three data types to string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lst = [1,2,3,\"A\",\"B\",True]\n",
    "str_vector = np.array(lst)\n",
    "str_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using numpy one can also create multidimensional arrays. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = np.array([[1,2,3],[4,5,6],[7,8,9]])\n",
    "matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## from scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For larger arrays it is usually more efficient to create arrays from scratch using internal NumPy functionality. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an array filled with a linear sequence starting at 0 and ending at 40, step size = 4\n",
    "array = np.arange(0,41,4) # Note that np.arange defines intervals [a,b), i.e. the upper limit is not included\n",
    "array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an array of 20 values evenly distributed between 10 and 40\n",
    "array = np.linspace(10,40,20)\n",
    "array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an array of dimension 4x5 filled with zeros\n",
    "np.zeros((4,5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an array of dimension 2x5 of random values between 0 and 1 (uniform distribution)\n",
    "# Note that numbers change if you execute cell again\n",
    "np.random.random((2,5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an array of dimension 3x3x4 of random integers between [5,12)\n",
    "# Note that numbers change if you execute cell again\n",
    "np.random.randint(5,12,(3,3,4))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking shape and size of an array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `.shape` returns the dimension of the array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = np.random.randint(10,20,(4,3))\n",
    "matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix_3d = np.zeros((4,5,6))\n",
    "matrix_3d.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `.size` function returns the size (i.e. the number of elements)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix_3d.size # 4 x 5 x 6 = 120"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accessing arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy arrays can be accessed via indexing and slicing.\n",
    "\n",
    "**Note**: as with lists the first element is at index level 0!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([6,1,42,2,8,9])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x[0] # get 0th element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x[2] # get second element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x[-2] # get second last element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multidimensional arrays can be accessed using indices separated by commas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([[1,2,3],[6,5,4],[100,200,300]])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x[0,2] # get element in 0th row and 2nd column "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x[2,0] # get element in 2nd row and 0th column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Value assigment on existing numpy arrays using the above index rules is possible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x[2,0] = 123\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slicing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy arrays can be sliced meaning taking elements from index $i_{start}$ to $i_{end}$.\n",
    "\n",
    "Instead of a single index we simply pass a slice: `start:end` or `start:end:step`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr = np.arange(0,21)\n",
    "arr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr[2:4] # get elements 2 and 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr[:5] # get first 5 elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr[-5:] # get last 5 elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr[5:16:2] # between elements 5 and 16 get every second element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same can be done using multidimensinal arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat = np.random.randint(0,10,(5,3))\n",
    "mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[:2,2] # from rows 0 and 1 get the second column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[1,:] # from row 1 get all columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[2:4,1:3] # from rows 2 and 3 get columns 1 and 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reshaping of arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes it is necessary to reshape arrays. Here are some examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(1,21,1) # create array of shape (20,), i.e. a column vector with 20 elements\n",
    "x, x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = x.reshape((1,20)) # convert column vector into row vector\n",
    "x, x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = x.reshape(5,4) # convert vector into 5x4 matrix\n",
    "x, x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Concatenation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can join two arrays in NumPy several ways. The three most useful functions are: `np.concatenate`, `np.vstack`, `np.hstack`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([10,11,12])\n",
    "b = np.array([100,200,300])\n",
    "c = np.random.random(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can use `np.concatenate` to join both arrays by passing both of them in a list (or tuple)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.concatenate([a,b])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also concatenate more than two arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.concatenate((a,b,c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When working with arrays of different dimensions you can use `np.vstack` and `np.hstack`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1,2,3])\n",
    "b = np.array([[4,5,6], [7,8,9], [10,11,12]])\n",
    "a, b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.vstack((a,b)) # Stack arrays vertical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = a.reshape(3,1) # Column vector\n",
    "np.hstack((b,a)) # stack arrays horizontally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Splitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of joining you may want to split arrays. The tree most useful functions are: `np.split`, `np.vsplit`, `np.hsplit`. \n",
    "\n",
    "All functions basically do the opposite of their concatenation counterparts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.arange(1,10,1)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1, a2 = np.split(a,[3])\n",
    "a1, a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1, a2, a3 = np.split(a,[4,7])\n",
    "a1,a2,a3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = np.arange(0,16).reshape((4, 4))\n",
    "matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.vsplit(matrix,[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.hsplit(matrix,[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculating with arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using NumPy for calculations has many advantages. Especially it is much faster and often more convenient than using lists and for loops. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using lists calculations can become quite tedious. \n",
    "\n",
    "Example: let's take a vector, add a scalar to that vector (elementwise) and then take the sum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [1,2,3,4,5]\n",
    "data_plus_5 = [i + 5 for i in data]\n",
    "sum(data_plus_5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is much easier and more intuitive in NumPy as we are not required to use for loops but instead apply the operation \"+ 5\" to each element of the vector automatically (note: this is done using vectorization)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.array([1,2,3,4,5])\n",
    "sum(data + 5) # alternatively you can also use np.sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turns out this approach is also much faster than using for loops (in below example iit is roughly $1000x$ times faster)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.arange(1,10_000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%timeit np.sum(data + 5) # takes about 10 micro seconds (10/1000000 seconds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_5(data):\n",
    "    output = np.empty(len(data))\n",
    "    for i in range(len(data)):\n",
    "        output[i] = data[i] + 5\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%timeit add_5(data) # takes 3.7 milli seconds (3.7/1000 seconds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arithmetic operators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can use all reguar arithmetic operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data = np.arange(10)\n",
    "print(\"data\", data)\n",
    "print(\"data + 5\", data + 5)\n",
    "print(\"data - 5\", data - 5)\n",
    "print(\"data * 5\", data * 5)\n",
    "print(\"data / 5\", data / 5)\n",
    "print(\"data ^ 5\", data ** 5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exponents and logarithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "x = np.array([1,2,3])\n",
    "print(\"x    =\", x)\n",
    "print(\"e^x  =\", np.exp(x))\n",
    "print(\"5^x  =\", np.power(5,x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "x = np.array([1,2,4,10])\n",
    "print(\"x        =\", x)\n",
    "print(\"ln(x)    =\", np.log(x))\n",
    "print(\"log2(x)  =\", np.log2(x))\n",
    "print(\"log10(x) =\", np.log10(x))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "**Note** Recap logarithm:\n",
    "\n",
    "$$a^b = c$$\n",
    "\n",
    "where:\n",
    "\n",
    "$a = $ base\n",
    "\n",
    "$b = $ exponent\n",
    "\n",
    "Logarithm:\n",
    "\n",
    "$$ log_a(c) = b$$\n",
    "\n",
    "**Example**:\n",
    "\n",
    "$$2^5 = 32$$\n",
    "\n",
    "$$log_2(32) = 5$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.log2(32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arithmetics on multiple arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(1,10)\n",
    "y = np.arange(11,20)\n",
    "x + y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregating values in an array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([1,2,3,4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(x) # 1 + 2 + 3 + 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.prod(x) # 1 * 2 * 3 * 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.min(x) # Minimum value of x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.max(x) # Maximum value of x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argmin(x) # index where minimum value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argmax(x) # index where maximum value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.std(x) # Standard deviation (of population)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.std(x,ddof=1) # Standard deviation (of sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For multidimensional arrays these functions can also be used and be applied to either rows or columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.arange(1,101).reshape(20,5)\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(X,axis=0) # take sum for each column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(X, axis=1) # take sum for each row"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: calculate z-score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's assume we have tabular data with 5 columns. Each column represents a certain feature each of which we want to normalize using a **z-score**.\n",
    "\n",
    "For this we need:\n",
    "\n",
    "- to know the mean and the standard deviation for each column \n",
    "- to apply (column specific) mean and standard deviation to each element of a column\n",
    "\n",
    "Using NumPy we can do this in one line of code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.random.random((20,5)) # Define data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "zscores = (X - X.mean(axis=0)) / X.std(axis=0)\n",
    "zscores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Mean by column = \", zscores.mean(axis=0)) # should be ~0\n",
    "print(\"Standard deviation by column = \", zscores.std(axis=0)) # should be ~1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparisons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boolean masks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you need to analyse an arrays with respect to some kind of criterion, masking helps. For example, you might wish to count all values greater than a certain value, or perhaps remove all outliers that are above some threshold. In NumPy, boolean masking is often the most efficient way to accomplish these types of tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create some random data to show how boolean masking works. \n",
    "\n",
    "Here: we create an array with 10_000 elements. Each element is chosen based on a draw from a sample of number $1, 2, \\ldots, 5$ with probability $p$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.random.choice([1,2,3,4,5], size=10_000, p=[0.2,0.1,0.1,0.3,0.3])\n",
    "data, data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's count the elements that are $\\geq 3$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(data >=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data >= 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's determine the sum of elements that are equal to 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(data[data == 4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many numbers are $\\neq 4$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(data != 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are there any numbers $> 5$ in the array?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.any(data > 5) # No!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.all(data <= 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this works also for multidimensional data. \n",
    "\n",
    "Let's reshape the data to a $2.000 \\times 5$ dimension and count number of elements equal to 2 per column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data.reshape(2000,5)\n",
    "np.sum(data==2, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data==2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boolean operators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use **boolean operators** if we have more than one condition. This is very helpfull when putting conditions on our data. Let's consider the three most useful ones:\n",
    "\n",
    "- AND `&`\n",
    "- OR `|`\n",
    "- NOT `~` (negation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.random.randint(1,11,size=100_000) # Random data between 1 and 10\n",
    "data[:5] # first 5 entries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's count the number of elements that are *above 3* `AND` *below 6*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cond = (data > 3) & (data < 6)\n",
    "np.sum(cond)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: given we are looking for the number of elements that are $4$ or $5$ this should sum up roughly to $2/10$ of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(cond) / len(cond)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's find the number of elements that have the value $1$ `OR` $7$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cond = (data == 1) | (data == 7)\n",
    "np.sum(cond)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take the opposite of the above condition and count the number of elements that are `NOT` $1$ `AND` `NOT`$7$. We can do this negating the above statement using `~`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(~cond)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check if the result makes sense. Together both statements should result in the entire data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(cond) + np.sum(~cond) == 100_000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using structured data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: one can use numpy to also work with structured data. Think of structured data as an excel spreadsheet with column headers. \n",
    "\n",
    "Let's read in a csv file with salaries data. The data set contains:\n",
    "\n",
    "- three columns with 100 elements each\n",
    "- columns represent work experience\n",
    "    - column 1 (0-5 years)\n",
    "    - column 2 (6-10 years)\n",
    "    - column 3 (>10 years)\n",
    "    \n",
    "> **NOTE:** we will later on use another library called `pandas` that is build on numpy and handels structured data even better"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"./data/salaries.csv\"\n",
    "salaries_by_experience = np.recfromcsv(path)\n",
    "salaries_by_experience"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = salaries_by_experience.dtype\n",
    "columns.names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can than access each individual column by its name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "salaries_by_experience[\"610yrs\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the mean salaries per experience group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = columns.names\n",
    "means = [np.mean(salaries_by_experience[n]) for n in names]\n",
    "means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Advanced** We can make this even more flexible and write a function that takes another function as an input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_stat(arr,f):\n",
    "    names = arr.dtype.names\n",
    "    return [f(arr[n]) for n in names]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calc_stat(salaries_by_experience, np.mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calc_stat(salaries_by_experience, np.std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def trimmed_mean(arr):\n",
    "    low, up = np.percentile(arr,5), np.percentile(arr,95)\n",
    "    return np.mean(arr[(arr>low) & (arr< up)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calc_stat(salaries_by_experience, trimmed_mean)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sneak Preview to `pandas`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above can be handeled even easier by using `pandas`. Pandas is built on numpy the functionality is very similar. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"./data/salaries.csv\"\n",
    "data = pd.read_csv(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.plot(kind=\"hist\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.plot(kind=\"line\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}