featureScaling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Transformation: Feature scaling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Data transformation is one of the fundamental steps in the part of data processing. In the first place you may think that  the terms scale, standardise, and normalise may be used interchangeably. However, it was pretty hard to find information about which of them we should use and also when to use. Therefore, I’m going to explain the following key aspects:\n",
    "\n",
    "- The difference between Standardisation and Normalisation\n",
    "- When to use Standardisation and when to use Normalisation\n",
    "- How to apply feature scaling in Python\n",
    "\n",
    "## Definitions\n",
    "\n",
    "### Feature Scaling\n",
    "\n",
    "In practice, we often encounter different types of variables in the same dataset. A significant issue is that the range of the variables may differ a lot. Using the original scale may put more weights on the variables with a large range. In order to deal with this problem, we need to apply the technique of features rescaling to independent variables or features of data in the step of data pre-processing. The terms normalisation and standardisation are sometimes used interchangeably, but they usually refer to different things.\n",
    "\n",
    "- The goal of applying Feature Scaling is to make sure features are on almost the same scale so that each feature is equally important and make it easier to process by most ML algorithms.\n",
    "\n",
    "#### Rescaling\n",
    "a vector means to add or subtract a constant and then multiply or divide by a constant, as you would do to change the units of measurement of the data, for example, to convert a temperature from Celsius to Fahrenheit.\n",
    "\n",
    "#### Normalizing\n",
    "a vector most often means dividing by a norm of the vector. It also often refers to rescaling by the minimum and range of the vector, to make all the elements lie between 0 and 1 thus bringing all the values of numeric columns in the dataset to a common scale.\n",
    "\n",
    "#### Standardizing \n",
    "a vector most often means subtracting a measure of location and dividing by a measure of scale. For example, if the vector contains random values with a Gaussian distribution, you might subtract the mean and divide by the standard deviation, thereby obtaining a “standard normal” random variable with mean 0 and standard deviation 1.\n",
    "\n",
    "#### Note 1 : \n",
    "The other issue is that standardization has two elements: Centering and scaling. Division by standard deviation is a type of scaling. Subtraction of mean is a type of centering. You don't have to use those. You could use median and Gini mean difference, for example. You can center without scaling and scale without centering.\n",
    "\n",
    "#### Note 2 : \n",
    "If an algorithm is not distance-based, feature scaling is unimportant, including Naive Bayes, Linear Discriminant Analysis, and Tree-Based models (gradient boosting, random forest, etc.).\n",
    "\n",
    "### Example\n",
    " \n",
    "Let's considere a dataset that contains an independent variable (Purchased) and 3 dependent variables (Country, Age, and Salary). We can easily notice that the variables are not on the same scale because the range of Age is from 27 to 50, while the range of Salary going from 48 K to 83 K. The range of Salary is much wider than the range of Age. This will cause some issues in our models since a lot of machine learning models such as k-means clustering and nearest neighbour classification are based on the Euclidean Distance.\n",
    "\n",
    "## Z-score standardization or Min-Max scaling\n",
    "\n",
    "There is no obvious answer to this question: it really depends on the application.\n",
    "\n",
    "Some machine learning models are fundamentally based on distance matrix, also known as the distance-based classifier, for example, K-Nearest-Neighbours, SVM, and Neural Network. Feature scaling is extremely essential to those models, especially when the range of the features is very different. Otherwise, features with a large range will have a large influence in computing the distance.\n",
    "\n",
    "Max-Min Normalisation typically allows us to transform the data with varying scales so that no specific dimension will dominate the statistics, and it does not require making a very strong assumption about the distribution of the data, such as k-nearest neighbours and artificial neural networks. However, Normalisation does not treat outliners very well. On the contrary, standardisation allows users to better handle the outliers and facilitate convergence for some computational algorithms like gradient descent. Therefore, we usually prefer standardisation over Min-Max Normalisation.\n",
    "\n",
    "For example, in clustering analyses, standardization may be especially crucial in order to compare similarities between features based on certain distance measures. Another prominent example is the Principal Component Analysis, where we usually prefer standardization over Min-Max scaling, since we are interested in the components that maximize the variance (depending on the question and if the PCA computes the components via the correlation matrix instead of the covariance matrix.\n",
    "\n",
    "However, this doesn’t mean that Min-Max scaling is not useful at all! A popular application is image processing, where pixel intensities have to be normalized to fit within a certain range (i.e., 0 to 255 for the RGB color range). Also, typical neural network algorithm require data that on a 0-1 scale.\n",
    "\n",
    "\n",
    "## Conclusions\n",
    "- When to normalize the data and when to standardize the data depends purely on the context of the problem we are working and the scale of features that are required for that particular problem.\n",
    "- If we want all the features to have values in the range [0,1], we go for normalization and if we want all the features with mean-centered variance scaling, we go for standardization.\n",
    "- It is recommended to go for Standardization because most of the models give better results when they are trained on standardized data over normalized data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "class FeatureScaling:\n",
    "    def __init__(self, df):\n",
    "        self.df = df\n",
    "    \n",
    "    \n",
    "    def standardize(self):\n",
    "        stdValues = self.df.std(axis=0)\n",
    "        mean = self.df.mean(axis=0)\n",
    "        self.df = (self.df-self.df.mean())/self.df.std()\n",
    "        return {\"df\":self.df,'mean':mean,'std':stdValues}\n",
    "    \n",
    "    def normalize(self):\n",
    "        minValues = self.df.min(axis=0)\n",
    "        maxValues = self.df.max(axis=0)\n",
    "        self.df = (self.df-minValues)/(maxValues-minValues)\n",
    "        return {'df':self.df,'min':minValues,'max':maxValues}\n",
    "    \n",
    "    #Scaling using median and quantiles consists of subtracting the median to all the observations \n",
    "    #and then dividing by the interquartile difference. It Scales features using statistics that are robust to outliers.\n",
    "    def ScaleToMedianAndQuartiles(self):\n",
    "        medianValues = self.df.median()\n",
    "        scaleValues = self.df.quantile(0.75)-self.df.quantile(0.25)\n",
    "        self.df = (self.df-medianValues)/scaleValues\n",
    "        return {'df':self.df,'median':medianValues,'scale':scaleValues}\n",
    "\n",
    "    # In case, we want to use our trained model using scaled input values, which have to be within the same scaler, of course after storing the metrics\n",
    "    def standardizeInputs(self,inputs,mean,std):\n",
    "        scaledInputs = []\n",
    "        for i in range(len(inputs)):\n",
    "            scaledInputs.append((inputs[i] - mean[i]) / std[i])\n",
    "        return scaledInputs\n",
    "    \n",
    "    def normalizeInputs(self,inputs,metrics):\n",
    "        scaledInputs = []\n",
    "        for i in range(len(inputs)):\n",
    "            scaledInputs.append((inputs[i] - metrics['min'][i]) /( metrics['max'][i] - metrics['min'][i]))\n",
    "        return scaledInputs\n",
    "\n",
    "    def ScaleToMedianAndQuartilesInputs(self,inputs,metrics):\n",
    "        scaledInputs = []\n",
    "        for i in range(len(inputs)):\n",
    "            scaledInputs.append((inputs[i] - metrics['median'][i]) / metrics['scale'][i])\n",
    "        return scaledInputs\n",
    "           \n",
    "    def unstandardize(self,scaler):\n",
    "        self.df = (self.df*scaler[\"std\"])+scaler[\"mean\"]\n",
    "        return self.df\n",
    "    \n",
    "    def unnormalize(self,scaler):\n",
    "        self.df = (self.df*(scaler[\"max\"]-scaler[\"min\"])+scaler[\"min\"])\n",
    "        return self.df\n",
    "    \n",
    "    def unscaleMedianAndQuartiles(self,scaler):\n",
    "        self.df = (self.df*scaler[\"median\"])/scaler[\"scale\"]\n",
    "        return self.df\n",
    "    \n",
    "    def read(self):\n",
    "        return self.df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    A   B     C    D    E\n",
      "0  12  70    20   14  900\n",
      "1   7   2    16    3  600\n",
      "2  11  54  1000  888   10\n",
      "3   8   3     3  188  222\n",
      "4   4   2     8    6    8\n",
      "          A         B         C         D         E\n",
      "0  1.121719  1.320500 -0.428498 -0.539264  1.408998\n",
      "1 -0.436224 -0.729591 -0.437548 -0.568088  0.643238\n",
      "2  0.810130  0.838125  1.788652  1.750906 -0.862756\n",
      "3 -0.124635 -0.699443 -0.466959 -0.083327 -0.321619\n",
      "4 -1.370989 -0.729591 -0.455647 -0.560227 -0.867861\n",
      "[1.1217185151295603, 1.3204996492840566, -0.42849823341848875, -0.5392644052612866, 1.408997922734886]\n",
      "******************************\n",
      "      A     B       C      D      E\n",
      "0  12.0  70.0    20.0   14.0  900.0\n",
      "1   7.0   2.0    16.0    3.0  600.0\n",
      "2  11.0  54.0  1000.0  888.0   10.0\n",
      "3   8.0   3.0     3.0  188.0  222.0\n",
      "4   4.0   2.0     8.0    6.0    8.0\n"
     ]
    }
   ],
   "source": [
    "dff = pd.DataFrame({\"A\":[12, 7, 11, 8, 4], \n",
    "                   \"B\":[70, 2, 54, 3, 2], \n",
    "                   \"C\":[20, 16, 1000, 3, 8], \n",
    "                   \"D\":[14, 3, 888, 188, 6],\n",
    "                   \"E\":[900, 600, 10, 222, 8]}) \n",
    "\n",
    "print(dff)\n",
    "\n",
    "sc = FeatureScaling(dff)\n",
    "inp = [12, 70, 20, 14, 900]\n",
    "scaler = sc.standardize()\n",
    "print(scaler['df'])\n",
    "print(sc.standardizeInputs(inp,scaler[\"mean\"],scaler['std']))\n",
    "print('******************************')\n",
    "sc.unstandardize(scaler)\n",
    "print(sc.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    A   B     C    D    E\n",
      "0  12  70    20   14  900\n",
      "1   7   2    16    3  600\n",
      "2  11  54  1000  888   10\n",
      "3   8   3     3  188  222\n",
      "4   4   2     8    6    8\n",
      "      A         B          C         D         E\n",
      "0  1.00  1.288462   0.333333  0.000000  1.149153\n",
      "1 -0.25 -0.019231   0.000000 -0.060440  0.640678\n",
      "2  0.75  0.980769  82.000000  4.802198 -0.359322\n",
      "3  0.00  0.000000  -1.083333  0.956044  0.000000\n",
      "4 -1.00 -0.019231  -0.666667 -0.043956 -0.362712\n",
      "**************************************************************\n",
      "[1.0, 1.2884615384615385, 0.3333333333333333, 0.0, 1.1491525423728814]\n",
      "     A         B           C         D         E\n",
      "0  2.0  0.074334    0.444444  0.000000  0.432393\n",
      "1 -0.5 -0.001109    0.000000 -0.004649  0.241069\n",
      "2  1.5  0.056583  109.333333  0.369400 -0.135203\n",
      "3  0.0  0.000000   -1.444444  0.073542  0.000000\n",
      "4 -2.0 -0.001109   -0.888889 -0.003381 -0.136478\n"
     ]
    }
   ],
   "source": [
    "dff = pd.DataFrame({\"A\":[12, 7, 11, 8, 4], \n",
    "                   \"B\":[70, 2, 54, 3, 2], \n",
    "                   \"C\":[20, 16, 1000, 3, 8], \n",
    "                   \"D\":[14, 3, 888, 188, 6],\n",
    "                   \"E\":[900, 600, 10, 222, 8]}) \n",
    "print(dff)\n",
    "sc = FeatureScaling(dff)\n",
    "scaler = sc.ScaleToMedianAndQuartiles()\n",
    "print(scaler['df'])\n",
    "inputs =[12, 70, 20, 14, 900]\n",
    "print('**************************************************************')\n",
    "print(sc.ScaleToMedianAndQuartilesInputs(inputs,scaler))\n",
    "print(sc.unscaleMedianAndQuartiles(scaler))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    A   B     C    D    E\n",
      "0  12  70    20   14  900\n",
      "1   7   2    16    3  600\n",
      "2  11  54  1000  888   10\n",
      "3   8   3     3  188  222\n",
      "4   4   2     8    6    8\n",
      "       A         B         C         D         E\n",
      "0  1.000  1.000000  0.017051  0.012429  1.000000\n",
      "1  0.375  0.000000  0.013039  0.000000  0.663677\n",
      "2  0.875  0.764706  1.000000  1.000000  0.002242\n",
      "3  0.500  0.014706  0.000000  0.209040  0.239910\n",
      "4  0.000  0.000000  0.005015  0.003390  0.000000\n",
      "[1.0, 1.0, 0.017051153460381142, 0.012429378531073447, 1.0]\n",
      "      A     B       C      D      E\n",
      "0  12.0  70.0    20.0   14.0  900.0\n",
      "1   7.0   2.0    16.0    3.0  600.0\n",
      "2  11.0  54.0  1000.0  888.0   10.0\n",
      "3   8.0   3.0     3.0  188.0  222.0\n",
      "4   4.0   2.0     8.0    6.0    8.0\n"
     ]
    }
   ],
   "source": [
    "dff = pd.DataFrame({\"A\":[12, 7, 11, 8, 4], \n",
    "                   \"B\":[70, 2, 54, 3, 2], \n",
    "                   \"C\":[20, 16, 1000, 3, 8], \n",
    "                   \"D\":[14, 3, 888, 188, 6],\n",
    "                   \"E\":[900, 600, 10, 222, 8]}) \n",
    "print(dff)\n",
    "sc = FeatureScaling(dff)\n",
    "scaler = sc.normalize()\n",
    "print(scaler['df'])\n",
    "inputs = [12, 70, 20, 14, 900]\n",
    "print(sc.normalizeInputs(inputs,scaler))\n",
    "sc.unnormalize(scaler)\n",
    "print(sc.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}