outliersHandling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handling Outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no censenus definition of outliers, but we can considere these data point as an observations that lies an abnormal distance from other values.\n",
    "\n",
    "Outlier can be of two types: Univariate and Multivariate. Above, we have discussed the example of univariate outlier. These outliers can be found when we look at distribution of a single variable. Multi-variate outliers are outliers in an n-dimensional space. In order to find them, you have to look at distributions in multi-dimensions.\n",
    "\n",
    "There is a bench of questions that should be asked when outliers are spotted, among these questions:\n",
    "#### 1) Is the outlier a mistke or legitimate point?\n",
    "#### 2) Is the outlier part of the population of interest?\n",
    "#### 3) Does the outlier have to be always dropped?\n",
    "#### 4) How to properly deal with the outliers?\n",
    "#### 5) What is the impact of outliers on a dataset?\n",
    "\n",
    "Sometimes outliers are simply caused by data recording errors, in other cases, outliers are legitimate observations. \n",
    "\n",
    "For the second question is whether the outlier is part of the population of interest. Depending on the answer to this question, we can decide whether outliers should be included in our analysis, which is the answer of the third question. \n",
    "\n",
    "There's no single solution. If an outlier is the result of data recording errors, we should recorrect the error if possible, otherwise we just remove it. If an outlier is outside of the population of interest, we should simply remove the outlier from further analysis. One should be cautious when removing outliers as removing them can sometime dramatically change the result of subsequent analysis. Outliers are not always outside of the population of interest. Sometimes they are actually the main focus of our analysis. As an example, it can be argued that all successful startups are outliers, as the success rate of startups is very low. However, in many analysis we're only interested in analyzing successful startups. \n",
    "\n",
    "As far as the last question, outliers can drastically change the results of the data analysis and statistical modeling. There are numerous unfavourable impacts of outliers in the data set:\n",
    "\n",
    "- It increases the error variance and reduces the power of statistical tests\n",
    "\n",
    "- If the outliers are non-randomly distributed, they can decrease normality\n",
    "\n",
    "- They can bias or influence estimates that may be of substantive interest\n",
    "\n",
    "- They can also impact the basic assumption of Regression, ANOVA and other statistical model assumptions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Identifying outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As far as identifying outliers, there is, of course, a degree of ambiguity. Qualifying a data point as an anomaly leaves it up to the analyst or model to determine what is abnormal and what to do with such data points.\n",
    "\n",
    "These outliers are typically easy to detect using straightforward methods like box plots, histograms and scatter-plots. In other cases, mathematical techniques are extremely valuable in fields which process large amounts of data and require a means to perform pattern recognition in larger datasets.\n",
    "\n",
    "\n",
    "### 2.1) Interquartile Range (IQR)\n",
    "The interquartile range (IQR) is a measure of statistical dispersion and is calculated as the difference between the 75th and 25th percentiles. It is represented by the formula IQR = Q3 − Q1, hence any value beyond the range of -1.5 x IQR to 1.5 x IQR may be considered as outlier.\n",
    "\n",
    "### 2.2) Skewness\n",
    "Several machine learning algorithms make the assumption that the data follow a normal (or Gaussian) distribution. This is easy to check with the skewness value, which explains the extent to which the data is normally distributed. Ideally, the skewness value should be between -1 and +1, and any major deviation from this range indicates the presence of outliers.\n",
    "\n",
    "### 2.3) Standard Deviation Method\n",
    "If a value is higher than the mean plus or minus three Standard Deviation is considered as outlier. It is based on the characteristics of a normal distribution for which 99.87% of the data appear within this range. \n",
    "\n",
    "This method has several shortcomings :\n",
    "- The mean and standard deviation are strongly affected by outliers.\n",
    "- It assumes that the distribution is normal (outliers included)\n",
    "- It does not detect outliers in small samples\n",
    "\n",
    "### 2.4) DBSCAN\n",
    "This technique is based on the DBSCAN clustering method. DBSCAN is a non-parametric, density based outlier detection method in a one or multi dimensional feature space.\n",
    "\n",
    "In the DBSCAN clustering technique, all data points are defined either as Core Points, Border Points or Noise Points.\n",
    "\n",
    "Core Points are data points that have at least MinPts neighboring data points within a distance ℇ.\n",
    "Border Points are neighbors of a Core Point within the distance ℇ but with less than MinPts neighbors within the distance ℇ.\n",
    "All other data points are Noise Points, also identified as outliers.\n",
    "Outlier detection thus depends on the required number of neighbors MinPts, the distance ℇ and the selected distance measure, like Euclidean or Manhattan.\n",
    "\n",
    "Dbscan pros:\n",
    "- It is a super effective method when the distribution of values in the feature space can not be assumed.\n",
    "- Works well if the feature space for searching outliers is multidimensional (ie. 3 or more dimensions)\n",
    "- Sci-kit learn’s implementation is easy to use and the documentation is superb.\n",
    "- Visualizing the results is easy and the method itself is very intuitive.\n",
    "\n",
    "Dbscan cons:\n",
    "- The values in the feature space need to be scaled accordingly.\n",
    "- Selecting the optimal parameters eps, MinPts and metric can be difficult since it is very sensitive to any of the three params.\n",
    "- It is an unsupervised model and needs to be re-calibrated each time a new batch of data is analyzed.\n",
    "- It can predict once calibrated but is strongly not recommended.\n",
    "\n",
    "#### Article ==> A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise\n",
    "\n",
    "### 2.5) Isolation Forest\n",
    "This is a non-parametric method for large datasets in a one or multi dimensional feature space.\n",
    "\n",
    "An important concept in this method is the isolation number.\n",
    "\n",
    "The isolation number is the number of splits needed to isolate a data point. This number of splits is ascertained by following these steps:\n",
    "\n",
    "A point “a” to isolate is selected randomly.\n",
    "A random data point “b” is selected that is between the minimum and maximum value and different from “a”.\n",
    "If the value of “b” is lower than the value of “a”, the value of “b” becomes the new lower limit.\n",
    "If the value of “b” is greater than the value of “a”, the value of “b” becomes the new upper limit.\n",
    "This procedure is repeated as long as there are data points other than “a” between the upper and the lower limit.\n",
    "It requires fewer splits to isolate an outlier than it does to isolate a non-outlier, i.e. an outlier has a lower isolation number in comparison to a non-outlier point. A data point is therefore defined as an outlier if its isolation number is lower than the threshold.\n",
    "\n",
    "The threshold is defined based on the estimated percentage of outliers in the data, which is the starting point of this outlier detection algorithm.\n",
    "#### An explanation with images ==> https://quantdare.com/isolation-forest-algorithm/.\n",
    "\n",
    "Isolation Forest pros:\n",
    "- There is no need of scaling the values in the feature space.\n",
    "- It is an effective method when value distributions can not be assumed.\n",
    "- It has few parameters, this makes this method fairly robust and easy to optimize.\n",
    "- Scikit-Learn’s implementation is easy to use and the documentation is superb.\n",
    "\n",
    "Isolation Forest cons:\n",
    "- The Python implementation exists only in the development version of Sklearn.\n",
    "- Visualizing results is complicated.\n",
    "- If not correctly optimized, training time can be very long and computationally expensive.\n",
    "\n",
    "### 2.6) Angle-Based Outlier Detection (ABOD)\n",
    "It considers the relationship between each point and its neighbor(s). It does not consider the relationships among these neighbors. The variance of its weighted cosine scores to all neighbors could be viewed as the outlying score\n",
    "ABOD performs well on multi-dimensional data\n",
    "PyOD provides two different versions of ABOD:\n",
    "\n",
    "- Fast ABOD: Uses k-nearest neighbors to approximate\n",
    "- Original ABOD: Considers all training points with high-time complexity\n",
    "\n",
    "### 2.7) k-Nearest Neighbors Detector\n",
    "For any data point, the distance to its kth nearest neighbor could be viewed as the outlying score\n",
    "PyOD supports three kNN detectors:\n",
    "\n",
    "- Largest: Uses the distance of the kth neighbor as the outlier score\n",
    "- Mean: Uses the average of all k neighbors as the outlier score\n",
    "- Median: Uses the median of the distance to k neighbors as the outlier score\n",
    "\n",
    "### 2.8) Local Correlation Integral (LOCI)\n",
    "- LOCI is very effective for detecting outliers and groups of outliers. It provides a LOCI plot for each point which summarizes a lot of the information about the data in the area around the point, determining clusters, micro-clusters, their diameters, and their inter-cluster distances\n",
    "- None of the existing outlier-detection methods can match this feature because they output only a single number for each point\n",
    "\n",
    "### 2.9) Automating outliers detection with SVM\n",
    "\n",
    "Support Vector Machines (SVM) is a powerful machine learning technique. OneClassSVM is an algorithm that specializes in learning the expected distributions in a dataset. OneClassSVM is especially useful as a novelty detector method if you can first provide data cleaned from outliers; otherwise, it’s effective as a detector of multivariate outliers. In order to have OneClassSVM work properly, you have two key parameters to fix:\n",
    "\n",
    "\n",
    "- gamma, telling the algorithm whether to follow or approximate the dataset distributions. For novelty detection, it is better to have a value of 0 or superior (follow the distribution); for outlier detection values, smaller than 0 values are preferred (approximate the distribution).\n",
    "\n",
    "- nu, which can be calculated by the following formula: nu_estimate = 0.95 * f + 0.05, where f is the percentage of expected outliers (a number from 1 to 0). If your purpose is novelty detection, f will be 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Dealing with outliers\n",
    "Should an outlier be removed from analysis? Should you keep outliers, or change them to another variable? The answer to these questions may seem straightforward, but isn’t so simple.\n",
    "\n",
    "There are many strategies for dealing with outliers in data. Depending on the situation and data set, any could be the right or the wrong way.\n",
    "### 3.1) Remove the outliers\n",
    "\n",
    "### 3.2) Change the value of outliers\n",
    "#### 3.2.1) Percentile Capping (Winsorization)\n",
    "In layman's terms, Winsorization (Winsorizing) at 1st and 99th percentile implies values that are less than the value at 1st percentile are replaced by the value at 1st percentile, and values that are greater than the value at 99th percentile are replaced by the value at 99th percentile. The winsorization at 5th and 95th percentile is also common.\n",
    "#### 3.2.2) Imputing\n",
    "\n",
    "#### 3.2.3) Transforming (Discretization)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.ensemble import IsolationForest\n",
    "from scipy import stats\n",
    "from pyod.models.abod import ABOD\n",
    "from pyod.models.cblof import CBLOF\n",
    "from pyod.models.feature_bagging import FeatureBagging\n",
    "from pyod.models.hbos import HBOS\n",
    "from pyod.models.iforest import IForest\n",
    "from pyod.models.knn import KNN\n",
    "from pyod.models.lof import LOF\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.cluster import DBSCAN\n",
    "from sklearn import svm\n",
    "\n",
    "\n",
    "class handlingOutliers:\n",
    "    def __init__(self, df):\n",
    "        self.df = df\n",
    "        \n",
    "    def isolationForest(self,column,n_estimators=100,outliers_fraction=0.01):\n",
    "        model=IsolationForest(n_estimators=n_estimators, max_samples='auto', contamination=float(outliers_fraction),max_features=1.0)\n",
    "        model.fit(self.df[[column]])\n",
    "        self.df['scores']=model.decision_function(self.df[[column]])\n",
    "        self.df['anomaly']=model.predict(self.df[[column]])\n",
    "        anomaly=self.df.loc[self.df['anomaly']==-1]\n",
    "        anomaly_index=list(anomaly.index)\n",
    "        return anomaly_index\n",
    "    \n",
    "    def KNN(self,outliers_fraction,column):\n",
    "        model = KNN(contamination=float(outliers_fraction))\n",
    "        model.fit(self.df[column].values.reshape(-1,1))\n",
    "        self.df['scores'] = model.decision_function(self.df[[column]])\n",
    "        self.df['anomaly']=model.predict(self.df[[column]])\n",
    "        anomaly=self.df.loc[self.df['anomaly']==1]\n",
    "        anomaly_index=list(anomaly.index)\n",
    "        return anomaly_index\n",
    "    \n",
    "    def predictOutliers(self,column1,column2,classifier,outliers_fraction=0.3,random_state=np.random.RandomState(100),n_neighbors=5,scale=True):      \n",
    "        classifiers = {\n",
    "        'Angle-based Outlier Detector (ABOD)': ABOD(contamination=float(outliers_fraction)),\n",
    "        'Cluster-based Local Outlier Factor (CBLOF)':CBLOF(contamination=float(outliers_fraction),check_estimator=False, random_state=random_state),\n",
    "        'Feature Bagging':FeatureBagging(LOF(n_neighbors=n_neighbors),contamination=float(outliers_fraction),check_estimator=False,random_state=random_state),\n",
    "        'Histogram-base Outlier Detection (HBOS)': HBOS(contamination=float(outliers_fraction)),\n",
    "        'Isolation Forest': IForest(contamination=float(outliers_fraction),random_state=random_state),\n",
    "        'K Nearest Neighbors (KNN)': KNN(contamination=float(outliers_fraction)),\n",
    "        'Average KNN': KNN(method='mean',contamination=float(outliers_fraction))\n",
    "         }\n",
    "        colPara = {column1:[self.df[column1].min(),self.df[column1].max()],column2:[self.df[column2].min(),self.df[column2].max()]}\n",
    "        if scale == True:\n",
    "            scaler = MinMaxScaler(feature_range=(0, 1))\n",
    "            self.df[[column1,column2]] = scaler.fit_transform(self.df[[column2,column2]])\n",
    "            \n",
    "        #multivariate outliers (2 columns like weigh and heigh)\n",
    "        X1 = df[column1].values.reshape(-1,1)\n",
    "        X2 = df[column2].values.reshape(-1,1)\n",
    "        X = np.concatenate((X1,X2),axis=1)\n",
    "        model = classifiers[classifier]\n",
    "        model.fit(X)\n",
    "        self.df['scores'] = model.decision_function(X)\n",
    "        self.df['anomaly']=model.predict(X)\n",
    "        anomaly=self.df.loc[self.df['anomaly']==1]\n",
    "        anomaly_index=list(anomaly.index)\n",
    "        return [anomaly_index,colPara]\n",
    "    \n",
    "    #EPS ==> The maximum distance between two samples for one to be considered as in the neighborhood of the other. \n",
    "    #This is not a maximum bound on the distances of points within a cluster. \n",
    "    #This is the most important DBSCAN parameter to choose appropriately for your data set and distance function.\n",
    "    \n",
    "    #min_samples ==> The number of samples (or total weight) in a neighborhood for a point to be considered as a core point.\n",
    "    #This includes the point itself.\n",
    "\n",
    "    def DBScan(self,eps=4,min_samples=2):\n",
    "        model = DBSCAN(eps=eps, min_samples=min_samples)\n",
    "        \n",
    "        float_col = self.df.select_dtypes(include=['float']) # This will select float columns only\n",
    "        for col in float_col.columns.values:\n",
    "            self.df[col] = self.df[col].round(0).astype(int)        \n",
    "        model.fit(self.df)\n",
    "        self.df['anomaly'] = model.labels_\n",
    "        return self.df[ self.df['anomaly'] == -1].index\n",
    "        \n",
    "    def OneClassSVM(self,outliers_fraction=0.01,gamma=0.1):\n",
    "        nu_estimate = 0.95 * outliers_fraction + 0.05\n",
    "        auto_detection = svm.OneClassSVM(kernel=\"rbf\", gamma=gamma, degree=3, nu=nu_estimate)\n",
    "        auto_detection.fit(self.df)\n",
    "        self.df['anomaly'] = auto_detection.predict(self.df)\n",
    "        anomaly=self.df.loc[self.df['anomaly']==-1]\n",
    "        anomaly_index=list(anomaly.index)\n",
    "        return anomaly_index\n",
    "    \n",
    "    def removeOutliers(self,anomaly_index):\n",
    "        self.df.drop(anomaly_index,inplace=True)\n",
    "        try:\n",
    "            self.df.drop(['anomaly'],axis=1,inplace=True)\n",
    "            self.df.drop(['scores'],axis=1,inplace=True)\n",
    "        except:\n",
    "            print(\"columns don't exist\")\n",
    "        \n",
    "    def imputeOutliers(self,anomaly_index,column):\n",
    "        self.df.loc[anomaly_index,column] = np.nan\n",
    "        try:\n",
    "            self.df.drop(['anomaly'],axis=1,inplace=True)\n",
    "            self.df.drop(['scores'],axis=1,inplace=True)        \n",
    "        except:\n",
    "            print(\"columns don't exist\")\n",
    "            \n",
    "    #factor ==> The common value for the factor k is the value 1.5. A factor k of 3 or more can be used to identify values \n",
    "    #that are extreme outliers or “far outs” when described in the context of box and whisker plots.       \n",
    "    def removeOutliersIQR(self,column,factor=1.5): \n",
    "        Q1=self.df[column].quantile(0.25)\n",
    "        Q3=self.df[column].quantile(0.75)\n",
    "        IQR=(Q3-Q1) * factor\n",
    "        Lower_Whisker = Q1 - IQR\n",
    "        Upper_Whisker = Q3 + IQR\n",
    "        self.df = self.df[self.df[column]< Upper_Whisker]\n",
    "        self.df = self.df[self.df[column]> Lower_Whisker]\n",
    "        return self.df\n",
    "\n",
    "    def removeOutliersZScore(self,column,threshold=3):\n",
    "        z_scores = stats.zscore(self.df)\n",
    "        abs_z_scores = np.abs(z_scores)\n",
    "        filtered_entries = (abs_z_scores < threshold).all(axis=1)\n",
    "        return self.df[filtered_entries]\n",
    "\n",
    "    def removeByTrimming(self,column,lowerW,upperW):\n",
    "        index = self.df[(self.df[column] >= upperW)|(self.df[column] <= lowerW)].index\n",
    "        self.df.drop(index, inplace=True)\n",
    "\n",
    "    def WinsorizeStats(self):\n",
    "        out = stats.mstats.winsorize(self.df, limits=[0.05, 0.05])\n",
    "        return out\n",
    "    \n",
    "    def replaceByMedian(self,anomaly_index,column):\n",
    "        self.df.loc[anomaly_index,column] = self.df[column].median()\n",
    "        try:\n",
    "            self.df.drop(['anomaly'],axis=1,inplace=True)\n",
    "            self.df.drop(['scores'],axis=1,inplace=True)\n",
    "        except:\n",
    "            print(\"columns are already removed\")\n",
    "\n",
    "    def replaceByMedianUp(self,column,upperWhisker):\n",
    "        self.df[column] = np.where(self.df[column] <upperWhisker, self.df[column].median(),self.df[column])\n",
    "\n",
    "    def replaceByMedianLow(self,column,lowerWhisker):\n",
    "        self.df[column] = np.where(self.df[column] >lowerWhisker, self.df[column].median(),self.df[column])\n",
    "\n",
    "\n",
    "    def read(self):\n",
    "        return self.df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    salary  age\n",
      "0      1.0   10\n",
      "1      2.0   11\n",
      "2      3.0   12\n",
      "3      4.0   15\n",
      "4      2.0   40\n",
      "5      4.0   90\n",
      "6      4.0    8\n",
      "7      3.0   20\n",
      "8      7.0   17\n",
      "9      4.0   19\n",
      "10     4.0   35\n"
     ]
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,35]})\n",
    "f = handlingOutliers(df)\n",
    "f.replaceByMedianLow(\"salary\",10)\n",
    "print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[6, 10]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,35]})\n",
    "f = handlingOutliers(df)\n",
    "f.isolationForest(\"salary\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    salary  age\n",
      "0        1   10\n",
      "1        2   11\n",
      "2        3   12\n",
      "3        4   15\n",
      "4        2   40\n",
      "5        4   90\n",
      "6      100    8\n",
      "7        3   20\n",
      "8        7   17\n",
      "9        4   19\n",
      "10     200   35\n",
      "columns don't exist\n",
      "None\n",
      "   salary  age\n",
      "1       2   11\n",
      "2       3   12\n",
      "3       4   15\n",
      "4       2   40\n",
      "7       3   20\n",
      "8       7   17\n",
      "9       4   19\n"
     ]
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,35]})\n",
    "f = handlingOutliers(df)\n",
    "print(df)\n",
    "print(f.removeOutliers(f.OneClassSVM()))\n",
    "print(f.read())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "columns don't exist\n",
      "   salary  age\n",
      "0       1   10\n",
      "1       2   11\n",
      "2       3   12\n",
      "3       4   15\n",
      "7       3   20\n",
      "8       7   17\n",
      "9       4   19\n"
     ]
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,35]})\n",
    "f = handlingOutliers(df)\n",
    "d = f.DBScan()\n",
    "f.removeOutliers(d)\n",
    "print(f.read())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    salary  age\n",
      "0        1   10\n",
      "1        2   11\n",
      "2        3   12\n",
      "3        4   15\n",
      "4        2   40\n",
      "5        4   90\n",
      "6      100    8\n",
      "7        3   20\n",
      "8        7   17\n",
      "9        4   19\n",
      "10     200   35\n",
      "None\n",
      "    salary  age\n",
      "0      1.0   10\n",
      "1      2.0   11\n",
      "2      3.0   12\n",
      "3      4.0   15\n",
      "4      2.0   40\n",
      "5      4.0   90\n",
      "6      4.0    8\n",
      "7      3.0   20\n",
      "8      4.0   17\n",
      "9      4.0   19\n",
      "10     4.0   35\n"
     ]
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,35]})\n",
    "f = handlingOutliers(df)\n",
    "print(df)\n",
    "ind =f.KNN(0.3,'salary')\n",
    "print(f.replaceByMedian(ind,'salary'))\n",
    "print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    salary  age\n",
      "0        1   10\n",
      "1        2   11\n",
      "2        3   12\n",
      "3        4   15\n",
      "4        2   40\n",
      "5        4   90\n",
      "6      100    8\n",
      "7        3   20\n",
      "8        7   17\n",
      "9        4   19\n",
      "10     200   35\n",
      "[90. 90.]\n",
      "okokokokokoko [0.01219512 0.01219512]\n",
      "[0, 2, 8]\n",
      "[1. 1.]\n",
      "okokokokokoko [1. 1.]\n",
      "[4, 5, 10]\n",
      "[1. 1.]\n",
      "okokokokokoko [1. 1.]\n",
      "[4, 5, 10]\n",
      "ABOD is done\n",
      "[1. 1.]\n",
      "okokokokokoko [1. 1.]\n",
      "[4, 5, 10]\n",
      "[1. 1.]\n",
      "okokokokokoko [1. 1.]\n",
      "[4, 5, 10]\n",
      "[1. 1.]\n",
      "okokokokokoko [1. 1.]\n",
      "[4, 5, 10]\n",
      "[1. 1.]\n",
      "okokokokokoko [1. 1.]\n",
      "[4, 5, 10]\n"
     ]
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,35]})\n",
    "f = handlingOutliers(df)\n",
    "print(df)\n",
    "print(f.predictOutliers('salary','age','Cluster-based Local Outlier Factor (CBLOF)'))\n",
    "print(f.predictOutliers('salary','age','Histogram-base Outlier Detection (HBOS)'))\n",
    "print(f.predictOutliers('salary','age','Angle-based Outlier Detector (ABOD)'))\n",
    "print(\"ABOD is done\")\n",
    "print(f.predictOutliers('salary','age','Feature Bagging'))\n",
    "print(f.predictOutliers('salary','age','Isolation Forest'))\n",
    "print(f.predictOutliers('salary','age','K Nearest Neighbors (KNN)'))\n",
    "print(f.predictOutliers('salary','age','Average KNN'))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "masked_array(\n",
       "  data=[[  2,  10],\n",
       "        [  2,  11],\n",
       "        [  3,  12],\n",
       "        [  4,  15],\n",
       "        [  2,  40],\n",
       "        [  4,  90],\n",
       "        [100,   8],\n",
       "        [  3,  20],\n",
       "        [  7,  17],\n",
       "        [  4,  19],\n",
       "        [200, 200]],\n",
       "  mask=False,\n",
       "  fill_value=999999,\n",
       "  dtype=int64)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,350]})\n",
    "f = handlingOutliers(df)\n",
    "f.WinsorizeStats()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    salary  age\n",
      "0      1.0   10\n",
      "1      2.0   11\n",
      "2      3.0   12\n",
      "3      4.0   15\n",
      "4      2.0   40\n",
      "5      4.0   90\n",
      "6      4.0    8\n",
      "7      3.0   20\n",
      "8      4.0   17\n",
      "9      4.0   19\n",
      "10     4.0  350\n"
     ]
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,350]})\n",
    "f = handlingOutliers(df)\n",
    "ind =f.KNN(0.3,'salary')\n",
    "f.replaceByMedian(ind,\"salary\")\n",
    "print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   salary  age\n",
      "1       2   11\n",
      "2       3   12\n",
      "3       4   15\n",
      "4       2   40\n",
      "5       4   90\n",
      "7       3   20\n",
      "8       7   17\n",
      "9       4   19\n"
     ]
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,35]})\n",
    "f = handlingOutliers(df)\n",
    "f.removeByTrimming(\"salary\",1,10)\n",
    "print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>4</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   salary  age\n",
       "0       1   10\n",
       "1       2   11\n",
       "2       3   12\n",
       "3       4   15\n",
       "4       2   40\n",
       "5       4   90\n",
       "7       3   20\n",
       "8       7   17\n",
       "9       4   19"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,35]})\n",
    "f = handlingOutliers(df)\n",
    "f.removeOutliersIQR(\"salary\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>100</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>4</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   salary  age\n",
       "0       1   10\n",
       "1       2   11\n",
       "2       3   12\n",
       "3       4   15\n",
       "4       2   40\n",
       "5       4   90\n",
       "6     100    8\n",
       "7       3   20\n",
       "8       7   17\n",
       "9       4   19"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,350]})\n",
    "f = handlingOutliers(df)\n",
    "f.removeOutliersZScore(\"salary\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>100</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>4</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>200</td>\n",
       "      <td>350</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    salary  age\n",
       "0        1   10\n",
       "1        2   11\n",
       "2        3   12\n",
       "3        4   15\n",
       "4        2   40\n",
       "5        4   90\n",
       "6      100    8\n",
       "7        3   20\n",
       "8        7   17\n",
       "9        4   19\n",
       "10     200  350"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.DataFrame({'salary':[1,2,3,4,2,4,100,3,7,4,200],'age':[10,11,12,15,40,90,8,20,17,19,350]})\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}