featureSelection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) What is Feature?\n",
    "Consider an example of data stored in sql database tables. Table is made up of\n",
    "rows and columns. Table contains Integer data, String data, Date fields etc. Now\n",
    "consider the date column. We want to do some analysis, but this field is not directly\n",
    "useful. So, first we write a program (or script) to extract day on any particular date\n",
    "and create a separate column with that information. Now we have 7 days (Monday,\n",
    "Sunday etc.) stored in a new column. Again, we create a script to check: is a day is\n",
    "weekend or a weekday. We create another field is_weekend. It contains True if the\n",
    "day is weekend otherwise False. We can use this for our analysis or to build\n",
    "predictive model. This is Feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Why Feature Engineering?\n",
    "In Machine Learning to get success in any modelling technique\n",
    "we need good features in the data. Features are very important to increase the\n",
    "predictive power of any model. When we try to solve real world problem, we may\n",
    "not always get the best features. As, Features may exist with a lot of problems like\n",
    "missing values, outlier, different type, error in data collection etc. Before training\n",
    "any machine learning model, we have to clean, transform and find right set of\n",
    "features.\n",
    "We can take an example of cooking food. We have given a lot of ingredient and we\n",
    "have to find which is needed to solve our problem. Without selection of right\n",
    "ingredient, we cannot make perfect food. Same is feature engineering in data\n",
    "science process.\n",
    "Data Scientist while solving any problem, spend more than half of their time in\n",
    "selecting the right features. Some of the benefits of spending time in feature\n",
    "engineering:\n",
    "1. Make simple model to perform much better than complex model.\n",
    "2. Reduce model selection time.They increase predictive power of simple\n",
    "models.\n",
    "3. Reduce training time by simplifying the model.\n",
    "It is all about decomposing the data in an intelligent fashion. We will discuss all the\n",
    "important aspects of feature engineering in this chapter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Feature Selection\n",
    "Feature selection are the techniques to select subset of the features from the data.\n",
    "It is different from the feature extracted where we have created new features. In\n",
    "this we find most useful features from the data itself. Feature selection is important\n",
    "because:\n",
    "1. Reduces Training Time: Model creation is faster due to less features.\n",
    "2. Easy to explain and interpret features.\n",
    "3. Better Generalized model: Predictive model behaves nearly same on generalized data.\n",
    "4. Reduces Overfitting: Less redundant data means less opportunity to make decisions based on noise.\n",
    "5. Improves Accuracy: Less misleading data means modeling accuracy improves.\n",
    "\n",
    "Feature selection is basically a search problem. We have to find ways to select\n",
    "features which can produce better results. Different techniques which are used in\n",
    "feature selection are:\n",
    "\n",
    "#### 1. Filter Methods: \n",
    "These methods are based on score of features by some\n",
    "statistical test. Each feature is evaluated with the outcome on statistical test\n",
    "(like pearson correlation, chi-squared test etc.) and a score is generated. Later\n",
    "features are ranked based on their score and lower score features are removed.\n",
    "\n",
    "#### 2. Wrapper Methods: \n",
    "These methods use machine learning algorithm to find\n",
    "the best feature. First, different subsets of features are created. Now with\n",
    "these set of features machine learning algorithm is trained on sample data\n",
    "and model performance is evaluated. The set of features which gives best\n",
    "performance are considered as selected features which are unconcerned with the variable types. These methods will take\n",
    "more time because they do actual training of algorithm with different set of\n",
    "features. RFE is a good example of a wrapper feature selection method.\n",
    "\n",
    "#### 3. Intrinsic feature selection methods\n",
    "There are some machine learning algorithms that perform feature selection automatically as part of learning the model. We might refer to these techniques as intrinsic feature selection methods.\n",
    "This includes algorithms such as penalized regression models like Lasso and decision trees, including ensembles of decision trees like random forest.\n",
    "\n",
    " #### Note: These previous methods are almost always supervised and are evaluated based on the performance of a resulting model on a hold out dataset.\n",
    "\n",
    "#### 4. Embedded methods: \n",
    "These methods will select best feature during the training itself. Some of the\n",
    "examples are ridge and lasso regression which we will see in later chapters.\n",
    "\n",
    "\n",
    "One way to think about feature selection methods are in terms of supervised and unsupervised methods.\n",
    "The difference has to do with whether features are selected based on the target variable or not. Unsupervised feature selection techniques ignores the target variable, such as methods that remove redundant variables using correlation. Supervised feature selection techniques use the target variable, such as methods that remove irrelevant variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistics for Filter-Based Feature Selection Methods\n",
    "\n",
    "It is common to use correlation type statistical measures between input and output variables as the basis for filter feature selection.\n",
    "\n",
    "Common data types include numerical (such as height) and categorical (such as a label), although each may be further subdivided such as integer and floating point for numerical variables, and boolean, ordinal, or nominal for categorical variables.\n",
    "\n",
    "Common input variable data types:\n",
    "#### Numerical Variables\n",
    "- Integer Variables.\n",
    "- Floating Point Variables.\n",
    "\n",
    "#### Categorical Variables.\n",
    "- Boolean Variables (dichotomous).\n",
    "- Ordinal Variables.\n",
    "- Nominal Variables.\n",
    "\n",
    "The more that is known about the data type of a variable, the easier it is to choose an appropriate statistical measure for a filter-based feature selection method.\n",
    "\n",
    "In this section, we will consider two broad categories of variable types: numerical and categorical; also, the two main groups of variables to consider: input and output.\n",
    "\n",
    "Input variables are those that are provided as input to a model. In feature selection, it is this group of variables that we wish to reduce in size. Output variables are those for which a model is intended to predict, often called the response variable.\n",
    "\n",
    "The type of response variable typically indicates the type of predictive modeling problem being performed. For example, a numerical output variable indicates a regression predictive modeling problem, and a categorical output variable indicates a classification predictive modeling problem.\n",
    "\n",
    "- Numerical Output: Regression predictive modeling problem.\n",
    "- Categorical Output: Classification predictive modeling problem.\n",
    "\n",
    "The statistical measures used in filter-based feature selection are generally calculated one input variable at a time with the target variable. As such, they are referred to as univariate statistical measures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Numerical Input, Numerical Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pearson’s correlation coefficient (linear)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import stats\n",
    "from scipy.stats import chi2_contingency\n",
    "import statsmodels.api as sm\n",
    "from statsmodels.formula.api import ols\n",
    "from sklearn.feature_selection import RFE\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from statsmodels.stats.multicomp import (pairwise_tukeyhsd,\n",
    "                                         MultiComparison)\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.model_selection import RepeatedStratifiedKFold\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.model_selection import cross_val_score\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>mpg</th>\n",
       "      <th>cyl</th>\n",
       "      <th>disp</th>\n",
       "      <th>hp</th>\n",
       "      <th>drat</th>\n",
       "      <th>wt</th>\n",
       "      <th>qsec</th>\n",
       "      <th>vs</th>\n",
       "      <th>am</th>\n",
       "      <th>gear</th>\n",
       "      <th>carb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mazda RX4</td>\n",
       "      <td>21.0</td>\n",
       "      <td>6</td>\n",
       "      <td>160.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.90</td>\n",
       "      <td>2.620</td>\n",
       "      <td>16.46</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mazda RX4 Wag</td>\n",
       "      <td>21.0</td>\n",
       "      <td>6</td>\n",
       "      <td>160.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.90</td>\n",
       "      <td>2.875</td>\n",
       "      <td>17.02</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Datsun 710</td>\n",
       "      <td>22.8</td>\n",
       "      <td>4</td>\n",
       "      <td>108.0</td>\n",
       "      <td>93</td>\n",
       "      <td>3.85</td>\n",
       "      <td>2.320</td>\n",
       "      <td>18.61</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Hornet 4 Drive</td>\n",
       "      <td>21.4</td>\n",
       "      <td>6</td>\n",
       "      <td>258.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3.08</td>\n",
       "      <td>3.215</td>\n",
       "      <td>19.44</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Hornet Sportabout</td>\n",
       "      <td>18.7</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>175</td>\n",
       "      <td>3.15</td>\n",
       "      <td>3.440</td>\n",
       "      <td>17.02</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  \\\n",
       "0          Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4   \n",
       "1      Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4   \n",
       "2         Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4   \n",
       "3     Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3   \n",
       "4  Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3   \n",
       "\n",
       "   carb  \n",
       "0     4  \n",
       "1     4  \n",
       "2     1  \n",
       "3     1  \n",
       "4     2  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars = pd.read_csv(\"mtcars.csv\")\n",
    "numCols = [\"cyl\",\"disp\",\"hp\",\"drat\",\"wt\",\"qsec\",\"vs\",\"am\",\"gear\",\"carb\"]\n",
    "numOutput = cars[\"mpg\"]\n",
    "cars.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pearsonrCorrelation(df,numInputCols,numOutput,numOfFeatu):\n",
    "    cofList = []\n",
    "    for inputCol in numInputCols:\n",
    "        pearsonr_cof, p_value = stats.pearsonr(numOutput,df[inputCol])\n",
    "        cofList.append((inputCol,pearsonr_cof))\n",
    "        cofList.sort(key=lambda col:col[1])\n",
    "    return cofList[:numOfFeatu]   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('wt', -0.8676593765172279),\n",
       " ('cyl', -0.8521619594266132),\n",
       " ('disp', -0.8475513792624787)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pearsonrCorrelation(cars,numCols,numOutput,3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spearman’s rank coefficient (nonlinear)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def spearmanrCorrelation(df,numInputCols,numOutput,numOfFeatu):\n",
    "    cofList = []\n",
    "    for inputCol in numInputCols:\n",
    "        pearsonr_cof, p_value = stats.spearmanr(numOutput,df[inputCol])\n",
    "        cofList.append((inputCol,pearsonr_cof))\n",
    "        cofList.sort(key=lambda col:col[1])\n",
    "    return cofList[:numOfFeatu]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('cyl', -0.9108013108624786),\n",
       " ('disp', -0.9088823637364655),\n",
       " ('hp', -0.8946646457499626)]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spearmanrCorrelation(cars,numCols,numOutput,3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numerical Input, Categorical Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ANOVA correlation coefficient (linear)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>weight</th>\n",
       "      <th>group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>6.03</td>\n",
       "      <td>trt1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>29</td>\n",
       "      <td>5.80</td>\n",
       "      <td>trt2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>3.59</td>\n",
       "      <td>trt1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>30</td>\n",
       "      <td>5.26</td>\n",
       "      <td>trt2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>3.83</td>\n",
       "      <td>trt1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>5.14</td>\n",
       "      <td>ctrl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22</td>\n",
       "      <td>5.12</td>\n",
       "      <td>trt2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>5.18</td>\n",
       "      <td>ctrl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>6.11</td>\n",
       "      <td>ctrl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>4.41</td>\n",
       "      <td>trt1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Unnamed: 0  weight group\n",
       "16          17    6.03  trt1\n",
       "28          29    5.80  trt2\n",
       "13          14    3.59  trt1\n",
       "29          30    5.26  trt2\n",
       "15          16    3.83  trt1\n",
       "9           10    5.14  ctrl\n",
       "21          22    5.12  trt2\n",
       "2            3    5.18  ctrl\n",
       "3            4    6.11  ctrl\n",
       "12          13    4.41  trt1"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"PlantGrowth.csv\")\n",
    "colInputs = ['weight']\n",
    "colOutput = \"group\"\n",
    "df.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def anovaCorOneWay(df,colInputs,colOutput,numOfFeatu):\n",
    "    cofList = []\n",
    "    for colInput in colInputs:\n",
    "        mod = ols(colInput+' ~ '+colOutput,data=df).fit() \n",
    "        aov_table = sm.stats.anova_lm(mod,typ=2)\n",
    "        cofList.append((colInput,aov_table['PR(>F)'][0]))\n",
    "        cofList.sort(key=lambda col:col[1])\n",
    "    return cofList[numOfFeatu:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anovaCorOneWay(df,colInputs,colOutput,1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Genotype</th>\n",
       "      <th>years</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>C</td>\n",
       "      <td>1_year</td>\n",
       "      <td>4.41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>F</td>\n",
       "      <td>3_year</td>\n",
       "      <td>9.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>D</td>\n",
       "      <td>1_year</td>\n",
       "      <td>3.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>C</td>\n",
       "      <td>1_year</td>\n",
       "      <td>3.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>E</td>\n",
       "      <td>1_year</td>\n",
       "      <td>2.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Genotype   years  value\n",
       "8         C  1_year   4.41\n",
       "51        F  3_year   9.84\n",
       "9         D  1_year   3.75\n",
       "6         C  1_year   3.99\n",
       "13        E  1_year   2.01"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = pd.read_csv(\"https://reneshbedre.github.io/assets/posts/anova/twowayanova.txt\", sep=\"\\t\")\n",
    "d_melt = pd.melt(d, id_vars=['Genotype'], value_vars=['1_year', '2_year', '3_year'])\n",
    "# replace column names\n",
    "d_melt.columns = ['Genotype', 'years', 'value']\n",
    "inputCols = ['Genotype', 'years']\n",
    "outputCol = \"value\"\n",
    "d_melt.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['Genotype', 'years'], 4.293214281878206e-21)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def anovaCortwoWay(df,colInputs,colOutput):\n",
    "    model = ols(colOutput+' ~  C('+colInputs[0]+' ):C( '+colInputs[1] +' )', data=df).fit()\n",
    "    anova_table = sm.stats.anova_lm(model, typ=2)\n",
    "    #anova_table['PR(>F)'][0] must be lower than 0.05\n",
    "    return (colInputs,anova_table['PR(>F)'][0])\n",
    "\n",
    "anovaCortwoWay(d_melt,inputCols,outputCol)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we know that genotype and time (years) differences are statistically significant, but ANOVA does not tell which genotype and time (years) are significantly different from each other. To know the pairs of significant different genotype and time (years), perform multiple pairwise comparison (Post-hoc comparison) analysis using Tukey HSD test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Multiple Comparison of Means - Tukey HSD, FWER=0.05</caption>\n",
       "<tr>\n",
       "  <th>group1</th> <th>group2</th> <th>meandiff</th>  <th>p-adj</th>  <th>lower</th>   <th>upper</th> <th>reject</th>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>A</td>      <td>B</td>     <td>2.04</td>   <td>0.5304</td> <td>-1.5094</td> <td>5.5894</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>A</td>      <td>C</td>    <td>2.7333</td>   <td>0.22</td>  <td>-0.816</td>  <td>6.2827</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>A</td>      <td>D</td>     <td>2.56</td>   <td>0.2847</td> <td>-0.9894</td> <td>6.1094</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>A</td>      <td>E</td>     <td>0.72</td>     <td>0.9</td>  <td>-2.8294</td> <td>4.2694</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>A</td>      <td>F</td>    <td>2.5733</td>  <td>0.2793</td> <td>-0.976</td>  <td>6.1227</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>B</td>      <td>C</td>    <td>0.6933</td>    <td>0.9</td>  <td>-2.856</td>  <td>4.2427</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>B</td>      <td>D</td>     <td>0.52</td>     <td>0.9</td>  <td>-3.0294</td> <td>4.0694</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>B</td>      <td>E</td>     <td>-1.32</td>  <td>0.8679</td> <td>-4.8694</td> <td>2.2294</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>B</td>      <td>F</td>    <td>0.5333</td>    <td>0.9</td>  <td>-3.016</td>  <td>4.0827</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>C</td>      <td>D</td>    <td>-0.1733</td>   <td>0.9</td>  <td>-3.7227</td>  <td>3.376</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>C</td>      <td>E</td>    <td>-2.0133</td> <td>0.5429</td> <td>-5.5627</td>  <td>1.536</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>C</td>      <td>F</td>     <td>-0.16</td>    <td>0.9</td>  <td>-3.7094</td> <td>3.3894</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>D</td>      <td>E</td>     <td>-1.84</td>  <td>0.6241</td> <td>-5.3894</td> <td>1.7094</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>D</td>      <td>F</td>    <td>0.0133</td>    <td>0.9</td>  <td>-3.536</td>  <td>3.5627</td>  <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>E</td>      <td>F</td>    <td>1.8533</td>  <td>0.6179</td> <td>-1.696</td>  <td>5.4027</td>  <td>False</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.table.SimpleTable'>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "###### A terminer #####\n",
    "def testAnova(df,inputCol,outputCol):\n",
    "    MultiComp = MultiComparison(df[outputCol],\n",
    "                                df[inputCol])\n",
    "    return MultiComp.tukeyhsd().summary()\n",
    "\n",
    "testAnova(d_melt,'Genotype','value')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical Input, Categorical Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chi-Squared test (for independence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.3232347637946903e-05"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def chiSquared(df,inputCol,outputCol):\n",
    "    table = pd.crosstab(df[inputCol],df[outputCol])\n",
    "    return chi2_contingency(table.values)[1]\n",
    "\n",
    "chiSquared(cars,'cyl','vs')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The P-value obtained from chi-square test for independence is significant (P<0.05), and therefore, we conclude that there is a significant association between cyl and vs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature selection with RFE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_model(model,X,y):\n",
    "    cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\n",
    "    scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\n",
    "    return scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\uchiha\\Anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['preg', 'mass', 'pedi']\n",
      "['plas', 'mass', 'pedi']\n",
      "['plas', 'mass', 'pedi']\n",
      "[0.74025974 0.81818182 0.74025974 0.81818182 0.76623377 0.81818182\n",
      " 0.75324675 0.74025974 0.80263158 0.69736842 0.7012987  0.75324675\n",
      " 0.66233766 0.76623377 0.71428571 0.77922078 0.72727273 0.77922078\n",
      " 0.76315789 0.76315789 0.68831169 0.76623377 0.76623377 0.7012987\n",
      " 0.85714286 0.72727273 0.72727273 0.75324675 0.77631579 0.82894737]\n",
      "['plas', 'mass', 'age']\n"
     ]
    }
   ],
   "source": [
    "url = \"https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.csv\"\n",
    "X = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age']\n",
    "Y = \"class\"\n",
    "cols = X+[Y]\n",
    "df = pd.read_csv(url, names=cols)\n",
    "def RfeMethod(df,inputCols,numFeatures,algorithm,outputCol):\n",
    "    array = df.values\n",
    "    X = array[:,0:len(inputCols)]\n",
    "    Y = array[:,len(inputCols)]\n",
    "    if algorithm == \"logisticRegression\":\n",
    "        model = LogisticRegression(solver='lbfgs')\n",
    "    elif algorithm == \"decisionTreeClassifier\":\n",
    "        model = DecisionTreeClassifier()\n",
    "    elif algorithm == \"DecisionTreeRegressor\":\n",
    "        model = DecisionTreeRegressor()\n",
    "    elif algorithm == \"RandomForestClassifier\":\n",
    "        model = RandomForestClassifier()\n",
    "    elif algorithm == \"GradientBoostingClassifier\":\n",
    "        model = GradientBoostingClassifier()\n",
    "        print(evaluate_model(model,df[inputCols],df[outputCol]))\n",
    "    rfe = RFE(model,numFeatures)\n",
    "    fit = rfe.fit(X, Y)\n",
    "    selectedFeMask = list(fit.support_)\n",
    "    selectedFe = [inputCols[i] for i in range(len(inputCols)) if selectedFeMask[i]]\n",
    "    return selectedFe\n",
    "\n",
    "print(RfeMethod(df,X,3,\"logisticRegression\",Y))\n",
    "\n",
    "print(RfeMethod(df,X,3,\"decisionTreeClassifier\",Y))\n",
    "\n",
    "print(RfeMethod(df,X,3,\"DecisionTreeRegressor\",Y))\n",
    "\n",
    "print(RfeMethod(df,X,3,\"GradientBoostingClassifier\",Y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Importance with ExtraTreesClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('age', 0.14524514648294043),\n",
       " ('mass', 0.14836462684830268),\n",
       " ('plas', 0.21677198482406776)]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def ExtraTreesClassifierMethod(df,inputCols,numFeatures):\n",
    "    array = df.values\n",
    "    X = array[:,0:len(inputCols)]\n",
    "    Y = array[:,len(inputCols)]\n",
    "    model = ExtraTreesClassifier(n_estimators=10)\n",
    "    model.fit(X, Y)\n",
    "    scores = list(model.feature_importances_)\n",
    "    listF = []\n",
    "    for i in range(len(scores)):\n",
    "        listF.append((inputCols[i],scores[i]))\n",
    "    listF.sort(key=lambda col:col[1])\n",
    "    return listF[-numFeatures:]\n",
    "\n",
    "ExtraTreesClassifierMethod(df,X,3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}