t81_558_class2_tensor_flow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Class 2: Introduction to TensorFlow.**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network Basics\n",
    "\n",
    "Neural networks were one of the first machine learning models.  Their popularity has fallen twice and is now on its third rise.  Deep learning implies the use of neural networks.  The \"deep\" in deep learning refers to a neural network with many hidden layers.  Because neural networks have been around for so long, they have quite a bit of baggage.  Many different training algorithms, activation/transfer functions, and structures have been added over the years.  This course is only concerned with the latest, most current state of the art techniques for deep neural networks.  I am not going to spend any time discussing the history of neural networks.  If you would like to learn about some of the more classic structures of neural networks, there are several chapters dedicated to this in your course book.  For the latest technology, I wrote an article for the Society of Actuaries on deep learning as the [third generation of neural networks](https://www.soa.org/Library/Newsletters/Predictive-Analytics-and-Futurism/2015/december/paf-iss12.pdf).\n",
    "\n",
    "Neural networks accept input and produce output.  The input to a neural network is called the feature vector.  The size of this vector is always a fixed length.  Changing the size of the feature vector means recreating the entire neural network.  Though the feature vector is called a \"vector,\" this is not always the case.  A vector implies a 1D array.  Historically the input to a neural network was always 1D.  However, with modern neural networks you might see inputs, such as:\n",
    "\n",
    "* **1D Vector** - Classic input to a neural network, similar to rows in a spreadsheet.  Common in predictive modeling.\n",
    "* **2D Matrix** - Grayscale image input to a convolutional neural network (CNN).\n",
    "* **3D Matrix** - Color image input to a convolutional neural network (CNN).\n",
    "* **nD Matrix** - Higher order input to a CNN.\n",
    "\n",
    "Prior to CNN's, the image input was sent to a neural network simply by squashing the image matrix into a long array by placing the image's rows side-by-side.  CNNs are different, as the nD matrix literally passes through the neural network layers.\n",
    "\n",
    "Initially this course will focus upon 1D input to neural networks.  However, later sessions will focus more heavily upon higher dimension input.\n",
    "\n",
    "**Dimensions** The term dimension can be confusing in neural networks.  In the sense of a 1D input vector, dimension refers to how many elements are in that 1D array.  For example a neural network with 10 input neurons has 10 dimensions.  However, now that we have CNN's, the input has dimensions too.  The input to the neural network will *usually* have 1, 2 or 3 dimensions.  4 or more dimensions is unusual.  You might have a 2D input to a neural network that has 64x64 pixels.  This would result in 4,096 input neurons.  This network is either 2D or 4,096D, depending on which set of dimensions you are talking about!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification or Regression\n",
    "\n",
    "Like many models, neural networks can function in classification or regression:\n",
    "\n",
    "* **Regression** - You expect a number as your neural network's prediction.\n",
    "* **Classification** - You expect a class/category as your neural network's prediction.\n",
    "\n",
    "The following shows a classification and regression neural network:\n",
    "\n",
    "![Neural Network Classification and Regression](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_ann_class_reg.png \"Neural Network Classification and Regression\")\n",
    "\n",
    "Notice that the output of the regression neural network is numeric and the output of the classification is a class.  Regression, or two-class classification, networks always have a single output.  Classification neural networks have an output neuron for each class. \n",
    "\n",
    "The following diagram shows a typical neural network:\n",
    "\n",
    "![Feedforward Neural Networks](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_ann.png \"Feedforward Neural Networks\")\n",
    "\n",
    "There are usually four types of neurons in a neural network:\n",
    "\n",
    "* **Input Neurons** - Each input neuron is mapped to one element in the feature vector.\n",
    "* **Hidden Neurons** - Hidden neurons allow the neural network to abstract and process the input into the output.\n",
    "* **Output Neurons** - Each output neuron calculates one part of the output.\n",
    "* **Context Neurons** - Holds state between calls to the neural network to predict.\n",
    "* **Bias Neurons** - Work similar to the y-intercept of a linear equation.  \n",
    "\n",
    "These neurons are grouped into layers:\n",
    "\n",
    "* **Input Layer** - The input layer accepts feature vectors from the dataset.  Input layers usually have a bias neuron.\n",
    "* **Output Layer** - The output from the neural network.  The output layer does not have a bias neuron.\n",
    "* **Hidden Layers** - Layers that occur between the input and output layers.  Each hidden layer will usually have a bias neuron.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neuron Calculation\n",
    "\n",
    "The output from a single neuron is calculated according to the following formula:\n",
    "\n",
    "$ f(x,\\theta) = \\phi(\\sum_i(\\theta_i \\cdot x_i)) $\n",
    "\n",
    "The input vector ($x$) represents the feature vector and the vector $\\theta$ (theta) represents the weights. To account for the bias neuron, a value of 1 is always appended to the end of the input feature vector.  This causes the last weight to be interpreted as a bias value that is simply added to the summation. The $\\phi$ (phi) is the transfer/activation function. \n",
    "\n",
    "Consider using the above equation to calculate the output from the following neuron:\n",
    "\n",
    "![Single Neuron](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_abstract_nn.png \"Single Neuron\")\n",
    "\n",
    "The above neuron has 2 inputs plus the bias as a third.  This neuron might accept the following input feature vector:\n",
    "\n",
    "```\n",
    "[1,2]\n",
    "```\n",
    "\n",
    "To account for the bias neuron, a 1 is appended, as follows:\n",
    "\n",
    "```\n",
    "[1,2,1]\n",
    "```\n",
    "\n",
    "The weights for a 3-input layer (2 real inputs + bias) will always have an additional weight, for the bias.  A weight vector might be:\n",
    "\n",
    "```\n",
    "[ 0.1, 0.2, 0.3]\n",
    "```\n",
    "\n",
    "To calculate the summation, perform the following:\n",
    "\n",
    "```\n",
    "0.1*1 + 0.2*2 + 0.3*3 = 1.4\n",
    "```\n",
    "\n",
    "The value of 1.4 is passed to the $\\phi$ (phi) function, which represents the activation function.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activation Functions\n",
    "\n",
    "Activation functions, also known as transfer functions, are used to calculate the output of each layer of a neural network.  Historically neural networks have used a hyperbolic tangent, sigmoid/logistic, or linear activation function.  However, modern deep neural networks primarily make use of the following activation functions:\n",
    "\n",
    "* **Rectified Linear Unit (ReLU)** - Used for the output of hidden layers.\n",
    "* **Softmax** - Used for the output of classification neural networks. [Softmax Example](http://www.heatonresearch.com/aifh/vol3/softmax.html)\n",
    "* **Linear** - Used for the output of regression neural networks (or 2-class classification).\n",
    "\n",
    "The ReLU function is calculated as follows:\n",
    "\n",
    "$ \\phi(x) = \\max(0, x) $\n",
    "\n",
    "The Softmax is calculated as follows:\n",
    "\n",
    "$ \\phi_i(z) = \\frac{e^{z_i}}{\\sum\\limits_{j \\in group}e^{z_j}} $\n",
    "\n",
    "The Softmax activation function is only useful with more than one output neuron.  It ensures that all of the output neurons sum to 1.0.  This makes it very useful for classification where it shows the probability of each of the classes as being the correct choice.\n",
    "\n",
    "To experiment with the Softmax, click [here](http://www.heatonresearch.com/aifh/vol3/softmax.html).\n",
    "\n",
    "The linear activation function is essentially no activation function:\n",
    "\n",
    "$ \\phi(x) = x $\n",
    "\n",
    "For regression problems, this is the activation function of choice.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why ReLU?\n",
    "\n",
    "Why is the ReLU activation function so popular?  It was one of the key improvements to neural networks that makes deep learning work. Prior to deep learning, the sigmoid activation function was very common:\n",
    "\n",
    "$ \\phi(x) = \\frac{1}{1 + e^{-x}} $\n",
    "\n",
    "The graph of the sigmoid function is shown here:\n",
    "\n",
    "![Sigmoid Activation Function](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_sigmoid.png \"Sigmoid Activation Function\")\n",
    "\n",
    "Neural networks are often trained using gradient descent.  To make use of gradient descent, it is necessary to take the derivative of the activation function.  This allows the partial derivatives of each of the weights to be calculated with respect to the error function.  A derivative is the instantaneous rate of change:\n",
    "\n",
    "![Derivative](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_deriv.png \"Derivative\")\n",
    "\n",
    "The derivative of the sigmoid function is given here:\n",
    "\n",
    "$ \\phi'(x)=\\phi(x)(1-\\phi(x)) $\n",
    "\n",
    "This derivative is often given in other forms.  The above form is used for computational efficiency. To see how this derivative was taken, see [this](http://www.heatonresearch.com/aifh/vol3/deriv_sigmoid.html).\n",
    "\n",
    "The graph of the sigmoid derivative is given here:\n",
    "\n",
    "![Sigmoid Derivative](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_deriv_sigmoid.png \"Sigmoid Derivative\")\n",
    "\n",
    "The derivative quickly saturates to zero as *x* moves from zero.  This is not a problem for the derivative of the ReLU, which is given here:\n",
    "\n",
    "$ \\phi'(x) = \\begin{cases} 1 & x > 0 \\\\ 0 & x \\leq 0 \\end{cases} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why are Bias Neurons Needed?\n",
    "\n",
    "The activation functions seen in the previous section specifies the output of a single neuron.  Together, the weight and bias of a neuron shape the output of the activation to produce the desired output.  To see how this process occurs, consider the following equation. It represents a single-input sigmoid activation neural network.\n",
    "\n",
    "$ f(x,w,b) = \\frac{1}{1 + e^{-(wx+b)}} $ \n",
    "\n",
    "The *x* variable represents the single input to the neural network.  The *w* and *b* variables specify the weight and bias of the neural network.  The above equation is a combination of the weighted sum of the inputs and the sigmoid activation function.  For this section, we will consider the sigmoid function because it clearly demonstrates the effect that a bias neuron has.\n",
    "\n",
    "The weights of the neuron allow you to adjust the slope or shape of the activation function.  The following figure shows the effect on the output of the sigmoid activation function if the weight is varied:\n",
    "\n",
    "![Adjusting Weight](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_bias_weight.png \"Bias 1\")\n",
    "\n",
    "The above diagram shows several sigmoid curves using the following parameters:\n",
    "\n",
    "```\n",
    "f(x,0.5,0.0)\n",
    "f(x,1.0,0.0)\n",
    "f(x,1.5,0.0)\n",
    "f(x,2.0,0.0)\n",
    "```\n",
    "\n",
    "To produce the curves, we did not use bias, which is evident in the third parameter of 0 in each case. Using four weight values yields four different sigmoid curves in the above figure. No matter the weight, we always get the same value of 0.5 when x is 0 because all of the curves hit the same point when x is 0.  We might need the neural network to produce other values when the input is near 0.5.  \n",
    "\n",
    "Bias does shift the sigmoid curve, which allows values other than 0.5 when x is near 0. The following figure shows the effect of using a weight of 1.0 with several different biases:\n",
    "\n",
    "\n",
    "![Adjusting Bias](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_bias_value.png \"Bias 1\")\n",
    "\n",
    "The above diagram shows several sigmoid curves with the following parameters:\n",
    "\n",
    "```\n",
    "f(x,1.0,1.0)\n",
    "f(x,1.0,0.5)\n",
    "f(x,1.0,1.5)\n",
    "f(x,1.0,2.0)\n",
    "```\n",
    "\n",
    "We used a weight of 1.0 for these curves in all cases.  When we utilized several different biases, sigmoid curves shifted to the left or right.  Because all the curves merge together at the top right or bottom left, it is not a complete shift.\n",
    "\n",
    "When we put bias and weights together, they produced a curve that created the necessary output from a neuron.  The above curves are the output from only one neuron.  In a complete network, the output from many different neurons will combine to produce complex output patterns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow - Google\n",
    "\n",
    "![TensorFlow](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_tensorflow.png \"TensorFlow\")\n",
    "\n",
    "TensorFlow is an open source software library for machine learning in various kinds of perceptual and language understanding tasks. It is currently used for both research and production by different teams in many commercial Google products, such as speech recognition, Gmail, Google Photos, and search, many of which had previously used its predecessor DistBelief. TensorFlow was originally developed by the Google Brain team for Google's research and production purposes and later released under the Apache 2.0 open source license on November 9, 2015.\n",
    "\n",
    "* [TensorFlow Homepage](https://www.tensorflow.org/)\n",
    "* [TensorFlow GitHib](https://github.com/tensorflow/tensorflow)\n",
    "* [TensorFlow Google Groups Support](https://groups.google.com/forum/#!forum/tensorflow)\n",
    "* [TensorFlow Google Groups Developer Discussion](https://groups.google.com/a/tensorflow.org/forum/#!forum/discuss)\n",
    "* [TensorFlow FAQ](https://www.tensorflow.org/resources/faq)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What version of TensorFlow do you have?\n",
    "\n",
    "TensorFlow is very new and changing rapidly.  It is very important that you run the same version of it that I am using.  For this semester we will use TensorFlow 0.12.1. It is very important to have the right version of TensorFlow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Self Driving Car](http://imgc-cn.artprintimages.com/images/P-473-488-90/94/9475/CFB6500Z/posters/paul-noth-does-your-car-have-any-idea-why-my-car-pulled-it-over-new-yorker-cartoon.jpg)\n",
    "\n",
    "[Wrong version of TensorFlow?](https://twitter.com/reza_zadeh/status/849160032608440320)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor Flow Version: 0.12.1\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(\"Tensor Flow Version: {}\".format(tf.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installing TensorFlow\n",
    "\n",
    "* [IBM Data Scientist Workbench](datascientistworkbench.com) - All platforms, use your browser.\n",
    "* Windows - Installing TensorFlow on Windows is not easy. Docker is your best option. [Hopefully this will improve in the future](https://github.com/tensorflow/tensorflow/issues/17).\n",
    "* Mac - Supported platform.\n",
    "* Linux - Supported platform.\n",
    "\n",
    "[TensorFlow Setup Instructions](https://www.tensorflow.org/versions/r0.10/get_started/os_setup.html#docker-based_installation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why TensorFlow\n",
    "\n",
    "* Supported by Google\n",
    "* Works well on Linux/Mac\n",
    "* Excellent GPU support\n",
    "* Python is an easy to learn programming language\n",
    "* Python is [extremely popular](http://www.kdnuggets.com/2014/08/four-main-languages-analytics-data-mining-data-science.html) in the data science community\n",
    "\n",
    "# Other Deep Learning Tools\n",
    "TensorFlow is not the only only game in town.  These are some of the best supported alternatives.  Most of these are written in C++. In order of my own preference (I have used all of these):\n",
    "\n",
    "* [Theano](http://deeplearning.net/software/theano/) - Python, from the academics that created deep learning.  GPU support.\n",
    "* [Computational Network Toolkit (CNTK)](https://github.com/Microsoft/CNTK) - Microsoft.  Support for Windows/Linux, command line only.  Bindings for predictions for C#/Python. GPU support.\n",
    "* [Caffe](http://caffe.berkeleyvision.org/) Academic project from Berkeley.  Python.  \n",
    "* [H2O](http://www.h2o.ai/) - Java based.  Supports all major platforms.  Limited support for computer vision. No GPU support.\n",
    "* [Deeplearning4J](http://deeplearning4j.org/) - Java based. Supports all major platforms. GPU support in Java!\n",
    "* [PaddlePaddle](https://github.com/baidu/Paddle) - [Baidu](http://www.baidu.com/)'s offering, very new.\n",
    "\n",
    "# Other/Advanced Deep Learning\n",
    "\n",
    "[Torch](http://torch.ch/) is used by Google DeepMind, the Facebook AI Research Group, IBM, Yandex and the Idiap Research Institute.  It has been used for some of the most advanced deep learning projects in the world.  However, it requires the [LUA](https://en.wikipedia.org/wiki/Lua_(programming_language)) programming language.  It is very advanced, but it is not mainstream.  I have not worked with Torch (yet!)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using TensorFlow\n",
    "\n",
    "TensorFlow is a low-level mathematics API, similar to [Numpy](http://www.numpy.org/).  However, unlike Numpy, TensorFlow is built for deep learning.  TensorFlow works by allowing you to define compute graphs with Python.  In this regard, it is similar to [Spark](http://spark.apache.org/).  TensorFlow compiles these compute graphs into highly efficient C++/[CUDA](https://en.wikipedia.org/wiki/CUDA) code.\n",
    "\n",
    "The [TensorBoard](https://www.tensorflow.org/versions/r0.10/how_tos/summaries_and_tensorboard/index.html) command line utility can be used to view these graphs.  The iris neural network's graph used in this class is shown here:\n",
    "\n",
    "![Iris Graph](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_graph_tf.png \"Iris Graph\")\n",
    "\n",
    "Expanding the DNN gives:\n",
    "\n",
    "\n",
    "![Iris DNN Graph](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_graph_dnn.png \"Iris DNN Graph\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using tf.contrib.learn\n",
    "\n",
    "TF Learn ([**tf.contrib.learn**](https://www.tensorflow.org/tutorials/tflearn/)) (formerly SKFLOW) is a layer on top of Tensorflow that makes it much easier to create neural networks.  Rather than define the graphs, like you see above, you define the individual layers of the network with a much more high level API.  Unless you are performing research into entirely new structures of deep neural networks it is unlikely that you need to program TensorFlow directly.  \n",
    "\n",
    "**For this class, we will use tf.contrib.learn, rather than direct TensorFlow**\n",
    "\n",
    "tf.contrib.learn is built into TensorFlow (as of v0.8).  This makes it very easy to use.\n",
    "\n",
    "All examples in this class will use tf.contrib.learn, and you are encouraged to use it for the programming assignments.\n",
    "\n",
    "* [TF Learn API Docs](https://www.tensorflow.org/api_docs/python/contrib.learn/)\n",
    "* [TF Learn Examples](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/learn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Functions for Tensorflow\n",
    "\n",
    "The following functions will be used in conjunction with TensorFlow to help preprocess the data.  It is okay to just use them. For better understanding, try to understand how they work.\n",
    "\n",
    "These functions allow you to build the feature vector for a neural network. Consider the following:\n",
    "\n",
    "* Predictors/Inputs \n",
    "    * Fill any missing inputs with the median for that column.  Use **missing_median**.\n",
    "    * Encode textual/categorical values with **encode_text_dummy**.\n",
    "    * Encode numeric values with **encode_numeric_zscore**.\n",
    "* Output\n",
    "    * Discard rows with missing outputs.\n",
    "    * Encode textual/categorical values with **encode_text_index**.\n",
    "    * Do not encode output numeric values.\n",
    "* Produce final feature vectors (x) and expected output (y) with **to_xy**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import shutil\n",
    "\n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df,name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = \"{}-{}\".format(name,x)\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "\n",
    "# Encode text values to a single dummy variable.  The new columns (which do not replace the old) will have a 1\n",
    "# at every location where the origional column (name) matches each of the target_values.  One column is added for\n",
    "# each target value.\n",
    "def encode_text_single_dummy(df,name,target_values):\n",
    "    for tv in target_values:\n",
    "        l = list(df[name].astype(str))\n",
    "        l = [1 if str(x)==str(tv) else 0 for x in l]\n",
    "        name2 = \"{}-{}\".format(name,tv)\n",
    "        df[name2] = l\n",
    "    \n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).\n",
    "def encode_text_index(df,name):\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "\n",
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df,name,mean=None,sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name]-mean)/sd\n",
    "\n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df,target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(target_type, '__iter__') else target_type\n",
    "    \n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        return df.as_matrix(result).astype(np.float32),df.as_matrix([target]).astype(np.int32)\n",
    "    else:\n",
    "        # Regression\n",
    "        return df.as_matrix(result).astype(np.float32),df.as_matrix([target]).astype(np.float32)\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)\n",
    "\n",
    "# Regression chart, we will see more of this chart in the next class.\n",
    "def chart_regression(pred,y):\n",
    "    t = pd.DataFrame({'pred' : pred, 'y' : y_test.flatten()})\n",
    "    t.sort_values(by=['y'],inplace=True)\n",
    "    a = plt.plot(t['y'].tolist(),label='expected')\n",
    "    b = plt.plot(t['pred'].tolist(),label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    \n",
    "# Get a new directory to hold checkpoints from a neural network.  This allows the neural network to be\n",
    "# loaded later.  If the erase param is set to true, the contents of the directory will be cleared.\n",
    "def get_model_dir(name,erase):\n",
    "    base_path = os.path.join(\".\",\"dnn\")\n",
    "    model_dir = os.path.join(base_path,name)\n",
    "    os.makedirs(model_dir,exist_ok=True)\n",
    "    if erase and len(model_dir)>4 and os.path.isdir(model_dir):\n",
    "        shutil.rmtree(model_dir,ignore_errors=True) # be careful, this deletes everything below the specified path\n",
    "    return model_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple TensorFlow Classification: Iris\n",
    "\n",
    "This is a very simple example of how to perform the Iris classification using TensorFlow.  The iris.csv file is used, rather than using the built-in files that many of the Google examples require.  \n",
    "\n",
    "**Make sure that you always run previous code blocks.  If you run the code block below, without the codeblock above, you will get errors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1\n",
      "}\n",
      ", '_save_summary_steps': 100, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001B67EE63390>, '_num_ps_replicas': 0, '_environment': 'local', '_master': '', '_task_type': None, '_keep_checkpoint_every_n_hours': 10000, '_tf_random_seed': None, '_keep_checkpoint_max': 5, '_task_id': 0, '_save_checkpoints_secs': 600, '_save_checkpoints_steps': None, '_evaluation_master': '', '_is_chief': True}\n",
      "WARNING:tensorflow:From <ipython-input-2-5f4562560747>:38: calling BaseEstimator.fit (from tensorflow.contrib.learn.python.learn.estimators.estimator) with x is deprecated and will be removed after 2016-12-01.\n",
      "Instructions for updating:\n",
      "Estimator is decoupled from Scikit Learn interface by moving into\n",
      "separate class SKCompat. Arguments x, y and batch_size are only\n",
      "available in the SKCompat class, Estimator will only accept input_fn.\n",
      "Example conversion:\n",
      "  est = Estimator(...) -> est = SKCompat(Estimator(...))\n",
      "WARNING:tensorflow:From <ipython-input-2-5f4562560747>:38: calling BaseEstimator.fit (from tensorflow.contrib.learn.python.learn.estimators.estimator) with y is deprecated and will be removed after 2016-12-01.\n",
      "Instructions for updating:\n",
      "Estimator is decoupled from Scikit Learn interface by moving into\n",
      "separate class SKCompat. Arguments x, y and batch_size are only\n",
      "available in the SKCompat class, Estimator will only accept input_fn.\n",
      "Example conversion:\n",
      "  est = Estimator(...) -> est = SKCompat(Estimator(...))\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\tf-latest\\lib\\site-packages\\tensorflow\\python\\util\\deprecation.py:247: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.\n",
      "  equality = a == b\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From C:\\Anaconda3\\envs\\tf-latest\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into .\\dnn\\iris\\model.ckpt.\n",
      "INFO:tensorflow:loss = 1.10515, step = 1\n",
      "INFO:tensorflow:global_step/sec: 734.759\n",
      "INFO:tensorflow:loss = 0.0763679, step = 101\n",
      "INFO:tensorflow:global_step/sec: 786.84\n",
      "INFO:tensorflow:loss = 0.0461095, step = 201\n",
      "INFO:tensorflow:global_step/sec: 567.768\n",
      "INFO:tensorflow:loss = 0.037618, step = 301\n",
      "INFO:tensorflow:global_step/sec: 640.559\n",
      "INFO:tensorflow:loss = 0.0339678, step = 401\n",
      "INFO:tensorflow:global_step/sec: 762.806\n",
      "INFO:tensorflow:loss = 0.031289, step = 501\n",
      "INFO:tensorflow:global_step/sec: 517.756\n",
      "INFO:tensorflow:loss = 0.0295143, step = 601\n",
      "INFO:tensorflow:global_step/sec: 587.782\n",
      "INFO:tensorflow:loss = 0.0279567, step = 701\n",
      "INFO:tensorflow:global_step/sec: 398.127\n",
      "INFO:tensorflow:loss = 0.0264107, step = 801\n",
      "INFO:tensorflow:global_step/sec: 366.024\n",
      "INFO:tensorflow:loss = 0.0248559, step = 901\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into .\\dnn\\iris\\model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.0231844.\n",
      "WARNING:tensorflow:From C:\\Anaconda3\\envs\\tf-latest\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\dnn.py:374: calling BaseEstimator.predict (from tensorflow.contrib.learn.python.learn.estimators.estimator) with x is deprecated and will be removed after 2016-12-01.\n",
      "Instructions for updating:\n",
      "Estimator is decoupled from Scikit Learn interface by moving into\n",
      "separate class SKCompat. Arguments x, y and batch_size are only\n",
      "available in the SKCompat class, Estimator will only accept input_fn.\n",
      "Example conversion:\n",
      "  est = Estimator(...) -> est = SKCompat(Estimator(...))\n",
      "Final score: 0.9866666666666667\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_l</th>\n",
       "      <th>sepal_w</th>\n",
       "      <th>petal_l</th>\n",
       "      <th>petal_w</th>\n",
       "      <th>expected</th>\n",
       "      <th>predicted</th>\n",
       "      <th>expected_str</th>\n",
       "      <th>predicted_str</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.897674</td>\n",
       "      <td>1.015602</td>\n",
       "      <td>-1.335752</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.139200</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>-1.335752</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.380727</td>\n",
       "      <td>0.327318</td>\n",
       "      <td>-1.392399</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.501490</td>\n",
       "      <td>0.097889</td>\n",
       "      <td>-1.279104</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.018437</td>\n",
       "      <td>1.245030</td>\n",
       "      <td>-1.335752</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-0.535384</td>\n",
       "      <td>1.933315</td>\n",
       "      <td>-1.165809</td>\n",
       "      <td>-1.048667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-1.501490</td>\n",
       "      <td>0.786174</td>\n",
       "      <td>-1.335752</td>\n",
       "      <td>-1.179859</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-1.018437</td>\n",
       "      <td>0.786174</td>\n",
       "      <td>-1.279104</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-1.743017</td>\n",
       "      <td>-0.360967</td>\n",
       "      <td>-1.335752</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-1.139200</td>\n",
       "      <td>0.097889</td>\n",
       "      <td>-1.279104</td>\n",
       "      <td>-1.442245</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-0.535384</td>\n",
       "      <td>1.474458</td>\n",
       "      <td>-1.279104</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-1.259964</td>\n",
       "      <td>0.786174</td>\n",
       "      <td>-1.222456</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-1.259964</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>-1.335752</td>\n",
       "      <td>-1.442245</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>-1.863780</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>-1.505695</td>\n",
       "      <td>-1.442245</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>-0.052331</td>\n",
       "      <td>2.162743</td>\n",
       "      <td>-1.449047</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-0.173094</td>\n",
       "      <td>3.080455</td>\n",
       "      <td>-1.279104</td>\n",
       "      <td>-1.048667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>-0.535384</td>\n",
       "      <td>1.933315</td>\n",
       "      <td>-1.392399</td>\n",
       "      <td>-1.048667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-0.897674</td>\n",
       "      <td>1.015602</td>\n",
       "      <td>-1.335752</td>\n",
       "      <td>-1.179859</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>-0.173094</td>\n",
       "      <td>1.703886</td>\n",
       "      <td>-1.165809</td>\n",
       "      <td>-1.179859</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>-0.897674</td>\n",
       "      <td>1.703886</td>\n",
       "      <td>-1.279104</td>\n",
       "      <td>-1.179859</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>-0.535384</td>\n",
       "      <td>0.786174</td>\n",
       "      <td>-1.165809</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>-0.897674</td>\n",
       "      <td>1.474458</td>\n",
       "      <td>-1.279104</td>\n",
       "      <td>-1.048667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>-1.501490</td>\n",
       "      <td>1.245030</td>\n",
       "      <td>-1.562342</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>-0.897674</td>\n",
       "      <td>0.556746</td>\n",
       "      <td>-1.165809</td>\n",
       "      <td>-0.917474</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>-1.259964</td>\n",
       "      <td>0.786174</td>\n",
       "      <td>-1.052513</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>-1.018437</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>-1.222456</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>-1.018437</td>\n",
       "      <td>0.786174</td>\n",
       "      <td>-1.222456</td>\n",
       "      <td>-1.048667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>-0.776911</td>\n",
       "      <td>1.015602</td>\n",
       "      <td>-1.279104</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>-0.776911</td>\n",
       "      <td>0.786174</td>\n",
       "      <td>-1.335752</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>-1.380727</td>\n",
       "      <td>0.327318</td>\n",
       "      <td>-1.222456</td>\n",
       "      <td>-1.311052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Iris-setosa</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>1.276066</td>\n",
       "      <td>0.327318</td>\n",
       "      <td>1.100097</td>\n",
       "      <td>1.443994</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>-0.293857</td>\n",
       "      <td>-0.590395</td>\n",
       "      <td>0.646916</td>\n",
       "      <td>1.050416</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>2.242172</td>\n",
       "      <td>-0.590395</td>\n",
       "      <td>1.666574</td>\n",
       "      <td>1.050416</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>0.551486</td>\n",
       "      <td>-0.819823</td>\n",
       "      <td>0.646916</td>\n",
       "      <td>0.788031</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>1.034539</td>\n",
       "      <td>0.556746</td>\n",
       "      <td>1.100097</td>\n",
       "      <td>1.181609</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>1.638355</td>\n",
       "      <td>0.327318</td>\n",
       "      <td>1.270040</td>\n",
       "      <td>0.788031</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>0.430722</td>\n",
       "      <td>-0.590395</td>\n",
       "      <td>0.590269</td>\n",
       "      <td>0.788031</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>0.309959</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>0.646916</td>\n",
       "      <td>0.788031</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>0.672249</td>\n",
       "      <td>-0.590395</td>\n",
       "      <td>1.043450</td>\n",
       "      <td>1.181609</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>1.638355</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>1.156745</td>\n",
       "      <td>0.525645</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>1.879882</td>\n",
       "      <td>-0.590395</td>\n",
       "      <td>1.326688</td>\n",
       "      <td>0.919223</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>2.483699</td>\n",
       "      <td>1.703886</td>\n",
       "      <td>1.496631</td>\n",
       "      <td>1.050416</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>0.672249</td>\n",
       "      <td>-0.590395</td>\n",
       "      <td>1.043450</td>\n",
       "      <td>1.312801</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133</th>\n",
       "      <td>0.551486</td>\n",
       "      <td>-0.590395</td>\n",
       "      <td>0.760211</td>\n",
       "      <td>0.394453</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Iris-versicolor</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>0.309959</td>\n",
       "      <td>-1.049251</td>\n",
       "      <td>1.043450</td>\n",
       "      <td>0.263260</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>2.242172</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>1.326688</td>\n",
       "      <td>1.443994</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>0.551486</td>\n",
       "      <td>0.786174</td>\n",
       "      <td>1.043450</td>\n",
       "      <td>1.575187</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>0.672249</td>\n",
       "      <td>0.097889</td>\n",
       "      <td>0.986802</td>\n",
       "      <td>0.788031</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>0.189196</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>0.590269</td>\n",
       "      <td>0.788031</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>1.276066</td>\n",
       "      <td>0.097889</td>\n",
       "      <td>0.930154</td>\n",
       "      <td>1.181609</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>1.034539</td>\n",
       "      <td>0.097889</td>\n",
       "      <td>1.043450</td>\n",
       "      <td>1.575187</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>1.276066</td>\n",
       "      <td>0.097889</td>\n",
       "      <td>0.760211</td>\n",
       "      <td>1.443994</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>-0.052331</td>\n",
       "      <td>-0.819823</td>\n",
       "      <td>0.760211</td>\n",
       "      <td>0.919223</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>1.155302</td>\n",
       "      <td>0.327318</td>\n",
       "      <td>1.213393</td>\n",
       "      <td>1.443994</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>1.034539</td>\n",
       "      <td>0.556746</td>\n",
       "      <td>1.100097</td>\n",
       "      <td>1.706379</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>1.034539</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>0.816859</td>\n",
       "      <td>1.443994</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>0.551486</td>\n",
       "      <td>-1.278680</td>\n",
       "      <td>0.703564</td>\n",
       "      <td>0.919223</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>0.793012</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>0.816859</td>\n",
       "      <td>1.050416</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>0.430722</td>\n",
       "      <td>0.786174</td>\n",
       "      <td>0.930154</td>\n",
       "      <td>1.443994</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>0.068433</td>\n",
       "      <td>-0.131539</td>\n",
       "      <td>0.760211</td>\n",
       "      <td>0.788031</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Iris-virginica</td>\n",
       "      <td>Iris-virginica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      sepal_l   sepal_w   petal_l   petal_w  expected  predicted  \\\n",
       "0   -0.897674  1.015602 -1.335752 -1.311052         0          0   \n",
       "1   -1.139200 -0.131539 -1.335752 -1.311052         0          0   \n",
       "2   -1.380727  0.327318 -1.392399 -1.311052         0          0   \n",
       "3   -1.501490  0.097889 -1.279104 -1.311052         0          0   \n",
       "4   -1.018437  1.245030 -1.335752 -1.311052         0          0   \n",
       "5   -0.535384  1.933315 -1.165809 -1.048667         0          0   \n",
       "6   -1.501490  0.786174 -1.335752 -1.179859         0          0   \n",
       "7   -1.018437  0.786174 -1.279104 -1.311052         0          0   \n",
       "8   -1.743017 -0.360967 -1.335752 -1.311052         0          0   \n",
       "9   -1.139200  0.097889 -1.279104 -1.442245         0          0   \n",
       "10  -0.535384  1.474458 -1.279104 -1.311052         0          0   \n",
       "11  -1.259964  0.786174 -1.222456 -1.311052         0          0   \n",
       "12  -1.259964 -0.131539 -1.335752 -1.442245         0          0   \n",
       "13  -1.863780 -0.131539 -1.505695 -1.442245         0          0   \n",
       "14  -0.052331  2.162743 -1.449047 -1.311052         0          0   \n",
       "15  -0.173094  3.080455 -1.279104 -1.048667         0          0   \n",
       "16  -0.535384  1.933315 -1.392399 -1.048667         0          0   \n",
       "17  -0.897674  1.015602 -1.335752 -1.179859         0          0   \n",
       "18  -0.173094  1.703886 -1.165809 -1.179859         0          0   \n",
       "19  -0.897674  1.703886 -1.279104 -1.179859         0          0   \n",
       "20  -0.535384  0.786174 -1.165809 -1.311052         0          0   \n",
       "21  -0.897674  1.474458 -1.279104 -1.048667         0          0   \n",
       "22  -1.501490  1.245030 -1.562342 -1.311052         0          0   \n",
       "23  -0.897674  0.556746 -1.165809 -0.917474         0          0   \n",
       "24  -1.259964  0.786174 -1.052513 -1.311052         0          0   \n",
       "25  -1.018437 -0.131539 -1.222456 -1.311052         0          0   \n",
       "26  -1.018437  0.786174 -1.222456 -1.048667         0          0   \n",
       "27  -0.776911  1.015602 -1.279104 -1.311052         0          0   \n",
       "28  -0.776911  0.786174 -1.335752 -1.311052         0          0   \n",
       "29  -1.380727  0.327318 -1.222456 -1.311052         0          0   \n",
       "..        ...       ...       ...       ...       ...        ...   \n",
       "120  1.276066  0.327318  1.100097  1.443994         2          2   \n",
       "121 -0.293857 -0.590395  0.646916  1.050416         2          2   \n",
       "122  2.242172 -0.590395  1.666574  1.050416         2          2   \n",
       "123  0.551486 -0.819823  0.646916  0.788031         2          2   \n",
       "124  1.034539  0.556746  1.100097  1.181609         2          2   \n",
       "125  1.638355  0.327318  1.270040  0.788031         2          2   \n",
       "126  0.430722 -0.590395  0.590269  0.788031         2          2   \n",
       "127  0.309959 -0.131539  0.646916  0.788031         2          2   \n",
       "128  0.672249 -0.590395  1.043450  1.181609         2          2   \n",
       "129  1.638355 -0.131539  1.156745  0.525645         2          2   \n",
       "130  1.879882 -0.590395  1.326688  0.919223         2          2   \n",
       "131  2.483699  1.703886  1.496631  1.050416         2          2   \n",
       "132  0.672249 -0.590395  1.043450  1.312801         2          2   \n",
       "133  0.551486 -0.590395  0.760211  0.394453         2          1   \n",
       "134  0.309959 -1.049251  1.043450  0.263260         2          2   \n",
       "135  2.242172 -0.131539  1.326688  1.443994         2          2   \n",
       "136  0.551486  0.786174  1.043450  1.575187         2          2   \n",
       "137  0.672249  0.097889  0.986802  0.788031         2          2   \n",
       "138  0.189196 -0.131539  0.590269  0.788031         2          2   \n",
       "139  1.276066  0.097889  0.930154  1.181609         2          2   \n",
       "140  1.034539  0.097889  1.043450  1.575187         2          2   \n",
       "141  1.276066  0.097889  0.760211  1.443994         2          2   \n",
       "142 -0.052331 -0.819823  0.760211  0.919223         2          2   \n",
       "143  1.155302  0.327318  1.213393  1.443994         2          2   \n",
       "144  1.034539  0.556746  1.100097  1.706379         2          2   \n",
       "145  1.034539 -0.131539  0.816859  1.443994         2          2   \n",
       "146  0.551486 -1.278680  0.703564  0.919223         2          2   \n",
       "147  0.793012 -0.131539  0.816859  1.050416         2          2   \n",
       "148  0.430722  0.786174  0.930154  1.443994         2          2   \n",
       "149  0.068433 -0.131539  0.760211  0.788031         2          2   \n",
       "\n",
       "        expected_str   predicted_str  \n",
       "0        Iris-setosa     Iris-setosa  \n",
       "1        Iris-setosa     Iris-setosa  \n",
       "2        Iris-setosa     Iris-setosa  \n",
       "3        Iris-setosa     Iris-setosa  \n",
       "4        Iris-setosa     Iris-setosa  \n",
       "5        Iris-setosa     Iris-setosa  \n",
       "6        Iris-setosa     Iris-setosa  \n",
       "7        Iris-setosa     Iris-setosa  \n",
       "8        Iris-setosa     Iris-setosa  \n",
       "9        Iris-setosa     Iris-setosa  \n",
       "10       Iris-setosa     Iris-setosa  \n",
       "11       Iris-setosa     Iris-setosa  \n",
       "12       Iris-setosa     Iris-setosa  \n",
       "13       Iris-setosa     Iris-setosa  \n",
       "14       Iris-setosa     Iris-setosa  \n",
       "15       Iris-setosa     Iris-setosa  \n",
       "16       Iris-setosa     Iris-setosa  \n",
       "17       Iris-setosa     Iris-setosa  \n",
       "18       Iris-setosa     Iris-setosa  \n",
       "19       Iris-setosa     Iris-setosa  \n",
       "20       Iris-setosa     Iris-setosa  \n",
       "21       Iris-setosa     Iris-setosa  \n",
       "22       Iris-setosa     Iris-setosa  \n",
       "23       Iris-setosa     Iris-setosa  \n",
       "24       Iris-setosa     Iris-setosa  \n",
       "25       Iris-setosa     Iris-setosa  \n",
       "26       Iris-setosa     Iris-setosa  \n",
       "27       Iris-setosa     Iris-setosa  \n",
       "28       Iris-setosa     Iris-setosa  \n",
       "29       Iris-setosa     Iris-setosa  \n",
       "..               ...             ...  \n",
       "120   Iris-virginica  Iris-virginica  \n",
       "121   Iris-virginica  Iris-virginica  \n",
       "122   Iris-virginica  Iris-virginica  \n",
       "123   Iris-virginica  Iris-virginica  \n",
       "124   Iris-virginica  Iris-virginica  \n",
       "125   Iris-virginica  Iris-virginica  \n",
       "126   Iris-virginica  Iris-virginica  \n",
       "127   Iris-virginica  Iris-virginica  \n",
       "128   Iris-virginica  Iris-virginica  \n",
       "129   Iris-virginica  Iris-virginica  \n",
       "130   Iris-virginica  Iris-virginica  \n",
       "131   Iris-virginica  Iris-virginica  \n",
       "132   Iris-virginica  Iris-virginica  \n",
       "133  Iris-versicolor  Iris-virginica  \n",
       "134   Iris-virginica  Iris-virginica  \n",
       "135   Iris-virginica  Iris-virginica  \n",
       "136   Iris-virginica  Iris-virginica  \n",
       "137   Iris-virginica  Iris-virginica  \n",
       "138   Iris-virginica  Iris-virginica  \n",
       "139   Iris-virginica  Iris-virginica  \n",
       "140   Iris-virginica  Iris-virginica  \n",
       "141   Iris-virginica  Iris-virginica  \n",
       "142   Iris-virginica  Iris-virginica  \n",
       "143   Iris-virginica  Iris-virginica  \n",
       "144   Iris-virginica  Iris-virginica  \n",
       "145   Iris-virginica  Iris-virginica  \n",
       "146   Iris-virginica  Iris-virginica  \n",
       "147   Iris-virginica  Iris-virginica  \n",
       "148   Iris-virginica  Iris-virginica  \n",
       "149   Iris-virginica  Iris-virginica  \n",
       "\n",
       "[150 rows x 8 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.contrib.learn as learn\n",
    "from sklearn import metrics\n",
    "import pandas as pd\n",
    "import os\n",
    "from sklearn import preprocessing\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "# Read iris dataset\n",
    "filename_read = os.path.join(path,\"iris.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "# Extract just the columns we shall predict on\n",
    "encode_numeric_zscore(df,'sepal_l')\n",
    "encode_numeric_zscore(df,'sepal_w')\n",
    "encode_numeric_zscore(df,'petal_l')\n",
    "encode_numeric_zscore(df,'petal_w')\n",
    "species = encode_text_index(df,'species')\n",
    "num_classes = len(species)\n",
    "\n",
    "# Create x(predictors) and y (expected outcome)\n",
    "x,y = to_xy(df,'species')\n",
    "\n",
    "# Get/clear a directory to store the neural network to\n",
    "model_dir = get_model_dir('iris',True)\n",
    "\n",
    "# Create a deep neural network with 3 hidden layers of 10, 20, 5\n",
    "feature_columns = [tf.contrib.layers.real_valued_column(\"\", dimension=x.shape[1])]\n",
    "classifier = learn.DNNClassifier(\n",
    "    model_dir= model_dir,\n",
    "    hidden_units=[10, 20, 5], n_classes=num_classes, feature_columns=feature_columns)\n",
    "\n",
    "# Might be needed in future versions of \"TensorFlow Learn\"\n",
    "#classifier = learn.SKCompat(classifier) # For Sklearn compatibility\n",
    "\n",
    "# Fit/train neural network\n",
    "classifier.fit(x, y, steps=1000)\n",
    "\n",
    "# Measure accuracy\n",
    "pred = list(classifier.predict(x, as_iterable=True))\n",
    "score = metrics.accuracy_score(y, pred)\n",
    "print(\"Final score: {}\".format(score))\n",
    "\n",
    "# How to convert predicted indexs to category labels (e.g. iris names)\n",
    "predDF = pd.DataFrame(pred)\n",
    "pred_nameDF = pd.DataFrame(species[pred])\n",
    "actual_nameDF = pd.DataFrame(species[df['species']])\n",
    "df2 = pd.concat([df,predDF,pred_nameDF,actual_nameDF],axis=1)\n",
    "df2.columns = ['sepal_l','sepal_w','petal_l','petal_w','expected','predicted','expected_str','predicted_str']\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the massive amount of warnings and other information that you are given?  All of the warnings do not apply, and are just \"growing pains\" of the rapid pace of the various parts of TensorFlow.  It can be helpful to instruct TensorFlow to only display errors.  The following code accomplishes this.  Run the line below and then return the above example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Only show errors\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, while debugging, you might like to see all of the information.  The following command turns the output back on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Only show errors\n",
    "tf.logging.set_verbosity(tf.logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a neural network trained, we would like to be able to use it.  The following code makes use of our neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predict that [[ 5.  3.  4.  2.]] is: ['Iris-virginica']\n",
      "Predict that [[ 5.   3.   4.   2. ]\n",
      " [ 5.2  3.5  1.5  0.8]] is: ['Iris-virginica' 'Iris-versicolor']\n"
     ]
    }
   ],
   "source": [
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "\n",
    "# ad hoc prediction\n",
    "sample_flower = np.array( [[5.0,3.0,4.0,2.0]], dtype=float)\n",
    "pred = list(classifier.predict(sample_flower, as_iterable=True))\n",
    "print(\"Predict that {} is: {}\".format(sample_flower,species[pred]))\n",
    "\n",
    "# predict two sample flowers\n",
    "sample_flower = np.array( [[5.0,3.0,4.0,2.0],[5.2,3.5,1.5,0.8]], dtype=float)\n",
    "pred = list(classifier.predict(sample_flower, as_iterable=True))\n",
    "print(\"Predict that {} is: {}\".format(sample_flower,species[pred]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple TensorFlow Regression: MPG\n",
    "\n",
    "This example shows how to encode the MPG dataset for regression.  This is slightly more complex than Iris, because:\n",
    "\n",
    "* Input has both numeric and categorical\n",
    "* Input has missing values\n",
    "\n",
    "To encode categorical values that are part of the feature vector, use the functions from above.  If the categorical value is the target (as was the case with Iris, use the same technique as Iris). The iris technique allows you to decode back to Iris text strings from the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1\n",
      "}\n",
      ", '_save_summary_steps': 100, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001B603241E80>, '_num_ps_replicas': 0, '_environment': 'local', '_master': '', '_task_type': None, '_keep_checkpoint_every_n_hours': 10000, '_tf_random_seed': None, '_keep_checkpoint_max': 5, '_task_id': 0, '_save_checkpoints_secs': 600, '_save_checkpoints_steps': None, '_evaluation_master': '', '_is_chief': True}\n",
      "WARNING:tensorflow:From <ipython-input-5-7d7e51d4a843>:40: calling BaseEstimator.fit (from tensorflow.contrib.learn.python.learn.estimators.estimator) with x is deprecated and will be removed after 2016-12-01.\n",
      "Instructions for updating:\n",
      "Estimator is decoupled from Scikit Learn interface by moving into\n",
      "separate class SKCompat. Arguments x, y and batch_size are only\n",
      "available in the SKCompat class, Estimator will only accept input_fn.\n",
      "Example conversion:\n",
      "  est = Estimator(...) -> est = SKCompat(Estimator(...))\n",
      "WARNING:tensorflow:From <ipython-input-5-7d7e51d4a843>:40: calling BaseEstimator.fit (from tensorflow.contrib.learn.python.learn.estimators.estimator) with y is deprecated and will be removed after 2016-12-01.\n",
      "Instructions for updating:\n",
      "Estimator is decoupled from Scikit Learn interface by moving into\n",
      "separate class SKCompat. Arguments x, y and batch_size are only\n",
      "available in the SKCompat class, Estimator will only accept input_fn.\n",
      "Example conversion:\n",
      "  est = Estimator(...) -> est = SKCompat(Estimator(...))\n",
      "WARNING:tensorflow:From C:\\Anaconda3\\envs\\tf-latest\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\tf-latest\\lib\\site-packages\\tensorflow\\python\\util\\deprecation.py:247: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.\n",
      "  equality = a == b\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into .\\dnn\\mpg\\model.ckpt.\n",
      "INFO:tensorflow:loss = 727.871, step = 1\n",
      "INFO:tensorflow:global_step/sec: 390.331\n",
      "INFO:tensorflow:loss = 14.5538, step = 101\n",
      "INFO:tensorflow:global_step/sec: 299.188\n",
      "INFO:tensorflow:loss = 12.9737, step = 201\n",
      "INFO:tensorflow:global_step/sec: 239.637\n",
      "INFO:tensorflow:loss = 12.3344, step = 301\n",
      "INFO:tensorflow:global_step/sec: 416.363\n",
      "INFO:tensorflow:loss = 11.9422, step = 401\n",
      "INFO:tensorflow:global_step/sec: 407.869\n",
      "INFO:tensorflow:loss = 11.6369, step = 501\n",
      "INFO:tensorflow:global_step/sec: 423.423\n",
      "INFO:tensorflow:loss = 11.4009, step = 601\n",
      "INFO:tensorflow:global_step/sec: 307.469\n",
      "INFO:tensorflow:loss = 11.2196, step = 701\n",
      "INFO:tensorflow:global_step/sec: 242.541\n",
      "INFO:tensorflow:loss = 11.0782, step = 801\n",
      "INFO:tensorflow:global_step/sec: 307.472\n",
      "INFO:tensorflow:loss = 10.9666, step = 901\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into .\\dnn\\mpg\\model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 10.874.\n",
      "WARNING:tensorflow:From C:\\Anaconda3\\envs\\tf-latest\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\dnn.py:692: calling BaseEstimator.predict (from tensorflow.contrib.learn.python.learn.estimators.estimator) with x is deprecated and will be removed after 2016-12-01.\n",
      "Instructions for updating:\n",
      "Estimator is decoupled from Scikit Learn interface by moving into\n",
      "separate class SKCompat. Arguments x, y and batch_size are only\n",
      "available in the SKCompat class, Estimator will only accept input_fn.\n",
      "Example conversion:\n",
      "  est = Estimator(...) -> est = SKCompat(Estimator(...))\n",
      "Final score (RMSE): 3.297440767288208\n",
      "WARNING:tensorflow:From C:\\Anaconda3\\envs\\tf-latest\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\dnn.py:692: calling BaseEstimator.predict (from tensorflow.contrib.learn.python.learn.estimators.estimator) with x is deprecated and will be removed after 2016-12-01.\n",
      "Instructions for updating:\n",
      "Estimator is decoupled from Scikit Learn interface by moving into\n",
      "separate class SKCompat. Arguments x, y and batch_size are only\n",
      "available in the SKCompat class, Estimator will only accept input_fn.\n",
      "Example conversion:\n",
      "  est = Estimator(...) -> est = SKCompat(Estimator(...))\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin-1</th>\n",
       "      <th>origin-2</th>\n",
       "      <th>origin-3</th>\n",
       "      <th>pred</th>\n",
       "      <th>ideal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.089233</td>\n",
       "      <td>0.672271</td>\n",
       "      <td>0.630077</td>\n",
       "      <td>-1.293870</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16.635473</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.501624</td>\n",
       "      <td>1.587959</td>\n",
       "      <td>0.853259</td>\n",
       "      <td>-1.475181</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14.806089</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.194728</td>\n",
       "      <td>1.195522</td>\n",
       "      <td>0.549778</td>\n",
       "      <td>-1.656492</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16.233561</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.060461</td>\n",
       "      <td>1.195522</td>\n",
       "      <td>0.546236</td>\n",
       "      <td>-1.293870</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16.070099</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.041280</td>\n",
       "      <td>0.933897</td>\n",
       "      <td>0.565130</td>\n",
       "      <td>-1.837804</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16.844168</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>2.259274</td>\n",
       "      <td>2.451322</td>\n",
       "      <td>1.618455</td>\n",
       "      <td>-2.019115</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12.304775</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>14.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>2.499036</td>\n",
       "      <td>3.026898</td>\n",
       "      <td>1.633806</td>\n",
       "      <td>-2.381737</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11.659739</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>14.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>2.364769</td>\n",
       "      <td>2.896085</td>\n",
       "      <td>1.584210</td>\n",
       "      <td>-2.563048</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12.143611</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>14.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>2.508627</td>\n",
       "      <td>3.157710</td>\n",
       "      <td>1.717647</td>\n",
       "      <td>-2.019115</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11.064786</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.885244</td>\n",
       "      <td>2.242022</td>\n",
       "      <td>1.038654</td>\n",
       "      <td>-2.563048</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14.233102</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.818111</td>\n",
       "      <td>1.718772</td>\n",
       "      <td>0.699747</td>\n",
       "      <td>-2.019115</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>15.099044</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>14.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.405719</td>\n",
       "      <td>1.457147</td>\n",
       "      <td>0.754067</td>\n",
       "      <td>-2.744360</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16.187143</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.981149</td>\n",
       "      <td>1.195522</td>\n",
       "      <td>0.933557</td>\n",
       "      <td>-2.200426</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>15.395020</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>2.508627</td>\n",
       "      <td>3.157710</td>\n",
       "      <td>0.136478</td>\n",
       "      <td>-2.019115</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13.741521</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>24.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.771324</td>\n",
       "      <td>-0.243417</td>\n",
       "      <td>-0.706655</td>\n",
       "      <td>-0.206002</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>25.938314</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>22.0</td>\n",
       "      <td>0.320532</td>\n",
       "      <td>0.043868</td>\n",
       "      <td>-0.243417</td>\n",
       "      <td>-0.162279</td>\n",
       "      <td>-0.024691</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20.217829</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>18.0</td>\n",
       "      <td>0.320532</td>\n",
       "      <td>0.053459</td>\n",
       "      <td>-0.191092</td>\n",
       "      <td>-0.231950</td>\n",
       "      <td>-0.024691</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20.258669</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>21.0</td>\n",
       "      <td>0.320532</td>\n",
       "      <td>0.063049</td>\n",
       "      <td>-0.505042</td>\n",
       "      <td>-0.452770</td>\n",
       "      <td>0.156620</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22.292265</td>\n",
       "      <td>21.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>27.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.924773</td>\n",
       "      <td>-0.426554</td>\n",
       "      <td>-0.992422</td>\n",
       "      <td>-0.387314</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>28.020178</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>26.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.924773</td>\n",
       "      <td>-1.525380</td>\n",
       "      <td>-1.340775</td>\n",
       "      <td>1.788421</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>32.602875</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>25.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.800096</td>\n",
       "      <td>-0.452717</td>\n",
       "      <td>-0.352397</td>\n",
       "      <td>0.700554</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>24.524057</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>24.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.828867</td>\n",
       "      <td>-0.374229</td>\n",
       "      <td>-0.638165</td>\n",
       "      <td>-0.387314</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>25.421770</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>25.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.857639</td>\n",
       "      <td>-0.243417</td>\n",
       "      <td>-0.703112</td>\n",
       "      <td>0.700554</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>25.130121</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>26.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.694600</td>\n",
       "      <td>0.227509</td>\n",
       "      <td>-0.869613</td>\n",
       "      <td>-1.112559</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>23.748926</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>21.0</td>\n",
       "      <td>0.320532</td>\n",
       "      <td>0.053459</td>\n",
       "      <td>-0.374229</td>\n",
       "      <td>-0.380738</td>\n",
       "      <td>-0.206002</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21.550219</td>\n",
       "      <td>21.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.597529</td>\n",
       "      <td>2.896085</td>\n",
       "      <td>1.942010</td>\n",
       "      <td>-0.568625</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10.532638</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.089233</td>\n",
       "      <td>2.503648</td>\n",
       "      <td>1.659785</td>\n",
       "      <td>-0.206002</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11.583697</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>11.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.194728</td>\n",
       "      <td>2.765273</td>\n",
       "      <td>1.666870</td>\n",
       "      <td>-0.749936</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11.585458</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>9.0</td>\n",
       "      <td>1.496308</td>\n",
       "      <td>1.060461</td>\n",
       "      <td>2.320510</td>\n",
       "      <td>2.080171</td>\n",
       "      <td>1.063176</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10.145028</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>27.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.924773</td>\n",
       "      <td>-0.426554</td>\n",
       "      <td>-0.992422</td>\n",
       "      <td>-0.387314</td>\n",
       "      <td>71</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>28.317135</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>368</th>\n",
       "      <td>27.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.780915</td>\n",
       "      <td>-0.426554</td>\n",
       "      <td>-0.390185</td>\n",
       "      <td>1.099439</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27.471544</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>369</th>\n",
       "      <td>34.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.780915</td>\n",
       "      <td>-0.426554</td>\n",
       "      <td>-0.679495</td>\n",
       "      <td>0.881865</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>28.578432</td>\n",
       "      <td>34.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>370</th>\n",
       "      <td>31.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.780915</td>\n",
       "      <td>-0.505042</td>\n",
       "      <td>-0.466940</td>\n",
       "      <td>0.229145</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>28.129114</td>\n",
       "      <td>31.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>371</th>\n",
       "      <td>29.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.560333</td>\n",
       "      <td>-0.531204</td>\n",
       "      <td>-0.525983</td>\n",
       "      <td>0.156620</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>28.066454</td>\n",
       "      <td>29.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>27.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.406885</td>\n",
       "      <td>-0.374229</td>\n",
       "      <td>-0.278003</td>\n",
       "      <td>0.881865</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.191919</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>373</th>\n",
       "      <td>24.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.512381</td>\n",
       "      <td>-0.321904</td>\n",
       "      <td>-0.124492</td>\n",
       "      <td>0.301669</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25.635857</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>374</th>\n",
       "      <td>23.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.406885</td>\n",
       "      <td>-0.282660</td>\n",
       "      <td>0.076254</td>\n",
       "      <td>1.788421</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24.433367</td>\n",
       "      <td>23.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>375</th>\n",
       "      <td>36.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.848048</td>\n",
       "      <td>-0.792829</td>\n",
       "      <td>-1.169551</td>\n",
       "      <td>-0.097216</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>32.644531</td>\n",
       "      <td>36.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>37.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.982316</td>\n",
       "      <td>-0.949804</td>\n",
       "      <td>-1.116412</td>\n",
       "      <td>0.954390</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>34.110847</td>\n",
       "      <td>37.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>377</th>\n",
       "      <td>31.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.982316</td>\n",
       "      <td>-0.949804</td>\n",
       "      <td>-1.181360</td>\n",
       "      <td>0.736816</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>34.371044</td>\n",
       "      <td>31.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>378</th>\n",
       "      <td>38.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.848048</td>\n",
       "      <td>-1.080617</td>\n",
       "      <td>-0.998327</td>\n",
       "      <td>-0.314789</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>32.546703</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>379</th>\n",
       "      <td>36.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.915182</td>\n",
       "      <td>-0.897479</td>\n",
       "      <td>-0.998327</td>\n",
       "      <td>0.628029</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>31.883390</td>\n",
       "      <td>36.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>380</th>\n",
       "      <td>36.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.704191</td>\n",
       "      <td>-0.426554</td>\n",
       "      <td>-0.956997</td>\n",
       "      <td>-0.387314</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>31.056963</td>\n",
       "      <td>36.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>381</th>\n",
       "      <td>36.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.828867</td>\n",
       "      <td>-0.766667</td>\n",
       "      <td>-0.903858</td>\n",
       "      <td>-0.387314</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>32.411007</td>\n",
       "      <td>36.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>382</th>\n",
       "      <td>34.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.819277</td>\n",
       "      <td>-0.897479</td>\n",
       "      <td>-0.856624</td>\n",
       "      <td>0.482980</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>32.667728</td>\n",
       "      <td>34.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>383</th>\n",
       "      <td>38.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.982316</td>\n",
       "      <td>-0.975967</td>\n",
       "      <td>-1.187264</td>\n",
       "      <td>-0.206002</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>34.561279</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>384</th>\n",
       "      <td>32.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.982316</td>\n",
       "      <td>-0.975967</td>\n",
       "      <td>-1.187264</td>\n",
       "      <td>0.047833</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>34.543659</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>385</th>\n",
       "      <td>38.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.982316</td>\n",
       "      <td>-0.975967</td>\n",
       "      <td>-1.151838</td>\n",
       "      <td>0.229145</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>34.397385</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>386</th>\n",
       "      <td>25.0</td>\n",
       "      <td>0.320532</td>\n",
       "      <td>-0.119170</td>\n",
       "      <td>0.149021</td>\n",
       "      <td>-0.030023</td>\n",
       "      <td>0.301669</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22.159510</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>38.0</td>\n",
       "      <td>0.320532</td>\n",
       "      <td>0.657660</td>\n",
       "      <td>-0.505042</td>\n",
       "      <td>0.052637</td>\n",
       "      <td>0.519243</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22.863890</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>26.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.358933</td>\n",
       "      <td>-0.321904</td>\n",
       "      <td>-0.455132</td>\n",
       "      <td>-0.387314</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.658047</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>22.0</td>\n",
       "      <td>0.320532</td>\n",
       "      <td>0.369945</td>\n",
       "      <td>0.201346</td>\n",
       "      <td>-0.159917</td>\n",
       "      <td>-0.314789</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22.432077</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>32.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.474019</td>\n",
       "      <td>-0.217254</td>\n",
       "      <td>-0.360663</td>\n",
       "      <td>-0.604887</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>27.591600</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>36.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.560333</td>\n",
       "      <td>-0.531204</td>\n",
       "      <td>-0.709016</td>\n",
       "      <td>-0.931247</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>28.832703</td>\n",
       "      <td>36.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>392</th>\n",
       "      <td>27.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.406885</td>\n",
       "      <td>-0.374229</td>\n",
       "      <td>-0.024119</td>\n",
       "      <td>0.628029</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25.251448</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>27.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.512381</td>\n",
       "      <td>-0.478879</td>\n",
       "      <td>-0.213056</td>\n",
       "      <td>0.011571</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.605188</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>394</th>\n",
       "      <td>44.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.924773</td>\n",
       "      <td>-1.368405</td>\n",
       "      <td>-0.992422</td>\n",
       "      <td>3.275173</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>34.133556</td>\n",
       "      <td>44.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>395</th>\n",
       "      <td>32.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.560333</td>\n",
       "      <td>-0.531204</td>\n",
       "      <td>-0.797581</td>\n",
       "      <td>-1.438919</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29.202166</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>396</th>\n",
       "      <td>28.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.704191</td>\n",
       "      <td>-0.662017</td>\n",
       "      <td>-0.407897</td>\n",
       "      <td>1.099439</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>28.324156</td>\n",
       "      <td>28.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>397</th>\n",
       "      <td>31.0</td>\n",
       "      <td>-0.855244</td>\n",
       "      <td>-0.713781</td>\n",
       "      <td>-0.583529</td>\n",
       "      <td>-0.295716</td>\n",
       "      <td>1.389537</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27.590269</td>\n",
       "      <td>31.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>398 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      mpg  cylinders  displacement  horsepower    weight  acceleration  year  \\\n",
       "0    18.0   1.496308      1.089233    0.672271  0.630077     -1.293870    70   \n",
       "1    15.0   1.496308      1.501624    1.587959  0.853259     -1.475181    70   \n",
       "2    18.0   1.496308      1.194728    1.195522  0.549778     -1.656492    70   \n",
       "3    16.0   1.496308      1.060461    1.195522  0.546236     -1.293870    70   \n",
       "4    17.0   1.496308      1.041280    0.933897  0.565130     -1.837804    70   \n",
       "5    15.0   1.496308      2.259274    2.451322  1.618455     -2.019115    70   \n",
       "6    14.0   1.496308      2.499036    3.026898  1.633806     -2.381737    70   \n",
       "7    14.0   1.496308      2.364769    2.896085  1.584210     -2.563048    70   \n",
       "8    14.0   1.496308      2.508627    3.157710  1.717647     -2.019115    70   \n",
       "9    15.0   1.496308      1.885244    2.242022  1.038654     -2.563048    70   \n",
       "10   15.0   1.496308      1.818111    1.718772  0.699747     -2.019115    70   \n",
       "11   14.0   1.496308      1.405719    1.457147  0.754067     -2.744360    70   \n",
       "12   15.0   1.496308      1.981149    1.195522  0.933557     -2.200426    70   \n",
       "13   14.0   1.496308      2.508627    3.157710  0.136478     -2.019115    70   \n",
       "14   24.0  -0.855244     -0.771324   -0.243417 -0.706655     -0.206002    70   \n",
       "15   22.0   0.320532      0.043868   -0.243417 -0.162279     -0.024691    70   \n",
       "16   18.0   0.320532      0.053459   -0.191092 -0.231950     -0.024691    70   \n",
       "17   21.0   0.320532      0.063049   -0.505042 -0.452770      0.156620    70   \n",
       "18   27.0  -0.855244     -0.924773   -0.426554 -0.992422     -0.387314    70   \n",
       "19   26.0  -0.855244     -0.924773   -1.525380 -1.340775      1.788421    70   \n",
       "20   25.0  -0.855244     -0.800096   -0.452717 -0.352397      0.700554    70   \n",
       "21   24.0  -0.855244     -0.828867   -0.374229 -0.638165     -0.387314    70   \n",
       "22   25.0  -0.855244     -0.857639   -0.243417 -0.703112      0.700554    70   \n",
       "23   26.0  -0.855244     -0.694600    0.227509 -0.869613     -1.112559    70   \n",
       "24   21.0   0.320532      0.053459   -0.374229 -0.380738     -0.206002    70   \n",
       "25   10.0   1.496308      1.597529    2.896085  1.942010     -0.568625    70   \n",
       "26   10.0   1.496308      1.089233    2.503648  1.659785     -0.206002    70   \n",
       "27   11.0   1.496308      1.194728    2.765273  1.666870     -0.749936    70   \n",
       "28    9.0   1.496308      1.060461    2.320510  2.080171      1.063176    70   \n",
       "29   27.0  -0.855244     -0.924773   -0.426554 -0.992422     -0.387314    71   \n",
       "..    ...        ...           ...         ...       ...           ...   ...   \n",
       "368  27.0  -0.855244     -0.780915   -0.426554 -0.390185      1.099439    82   \n",
       "369  34.0  -0.855244     -0.780915   -0.426554 -0.679495      0.881865    82   \n",
       "370  31.0  -0.855244     -0.780915   -0.505042 -0.466940      0.229145    82   \n",
       "371  29.0  -0.855244     -0.560333   -0.531204 -0.525983      0.156620    82   \n",
       "372  27.0  -0.855244     -0.406885   -0.374229 -0.278003      0.881865    82   \n",
       "373  24.0  -0.855244     -0.512381   -0.321904 -0.124492      0.301669    82   \n",
       "374  23.0  -0.855244     -0.406885   -0.282660  0.076254      1.788421    82   \n",
       "375  36.0  -0.855244     -0.848048   -0.792829 -1.169551     -0.097216    82   \n",
       "376  37.0  -0.855244     -0.982316   -0.949804 -1.116412      0.954390    82   \n",
       "377  31.0  -0.855244     -0.982316   -0.949804 -1.181360      0.736816    82   \n",
       "378  38.0  -0.855244     -0.848048   -1.080617 -0.998327     -0.314789    82   \n",
       "379  36.0  -0.855244     -0.915182   -0.897479 -0.998327      0.628029    82   \n",
       "380  36.0  -0.855244     -0.704191   -0.426554 -0.956997     -0.387314    82   \n",
       "381  36.0  -0.855244     -0.828867   -0.766667 -0.903858     -0.387314    82   \n",
       "382  34.0  -0.855244     -0.819277   -0.897479 -0.856624      0.482980    82   \n",
       "383  38.0  -0.855244     -0.982316   -0.975967 -1.187264     -0.206002    82   \n",
       "384  32.0  -0.855244     -0.982316   -0.975967 -1.187264      0.047833    82   \n",
       "385  38.0  -0.855244     -0.982316   -0.975967 -1.151838      0.229145    82   \n",
       "386  25.0   0.320532     -0.119170    0.149021 -0.030023      0.301669    82   \n",
       "387  38.0   0.320532      0.657660   -0.505042  0.052637      0.519243    82   \n",
       "388  26.0  -0.855244     -0.358933   -0.321904 -0.455132     -0.387314    82   \n",
       "389  22.0   0.320532      0.369945    0.201346 -0.159917     -0.314789    82   \n",
       "390  32.0  -0.855244     -0.474019   -0.217254 -0.360663     -0.604887    82   \n",
       "391  36.0  -0.855244     -0.560333   -0.531204 -0.709016     -0.931247    82   \n",
       "392  27.0  -0.855244     -0.406885   -0.374229 -0.024119      0.628029    82   \n",
       "393  27.0  -0.855244     -0.512381   -0.478879 -0.213056      0.011571    82   \n",
       "394  44.0  -0.855244     -0.924773   -1.368405 -0.992422      3.275173    82   \n",
       "395  32.0  -0.855244     -0.560333   -0.531204 -0.797581     -1.438919    82   \n",
       "396  28.0  -0.855244     -0.704191   -0.662017 -0.407897      1.099439    82   \n",
       "397  31.0  -0.855244     -0.713781   -0.583529 -0.295716      1.389537    82   \n",
       "\n",
       "     origin-1  origin-2  origin-3       pred  ideal  \n",
       "0           1         0         0  16.635473   18.0  \n",
       "1           1         0         0  14.806089   15.0  \n",
       "2           1         0         0  16.233561   18.0  \n",
       "3           1         0         0  16.070099   16.0  \n",
       "4           1         0         0  16.844168   17.0  \n",
       "5           1         0         0  12.304775   15.0  \n",
       "6           1         0         0  11.659739   14.0  \n",
       "7           1         0         0  12.143611   14.0  \n",
       "8           1         0         0  11.064786   14.0  \n",
       "9           1         0         0  14.233102   15.0  \n",
       "10          1         0         0  15.099044   15.0  \n",
       "11          1         0         0  16.187143   14.0  \n",
       "12          1         0         0  15.395020   15.0  \n",
       "13          1         0         0  13.741521   14.0  \n",
       "14          0         0         1  25.938314   24.0  \n",
       "15          1         0         0  20.217829   22.0  \n",
       "16          1         0         0  20.258669   18.0  \n",
       "17          1         0         0  22.292265   21.0  \n",
       "18          0         0         1  28.020178   27.0  \n",
       "19          0         1         0  32.602875   26.0  \n",
       "20          0         1         0  24.524057   25.0  \n",
       "21          0         1         0  25.421770   24.0  \n",
       "22          0         1         0  25.130121   25.0  \n",
       "23          0         1         0  23.748926   26.0  \n",
       "24          1         0         0  21.550219   21.0  \n",
       "25          1         0         0  10.532638   10.0  \n",
       "26          1         0         0  11.583697   10.0  \n",
       "27          1         0         0  11.585458   11.0  \n",
       "28          1         0         0  10.145028    9.0  \n",
       "29          0         0         1  28.317135   27.0  \n",
       "..        ...       ...       ...        ...    ...  \n",
       "368         1         0         0  27.471544   27.0  \n",
       "369         1         0         0  28.578432   34.0  \n",
       "370         1         0         0  28.129114   31.0  \n",
       "371         1         0         0  28.066454   29.0  \n",
       "372         1         0         0  26.191919   27.0  \n",
       "373         1         0         0  25.635857   24.0  \n",
       "374         1         0         0  24.433367   23.0  \n",
       "375         0         1         0  32.644531   36.0  \n",
       "376         0         0         1  34.110847   37.0  \n",
       "377         0         0         1  34.371044   31.0  \n",
       "378         1         0         0  32.546703   38.0  \n",
       "379         1         0         0  31.883390   36.0  \n",
       "380         0         0         1  31.056963   36.0  \n",
       "381         0         0         1  32.411007   36.0  \n",
       "382         0         0         1  32.667728   34.0  \n",
       "383         0         0         1  34.561279   38.0  \n",
       "384         0         0         1  34.543659   32.0  \n",
       "385         0         0         1  34.397385   38.0  \n",
       "386         1         0         0  22.159510   25.0  \n",
       "387         1         0         0  22.863890   38.0  \n",
       "388         1         0         0  26.658047   26.0  \n",
       "389         1         0         0  22.432077   22.0  \n",
       "390         0         0         1  27.591600   32.0  \n",
       "391         1         0         0  28.832703   36.0  \n",
       "392         1         0         0  25.251448   27.0  \n",
       "393         1         0         0  26.605188   27.0  \n",
       "394         0         1         0  34.133556   44.0  \n",
       "395         1         0         0  29.202166   32.0  \n",
       "396         1         0         0  28.324156   28.0  \n",
       "397         1         0         0  27.590269   31.0  \n",
       "\n",
       "[398 rows x 12 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow.contrib.learn as skflow\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "# create feature vector\n",
    "missing_median(df, 'horsepower')\n",
    "df.drop('name',1,inplace=True)\n",
    "encode_numeric_zscore(df, 'horsepower')\n",
    "encode_numeric_zscore(df, 'weight')\n",
    "encode_numeric_zscore(df, 'cylinders')\n",
    "encode_numeric_zscore(df, 'displacement')\n",
    "encode_numeric_zscore(df, 'acceleration')\n",
    "encode_text_dummy(df, 'origin')\n",
    "\n",
    "# Display training data\n",
    "df\n",
    "\n",
    "# Encode to a 2D matrix for training\n",
    "x,y = to_xy(df,'mpg')\n",
    "\n",
    "# Get/clear a directory to store the neural network to\n",
    "model_dir = get_model_dir('mpg',True)\n",
    "\n",
    "# Create a deep neural network with 3 hidden layers of 50, 25, 10\n",
    "feature_columns = [tf.contrib.layers.real_valued_column(\"\", dimension=x.shape[1])]\n",
    "regressor = skflow.DNNRegressor(\n",
    "    model_dir= model_dir,\n",
    "    feature_columns=feature_columns,\n",
    "    hidden_units=[50, 25, 10])\n",
    "\n",
    "# Fit/train neural network\n",
    "regressor.fit(x, y,steps=1000)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "pred = list(regressor.predict(x, as_iterable=True))\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(\"Final score (RMSE): {}\".format(score))\n",
    "\n",
    "# How to make many predictions\n",
    "pred = list(regressor.predict(x, as_iterable=True))\n",
    "predDF = pd.DataFrame(pred)\n",
    "df2 = pd.concat([df,predDF,pd.DataFrame(y)],axis=1)\n",
    "\n",
    "df2.columns = list(df.columns)+['pred','ideal']\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Load/Save Trained Network\n",
    "\n",
    "When the neural network is created we must specify a directory that the neural network is saved to.  Often we clear out that directory so that training can start over.  This clearing is accomplished by calling the **get_model_dir** method with the second paramater of True.  The code below trains an iris neural network, exactly as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1\n",
      "}\n",
      ", '_save_summary_steps': 100, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001B601F411D0>, '_num_ps_replicas': 0, '_environment': 'local', '_master': '', '_task_type': None, '_keep_checkpoint_every_n_hours': 10000, '_tf_random_seed': None, '_keep_checkpoint_max': 5, '_task_id': 0, '_save_checkpoints_secs': 600, '_save_checkpoints_steps': None, '_evaluation_master': '', '_is_chief': True}\n",
      "WARNING:tensorflow:From <ipython-input-6-8486842bfe98>:38: calling BaseEstimator.fit (from tensorflow.contrib.learn.python.learn.estimators.estimator) with x is deprecated and will be removed after 2016-12-01.\n",
      "Instructions for updating:\n",
      "Estimator is decoupled from Scikit Learn interface by moving into\n",
      "separate class SKCompat. Arguments x, y and batch_size are only\n",
      "available in the SKCompat class, Estimator will only accept input_fn.\n",
      "Example conversion:\n",
      "  est = Estimator(...) -> est = SKCompat(Estimator(...))\n",
      "WARNING:tensorflow:From <ipython-input-6-8486842bfe98>:38: calling BaseEstimator.fit (from tensorflow.contrib.learn.python.learn.estimators.estimator) with y is deprecated and will be removed after 2016-12-01.\n",
      "Instructions for updating:\n",
      "Estimator is decoupled from Scikit Learn interface by moving into\n",
      "separate class SKCompat. Arguments x, y and batch_size are only\n",
      "available in the SKCompat class, Estimator will only accept input_fn.\n",
      "Example conversion:\n",
      "  est = Estimator(...) -> est = SKCompat(Estimator(...))\n",
      "WARNING:tensorflow:From C:\\Anaconda3\\envs\\tf-latest\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\tf-latest\\lib\\site-packages\\tensorflow\\python\\util\\deprecation.py:247: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.\n",
      "  equality = a == b\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into .\\dnn\\iris\\model.ckpt.\n",
      "INFO:tensorflow:loss = 1.09609, step = 1\n",
      "INFO:tensorflow:global_step/sec: 729.287\n",
      "INFO:tensorflow:loss = 0.228442, step = 101\n",
      "INFO:tensorflow:global_step/sec: 724.2\n",
      "INFO:tensorflow:loss = 0.135569, step = 201\n",
      "INFO:tensorflow:global_step/sec: 718.924\n",
      "INFO:tensorflow:loss = 0.102327, step = 301\n",
      "INFO:tensorflow:global_step/sec: 799.427\n",
      "INFO:tensorflow:loss = 0.0853535, step = 401\n",
      "INFO:tensorflow:global_step/sec: 825.845\n",
      "INFO:tensorflow:loss = 0.0745204, step = 501\n",
      "INFO:tensorflow:global_step/sec: 786.828\n",
      "INFO:tensorflow:loss = 0.0664632, step = 601\n",
      "INFO:tensorflow:global_step/sec: 812.428\n",
      "INFO:tensorflow:loss = 0.060262, step = 701\n",
      "INFO:tensorflow:global_step/sec: 799.419\n",
      "INFO:tensorflow:loss = 0.0549611, step = 801\n",
      "INFO:tensorflow:global_step/sec: 819.083\n",
      "INFO:tensorflow:loss = 0.050359, step = 901\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into .\\dnn\\iris\\model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.0451988.\n",
      "WARNING:tensorflow:From C:\\Anaconda3\\envs\\tf-latest\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\dnn.py:374: calling BaseEstimator.predict (from tensorflow.contrib.learn.python.learn.estimators.estimator) with x is deprecated and will be removed after 2016-12-01.\n",
      "Instructions for updating:\n",
      "Estimator is decoupled from Scikit Learn interface by moving into\n",
      "separate class SKCompat. Arguments x, y and batch_size are only\n",
      "available in the SKCompat class, Estimator will only accept input_fn.\n",
      "Example conversion:\n",
      "  est = Estimator(...) -> est = SKCompat(Estimator(...))\n",
      "Accuarcy before save: 0.9866666666666667\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.contrib.learn as learn\n",
    "from sklearn import metrics\n",
    "import pandas as pd\n",
    "import os\n",
    "from sklearn import preprocessing\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "# Read iris dataset\n",
    "filename_read = os.path.join(path,\"iris.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "# Extract just the columns we shall predict on\n",
    "encode_numeric_zscore(df,'sepal_l')\n",
    "encode_numeric_zscore(df,'sepal_w')\n",
    "encode_numeric_zscore(df,'petal_l')\n",
    "encode_numeric_zscore(df,'petal_w')\n",
    "species = encode_text_index(df,'species')\n",
    "num_classes = len(species)\n",
    "\n",
    "# Create x(predictors) and y (expected outcome)\n",
    "x,y = to_xy(df,'species')\n",
    "\n",
    "# Get/clear a directory to store the neural network to\n",
    "model_dir = get_model_dir('iris',True)\n",
    "\n",
    "# Create a deep neural network with 3 hidden layers of 10, 20, 5\n",
    "feature_columns = [tf.contrib.layers.real_valued_column(\"\", dimension=x.shape[1])]\n",
    "classifier = learn.DNNClassifier(\n",
    "    model_dir= model_dir,\n",
    "    hidden_units=[10, 20, 5], n_classes=num_classes, feature_columns=feature_columns)\n",
    "\n",
    "# Might be needed in future versions of \"TensorFlow Learn\"\n",
    "#classifier = learn.SKCompat(classifier) # For Sklearn compatibility\n",
    "\n",
    "# Fit/train neural network\n",
    "classifier.fit(x, y, steps=1000)\n",
    "\n",
    "# Measure accuracy\n",
    "pred = list(classifier.predict(x, as_iterable=True))\n",
    "score = metrics.accuracy_score(y, pred)\n",
    "print(\"Accuarcy before save: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below sets up a neural network and reads the data (for predictions), but it does not clear the model directory or fit the neural network.  The weights from the previous fit are used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuarcy after load: 0.9866666666666667\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.contrib.learn as learn\n",
    "from sklearn import metrics\n",
    "import pandas as pd\n",
    "import os\n",
    "from sklearn import preprocessing\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "# Read iris dataset\n",
    "filename_read = os.path.join(path,\"iris.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "# Extract just the columns we shall predict on\n",
    "encode_numeric_zscore(df,'sepal_l')\n",
    "encode_numeric_zscore(df,'sepal_w')\n",
    "encode_numeric_zscore(df,'petal_l')\n",
    "encode_numeric_zscore(df,'petal_w')\n",
    "species = encode_text_index(df,'species')\n",
    "num_classes = len(species)\n",
    "\n",
    "# Create x(predictors) and y (expected outcome)\n",
    "x,y = to_xy(df,'species')\n",
    "\n",
    "# Get/clear a directory to store the neural network to\n",
    "model_dir = get_model_dir('iris',False)  # False, do not clear!!! Use saved network.\n",
    "\n",
    "# Create a deep neural network with 3 hidden layers of 10, 20, 5\n",
    "feature_columns = [tf.contrib.layers.real_valued_column(\"\", dimension=x.shape[1])]\n",
    "classifier = learn.DNNClassifier(\n",
    "    model_dir= model_dir,\n",
    "    hidden_units=[10, 20, 5], n_classes=num_classes, feature_columns=feature_columns)\n",
    "\n",
    "# Do not fit!!!\n",
    "\n",
    "# Measure accuracy\n",
    "pred = list(classifier.predict(x, as_iterable=True))\n",
    "score = metrics.accuracy_score(y, pred)\n",
    "print(\"Accuarcy after load: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# View Graph for Neural Network\n",
    "\n",
    "TensorFlow includes the command line utility called [**tensorboard**](https://www.tensorflow.org/versions/r0.10/how_tos/summaries_and_tensorboard/index.html) that can be used to visualize the neural networks.  It is not needed for this course, but it can be handy to see your neural network, and I will use it in lecture a few times.  It does not work with IBM Data Scientist Workbench, so you will need a native install if you would like to use it.\n",
    "\n",
    "To make use of it, you must specify a **logdir** on the fit command, for example:\n",
    "\n",
    "```\n",
    "classifier.fit(x, y, logdir='./log/')\n",
    "```\n",
    "\n",
    "Once the **fit** occurs, the logdir will be filled with files that **tensorboard** will use.  To view the graph, issue the following command:\n",
    "\n",
    "```\n",
    "tensorboard --logdir ./log\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}