1003_hyperparameter_tuning_tf_scikit.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "1003-hyperparameter_tuning_tf_scikit.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/parekhakhil/pyImageSearch/blob/main/1003_hyperparameter_tuning_tf_scikit.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j4RpK9pawQzP"
      },
      "source": [
        "# Hyperparameter tuning for Deep Learning with scikit-learn, Keras, and TensorFlow\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ntZ1AkXZIxY"
      },
      "source": [
        "\n",
        "\n",
        "This notebook is associated with the [Hyperparameter tuning for Deep Learning with scikit-learn, Keras, and TensorFlow](https://www.pyimagesearch.com/2021/05/31/hyperparameter-tuning-for-deep-learning-with-scikit-learn-keras-and-tensorflow/) blog post published on 2021-05-31.\n",
        "\n",
        "Only the code for the blog post is here. Most codeblocks have a 1:1 relationship with what you find in the blog post with two exceptions: (1) Python classes are not separate files as they are typically organized with PyImageSearch projects, and (2) Command Line Argument parsing is replaced with an `args` dictionary that you can manipulate as needed.\n",
        "\n",
        "We recommend that you execute (press ▶️) the code block-by-block, as-is, before adjusting parameters and `args` inputs. Once you've verified that the code is working, you are welcome to hack with it and learn from manipulating inputs, settings, and parameters. For more information on using Jupyter and Colab, please refer to these resources:\n",
        "\n",
        "*   [Jupyter Notebook User Interface](https://jupyter-notebook.readthedocs.io/en/stable/notebook.html#notebook-user-interface)\n",
        "*   [Overview of Google Colaboratory Features](https://colab.research.google.com/notebooks/basic_features_overview.ipynb)\n",
        "\n",
        "\n",
        "\n",
        "Happy hacking!\n",
        "\n",
        "\n",
        "\n",
        "<hr>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NFhAzQB3aNMa"
      },
      "source": [
        "### Download the code zip file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7y0LG1EuaRlB"
      },
      "source": [
        "!wget https://pyimagesearch-code-downloads.s3-us-west-2.amazonaws.com/hyperparameter-tuning-tf-scikit/hyperparameter-tuning-tf-scikit.zip\n",
        "!unzip -qq hyperparameter-tuning-tf-scikit.zip\n",
        "%cd hyperparameter-tuning-tf-scikit"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_SgTVT3HagGZ"
      },
      "source": [
        "## Blog Post Code"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wcrOk6pURp50"
      },
      "source": [
        "### Import Packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VJaCNlDDRz6d"
      },
      "source": [
        "# import tensorflow and fix the random seed for better reproducibility\n",
        "import tensorflow as tf\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# import the necessary packages\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Flatten\n",
        "from tensorflow.keras.layers import Dropout\n",
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n",
        "from sklearn.model_selection import RandomizedSearchCV\n",
        "from tensorflow.keras.datasets import mnist"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UPcRICv15Jad"
      },
      "source": [
        "### Implementing our basic feedforward neural network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BbS0i1lC5Kij"
      },
      "source": [
        "def get_mlp_model(hiddenLayerOne=784, hiddenLayerTwo=256,\n",
        "\tdropout=0.2, learnRate=0.01):\n",
        "\t# initialize a sequential model and add layer to flatten the\n",
        "\t# input data\n",
        "\tmodel = Sequential()\n",
        "\tmodel.add(Flatten())\n",
        "\n",
        "\t# add two stacks of FC => RELU => DROPOUT\n",
        "\tmodel.add(Dense(hiddenLayerOne, activation=\"relu\",\n",
        "\t\tinput_shape=(784,)))\n",
        "\tmodel.add(Dropout(dropout))\n",
        "\tmodel.add(Dense(hiddenLayerTwo, activation=\"relu\"))\n",
        "\tmodel.add(Dropout(dropout))\n",
        "\n",
        "\t# add a softmax layer on top\n",
        "\tmodel.add(Dense(10, activation=\"softmax\"))\n",
        "\n",
        "\t# compile the model\n",
        "\tmodel.compile(\n",
        "\t\toptimizer=Adam(learning_rate=learnRate),\n",
        "\t\tloss=\"sparse_categorical_crossentropy\",\n",
        "\t\tmetrics=[\"accuracy\"])\n",
        "\n",
        "\t# return compiled model\n",
        "\treturn model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jppw5-Bd56H-"
      },
      "source": [
        "### Creating our basic training script (no hyperparameter tuning)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "okM7Bpyeq8Kc"
      },
      "source": [
        "# load the MNIST dataset\n",
        "print(\"[INFO] downloading MNIST...\")\n",
        "((trainData, trainLabels), (testData, testLabels)) = mnist.load_data()\n",
        "\n",
        "# scale data to the range of [0, 1]\n",
        "trainData = trainData.astype(\"float32\") / 255.0\n",
        "testData = testData.astype(\"float32\") / 255.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "jqilucXI6EyI",
        "outputId": "bd5ca674-a58f-478a-a12b-83d27394a6d2"
      },
      "source": [
        "# initialize our model with the default hyperparameter values\n",
        "print(\"[INFO] initializing model...\")\n",
        "model = get_mlp_model()\n",
        "\n",
        "# train the network (i.e., no hyperparameter tuning)\n",
        "print(\"[INFO] training model...\")\n",
        "H = model.fit(x=trainData, y=trainLabels,\n",
        "\tvalidation_data=(testData, testLabels),\n",
        "\tbatch_size=8,\n",
        "\tepochs=20)\n",
        "\n",
        "# make predictions on the test set and evaluate it\n",
        "print(\"[INFO] evaluating network...\")\n",
        "accuracy = model.evaluate(testData, testLabels)[1]\n",
        "print(\"accuracy: {:.2f}%\".format(accuracy * 100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[INFO] initializing model...\n",
            "[INFO] training model...\n",
            "Epoch 1/20\n",
            "7500/7500 [==============================] - 50s 7ms/step - loss: 0.8856 - accuracy: 0.7734 - val_loss: 0.4093 - val_accuracy: 0.9022\n",
            "Epoch 2/20\n",
            "7500/7500 [==============================] - 47s 6ms/step - loss: 0.6730 - accuracy: 0.8408 - val_loss: 0.4901 - val_accuracy: 0.8644\n",
            "Epoch 3/20\n",
            "7500/7500 [==============================] - 49s 7ms/step - loss: 0.6676 - accuracy: 0.8507 - val_loss: 0.4805 - val_accuracy: 0.9021\n",
            "Epoch 4/20\n",
            "7500/7500 [==============================] - 46s 6ms/step - loss: 0.6591 - accuracy: 0.8534 - val_loss: 0.4451 - val_accuracy: 0.8951\n",
            "Epoch 5/20\n",
            "7500/7500 [==============================] - 47s 6ms/step - loss: 0.6643 - accuracy: 0.8514 - val_loss: 0.4343 - val_accuracy: 0.8898\n",
            "Epoch 6/20\n",
            "7500/7500 [==============================] - 48s 6ms/step - loss: 0.6509 - accuracy: 0.8541 - val_loss: 0.4318 - val_accuracy: 0.8924\n",
            "Epoch 7/20\n",
            "7500/7500 [==============================] - 48s 6ms/step - loss: 0.6057 - accuracy: 0.8589 - val_loss: 0.4557 - val_accuracy: 0.8798\n",
            "Epoch 8/20\n",
            "7500/7500 [==============================] - 49s 7ms/step - loss: 0.7062 - accuracy: 0.8584 - val_loss: 0.5056 - val_accuracy: 0.8980\n",
            "Epoch 9/20\n",
            "7500/7500 [==============================] - 48s 6ms/step - loss: 0.6514 - accuracy: 0.8564 - val_loss: 0.4801 - val_accuracy: 0.9123\n",
            "Epoch 10/20\n",
            "7500/7500 [==============================] - 47s 6ms/step - loss: 0.6204 - accuracy: 0.8623 - val_loss: 0.4955 - val_accuracy: 0.9041\n",
            "Epoch 11/20\n",
            "7500/7500 [==============================] - 48s 6ms/step - loss: 0.6904 - accuracy: 0.8597 - val_loss: 0.5065 - val_accuracy: 0.9105\n",
            "Epoch 12/20\n",
            "7500/7500 [==============================] - 48s 6ms/step - loss: 0.6963 - accuracy: 0.8583 - val_loss: 0.4663 - val_accuracy: 0.9067\n",
            "Epoch 13/20\n",
            "7500/7500 [==============================] - 49s 7ms/step - loss: 0.6357 - accuracy: 0.8642 - val_loss: 0.5283 - val_accuracy: 0.9025\n",
            "Epoch 14/20\n",
            "7500/7500 [==============================] - 48s 6ms/step - loss: 0.6351 - accuracy: 0.8530 - val_loss: 0.5744 - val_accuracy: 0.9064\n",
            "Epoch 15/20\n",
            "7500/7500 [==============================] - 49s 7ms/step - loss: 0.6618 - accuracy: 0.8579 - val_loss: 0.4393 - val_accuracy: 0.9071\n",
            "Epoch 16/20\n",
            "7500/7500 [==============================] - 48s 6ms/step - loss: 0.6151 - accuracy: 0.8645 - val_loss: 0.6751 - val_accuracy: 0.9054\n",
            "Epoch 17/20\n",
            "7500/7500 [==============================] - 48s 6ms/step - loss: 0.6852 - accuracy: 0.8593 - val_loss: 0.5924 - val_accuracy: 0.8966\n",
            "Epoch 18/20\n",
            "7500/7500 [==============================] - 48s 6ms/step - loss: 0.6528 - accuracy: 0.8445 - val_loss: 0.5882 - val_accuracy: 0.9117\n",
            "Epoch 19/20\n",
            "7500/7500 [==============================] - 48s 6ms/step - loss: 0.6836 - accuracy: 0.8536 - val_loss: 1.0516 - val_accuracy: 0.8870\n",
            "Epoch 20/20\n",
            "7500/7500 [==============================] - 49s 6ms/step - loss: 0.7327 - accuracy: 0.8463 - val_loss: 0.8194 - val_accuracy: 0.8954\n",
            "[INFO] evaluating network...\n",
            "313/313 [==============================] - 1s 4ms/step - loss: 0.8194 - accuracy: 0.8954\n",
            "accuracy: 89.54%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KDlYU6RF6gXO"
      },
      "source": [
        "### Implementing our Keras/TensorFlow hyperparameter tuning script"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "imFpeyug6wDn"
      },
      "source": [
        "# wrap our model into a scikit-learn compatible classifier\n",
        "print(\"[INFO] initializing model...\")\n",
        "model = KerasClassifier(build_fn=get_mlp_model, verbose=0)\n",
        "\n",
        "# define a grid of the hyperparameter search space\n",
        "hiddenLayerOne = [256, 512, 784]\n",
        "hiddenLayerTwo = [128, 256, 512]\n",
        "learnRate = [1e-2, 1e-3, 1e-4]\n",
        "dropout = [0.3, 0.4, 0.5]\n",
        "batchSize = [4, 8, 16, 32]\n",
        "epochs = [10, 20, 30, 40]\n",
        "\n",
        "# create a dictionary from the hyperparameter grid\n",
        "grid = dict(\n",
        "\thiddenLayerOne=hiddenLayerOne,\n",
        "\tlearnRate=learnRate,\n",
        "\thiddenLayerTwo=hiddenLayerTwo,\n",
        "\tdropout=dropout,\n",
        "\tbatch_size=batchSize,\n",
        "\tepochs=epochs\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FYtnhD3s63Yr"
      },
      "source": [
        "# initialize a random search with a 3-fold cross-validation and then\n",
        "# start the hyperparameter search process\n",
        "print(\"[INFO] performing random search...\")\n",
        "searcher = RandomizedSearchCV(estimator=model, n_jobs=-1, cv=3,\n",
        "\tparam_distributions=grid, scoring=\"accuracy\")\n",
        "searchResults = searcher.fit(trainData, trainLabels)\n",
        "\n",
        "# summarize grid search information\n",
        "bestScore = searchResults.best_score_\n",
        "bestParams = searchResults.best_params_\n",
        "print(\"[INFO] best score is {:.2f} using {}\".format(bestScore,\n",
        "\tbestParams))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jILeoaLP66Pu"
      },
      "source": [
        "# extract the best model, make predictions on our data, and show a\n",
        "# classification report\n",
        "print(\"[INFO] evaluating the best model...\")\n",
        "bestModel = searchResults.best_estimator_\n",
        "accuracy = bestModel.score(testData, testLabels)\n",
        "print(\"accuracy: {:.2f}%\".format(accuracy * 100))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ogkNauArL6u"
      },
      "source": [
        "For a detailed walkthrough of the concepts and code, be sure to refer to the full tutorial, [*Hyperparameter tuning for Deep Learning with scikit-learn, Keras, and TensorFlow*](https://www.pyimagesearch.com/2021/05/31/hyperparameter-tuning-for-deep-learning-with-scikit-learn-keras-and-tensorflow/) published on 2021-05-31."
      ]
    }
  ]
}