Post_training_quantization_Guides.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Post-training quantization Guides.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyOQUtAWiSarhzouY8fqMe8S",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jvishnuvardhan/TF_Lite/blob/master/Post_training_quantization_Guides.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ofb5T4ayBiMf",
        "colab_type": "text"
      },
      "source": [
        "# Post-training quantization Guides\n",
        "\n",
        "Without degrading the model accuracy, Post-training quantization includes general techniques to reduce\n",
        "*   CPU and hardware accelerator latency\n",
        "*   processing\n",
        "*   power, and\n",
        "*   model size\n",
        "\n",
        "These techniques can be performed on an already-trained float TensorFlow model and applied during TensorFlow Lite conversion. These techniques are enabled as options in the TensorFlow Lite converter.\n",
        "\n",
        "Different post training quantization types\n",
        "*   float quantized model\n",
        "*   Post-training float16 quantization\n",
        "*   Post-training dynamic range quantization\n",
        "*   Post-training full integer quantization\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SKAEVXg4EOK_",
        "colab_type": "text"
      },
      "source": [
        "**1. Quantizing weights**.  \n",
        "\n",
        "Weights can be converted to types with reduced precision, such as 16 bit floats or 8 bit integers. We generally recommend 16-bit floats for GPU acceleration and 8-bit integer for CPU execution.\n",
        "\n",
        "For example, here is how to specify 8 bit integer weight quantization:\n",
        "\n",
        "\n",
        "\n",
        "At inference, the most critically intensive parts are computed with 8 bits instead of floating point. There is some inference-time performance overhead, relative to quantizing both weights and activations below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O1p2MKTQEM-R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)\n",
        "converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE].  # specifies int8 weight quatization (activations still are float)\n",
        "tflite_quant_model = converter.convert()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_r0jSPlHFSIJ",
        "colab_type": "text"
      },
      "source": [
        "**2. Full integer quantization of weights and activations**.  \n",
        "Improve latency, processing, and power usage, and get access to integer-only hardware accelerators by making sure both weights and activations are quantized. This requires a small representative data set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ut8sDQi5BhIv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "def representative_dataset_gen():\n",
        "  for _ in range(num_calibration_steps):\n",
        "    # Get sample input data as a numpy array in a method of your choosing.\n",
        "    yield [input]\n",
        "\n",
        "converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)\n",
        "converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
        "converter.representative_dataset = representative_dataset_gen.    # this makes the activations to be converted to int8\n",
        "tflite_quant_model = converter.convert()\n",
        "# The resulting model will still take float input and output for convenience"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}