GPT_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "GPT-2.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ak9250/gpt-2-colab/blob/master/GPT_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "Pzxl1vYX-1kk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Setup:\n",
        "\n",
        "1) Make sure GPU is enabled, go to edit->notebook settings->Hardware Accelerator GPU\n",
        "\n",
        "2) make a copy to your google drive, click on copy to drive in panel"
      ]
    },
    {
      "metadata": {
        "id": "iW0abT07ZkhZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note: colab will reset after 12 hours make sure to save your model checkpoints to google drive around 10-11 hours mark or before, then go to runtime->reset all runtimes. Now copy your train model back into colab and start training again from the previous checkpoint."
      ]
    },
    {
      "metadata": {
        "id": "iLXW02eIYpcB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "clone and cd into repo"
      ]
    },
    {
      "metadata": {
        "id": "ICYu3w9hIJkC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/nshepperd/gpt-2.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6eEIs3ApZUVO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cd gpt-2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Qtn1qZPgZLb0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "install requirements"
      ]
    },
    {
      "metadata": {
        "id": "434oOx0bZH6J",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip3 install -r requirements.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WvUQhgK3PQ4L",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "mount drive to access google drive for saving and accessing checkpoints later"
      ]
    },
    {
      "metadata": {
        "id": "FNpf6R4ahYSN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "o1hrgeKFYsuE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "download the model"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "A498TySgHYyF",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!sh download_model.sh 117M"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7oJPQtdLbbeK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!export PYTHONIOENCODING=UTF-8"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0KzSbAvePgsI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "fetch checkpoints if you have them saved in google drive"
      ]
    },
    {
      "metadata": {
        "id": "cA2Wk7yIPmS6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!cp -r /content/drive/My\\ Drive/checkpoint/ /content/gpt-2/ "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0p--9zwqQRTc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "lets get our text to train on, in this case from project gutenberg, A Tale of Two Cities, by Charles Dickens"
      ]
    },
    {
      "metadata": {
        "id": "QOCvrs-DHvxa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!wget https://www.gutenberg.org/files/98/98-0.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yPfJ5b3CQXqr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "start training"
      ]
    },
    {
      "metadata": {
        "id": "pEn_ihcGI00T",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!PYTHONPATH=src ./train.py --dataset /content/gpt-2/98-0.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vS1RJJDFOPnb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "save our checkpoints to start training again later"
      ]
    },
    {
      "metadata": {
        "id": "JretqG1zOXdi",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!cp -r /content/gpt-2/checkpoint/ /content/drive/My\\ Drive/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6D-i7vERWbNS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "use your trained model"
      ]
    },
    {
      "metadata": {
        "id": "VeETvWvrbKga",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!cp -r /content/gpt-2/checkpoint/run1/* /content/gpt-2/models/117M/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GmnSrXqtfRbq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Conditional samples"
      ]
    },
    {
      "metadata": {
        "id": "utJj-iY4gHwE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!python3 src/interactive_conditional_samples.py"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "K8rSqkGxg5OK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "unconditional samples"
      ]
    },
    {
      "metadata": {
        "id": "LaQUEnRxWc3c",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!python3 src/generate_unconditional_samples.py"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}