Demo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 2
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython2",
      "version": "2.7.6"
    },
    "colab": {
      "name": "Demo.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "MoHa0jLjxOdb"
      },
      "source": [
        "# COMP0087 Group Project - Python Code Generator\n",
        "Code generation is becoming an important and trending field in natural language processing (NLP), as it could potentially help to improve programming productivity by developing automatic code. Given some natural language (NL) utterances, the code generator aims to output some source code that completes the task described in the NL intents. Many models for the code generation task have been proposed by the researchers. In particular, TranX is a transition-based neural abstract syntax parser for code generation, it achieves state-of-the-art results on the CoNaLa dataset.\n",
        "\n",
        "However, existing code generation models suffer from various problems. For example, TRANX often leads to disadvantageous performance when dealing with long and complex code generation tasks. Furthermore, current code generators suffer from learning dependencies between distant positions. In particular, TRANX uses standard bidirectional Long Short-term Memory (LSTM) network as the encoder and decoder, which may lead to this issue due to its sequential computation. TranX also has high complexity and high computational cost due to its recurrent layer type.\n",
        "\n",
        "To solve these problems, this project explores potential solutions by using TRANX as the baseline, experimenting and modifying the encoder with different networks like Gated Recurrent Units (GRUs) and attentional encoder. In particular, for the code generation task on CoNaLa dataset, both candidate models achieved slightly better results than TRANX in terms of the exact match. In addition, the training time per epoch for our candidate models is less if compared with TRANX. Furthermore, we found that GRU is less inclined to use implicit information. The attentional encoder has a stronger ability in dealing with long context intent and distinguishing ambiguous words.\n",
        "\n",
        "![jupyter](https://i.postimg.cc/X7J8xftm/result-table.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "o-BL3pVnxOde"
      },
      "source": [
        "----\n",
        "## 1 System Architecture\n",
        "TRANX is a seq-to-action model, in which the input is the natural language utterances that described the task and the output is a series of actions corresponding to some Python source code that completes the task. Please find the workflow of TRANX below.\n",
        "\n",
        "![jupyter](https://i.postimg.cc/JzDvG5RY/Work-flow.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "eBrca52-xOdf"
      },
      "source": [
        "TRANX employs an encoder-decoder structure to score AST by measuring the probability of a series of actions. TRANX uses a Bi-LSTM network for the encoder and a standard LSTM for the decoder. This project explores and replaces the encoder with two different network structures: Gated Recurrent Units (GRUs) and attentional encoder.\n",
        "\n",
        "Figure below gives brief overview of the partial system for the original TRANX model.\n",
        "\n",
        "![jupyter](https://i.postimg.cc/25tKYHvs/TRANX.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C_VQgytfsUiA"
      },
      "source": [
        "For TRANX_GRU, we replace the encoder part with a GRU network. In graphical representations, we change the LSTM encoder (highlighted by the dotted squared) in the TRANX architecture above with a GRU network as shown in the figure below.\n",
        "![jupyter](https://i.postimg.cc/7hRW2Wsf/GRU.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "fygiei2ExOdg"
      },
      "source": [
        "For TRANX_attentional_encoder, we change the encoder part with an attentional encoder, which is also the encoder of the transformer. The corresponding changed part is shown below.\n",
        "\n",
        "![jupyter](https://i.postimg.cc/N0GSZmHG/Transformer-enc.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "XAjAshgwxOdh"
      },
      "source": [
        "----\n",
        "## 2 Project Setup and Download Project Repo\n",
        "This project can be either run on colab or the local machine. Please find the project set up in the corresponding subsection below. To run it without CUDA, please simply remove the \"--cuda\" flag from the command line argument in all shell scripts under the file named \"scripts\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "aI9iWC2ExOdi"
      },
      "source": [
        "### 2.1 Colab Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p0SOln6mxVXQ"
      },
      "source": [
        "Running the cell below will require you to enter an authorization code. Please kindly follow the instructions and go to the URL shown below, log in to your google account to get your own authorization code. Note that the authorization code will be asked **TWICE**, which means you may need to log in twice. Thanks for your patient."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i7H_4JM1xOdi",
        "outputId": "0bb7aeb2-8a4d-4dbc-aff5-49da703c3b04"
      },
      "source": [
        "# Setup\n",
        "from google import colab\n",
        "colab.drive.mount('/content/drive')\n",
        "\n",
        "# Imports, login, connect drive\n",
        "import os\n",
        "from pathlib import Path\n",
        "import requests\n",
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "from googleapiclient.discovery import build\n",
        "drive = build('drive', 'v3').files()\n",
        "\n",
        "# Recursively get names\n",
        "def get_path(file_id):\n",
        "    f = drive.get(fileId=file_id, fields='name, parents').execute()\n",
        "    name = f.get('name')\n",
        "    if f.get('parents'):\n",
        "        parent_id = f.get('parents')[0]  # assume 1 parent\n",
        "        return get_path(parent_id) / name\n",
        "    else:\n",
        "        return Path(name)\n",
        "\n",
        "# Change directory\n",
        "def chdir_notebook():\n",
        "    d = requests.get('http://172.28.0.2:9000/api/sessions').json()[0]\n",
        "    file_id = d['path'].split('=')[1]\n",
        "    path = get_path(file_id)\n",
        "    nb_dir = 'drive' / path.parent\n",
        "    os.chdir(nb_dir)\n",
        "    return nb_dir\n",
        "\n",
        "!cd /\n",
        "chdir_notebook()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PosixPath('drive/My Drive/Colab Notebooks')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oH5-km1ccMay"
      },
      "source": [
        "The code that used to train and evaluate our models is available on https://github.com/kzCassie/ucl\\_nlp. Note that you should only run the following code **ONCE**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nCqVDIhycLF-",
        "outputId": "1e8776d6-3dc6-49ad-bf0b-731ac6bc770c"
      },
      "source": [
        "# Clone the project repo\n",
        "%%shell\n",
        "git clone 'https://github.com/kzCassie/ucl_nlp.git'"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'ucl_nlp'...\n",
            "remote: Enumerating objects: 617, done.\u001b[K\n",
            "remote: Counting objects: 100% (617/617), done.\u001b[K\n",
            "remote: Compressing objects: 100% (446/446), done.\u001b[K\n",
            "remote: Total 617 (delta 300), reused 421 (delta 154), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (617/617), 128.51 MiB | 17.52 MiB/s, done.\n",
            "Resolving deltas: 100% (300/300), done.\n",
            "Checking out files: 100% (160/160), done.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              ""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "945h8XRHfbz1",
        "outputId": "1a6ba1b6-a594-40f5-d822-10b1b8736759"
      },
      "source": [
        "cd ucl_nlp"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/Colab Notebooks/ucl_nlp\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "xYH_LqFKxOdj"
      },
      "source": [
        "### 2.2 Local Machine Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LeFi-Qfxb7jI"
      },
      "source": [
        "When runing the project on local machine, please follow the instructions below. If you are using Colab, then please **IGNORE** section 2.2."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "iYuRK7lDxOdj"
      },
      "source": [
        "# Clone our project repository into the local machine\n",
        "git clone https://github.com/kzCassie/ucl_nlp\n",
        "# Enter the project file\n",
        "cd ucl_nlp\n",
        "\n",
        "# Create virtual environments\n",
        "python3 -m venv config/env\n",
        "# Activate virtual environment\n",
        "source config/env/bin/activate\n",
        "# Install all the required packages\n",
        "pip install -r requirements.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "cdwP2nnBxOdj"
      },
      "source": [
        "----\n",
        "## 3 Data Loading & Data Preprocessing\n",
        "\n",
        "Run the following shell script to get the Conala json file from http://www.phontron.com/download/conala-corpus-v1.1.zip, and download the preprocessed Conala zipfile from GoogleDrive.\n",
        "\n",
        "The CoNaLa dataset was originally released by CMU. This dataset contains mined corpus of natural language intents and Python code snippets from Stack Overflow. The original data is in JSON format and comes in 3 parts: first, 2379 manually-curated training examples, second 500 human-curated test examples and thirdly 600k automatically-mined examples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L8gLoTJ2xOdk",
        "outputId": "53a4d1e9-1d0d-421d-ac54-ae00b6b6b7af"
      },
      "source": [
        "# Download original Conala json dataset\n",
        "# Download pre-processed Colana zipfile from GoogleDrive\n",
        "!bash pull_data.sh"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "download original Conala json dataset\n",
            "--2021-06-01 05:19:51--  http://www.phontron.com/download/conala-corpus-v1.1.zip\n",
            "Resolving www.phontron.com (www.phontron.com)... 208.113.196.149\n",
            "Connecting to www.phontron.com (www.phontron.com)|208.113.196.149|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 52105440 (50M) [application/zip]\n",
            "Saving to: ‘data/conala-corpus-v1.1.zip’\n",
            "\n",
            "conala-corpus-v1.1. 100%[===================>]  49.69M  25.1MB/s    in 2.0s    \n",
            "\n",
            "2021-06-01 05:19:53 (25.1 MB/s) - ‘data/conala-corpus-v1.1.zip’ saved [52105440/52105440]\n",
            "\n",
            "Archive:  data/conala-corpus-v1.1.zip\n",
            "   creating: data/conala-corpus/\n",
            "  inflating: data/conala-corpus/conala-mined.jsonl  \n",
            "  inflating: data/conala-corpus/conala-train.json  \n",
            "  inflating: data/conala-corpus/conala-test.json  \n",
            "mv: missing destination file operand after 'data/conala-corpus'\n",
            "Try 'mv --help' for more information.\n",
            "download preprocessed Conala zip from GoogleDrive\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100   408    0   408    0     0   3138      0 --:--:-- --:--:-- --:--:--  3114\n",
            "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
            "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
            "100  188M    0  188M    0     0  50.9M      0 --:--:--  0:00:03 --:--:-- 69.6M\n",
            "Archive:  data/100000.zip\n",
            "   creating: data/conala/100000/\n",
            "  inflating: data/conala/__MACOSX/._100000  \n",
            "  inflating: data/conala/100000/.DS_Store  \n",
            "  inflating: data/conala/__MACOSX/100000/._.DS_Store  \n",
            "  inflating: data/conala/100000/dev.bin  \n",
            "  inflating: data/conala/100000/train.gold.full.bin  \n",
            "  inflating: data/conala/100000/train.all_100000.bin  \n",
            "  inflating: data/conala/100000/mined_100000.bin  \n",
            "  inflating: data/conala/100000/test.bin  \n",
            "  inflating: data/conala/100000/vocab.src_freq3.code_freq3.mined_100000.bin  \n",
            "   creating: data/conala/100000/debug/\n",
            "  inflating: data/conala/100000/debug/conala-test.json.debug  \n",
            "  inflating: data/conala/100000/debug/conala-train.json.debug  \n",
            "  inflating: data/conala/100000/debug/conala-mined.jsonl.debug  \n",
            "Done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "Df69_g7bxOdk"
      },
      "source": [
        "**Clarification on Data Preprocessing**\n",
        "\n",
        "Hint: You don't need to preprocess it again **(YOU DON'T NEED TO RUN THE FOLLOWING CODE)**, as the code above has already helped you to download our preprocessed data. It is used for illustration and clarification.\n",
        "\n",
        "The data were preprocessed with the downloaded mined file and topk=100000 (First k number from mined file) using the code below. This means we use all of the human-curated data, but only 100k automatically-mined examples for training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7N2hlz7YgjWX"
      },
      "source": [
        "mined_data_file = \"data/conala-corpus/conala-mined.jsonl\" # path to the downloaded mined file\n",
        "topk = 100000 # number of pretraining data to be preprocessed\n",
        "!python datasets/conala/dataset.py --pretrain=$mined_data_file --topk=$topk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i1IweRirLid7"
      },
      "source": [
        "After data pre-processing, the NL intents were transformed into lists of tokens, while the Python source code snippets become series of actions through the AST parsing process as explained before. Please see the example of the preprocessed data below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "stdHy7U4h13U",
        "outputId": "dfd710b5-6ad1-4fa7-b647-13a235d5d7eb"
      },
      "source": [
        "# example of pre-processed data.\n",
        "from components.dataset import Dataset\n",
        "n_example = 3\n",
        "train_set = Dataset.from_bin_file(\"data/conala/100000/train.gold.full.bin\")\n",
        "for src, tgt in zip(train_set.all_source[:n_example],train_set.all_targets[:n_example]):\n",
        "    print(f'Source:{src} \\nTarget:{tgt} \\n')\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Source:['concatenate', 'elements', 'of', 'a', 'list', 'str_0', 'of', 'multiple', 'integers', 'to', 'a', 'single', 'integer'] \n",
            "Target:sum(d * 10 ** i for i, d in enumerate(str_0[::-1])) \n",
            "\n",
            "Source:['convert', 'a', 'list', 'of', 'integers', 'into', 'a', 'single', 'integer'] \n",
            "Target:r = int(''.join(map(str, x))) \n",
            "\n",
            "Source:['convert', 'a', 'datetime', 'string', 'back', 'to', 'a', 'datetime', 'object', 'of', 'format', 'str_0'] \n",
            "Target:datetime.strptime('2010-11-13 10:33:54.227806', 'str_0') \n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vL0LxpWbL3KZ"
      },
      "source": [
        "We preprocess the json files into several bin files and save them to the folder named data/canola/${topk}. These preprocessed files are then used in the next section for training, fine-tuning and testing. In particular, we held out 200 examples from the 2379 manually-curated training data as the validation set. We first trained the model from scratch with the 100k un-curated examples along with the 200 validation examples. Next, we fine-tuned the obtained model using the 2179 remaining training data and evaluated the same 200 evaluation examples to form the final model. In the end, we applied the fine-tuned model on the 500 test set on which the results are reported."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "LhjrGXivxOdm"
      },
      "source": [
        "----\n",
        "## 4 Model Training & Fine-tuning\n",
        "\n",
        "Hint: It takes several hours to train and fine-tune the model. The best-pre-trained models can be downloaded using the following shell script, **if you just want to test the model to see the model performance, please go to section 5.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UdyzU0aJBhTA"
      },
      "source": [
        "### Download best pretrained models from GoogleDrive\n",
        "\n",
        "We have uploaded the best-pretrained models to an open-accessed GoogleDrive file, please kindly use the shell script *pull_best_models.sh* to download these models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t2p13S-RBmeM",
        "outputId": "4fd69455-846f-4888-b6cd-5eaf5ef31bf4"
      },
      "source": [
        "# Download best pretrained models zipfile from GoogleDrive\n",
        "!bash pull_best_models.sh"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "download trained_best_models from GoogleDrive\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100   408    0   408    0     0   2775      0 --:--:-- --:--:-- --:--:--  2756\n",
            "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
            "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
            "100 54.9M    0 54.9M    0     0  32.8M      0 --:--:--  0:00:01 --:--:-- 80.3M\n",
            "Done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XFMzFEw9r4os"
      },
      "source": [
        "### TRANX_LSTM (Baseline)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "a9JToqhGxOdm"
      },
      "source": [
        "# tranX baseline model\n",
        "# pretrain with mined_num = 100000\n",
        "！bash scripts/tranX/pretrain.sh 100000\n",
        "\n",
        "# fine-tune with the gold set\n",
        "! bash scripts/tranX/finetune.sh 100000"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "h1CpJYXnxOdm"
      },
      "source": [
        "### TRANX_GRU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "DyYSHMPHxOdn"
      },
      "source": [
        "# GRU model\n",
        "# pretrain with mined_num = 100000\n",
        "！bash scripts/GRU/pretrain.sh 100000\n",
        "\n",
        "# fine-tune with the gold set\n",
        "! bash scripts/GRU/finetune.sh 100000"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "ctzjJ4l6xOdn"
      },
      "source": [
        "### TRANX_attentional_encoder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "bBW8CpSYxOdn"
      },
      "source": [
        "# attentional_encoder\n",
        "# pretrain with mined_num = 100000\n",
        "！bash scripts/transformer/pretrain.sh 100000\n",
        "\n",
        "# fine-tune with the gold set\n",
        "! bash scripts/transformer/finetune.sh 100000"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "GRyJwe-9xOdn"
      },
      "source": [
        "----\n",
        "## 5 Model Testing with the test set provided in CoNaLa dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OElBCalpr1TS"
      },
      "source": [
        "### tranX_LSTM (Baseline)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XNvJVBfoxOdo",
        "outputId": "9524c1fe-238e-43c6-d82d-27ec250f16e2"
      },
      "source": [
        "# tranX baseline model\n",
        "!bash scripts/tranX/test.sh best_pretrained_models/finetune.conala.default_parser.hidden256.embed128.action128.field64.type64.dr0.3.lr0.001.lr_de0.5.lr_da15.beam15.vocab.src_freq3.code_freq3.mined_100000.bin.mined_100000.bin.glorot.par_state.seed0.bin 100000 default_parser"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "load model from [best_pretrained_models/finetune.conala.default_parser.hidden256.embed128.action128.field64.type64.dr0.3.lr0.001.lr_de0.5.lr_da15.beam15.vocab.src_freq3.code_freq3.mined_100000.bin.mined_100000.bin.glorot.par_state.seed0.bin]\n",
            "Decoding: 100% 466/466 [02:46<00:00,  2.80it/s]\n",
            "{'corpus_bleu': 0.3011020946247816, 'oracle_corpus_bleu': 0.42743396862112837, 'avg_sent_bleu': 0.2257651682863722, 'oracle_avg_sent_bleu': 0.38993023014444694, 'exact_match': 0.017167381974248927, 'oracle_exact_match': 0.06652360515021459}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "ZCWkscq9xOdo"
      },
      "source": [
        "### tranX_GRU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zzp0ENgfxOdo",
        "outputId": "41b4af3e-4c30-4397-85a8-7c5dc2357dd1"
      },
      "source": [
        "# GRU model\n",
        "!bash scripts/GRU/test.sh best_pretrained_models/finetune.conala.gru_parser.hidden256.embed128.action128.field64.type64.dr0.3.lr0.001.lr_de0.5.lr_da15.beam15.vocab.src_freq3.code_freq3.mined_100000.bin.mined_100000.bin.glorot.par_state.seed0.bin 100000 gru_parser"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "load model from [best_pretrained_models/finetune.conala.gru_parser.hidden256.embed128.action128.field64.type64.dr0.3.lr0.001.lr_de0.5.lr_da15.beam15.vocab.src_freq3.code_freq3.mined_100000.bin.mined_100000.bin.glorot.par_state.seed0.bin]\n",
            "Decoding: 100% 466/466 [02:41<00:00,  2.88it/s]\n",
            "{'corpus_bleu': 0.2856295136099089, 'oracle_corpus_bleu': 0.4230469350759587, 'avg_sent_bleu': 0.2249070823860801, 'oracle_avg_sent_bleu': 0.39904222801804207, 'exact_match': 0.030042918454935622, 'oracle_exact_match': 0.08369098712446352}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "pseArDnpxOdo"
      },
      "source": [
        "### tranX_attentional_encoder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "EGv2-_tSxOdp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b744ef6f-3a95-4d46-b8e2-eddf0c2d51b1"
      },
      "source": [
        "# attentional_encoder\n",
        "!bash scripts/transformer/test.sh best_pretrained_models/finetune.conala.transformer_enc_parser.enc_nhead2.enc_nlayer1.hidden256.embed128.action128.field64.type64.dr0.3.lr0.001.lr_de0.5.lr_da15.beam15.vocab.src_freq3.code_freq3.mined_100000.bin.mined_100000.bin.glorot.par_state.seed0.bin 100000 transformer_enc_parser"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "load model from [best_pretrained_models/finetune.conala.transformer_enc_parser.enc_nhead2.enc_nlayer1.hidden256.embed128.action128.field64.type64.dr0.3.lr0.001.lr_de0.5.lr_da15.beam15.vocab.src_freq3.code_freq3.mined_100000.bin.mined_100000.bin.glorot.par_state.seed0.bin]\n",
            "Decoding: 100% 466/466 [03:01<00:00,  2.56it/s]\n",
            "{'corpus_bleu': 0.2819106289394482, 'oracle_corpus_bleu': 0.4184672856242742, 'avg_sent_bleu': 0.21514487082307537, 'oracle_avg_sent_bleu': 0.36651935234022703, 'exact_match': 0.023605150214592276, 'oracle_exact_match': 0.06223175965665236}\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}