10_bert_farsi_sentiment.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# default_exp bert_farsi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from nbdev.showdoc import *\n",
    "# all_flag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis with Multilingual Transformers\n",
    "> Summary: Transformers, BERT, Bert Tokenizer, Pretrained Models, Farsi Sentiment Analysis, Multiligual Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Transformer** revolves around the idea of a model that uses *attention* to increase the speed with which it can be trained. The primary motivation for designing a transformer was to enable parallel processing of the words in the sentences, i.e. to process the entire sentence at once. This parallel processing is not possible in LSTMs or RNNs or GRUs as they take words of the input sentence one by one. The first transformer was proposed in the paper [Attention is All You Need](https://arxiv.org/abs/1706.03762). There is a TensorFlow implementation of it is available [here](https://github.com/tensorflow/tensor2tensor). Also, Harvard’s NLP group provided a [guide](http://nlp.seas.harvard.edu/2018/04/03/attention.html) annotating the paper with PyTorch implementation.\n",
    "These transformer models come in different shapes, sizes, and architectures and have their own ways of accepting input data: via *tokenization*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT Overview\n",
    "\n",
    "BERT (Bidirectionnal Encoder Representations for Transformers) is a “new method of pre-training language representations” developed by Google in [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) and released in late 2018. Since it is pre-trained on generic large datasets (from Wikipedia and BooksCorpus), it can be used for a wide variety of NLP tasks like text classification, translation, summarization, and question answering. Here is the abstract from the paper:\n",
    "\n",
    "*We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications.*\n",
    "\n",
    "BERT provides pre-trained language models for English and 103 other languages that you can fine-tune to fit your needs. Fine-tuning a model means that we will slightly train it using our dataset on top of an already trained checkpoint. Here, we’ll see how to fine-tune the multilingual model to do sentiment analysis. To do that, we follow the steps below:\n",
    "\n",
    "1. Load and preprocess the data so that it can be used by the model.\n",
    "\n",
    "2. Set-up a training loop using Keras' fit API; train the model on the training data\n",
    "\n",
    "3. Evaluate the model on the testing data and compare to the actual results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Analysis on Farsi Text\n",
    "\n",
    "The following implementation shows how to use the [Transformers](https://huggingface.co/transformers/index.html) library to obtain state-of-the-art results on the sequence classification task. This library \"*provides general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet…) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between TensorFlow 2.0 and PyTorch*\". \n",
    "\n",
    "There are plenty of applications of classification on English text. Thus, the goal of my implementation is to perform sentiment classification on non-English text, in this case Farsi(Persian) language. As it takes time to run it on CPU, I created a Google Colab notebook version of the entire implementation, which you can access it [here](https://colab.research.google.com/drive/12lM8FCqEQdCUCj1b52mN3Ggw85Mj0qYV?usp=sharing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import csv\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import re\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from transformers import BertTokenizer, BasicTokenizer\n",
    "from transformers import TFBertModel, TFBertPreTrainedModel, TFBertForSequenceClassification\n",
    "from transformers import glue_convert_examples_to_features, InputExample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2.0\n"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "# Check the version of the tensorflow\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the data\n",
    "For this tutorial, I used a [Farsi dataset](https://www.kaggle.com/saeedtqp/taaghche), from Kaggle. It contains over 68000 comments about books gathered from a book website called [taaghche](https://taaghche.com/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>comment</th>\n",
       "      <th>bookname</th>\n",
       "      <th>rate</th>\n",
       "      <th>bookID</th>\n",
       "      <th>like</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1395/11/14</td>\n",
       "      <td>اسم کتاب   No one writes to the Colonel\\nترجمش...</td>\n",
       "      <td>سرهنگ کسی ندارد برایش نامه بنویسد</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1395/11/14</td>\n",
       "      <td>طاقچه عزیز،نام کتاب\"کسی به سرهنگ نامه نمینویسد...</td>\n",
       "      <td>سرهنگ کسی ندارد برایش نامه بنویسد</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1394/06/06</td>\n",
       "      <td>بنظرم این اثر مارکز خیلی از صد سال تنهایی که ب...</td>\n",
       "      <td>سرهنگ کسی ندارد برایش نامه بنویسد</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1393/09/02</td>\n",
       "      <td>به نظر کتاب خوبی میومد اما من از ترجمش خوشم نی...</td>\n",
       "      <td>سرهنگ کسی ندارد برایش نامه بنویسد</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1393/06/29</td>\n",
       "      <td>کتاب خوبی است</td>\n",
       "      <td>سرهنگ کسی ندارد برایش نامه بنویسد</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date                                            comment  \\\n",
       "0  1395/11/14  اسم کتاب   No one writes to the Colonel\\nترجمش...   \n",
       "1  1395/11/14  طاقچه عزیز،نام کتاب\"کسی به سرهنگ نامه نمینویسد...   \n",
       "2  1394/06/06  بنظرم این اثر مارکز خیلی از صد سال تنهایی که ب...   \n",
       "3  1393/09/02  به نظر کتاب خوبی میومد اما من از ترجمش خوشم نی...   \n",
       "4  1393/06/29                                      کتاب خوبی است   \n",
       "\n",
       "                            bookname  rate  bookID  like  \n",
       "0  سرهنگ کسی ندارد برایش نامه بنویسد   0.0     3.0   2.0  \n",
       "1  سرهنگ کسی ندارد برایش نامه بنویسد   5.0     3.0   2.0  \n",
       "2  سرهنگ کسی ندارد برایش نامه بنویسد   5.0     3.0   0.0  \n",
       "3  سرهنگ کسی ندارد برایش نامه بنویسد   2.0     3.0   0.0  \n",
       "4  سرهنگ کسی ندارد برایش نامه بنویسد   3.0     3.0   0.0  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#export\n",
    "def load_data(file_name):\n",
    "    \"\"\"\n",
    "        Read the CSV and creates a Panda dataframe fro the file content.\n",
    "    \"\"\"\n",
    "    f = open(file_name, 'r')\n",
    "    data = pd.read_csv(f, delimiter=',', encoding='utf-8')\n",
    "    return data\n",
    "\n",
    "\n",
    "file_name = os.path.join('data','taghche.csv')\n",
    "df = load_data(file_name)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>comment</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>اسم کتاب   No one writes to the Colonel\\nترجمش...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>طاقچه عزیز،نام کتاب\"کسی به سرهنگ نامه نمینویسد...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>بنظرم این اثر مارکز خیلی از صد سال تنهایی که ب...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>به نظر کتاب خوبی میومد اما من از ترجمش خوشم نی...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>کتاب خوبی است</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             comment label\n",
       "0  اسم کتاب   No one writes to the Colonel\\nترجمش...     0\n",
       "1  طاقچه عزیز،نام کتاب\"کسی به سرهنگ نامه نمینویسد...     1\n",
       "2  بنظرم این اثر مارکز خیلی از صد سال تنهایی که ب...     1\n",
       "3  به نظر کتاب خوبی میومد اما من از ترجمش خوشم نی...     0\n",
       "4                                      کتاب خوبی است     1"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remove the unnecessary columns\n",
    "df = df.drop(columns=['date', 'bookname', 'bookID', 'like'])\n",
    "# df = df.rename(columns={'Text':'text','Suggestion': 'label'})\n",
    "df.loc[(df.rate < 3), 'label'] = '0'\n",
    "df.loc[(df.rate >= 3), 'label'] = '1'\n",
    "df = df.drop(columns=['rate'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Postive examples: 55922  Negative examples: 13868\n"
     ]
    }
   ],
   "source": [
    "df_pos = df.loc[df.label == '1']\n",
    "df_neg = df.loc[df.label == '0']\n",
    "print('Postive examples: {}  Negative examples: {}'.format(len(df_pos),len(df_neg)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Balanced Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def create_small_dataset(df_pos, df_neg, n_samples):\n",
    "    \"\"\" Create a custom dataset of size `n_samples` from positive `df_pos` and negative `df-neg`\n",
    "        examples.\n",
    "    \"\"\"\n",
    "    duplicates = set()\n",
    "    counter = 0\n",
    "    data = {}\n",
    "    data['comment'] = []\n",
    "    data['polarity'] = []\n",
    "    while counter < n_samples:\n",
    "        index = np.random.randint(0, len(df_pos))\n",
    "        if index in duplicates:\n",
    "            continue\n",
    "        row = df_pos.iloc[index]\n",
    "        comment = remove_emoji(row['comment'])\n",
    "        label = row['label']\n",
    "        data['comment'].append(comment)\n",
    "        data['polarity'].append(label)\n",
    "        duplicates.add(index)\n",
    "        counter += 1\n",
    "    \n",
    "    duplicates.clear()\n",
    "    counter = 0\n",
    "    while counter < n_samples:\n",
    "        index = np.random.randint(0, len(df_neg))\n",
    "        if index in duplicates:\n",
    "            continue\n",
    "        row = df_neg.iloc[index]\n",
    "        comment = remove_emoji(row['comment'])\n",
    "        label = row['label']\n",
    "        data['comment'].append(comment)\n",
    "        data['polarity'].append(label)\n",
    "        duplicates.add(index)\n",
    "        counter += 1\n",
    "    return pd.DataFrame.from_dict(data)\n",
    "    \n",
    "def remove_emoji(text):\n",
    "    \"\"\" Remove a number of emojis from text.\"\"\"\n",
    "    emoji_pattern = re.compile(\"[\"\n",
    "        u\"\\U0001F600-\\U0001F64F\"  # emoticons\n",
    "        u\"\\U0001F300-\\U0001F5FF\"  # symbols & pictographs\n",
    "        u\"\\U0001F680-\\U0001F6FF\"  # transport & map symbols\n",
    "        u\"\\U0001F1E0-\\U0001F1FF\"  # flags (iOS)\n",
    "                           \"]+\", flags=re.UNICODE)\n",
    "    text = re.sub(emoji_pattern, ' ', text).replace('.','')\n",
    "    return re.sub(r'[a-z]+[A-Z]+', '', text, re.I)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>comment</th>\n",
       "      <th>polarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>اصلا خوب نبود</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>لطفا کتاب های بیشتری بزارید رشته حسابداری ممنو...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>با وجود اینکه ساده و روان نوشته شده و گوینده ه...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>بسیار عجیب و باورنکردنی</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>خوب نبود</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9995</th>\n",
       "      <td>خوبه</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9996</th>\n",
       "      <td>سلام\\nلطفا براش تخفیف بگزارید\\nبهش احتیاج داریم!</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9997</th>\n",
       "      <td>قیمتش حتی با تخفیف چهل درصد هم غیرمنطقیه برای ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td>افتضاح، بد، مزخرف</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td>فوق العادست</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                comment polarity\n",
       "0                                         اصلا خوب نبود        0\n",
       "1     لطفا کتاب های بیشتری بزارید رشته حسابداری ممنو...        0\n",
       "2     با وجود اینکه ساده و روان نوشته شده و گوینده ه...        0\n",
       "3                               بسیار عجیب و باورنکردنی        1\n",
       "4                                              خوب نبود        0\n",
       "...                                                 ...      ...\n",
       "9995                                               خوبه        1\n",
       "9996   سلام\\nلطفا براش تخفیف بگزارید\\nبهش احتیاج داریم!        1\n",
       "9997  قیمتش حتی با تخفیف چهل درصد هم غیرمنطقیه برای ...        0\n",
       "9998                                  افتضاح، بد، مزخرف        0\n",
       "9999                                        فوق العادست        1\n",
       "\n",
       "[10000 rows x 2 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset_size = 5000\n",
    "dataset = create_small_dataset(df_pos, df_neg, dataset_size)\n",
    "\n",
    "#shuffle the dataset\n",
    "dataset = dataset.sample(frac=1).reset_index(drop=True)\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train data size: 8000    test data size: 2000\n"
     ]
    }
   ],
   "source": [
    "train_data, test_data = train_test_split(dataset, test_size=0.2)\n",
    "print('train data size: {}    test data size: {}'.format(len(train_data), len(test_data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert the Data into Bert Specific Format\n",
    "\n",
    "We need to transform our data into a format BERT understands. This involves two steps. First, we create a list of `InputExample` objects using the constructor provided by Transformers library. Every `InputExample` must have the following structure:\n",
    "\n",
    "- `text_a` is the text we want to classify\n",
    "\n",
    "- `text_b` is used if we're training a model to understand the relationship between sentences (i.e. is `text_b` a translation of `text_a`)? Is `text_b` an answer to the question asked by `text_a`?). This doesn't apply to our task, so we can leave `text_b` blank.\n",
    "\n",
    "- `label` is the label of our example, i.e. True or False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def convert_data_into_input_example(data):\n",
    "    \"\"\" Covert the list of examples into a list of `InputExample` objects that is suitable\n",
    "        for BERT model.\"\"\"\n",
    "    input_examples = []\n",
    "    for i in range(len(data)):\n",
    "        example = InputExample(\n",
    "            guid= None,\n",
    "            text_a= data.iloc[i]['comment'],\n",
    "            text_b= None,\n",
    "            label= data.iloc[i]['polarity']\n",
    "        )\n",
    "        input_examples.append(example)\n",
    "    return input_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_input_examples = convert_data_into_input_example(train_data)\n",
    "val_input_examples = convert_data_into_input_example(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to preprocess our data (i.e. `InputExample`'s) so that it matches the data BERT was trained on. Therefore, we need to do a few things:\n",
    "\n",
    "1. Lowercase our text (if we're using a BERT lowercase model)\n",
    "\n",
    "2. Tokenize it (i.e. \"sally says hi\" -> [\"sally\", \"says\", \"hi\"])\n",
    "\n",
    "3. Break words into WordPieces (i.e. \"calling\" -> [\"call\", \"##ing\"])\n",
    "\n",
    "4. Map our words to indexes using a vocab file that BERT provides\n",
    "\n",
    "5. Add special \"CLS\" and \"SEP\" tokens (see the [readme](https://github.com/google-research/bert))\n",
    "\n",
    "6. Append \"index\" and \"segment\" tokens to each input (see the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization\n",
    "\n",
    "Deep learning models accept certain kinds of inputs, which is vectors of integers, each value representing a token. Each string of text must first be converted to a list of indices to be fed to the model. The tokenizer takes care of that for us. We also need to add special tokens to the list of ids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', 'like', '##d', 'that', 'book', 'very', 'much', '!']\n",
      "text ids: [146, 11850, 10162, 10189, 12748, 12558, 13172, 106]\n",
      "text ids with special tokens:  [101, 146, 11850, 10162, 10189, 12748, 12558, 13172, 106, 102]\n"
     ]
    }
   ],
   "source": [
    "tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased', do_lower_case=False)\n",
    "\n",
    "text = 'I liked that book very much!'\n",
    "tokenized_text = tokenizer.tokenize(text)\n",
    "print(tokenized_text)\n",
    "text_ids = tokenizer.convert_tokens_to_ids(tokenized_text)\n",
    "print('text ids:', text_ids)\n",
    "text_ids_with_special_tokens = tokenizer.build_inputs_with_special_tokens(text_ids)\n",
    "print('text ids with special tokens: ', text_ids_with_special_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Happily, there is a much simpler method that can do all the previous steps (i.e. tokenize, convert to indices and add special tokens) altogether. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoded text:  [101, 146, 11850, 10162, 10189, 12748, 12558, 13172, 106, 102]\n",
      "decoded text with special token:  [CLS] I liked that book very much! [SEP]\n",
      "decoded text without special token:  I liked that book very much!\n"
     ]
    }
   ],
   "source": [
    "MAX_SEQ_LENGTH = 128\n",
    "encoded_bert_text = tokenizer.encode(text, add_special_tokens=True, max_length=MAX_SEQ_LENGTH)\n",
    "# encoded_bert_text = tokenizer.encode(text, add_special_tokens=True, max_length=MAX_SEQ_LENGTH, return_tensors='tf')\n",
    "\n",
    "print('encoded text: ', encoded_bert_text)\n",
    "decoded_text_with_special_token = tokenizer.decode(encoded_bert_text)\n",
    "decoded_text_without_special_token = tokenizer.decode(encoded_bert_text, skip_special_tokens=True)\n",
    "\n",
    "print('decoded text with special token: ', decoded_text_with_special_token)\n",
    "print('decoded text without special token: ', decoded_text_without_special_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, we still require other addtional information to deal and manage. Thankfully, the Transformer library has a method to directly convert a dataset of InputExamples into features BERT understands. This method is called `glue_convert_examples_to_features`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example: {\n",
      " Input_ids: [101, 66847, 11892, 41002, 14495, 99500, 29869, 37951, 20208, 60230, 774, 92289, 11892, 10498, 763, 101420, 39387, 10700, 17197, 11626, 55532, 26973, 10700, 29869, 76528, 10700, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      " attention_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      " token_type_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      " label: 1\n",
      "}\n",
      "Example: {\n",
      " Input_ids: [101, 40160, 10429, 19015, 12455, 13141, 54731, 788, 71222, 54862, 80493, 10278, 26973, 10388, 756, 770, 11086, 10327, 53880, 54862, 10278, 756, 10289, 23155, 54862, 808, 17821, 19015, 55532, 29869, 86598, 10700, 10498, 12084, 10582, 10700, 789, 22929, 86598, 10700, 10498, 10641, 41883, 17821, 10327, 10700, 773, 36990, 26973, 10388, 12218, 24076, 756, 29869, 11626, 10388, 15511, 31688, 10582, 67125, 823, 24076, 756, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      " attention_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      " token_type_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      " label: 0\n",
      "}\n",
      "Example: {\n",
      " Input_ids: [101, 11800, 34353, 10700, 772, 22900, 29413, 763, 73478, 68625, 22955, 789, 53001, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      " attention_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      " token_type_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      " label: 0\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "label_list = ['0', '1']\n",
    "\n",
    "bert_train_dataset = glue_convert_examples_to_features(examples=train_input_examples, tokenizer=tokenizer, max_length=MAX_SEQ_LENGTH, task='mrpc', label_list=label_list)\n",
    "bert_val_dataset = glue_convert_examples_to_features(examples=val_input_examples, tokenizer=tokenizer, max_length=MAX_SEQ_LENGTH, task='mrpc', label_list=label_list)\n",
    "\n",
    "for i in range(3):\n",
    "#     print('Example: {}'.format(bert_train_dataset[i]))\n",
    "    print('Example: {')\n",
    "    print(' Input_ids: {}'.format(bert_train_dataset[i].input_ids))\n",
    "    print(' attention_mask: {}'.format(bert_train_dataset[i].attention_mask))\n",
    "    print(' token_type_ids: {}'.format(bert_train_dataset[i].token_type_ids))\n",
    "    print(' label: {}'.format(bert_train_dataset[i].label))\n",
    "    print('}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "عالی بودحتما پیشنهاد میکنمانقدر زیبا بود که احساس میکردم فیلمش رو دارم میبینم\n"
     ]
    }
   ],
   "source": [
    "# Let's take a look at one example from the dataset\n",
    "ex = bert_train_dataset[0]\n",
    "in_ids = ex.input_ids\n",
    "decoded_sentence = tokenizer.decode(in_ids, skip_special_tokens=True)\n",
    "print(decoded_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the Hyperparameters\n",
    "\n",
    "Before fine-tuning the model, we must define a few hyperparameters that will be used during the training such as the optimizer, the loss and the evaluation metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = TFBertForSequenceClassification.from_pretrained('bert-base-multilingual-cased')\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=2e-5, epsilon=1e-08, clipnorm=1.0)\n",
    "loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
    "metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')\n",
    "model.compile(optimizer=optimizer, loss=loss, metrics=[metric])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Note: For some reason, when passing the `bert_train_dataset`, (which is supposed to work), to the `model.fit()` did NOT work, so I had to workaround it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This way did NOT work, so I had to workaround it.\n",
    "\n",
    "model.fit(bert_train_dataset, validation_data=bert_val_dataset, epochs=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workaround\n",
    "\n",
    "I used the `bert_train_dataset` and created a list for each feature (i.e. `input_ids`, `attention_mask`, `token_type_ids` and `label`) and passed them as the arguments of the model. Please see transformers' [documentation](https://huggingface.co/transformers/model_doc/bert.html) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def my_solution(bdset):\n",
    "    \"\"\" Create a list of input tensors required to be in the first argument of the \n",
    "        model call function for training. e.g. `model([input_ids, attention_mask, token_type_ids])`.\n",
    "    \"\"\"\n",
    "    input_ids, attention_mask, token_type_ids, label = [], [], [], []\n",
    "    for in_ex in bdset:\n",
    "        input_ids.append(in_ex.input_ids)\n",
    "        attention_mask.append(in_ex.attention_mask)\n",
    "        token_type_ids.append(in_ex.token_type_ids)\n",
    "        label.append(in_ex.label)\n",
    "\n",
    "    input_ids = np.vstack(input_ids)\n",
    "    attention_mask = np.vstack(attention_mask)\n",
    "    token_type_ids = np.vstack(token_type_ids)\n",
    "    label = np.vstack(label)\n",
    "    return ([input_ids, attention_mask, token_type_ids], label)\n",
    "\n",
    "def example_to_features(input_ids, attention_masks, token_type_ids, y):\n",
    "    \"\"\" Convert a training example into the Bert compatible format.\"\"\"\n",
    "    return {\"input_ids\": input_ids,\n",
    "            \"attention_mask\": attention_masks,\n",
    "            \"token_type_ids\": token_type_ids},y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (8000, 128)\n",
      "x_val shape: (2000, 128)\n",
      "Format of model input examples: <TakeDataset shapes: ({input_ids: (None, 128), attention_mask: (None, 128), token_type_ids: (None, 128)}, (None, 1)), types: ({input_ids: tf.int64, attention_mask: tf.int64, token_type_ids: tf.int64}, tf.int64)> \n"
     ]
    }
   ],
   "source": [
    "x_train, y_train = my_solution(bert_train_dataset)\n",
    "x_val, y_val = my_solution(bert_val_dataset)\n",
    "\n",
    "print('x_train shape: {}'.format(x_train[0].shape))\n",
    "print('x_val shape: {}'.format(x_val[0].shape))\n",
    "\n",
    "train_ds = tf.data.Dataset.from_tensor_slices((x_train[0], x_train[1], x_train[2], y_train)).map(example_to_features).shuffle(100).batch(32)\n",
    "val_ds   = tf.data.Dataset.from_tensor_slices((x_val[0], x_val[1], x_val[2], y_val)).map(example_to_features).batch(64)\n",
    "\n",
    "print('Format of model input examples: {} '.format(train_ds.take(1)))\n",
    "\n",
    "EPOCHS = 5\n",
    "\n",
    "history = model.fit(train_ds, validation_data=val_ds, epochs=EPOCHS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Details\n",
    "\n",
    "```\n",
    "Epoch 1/5\n",
    "250/250 [==============================] - 137s 548ms/step - loss: 0.5838 - accuracy: 0.6894 - val_loss: 0.5079 - val_accuracy: 0.7665\n",
    "Epoch 2/5\n",
    "250/250 [==============================] - 134s 535ms/step - loss: 0.4867 - accuracy: 0.7747 - val_loss: 0.5002 - val_accuracy: 0.7650\n",
    "Epoch 3/5\n",
    "250/250 [==============================] - 134s 534ms/step - loss: 0.4131 - accuracy: 0.8227 - val_loss: 0.5112 - val_accuracy: 0.7685\n",
    "Epoch 4/5\n",
    "250/250 [==============================] - 134s 534ms/step - loss: 0.3515 - accuracy: 0.8558 - val_loss: 0.5692 - val_accuracy: 0.7630\n",
    "Epoch 5/5\n",
    "250/250 [==============================] - 134s 535ms/step - loss: 0.2901 - accuracy: 0.8871 - val_loss: 0.6147 - val_accuracy: 0.7575\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def plot_history(history):\n",
    "    acc = history.history['accuracy']\n",
    "    val_acc = history.history['val_accuracy']\n",
    "    loss = history.history['loss']\n",
    "    val_loss = history.history['val_loss']\n",
    "    x = range(1, len(acc) + 1)\n",
    "\n",
    "    plt.figure(figsize=(12, 5))\n",
    "    plt.subplot(1, 2, 1)\n",
    "    plt.plot(x, acc, 'b', label='Training acc')\n",
    "    plt.plot(x, val_acc, 'r', label='Validation acc')\n",
    "    plt.title('Training and validation accuracy')\n",
    "    plt.legend()\n",
    "    plt.subplot(1, 2, 2)\n",
    "    plt.plot(x, loss, 'b', label='Training loss')\n",
    "    plt.plot(x, val_loss, 'r', label='Validation loss')\n",
    "    plt.title('Training and validation loss')\n",
    "    plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the Loss and Accuracy\n",
    "\n",
    "We can see the training and validation loss as well as accuracy in the plots below. Please note that after epoch 3 `validation loss` starts to grow, which means the model begins to overfit. Therefore 3 epochs would be enough and work better since it has better accuracy of `0.7685`.\n",
    "\n",
    "![](images/bert_5_epochs_lr2-e5_sparse.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(val_ds)\n",
    "print(predictions[0].shape)\n",
    "print()\n",
    "predictions_classes = np.argmax(predictions[0], axis = 1)\n",
    "for i in range(10):\n",
    "    print('comment: {}\\n, actual label: {}, predicted label: {}'.format(test_data.iloc[i]['comment'], val_input_examples[i].label, predictions_classes[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "comment: کسالت آور بود \n",
    "ولی چون یک داستان نسباتا واقعیست ، به خاطر حقایقی که در اون وجود داشت بخوبی ما رو با فرهنگ و اعتقادات مردم اون دوره انگلیس آشنا میکنه\n",
    "داستان کشتی تایتانیک\n",
    ", actual label: 0, predicted label: 1\n",
    "comment: شیوایی کلام میتوانست بهتر باشد\n",
    ", actual label: 0, predicted label: 0\n",
    "comment: باسلام،برای قشرخاصی طراحی شده کتاب ودرکش برای عموم کمی سخت وگیج کننده است\n",
    ", actual label: 0, predicted label: 1\n",
    "comment: بد نبود- راضی کننده بود\n",
    ", actual label: 1, predicted label: 0\n",
    "comment: من به عنوان یه نویسنده این مجله تقاضا میکنم همه ازش حمایت کنند! این یه کار نمونه ای علمی ترویجی نو تو زمینه زبان کوردی هست و نیاز به حمایت همه داره\n",
    ", actual label: 1, predicted label: 0\n",
    "comment: دوستشدارمدوستش خواهی داشت✌\n",
    ", actual label: 1, predicted label: 0\n",
    "comment: خیلی قشنگ بود  \n",
    "مرسی طاقچه جون\n",
    ", actual label: 0, predicted label: 1\n",
    "comment: همه کتاب داره درباره جزییات ظاهر بقیه و محیط صحبت میکنهچنگی ب دل نمیزنهبسیار خسته کنندس\n",
    ", actual label: 0, predicted label: 0\n",
    "comment: کتاب رو خوندم\n",
    "به نظرم کتاب مفیدی میتونه باشه\n",
    "فقط طیق معمول کتب روانشناسی داستان زیاد داره\n",
    "خودم فقط قسمتهای غیر داستانی رو خوندم اگر ابهام ایجاد میشد داستانش رو هم میخوندم\n",
    "کلا خوبه\n",
    "حداقل متوجه میشید زبان عشق خودتون چیه و ب طرف مقابلتون میتوتید بگید با من اینطوری باش \n",
    ", actual label: 1, predicted label: 1\n",
    "comment: کتاب با بررسی وقایع کربلا و تطبیق آنها با زندگی امروز به ما هشدار می‌دهد تا به سمت همان فرهنگی که فرزند رسول خدا را به شهادت رساند حرکت نکنیم و این کار را با دلایل منطقی و بسیار هوشمندانه انجام داده است\n",
    "پیشنهاد میکنم این کتاب رو حتماً بخونید چون جواب بسیاری از سوالاتتون رو خواهید گرفت\n",
    ", actual label: 1, predicted label: 1\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def example_to_features_predict(input_ids, attention_masks, token_type_ids):\n",
    "    \"\"\"\n",
    "        Convert the test examples into Bert compatible format.\n",
    "    \"\"\"\n",
    "    return {\"input_ids\": input_ids,\n",
    "            \"attention_mask\": attention_masks,\n",
    "            \"token_type_ids\": token_type_ids}\n",
    "\n",
    "\n",
    "def get_prediction(in_sentences):\n",
    "    \"\"\"\n",
    "        Prepare the test comments and return the predictions.\n",
    "    \"\"\"\n",
    "    labels = [\"0\", \"1\"]\n",
    "    input_examples = [InputExample(guid=\"\", text_a = x, text_b = None, label = '0') for x in in_sentences] # here, \"\" is just a dummy label\n",
    "    predict_input_fn = glue_convert_examples_to_features(examples=input_examples, tokenizer=tokenizer, max_length=MAX_SEQ_LENGTH, task='mrpc', label_list=label_list)\n",
    "    x_test_input, y_test_input = my_solution(predict_input_fn)\n",
    "    test_ds   = tf.data.Dataset.from_tensor_slices((x_test_input[0], x_test_input[1], x_test_input[2])).map(example_to_features_predict).batch(32)\n",
    "\n",
    "    predictions = model.predict(test_ds)\n",
    "    #   print('predictions:', predictions[0].shape)\n",
    "    predictions_classes = np.argmax(predictions[0], axis = 1)\n",
    "    return [(sentence, prediction) for sentence, prediction in zip(in_sentences, predictions_classes)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pred_sentences = ['به نظر کتاب خوبی نمی ومد', ' رایگان بودنش فوق العادش میکند']\n",
    "pred_sentences = [comment for comment,l in test_data.sample(20).values]\n",
    "predictions = get_prediction(pred_sentences)\n",
    "for p in predictions:\n",
    "    print(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "('واقعا چطور همچین کتابایی اجازه نشر پیدا میکنن؟', 0)\n",
    "('روش چی ؟', 0)\n",
    "('بنظرم کتاب زندگینامه شهید رو رایگان کردن صورت خوشی نداره! چون اونی که براش مهمه بخونه پولشو میده میخره کسی هم که سلیقه مطالعه کتابش این مدلی نیست، رایگانم بشه نمیخونه', 0)\n",
    "('سوالات کجا میاد کجا باید جواب بدیم', 0)\n",
    "('سمنوپزان ، مسلول و  زن زیادی  خوب بودند ؛ در کل ادم رو به گذشته های دور و شیوه زندگی اونها میبره و از این جهت فوق العاده است\\nممنون از طاقچه بابت برنامه خوبش و به خاطر کتابهایی که رایگان کردید', 1)\n",
    "('کتاب فوق العاده ای بود به خیلی از شبهات در غالب داستان جواب داده  حتما پیشنهاد میکنم', 1)\n",
    "('چقدر قشنگ بالا و پایین زندگی رو نشون داده بود  ', 0)\n",
    "('ترجمه اش اصلا خوب نیست', 0)\n",
    "('واقعا این کتاب عالییییییییییییییییییییییییییییییییییییییییییییییییه یه حرف نگفته اس که غمباد شده تو دلمون واقعا دست مریزاد سرکاره خانوم عالیشاهی با آرزوی بهترین ها و بالاترین درجات برای شما نویسنده محترمه', 1)\n",
    "('واقعاً قیمت ها رو بر چه اساس تعیین میکنید نمیدونم قیمت نسخه چاپی ۵۰ تومان هست که با توجه به قیمت کاغذ منطقی هست اما نسخه الکترونیک برای یک کتاب نباید انقدر گران باشد بنظرم طاقچه باید محدودیت های مشخص شده قرار بده برای قیمت', 0)\n",
    "('واقعا صداش خوب نیست چون انگار داره برا بچه ها قصه میگه همچین متن هایی رو باید یه نفر با صدای قوی ومحکم وبا جذبه قرائت کنه ', 0)\n",
    "('خیلی داستان جذابی نداشت و در آخر داستان هم\\u200c اسلام هراسی بیداد میکرد', 0)\n",
    "('سلام من این کتاب را قبلا خریداری کردم نمی دانم چرا الان فایل کتاب را باز نمی کند ؟', 0)\n",
    "('آیا حرف دلواپسان درست نبود؟', 0)\n",
    "('عالی', 1)\n",
    "('کتاب سطح پایین با نثری درهم پیچیده البته ترجمه نامناسب', 0)\n",
    "('قشنگ بود', 1)\n",
    "('ترجمه خیلی خیلی خیلی بد و غیرقابل فهم', 0)\n",
    "('یکی از بهترین و ساده ترین کتاب ها برای کسایی که دنبال راه حلی هستند تا در زندگی مالی موفق شوند', 1)\n",
    "('خیلی عالی بود لذت بردم امیدوارم شما هم فراموش نکنید اولین نفری بودم که به ازادیش فکر کردم به درخت، به ماهی، به دلیران تنگسیر، به پایانی خوش', 1)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Evaluating the BERT model\")\n",
    "model.evaluate(val_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Evaluating the BERT model\n",
    "32/32 [==============================] - 9s 293ms/step - loss: 0.6147 - accuracy: 0.7575\n",
    "[0.6146795749664307, 0.7574999928474426]\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}