Lab1_NNLM-ZH-Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 將 Google-NNLM-ZH 以 Keras Embedding Layer 的形式讀取\n",
    "+ 從[這個網頁](https://tfhub.dev/google/nnlm-zh-dim128-with-normalization/2)下載模型檔案\n",
    "+ 使用 `tensorflow_hub` 套件讀取存在本地端的模型\n",
    "+ 該資料夾要包含 `.pb` 檔\n",
    "+ 檔案資料夾結構：\n",
    "    ```\n",
    "    .\n",
    "    ├── NNLM-ZH\n",
    "    │   ├── assets\n",
    "    │   │   └── tokens.txt\n",
    "    │   ├── saved_model.pb\n",
    "    │   └── variables\n",
    "    │       ├── variables.data-00000-of-00001\n",
    "    │       └── variables.index\n",
    "    └── NNLM-ZH-Demo.ipynb\n",
    "    ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow_hub as hub\n",
    "\n",
    "emb_layer = hub.KerasLayer('./NNLM-ZH')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 讀取純文字字典檔\n",
    "+ 新鮮的 `./assets/tokens.txt` 裡面基本上都是簡體中文\n",
    "+ 此例字典檔內有 968,075 個詞"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total tokens: 968075\n"
     ]
    }
   ],
   "source": [
    "tokens = open('./NNLM-ZH/assets/tokens.txt', 'r', encoding='UTF-8').read().strip().split('\\n')\n",
    "print(f'Total tokens: {len(tokens)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 探索 Embedding Layer\n",
    "+ 檢視 `emb_layer.get_weights()` 可以發現只有一組權重\n",
    "+ 檢視 `weights[0].shape` 發現大小為 971177 x 128\n",
    "  + **!! 大小與字典檔並不一致 !!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of weights: 1\n",
      "Shape of the weights: (971177, 128)\n"
     ]
    }
   ],
   "source": [
    "weights = emb_layer.get_weights()\n",
    "print(f'Number of weights: {len(weights)}')\n",
    "print(f'Shape of the weights: {weights[0].shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 基本使用方法\n",
    "+ 在 TF2.0 裡面可以直接使用 Layer 呼叫來獲得 Text Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([1, 128])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = emb_layer(['今天天氣真好'])\n",
    "result.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 測試單詞\n",
    "1. 使用 Layer Call\n",
    "2. 使用 Token Index 定位 Layer Weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token = '猫'\n",
    "\n",
    "# Layer Call\n",
    "t1 = emb_layer([token])\n",
    "\n",
    "# Token Index\n",
    "idx = tokens.index(token)\n",
    "t2 = emb_layer.get_weights()[0][idx]\n",
    "\n",
    "# Compare\n",
    "(t1 == t2).numpy().all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 大小寫敏感測試\n",
    "+ Runa 有出現在字典裡\n",
    "+ 結果顯示是 Case-Sensitive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t1 = emb_layer(['runa'])\n",
    "idx = tokens.index('Runa')\n",
    "t2 = emb_layer.get_weights()[0][idx]\n",
    "(t1 == t2).numpy().all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OOV 測試\n",
    "+ runa 與貓並未出現在字典檔內"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t1 = emb_layer(['runa'])\n",
    "idx = tokens.index('<UNK>')\n",
    "t2 = emb_layer.get_weights()[0][idx]\n",
    "(t1 == t2).numpy().all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t1 = emb_layer(['貓'])\n",
    "idx = tokens.index('<UNK>')\n",
    "t2 = emb_layer.get_weights()[0][idx]\n",
    "(t1 == t2).numpy().all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 從字典檔內隨機挑選字詞測試\n",
    "+ 隨機檢視的結果發現有些 Tokens 裡面有 `##` 這種與 BERT Tokenizer 相似的處理符號\n",
    "+ 甚至可能比 BERT Tokenizer 更複雜"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Token: 蔡学胜\n",
      "Token ID: 335996\n",
      "Pass Test: True\n",
      "\n",
      "Test Token: PiPO\n",
      "Token ID: 378640\n",
      "Pass Test: True\n",
      "\n",
      "Test Token: 陆建德\n",
      "Token ID: 243462\n",
      "Pass Test: True\n",
      "\n",
      "Test Token: 煤耗\n",
      "Token ID: 53311\n",
      "Pass Test: True\n",
      "\n",
      "Test Token: 雷长林\n",
      "Token ID: 327671\n",
      "Pass Test: True\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "for i in range(5):\n",
    "    token = random.choice(tokens)\n",
    "    print(f'Test Token: {token}')\n",
    "    token_id = tokens.index(token)\n",
    "    print(f'Token ID: {token_id}')\n",
    "\n",
    "    t1 = emb_layer([token])\n",
    "    t2 = emb_layer.get_weights()[0][token_id]\n",
    "    print(f'Pass Test: {(t1 == t2).numpy().all()}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Token: ###.##a/b/g/n\n",
      "Token ID: 99382\n",
      "Pass Test: False\n",
      "\n"
     ]
    }
   ],
   "source": [
    "token = '###.##a/b/g/n'\n",
    "print(f'Test Token: {token}')\n",
    "token_id = tokens.index(token)\n",
    "print(f'Token ID: {token_id}')\n",
    "\n",
    "t1 = emb_layer([token])\n",
    "t2 = emb_layer.get_weights()[0][token_id]\n",
    "print(f'Pass Test: {(t1 == t2).numpy().all()}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentence Embedding Re-implement\n",
    "+ 參考 TF-Hub 的介紹頁面使用 `tf.nn.embedding_lookup_sparse` 與 sqrtn combiner 重現 Google NNLM 的 Sentence Embedding 製作方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "emb_weights = emb_layer.get_weights()[0]\n",
    "\n",
    "def to_ids(sent):\n",
    "    segs = sent.split(' ')\n",
    "    ids = [list(map(tokens.index, segs))]\n",
    "    return ids\n",
    "\n",
    "def to_sparse(sent):\n",
    "    ids = to_ids(sent)\n",
    "    ids = tf.convert_to_tensor(ids)\n",
    "    return tf.sparse.from_dense(ids)\n",
    "\n",
    "def op_weights(sent):    \n",
    "    return tf.nn.embedding_lookup_sparse(emb_weights, to_sparse(sent), None, combiner='sqrtn')\n",
    "\n",
    "def op_hand(sent):\n",
    "    import numpy as np\n",
    "    ids = to_ids(sent)\n",
    "    arr = np.array([emb_weights[i] for i in ids]).sum(axis=1)\n",
    "    arr /= ((len(ids) + 2) ** 0.5) # 此處尚不知為何要 +2\n",
    "\n",
    "    return arr\n",
    "\n",
    "def equal(t1, t2):\n",
    "    return (t1 == t2).numpy().all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent = '猫 和 狗'\n",
    "\n",
    "t1 = emb_layer([sent])\n",
    "t2 = op_weights(sent)\n",
    "t3 = op_hand(sent)\n",
    "\n",
    "equal(t1, t2), equal(t2, t3), equal(t1, t3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 結論\n",
    "+ 九成的 Google NNLM 操作已經被成功解析\n",
    "+ 證實了字典檔與模型權重之間確實有關聯\n",
    "+ 尚無法理解 UNK Token 的行為"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t1 = emb_layer(['貓'])\n",
    "\n",
    "emb_weights = emb_layer.get_weights()[0]\n",
    "ids = to_sparse('<UNK>')\n",
    "t2 = tf.nn.embedding_lookup_sparse(emb_weights, ids, None, combiner='sqrtn')\n",
    "(t1 == t2).numpy().all()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}