language-identification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Language Identification\n",
    "\n",
    "[Language identification](https://en.wikipedia.org/wiki/Language_identification) is one of the most common feature of every Social Network or Web application, this is commonly paired with [Machine Translation](https://en.wikipedia.org/wiki/Machine_translation) to improve the user experience and content accesibility(a must have in the 2.0 society). The goal of the task  is to detect the natural language of a given piece of text. *What can you use it for?* This is a foundation for other features such as Machine Translation (as mentioned before) and post/tweets/articles and documents analysis.\n",
    "\n",
    "### Language identification of short pieces of text from Wikipedia\n",
    "\n",
    "In this notebook we will build a deep learning model able to [detect the languages from short piceces of text (140 characters, old Tweets lenght) with high accuracy using neural networks](http://machinelearningexp.com/deep-learning-language-identification-using-keras-tensorflow/). The task is commonly solved using hard-coded rules or NLP library, but we will attack the problem using Deep Learning. \n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/floydhub/language-identification-template/master/images/fhchat.png\" width=\"500\" height=\"500\" align=\"center\"/>\n",
    "\n",
    "*Made with [Sketch Group Chat](https://www.sketchappsources.com/free-source/1558-group-chat-sketch-freebie-resource.html)*\n",
    "\n",
    "We have [already gathered and extract the raw dataset](https://floydhub.com/floydhub/datasets/language-identification/1) from https://dumps.wikimedia.org for 7 languages: *Italian*, *Spanish* and *French* which are considered to be in Latin language group, *English* and *German* have also common roots. *Czech* and *Slovakian* are extremely similar and are considered to be one of major challenged in the language recognition.\n",
    "\n",
    "iso-code | language | example\n",
    "---------|----------|--------\n",
    "en | English | Hello world!\n",
    "fr | French | Bonjour tout le monde!\n",
    "es | Spanish | Hola mundo!\n",
    "it | Italian | Ciao mondo!\n",
    "de | German | Hallo welt!\n",
    "cz | Czech | Ahoj světe!\n",
    "sk | Slovakian | Dobrý deň svet!\n",
    "\n",
    "We will:\n",
    "\n",
    "- Preprocess text data for NLP\n",
    "- Build and train Deep Neural Network using Keras and Tensorflow\n",
    "- Evaluate our model on the test set\n",
    "- Run the model on your own text!\n",
    "\n",
    "\n",
    "### Instructions\n",
    "- To execute a code cell, click on the cell and press `Shift + Enter` (shortcut for Run).\n",
    "- To learn more about Workspaces, check out the [Getting Started Notebook](get_started_workspace.ipynb).\n",
    "- **Tip**: *Feel free to try this Notebook with your own data and on your own super awesome regression task.*\n",
    "\n",
    "Now, let's get started! 🚀"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial Setup\n",
    "Let's start by importing some packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import random\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import time\n",
    "\n",
    "from sklearn import preprocessing\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import Dropout\n",
    "import keras.optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Parameters\n",
    "\n",
    "We'll set the hyperparameters for training our model. If you understand what they mean, feel free to play around - otherwise, we recommend keeping the defaults for your first run 🙂"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperparams if GPU is available\n",
    "if tf.test.is_gpu_available():\n",
    "    # GPU\n",
    "    BATCH_SIZE = 512  # Number of images used in each iteration\n",
    "    EPOCHS = 12  # Number of passes through entire dataset\n",
    "    \n",
    "# Hyperparams for CPU training\n",
    "else:\n",
    "    # CPU\n",
    "    BATCH_SIZE = 64\n",
    "    EPOCHS = 12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation\n",
    "\n",
    "*WARNING*\n",
    "\n",
    "Make sure that the dataset has been mounted before running the next Code Cells. The data mounting should take about 3 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ALPHABET:\n",
      "abcdefghijklmnopqrstuvwxyzßàáâäæçèéêìíîïñòóôöùúûüýÿčďěĺľňœŕřšťůž !?¿¡ABCDEFGHIJKLMNOPQRSTUVWXYZÀÁÂÄÆÇÈÉÊÌÍÎÏÑÒÓÔÖÙÚÛÜÝČĎĚĹĽŇŒŔŘŠŤŮŸŽ\n",
      "ALPHABET LEN(VOCAB SIZE): 132\n"
     ]
    }
   ],
   "source": [
    "#################\n",
    "# Configuration #\n",
    "#################\n",
    "\n",
    "# dictionary of languages that our classifier will cover\n",
    "LANGUAGES_DICT = {'en':0,'fr':1,'es':2,'it':3,'de':4,'sk':5,'cs':6}\n",
    "\n",
    "# Length of cleaned text used for training and prediction - 140 chars\n",
    "MAX_LEN = 140\n",
    "\n",
    "# number of language samples per language that we will extract from source files\n",
    "NUM_SAMPLES = 250000\n",
    "\n",
    "# For reproducibility\n",
    "SEED = 42\n",
    "\n",
    "from support import define_alphabet\n",
    "# Load the Alphabet\n",
    "alphabet = define_alphabet()\n",
    "print('ALPHABET:')\n",
    "print(alphabet[2])\n",
    "\n",
    "VOCAB_SIZE = len(alphabet[2])\n",
    "print('ALPHABET LEN(VOCAB SIZE):', VOCAB_SIZE)\n",
    "\n",
    "# Folders from where load / store the raw, source, cleaned, samples and train_test data\n",
    "data_directory = \"/floyd/input/languageidentification/data\"\n",
    "source_directory = os.path.join(data_directory, 'source')\n",
    "cleaned_directory = os.path.join(data_directory, 'cleaned')\n",
    "samples_directory = os.path.join('/tmp', 'samples')\n",
    "train_test_directory = os.path.join('/tmp', 'train_test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before feeding the data into the model, we have to preprocess the text.\n",
    "\n",
    "We will use the characters frequency as features to our model. This representation is similar to the [Bag of Words](https://en.wikipedia.org/wiki/Bag-of-words_model) model, with the exception that we are using characters and not words for defining the Vocabulary. You can see an example below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. SAMPLE TEXT: \n",
      " die Fähre \"Ibn Batouta\", und den Mondkrater Ibn Battuta. Liste der Gemeinden in Österreich Dies ist eine Zusammenstellung von Listen der\n",
      "\n",
      "2. REFERENCE ALPHABET: \n",
      " ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'ß', 'à', 'á', 'â', 'ä', 'æ', 'ç', 'è', 'é', 'ê', 'ì', 'í', 'î', 'ï', 'ñ', 'ò', 'ó', 'ô', 'ö', 'ù', 'ú', 'û', 'ü', 'ý', 'ÿ', 'č', 'ď', 'ě', 'ĺ', 'ľ', 'ň', 'œ', 'ŕ', 'ř', 'š', 'ť', 'ů', 'ž', ' ', '!', '?', '¿', '¡', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'À', 'Á', 'Â', 'Ä', 'Æ', 'Ç', 'È', 'É', 'Ê', 'Ì', 'Í', 'Î', 'Ï', 'Ñ', 'Ò', 'Ó', 'Ô', 'Ö', 'Ù', 'Ú', 'Û', 'Ü', 'Ý', 'Č', 'Ď', 'Ě', 'Ĺ', 'Ľ', 'Ň', 'Œ', 'Ŕ', 'Ř', 'Š', 'Ť', 'Ů', 'Ÿ', 'Ž']\n",
      "\n",
      "3. SAMPLE INPUT ROW: \n",
      " [6, 4, 1, 8, 18, 1, 2, 2, 11, 0, 1, 4, 4, 13, 3, 0, 0, 7, 7, 11, 5, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0, 0, 2, 0, 1, 0, 1, 1, 0, 2, 0, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "\n",
      "4. INPUT SIZE (VOCAB SIZE):  132\n"
     ]
    }
   ],
   "source": [
    "from support import get_sample_text, get_input_row\n",
    "    \n",
    "# let's see if our processing is returning counts\n",
    "# last part calculates also input_size for DNN so this code must be run before DNN is trained\n",
    "path = os.path.join(cleaned_directory, \"de_cleaned.txt\")\n",
    "with open(path, 'r') as f:\n",
    "    content = f.read()\n",
    "    random_index = random.randrange(0,len(content)-2*MAX_LEN)\n",
    "    sample_text = get_sample_text(content,random_index,MAX_LEN)\n",
    "    print (\"1. SAMPLE TEXT: \\n\", sample_text)\n",
    "    print (\"\\n2. REFERENCE ALPHABET: \\n\", alphabet[0]+alphabet[1])\n",
    "    \n",
    "    sample_input_row = get_input_row(content, random_index, MAX_LEN, alphabet)\n",
    "    print (\"\\n3. SAMPLE INPUT ROW: \\n\",sample_input_row)\n",
    "    \n",
    "    input_size = len(sample_input_row)\n",
    "    if input_size != VOCAB_SIZE:\n",
    "        print(\"Something strange happened!\")\n",
    "        \n",
    "    print (\"\\n4. INPUT SIZE (VOCAB SIZE): \", input_size)\n",
    "    del content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will apply the transformation from raw text to Bag of Characters representation for all the data we have collected. At the end of the proprocessing, we will have 250k samples per language where every sample will be piece of text 140 characters long, represented using the Bag of Characters model.\n",
    "\n",
    "Dataset dimension (1750k, 133):\n",
    "- rows: 1750k (250k * 7) or (NUM_SAMPLES * num_languages)\n",
    "- columns: 133 (132 + 1) or (VOCAB_SIZE + language_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing file : /floyd/input/languageidentification/data/cleaned/en_cleaned.txt\n",
      "File size :  101.42 MB  | # possible samples :  768340 | # skip chars : 199\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Processing file : /floyd/input/languageidentification/data/cleaned/fr_cleaned.txt\n",
      "File size :  98.72 MB  | # possible samples :  747899 | # skip chars : 191\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Processing file : /floyd/input/languageidentification/data/cleaned/es_cleaned.txt\n",
      "File size :  97.56 MB  | # possible samples :  739111 | # skip chars : 187\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Processing file : /floyd/input/languageidentification/data/cleaned/it_cleaned.txt\n",
      "File size :  101.89 MB  | # possible samples :  771891 | # skip chars : 200\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Processing file : /floyd/input/languageidentification/data/cleaned/de_cleaned.txt\n",
      "File size :  101.22 MB  | # possible samples :  766797 | # skip chars : 198\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Processing file : /floyd/input/languageidentification/data/cleaned/sk_cleaned.txt\n",
      "File size :  85.65 MB  | # possible samples :  648847 | # skip chars : 151\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Processing file : /floyd/input/languageidentification/data/cleaned/cs_cleaned.txt\n",
      "File size :  90.56 MB  | # possible samples :  686098 | # skip chars : 166\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Vocab Size :  132\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Samples array size :  (1750000, 133)\n",
      "/tmp/samples/lang_samples_132.npz size :  57.21 MB\n"
     ]
    }
   ],
   "source": [
    "# Utility function to return file Bytes size in MB\n",
    "def size_mb(size):\n",
    "    size_mb =  '{:.2f}'.format(size/(1000*1000.0))\n",
    "    return size_mb + \" MB\"\n",
    "\n",
    "# Now we have preprocessing utility functions ready. Let's use them to process each cleaned language file\n",
    "# and turn text data into numerical data samples for our neural network\n",
    "# prepare numpy array\n",
    "sample_data = np.empty((NUM_SAMPLES*len(LANGUAGES_DICT),input_size+1),dtype = np.uint16)\n",
    "lang_seq = 0 # offset for each language data\n",
    "jump_reduce = 0.2 # part of characters removed from jump to avoid passing the end of file\n",
    "\n",
    "for lang_code in LANGUAGES_DICT:\n",
    "    start_index = 0\n",
    "    path = os.path.join(cleaned_directory, lang_code+\"_cleaned.txt\")\n",
    "    with open(path, 'r') as f:\n",
    "        print (\"Processing file : \" + path)\n",
    "        file_content = f.read()\n",
    "        content_length = len(file_content)\n",
    "        remaining = content_length - MAX_LEN*NUM_SAMPLES\n",
    "        jump = int(((remaining/NUM_SAMPLES)*3)/4)\n",
    "        print (\"File size : \",size_mb(content_length),\\\n",
    "               \" | # possible samples : \",int(content_length/VOCAB_SIZE),\\\n",
    "              \"| # skip chars : \" + str(jump))\n",
    "        for idx in range(NUM_SAMPLES):\n",
    "            input_row = get_input_row(file_content, start_index, MAX_LEN, alphabet)\n",
    "            sample_data[NUM_SAMPLES*lang_seq+idx,] = input_row + [LANGUAGES_DICT[lang_code]]\n",
    "            start_index += MAX_LEN + jump\n",
    "        del file_content\n",
    "    lang_seq += 1\n",
    "    print (100*\"-\")\n",
    "     \n",
    "# Let's randomy shuffle the data\n",
    "np.random.shuffle(sample_data)\n",
    "# reference input size\n",
    "print (\"Vocab Size : \",VOCAB_SIZE )\n",
    "print (100*\"-\")\n",
    "print (\"Samples array size : \",sample_data.shape )\n",
    "\n",
    "# Create the the sample dirctory if not exists\n",
    "if not os.path.exists(samples_directory):\n",
    "    os.makedirs(samples_directory)\n",
    "\n",
    "# Save compressed sample data to disk\n",
    "path_smpl = os.path.join(samples_directory,\"lang_samples_\"+str(VOCAB_SIZE)+\".npz\")\n",
    "np.savez_compressed(path_smpl,data=sample_data)\n",
    "print(path_smpl, \"size : \", size_mb(os.path.getsize(path_smpl)))\n",
    "del sample_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sanity check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample record : \n",
      " [10  2  3  6 13  4  4  2 10  0  0  8  3  6  5  3  0  4 10  6  4  0  0  1\n",
      "  2  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 18  0  0  0  0  0  0  1\n",
      "  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  0  0  0  0  0]\n",
      "\n",
      "Sample language :  en\n",
      "\n",
      "Dataset shape (Total_samples, Alphabet): (1750000, 133)\n",
      "Language bins count (samples per language): \n",
      "en 250000\n",
      "fr 250000\n",
      "es 250000\n",
      "it 250000\n",
      "de 250000\n",
      "sk 250000\n",
      "cs 250000\n"
     ]
    }
   ],
   "source": [
    "# utility function to turn language id into language code\n",
    "def decode_langid(langid):    \n",
    "    for dname, did in LANGUAGES_DICT.items():\n",
    "        if did == langid:\n",
    "            return dname\n",
    "\n",
    "# Loading the data\n",
    "path_smpl = os.path.join(samples_directory,\"lang_samples_\"+str(VOCAB_SIZE)+\".npz\")\n",
    "dt = np.load(path_smpl)['data']\n",
    "\n",
    "# Sanity chech on a random sample\n",
    "random_index = random.randrange(0,dt.shape[0])\n",
    "print (\"Sample record : \\n\",dt[random_index,])\n",
    "print (\"\\nSample language : \",decode_langid(dt[random_index,][VOCAB_SIZE]))\n",
    "\n",
    "# Check if the data have equal share of different languages\n",
    "print (\"\\nDataset shape (Total_samples, Alphabet):\", dt.shape)\n",
    "bins = np.bincount(dt[:,input_size])\n",
    "\n",
    "print (\"Language bins count (samples per language): \") \n",
    "for lang_code in LANGUAGES_DICT: \n",
    "    print (lang_code, bins[LANGUAGES_DICT[lang_code]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preprocessing\n",
    "\n",
    "Even if our data is ready for the training, we have to [Standardize](https://en.wikipedia.org/wiki/Feature_scaling#Standardization) the dataset. This will help our model to converge faster since it makes the data more \"computational friendly\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example data before processing:\n",
      "X : \n",
      " [ 8.  0.  5.  5. 21.  2.  1.  7.  6.  0.  0.  6.  1.  8.  5.  0.  0.  7.\n",
      " 10. 10.  4.  3.  2.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0. 24.  0.  0.  0.  0.  1.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  1.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.]\n",
      "Y : \n",
      " 0.0\n",
      "X preprocessed shape : (1750000, 132)\n",
      "\n",
      "Example data after processing:\n",
      "X : \n",
      " [-4.17050153e-01 -1.13270748e+00  6.38392270e-01  1.75331384e-01\n",
      "  1.73380041e+00  5.33405364e-01 -3.80145878e-01  1.91697395e+00\n",
      " -5.14976740e-01 -6.19032502e-01 -7.30089128e-01  1.92491740e-01\n",
      " -1.09850657e+00 -2.04314422e-02 -7.94259489e-01 -1.46870685e+00\n",
      " -4.61331427e-01  5.33611178e-02  1.16716337e+00  1.14771831e+00\n",
      "  3.73478569e-02  2.73657739e-01  1.30994463e+00 -3.63413811e-01\n",
      " -1.49887251e-02 -7.82141924e-01 -1.45357341e-01 -2.73461759e-01\n",
      " -5.23208618e-01 -8.40839669e-02 -2.57696390e-01 -1.61323845e-02\n",
      " -9.63160172e-02 -2.67638355e-01 -5.72866321e-01 -1.24845751e-01\n",
      " -8.58770609e-02 -4.83961701e-01 -7.90864602e-02 -4.38546538e-02\n",
      " -1.56157464e-01 -1.38136834e-01 -3.22708428e-01 -1.61213249e-01\n",
      " -1.93664357e-01 -1.26288012e-01 -3.17232043e-01 -4.67012003e-02\n",
      " -2.45021537e-01 -3.98998171e-01 -6.17011636e-03 -3.92622024e-01\n",
      " -1.21239848e-01 -3.00596684e-01 -4.27031331e-02 -2.16784000e-01\n",
      " -1.39083400e-01 -5.39894253e-02 -2.65944358e-02 -2.88298607e-01\n",
      " -3.61458272e-01 -1.90843359e-01 -2.18461171e-01 -3.85120064e-01\n",
      "  1.30634761e+00 -3.31691280e-02 -3.66677120e-02 -1.20478580e-02\n",
      " -9.79928020e-03  9.66669083e-01 -4.30910617e-01 -4.42056417e-01\n",
      " -4.23938990e-01 -4.10972983e-01 -3.52509290e-01 -3.54364634e-01\n",
      " -3.37542146e-01 -3.59003991e-01 -3.15173328e-01 -3.17866415e-01\n",
      " -4.55488026e-01 -4.60290730e-01 -3.92121077e-01 -3.03513795e-01\n",
      " -4.86154377e-01 -1.12909846e-01  1.68673170e+00 -5.44467688e-01\n",
      "  1.40097451e+00 -2.54374862e-01 -3.80121171e-01 -2.62794465e-01\n",
      " -9.93090570e-02 -1.18509911e-01 -2.20897570e-01 -5.71699217e-02\n",
      " -4.62426692e-02 -1.81627274e-02 -3.68494987e-02 -6.64461171e-03\n",
      " -8.24078172e-03 -4.20710333e-02 -8.09805691e-02 -2.85832421e-03\n",
      " -7.55929330e-04 -2.11150870e-02 -2.74087414e-02 -1.60357123e-03\n",
      " -6.56590983e-03 -2.61863228e-03 -1.99632142e-02 -4.02917853e-03\n",
      " -4.37377803e-02 -1.51186134e-03 -5.64606264e-02  0.00000000e+00\n",
      " -5.21061346e-02 -5.66610694e-03 -1.08200289e-01 -2.99859717e-02\n",
      " -2.85716611e-03 -1.06904586e-03 -2.65672822e-02 -4.56597749e-03\n",
      " -6.93865819e-03 -1.01418595e-03 -4.24020365e-02 -8.98558497e-02\n",
      " -1.53618287e-02 -1.76383916e-03  0.00000000e+00 -7.48927444e-02]\n",
      "Y : \n",
      " [1. 0. 0. 0. 0. 0. 0.]\n",
      "/tmp/train_test/train_test_data_132.npz size :  94.82 MB\n"
     ]
    }
   ],
   "source": [
    "# we need to preprocess data for DNN yet again - scale it \n",
    "# scaling will ensure that our optimization algorithm (variation of gradient descent) will converge well\n",
    "# we need also ensure one-hot econding of target classes for softmax output layer\n",
    "# let's convert datatype before processing to float\n",
    "dt = dt.astype(np.float32)\n",
    "# X and Y split\n",
    "X = dt[:, 0:input_size] # Samples\n",
    "Y = dt[:, input_size] # The last element is the label\n",
    "del dt\n",
    "\n",
    "# Random index to check random sample\n",
    "random_index = random.randrange(0,X.shape[0])\n",
    "print(\"Example data before processing:\")\n",
    "print(\"X : \\n\", X[random_index,])\n",
    "print(\"Y : \\n\", Y[random_index])\n",
    "\n",
    "# X PREPROCESSING\n",
    "# Feature Standardization - Standar scaler will be useful later during DNN prediction\n",
    "standard_scaler = preprocessing.StandardScaler().fit(X)\n",
    "X = standard_scaler.transform(X)   \n",
    "print (\"X preprocessed shape :\", X.shape)\n",
    "\n",
    "# Y PREPROCESSINGY \n",
    "# One-hot encoding\n",
    "Y = keras.utils.to_categorical(Y, num_classes=len(LANGUAGES_DICT))\n",
    "\n",
    "# See the sample data\n",
    "print(\"\\nExample data after processing:\")\n",
    "print(\"X : \\n\", X[random_index,])\n",
    "print(\"Y : \\n\", Y[random_index])\n",
    "\n",
    "# Train/test split. Static seed to have comparable results for different runs\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.20, random_state=SEED)\n",
    "del X, Y\n",
    "\n",
    "# Create the train / test directory if not extists\n",
    "if not os.path.exists(train_test_directory):\n",
    "    os.makedirs(train_test_directory)\n",
    "\n",
    "# Save compressed train_test data to disk\n",
    "path_tt = os.path.join(train_test_directory,\"train_test_data_\"+str(VOCAB_SIZE)+\".npz\")\n",
    "np.savez_compressed(path_tt,X_train=X_train,Y_train=Y_train,X_test=X_test,Y_test=Y_test)\n",
    "print(path_tt, \"size : \",size_mb(os.path.getsize(path_tt)))\n",
    "del X_train,Y_train,X_test,Y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train - Test Split\n",
    "\n",
    "Split: 80% for Train and 20% for Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (1400000, 132)\n",
      "Y_train:  (1400000, 7)\n",
      "X_test:  (350000, 132)\n",
      "Y_test:  (350000, 7)\n"
     ]
    }
   ],
   "source": [
    "# Load train data first from file\n",
    "path_tt = os.path.join(train_test_directory, \"train_test_data_\"+str(VOCAB_SIZE)+\".npz\")\n",
    "train_test_data = np.load(path_tt)\n",
    "\n",
    "# Train Set\n",
    "X_train = train_test_data['X_train']\n",
    "print (\"X_train: \",X_train.shape)\n",
    "Y_train = train_test_data['Y_train']\n",
    "print (\"Y_train: \",Y_train.shape)\n",
    "\n",
    "# Test Set\n",
    "X_test = train_test_data['X_test']\n",
    "print (\"X_test: \",X_test.shape)\n",
    "Y_test = train_test_data['Y_test']\n",
    "print (\"Y_test: \",Y_test.shape)\n",
    "\n",
    "del train_test_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model\n",
    "\n",
    "We will implement a really simple 3 layers Neural Network with [Droput](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) for preventing Overfitting. We will also use the [Xavier initializer](https://www.quora.com/What-is-an-intuitive-explanation-of-the-Xavier-Initialization-for-Deep-Neural-Networks)(one of the best initialization scheme, this will improve the chance to converge in a better \"place\": a point with better accuracy).\n",
    "\n",
    "![nn](http://neuralnetworksanddeeplearning.com/images/tikz40.png)\n",
    "\n",
    "*From http://neuralnetworksanddeeplearning.com/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 500)               66500     \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 300)               150300    \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 300)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 7)                 707       \n",
      "=================================================================\n",
      "Total params: 247,607\n",
      "Trainable params: 247,607\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "# Note: glorot_uniform is the Xavier uniform initializer.\n",
    "\n",
    "model.add(Dense(500,input_dim=input_size, kernel_initializer=\"glorot_uniform\", activation=\"sigmoid\"))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(300, kernel_initializer=\"glorot_uniform\", activation=\"sigmoid\"))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(100, kernel_initializer=\"glorot_uniform\", activation=\"sigmoid\"))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(len(LANGUAGES_DICT), kernel_initializer=\"glorot_uniform\", activation=\"softmax\"))\n",
    "model_optimizer = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=model_optimizer,\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train & Evaluate\n",
    "\n",
    "If you left the default hyperpameters in the Notebook untouched, your training should take approximately:\n",
    "\n",
    "- On CPU machine: 24 minutes for 12 epochs.\n",
    "- On GPU machine: 3 minute for 12 epochs.\n",
    "\n",
    "*Note*: You can follow the execution on Tensorboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.callbacks import TensorBoard\n",
    "\n",
    "# Tensorboard\n",
    "tensorboard = TensorBoard(log_dir=\"run\")\n",
    "\n",
    "# let's fit the data\n",
    "# history variable will help us to plot results later\n",
    "history = model.fit(X_train,Y_train,\n",
    "                  epochs=EPOCHS,\n",
    "                  validation_split=0.1,\n",
    "                  batch_size=BATCH_SIZE,\n",
    "                  callbacks=[tensorboard],\n",
    "                  shuffle=True,\n",
    "                  verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "350000/350000 [==============================] - 15s 42us/step\n",
      "acc: 97.60%\n"
     ]
    }
   ],
   "source": [
    "# Evaluation on Test set\n",
    "scores = model.evaluate(X_test, Y_test, verbose=1)\n",
    "print(\"%s: %.2f%%\" % (model.metrics_names[1], scores[1]*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and now we will prepare data for scikit-learn confusion matrix and classification report\n",
    "Y_pred = model.predict_classes(X_test)\n",
    "Y_pred = keras.utils.to_categorical(Y_pred, num_classes=len(LANGUAGES_DICT))\n",
    "LABELS =  list(LANGUAGES_DICT.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAHGCAYAAACPTHSaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4FFXbx/HvSaFDqNKlgwpKkdBBIBh6E0R9BBVBfGjy0ESKCgiIClixgAgWOgoq0psUKQFC7x0CSA8QSkhy3j92E0NCe2ULSX6f69qL7Jkzs/cwO5M7p8wYay0iIiIi8g8fbwcgIiIi8qBRgiQiIiKSgBIkERERkQSUIImIiIgkoARJREREJAElSCIiIiIJKEESSQaMMWmNMb8bY8KNMdPvYzsvGmMWuDI2bzDGzDXGvOztOEQk6VKCJOJBxpj/GGPWG2MuG2NOOH+RV3PBplsCOYFs1tpn/+1GrLUTrbXBLojnJsaYmsYYa4yZmaC8tLN82T1uZ6Ax5qe71bPW1rfWfv8vwxURUYIk4inGmB7AJ8AwHMnMw8CXQFMXbL4AsMdaG+WCbbnLaaCyMSZbvLKXgT2u+gDjoOuaiNw3XUhEPMAYEwAMBjpba3+x1kZYa29Ya3+31vZ21kltjPnEGHPc+frEGJPauaymMeaYMaanMeaUs/WprXPZIOAd4Dlny1S7hC0txpiCzpYaP+f7V4wxB4wxl4wxB40xL8YrXxlvvSrGmBBn112IMaZKvGXLjDHvGWNWObezwBiT/Q7/DZHALOB55/q+wHPAxAT/V58aY44aYy4aYzYYY6o7y+sB/eLt5+Z4cQw1xqwCrgCFnWXtncu/Msb8HG/7HxhjFhtjzD0fQBFJcZQgiXhGZSANMPMOdfoDlYAyQGmgAjAg3vJcQACQF2gHjDbGZLHWvoujVWqqtTaDtXbcnQIxxqQHPgPqW2szAlWATbeolxX4w1k3GzAK+CNBC9B/gLbAQ0AqoNedPhv4AXjJ+XNdYBtwPEGdEBz/B1mBScB0Y0waa+28BPtZOt46bYAOQEbgcILt9QQedyZ/1XH8371s9ZwlEbkDJUginpENOHOXLrAXgcHW2lPW2tPAIBy/+GPdcC6/Ya2dA1wGSvzLeGKAUsaYtNbaE9ba7beo0xDYa6390VobZa2dDOwCGserM95au8daexWYhiOxuS1r7V9AVmNMCRyJ0g+3qPOTtfas8zNHAqm5+35OsNZud65zI8H2ruD4fxwF/AR0tdYeu8v2RCSFU4Ik4hlngeyxXVy3kYebWz8OO8vitpEgwboCZPj/BmKtjcDRtfVf4IQx5g9jzCP3EE9sTHnjvT/5L+L5EegC1OIWLWrGmF7GmJ3Obr0LOFrN7tR1B3D0TguttWuBA4DBkciJiNyREiQRz1gNXAea3aHOcRyDrWM9TOLup3sVAaSL9z5X/IXW2vnW2qeB3DhahcbeQzyxMYX9y5hi/Qh0AuY4W3fiOLvA3gRaAVmstZmBcByJDcDtusXu2F1mjOmMoyXquHP7IiJ3pARJxAOsteE4BlKPNsY0M8akM8b4G2PqG2M+dFabDAwwxuRwDnZ+B0eX0L+xCahhjHnYOUC8b+wCY0xOY0xT51ik6zi66mJusY05QHHnrQn8jDHPAY8Bs/9lTABYaw8CT+EYc5VQRiAKx4w3P2PMO0CmeMv/Bgr+f2aqGWOKA0OA1ji62t40xtyxK1BERAmSiIc4x9P0wDHw+jSObqEuOGZ2geOX+HpgC7AV2Ogs+zeftRCY6tzWBm5OanyccRwHzuFIVjreYhtngUY4BjmfxdHy0shae+bfxJRg2yuttbdqHZsPzMMx9f8wcI2bu89ib4J51hiz8W6f4+zS/An4wFq72Vq7F8dMuB9jZwiKiNyK0UQOERERkZupBUlEREQkASVIIiIiIgkoQRIRERFJQAmSiIiISAJKkEREREQSuNNdfR9oN07sTLHT79IWqOPtELwqpT9hVM9YTbliNOs4RYuKDPPoyX/jzAGXfeH8sxdOchcutSCJiIiIJJBkW5BERETEjWKivR2BVylBEhERkcTsrZ5AlHKoi01EREQkAbUgiYiISGIxKbsFSQmSiIiIJGLVxSYiIiIi8akFSURERBJTF5uIiIhIAupiExEREZH41IIkIiIiielGkSIiIiIJqItNREREROJTC5KIiIgkpllsIiIiIjfTjSJFRERE5CZqQRIREZHE1MUmIiIikoC62EREREQkPrUgiYiISGK6UaSIiIhIAim8iy3FJ0jR0dE893ovHsqejS+HD2DNhs2M/Pp7YmJiSJc2LUPfeoOH8+Xmgy/GsS50KwDXrkdy7vwFVv8xiV17D/Dex99w+coVfHx86ND6WerXrgbAsRN/03vwCC6EX+KxEkUY3u9/+Pv7e3N378nYMSNp2KAOp06foUzZIAAmTfyK4sWLAJA5IBMXwi9SPjAYgMcff5SvRn9AxkwZiImJoVLlhly/ft1r8d+P1KlTs3TJz6ROnRpfP19++eUPBg8eSc2aVfnwg7fxT+VP6MatvNahJ9HRjr+uPh41mHr1anP16lXatetO6KZtXt6L++fj48Oa1XMIO36S5s1f4duxo6heoxIXwy8B0L59dzZv2UGmTBn5fsJn5M+fFz8/X0Z9/A0//DDNy9Hfnz27V3P5cgTR0dFERUVRuUpDBr7bi8aN6xITE8Op02do374HJ078DcCo2ON/5Srt2ndnUzI4/rG6vfEar776AtZatm3bRbv2PWj36gu80bU9RYsWImfuUpw9e97bYbrNvj1ruHT5MtHRMURFRVGpcgM+eH8ADRs9TWRkJAcOHKZd+x6Eh1/0dqjiBsZa6+0Y/pUbJ3a6JPDvp/3K9t37uBxxlS+HD6Bh6058NrQvRQrkZ8qsOWzduZehfbvdtM7EX2azc+9BhvTpyqGjYRhjKJAvD6fOnKNVh5789v3nZMqYgZ4DPySoemUaBFVn0MivKFG0IM83rX/fMactUOe+t3En1atV5PLlCMaP/zQuQYrvow/eIfziRYYM/QRfX19C1s3jlbbd2LJlB1mzZuHChXBi3Dj7wbhtyw7p06cjIuIKfn5+/LlsJr16DWLixK+oW+859u49wLvv9uLI4WOMnzCFevVq07lTWxo3aUPFCuUYNWoQVas1dmt8xrj7fwC6dXuNJ8uVJmOmDHEJ0pw5i/ll5h831evzZhcCAjLRr/8wsmfPyraty8n/cFlu3Ljh9hjdZc/u1VSu0uCmX/wZM2bg0qXLAHTu/CqPPlqMLl36Uq9ebTp1akuTJm2oUKEco0YOolp19x3/GA9er/PkycWfS2fyeOlaXLt2jcmTvmbu3CVs2bqd8+fDWbxwBhUr10/2CVLCfXy6Tg2WLF1FdHQ07w/rB0DffsM8Ek9UZJj7T/54rm9f7LIvXOqSQR6N3RVS9CDtk6fOsHzNelo0fDquzBiIiLgKwKWIK+TInjXRenMWr6BBUHUACubPS4F8eQB4KHtWsmYJ4Hz4Ray1rN24leCnqgDQtF4tlqxc6+5dcokVK9dy7vyF2y5v2bIxU6b+CkDw00+xdetOtmzZAcC5c+fdmhx5QkTEFQD8/f3w9/cnOjqayMhI9u49AMCiRctp3rwBAE0a1+WniTMAWLtuIwGZA8iV6yHvBO4iefPmpn79IL4bP+muda21ZMiYHoAMGdJz7vwFoqKi3B2ix8UmRwDp06Ul9g/Lxo2DmfiT4/ivW7eRzJkzJfnjH5+fnx9p06bB19eXdGnTcuLESTZt2s7hw8e8HZrXLFy0PK71eM3ajeTNm9vLEbmRjXHdKwlya4JkHN40xuw3xlw1xmw1xrR2LitojLHGmBbGmIXGmCvGmB3GmKfvtl1X+eCLcfR4/eWb/iIf1LszHd96j6CW7fh9wTLa/6fFTescP3mKsBOnqFj28UTb27pzDzduRJE/Ty4uhF8iY4b0+Pn5ApAzRzZOnT7n3h3ygOrVKvL3qdPs23cQgGLFCmMtzJk9kXVr59GrZ0cvR3j/fHx8WB+ygONhW1i0eDnrQkLx8/PjyXJPANDimYbkz+9IivPkycWxo8fj1g07doK8eXJ5JW5XGTliIH37DiUm5uY/HgcPfpMN6xfy0UfvkipVKgC+/GoCj5QoxuFDG9i4YRE9e75DUm2VjmWxzPljEmtWz6FduxfjygcPepP9+9bxwgvNGTRoBOA4/keP/XP8j4WdIE8SP/6xjh8/yaiPv+bg/nUcOxJK+MWLLFy03NtheZS1lrlzJrN2zVzax/suxGr7yvPMm7/UC5GJJ7i7BWkI0A7oDDwGvA98Y4xpGK/OUOAzoDQQAkwxxmRwc1ws+yuErFkCKFmi6E3lP0z/na+Gv83iGeNoVj+ID0d/d9PyuUtWEvxUZXx9fW8qP332HH2HfcKQPl3x8Um+DXPPPdeMqc7WIwA/P1+qVgmkzctdeKpmM5o1rU/tWtW8GOH9i4mJoXxgMAULlSewfFlKlixB69adGDFiIH+tms2lyxFERyfNv4jupkGDIE6dPkOoc7xdrAFvD6fU409RuUpDsmbJTO9enQAIfromm7dsp0DBJwmsUJdPPhlCxoxuP33dqlatZ6hYqT6Nm7Sh439fplq1igC88+6HFClagcmTZ9KpY1svR+l+mTMH0KRxXYoWr0T+AuVInz4d//nPM94Oy6OeqtWcChXr0ahxazp2fIXqzu8CQN+33iAqKopJk37xYoRuFhPjulcS5Lbf5MaY9EAPoL21dp619qC1dhIwFkfCFOtja+3v1tq9QD8gK1DmNtvsYIxZb4xZ/+1P9zcQNHTbLpatCiH4udfoPXgk60K30PGt99i9/yBPPFYcgPq1qrFp+66b1pu7ZAX1g2rcVHY54gqd3hrCG+1aU7pkCQAyB2Tk0uUIoqIcTbF/nz7LQzkSd9clJb6+vjRvVp9p03+LKzsWdoIVK9dy9ux5rl69xtx5SyhbtpQXo3Sd8PCLLPtzFcHBNVmzdgO1aj9DlaqNWLFiDXuc3W3Hj58kn7M1CSBvvtyEHT/prZDvW5XKgTRqGMye3av56cfR1KpZlQnjP+PkyVMAREZG8v0P0ygf6DhFX3q5FbNmzQVg//5DHDp4lBIJ/uhIao47j9/p02f59dd5BAbefDmaPGUmzZvXj6ubP98/xz9f3txx6yd1QUHVOXjoCGfOnCMqKoqZs+ZSuVJ5b4flUTd/F+bGfRdeatOKhg3q0OalLt4Mz+2sjXbZKylyZ1PHY0AaYJ4x5nLsC+gIFIlXb0u8n2Pbqm/ZiW+tHWOtLW+tLd++dav7Cq57hzYsnjGOBVPH8tE7PalQ9gk+H9KPy5evcOhoGAB/rd9E4QL54tY5cPgYFy9dpowzCQK4ceMG3d5+nybBNQmuWSWu3BhDhbKPs+DPvwD4dd5SaletcF8xe1udoOrs3r2PsLATcWULFvxJqVKPxI1TqFG9Ejt37vVilPcne/asBARkAiBNmjTUCarB7t37yZEjGwCpUqWid6/OjBnzIwC/z15A6xdbAlCxQjkuhl+MSyaSogFvD6dwkUCKl6hM6zadWbpsFa+0feOmcTVNmtRlx/bdABw9GhbXYvjQQ9kpXrwIBw8e9krsrpAuXVoyZEgf93OdOjXYvn03RYsWiqvTuHFddu/eD8Ds2Qt4sbXj+FeoUI7w8EtJ+vjHd/RIGBUrliNt2jQA1K5VjV27ku65/f+V8LvwdJ2n2L59N3WDa9KrV0eaPfMKV69e83KU4k7unOYfm3w1Bo4kWHaDfyYjxU13sdZa53ggr/RR+fn5MrB3Z7q/8wHGx4dMGdLzXp+uccvnLllB/drVbxqzNG/pKjZs3sGF8EvMmrcEgKFvvcEjxQrT/fWX6D14JJ+Pm8ijxQrzTAOPDa+6Lz/9OJqnalQme/asHDqwnkGDRzB+whRatWoaNzg71oUL4Xzy6RjWrJ6DtZZ585YwZ+5iL0V+/3Lnzsl34z7B19cH4+PDjBm/M2fOIoa/P4AGDevg4+PDmG9+YNmyVQDMnbuY+vVqs2vnKq5evUr79j28vAfu8f2Ez8mRIxvGwObNO+jc5S0Ahg37lG+/HcXGDYswBvr3H5akZzXlzJmD6dO+BRzXgylTZrFgwTKmThlD8eKFiYmxHDlyjM5d+gIwd+4S6tWrzc6dK7l65RrtX0s+x39dSCi//PIHIevmExUVxaZN2xn77US6dH6VXj07kStXDkI3LGLuvCW8/t/e3g7X5XLmzMGM6eOAf74L8xcsY9eOlaROnZp5c6cAsHbtxrjzIdlJooOrXcVt0/yNMRmB00BHa+34WywvCBwEAq216+OVW+BZa+2MO23fVdP8kyJ3T/N/0CW5uaIu5olp/vJg8uQ0f3nweHqa/7WNv7nsC5emXJMkd+FyWwuStfaSMWYEMMI4rujLgQxAJSAGWOCuzxYRERG5H+6+k/bbwN9AL+Ar4CKwCfjQzZ8rIiIi9yOFd7G5NUGyjv67z52vW0nU5GatTXLNcCIiIslOCn9YbfK9YY+IiIjIv5TiH1YrIiIit6AuNhEREZEEkugdsF1FXWwiIiIiCagFSURERBJTF5uIiIhIAupiExEREZH41IIkIiIiiaXwFiQlSCIiIpKItbpRpIiIiIjEoxYkERERSUxdbCIiIiIJpPBp/upiExEREUlALUgiIiKSmLrYRERERBJQF5uIiIiIxKcWJBEREUlMXWwiIiIiCaiLTURERETiUwuSiIiIJKYutqQpbYE63g7Ba64eXeLtELwqXf7a3g5BvCTGWm+HIJJypPAESV1sIiIiIgkk2RYkERERcaMUPkhbCZKIiIgklsK72JQgiYiISGIpvAVJY5BEREREElALkoiIiCSmLjYRERGRBNTFJiIiIiLxqQVJREREElMXm4iIiEgCKTxBUhebiIiISAJqQRIREZHEUvizD5UgiYiISGLqYhMRERGR+JQgiYiISGIxMa573QNjjK8xJtQYM9v5vpAxZq0xZp8xZqoxJpWzPLXz/T7n8oLxttHXWb7bGFM3Xnk9Z9k+Y8xb9xKPEiQRERFJzMa47nVvugE7473/APjYWlsUOA+0c5a3A847yz921sMY8xjwPFASqAd86Uy6fIHRQH3gMeAFZ907UoIkIiIiXmWMyQc0BL51vjdAbWCGs8r3QDPnz02d73EuD3LWbwpMsdZet9YeBPYBFZyvfdbaA9baSGCKs+4dKUESERGRxFzYxWaM6WCMWR/v1SHBp30CvAnENjdlAy5Ya6Oc748BeZ0/5wWOAjiXhzvrx5UnWOd25XekWWwiIiKSmAun+VtrxwBjbrXMGNMIOGWt3WCMqemyD71PSpBERETEm6oCTYwxDYA0QCbgUyCzMcbP2UqUDwhz1g8D8gPHjDF+QABwNl55rPjr3K78ttTFJiIiIol5aBabtbavtTaftbYgjkHWS6y1LwJLgZbOai8Dvzp//s35HufyJdZa6yx/3jnLrRBQDFgHhADFnLPiUjk/47e77b5XEiRjjI8x5htjzFljjH2QmtREREQEj0/zv4U+QA9jzD4cY4zGOcvHAdmc5T2AtwCstduBacAOYB7Q2Vob7WyB6gLMxzFLbpqz7h15qwWpAdAWaAzkBv7yUhy3NHbMSI4f28ym0MWJlnX/3+tERYaRLVuWuLKPRw1m146VbNywkLJlSnky1PsWHR1Ny3bd6NRnMABrNmzm2Xb/o8Wr3WjTuQ9Hjh0H4MTfp2nbrT8t23Wj+StdWb56PQA3oqLoN/Rjmr/clcatOzH2p+lx2x4w/FNqNGlDs5e7eH7H/qXUqVPz16rZbFi/kE2blvDOOz0BWLrkF9aHLGB9yAIOH9rAjBnj4tb5eNRgdibR4387Pj4+rFs7j5kzJwDw/YTP2bb1T0I3LmLMNyPw83P0zteoUZnTp3YQsm4+Ievm07/f/7wYtWsVL14k7pivD1nAuTO7eKNrewA6d2rLtq1/snnTEoa/39/LkbpHvnx5WLRgOls2L2XzpiV07eKYYV26dElWrfid9SELWLN6DoHly3g5Uve41e+BFi0asXnTEiKvHeXJck94Mbrky1q7zFrbyPnzAWttBWttUWvts9ba687ya873RZ3LD8Rbf6i1toi1toS1dm688jnW2uLOZUPvJRZvJUhFgRPW2r+stSed0+7ixN4Mylt++GEaDRu9mKg8X748PF2nBocPH4srq1+vNsWKFuKRx6rRsWMfRn/xvidDvW8/zfidwgX+6Zp9b+RXDH+7Jz9/9ykN6zzFNz9MA+CbH6ZSt1ZVZoz7lBEDezPk468BWLB0FZE3opj5/edM+/Zjpv82n7ATfwPQrF4QX3800OP7dD+uX7/O08GteLL805QvH0zd4JpUrFCOWrWfoXxgMOUDg1mzdgOzZjnOu3r1alO0aCEedR7/L5LY8b+drl3bsWvXvrj3k6fMpNTjT1G2XB3Spk3Dq6++ELds5ap1BFaoS2CFugwd9ok3wnWLPXv2xx3zChXrceXKVWb9OpeaT1WhSeO6lHvyaUqXqc3IUV97O1S3iIqKovebg3iidC2qVmtMx46v8OijxRg+rD/vDRlF+cBgBg0akWwTxFv9Hti+fRfPtnqNFSvWeCkqD/P8fZAeKB5PkIwxE3Dc2OlhZ/faIWPMMmPMV8aYEcaY08AqT8cV34qVazl3/kKi8pEjBvJWv6HYeCP7Gzeuy48THbdpWLtuIwGZA8iV6yGPxXo/Tp46w/LV62nR8Om4MmMMEVeuAHApIoIc2bM6yjFERFx1lF++Qo5sznIDV69dIyoqmuvXr+Pv50eG9OkAKF+mFAGZMnhyl1wiIsKx//7+fvj7+990vDNmzECtmlX59dd5ADRpXJefkujxv528eXNTv34Q342fFFc2b96SuJ9D1m8iX97c3gjNa4JqV+PAgcMcORLG66+/xIcfjSYy0vF33enTZ70cnXucPHmK0E3bALh8OYJdu/aSN08urLVkzJQRgEwBGTnu/IMoubnV74Fdu/axZ89+L0XkeTbGuuyVFHmjBakbMBjHfQhyA4HO8taAAaoDL3khrjtq3DiYsLATbNmy46byvHlycezo8bj3YcdOkDdPLk+H96988Pm39Oj4Csbnn6/BoDe70PHNwQS1aMvv85fR/kXH+LhObV9g9oJlBLVoS6c3B9Hvf45bWDxdsypp06ShVvOXefrZdrzyfDMCnBfPpMrHx4f1IQs4HraFRYuXsy4kNG5Z06b1WLJ0FZcuXQYgTxI+/rczcsRA+vYdSswtLmp+fn68+J8WzF+wLK6sUsUnWR+ygN9++5HHHi3uwUg9p1WrpkyZOguAYsUKU61aBf5a+TtLFs2g/JOlvRyd+xUokI8ypUuxdl0oPXq9ywfvD+Dg/hA+HP42/Qckj1ZTkYQ8niBZa8OBS0C0s3vttHPRQWttT2vtLmvtzjtswuPSpk1D3z5dGThohLdDcZllf4WQNUsAJUsUvan8h2m/8tWH77D45/E0axDEh184xtrMWbycpvVrs/jn8Xz54bv0HfIxMTExbN25B18fH5bMnMC8qWP5fuqvHD1+0hu75DIxMTGUDwymYKHyBJYvS8mSJeKWPdeqKVOdvyiTowYNgjh1+gyhoVtvufzzz4axYuVaVq1aB0Bo6FaKFqtI+cBgvvxyPNPjjc1KLvz9/WncKJgZP88GwM/PlyxZMlOlWmP6vDWEyZOSZxdbrPTp0zFt6lh69HqXS5cu83qHl+jZeyCFigTSs/cgxn4z0tshirt4f5C2Vz1I0/w33K1C/DtxxsREeCImAIoUKUjBgg+zcf1C9u1ZQ758uQlZO5+cOXMQdvwk+fLniaubN19uwpJAghC6dQfLVq0juFV7eg/6iHUbt9DxzcHs3n+IJx5zJAT1a1dn07ZdAPzyx0Lq1qoGQJlSjxAZGcn58IvMWbicqhXL4e/nR7YsmSnz+CNsjzd2JSkLD7/Isj9XERxcE4Bs2bIQGFiWOXP+GbR5PIke/9upUjmQRg2D2bN7NT/9OJpaNasyYfxnAAzo350cObLSu/eguPqXLl2O65KcN2+J43sQbwJDclCvXi1CQ7dy6tQZwNFKGDsGLWT9JmJiYsju7IpObvz8/Jg+dSyTJ8+M2+eX2jzLzJlzAJgx43cCA5PnIG1BY5C8HUA8d814rLVjrLXlrbXlfXzSeyImALZt20WefKUpWrwSRYtX4tixEwRWrMvff59m9uwFtHF2Q1WsUI6L4Rc5efKUx2L7t7q//jKLfx7Pgmnf8tG7valQ7gk+H9afyxERHDrquH/WXyGhFC6QD4DcOXOwduMWAPYfOsr1yBtkzRxA7pw5WOcsv3L1Glu276FQgbvewf2BlT17VgICMgGQJk0a6gTVYPdux5iDFs80Ys6cRVy/fj2u/u+zF9A6CR7/2xnw9nAKFwmkeInKtG7TmaXLVvFK2zdo2/YFnn76KVq36XLTmKycOXPE/Vy+fBl8fHw4e/a8N0J3m+efaxbXvQbw62/zqVmzCuDobkuVKhVnzpzzVnhuNXbMSHbu2scnn/5zA+TjJ/7mqRqVAahdqxp79x30VngibqU7ad/CTz+O5qkalcmePSuHDqxn0OARjJ8w5ZZ158xdTL16tdm9cxVXrl6lffseHo7Wdfz8fBnYuwvdBwzH+BgyZczAe2+9AUDvzq/y7odf8MO0XzHGMKRvN4wxvNC8AQOGf0rTlzpjLTRrEESJIoUc6wz6iJDQbVwIv+gYu9T2BVo0CvbmLt5V7tw5+W7cJ/j6+mB8fJgx43fmzFkEQKtWTfjwo9E31Z87dzH169Vm185VXE3ix/9ORn/xPoePHGPFcsd92mbNmsvQYZ/wzDMNeb1DG6Kiorl69Rqt23TycqSulS5dWuoE1aBjpz5xZeMnTOHbsSPZFLqYyMgbvNou+dzaIL6qVQJp07olW7buYH3IAgDefns4//1vb0aNGoyfnx/Xr12jY8c3vRype9zq98C58xf49OMh5MiRld9+/YHNm7fT4BYznpONJDq42lWMdeGzVu75Q43pBXRx3jUTY8wyYJu19p5vmOOXKm+KPXJXjy65e6VkLF3+2t4OwascD61OmWK8cL20+9ZRAAAgAElEQVQSeVBERYZ59OS/8nknl51w6bp+meQuXA9SF5uIiIjIA8ErXWzW2hHAiHjva3ojDhEREbmNJDr7zFU0BklEREQSS+Fd2upiExEREUlALUgiIiKSmLrYRERERBJI4dP81cUmIiIikoBakERERCSxJPqIEFdRgiQiIiKJqYtNREREROJTC5KIiIgkYjWLTURERCQBdbGJiIiISHxqQRIREZHENItNREREJAF1sYmIiIhIfGpBEhERkcQ0i01EREQkAXWxiYiIiEh8akESERGRxDSLTURERCQBdbGJiIiISHxJtgXJeDsAL0qXv7a3Q/CqiAPzvB2CV2UoUt/bIYhICqBnsYmIiIgklMK72JQgiYiISGIpPEHSGCQRERGRBNSCJCIiIolpmr+IiIhIAupiExEREZH41IIkIiIiidgU3oKkBElEREQSS+EJkrrYRERERBJQC5KIiIgkpjtpi4iIiCSgLjYRERERiU8tSCIiIpJYCm9BUoIkIiIiiVibshMkdbGJiIiIJKAWJBEREUlMXWwiIiIiCaTwBEldbCIiIiIJqAVJREREEtGz2EREREQSSuEJkrrYRERERBJQC5KIiIgklrIfxaYE6Vby5cvD+O8+5aGc2bHWMu7biXz+xTgmTvyKEsWLABAQkInw8IuUDwwmKKg6w4b2I1UqfyIjb9DnrSEsW7bKy3vx76VOnZqlS34mderU+Pr58ssvfzB48EjGffsx1atX4uLFSwC0a9+dzZu3A1CjRmVGjRyEn78fZ8+cI6hOS2/uwj2Ljo7m+U59eChbVkYP68eajVsY9c2PxFhLurRpGPJmZx7Om5tpv89n8q/z8fXxIV3aNLzb/XWKFMwPwO79hxj88RgirlzB+Pgw5cvhpE6Vis/GTeK3hX9y8VIE6/74yct7+v/j4+PDmtVzCDt+kubNX4krHzVqMK+8/BxZs5UA4OGH8zJmzEhyZM/GuXMXeKXtG4SFnfBS1K6VOnVqli35mVSpU+PnPA8GOc+DGtUrEX6L8yA5KV68CJMmfhX3vnChhxk4aATZsmWhceNgYmIsp0+d4dX23Tlx4m8vRuoetzv+tWpW5YMP3iZVKn82btzKax16Eh0d7e1w3SKlj0EySfVOmf6p8rot8Fy5HiJ3rocI3bSNDBnSs3btPFq2fJWdO/fG1fnwg3cIv3iRoUM/oUyZkvz99xlOnPibkiVL8MfsiRQsVN5d4XlE+vTpiIi4gp+fH38um0mPHu/SoUMb/piziF9++eOmugEBmVi+/FcaNXqRo0ePkyNHNk6fPuu22CIOzHPZtr6f/jvb9+wnIuIKo4f1o9FLXfnsvT4ULpCPKb/OY+uufQzt04XLEVfIkD4dAEv/CmHqb/P5evgAoqKjafV6b97v+wYlihTkQvglMmZIh6+vL5t37CFPzhw0fKmrSxOkDEXqu2xbt9Ot22s8Wa40GTNliEuQypV7gq5d2tG0ab24BGnypK+ZM2cRP/40g5o1q/DyS8/R9tVubosrxsPXq/jnwfJlM+l+h/MgOfPx8eHIoQ1UqdaI8+fDuXTpMgBdOr/Ko48Wp3OXt7wcoXskPP49ew1i0sSvCK73HHv3HmDgu704fPgY4ydM8Ug8UZFhxiMf5HThxdouO+EyT1zi0dhdQWOQbuHkyVOEbtoGwOXLEezatZc8eXLdVKdly8ZMnforAJs2bY/7C2r79t2kTZuGVKlSeTZoF4uIuAKAv78f/v7+d7zl/AvPN2fWrLkcPXocwK3JkSudPH2WFWs30KJBUFyZMYbLVxz7fjniCg9lywIQlxwBXL12Pe7nv9ZvpnjhApQoUhCAzAEZ8fX1BaD0Y8XJ4Vw/KcmbNzf16wfx3fhJcWU+Pj4Mf38AffsNvanuo48WY6mztXTZsr9o3DjYo7G6W/zzwO8u50FyFlS7GgcOHObIkbC45AgcCURy/j9JePyjo6OJjIxk794DACxatJxnmjfwZojuFWNd90qC3J4gGYc3jTH7jTFXjTFbjTGt4y1/xxhz2Bhz3Rhz0hjzg7tj+v8oUCAfZUqXYt260LiyatUqcurUafbtO5io/jPPNCQ0dBuRkZGeDNPlfHx8WB+ygONhW1i0eDnrQhz7P3hwHzZuWMiIjwbGJYHFihUmS+YAFi2czto1c2ndOml0r304ejzdO7TBx/zzh83Anv+lU99hBD3Xgd8XLqfdC83jlk2eNZf6rTszasyP9O3SDoDDx45jjOH1Pu/R6vXefDdllsf3w9VGjhhI375DiYl3UevUqS2z/1jAyZOnbqq7ZctOmjVz/IJo1rQ+mTJlJGvWzB6N151iz4MTYVtYHO88eM95HoyMdx4kZ61aNWXK1H++2+8N7sPB/SG88EJzBg76yIuRudetjr+fnx9PlnsCcFzv8+XP4+Uo3SjGha8kyBMtSEOAdkBn4DHgfeAbY0xDY0wLoBfQCSgGNALWeSCme5I+fTqmTR1Lz17v3vRX0/PPNWOKs/UovsceK86wof3o1LmPJ8N0i5iYGMoHBlOwUHkCy5elZMkS9B/wPqVK1aBS5YZkzZqZ3r07AeDn50u5ck/QpOlLNGj4H/r1/R/FihX28h7c2Z+r15M1SwAlnWPKYv3482y+fL8fi6eOoVm9Wnz01fdxy15oVp+5P42m+2utGfPTDMAxhil02y6G9+vG958OYfHKdazZuMWj++JKDRoEcer0GUJDt8aV5c6dkxbPNGT06PGJ6vd56z1qVK/EurXzqF6jEseOnSA6OoleDW8h9jwokOA8KOk8D7JkzcybzvMgufL396dxo2Bm/Dw7ruztdz6gUJFAJk+eSedObb0YnXvd6vi/2LoTI0cMZPWq2Vy+HJGsvu9yM7cmSMaY9EAPoL21dp619qC1dhIwFkfCVAA4ASyw1h6x1q631n5xh+11MMasN8asj4mJcGfo+Pn5MW3qWCZPnsmsWXPjyn19fWnWrD7Tp/92U/28eXMzffo4Xn21GwcOHHZrbJ4UHn6RZX+uIji4ZlzrQWRkJBO+n0pg+bIAHAs7wYKFy7hy5Spnz55n5co1PPHEY94M+65Ct+9m6V8h1P1PR3oP+YR1m7bRqd8wdu8/zBOPFgegXs0qbNq+O9G69WtVZclfIQDkzJ6NJx9/lCwBmUibJjXVK5Zl597ELYtJRZXKgTRqGMye3av56cfR1KpZlU2hiylSpCA7d6xkz+7VpEuXlh07VgJw4sTftHruNSpUrMc773wAOL4zyU3seVA3wXnwfbzzILmqV68WoaFbOXXqTKJlkyb/QvPk3MXkFP/4r1m7gZq1n6Fy1UasWLEmrrstObIx1mWvpMjdLUiPAWmAecaYy7EvoCNQBJjuXH7QGDPOGPOsMSb17TZmrR1jrS1vrS3v45PerYGPHTOSXbv28cmnY24qDwqqzu7d+26aqRMQkInffv2B/v2H8dfq9W6NyxOyZ89KQEAmANKkSUOdoBrs3r2fXLkeiqvTtEk9tu/YBcDvv8+napUK+Pr6kjZtGgIrlGXXrr233PaD4n/tX2Tx1DHMn/QVHw34HxXKlOKz9/pwOeIKh5xjqVZv2ELhAnkBOHzsn+O9fM1GHs7rGJNWJbAMew8e4eq160RFR7N+yw6KFMjn+R1ykQFvD6dwkUCKl6hM6zadWbpsFTlzleLhAuUoXqIyxUtU5sqVqzz2WDUAsmXLgnF2UfZ5swvffz/Vm+G71L2cB03inQfJlaPF/J/utaJFC8X93KRxXXbv3u+NsNzudsc/R45sAKRKlYrevTozZsyP3gzTvVJ4F5u7p/nHJmCNgSMJlt2w1h41xpQAgoA6wEjgXWNMRWute5uI7qBqlUBat27J1q07WB+yAHD84pg3bwnPtWoaNzg7VqdObSlSpCAD+ndnQP/uANRv8EKSGaycUO7cOflu3Cf4+vpgfHyYMeN35sxZxIL508iRIysYw5bN2+nU2TFzZdeufcxfsJSNGxcRExPD+O8ms/0WLS8POj9fXwb2/C/dB43AxxgyZUzP4F6dAcf4ozUbt+Dn50emDOkZ2qcrAAEZM9CmZWNe6NQHYwzVK5SjRqUnARj1zY/8sWQF165fJ+i5DrRoEESnl5/z2v65w1M1qvDekLfAWlasWMsb3fp7OySXiX8e+DjPgz/mLGLh/Glkz5EVYwyb450HyVG6dGmpE1SDjp3+GTYwbGhfihcvQkxMDEeOhCXb/b/d8f/g/QE0aFgHHx8fvvnmh7hJCpL8uHWavzEmI3Aa6GitTTyAIXH9nMBJoK61dsGd6rpzmr882Fw5zT8p8sQ0/weVp6f5izxIPD3N/1zzp1x2wmWd+WeSm+bv1hYka+0lY8wIYIRxtMMvBzIAlXA0ukU6Y1gLXAaeA24AD3b/jIiISHKXRLvGXMUTd9J+G/gbx2y1r4CLwCbgQyA90AcYAfgDO4BnrLVJd5SriIiIJHluT5Csow/vc+frVpL+jWNERESSGasWJBEREZEEUniCpEeNiIiIiCSgFiQRERFJJKV3sakFSURERBLz4I0ijTFpjDHrjDGbjTHbjTGDnOWFjDFrjTH7jDFTjTGpnOWpne/3OZcXjLetvs7y3caYuvHK6znL9hlj7noDLyVIIiIi4m3XgdrW2tJAGaCeMaYS8AHwsbW2KHAex7Ndcf573ln+sbMexpjHgOeBkkA94EtjjK8xxhcYDdTH8ZSPF5x1b0sJkoiIiCRiY1z3uutnOcQ+Fd7f+bJAbWCGs/x7oJnz56bO9ziXBznvt9gUmGKtve68ZdA+oILztc9ae8BaGwlMcda9LSVIIiIikogrE6T4D5t3vjok/DxnS88m4BSwENgPXLDWRjmrHAPyOn/OCxwFcC4PB7LFL0+wzu3Kb0uDtEVERMStrLVjgDF3qRMNlDHGZAZmAo94IrbbUYIkIiIiiXhrFpu19oIxZilQGchsjPFzthLlA8Kc1cKA/MAxY4wfEACcjVceK/46tyu/JXWxiYiISGLWuO51F8aYHM6WI4wxaYGngZ3AUqCls9rLwK/On39zvse5fInzyR2/Ac87Z7kVAooB64AQoJhzVlwqHAO5f7tTTGpBEhEREW/LDXzvnG3mA0yz1s42xuwAphhjhgChwDhn/XHAj8aYfcA5HAkP1trtxphpOJ7tGgV0dnbdYYzpAswHfIHvrLXb7xSQcSRcSY9/qrxJM3C5bxEH5nk7BK/KUKS+t0Pwmpgker0ScYWoyLC7N8W40MkaNV12wuVavsyjsbuCWpBEREQkERuT5HIal9IYJBEREZEE1IIkIiIiiaT0Z7EpQRIREZFE7D3MPkvO1MUmIiIikoBakERERCQRdbGJiIiIJKBZbCIiIiJyE7UgJUHGpOysPmPRBt4OwasuH5zv7RC8Jl3BYG+HIJJipPT7sipBEhERkURSehebEiQRERFJRAnSbRhjMt1pRWvtRdeHIyIiIuJ9d2pB2g5YIH4KGfveAg+7MS4RERHxIo1Bug1rbX5PBiIiIiIPjpTexXZP0/yNMc8bY/o5f85njHnSvWGJiIiIeM9dEyRjzBdALaCNs+gK8LU7gxIRERHvsta47JUU3csstirW2nLGmFAAa+05Y0wqN8clIiIiXpTSHzVyL11sN4wxPjgGZmOMyQak8P82ERERSc7upQVpNPAzkMMYMwhoBQxya1QiIiLiVTFJtGvMVe6aIFlrfzDGbADqOIuetdZuc29YIiIi4k1JdeyQq9zrnbR9gRs4utn0gFsRERFJ1u5lFlt/YDKQB8gHTDLG9HV3YCIiIuI9Nsa47JUU3UsL0ktAWWvtFQBjzFAgFHjfnYGJiIiI96T0O2nfS3fZCW5OpPycZSIiIiLJ0p0eVvsxjjFH54Dtxpj5zvfBQIhnwhMRERFvSKpdY65ypy622Jlq24E/4pWvcV84IiIi8iDQNP/bsNaO82QgIiIiIg+Kuw7SNsYUAYYCjwFpYsuttcXdGJeIiIh4UUq/D9K9DNKeAIwHDFAfmAZMdWNMIiIi4mXWuu6VFN1LgpTOWjsfwFq731o7AEeiJCIiIpIs3UuCdN35sNr9xpj/GmMaAxld8eHGmAnGmNmu2JYrpU6dmr9WzWbD+oVs2rSEd97pedPyj0cN5vy5PXHvO7zWhtCNi1gfsoBlS2fy6KPFPB2yy/n4+LBu7TxmzpwAQMeOr7Bjx0oirx8jW7YscfUyZcrIzF/Gsz5kAZtCF/PSS628FLHrBARkYvKkr9myeSmbNy2hYsVyccv+160D168djfs/aNwomPUhC1i3dh5/rfqDKlUCvRX2vxIdHc2zr/Wkc9+hAKzZsIVWHXrSsn0PXurajyNhjjt6zJq3hBrNXqFl+x60bN+Dn/9YGLeN0kEt48q79h8WV75241ZadehJ87bd6P/+Z0RFR3t25+5T6tSpWe28DmzetIR3ndeBH77/nO3blrMpdDFjx4zEz+9eH0iQtOTLl4dFC6bHnQddu7QDYNDA3mzcsJD1IQuY+8ckcufO6eVIPaNrl3ZsCl3M5k1LeKNre2+H4xEx1rjslRQZe5e2L2NMRWAHkAXHWKQA4ANr7ar7/nBjApwxXDDGLAO2WWu73Mu6/qnyurXRLn36dEREXMHPz48/l82kR493WbtuI0+We4KuXdvTtGk9smR1DMPKmDEDly5dBqBRo6f57+sv06hxa7fFZoz7v2zdur3Gk+VKkzFTBpo3f4UypUty/kI4CxdMp3KVBpw9ex6APm92ISAgE/36DyN79qxs27qc/A+X5caNG26Lzd37/+23o1i1ah3jx0/B39+fdOnSEh5+kXz5cvP1Vx9RvEQRKld2/B/Efk8ASpV6hEkTv+KJ0rXcGt+lA/Nctq3vp/3G9t37ibhyhdHv96dRm858NqQvhQvkY8qsuWzdtY+hb3Vl1rwlbN+9n/7dXku0jQr1/8O6uZNuKouJiSH4+df5duQgCubPwxffTSZPzhw807BOovX/P9IVDL6v9f+/4l8Hli+bSfce75I1a2bmzlsCwE8/jmbFirV8M+YHj8blCblyPUTuXA8RumkbGTKkZ93aebRo+SrHjp2Iu9516fwqjz5anM5d3vJytO5VsmQJJv70JZWrNCQy8gZzZk+kU5e32L//kEfjiIoM82imEfpwU5f9ni175NcklyXdtQXJWrvWWnvJWnvEWtvGWtvEFcmRc9vh1toLrtiWq8X+0vP398Pf3x9rLT4+Pgwf/jZv9R1yU93YiwU4Lqh3SzofdHnz5qZ+/SC+G//PL71Nm7dz+PCxRHWttWTImB6ADBnSc+78BaKiojwWq6tlypSR6tUqMn78FABu3LhBePhFAD768F369ht60/GN/Z5A0jv2J0+fYcWaDbSIl7QYY7js3KfLEVd4KF5r4f/HhYuX8Pf3o2D+PABULl+ahSuS3h1C4l8H/JzXgdjkCCAkZBP58uX2VnhudfLkKUI3Oe72cvlyBLt27SVvnlzJ7np3Lx55pBjr1oVy9eo1oqOjWb5iDc2baaRJcnenG0XOxHFjyFuy1j5zvx9ujJkAZAfOAE8BTxljOjsXF7LWHrrfz/i3YruYihQpyFdfT2BdSChdu7Rj9uwFnDx5KlH9jv99mW7dOpAqVSqC6ybtbqaRIwbSt+9QMmbMcNe6X341gV9+Hs/hQxvImDEDL7bumKQvmAUL5uf06XOMHTuKJx5/lI2hW+nZ812Calfn+PGTbN26M9E6TZrUY8h7fciRIzvNmr/shaj/nQ+/+I7ur7/ElatX48oG9upEp75DSJ0qFRnSp2Pi6OFxyxYtX82GLTsomC83b3Z+lVwPZQcgMjKS517vjZ+vD6/+5xmCqlUkS0AmoqOj2b57HyVLFGXhn6s5eeqMx/fxfsVeB4rGuw7E8vPz48UXW9CjxztejNAzChTIR5nSpVi7zrH/7w3uQ+sXWxJ+8SJ1nn7Wy9G53/btu3hvcB+yZs3C1atXqV+vNus3bPZ2WG6XhC/lLnGnFqQvgNF3eLlSN2A1jtlyuZ2voy7+jP+XmJgYygcGU7BQeQLLl6VatYq0aNGIL0Z/d8v6X339PY88WpV+/YfSr283D0frOg0aBHHq9BlCQ7feU/3gp2uyect2ChR8ksAKdfnkkyH3lFg9qPz8/ChbthRjxvxAxUr1uRJxhbcH9ODNN7swaPDIW67z22/zeKJ0LZ5t1Z6B7/bycMT/zp+r15M1cwAlSxS5qfzHGb/z5fsDWDz9W5rVq81HX44HoGblQOZP/oZfxn1MpSdL03/4Z3HrzJ/yDVO/+YjhA7rz4RffcTTsJMYYPny7Jx+OHs8LHd8kXbq0+Prcy5DHB0vsdaCA8zpQsmSJuGVffD6MFSvWsnLVOi9G6H7p06dj2tSx9Oj1blzr0dvvfEChIoFMnjyTzp3aejlC99u1ax8ffTSauXMmMWf2RDZt3k50dIy3w3K7lD4G6bZXLGvt4ju9XBmEtTYciASuWGtPOl+JRnQaYzoYY9YbY9bHxES4MoTbCg+/yLI/V1GzZhWKFCnIrp2r2LtnDenSpWXnjpWJ6k+d+itNmtT1SGzuUKVyII0aBrNn92p++nE0tWpWZcL4z25b/6WXWzFr1lwA9u8/xKGDRylRoqinwnW5sLATHAs7QUjIJgB+mTmHsmVLUbBgfkJC5rN791/ky5ubNWvmkjNnjpvWXblyLYUKPXzTIPYHVei2XSz9K4S6z79O78GjWBe6lU5vDWH3/kM88ZhjbF29WlXZtH03AJkDMpIqlT8ALRrWYceeA3HbypkjGwD58+SifJlS7NznWFamZAm+/2wok7/6kPJPPEaBfHk8uYsuFXsdqBtcE4C3B3QnR45s9Oo90KtxuZufnx/Tp45l8uSZced5fJMm/0Lz5g28EJnnjZ8whYqV6lMrqAUXLoSzd++Bu68kSVqS+pPOWjvGWlveWlvexye92z4ne/asBARkAiBNmjTUCarBxo1byf9wWYoVr0Sx4pW4cuUqjz5WDYCiRQvFrdugQR327TvottjcbcDbwylcJJDiJSrTuk1nli5bxStt37ht/aNHw6hdy/H/8NBD2SlevAgHDx72VLgu9/ffpzl27ATFixUGoFatqoSGbiP/w2UpUaIKJUpU4VjYCSpVqs/ff5+mSOGCceuWKVOKVKlSxw1gf5D977XWLJ7+LfOnfMNH7/SgQtnH+WxoXy5fvsKho8cBWL1+M4UfzgfA6bPn4tZd9lcIhR/OC0D4pctERjoG5J8Pv8imbbsoUiA/AGfPO4YXRkbe4LvJM2mVxP5wuNV1YPfu/bza9gWCn67Ji607J+nu5HsxdsxIdu7axyefjokri3+9a9K4Lrt37/dGaB6XI/YPgfx5aNasPpOnzPRyRO5nrXHZKylKnvNT71Pu3Dn5btwn+Pr6YHx8mDHjd+bMWXTb+p06vkLtoOpE3Yji/PlwXm33Pw9G6xmdO79Kzx4dyZUrBxvWL2TevKX8t2Nvhg37lG+/HcXGDYswBvr3H5YkEoQ76d79bSZM+JxUqfw5ePAIr3Xoedu6zZrXp/WLLbhxI4qrV6/Ruk0nD0bqWn6+vgzs1ZHu736IjzFkypiBwW86hgRO/GUOy1aF4OvrQ0CmjLz3VlcADh4+xqBRX+NjDDHW0u6F5hQp6EiQJkz9lT9Xr8daS6smdalY7nGv7du/Ef864OO8DvwxZxHXrhzm8OFjrFzxGwCzZs1hyNBPvByt61WtEkib1i3ZsnUH60MWAPD228Np2/Z5ihcvQkxMDEeOhNGpc/KewRZr+tSxZM2WhRs3onjjjf5xkzeSs6TaNeYqd53mH1fRmNTW2usu/XDnIG1rbSNjzAJgv7W2472s6+5p/g8yT0zzf5Cl9P135TT/pMbT0/xFHiSenua/Ns8zLvs9W/H4L0nuwn3XLjZjTAVjzFZgr/N9aWPM526I5RBQwRhT0BiT3XlzShEREfEC68JXUnQvSchnQCPgLIC1djPgjjvhjcAxUHsHcBp42A2fISIiIvcgpc9iu5cxSD7W2sMJujVc8swAa+0r8X7eA1R2xXZFRERE7se9JEhHjTEVAGuM8QW6Anvuso6IiIgkYUl19pmr3EuC1BFHN9vDwN/AImeZiIiIJFPJ/1aYd3bXBMlaewp43gOxiIiIiDwQ7pogGWPGcotB6NbaDm6JSERERLzOoi62u4l/h8Q0QHO8/Jw0ERERca+YpDo/30XupYttavz3xpgfgcQPIRMRERFJJv7No0YKATldHYiIiIg8OGLUxXZnxpjz/DMGyQc4B6SMh++IiIikUBqDdAfGcXfI0kCYsyjGJvfHV4uIiEiKd8cEyVprjTFzrLWlPBWQiIiIeF9Kvw/SvTyLbZMxpqzbIxEREZEHhsW47JUU3bYFyRjjZ62NAsoCIcaY/UAEYHA0LpXzUIwiIiIiHnWnLrZ1QDmgiYdiERERkQdESu9iu1OCZACstfs9FIuIiIg8IJQg3V4OY0yP2y201o5yQzwiIiIiXnenBMkXyABJdHSViIiI/GtJdXC1q9wpQTphrR3ssUhERETkgRGTsvOj/2vvzuNtqv4/jr8+XGQeUgkNxgZ9KWMkyRgiMjR8M30VIURE86C+KWmeEKHBXIYyT6EM1zyLkCGSzEOmu35/7O3+7mTqe8/Z7j3vp8d5uHftfc75rHvuPftzPmutvc+5zD/CfzQiIiISqc5VQaoatihERETkkqJrsZ2Fc25vOAMRERGRS0ekX1fsvBervVR5l4mLTDGRfjm8CO9/putrBB1CYI79PifoEAKVMe+dQYcgEjFSbIIkIiIioaPzIImIiIgkEBPBIzWgBElERESSENmTGc69zF9EREQkIqmCJCIiIolE+hwkVZBEREQkkRhLvtv5mNk1ZjbTzNaY2Woz6+S35zKzqWa2wf8/p99uZvaBmW00sxVmVjLOYzX3999gZs3jtJcys5X+fT6w87pOaMkAACAASURBVCyHV4IkIiIiQTsFPOWcuxm4HWhvZjcDPYDpzrkiwHT/e4BaQBH/1hr4FLyECngJKAeUBV46k1T5+zwW5373nCsgJUgiIiKSSAyWbLfzcc7tdM4t8b8+BKwF8gH3AYP93QYD9f2v7wOGOM98IIeZXQ3UBKY65/Y65/YBU4F7/G3ZnHPznXMOGBLnsZKkBElEREQSccl4M7PWZrYozq312Z7XzK4HbgMWAFc553b6m3YBV/lf5wO2xbnbdr/tXO3bk2g/K03SFhERkZByzvUD+p1vPzPLAowGnnTOHYw7Tcg558wsbGcfUAVJREREEgnnJG0AM0uHlxx97Zz71m/+wx8ew/9/t9++A7gmzt3z+23nas+fRPtZKUESERGRRGKS8XY+/oqyAcBa59w7cTaNA86sRGsOjI3T3sxfzXY7cMAfipsM1DCznP7k7BrAZH/bQTO73X+uZnEeK0kaYhMREZGg3QE0BVaa2TK/7VmgFzDCzFoBvwFN/G0TgNrARuAo0BLAObfXzHoC0f5+rzrn9vpftwMGARmBif7trJQgiYiISCLhvNSIc24unHW5W9Uk9ndA+7M81kBgYBLti4BbLjQmJUgiIiKSyIXOHUqtNAdJREREJAFVkERERCSRSL8WmxIkERERSSTSEyQNsYmIiIgkoAqSiIiIJOIifJK2EiQRERFJRENsYWZm35vZoHA/r4iIiMiF0hykc0iTJg0LF0ziu+8GATBj+miiF04meuFktmxexKiRnwPw0IMNWLxoKksWT+PHWWMo/q+bAow6eRUtWohF0VNib3v3rKNjh0fJmTMHkyYMZe3quUyaMJQcObIHHWrIZM+ejeHD+rFq5Y+sXDGL28uVAqB9u5asWvkjy5fNoNcbzwUcZWgk1fcSJYrx05zxLIqewvx5EyhT+tagw/xHTp8+TaMW7WnX7SUAFixeRuOWT1D/kcd5tufbnDp1Ot7+K9eup0SlOkyZOSe2beyEqdR+oBW1H2jF2AlTY9vf7zuIqg2aUqZag/B0JgQyZMjAvJ++Z/GiqSxfNoOXXnwqdlvPV7uzZvUcVq6YxRPt/xNglKGTP39epk0ZyYrlM1m+bAYdnmgFQPHiNzN39jiWLpnGmO8GkTVrloAjDZ1wXmrkUqQhtnPo0KEV69ZtJGs27w+gStWGsduGD+vH+PGTAdi8ZStVqzVi//4D1Kx5N5988hYV76wbSMzJ7ZdffqV0mRqAlzBu3bKYMWMn0v3p9syYOZe3en/M093a0/3p9jzz7H8DjjY03n3nVSZPnskDD7YmXbp0ZMqUkcp3VaBe3ZqULFWdEydOcMUVlwcdZkgk1fdh33xGz9feYdLkmdS6pwq93niOqtUbBx3qRftq5FgKXn8th48cJSYmhmdf68OA99/g+mvz81H/IYydOI2GdWsCXjL17idfUKFMydj7Hzh4iE+/+IbhAz4A4IFWHalc8XayZ8tK5TvK8XDDetR+sFUgfUsOx48fp1qNJhw5cpSoqChmz/qOSZNmcuONhcmfPy/FbqmEcy7V/u6fOnWKbk+/wtJlq8iSJTMLF0xi2vTZ9P2sN92792T2nPm0aP4AXZ9qy0sv9w463JAI55m0L0UhrSCZWSYzG2Rmh83sDzN7NsH29Gb2ppltN7OjZhZtZjVDGdOFypfvamrVqsrAL75JtC1r1ixUrlyBseO8BGn+/MXs338AgAULlpAv39VhjTVcqlapyKZNv7F16w7q1q3JkC9HAjDky5HUq3dPwNGFRrZsWbmzYjkGfjEUgJMnT3LgwEHatGnGW70/5sSJEwD8+edfQYYZEmfru3OOrNmyevtkz8rvO/8IMsx/ZNfuP5n988LYBGj/gYOki4ri+mu9i32XL1OSabPmxu7/zahxVK98B7ly5oht+2nBYsqXuY3s2bKSPVtWype5jZ8WLAagxC03cUXuXGHsUWgcOXIUgHTpoohKlw7nHI+3acZrr7+Ld6WH1Pm7D7Br126WLlsFwOHDR1i3bgP58uahaJGCzJ4zH4Bp0+fQoEHtIMOUEAr1ENvbQHWgId61VG4DKsXZ/gVwF/Aw3vVRBgPjzaxEiOM6rz5vv8wzz7xOTEziHPq+ejWZOfMnDh06nGhby5YPMnnyzHCEGHZNmtzHsOFjALjqytzs2rUb8N5Irroyd5ChhUyBAteyZ89fDPj8XaIXTqbvZ73JlCkjRYoUpGLFsvw8dzwzpo2idKnAf2WT3dn63qXrS7z5xvNs/jWat3q9wHPPvxF0qBftzff70qVdK8y8t8CcObJz+nQMq9b+AsCUWXPZtXsPAH/8uYfps3/mgQZ14j3GH3/uIc+VV8R+f9UVufnjzz1h6kF4pEmThkXRU9i5YwXTp89mYfRSCha8niaN6zF/3gS+H/clhQsXCDrMkLvuuvzcWuIWFixcypo1v1CvnpdYN2p4L9fkzxtwdKETY8l3S4lCliCZWRagFfC0c26yc24V3tV2Y/zthYCHgCbOudnOuU3OuY/wrtDbJlRxXYjatauy+889LF26MsntTR6oz/DhYxO133VXBVq2eJBnn3s91CGGXbp06ah7bw1Gjf4+ye1nPk2mNlFp03Lbbf+ib98hlClbkyNHjtL96SeIikpLzpw5qFCxLt17vMbQbz4LOtRkd7a+t2ndjKe6vUyBQmV4qtsr9O/bJ+hQL8qsnxaQK2cOit1YJLbNzOj9ag/e+qAfDz7aicyZMpImjff2+Ob7fenc9j+x30eSmJgYSpepwXUFSlOm9G0UK3YDGTKk5++/j3N7+dp8PvAbPu+Xsl7/i5U5cyZGDO9Pl64vcejQYR5t3YW2bZqzYP5EsmbNzIkTJ4MOMWQ0Byl0CgHpgXlnGpxzh83sTNZREu/KvWvM4qWXGYAZST2gmbUGWgOkTZuDNGkzhyBsqFC+DPfWqcE9Natw2WUZyJYtK4O++IAWLTty+eU5KVP6Vho3fjTeff51y0189tlb1KvXlL1794ckriDdc8/dLF26kt1nPlXv3kOePFeya9du8uS5kt2ptMy+fcdOtm/fycLopQB8++0PPN3tCXZs38mYMRMBiF60jJiYGHLnzsWePXuDDDdZna3vd9xRhs5dXgRg1Kjx9PssZc2/WLpiDbPmzmfOvGiOnzjpJX6vvMWbLz3NkE/fBrzhs9+27QBg9boNdHupFwD7Dhxkzrxo0qZNy1VX5CZ66YrYx/3jzz2Uua14+DsUBgcOHGTWjz9Rs0Zltu/YyXdjJgAwZsxEBvR/J+DoQicqKoqRw/szdOh3sX/v69f/Sq06DwNQpEhBatdKdKF5SSWC/EiUBm8OWBng1ji3m4Akl0U45/o550o750qHKjkCeP6FXhQsVIaiN5TnkabtmTnrJ1q07AjA/ffXYcKEaRw/fjx2/2uuycvwEf1p2bITGzZsDllcQXrwgfqxw2sA34+fQrOm3sTcZk0bx05YT23++ONPtm//naJFCwFQpUpF1q79hbHjJlO5cgXAe5NMnz59qkqO4Ox9/33nH9xVqbzXdndFNmxMWb/zndu2ZPqYr5gyejC9X+lB2VIlePOlp/lrn/fB5sSJEwz8eiRN6ntzSyaPGsSU0YOZMnowNSpX5Pmu7alaqQJ3lCvFzwuXcODgIQ4cPMTPC5dwh7/CMTXInTsX2bNnA+Cyyy6jWtVKrF//K+PGTaLyXd7v/l2VyvPLhk1BhhlS/fv1Ye26jbz3fr/YtjOT0s2MZ5/pRN9+XwYVXsipghQ6vwIngduBTQBmlhlvrtGvwFK8ClIe51yKmbTTpPF99H7743htzz3bmctz5eDDD7xVXKdOnaJ8hTpJ3T1FypQpI9WqVqJtu+6xbW/2/phh33xGyxYPsXXrdh58+PEAIwytTp1fYMjgD0mfPh2bN2+l1aNdOHLkKJ/378OypdM5ceIk/2n1ZNBhhkRSfR83fjLvvPMqUVFRHP/7b9q2fTroMJPFF1+P4sefF+JiYnigQR3KlTr36QuyZ8tKmxYP8eCjnQB4vOXDZPcnr/f5eAATps7k77+PU7X+I9xf9x7at3ok5H1ITldffRUDB7xH2rRpSJMmDaNGjeeHCdOY+9NCvhz8EZ06PcaRw0dp83i3oEMNiTsqlKHpI41YsXINi6KnAPDCC70oXLgAbdu2AGDMmAkMGjw8wChDK3VOnLhwFsq5I2b2KXAvXkXod+BFoBbwrXOuhZl9BdwJPAUsAXIBlYFNzrlvz/XY6TPkj9jXLiaVzvcROZ9jv885/06pWMa8dwYdggTo1IkdYZ3u/Pa1jyTbwabr1q9S3FTtUJ8HqSuQGfgOOAp86H9/RkvgOeAtID+wF1gIpJiKkoiISGqUUlefJZeQJkjOuSNAM/+W1PaTwMv+TURERC4RKXXuUHKJvHWrIiIiIuehS42IiIhIIpE+21UJkoiIiCQSE+EpkobYRERERBJQBUlEREQSifRJ2kqQREREJJHIHmDTEJuIiIhIIqogiYiISCIaYhMRERFJINLPpK0hNhEREZEEVEESERGRRCL9PEhKkERERCSRyE6PNMQmIiIikogqSCIiIpKIVrGJiIiIJBDpc5A0xCYiIiKSgCpIIiIikkhk14+UIImIiEgSIn0OkobYRERERBJQBUlEREQSifRJ2ik2QYpxkf3CiUSijHnvDDqEQB3bNiPoEAKVo0DNoEOIKJF+lNUQm4iIiEgCKbaCJCIiIqET6ZO0lSCJiIhIIi7CB9mUIImIiEgikV5B0hwkERERkQRUQRIREZFEtMxfREREJIHITo80xCYiIiKSiCpIIiIikoiG2EREREQS0Co2EREREYlHFSQRERFJRCeKFBEREUlAQ2wiIiIiEo8qSCIiIpKIhthEREREEtAQm4iIiIjEowqSiIiIJBLjNMQmIiIiEk9kp0caYhMRERFJRBUkERERSUTXYhMRERFJINKX+Qc6xGZmLczscJAxXKg0adIQvXAyY78bDMCAz99lw/p5LIqewqLoKZQoUSzgCMOjwxOtWLZ0OsuXzaBjh0eDDicsEr72Ve6uyMIFk1gUPYUfZ35HoULXBxtgCPXv14ffty9n2dLpsW2vvNyNJYunsih6ChN/+Iarr74qwAhDJ3/+vEybMpIVy2eyfNkMOjzRCoA333ieVSt/ZMniqYwa+TnZs2cLONJ/5vTp0zRq1Yl23V8FYP7i5TRu9SQN/9OJpu27s3X77wD8vms3rZ58ngYtOtCi47Ps2r0HgHUbNvHvtt24r1l7GrTowMTpc2If+5vR31ProdbcUqke+/YfDH/nLsKnn73Fli2LiI6eHNvWoEFtohdN4dDhTdxW8l+x7Q88cB/z5k+IvR06vInixW8GoHHjeixcOIkFCyYyZuxgLr88Z9j7IslLc5AuUMcOj7Ju3YZ4bd2feY3SZWpQukwNli9fHVBk4VOs2A20avUw5SvUoWSp6tSpXS1VJwdnJHztP/roDZo1f4LSZWowdNgYnn2mU4DRhdaQISOoc++/47W93edTSpaqTukyNfhhwjSef65zQNGF1qlTp+j29CsUL3E3d1SsS9u2LbjppiJMmz6bErdWoWSp6mzYsIke3Z8IOtR/5KtR4yl43TWx3/fs8ym9XniK0QPfp061u+g7ZAQAb38ykHo17+a7QR/StvkDvNdvCACXXZaB/z7bmbFDPqbv2y/z5oefc/CQ93n3tn/dxOfv9CRvnivD37GL9NWXo6hfv3m8tjVr1vPwQ48zd+7CeO3Dh4+l/O21KX97bR5t1ZktW7axYsUa0qZNS+/eL1Kr1kOUK1eLVSvX0ubx+I+ZEsUk4y0lUoJ0AfLlu5rataoycODQoEMJ1I03FmHhwqUcO/Y3p0+fZvac+TSoXyvosEIqqdfeOUe2rFkByJ49Kzt3/hFUeCE3Z+4C9u7bH6/t0KH/L/pmzpwJl0qXAu/atZuly1YBcPjwEdat20C+vHmYOm02p0+fBmD+giXky3d1kGH+I7t272H2vEU0rFM9ts3MOHL0KACHjhzhity5APh1yzbKliwOQNmSxZk5dwEA11+Tj+uuyQvAlbkvJ1fO7LHVopuKFiJfCqks/vTTQvbuPRCvbf36X9mwYdM579e4ST1GjRoPeD87zMiUKRMAWbOljveFGFyy3VKisCRIZlbJzOab2WEzO2BmC83sliT2y2lmP5nZZDPLHI7YLsQ7fV6hxzOvERMTPw/u+Wp3liyeSp/eL5M+ffqAoguf1avXUbFiOXLlyknGjJdR654q5M+fN+iwQiqp175Nm66MH/clWzYt4t//bsibb30UYITB6Plqdzb/Gs1DDzXg5Vd6Bx1OyF13XX5uLXELCxYujdfessWDTJo8M6Co/rk3P/ycLm1bYGn+/xDwytNP0PbpV6nasCXjJ8/i0X83AuCGwgWYNnseANNmz+PI0WPsPxB/2Gzlml84efIU1+TLE75OBKxhw3sZOWIc4FUbn+z0PAujJ/HrpoXceGNhBg8aHnCE8r8KeYJkZlHAWGAuUAIoB7wHnE6wX15gNrAdqOucOxLq2C5EndrV2L17D0uWrozX/tzzb1DslkrcXr4OOXPl4Olu7QKKMHzWrdtI794fM3HCN0z4/muWLV/N6dMptXh6fmd77Tt1eoy69ZpyfcHSDB48nLd7vxRQhMF54cU3KVCoDEOHfkf7di2DDiekMmfOxIjh/enS9aV41bNnenTk1KlTfPPNtwFGd/Fm/RxNrpzZKXZD4XjtQ0aM5dO3XmT66C+oX7sqb300AICu7VqyaNkqGrXqxKJlq7nqistJEyex+nPPXp55/V1ee6ZjvPbUrHSZWzl29Bhr1vwCQFRUFI899ggVytehUMGyrFq1jq6p4JjgkvFfShSOVWzZgBzAeOfcr37bOgAzK+f/XxiYCkwG2jnnkjzqmllroDWApc1OmjShLzJVqFCauvfWoNY9Vbjssgxky5aVwYM+oHmLjgCcOHGCwYOH06Xz4yGP5VLwxaBhfDFoGACv9ezB9u07A44odJJ67ceNGcINNxRiYbRXSRgxchw/fP91wJEG55uh3zJ+3Je88mqfoEMJiaioKEYO78/Qod8xZszE2PZmTZtQp3Y1qtdsEmB0/8zSlWuY9dNC5sxfzPETJzhy5Chtn36VzVu3U/zmGwCoVeVO2nR9GfCGz95//VkAjh49xrTZP5MtaxYADh85Srvur9LxsUcoUezGQPoThMaN6jJi5LjY74uX8CZqb968FYBvR//AU13bBhJbckq9H38vTMjTfefcXmAQMNnMfjCzLmZ2bZxd0uNVlyY45x4/W3LkP1Y/51xp51zpcCRHAM8934vrC5amcNHb+fcj7Zg58yeat+hInjiTD+vVu4fVa9aFJZ6gXXHF5QBcc01e6tevxdBh3wUcUegk9do3aNiS7NmzUaRIQQCqVa2UaPJ+ale4cIHYr+vVrcn69b+eY++UrX+/Pqxdt5H33u8X21azRmW6dm1L/ftbcOzY3wFG9890btOc6aO/YMqIz+n9UjfKlizOh/99jsNHjrBl2w4Afo5eSsHr8gOwb//B2CHm/l+PokHtagCcPHmSTs/9l3o176ZG5TuC6UwAzIz7G9Zh1MjxsW2//76LG28qQm5/3laVqhVZv25jUCGmSGY20Mx2m9mqOG25zGyqmW3w/8/pt5uZfWBmG81shZmVjHOf5v7+G8yseZz2Uma20r/PB2Zm54spLOdBcs61NLP3gHuAesDrZlbf33wSmALUNrPrnHO/hSOm/9WXgz8i9xW5MDOWL19Nu/Y9gg4pLEYO70+uy3Ny8uQpOnZ8jgMHLu0lvMnt9OnTtGnbjRHD+xET49i/bz+Ptn4q6LBC5qsvP+auSuXJnTsXWzYt4pVX36ZWrSoULVqImJgYtm7dkWp/9++oUIamjzRixco1LIqeAsALL/Ti3XdeJUOGDEya6FVSFyxYQvsnUvbPICoqLS93e4LOz/fC0hjZsmahZw+vSh69bCXv9R2CmVGqRDGe96vlk2bOZfHy1ew/eIgxk2YA8PoznbixSEG+GjWeL4Z+y569+7i/ZUfuvL0Ur3bvEFj/zmXQoA+4s9LtXH55Tn7ZMI/XXnuXffsO0KfPy+TOnYtvRw9kxYq13HdfMwAqVizH9u072bJlW+xj7Nq5m//+930mTxnBqZMn2bptB21adw2qS8kmzAswBgEfAUPitPUApjvneplZD//77kAtoIh/Kwd8CpQzs1zAS0BpvCulLDazcc65ff4+jwELgAl4+chEzsGCWIFiZhOBfXiJ0Ud4w3CDgTuAys65red7jKj0+VLmoKaIyD90bNuMoEMIVI4CNYMOIVBHjm45b9UjOd137b3Jdpwdu/X788ZuZtcD3zvnbvG/X4+XE+w0s6uBWc65G8ysr//10Lj7nbk559r47X2BWf5tpnPuRr/9obj7nU04JmkXMLNeZlbBzK4zs7uB4sCaM/v4w2rNgZ+BWQmG4ERERCTyXOWcOzPRdRdw5twR+YBtcfbb7redq317Eu3nFI4lB0eBosBI4Be8StHXwJtxd0qQJM1UkiQiIhKc5DxRpJm1NrNFcW6tLyYW5w13hXXkKORzkJxzfwD3n2XzIP92Zt/TwCOhjklERETOLTmX5zvn+gH9zrtjfH+Y2dVxhth2++07gGvi7Jffb9uBN8wWt32W354/if3PKTJOWiEiIiIpzTi8kSX8/8fGaW/mr2a7HTjgD8VNBmr4J53OCdQAJvvbDprZ7f7qtWZxHuuswrKKTURERFKWcF4ixMyG4lV/cpvZdrzVaL2AEWbWCvgNOHPisQlAbWAj3jSeluCdVsjMegLR/n6v+qcaAmiHN2KVEW/12jlXsIESJBEREUlCOFe5O+ceOsumqkns64D2Z3mcgcDAJNoXAYkucXYuGmITERERSUAVJBEREUkk0i81ogRJREREEkmpF5lNLhpiExEREUlAFSQRERFJJJyr2C5FSpBEREQkkSCu1Xop0RCbiIiISAKqIImIiEgiGmITERERSUCr2EREREQkHlWQREREJJGYCJ+krQRJREREEons9EhDbCIiIiKJqIIkIiIiiWgVm4iIiEgCkZ4gaYhNREREJAFVkERERCSRSL/UiBIkEZEUIut11YMOIVCHNk8OOoSIoiE2EREREYlHFSQRERFJJNIvNaIESURERBLRHCQRERGRBDQHSURERETiUQVJREREEtEQm4iIiEgCGmITERERkXhUQRIREZFEtMxfREREJIGYCJ+DpCE2ERERkQRUQRIREZFENMQmIiIikoCG2EREREQkHlWQREREJBENsYmIiIgkoCE2EREREYlHFSQRERFJRENsIiIiIgloiE1ERERE4lEFSURERBLREJuIiIhIAs7FBB1CoDTEdhGKFi3Eougpsbe9e9bRscOjQYcVNpHY//79+vD79uUsWzo9tq1hw3tZvmwGJ/7eRqmSxQOMLvw2/jKfpUumsSh6CvPnTQg6nJBK6rV/8YUu/LZ5UezfQK17qgQYYWh06NCKJUumsXjxVIYM+ZAMGTJw9913MG/eDyxYMJEZM0ZTsOB1ADRt2oht25ayYMFEFiyYSMuWDwYc/YU7ffo0jR7tTLserwEwf/FyGj/WhYatnqTpE8+wdftOAN78aAANWz1Jw1ZPUueRdpSv83C8xzl85ChVG7Xi9ff6AXDs7+O07dGTuk3bc1+LDrzbd0h4OybJRhWki/DLL79SukwNANKkScPWLYsZM3ZiwFGFTyT2f8iQEXzyyRd88cX7sW2rV6+jcZPH+PTjXgFGFpxq1Rvz11/7gg4j5JJ67QHe/6A/77zbN6CoQitv3qto374lt95alb//Ps5XX31CkyZ1efrpJ2jU6FHWr99I69ZNeeaZjjz22FMAjBo1ns6dXww48ov31ejvKXhdfg4fOQZAz3f78sHrz1DoumsYNmYCfb8cwevPdKL7E61i7/P1t9+zdsPmeI/z4cBvKFXi5nhtLR+oT9nb/sXJkydp1eVF5ixYzJ3lSoW+U8ksJsKH2MJSQTLPU2a2wcyOm9l2M3vD3/aimf3mt+8ysxSRbletUpFNm35j69YdQYcSiEjp/5y5C9i7b3+8tnXrNvLLL78GFJGES1KvfSSIiooiY8bLSJs2LZkyZWTnzj9wzpEtWxYAsmfPxs6dfwQc5f9m1+49zJ6/iIZ1qse2mcERP1k6dOQoV+TOleh+E6bPoXbVO2O/X71+I3/t3U+F0rfGtmW8LANlb/sXAOnSpeOmooX448+/QtWVkHLOJdstJQrXENt/gReAN4BiQGNgm5k1BLoC7YAiwL3AwjDF9D9p0uQ+hg0fE3QYgYn0/kcq5xwTJwxlwfyJPNrq30GHE4h2bVuyZPFU+vfrQ44c2YMOJ1n9/vsfvPtuPzZsmM+WLYs4ePAg06bNoW3b7owZM5iNGxfw8MP307v3J7H3qV+/NtHRk/nmm8/In//qAKO/cG9+NIAubZpjZrFtr3RrT9sePanaqBXjp8zi0YcbxrvP77t2s2Pnbsr5yU9MTAy9P/mCrm1bnPV5Dh46zI8/R1MuwobiU4uQJ0hmlgXoDPRwzg10zm10zs1zzn0CXAfsBKY457Y65xY55z46x2O1NrNFZrYoJuZIqEM/q3Tp0lH33hqMGv19YDEEKdL7H8nuursBZcvdw711H6Ft2xbcWbFc0CGF1Wd9h1D0xgqUKl2DXbt20/utlDe0dC45cmSnbt3q3HjjHRQoUIZMmTLx0EMN6NChFfXrN6dw4XIMGTKCt956AYAffpjGDTdUoEyZmsyYMYfPP38n4B6c36yfo8mVMzvFbigcr33IyPF82usFpo8aQP1aVXnr44Hxtk+cMZcad5Unbdq0AAwbM5FKt5ciz5W5k3yeU6dO83TPd/j3/XW4Jm+e0HQmxGJwyXZLicIxB+lmIAMwPYltI4FOwGYzmwxMAsY5fxPi8wAAEQdJREFU544n9UDOuX5AP4Co9PkC+4nfc8/dLF26kt279wQVQqAivf+R7PffdwHw559/MXbsRMqUuZU5cxcEHFX4xP2d/3zA14wdMzjAaJJflSoV2bJlG3v27AVg7NhJlC9fmuLFbyY6ehngzTkaN+5LAPbu/f8hyIEDh/L668+EP+iLtHTVOmb9FM2c+Ys5fuIkR44epW2Pnmzeup3iNxcFoNbdFWnz9Cvx7jdxxhyee7JN7PfL16xn8Yo1DBszkaPH/ubkqVNkyngZnds0A+DlPp9wbf6radq4Xvg6l8xS6tBYcgl0FZtzbhtwA9AGOAj0ARabWeYg4zqfBx+oH9HDS5He/0iVKVNGsmTJHPt19Wp3sXr1+oCjCq88ea6M/br+fbVSXf+3bdtB2bIlyZjxMgDuvvsO1q7dQLZsWSlcuAAAVaveybp1G4D4P497763OunUbwx/0RercuinTRw1gyvD+9H7xKcreVpwPX3uWw4ePsmWbN6fy50XLKHhd/tj7bPptOwcPHebWYjfEtr35fBemjficKcP707VtC+rVuDs2Ofrg8685fOQIPeJM8JaUJxwVpLXAcaAqsCHhRufc38APwA9m1gvYBdwBTAlDbBctU6aMVKtaibbtugcdSiAirf9fffkxd1UqT+7cudiyaRGvvPo2e/ft5/13X+OKK3IxbuwQli9fTe17U/98nKuuuoJRIwcAEBWVlmHDxjB5yqxggwqhpF77u+6qQIkSN+Oc47fftqe6v4Po6GV8990E5s+fwKlTp1m+fDUDBnzDjh07GTasLzExMezff4A2bboB0L59S+rUqc6pU6fYt29/7Mq2lCYqKi0vd2tP5xffxNKkIVuWzPTs3iF2+8QZc6hV5c54c5bOZtfuPfT7aiQFrs1P48e6APBQgzo0urf6ee556Yn0S41YOEpoZvYm0BroAswGLgdKAcfwkrQFwGHgAeBV4Abn3OakH80T5BCbiEgQotKkDTqEQB3aPDnoEAKV7uqbzp+hJaM8OW5KtuPsrv1rwxp7cgjXeZCeAfbhrWTLD/wBDAGige7A20A6YA1w//mSIxEREZFQCkuC5LzzlffybwlpMouIiMglJtInaetM2iIiIpJISl2en1x0LTYRERGRBFRBEhERkUQ0xCYiIiKSQKQv89cQm4iIiEgCqiCJiIhIIhpiExEREUlAq9hEREREJB5VkERERCQRDbGJiIiIJKBVbCIiIiISjypIIiIikoiL8EnaSpBEREQkEQ2xiYiIiEg8qiCJiIhIIlrFJiIiIpJApM9B0hCbiIiISAKqIImIiEgiGmITERERSSDSEyQNsYmIiIgkoAqSiIiIJBLZ9SOwSC+h/VNm1to51y/oOIIQyX0H9V/9j9z+R3LfQf2PNBpi++daBx1AgCK576D+q/+RK5L7Dup/RFGCJCIiIpKAEiQRERGRBJQg/XORPA4dyX0H9V/9j1yR3HdQ/yOKJmmLiIiIJKAKkoiIiEgCSpBEREREElCCJEkyMws6BhERkaAoQRLM5399M4DT5LSIYmZ6L4hQcf72rww6lqDpg6HEpTfFs4iEPxQzu9LMzPnMrC4w18zKBx1bEMwsS9AxhMuZhMjMMgM452LM7JZgo5JwOvMe5//t3w2MMLNMkZgsm1khM0unD4YSV8T9IZyLmeUxs+sh9VdQzKwv8B5wmf/9tUAT4Bnn3LwgYwuCmdUEvjCzEkHHEg5+QnQtMMDMbjez+4EVZnZr0LGFU5zqyVVmVtjMcprZmb+JVPn+aGZdzay+nxid6eOtwAHn3NEgYwuCmdUDvgU6pNbXXP4Z/TL4zKwB8BPwo5ktNLMyZpYqL+ZrZg8A9YG3nHPHzKwk8DJQAPjR3yfVV9DOMLOGwChgNf+fMKba/ptZOzO7HbgCKAR8BHwNNHPOLYuUg8SZ6qmZ1Qem+LfpwIdmVtA5FxNshMnPzHIAZYBhZlYrTh+vBo6DlzwHFV+4+a/9cLzzG30fSX2X84uIN8LzMbPiwCdAf6ALcAwYA1RLpUnStcBf/sHwHmAQUAEoDVwPqb+Cdob/2n8EdHHOveycW+Bvui7AsELGzK4AqgN7nHOL8Q4MJYFfgd8gtrqUahNE8KpDfnJUFfgKGAj8CxgLNAfuDDK+UHHO7QeeBQbjDanVibP5GICZZQgitnAzs7zAC0A359zHwBYzy25mD/rVxNT43i8XIeITJDMrhVc56e+c6+WcG+2cuwtYBXxB6kySZgNRZjYD+AHoDDyJd5DsaGZlgwwuHOIkAEWB3c65/maW2cxamtlUYJWZ9TWz9AGGmeycc38CDzrnNppZGaAsXkKwB3jWzGr5+7nUliSZWXMzew5ik8B0eMPK/Z1z7wNZgJb+94P9+2QMLOBkdqYy6Jz7FXgLGIpXSboT2AL85fc385n9zaxoQOGGlP93fRjIiZcYpcVLHH8A+gKLgTuCi1AuBRGbIPkLtzLgvUl8BxSKu5rLOVcTWIH3Cbt2akqS/CrJDKAyEO2cm+6cmwT0BK4EnjSz0gGGGHJxKmTbgOxmNhhveOU+vOT4EeAxoFowEYaOc+64Pzn7JaAYMAd4HMiKlyDX9PdzZnZXcJEmH//AXw+ob2YdAZxzJ4E8QLRfWVsCTAae8O9TF6iaioYcHYD/t30lXuV0GDAReBHv57MGWGtmG/GSph/M7PJAog0RM6uB1/dyeB8W3wN2AiWA0c657HjD7c0CC1IuCRF/qZEzE1WBIkBN59x6v/we42+fD+QASjnnjgQYarLxDxbf470BVgCWOuce9rc9jDfMuAb4NDVO2DazCnjzMHICm4FMQFW8ZGkAsNY5d9rMZgM9nXNTAws2hMzsRuAdIAPQHu8A+jneJ+uRQH68uWn5nHM7Awoz2Zi3jP0NvKrhOOdcbzMbBNyMNx9rAtDJOXfK/xvpD6wDejnnTgUUdrKIM9+qAd5r/AHwKV7VrBPwKF5/38GrqKcFYoDNzrnfgok6+fmLEb4CXsN7vU/gTS3IhPdh+ZBfXRwOrHLO9QwsWAlcRCZIcQ6QVwBz8ZKB4UB6oJFzbnOCJOla59zWwAIOATPLhDfn4DHgKWBxnCTpQbw3kJlAB+fc34EFmsz8N8iBwHi8ialZgE1Ac7+icGa/nnifIO9wzm0PItZwMLMiwMd4B8T2wGm8g2Rhv+0hf65SiuVXhaOccyfNO8/Xy3hJ0rvAArwJ6nmcc/ni7P8aXhWxmnNuQyCBJzO/cjIaLyEa7Zw74LffCLTBS5LqOedmBhdl6Pi/65OAd/w5R0ntkxvv59MWqOicWxfGEOUSE3EJUoIDZB7gKrzS+gt45fXDQBPn3Ja4SVJqZWZZgQeBrsRPkhr5328OMr7k5B8IJuJVBPr63y/Em3PylL9PZaAFUBuvorg0oHDDJkGS1AFYjzdJ/Yhz7o8gY0sOcaonTYCHgFzAbcABvKGWHcB/gf3ARv9ulYDqqeX194cJPwVOOuee8D8g3YSXBO4CduNV0p7C+71PdVVTM6uCN7+ohv8h2PCOgTFxtjcH7gbuSy2vvfxzqWVs/YL4B8Q+QHfnXFO8uQbX463o2gbUwDtITPerRqk6OQJwzh3CKy2/DZQws/F++6jUlBz58gN7/eSoAF5CPDROclQT76BxAqgcKW+QfoWkPd4y7yFAWefcptSQHEHsXKqyeIsuvsdLgIsBy4G6eB+SqgJTgYPAMqB8Knv90+AlvXn9KtqHeMONNfDmHtXCO9XFJ3hDzalRFiAjELv4IE5yVAlvFeMM4O5U9trLPxRRCRKJD5BTiHOABC4H7sf7RJU2oBjDzjl3GC9J+gzIY2b5Ag4pVDICe82sEN75niYB7QD8A2hFvHNhPemcWxNYlAHwk6QuePPSdgQbTUgUxzvwD3fObfY/ELXGqxh3Bio4555yzrVwzr2aWobVzvDnUPXCq47MwZuQ3885Vwz4BrgGb+VWh1Q8rLQSr3rYGhKdyuQ+IC8wyl/lJ0KqWZl1geIeIGfiDbfEPUA+iFdhuiulT8q8WM65w2Y2EPjSP1dKarQGb+nuBuBj51yHONseBm7Bm58QcWcTBnDOrTOzh51zJ4KOJQSO473fZQMOm3dZid/9FW0LgRfNLLtz7oMzQ3KBRhsCzrlZ5p0pPo9zbmGc0zhcj1c5u8z/sJQq+cNq7YG+/qrkL/Hm3DUDWuFVDVPFQhxJHpGWIF3IAfJYpCVHZ6T2Nwfn3K9m9h+8VTx/+0Ou6YGm+CcHdM7tCzLGoKXS5AhgHl6FoD3wXJwJ+ZfhVU424Z0cNlWfJNVfbHJmwUlJ884i3xrvdz/VJkdxDAFO4Q0xPoKXGJ7CG1ZbG2RgcumJqATpAg+Qe4OMUUJuNN5r/iHehN2DwN94b5CrggxMQsd5J8ZsjXftubR47wH7gMZ4E5S7nlnVFQnM7AagO95qvkrOuRUBhxQWzrnTwJdmNgvvMjtHgS3Oud2BBiaXpEhcxZYO78D4IXCI/z9A/sc5tyzI2CR8zCw/3tDCIWCHc25PsBFJqPlDSg/infx1L97wSja8VU1Lgowt3PyfRTG8OZm/Bx2PyKUo4hKkM3SAFIlMZnY93oqljMBC59yWIOMRkUtTxCZIIiIiImcTacv8RURERM5LCZKIiIhIAkqQRERERBJQgiQiIiKSgBIkERERkQSUIImIiIgkoARJ5BJnZqfNbJmZrTKzkWaW6X94rMpm9r3/dT0z63GOfXOYWbt/8Bwvm1nXC21PsM8gM2t0Ec91vZnpDOgikuyUIIlc+o455251zt0CnAAej7vRPBf9t+ycG+ec63WOXXLgX8xZRCTSKEESSVnmAIX9ysl6MxsCrAKuMbMaZjbPzJb4laYsAGZ2j5mtM7MlwP1nHsjMWpjZR/7XV5nZd2a23L9VAHoBhfzqVW9/v25mFm1mK8zslTiP9ZyZ/WJmc4EbztcJM3vMf5zlZjY6QVWsmpkt8h/vXn//tGbWO85zt/lff5AiIueiBEkkhTCzKKAWsNJvKgJ84pwrBhwBngeqOedKAouALmZ2GdAfqAuUAvKc5eE/AH50zpUASgKrgR7Ar371qpuZ1fCfsyxwK1DKzCqZWSm8a5zdCtQGylxAd751zpXxn28t0CrOtuv956gDfOb3oRVwwDlXxn/8x8yswAU8j4jIPxIVdAAicl4ZzezMhZTnAAOAvMBvzrn5fvvtwM3AT951SEkPzANuBDY75zYAmNlXQOsknqMK0Axir3h+wMxyJtinhn9b6n+fBS9hygp855w76j/HuAvo0y1m9hreMF4WYHKcbSOcczHABjPb5PehBlA8zvyk7P5z/3IBzyUictGUIIlc+o45526N2+AnQUfiNgFTnXMPJdgv3v3+Rwa84Zzrm+A5nvwHjzUIqO+cW25mLYDKcbYlvECk85+7g3MubiJ15sKzIiLJTkNsIqnDfOAOMysMYGaZzawosA643swK+fs9dJb7Twfa+vdNa2bZgUN41aEzJgP/iTO3KZ+ZXQnMBuqbWUYzy4o3nHc+WYGdZpYO+HeCbY3NLI0fc0Fgvf/cbf39MbOiZpb5Ap5HROQfUQVJJBVwzv3pV2KGmlkGv/l559wvZtYa+MHMjuIN0WVN4iE6Af3MrBVwGmjrnJtnZj/5y+gn+vOQbgLm+RWsw8AjzrklZjYcWA7sBqIvIOQXgAXAn/7/cWPaCiwEsgGPO+f+NrPP8eYmLTHvyf8E6l/YT0dE5OKZcwmr2SIiIiKRTUNsIiIiIgkoQRIRERFJQAmSiIiISAJKkEREREQSUIIkIiIikoASJBEREZEElCCJiIiIJKAESURERCSB/wNEVQCgeou1VgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x504 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot confusion matrix \n",
    "from sklearn.metrics import confusion_matrix\n",
    "from support import print_confusion_matrix\n",
    "\n",
    "cnf_matrix = confusion_matrix(np.argmax(Y_pred,axis=1), np.argmax(Y_test,axis=1))\n",
    "_ = print_confusion_matrix(cnf_matrix, LABELS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "         en       0.97      0.97      0.97     49999\n",
      "         fr       0.98      0.98      0.98     49917\n",
      "         es       0.98      0.97      0.97     49889\n",
      "         it       0.97      0.97      0.97     49977\n",
      "         de       0.99      0.99      0.99     50111\n",
      "         sk       0.97      0.98      0.98     50047\n",
      "         cs       0.98      0.97      0.98     50060\n",
      "\n",
      "avg / total       0.98      0.98      0.98    350000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Classification Report\n",
    "print(classification_report(Y_test, Y_pred, target_names=LABELS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## It's your turn\n",
    "\n",
    "Test out the model you just trained. Run the code Cell below and type your text in the widget, Have fun!🎉\n",
    "\n",
    "Here are some inspirations:\n",
    "\n",
    "##### EN - Frank Baum, The Wonderful Wizard of Oz, Project Gutenberg, public domain\n",
    "You are welcome, most noble Sorceress, to the land of the Munchkins. We are so grateful to you for having killed the Wicked Witch of the East, and for setting our people free from bondage.\n",
    "\n",
    "##### DE - Johann Wolfgang von Goethe, Faust: Der Tragödie erster Teil, Project Gutenberg, public domain\n",
    "Habe nun, ach! Philosophie, Juristerei und Medizin, Und leider auch Theologie  Durchaus studiert, mit heißem Bemühn. Da steh ich nun, ich armer Tor! Und bin so klug als wie zuvor.\n",
    "\n",
    "##### FR - Pierre Benoît, L'Atlantide, \n",
    "Voilà cinq mois que j'en faisais fonction, et, ma foi, je supportais bien cette responsabilité et goûtais fort cette indépendance. Je puis même affirmer, sans me flatter.\n",
    "\n",
    "##### IT - Alberto Boccardi, Il peccato di Loreta, Project Gutenberg, public domain\n",
    "Giovanni Sant'Angelo, che negli anni passati a Padova in mezzo alla baraonda tanto gioconda degli studenti, aveva appreso ad amare con foga di giovane qualche alto ideale, tornato in famiglia dovette fare uno sforzo.\n",
    "\n",
    "##### ES - Fernando Callejo Ferrer, Música y Músicos Portorriqueños, Project Gutenberg, public domain\n",
    "Dedicada esta sección a la reseña de los compositores nativos y obras que han producido, con ligeros comentarios propios a cada uno, parécenos oportuno dar ligeras noticias sobre el origen de la composición.\n",
    "\n",
    "##### CS - František Omelka, Blesky nad Beskydami, Project Gutenberg, public domain\n",
    "A Slávek, jsa povzbuzen, se ptal a otec odpovídal. Přestože byl prostým venkovským listonošem, nepřivedla jej žádná synova otázka do rozpaků. Od mládí se zajímal o dějepis a literaturu.\n",
    "\n",
    "##### SK - Janko Matúška, Nad Tatrou sa blýska,  national anthem of Slovakia, https://en.wikipedia.org/wiki/Nad_Tatrou_sa_blýska\n",
    "Nad Tatrou sa blýska Hromy divo bijú Zastavme ich, bratia Veď sa ony stratia Slováci ožijú To Slovensko naše Posiaľ tvrdo spalo Ale blesky hromu Vzbudzujú ho k tomu Aby sa prebralo.\n",
    "\n",
    "Can you do better? Play around with the model hyperparameters!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51a58e6a7b0f497aa907a3df96d293fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Textarea(value='', description='TEXT', placeholder='Type the text to identify here'), Bu…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# and now we will have some fun. Seeing is believing!\n",
    "# We will take some texts and try to predict the text's language using our trained neural network.\n",
    "\n",
    "from ipywidgets import interact_manual\n",
    "from ipywidgets import widgets\n",
    "from support import clean_text\n",
    "\n",
    "\n",
    "def get_prediction(TEXT):\n",
    "    if len(TEXT) < MAX_LEN:\n",
    "        print(\"Text has to be at least {} chars long, but it is {}/{}\".format(MAX_LEN, len(TEXT), MAX_LEN))\n",
    "        return(-1)\n",
    "    # Data cleaning\n",
    "    cleaned_text = clean_text(TEXT)\n",
    "    \n",
    "    # Get the MAX_LEN char\n",
    "    input_row = get_input_row(cleaned_text, 0, MAX_LEN, alphabet)\n",
    "    \n",
    "    # Data preprocessing (Standardization)\n",
    "    test_array = standard_scaler.transform([input_row])\n",
    "    \n",
    "    raw_score = model.predict(test_array)\n",
    "    pred_idx= np.argmax(raw_score, axis=1)[0]\n",
    "    score = raw_score[0][pred_idx]*100\n",
    "    \n",
    "    # Prediction\n",
    "    prediction = LABELS[model.predict_classes(test_array)[0]]\n",
    "    print('TEXT:', TEXT, '\\nPREDICTION:', prediction.upper(), '\\nSCORE:', score)\n",
    "\n",
    "interact_manual(get_prediction, TEXT=widgets.Textarea(placeholder='Type the text to identify here'));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save your model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving Model Weight\n",
    "model.save_weights('models/lang_identification_weights.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### That's all folks - don't forget to shutdown your workspace once you're done 🙂"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}