work2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0345cd70",
   "metadata": {
    "papermill": {
     "duration": 0.010055,
     "end_time": "2022-07-05T00:56:35.056496",
     "exception": false,
     "start_time": "2022-07-05T00:56:35.046441",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 作业二：实战CNN —— CIFAR-10图像分类"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a33c787",
   "metadata": {
    "papermill": {
     "duration": 0.00847,
     "end_time": "2022-07-05T00:56:35.074174",
     "exception": false,
     "start_time": "2022-07-05T00:56:35.065704",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "CIFAR-10 是由 Hinton 的学生 Alex Krizhevsky 和 Ilya Sutskever 整理的一个用于识别普适物体的小型数据集。\n",
    "\n",
    "一共包含 10 个类别的 RGB 彩色图片：飞机（ airlane ）、汽车（ automobile ）、鸟类（ bird ）、猫（ cat ）、鹿（ deer ）、狗（ dog ）、蛙类（ frog ）、马（ horse ）、船（ ship ）和卡车（ truck ）。图片的尺寸为 32×32 ，数据集中一共有 50000 张训练图片和 10000 张测试图片。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76305578",
   "metadata": {},
   "source": [
    "本次作业中，通过对`torch.nn`中各模块的手写实现（方式与官方**不同**），分别实现了LeNet、ResNet9、ResNet18、ResNet50、WRN50_2**五种**骨干网络提取图像的特征，以解决CIFAR-10数据集的分类问题，准确率达到了**97.7%**。\n",
    "在代码编写过程中，采用面向对象的思想将各个功能模块化，整体结构如下：\n",
    "* `Datamanager`类，负责数据加载和预处理；\n",
    "* `BaseModule`类，是所有模型的模版类；\n",
    "* `Sequential`类，继承自`BaseModule`，实现与`nn.Sequential`类相同的功能；\n",
    "* `Layer`类，继承自`BaseModule`，是所有网络层的基类，派生有`ReLU`、`Dropout`、`Flatten`、`Linear`、`Conv2d`、`MaxPool2d`、`AdaptiveAvg2d`、`BatchNorm2d`、`ZeroPad2d`网络层；\n",
    "* `BasicBlock`、`Bottleneck`、`ResNet9Block`类，构成各模型的基本模块；\n",
    "* `CrossEntropyLoss`类，实现了`nn.CrossEntropyLoss`的基本功能；\n",
    "* `Optimizer`类，为优化器的基类，派生有随机梯度下降`SGD`（包含动量`momentum`和L2权值衰减`weight_decay`）、`Adam`（包含L2权值衰减）；\n",
    "* `OneCycleLR`类，单周期学习率调节策略；\n",
    "* `LeNet`、`ResNet9`、`ResNet18`、`ResNet50`，为模型的主体；\n",
    "* `Trainer`类，将模型训练、测试、模型存储、训练日志存储、训练过程的准确率和损失值可视化集成在一个类中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06719eb7",
   "metadata": {
    "papermill": {
     "duration": 0.008394,
     "end_time": "2022-07-05T00:56:35.091377",
     "exception": false,
     "start_time": "2022-07-05T00:56:35.082983",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 导入相关包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ff3bbea4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:36.612138Z",
     "iopub.status.busy": "2022-07-05T00:56:36.611795Z",
     "iopub.status.idle": "2022-07-05T00:56:38.872825Z",
     "shell.execute_reply": "2022-07-05T00:56:38.871833Z"
    },
    "papermill": {
     "duration": 2.27449,
     "end_time": "2022-07-05T00:56:38.875907",
     "exception": false,
     "start_time": "2022-07-05T00:56:36.601417",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import wget\n",
    "import math\n",
    "import copy\n",
    "import torch\n",
    "import datetime\n",
    "import numpy as np\n",
    "from torch import nn\n",
    "from PIL import Image\n",
    "import torchvision as tv\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "import matplotlib.pyplot as plt\n",
    "from torch.utils.data import DataLoader\n",
    "import torchvision.transforms as transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b60e4e7",
   "metadata": {
    "papermill": {
     "duration": 0.008595,
     "end_time": "2022-07-05T00:56:38.894837",
     "exception": false,
     "start_time": "2022-07-05T00:56:38.886242",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 文件夹结构搭建 & 下载和缓存预训练模型\n",
    "\n",
    "为了减少训练时间、提高泛化能力，本次作业中引入了pytorch官方的ResNet18、ResNet50、WRN50_2的预训练模型，对本次作业的模型进行初始化。以下`setup`函数实现了文件夹结构搭建、预训练模型下载、预训练模型的调整的功能。CIFAR-10数据集的下载将使用torchvision官方代码实现。完成后，文件夹结构如下：\n",
    "```\n",
    "project   \n",
    "│\n",
    "└───PretrainedModel（存储预训练模型）\n",
    "│       │   ResNet18.pth \n",
    "│       │   ResNet50.pth  \n",
    "│       │   WRN50_2.pth\n",
    "│\n",
    "└───Model（存储训练好的模型参数）\n",
    "│   \n",
    "└───TrainingLog（存储训练日志）\n",
    "│   \n",
    "└───Figure（存储训练过程、结果的可视化文件）\n",
    "    \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8ac66b1f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:38.914264Z",
     "iopub.status.busy": "2022-07-05T00:56:38.913748Z",
     "iopub.status.idle": "2022-07-05T00:56:38.930901Z",
     "shell.execute_reply": "2022-07-05T00:56:38.929991Z"
    },
    "papermill": {
     "duration": 0.029339,
     "end_time": "2022-07-05T00:56:38.932955",
     "exception": false,
     "start_time": "2022-07-05T00:56:38.903616",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def setup():\n",
    "    resnet18_info = {'path': './PretrainedModel/ResNet18.pth',\n",
    "                     'url': 'https://download.pytorch.org/models/resnet18-5c106cde.pth',\n",
    "                     'original_path': './PretrainedModel/resnet18-5c106cde.pth'}\n",
    "    resnet50_info = {'path': './PretrainedModel/ResNet50.pth',\n",
    "                     'url': 'https://download.pytorch.org/models/resnet50-19c8e357.pth',\n",
    "                     'original_path': './PretrainedModel/resnet50-19c8e357.pth'}\n",
    "    wrn50_2_info = {'path': './PretrainedModel/WRN50_2.pth',\n",
    "                    'url': 'https://download.pytorch.org/models/wide_resnet50_2-95faca4d.pth',\n",
    "                    'original_path': './PretrainedModel/wide_resnet50_2-95faca4d.pth'}\n",
    "    model_list = [resnet18_info, resnet50_info, wrn50_2_info]\n",
    "    folders = ['./PretrainedModel', './Model', './Figure', './TrainingLog']\n",
    "\n",
    "    for folder in folders:\n",
    "        if not os.path.exists(folder):\n",
    "            os.mkdir(folder)\n",
    "\n",
    "    for model in model_list:\n",
    "        if not os.path.exists(model['path']):\n",
    "            print('Downloading model from', model['url'], '.')\n",
    "            wget.download(model['url'], model['original_path'])\n",
    "            state_dict = torch.load(model['original_path'])\n",
    "            key_list = []\n",
    "            for key in state_dict:\n",
    "                info = key.split('.')\n",
    "                if info[-1] == 'num_batches_tracked':\n",
    "                    key_list.append(key)\n",
    "            for key in key_list:\n",
    "                del state_dict[key]\n",
    "\n",
    "            del state_dict['fc.weight'], state_dict['fc.bias']\n",
    "            if model['path'] == './PretrainedModel/ResNet18.pth':\n",
    "                fc1 = nn.Linear(in_features=512, out_features=128)\n",
    "                fc2 = nn.Linear(in_features=128, out_features=10)\n",
    "                fc1_params = list(fc1.parameters())\n",
    "                fc2_params = list(fc2.parameters())\n",
    "                state_dict['fc.0.weight'] = fc1_params[0]\n",
    "                state_dict['fc.0.bias'] = fc1_params[1]\n",
    "                state_dict['fc.2.weight'] = fc2_params[0]\n",
    "                state_dict['fc.2.bias'] = fc2_params[1]\n",
    "            elif model['path'] == './PretrainedModel/ResNet50.pth' or model['path'] == './PretrainedModel/WRN50_2.pth':\n",
    "                fc1 = nn.Linear(in_features=2048, out_features=512)\n",
    "                fc2 = nn.Linear(in_features=512, out_features=128)\n",
    "                fc3 = nn.Linear(in_features=128, out_features=10)\n",
    "                fc1_params = list(fc1.parameters())\n",
    "                fc2_params = list(fc2.parameters())\n",
    "                fc3_params = list(fc3.parameters())\n",
    "                state_dict['fc.0.weight'] = fc1_params[0]\n",
    "                state_dict['fc.0.bias'] = fc1_params[1]\n",
    "                state_dict['fc.2.weight'] = fc2_params[0]\n",
    "                state_dict['fc.2.bias'] = fc2_params[1]\n",
    "                state_dict['fc.4.weight'] = fc3_params[0]\n",
    "                state_dict['fc.4.bias'] = fc3_params[1]\n",
    "\n",
    "            torch.save(state_dict, model['path'])\n",
    "            os.remove(model['original_path'])\n",
    "            print('')\n",
    "    print('Pretrained models have been downloaded and converted to the correct format.')\n",
    "\n",
    "setup()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b105b742",
   "metadata": {
    "papermill": {
     "duration": 0.008723,
     "end_time": "2022-07-05T00:56:38.950758",
     "exception": false,
     "start_time": "2022-07-05T00:56:38.942035",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 训练"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1ed7418",
   "metadata": {
    "papermill": {
     "duration": 0.009403,
     "end_time": "2022-07-05T00:56:38.985921",
     "exception": false,
     "start_time": "2022-07-05T00:56:38.976518",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 1.模型配置类：Config\n",
    "包括模型各项必要的配置，包括与训练相关的公有属性、选择相应的预训练模型路径等，并实现了对结果的保存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c1938c5f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.005620Z",
     "iopub.status.busy": "2022-07-05T00:56:39.005321Z",
     "iopub.status.idle": "2022-07-05T00:56:39.017565Z",
     "shell.execute_reply": "2022-07-05T00:56:39.016676Z"
    },
    "papermill": {
     "duration": 0.024455,
     "end_time": "2022-07-05T00:56:39.019727",
     "exception": false,
     "start_time": "2022-07-05T00:56:38.995272",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Config:\n",
    "    def __init__(self, name='WRN50_2', mode='Train'):\n",
    "        self.start_time = time.time()  # 记录代码开始运行的时间\n",
    "        self.model_name = name         # 模型名称，可以选择'LeNet'、'ResNet9'、'ResNet18'、'ResNet50'、`WRN50_2`\n",
    "        self.mode = mode               # 当前模式，可从'Train'和'Test'中选择\n",
    "        \n",
    "        # 与训练相关的公有属性\n",
    "        self.class_num = 10            # 类别数量\n",
    "        self.pretrain = True           # 是否使用预训练模型\n",
    "        self.epoch_num = 15            # 训练的轮次数量\n",
    "        self.batch_size = 64           # 一个批次数据包含的样本数\n",
    "        self.lr = 3e-3                 # 优化器的学习率\n",
    "        self.momentum = 0.9            # SGD优化器的momentum参数\n",
    "        self.weight_decay = 1e-4       # SGD和Adam优化器的weight_decay参数\n",
    "        self.train_batch_num = int(50000 / self.batch_size)    # 训练集的数据批次数量\n",
    "        \n",
    "        # 根据模型种类配置width属性，代表ResNet的Bottleneck的宽度倍率（1代表标准宽度）\n",
    "        # 选择相应的预训练模型路径\n",
    "        if self.model_name == 'WRN50_2':\n",
    "            self.width = 2\n",
    "            self.pretrain_model_path = './PretrainedModel/WRN50_2.pth'\n",
    "        elif self.model_name == 'ResNet50':\n",
    "            self.width = 1\n",
    "            self.pretrain_model_path = './PretrainedModel/ResNet50.pth'\n",
    "        elif self.model_name == 'ResNet18':\n",
    "            self.pretrain_model_path = './PretrainedModel/ResNet18.pth' \n",
    "        \n",
    "        # 选择使用的设备，当存在cuda处理器时优先使用cuda，否则使用cpu\n",
    "        self.device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
    "        self.model_path='./Model/'     # 测试模型的路径（仅在测试模式下需要）\n",
    "    \n",
    "    # 获得网络模型\n",
    "    def get_network(self):\n",
    "        if self.model_name == 'ResNet9':\n",
    "            return ResNet9(self)\n",
    "        elif self.model_name == 'ResNet18':\n",
    "            return ResNet18(self)\n",
    "        elif self.model_name == 'ResNet50' or self.model_name == 'WRN50_2':\n",
    "            return ResNet50(self)\n",
    "        else:\n",
    "            return LeNet(self)\n",
    "        \n",
    "    # 保存训练结果\n",
    "    def get_training_info(self):\n",
    "        training_info = []\n",
    "        for key in self.__dict__:\n",
    "            if key == 'start_time':\n",
    "                continue\n",
    "            info = key + ' : ' + str(self.__dict__[key])\n",
    "            training_info.append(info)\n",
    "        return training_info"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bcbc768",
   "metadata": {
    "papermill": {
     "duration": 0.008858,
     "end_time": "2022-07-05T00:56:39.037360",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.028502",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 2.数据预处理类：DataManager\n",
    "首先，导入```CIFAR10数据集```，根据不同网络模型的需求，对数据进行重塑宽高、翻转、规范化等预处理后，载入训练集和测试集的迭代器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "be0c9c69",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.055784Z",
     "iopub.status.busy": "2022-07-05T00:56:39.055511Z",
     "iopub.status.idle": "2022-07-05T00:56:39.066955Z",
     "shell.execute_reply": "2022-07-05T00:56:39.065843Z"
    },
    "papermill": {
     "duration": 0.022928,
     "end_time": "2022-07-05T00:56:39.068975",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.046047",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class DataManager:\n",
    "    def __init__(self, config):\n",
    "        # 根据网络模型选取不同的图像预处理策略，包含重塑宽高、翻转、张量化、规范化\n",
    "        if config.model_name == 'ResNet50' or config.model_name == 'WRN50_2':\n",
    "            transform = transforms.Compose(\n",
    "                [\n",
    "                    transforms.Resize((224, 224)),\n",
    "                    transforms.RandomHorizontalFlip(p=0.5),\n",
    "                    transforms.ToTensor(),\n",
    "                    transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))\n",
    "                ]\n",
    "            )\n",
    "        elif config.model_name == 'ResNet18':\n",
    "            transform = transforms.Compose(\n",
    "                [\n",
    "                    transforms.Resize((64, 64)),\n",
    "                    transforms.RandomHorizontalFlip(p=0.5),\n",
    "                    transforms.ToTensor(),\n",
    "                    transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))\n",
    "                ]\n",
    "            )\n",
    "        else:\n",
    "            transform = transforms.Compose(\n",
    "                [\n",
    "                    transforms.ToTensor(),\n",
    "                    transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))\n",
    "                ]\n",
    "            )\n",
    "        \n",
    "        # 加载数据集，生成DataLoader\n",
    "        training_set = tv.datasets.CIFAR10(root='./', train=True, download=True, transform=transform)\n",
    "        self.train_dataloader = DataLoader(dataset=training_set, batch_size=config.batch_size, shuffle=True, num_workers=2)\n",
    "        testing_set = tv.datasets.CIFAR10(root='./', train=False, download=True, transform=transform)\n",
    "        self.test_dataloader = DataLoader(dataset=testing_set, batch_size=config.batch_size, shuffle=False, num_workers=2)\n",
    "\n",
    "    def get_train_dataloader(self):\n",
    "        return self.train_dataloader\n",
    "\n",
    "    def get_test_dataloader(self):\n",
    "        return self.test_dataloader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8716a400",
   "metadata": {
    "papermill": {
     "duration": 0.008753,
     "end_time": "2022-07-05T00:56:39.086443",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.077690",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 3.神经网络组成模块\n",
    "在本节中，手写完成了以下神经网络组成模块（实现方式与官方**不同**）：\n",
    "- 神经网络基类：`BaseModule`，是所有模型的模版类，实现了`nn.Module`类的部分功能；\n",
    "- 神经网络模块序列：`Sequential`，实现与`nn.Sequential`类相同的功能（继承自`BaseModule`）；\n",
    "- 神经网络层：网络层基类`Layer`、激活函数`ReLU`、线性层`Linear`、正则化层`Dropout`等等；\n",
    "- 损失函数：交叉熵损失函数类`CrossEntropyLoss`，实现了`nn.CrossEntropyLoss`的基本功能；\n",
    "- 优化器：优化器基类`Optimizer`、自适应矩估计`Adam`优化器（包含L2权值衰减）和随机梯度下降`SGD`优化器（包含动量`momentum`和L2权值衰减`weight_decay`）；\n",
    "- 学习率调节器：`OneCycleLR`（单周期学习率调节策略）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38a84f0a",
   "metadata": {
    "papermill": {
     "duration": 0.009105,
     "end_time": "2022-07-05T00:56:39.104180",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.095075",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 3.1 神经网络基类：BaseModule"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cd8adcbc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.123527Z",
     "iopub.status.busy": "2022-07-05T00:56:39.123268Z",
     "iopub.status.idle": "2022-07-05T00:56:39.150377Z",
     "shell.execute_reply": "2022-07-05T00:56:39.149435Z"
    },
    "papermill": {
     "duration": 0.039355,
     "end_time": "2022-07-05T00:56:39.152265",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.112910",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class BaseModule:\n",
    "    def __init__(self, config=None):\n",
    "        self._parameters = None       # 网络模块中所有张量组成的字典\n",
    "        self.loss = None              # 网络的损失函数\n",
    "        self.optimizer = None         # 网络的优化器\n",
    "        self.scheduler = None         # 网络的学习率调节器\n",
    "        \n",
    "        # 若传入config（顶层网络），则获取网络所处的设备和预训练信息\n",
    "        if config is not None:\n",
    "            self.device = config.device\n",
    "            self.pretrain = config.pretrain\n",
    "            if self.pretrain:\n",
    "                self.pretrain_model_path = config.pretrain_model_path\n",
    "        else:\n",
    "            self.device = None\n",
    "            self.pretrain = False\n",
    "\n",
    "    # 网络模块的前向传播\n",
    "    def forward(self, data) -> torch.Tensor:\n",
    "        raise NotImplementedError\n",
    "\n",
    "    # 顶层网络一个batch的训练过程，返回该batch的loss之和\n",
    "    def step(self, data) -> float:\n",
    "        scores = self.forward(data[0])\n",
    "        loss = self.loss(scores, data[1])\n",
    "        loss.backward()\n",
    "        self.optimizer.step()\n",
    "        self.optimizer.zero_grad()\n",
    "        if self.scheduler is not None:\n",
    "            self.scheduler.step()\n",
    "        return loss.data.tolist()\n",
    "\n",
    "    # 顶层网络一个batch的测试过程，返回正确预测数和预测总数\n",
    "    @torch.no_grad()\n",
    "    def evaluate(self, data):\n",
    "        scores = self.forward(data[0])\n",
    "        predicted_result = torch.argmax(scores, dim=-1)\n",
    "        gt_result = data[1]\n",
    "        correct_num = torch.sum(predicted_result == gt_result)\n",
    "        return correct_num.data.tolist(), len(predicted_result)\n",
    "\n",
    "    # 获取当前网络及其所有子网络的参数字典\n",
    "    def get_parameters(self) -> dict:\n",
    "        for key in self.__dict__:\n",
    "            # 递归构造网络参数字典\n",
    "            if isinstance(self.__dict__[key], BaseModule):\n",
    "                sub_state_dict = self.__dict__[key].get_parameters()\n",
    "                for sub_key in sub_state_dict:\n",
    "                    new_key = key + '.' + sub_key\n",
    "                    self._parameters[new_key] = sub_state_dict[sub_key]\n",
    "        return self._parameters\n",
    "\n",
    "    # 将参数字典中的所有参数加载到当前网络的对应位置上\n",
    "    def set_parameters(self, state_dict):\n",
    "        self._parameters = state_dict\n",
    "        for key in self.__dict__:\n",
    "            # 从当前网络的参数名称中剥离出子网络的参数名称，并据此构造子网络参数字典\n",
    "            if isinstance(self.__dict__[key], BaseModule):\n",
    "                sub_state_dict = {}\n",
    "                for param_key in self._parameters:\n",
    "                    param_key_copy = copy.deepcopy(param_key)\n",
    "                    param_key = param_key.split(sep='.')\n",
    "                    module_name = param_key[0]          # 键的第一部分为子模块名称\n",
    "                    param_key.pop(0)                    # 去掉模块名称\n",
    "                    param_key = '.'.join(param_key)     # 其余部分为该参数在子模块字典中的键\n",
    "                    if module_name == key:              # 若模块名称匹配，则将当前参数加入到当前子模块的字典中\n",
    "                        sub_state_dict[param_key] = self._parameters[param_key_copy]\n",
    "                self.__dict__[key].set_parameters(sub_state_dict)  # 递归调用\n",
    "\n",
    "    # 获取顶层网络包含的全部需要计算梯度的张量列表\n",
    "    def parameters(self) -> list:\n",
    "        # 初次调用时初始化网络参数，并加载预训练模型\n",
    "        if self._parameters is None:\n",
    "            self.init_parameters()\n",
    "            self.get_parameters()\n",
    "        if self.pretrain is True:\n",
    "            state_dict = torch.load(self.pretrain_model_path)\n",
    "            self.load_state_dict(state_dict)\n",
    "        grad_params = []\n",
    "        for key in self._parameters:\n",
    "            if isinstance(self._parameters[key], torch.Tensor):\n",
    "                if self._parameters[key].requires_grad is True:\n",
    "                    grad_params.append(self._parameters[key])\n",
    "        return grad_params\n",
    "\n",
    "    # 递归初始化网络参数\n",
    "    def init_parameters(self, device=None):\n",
    "        if device is None:\n",
    "            device = self.device\n",
    "        self._parameters = {}\n",
    "        for key in self.__dict__:\n",
    "            if isinstance(self.__dict__[key], BaseModule):\n",
    "                self.__dict__[key].init_parameters(device)\n",
    "\n",
    "    # 获取需要存储的网络参数字典\n",
    "    def state_dict(self) -> dict:\n",
    "        self.get_parameters()\n",
    "        for key in self._parameters:\n",
    "            info = key.split('.')\n",
    "            if info[-1] == 'running_mean' or info[-1] == 'running_var':\n",
    "                self._parameters[key] = self._parameters[key].detach()\n",
    "        return self._parameters\n",
    "\n",
    "    # 加载网络参数字典\n",
    "    def load_state_dict(self, state_dict):\n",
    "        # 判断网络参数字典是否与网络结构相符\n",
    "        if len(self._parameters) != len(state_dict):\n",
    "            print('The state dict does not match the model.')\n",
    "            exit(1)\n",
    "        for key in self._parameters:\n",
    "            if self._parameters[key] is not None:\n",
    "                if self._parameters[key].size() == state_dict[key].size():\n",
    "                    continue\n",
    "                else:\n",
    "                    print('The state dict does not match the model.')\n",
    "                    exit(1)\n",
    "        del self._parameters\n",
    "        self._parameters = state_dict\n",
    "        for key in self._parameters:\n",
    "            # 将所有网络参数转化成float32格式\n",
    "            if self._parameters[key].dtype is not torch.float:\n",
    "                self._parameters[key] = self._parameters[key].float()\n",
    "            self._parameters[key] = self._parameters[key].to(self.device)\n",
    "            self._parameters[key].detach_()    # 将需要求梯度的张量从cpu转移到gpu后需要重新将其设置为叶子结点\n",
    "            info = key.split('.')\n",
    "            if info[-1] != 'running_mean' and info[-1] != 'running_var':\n",
    "                self._parameters[key].requires_grad = True\n",
    "        self.set_parameters(self._parameters)\n",
    "\n",
    "    # 递归将网络设置为评估模式\n",
    "    def eval(self):\n",
    "        for key in self.__dict__:\n",
    "            if isinstance(self.__dict__[key], BaseModule):\n",
    "                self.__dict__[key].eval()\n",
    "\n",
    "    # 递归将网络设置为训练模式\n",
    "    def train(self):\n",
    "        for key in self.__dict__:\n",
    "            if isinstance(self.__dict__[key], BaseModule):\n",
    "                self.__dict__[key].train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd10a8ac",
   "metadata": {
    "papermill": {
     "duration": 0.008971,
     "end_time": "2022-07-05T00:56:39.169873",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.160902",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 3.2 神经网络模块序列：Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "57637647",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.189177Z",
     "iopub.status.busy": "2022-07-05T00:56:39.188898Z",
     "iopub.status.idle": "2022-07-05T00:56:39.202402Z",
     "shell.execute_reply": "2022-07-05T00:56:39.201508Z"
    },
    "papermill": {
     "duration": 0.025714,
     "end_time": "2022-07-05T00:56:39.204408",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.178694",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Sequential(BaseModule):\n",
    "    # 传入一个网络模块列表对Sequential类进行初始化\n",
    "    def __init__(self, *layers):\n",
    "        super(Sequential, self).__init__()\n",
    "        self._parameters = {}\n",
    "        self.layers = []\n",
    "        for layer in layers:\n",
    "            self.layers.append(layer)\n",
    "    \n",
    "    # 将神经网络模块序列的前向传播写入__call__方法\n",
    "    def __call__(self, x):\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        return x\n",
    "\n",
    "    # 函数重载，递归初始化网络参数\n",
    "    def init_parameters(self, device=None):\n",
    "        for layers in self.layers:\n",
    "            layers.init_parameters(device)\n",
    "\n",
    "    # 函数重载，获取当前网络及其所有子网络的参数字典\n",
    "    def get_parameters(self):\n",
    "        for i, layer in enumerate(self.layers):\n",
    "            state_dict = layer.get_parameters()\n",
    "            for key in state_dict:\n",
    "                # Sequential实例中的层从前到后以数字顺序命名，从0开始\n",
    "                new_key = str(i) + '.' + key\n",
    "                self._parameters[new_key] = state_dict[key]\n",
    "        return self._parameters\n",
    "\n",
    "    # 函数重载，将参数字典中的所有参数加载到当前网络的对应位置上\n",
    "    def set_parameters(self, state_dict):\n",
    "        self._parameters = state_dict\n",
    "        for i in range(len(self.layers)):\n",
    "            sub_state_dict = {}\n",
    "            for key in self._parameters:\n",
    "                sub_key = key.split(sep='.')\n",
    "                index = int(sub_key[0])\n",
    "                sub_key.pop(0)\n",
    "                sub_key = '.'.join(sub_key)\n",
    "                if index == i:\n",
    "                    sub_state_dict[sub_key] = state_dict[key]\n",
    "                elif index > i:\n",
    "                    break\n",
    "            self.layers[i].set_parameters(sub_state_dict)\n",
    "\n",
    "    # 函数重载，递归将网络设置为评估模式\n",
    "    def eval(self):\n",
    "        for layer in self.layers:\n",
    "            if isinstance(layer, BaseModule):\n",
    "                layer.eval()\n",
    "    \n",
    "    # 函数重载，递归将网络设置为训练模式\n",
    "    def train(self):\n",
    "        for layer in self.layers:\n",
    "            if isinstance(layer, BaseModule):\n",
    "                layer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ba53131",
   "metadata": {
    "papermill": {
     "duration": 0.008702,
     "end_time": "2022-07-05T00:56:39.222210",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.213508",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 3.3 神经网络层的实现\n",
    "继承自`BaseModule`，是所有网络层的基类，派生有`ReLU`、`Dropout`、`Flatten`、`Linear`、`Conv2d`、`MaxPool2d`、`AdaptiveAvg2d`、`BatchNorm2d`、`ZeroPad2d`网络层。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f6675f44",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.241353Z",
     "iopub.status.busy": "2022-07-05T00:56:39.241072Z",
     "iopub.status.idle": "2022-07-05T00:56:39.287699Z",
     "shell.execute_reply": "2022-07-05T00:56:39.286875Z"
    },
    "papermill": {
     "duration": 0.058772,
     "end_time": "2022-07-05T00:56:39.289765",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.230993",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 网络层基类，定义通用的接口\n",
    "class Layer(BaseModule):\n",
    "    def __init__(self):\n",
    "        super(Layer, self).__init__()\n",
    "        self._parameters = {}\n",
    "        self.mode = 'Train'\n",
    "        self.device = None\n",
    "\n",
    "    def __call__(self, x) -> torch.Tensor:\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def forward(self, data) -> torch.Tensor:\n",
    "        return self.__call__(data)\n",
    "\n",
    "    # 根据权重矩阵的尺寸计算扇入扇出，据此初始化权重矩阵和偏置矩阵，参考了pytorch官方代码的实现\n",
    "    @staticmethod\n",
    "    def _calculate_fan_in_and_fan_out(tensor):\n",
    "        num_input = tensor.size(1)\n",
    "        num_output = tensor.size(0)\n",
    "        receptive_field_size = 1\n",
    "        if tensor.dim() > 2:\n",
    "            for s in tensor.shape[2:]:\n",
    "                receptive_field_size *= s\n",
    "        fan_in = num_input * receptive_field_size\n",
    "        fan_out = num_output * receptive_field_size\n",
    "        return fan_in, fan_out\n",
    "\n",
    "    def init_parameters(self, device=None):\n",
    "        self.device = device\n",
    "\n",
    "    def get_parameters(self):\n",
    "        self.get_no_grad_parameters()\n",
    "        return self._parameters\n",
    "\n",
    "    def set_parameters(self, state_dict):\n",
    "        self._parameters = state_dict\n",
    "\n",
    "    # 获取当前层中不需要计算梯度的参数\n",
    "    def get_no_grad_parameters(self):\n",
    "        pass\n",
    "\n",
    "    def eval(self):\n",
    "        self.mode = 'Eval'\n",
    "\n",
    "    def train(self):\n",
    "        self.mode = 'Train'\n",
    "\n",
    "    @staticmethod\n",
    "    def _pair(x):\n",
    "        if isinstance(x, tuple):\n",
    "            return x\n",
    "        else:\n",
    "            return x, x\n",
    "\n",
    "\n",
    "class ReLU(Layer):\n",
    "    def __init__(self):\n",
    "        super(ReLU, self).__init__()\n",
    "\n",
    "    def __call__(self, x):\n",
    "        return torch.where(x < 0.0, torch.tensor([0.0], device=self.device), x)\n",
    "\n",
    "\n",
    "class Dropout(Layer):\n",
    "    def __init__(self, dropout_ratio=0.5):\n",
    "        super(Dropout, self).__init__()\n",
    "        self.dropout_ratio = dropout_ratio\n",
    "        self.mask = None\n",
    "\n",
    "    def __call__(self, x):\n",
    "        # 训练模式下随机丢弃一定比例的信息，测试模式下保证数学期望不变\n",
    "        if self.mode == 'Train':\n",
    "            self.mask = torch.rand_like(x).to(self.device) > self.dropout_ratio\n",
    "            return x * self.mask\n",
    "        else:\n",
    "            return x * (1.0 - self.dropout_ratio)\n",
    "\n",
    "\n",
    "class Flatten(Layer):\n",
    "    def __init__(self):\n",
    "        super(Flatten, self).__init__()\n",
    "\n",
    "    def __call__(self, x):\n",
    "        return torch.flatten(x, 1)\n",
    "\n",
    "\n",
    "class Linear(Layer):\n",
    "    def __init__(self, in_features, out_features, bias=True):\n",
    "        super(Linear, self).__init__()\n",
    "        self.in_features = in_features\n",
    "        self.out_features = out_features\n",
    "        self.bias = bias\n",
    "        self.W = None\n",
    "        self.B = None\n",
    "\n",
    "    def __call__(self, x):\n",
    "        if self.B is not None:\n",
    "            return torch.matmul(x, self.W.T) + self.B\n",
    "        else:\n",
    "            return torch.matmul(x, self.W.T)\n",
    "\n",
    "    def init_parameters(self, device=None):\n",
    "        self.device = device\n",
    "        self.W = torch.empty(self.out_features, self.in_features, device=self.device, requires_grad=True)\n",
    "        nn.init.kaiming_uniform_(self.W, a=math.sqrt(5))\n",
    "        self._parameters['weight'] = self.W\n",
    "        if self.bias is True:\n",
    "            self.B = torch.empty(self.out_features, device=self.device, requires_grad=True)\n",
    "            fan_in, _ = self._calculate_fan_in_and_fan_out(self.W)\n",
    "            if fan_in != 0:\n",
    "                bound = 1 / math.sqrt(fan_in)\n",
    "                nn.init.uniform_(self.B, -bound, bound)\n",
    "                self._parameters['bias'] = self.B\n",
    "\n",
    "    def set_parameters(self, state_dict):\n",
    "        self._parameters = state_dict\n",
    "        self.W = self._parameters['weight']\n",
    "        if self.bias is True:\n",
    "            self.B = self._parameters['bias']\n",
    "\n",
    "\n",
    "class Conv2d(Layer):\n",
    "    def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, bias=True, dilation=1, groups=1):\n",
    "        super(Conv2d, self).__init__()\n",
    "        self.in_channels = in_channels\n",
    "        self.out_channels = out_channels\n",
    "        self.kernel_size = self._pair(kernel_size)\n",
    "        self.stride = self._pair(stride)\n",
    "        self.padding = self._pair(padding)\n",
    "        self.dilation = self._pair(dilation)\n",
    "        self.filter = None    # 二维卷积层的卷积核\n",
    "        self.B = None         # 二维卷积层的偏置\n",
    "        self.bias = bias\n",
    "        self.groups = groups\n",
    "\n",
    "    def __call__(self, x):\n",
    "        return F.conv2d(input=x, weight=self.filter, bias=self.B, stride=self.stride, padding=self.padding,\n",
    "                        dilation=self.dilation, groups=self.groups)\n",
    "\n",
    "    def init_parameters(self, device=None):\n",
    "        self.device = device\n",
    "        self.filter = torch.empty(self.out_channels, self.in_channels, self.kernel_size[0], self.kernel_size[1],\n",
    "                                  device=self.device, requires_grad=True)\n",
    "        nn.init.kaiming_uniform_(self.filter, a=math.sqrt(5))\n",
    "        self._parameters['weight'] = self.filter\n",
    "\n",
    "        if self.bias is True:\n",
    "            self.B = torch.empty(self.out_channels, device=self.device, requires_grad=True)\n",
    "            fan_in, _ = self._calculate_fan_in_and_fan_out(self.filter)\n",
    "            if fan_in != 0:\n",
    "                bound = 1 / math.sqrt(fan_in)\n",
    "                nn.init.uniform_(self.B, -bound, bound)\n",
    "                self._parameters['bias'] = self.B\n",
    "\n",
    "    def set_parameters(self, state_dict):\n",
    "        self._parameters = state_dict\n",
    "        self.filter = self._parameters['weight']\n",
    "        if self.bias is True:\n",
    "            self.B = self._parameters['bias']\n",
    "\n",
    "\n",
    "class MaxPool2d(Layer):\n",
    "    def __init__(self, kernel_size, stride=1, padding=0, dilation=1, return_indices=False, ceil_mode=False):\n",
    "        super(MaxPool2d, self).__init__()\n",
    "        self.kernel_size = self._pair(kernel_size)\n",
    "        self.stride = self._pair(stride)\n",
    "        self.padding = self._pair(padding)\n",
    "        self.dilation = self._pair(dilation)\n",
    "        self.return_indices = return_indices\n",
    "        self.ceil_mode = ceil_mode\n",
    "\n",
    "    def __call__(self, x):\n",
    "        return F.max_pool2d(x, self.kernel_size, self.stride, self.padding, self.dilation, self.ceil_mode,\n",
    "                            self.return_indices)\n",
    "\n",
    "\n",
    "class AdaptiveAvgPool2d(Layer):\n",
    "    def __init__(self, output_size):\n",
    "        super(AdaptiveAvgPool2d, self).__init__()\n",
    "        self.output_size = output_size\n",
    "\n",
    "    def __call__(self, x):\n",
    "        return F.adaptive_avg_pool2d(x, self.output_size)\n",
    "\n",
    "\n",
    "class BatchNorm2d(Layer):\n",
    "    def __init__(self, num_features, eps=1e-5, momentum=0.1):\n",
    "        super(BatchNorm2d, self).__init__()\n",
    "        self.num_features = num_features\n",
    "        self.eps = eps\n",
    "        self.momentum = momentum\n",
    "        self.gamma = None         # 批量归一化层的权重\n",
    "        self.beta = None          # 批量归一化层的偏置\n",
    "        self.running_mean = None  # 数据样本均值\n",
    "        self.running_var = None   # 数据样本方差\n",
    "\n",
    "    def __call__(self, x):\n",
    "        if self.running_mean is None:\n",
    "            self.running_mean = torch.zeros(self.num_features, device=self.device)\n",
    "            self.running_var = torch.zeros(self.num_features, device=self.device)\n",
    "        return F.batch_norm(input=x, running_mean=self.running_mean, running_var=self.running_var, weight=self.gamma,\n",
    "                            bias=self.beta, training=(self.mode == 'Train'), momentum=self.momentum, eps=self.eps)\n",
    "\n",
    "    def init_parameters(self, device=None):\n",
    "        self.device = device\n",
    "        self.gamma = torch.ones(self.num_features, device=self.device, requires_grad=True)\n",
    "        self.beta = torch.zeros(self.num_features, device=self.device, requires_grad=True)\n",
    "        self._parameters['running_mean'] = None\n",
    "        self._parameters['running_var'] = None\n",
    "        self._parameters['weight'] = self.gamma\n",
    "        self._parameters['bias'] = self.beta\n",
    "\n",
    "    def get_no_grad_parameters(self):\n",
    "        self._parameters['running_mean'] = self.running_mean\n",
    "        self._parameters['running_var'] = self.running_var\n",
    "\n",
    "    def set_parameters(self, state_dict):\n",
    "        self._parameters = state_dict\n",
    "        self.running_mean = self._parameters['running_mean']\n",
    "        self.running_var = self._parameters['running_var']\n",
    "        self.gamma = self._parameters['weight']\n",
    "        self.beta = self._parameters['bias']\n",
    "\n",
    "\n",
    "class ZeroPad2d(Layer):\n",
    "    def __init__(self, padding):\n",
    "        super(ZeroPad2d, self).__init__()\n",
    "        if isinstance(padding, tuple):\n",
    "            self.padding = padding\n",
    "        else:\n",
    "            self.padding = (padding, padding, padding, padding)\n",
    "\n",
    "    def __call__(self, x):\n",
    "        return F.pad(input=x, pad=self.padding, mode='constant', value=0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b72ff6e",
   "metadata": {
    "papermill": {
     "duration": 0.008588,
     "end_time": "2022-07-05T00:56:39.307107",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.298519",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 3.4 损失函数：CrossEntropyLoss\n",
    "在本小节，我们实现了交叉熵损失函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9b0b3551",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.325540Z",
     "iopub.status.busy": "2022-07-05T00:56:39.325283Z",
     "iopub.status.idle": "2022-07-05T00:56:39.334104Z",
     "shell.execute_reply": "2022-07-05T00:56:39.333221Z"
    },
    "papermill": {
     "duration": 0.020286,
     "end_time": "2022-07-05T00:56:39.335997",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.315711",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class CrossEntropyLoss:\n",
    "    def __init__(self, reduction='mean'):\n",
    "        super(CrossEntropyLoss, self).__init__()\n",
    "        self.reduction = reduction\n",
    "\n",
    "    def __call__(self, scores, target):\n",
    "        if scores.dim() > 2:\n",
    "            scores = scores.view(scores.size(0), scores.size(1), -1)\n",
    "            scores = scores.transpose(1, 2)\n",
    "            scores = scores.contiguous().view(-1, scores.size(2))\n",
    "        target = target.view(-1, 1)\n",
    "\n",
    "        scores = F.log_softmax(scores, 1)\n",
    "        scores = scores.gather(1, target)\n",
    "        loss = -1 * scores\n",
    "\n",
    "        if self.reduction == 'mean':\n",
    "            loss = loss.mean()\n",
    "        elif self.reduction == 'sum':\n",
    "            loss = loss.sum()\n",
    "        return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85dbf8b7",
   "metadata": {
    "papermill": {
     "duration": 0.008466,
     "end_time": "2022-07-05T00:56:39.353836",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.345370",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 3.5 优化器：Optimizer\n",
    " `Optimizer`类为优化器的基类，派生有`SGD`（随机梯度下降，包含动量`momentum`和L2权值衰减`weight_decay`）、`Adam`（包含L2权值衰减）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ebb67fc9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.372621Z",
     "iopub.status.busy": "2022-07-05T00:56:39.372361Z",
     "iopub.status.idle": "2022-07-05T00:56:39.395030Z",
     "shell.execute_reply": "2022-07-05T00:56:39.394055Z"
    },
    "papermill": {
     "duration": 0.035272,
     "end_time": "2022-07-05T00:56:39.397754",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.362482",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Optimizer:\n",
    "    def __init__(self):\n",
    "        self.params = None\n",
    "        self.lr = None\n",
    "\n",
    "    # 优化器更新参数的函数\n",
    "    @torch.no_grad()\n",
    "    def step(self):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    # 优化器参数梯度归零的函数\n",
    "    @torch.no_grad()\n",
    "    def zero_grad(self):\n",
    "        for param in self.params:\n",
    "            if param.grad.grad_fn is not None:\n",
    "                param.grad.detach_()\n",
    "            else:\n",
    "                param.grad.requires_grad_(False)\n",
    "            param.grad.zero_()\n",
    "\n",
    "\n",
    "# Adam优化器，参考了pytorch官方的实现\n",
    "class Adam(Optimizer):\n",
    "    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0):\n",
    "        super(Adam, self).__init__()\n",
    "        self.params = params\n",
    "        self.lr = lr\n",
    "        self.betas = betas\n",
    "        self.eps = eps\n",
    "        self.weight_decay = weight_decay\n",
    "        self.iter = 0\n",
    "        self.m = None\n",
    "        self.v = None\n",
    "        self.state = {}\n",
    "        for param in self.params:\n",
    "            self.state[param] = {}\n",
    "\n",
    "    @torch.no_grad()\n",
    "    def step(self):\n",
    "        grads = []\n",
    "        exp_avgs = []\n",
    "        exp_avg_sqs = []\n",
    "        state_steps = []\n",
    "        beta1, beta2 = self.betas\n",
    "\n",
    "        for param in self.params:\n",
    "            grads.append(param.grad)\n",
    "\n",
    "            state = self.state[param]\n",
    "            if len(state) == 0:\n",
    "                state['step'] = 0\n",
    "                state['exp_avg'] = torch.zeros_like(param, memory_format=torch.preserve_format)\n",
    "                state['exp_avg_sq'] = torch.zeros_like(param, memory_format=torch.preserve_format)\n",
    "\n",
    "            exp_avgs.append(state['exp_avg'])\n",
    "            exp_avg_sqs.append(state['exp_avg_sq'])\n",
    "\n",
    "            state['step'] += 1\n",
    "            state_steps.append(state['step'])\n",
    "\n",
    "        for i, param in enumerate(self.params):\n",
    "            grad = grads[i]\n",
    "            exp_avg = exp_avgs[i]\n",
    "            exp_avg_sq = exp_avg_sqs[i]\n",
    "            step = state_steps[i]\n",
    "\n",
    "            bias_correction1 = 1 - beta1 ** step\n",
    "            bias_correction2 = 1 - beta2 ** step\n",
    "\n",
    "            if self.weight_decay != 0:\n",
    "                grad = grad.add(param, alpha=self.weight_decay)\n",
    "\n",
    "            exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)\n",
    "            exp_avg_sq.mul_(beta2).addcmul_(grad, grad.conj(), value=1 - beta2)\n",
    "            denom = (exp_avg_sq.sqrt() / math.sqrt(bias_correction2)).add_(self.eps)\n",
    "\n",
    "            step_size = self.lr / bias_correction1\n",
    "            param.addcdiv_(exp_avg, denom, value=-step_size)\n",
    "\n",
    "\n",
    "# SGD优化器，参考了pytorch官方的实现\n",
    "class SGD(Optimizer):\n",
    "    def __init__(self, params, lr, momentum=0, weight_decay=0):\n",
    "        super(SGD, self).__init__()\n",
    "        self.params = params\n",
    "        self.lr = lr\n",
    "        self.momentum = momentum\n",
    "        self.weight_decay = weight_decay\n",
    "        self.state = {}\n",
    "        for param in self.params:\n",
    "            self.state[param] = {}\n",
    "\n",
    "    @torch.no_grad()\n",
    "    def step(self):\n",
    "        grads = []\n",
    "        momentum_buffer_list = []\n",
    "        weight_decay = self.weight_decay\n",
    "        momentum = self.momentum\n",
    "        lr = self.lr\n",
    "\n",
    "        for param in self.params:\n",
    "            grads.append(param.grad)\n",
    "\n",
    "            state = self.state[param]\n",
    "            if 'momentum_buffer' not in state:\n",
    "                momentum_buffer_list.append(None)\n",
    "            else:\n",
    "                momentum_buffer_list.append(state['momentum_buffer'])\n",
    "\n",
    "        for i, param in enumerate(self.params):\n",
    "            grad = grads[i]\n",
    "            if weight_decay != 0:\n",
    "                grad = grad.add(param, alpha=weight_decay)\n",
    "\n",
    "            if momentum != 0:\n",
    "                buf = momentum_buffer_list[i]\n",
    "                if buf is None:\n",
    "                    buf = torch.clone(grad).detach()\n",
    "                    momentum_buffer_list[i] = buf\n",
    "                else:\n",
    "                    buf.mul_(momentum).add_(grad, alpha=1)\n",
    "                grad = buf\n",
    "\n",
    "            alpha = -lr\n",
    "            param.add_(grad, alpha=alpha)\n",
    "\n",
    "        for param, momentum_buffer in zip(self.params, momentum_buffer_list):\n",
    "            state = self.state[param]\n",
    "            state['momentum_buffer'] = momentum_buffer"
   ]
  },
  {
   "attachments": {
    "OneCycleLR.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAYAAAA10dzkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABl7klEQVR4nO3dd3xT9eI+8OckbZLuvdKWtkBZskcRQaSA0DKEKoIL1Csogl5FvUovLlygonIvIriviAiiKMPBKi2bqojIplDopnRldCRNcn5/8KU/KtCZ9qQ5z/v1ykubc5I+oW3Ok/M553MEURRFEBEREZFsKKQOQEREREStiwWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZcpA7QltlsNuTl5cHLywuCIEgdh4iIiBpAFEUYDAZotVooFPLcF8YC2Ax5eXmIjIyUOgYRERE1QXZ2NiIiIqSOIQkWwGbw8vICcOkXyNvbW+I0RERE1BB6vR6RkZE123E5YgFshsvDvt7e3iyAREREbYycD9+S58A3ERERkYyxABIRERHJDAsgERERkczwGEAiIiI7EUURFosFVqtV6iiyplQq4eLiIutj/OrDAkhERGQHZrMZ+fn5qKiokDoKAXB3d0dYWBhUKpXUURwSCyAREVEz2Ww2ZGZmQqlUQqvVQqVSce+TRERRhNlsxsWLF5GZmYnY2FjZTvZcFxZAIiKiZjKbzbDZbIiMjIS7u7vUcWTPzc0Nrq6uOH/+PMxmMzQajdSRHA4rMRERkZ1wT5Pj4M+ibvzXISIiIpIZhxgCLi8vx7PPPgs/Pz8YjUa8+eabUKvVV623YcMGbN++HSaTCZMmTcLIkSPrXWa1WjF79mysWrUKUVFRWLFiBfr06dOg5yQiIqKGmzVrFqKjo/Hss89KHYXqIzqAqVOniuvWrRNFURS/+OILcc6cOVetc/z4cXHAgAGizWYTrVar2KdPHzEnJ6feZStXrhT37dsnFhYWimPGjBEHDx7coOdsCJ1OJwIQdTpdc14+ERG1cZWVleKxY8fEyspKqaM0yksvvSR2795dvOeee8RBgwaJ7u7u4r333iuOGTNG1Gg0YmZmZqOeb9euXeJff/3VqMccPXpUvPvuu0UA4tNPPy0eO3as1nKdTid+/PHHoiAIYv/+/cXZs2eLQ4YMEadOnSqWlZVd93nr+plw+y2Kkg8B5+XlYe3atUhMTAQAJCYmYvny5TAYDLXWW7x4MRISEiAIAhQKBQYNGoRly5bVu+z222/HjTfeiKCgIDz66KNQKpUNek4iIiJnFx4ejl9//RVfffUVHn74Yfj5+WHlypX48ccf8dlnnzX6+YYMGYLu3bs36jHdunXDP/7xDwDAq6++iq5du9Za7u3tjenTp0Or1eKRRx7B+++/j19++QV79uzBww8/3OiMdInkBTA1NRWBgYE1Z+gEBQVBpVIhPT291nopKSmIioqq+To2NhZpaWn1LnNzc6u5/9ixY3jjjTca9JzXYjKZoNfra92IWotZb8WF9ApkbzUia7MBuWnlKDhQgeK/qlBeUA2bVZQ6IhG1MUlJSdc9Q3bixIkICgpqlRwuLpeOSHN1da13HQDw8PDAbbfdhp9++qnFszkryY8BzM3Nhb+/f637vLy8kJeXV+d6V65T1zIAKC4uxtKlS/Hpp5/W+mRS3+P+bsGCBZg/f34jXyFR0xlzqnH881KcWatD8V+mOtcVFIAmyAWeES7w7aSGXxc1fLuoEdhTA99OKggKzklGRLUFBgZed5mbmxvWrFmDRYsWYebMmXj99dcxffp03H///Zg3bx5iY2Px888/46WXXsKtt96K3377Da+99hr69euHZ599Fh988AHeeecdrF+/HtOnT0dJSQm2b9+Ojh072iV7WVkZAgIC7PJcciR5ARQE4apPH2az+apPAX9f78p16loGXCp2Y8aMweHDh5GUlISMjAxERkbW+7i/S05OxlNPPVXztV6vR2RkZCNfMVH9Ki9acODFQhz/tBRKtYCYCV7o/XQgAnpqoPZXQqEUYKm0wVIpwlJhQ+VFCyoKLKjIt8CQVY2ykybkbDOi8uKly1GpfBQIiXNDyI3uCI/3QNhgdyhVkg8AEDm96gobyk7U/eGtJfh2UcPVvfl/44mJiXjkkUdw9OhRrF69GmazGe+++y46dOiAefPmQa1W47///S9uvfVW9OrVCyaTCVarFSqVCsOGDcNTTz2F06dP4/fff8eECRPwySefYOHChc3OtXPnTqxZswZLly5t9nPJleQFUKvVQqfT1brPaDRCq9XWuZ7BYKhZp65lAKBSqdC/f3+sWbMGnTt3xv79+xEZGVnv4/5OrVZf8+xkIns6vboMOx/Lh2gDBr4ajO6z/KHyUtb/wGuoKrbg4h9VuLC/AhcOVOLIByX47dWLcPVUIDzeA+0SPNE+yRseYdf/4ENETVd2woRv+p1p9e87+fcOCOrrVv+K9fD29oaPjw/Gjx+PuLg4AEBoaCg8PDyQk5ODv/76C0ajEcCl4dvg4GAAl3bM+Pr6AgDuuusuAECvXr3qHGVriI0bN+L777/Hvn37sGPHDgwcOLBZzydnkhfAYcOG4eGHH4bZbIZKpar55bj8i3bZiBEjcOrUqZqvMzIyEB8fX++yKymVSvTq1Qvh4eGNehxRa7BWi9jzVD7+er8EHSd74+YlWrgHN+9PVBPggsiRnogc6QkAEG0iig5VIWvzpeMIdz+Rj52P5UM71B2xU3zQ/g6fZn9PIvr/fLuoMfn3DpJ8X3sRBKHWZe3CwsLw2muvoXPnzrjppptw/vz5Wute6/+BS8fw2Wy2ZmWJj4/HxIkT0adPH+zbt48FsBkkHwPSarVISEioOfliy5YtmDVrFjQaDRYtWoSTJ08CAGbOnIlt27YBACwWC9LT0zFjxox6lx07dgwlJSUALh0vYLFYMGjQoHofR9Saqstt2DTmHI4uL8Ety7QYvaZdixQxQSEgqK8b+iUHISm1PR680AXxn4RDqVZg5+P5+CLiJH6ZnIXs7UaINp5UQtRcru4KBPV1a/WbPYZ/r+fRRx9FbGwsHnjggRYfFSsuLkZGRkat+6Kjo/Hpp59i7ty5OHjwYIt+f2fmEB/1ly9fjrlz5+LAgQMoKSmpOT7g66+/RnR0NDp37oxevXrhwQcfxDPPPAOz2Yz33nsPoaGhAFDnskWLFmHDhg2YMGECQkND8fHHH9d8KqnrcUStxay3YtPY8yg6VIXbtkYjfJhnq31vjb8Luv3DD93+4YfKixacWlWGox+WYsPIc/DpqEL3R/3RbYZfk4egiajtsFgssFgs11xmtVpr/v+PP/5A9+7dUVFRgT179qCyshKZmZmIiYmBKIoQxUsfHi/v7RNFsWa7e3nZtb43AFRXV9ec7Wuz2fDGG2/grbfeqlnnco7bb78d06dPx+TJk/Hrr7/Cz8+vuS9ffqSdhrBt40SS1FzVFVbxu8FnxI98jor5+8qljiOKoijabDYxd6dR3HJvlviBy1/iRz5Hxb3J+aIx3yx1NCKH1VYngr7st99+E0eOHCkKgiB+9NFHNdu1b7/9VlSpVOJdd90l5ufni6Ioih999JHo7e0t3nbbbeJ3330nBgcHixs3bhSPHTsmdurUSRw6dKh4+vRp8aWXXhIBiB9//LF49uxZcfjw4WKXLl3EI0eO1PreR48eFe+66y4RgDhs2DDx3nvvFadMmSLecMMN4pQpU0SdTid+9NFHokKhEG+++WZx7969oiiKYlVVldi7d2+xW7du4v/+97+rXhMngq6bIIrXqeNUL71eDx8fH+h0Onh7e0sdh9oY0SZi813ZOLfRgIkpMQgd5C51pKsYc6rx5+IiHP2wFFaziBtm+KHfvCCeNEL0N1VVVTV7wa43rx61rrp+Jtx+O8AxgERytf/fF3DmWz1GrYp0yPIHAJ4Rrhi8KAz3Z3fGgJeCcOqrMqzscAr7kgtQVWqt/wmIiMghsQASSeDsej0OvlmEm94KRfskx//0qfZVov+/gzE1szN6PRWIw/8txpcxJ3Hw7Yuwmpt3Vh8REbU+FkCiVqbPNCPlgRzETPRC76fb1iz2al8lbnwtBFPPdkbnqb7Yn3wBq3tk4PwvhvofTEREDoMFkKgV2SyXjvtT+ykx4vOIq+bJaivcQ1wwdIkWU/7oCI9wV2xKPI8fJ5yH7qxZ6mhERNQALIBEreiPRUW4+Fslbl0VCbVv259aJaCHBhO2R2P02kgU/VGF1T1O48/FRbBZeW4ZEZEjYwEkaiUlx6qQ/lIhej8TiNAbHfOkj6YQBAEdJ/ng7mMd0W26P3Y/VYB1g8+i+GiV1NGIWh0n1nAc/FnUjQWQqBXYrCK2P5gLn/YqxM0PljpOi1B5KnHzf8Jw+64YmMqs+KbPGfz2RiH3BpIsuLpemhqpoqJC4iR02eWfxeWfDdXmEFcCIXJ2xz4qQWF6Je7Y2x4uGuf+3BU22ANTDnXEr68UIv2FQmT9YsTILyPgHaWSOhpRi1EqlfD19UVhYSEAwN3dvc0e49vWiaKIiooKFBYWwtfXF0pl2z/cpiVwIuhm4ESS1BBVxRas7HQaMRO8MOKzCKnjtKq8neXYOjUHZp0Vw5ZrEXuXr9SRiFqMKIooKChAWVmZ1FEIgK+vL0JDQ69ZxLn95h5Aoha3//lCiBYRgxbI7zrT2qEeuOvPjkidmYctd+cg6xcjhn6gbdEL1RNJRRAEhIWFITg4GNXV1VLHkTVXV1fu+asHCyBRC7p4qBJHPyzBkHdD4R4izz83ta8So76OQFSiJ9Jm5eHiwUokrGsH345qqaMRtQilUsnyQQ6PH8OJWtC+Zwvg11mN7rPb1oTP9iYIArrc74dJBzrAUiVibf8zyNyglzoWEZFssQAStZCcFCOyt5Zj4OvBULryYHAACOiuwZ2/dkDEcE/8NCEL++dd4FnCREQSYAEkagGiKGJf8gUED3BrE9f6bU1qHyUSvovEoDdDcHDhRfw8MQtmg1XqWEREssICSNQCMtcbUJheiRsXhHAqiGsQBAF9nw3C2B+jkJtWjnWDz0J/npeRIyJqLSyARHYm2kQceP4CIkZ4IHKEp9RxHFpUghfu2NceZoMN38adQcF+TqJLRNQaWACJ7OzsD3qUHDVh4KshUkdpEwJu0ODO9A7wjVXjh2GZOL26TOpIREROjwWQyI5EUcTvr19EeLwHQgc5z/V+W5pbkAsmbI9Ghzu9seXuHPy5uEjqSERETk2eE5MRtZDsLUZcPFiF27ZGSx2lzVGqFRi5IgKe4a7YPacAFQUWHkNJRNRCWACJ7Oi31y8iOM4NESM8pI7SJgmCgEELQ+EW4oI9T10qgcM+Duc0OkREdsYCSGQnebvKkb+rAok/tONeq2bqPScQbsEuSHkgB5UXLRi9th0vH0dEZEd8RyWykz/eLoL/DWrEjPeSOopT6Hyv7/9NE1OBTWPOwWzkXIFERPbCAkhkB2WnTTi3yYBecwIgKLj3z17ajfLCbZujcPFgFTaOPg+TjiWQiMgeWACJ7ODwkmJoApTodI+v1FGcTthgD0zYFo3SY1XYMPIcqkosUkciImrzWACJmsmks+LE52W44RF/uLjxT6olhMS5Y0JKDPSZZqwfcQ6VRSyBRETNwa0VUTMd/6wU1iobeszylzqKUwvq44aJqTEoz6vGD8MyUXmRJZCIqKlYAImawWYV8deSYnSY7AMPravUcZxeQHcNktLao7LIivW3cjiYiKipWACJmuHcJgP0mdXo9USA1FFkw6+LGhO2R6M8txobE3hiCBFRU7AAEjXD0eUlCB7ghpA4XvatNQXcoMFtW6OhO23CpjHnOUUMEVEjsQASNZH+nBlZm4244RE/qaPIUlBvN4zfHI3iv6rw0/gsVFfYpI5ERNRmsAASNdHxT0vh6qlAxyk+UkeRrZA4d4z7KQoX0ivwy+1ZsJpZAomIGoIFkKgJbBYRxz4tRef7fKHyVEodR9a0QzwwdmMUcnaUY/sDuRBtotSRiIgcHgsgUROc+9GAinwLuj3M4V9HEDHcE7d+FYHTq3XYPacAosgSSERUFxepAxC1Rcc+unTyR1BvN6mj0P/pOMkHVR9YkfZoHtxClOj/72CpIxEROSwWQKJG0p834/zPRsR/pJU6Cv1N95n+qLhgwYF5hXAPdkG36Zycm4joWlgAiRrpxP/K4OqhQMe7ePKHIxrwYhAqCy1IfSQPmkAXtJ/oLXUkIiKHw2MAiRpBFEWcXFGKDnd68+QPByUIAm7+bxja3+GNLXdlo2BfhdSRiIgcDgsgUSPk76mA/mw1ukzzlToK1UGhFHDrlxEIHuCGH287D91Zs9SRiIgcCgsgUSOcXFEGryhXaId6SB2F6qFUKzDmh3ZQ+ymxaQyvG0xEdCUWQKIGslTakPGNDp2n+kJQCFLHoQbQBLhg3E9RqCqy4ufbs2A1caJoIiKABZCowTI3GGDW2dB5qq/UUagRfDuqkfhDOxTsq8SOGXmcI5CICCyARA12ckUpQge5wbeTWuoo1EjaIR4Y8b9wnPyyDL++clHqOEREkuM0MEQNUF5QjazNRgx9n3P/tVWd7vaF/qwZB54vhG8nFTrd7St1JCIiybAAEjXA6VU6CEoBHSdzTrm2rN+/g1B6woSUf+TCt5Mawf14JRcikieHGAIuLy/H7Nmz8fzzz+PJJ5+EyWS65nobNmzAE088gZkzZ2Lbtm0NWqbT6XDnnXfC29sbffr0wf79+6/63v7+/hAEAYIg4Pvvv7f/C6Q279SqMkSP84LGn5+Z2jJBEBD/UTgCemjw88TzKC+oljoSEZEkBNEBjoieNm0akpKSkJSUhBUrVuDQoUN49913a61z4sQJTJs2DQcOHIAoiujfvz82btyI8PDwOpf9+9//Ro8ePdCpUyfMnz8fBw4cwNmzZ+HhcWkajyVLliA0NBR+fn4AgPj4eCiVDZvgV6/Xw8fHBzqdDt7e3DPkrMoyTPgq9jRGfxOJjnfy6h/OwJhbjbUDzsA7RoWJKdFQqh3iszARtRJuvx1gD2BeXh7Wrl2LxMREAEBiYiKWL18Og8FQa73FixcjISEBgiBAoVBg0KBBWLZsWb3LRo4cibvvvhv9+vXDypUrUVpaimPHjgEArFYrNm3ahJ49e2LkyJEYOXJkg8sfyUfGGh1cPBSIGusldRSyE89wVyR+3w4Xf69E2ux8nhlMRLIjeQFMTU1FYGAgNBoNACAoKAgqlQrp6em11ktJSUFUVFTN17GxsUhLS6t32fDhw2vu9/b2hre3NyIiIgAAW7ZswZ49e9ClSxeMHj0ahYWFdWY1mUzQ6/W1buT8Tq/WIWaCF1zdJf9zITsKHeiOYR9pcfzTUvz1fonUcYiIWpXkW7Tc3Fz4+/vXus/Lywt5eXl1rnflOnUtu9KpU6cwbNgwhIWFAbi0t9FgMCAtLQ25ubkYP348bLbrTxS7YMEC+Pj41NwiIyMb/4KpTSk+WoWSIybETuHQrzPqMs0PvZ4KwO45+cjebpQ6DhFRq5G8AAqCULP37zKz2QxXV9c617tynbqWXWnp0qV4++23r3reoUOHYseOHcjIyLjqJJErJScnQ6fT1dyys7Mb/kKpTcpYo4PKR4F2oz2ljkIt5KY3QxExwhNbpmRDf57XDCYieZC8AGq1Wuh0ulr3GY1GaLXaOtczGAw169S17LLNmzdj+PDhiImJuWaOoKAgTJ48uc5Sp1ara4aRL9/IeYmiiNOrdWh/uzdPEnBiChcBo1ZFwNVTgc13ZvNycUQkC5Jv1YYNG4acnByYzZc+eV8euo2Li6u13ogRI3Dq1KmarzMyMhAfH1/vMgA4evQosrKyMGHChDqzuLi4oHfv3s16PeQ8ig5VQXfazOFfGdAEuCDhu3YoPlyFXU/kSx2HiKjFSV4AtVotEhISak7a2LJlC2bNmgWNRoNFixbh5MmTAFBrfj+LxYL09HTMmDGj3mVnzpzBxx9/jFtvvRXnzp3Dn3/+WXOG8Pr163H8+HEAwMmTJ+Ht7Y3OnTu33osnh3Z6tQ6aQCXCh3P4Vw6C+7lh6PthOPphKU58USp1HCKiFuUQ8wAWFRVh7ty5iI6ORklJCRYuXAiVSoV+/fohOTkZkyZNAgB8/vnnOHr0KMxmM5KSkmrt5bvWsvz8fMTFxSEnJ6fW91u9ejWmTJmCefPmYcmSJRg6dCji4+MxZ84cKBQN78ScR8h5iaKIL2NOoV2iF4Yt4+Xf5CTloRycWqXDHfvaI6g3rxRC5Iy4/XaQAthW8RfIeV34tQLfxp3FhJRoRMRzD6CcWCpt+G7wWZh1Vtz5W0do/Dg3KJGz4fbbAYaAiRzR2XV6aAKV0N7sIXUUamUubgokftcOplIbtk/LgWjjZ2Qicj4sgER/I4oiznynR8wEbyhcBKnjkAS8Y1QYuTIC53404OCbRVLHISKyOxZAor8pOWaC7rQZ7W+X57AAXRI9xgv9/h2EAy9cQN7ucqnjEBHZFQsg0d+cXaeHq5cCkSM4/Ct3cS8HI/Qmd2y9OxtVxRap4xAR2Q0LINHfnF2nR/RYL07+TP83SXQkLJUitt2fC54zR0TOgls4oivoM80oOlTF4V+q4RnhihFfROD8jwb8+V6x1HGIiOyCBZDoCme/10OpFtAukVO/0P8XPdYLvZ8JxL7nCnAhvULqOEREzcYCSHSFs+v0iBztCZUn536j2m58IwRB/d2weUo2TGVWqeMQETULCyDR/ynPr0b+3gq0T+LwL11N6Spg1NeRMJdZsWM6jwckoraNBZDo/2SuN0BQADHjvaSOQg7KO1qF4Z9F4Mx3ehxZViJ1HCKiJmMBJPo/Z9fpED7MA5oAF6mjkANrn+SNHrP9sefpAhQfrZI6DhFRk7AAEgEw663I3VGOmIkc/qX63fR2KLzbq7D1nhxYTTap4xARNRoLIBGArM1G2CxANId/qQFc3BQYtSoCpSdM2PfvC1LHISJqNBZAIgDnNhoQ0EMN7yiV1FGojQjs5YZBC0Pw57vFyNpikDoOEVGjsACS7NmsIs7/ZED0eA7/UuP0eiIAkaM8sf3+XFQW8VJxRNR2sACS7BXsq0BVsZXDv9RogkLAiP+Fw1YtcmoYImpTWABJ9s5tNMAtSImQODepo1Ab5BHmivhPw5G53oBjH5dKHYeIqEFYAEn2zm00IGqsFwSFIHUUaqPaT/DGDY/4YfeT+Sg9YZI6DhFRvVgASdZ0Z0woPW7i8C812+B3wuDZzhVb782G1cypYYjIsbEAkqyd22SAQiUgcpSn1FGojXP1UODWryJRfLgKv712Ueo4RER1YgEkWTu30YCIeA+oPJVSRyEnENzPDf1fCMbvb1zEhfQKqeMQEV0XCyDJlklnRV5aOYd/ya76JgchqI8btk3LgaWSQ8FE5JhYAEm2Ll/9I2ocCyDZj9JVwIgV4TCcq8a+ZF4lhIgcEwsgyda5jXoE9NTw6h9kd/5dNbhxQQgO/6cYOTuMUschIroKCyDJks0qIutnI4d/qcX0eiIA2lvckfJgLsx6q9RxiIhqYQEkWSpMr0RVsRVRY1gAqWUICgEjPo9AVbEVu+fkSx2HiKgWFkCSpfO/GKD2UyJkIK/+QS3HO0aFIe+F4vhnZcjcqJc6DhFRDRZAkqWsn42IHOUJhZJX/6CW1fUhP0SN9cKOGXmoLLJIHYeICAALIMlQ5UULCn+rRFQiJ3+mlicIAuI/1sJWLSJtVp7UcYiIALAAkgxlbTYCIhA5mgWQWodHmCtu+UCLM2v1yPhWJ3UcIiIWQJKfrF8MCOyjgUeoq9RRSEY6TvZG+yRv7JzFoWAikh4LIMmKaBORtdmIqESe/UutSxAEDP0gDDYrsOsJnhVMRNJiASRZKfytElVFVrTj8X8kAY9QV9z8nzCcXqVD5gaeFUxE0mEBJFnJ+sUIlY8CoTe6Sx2FZKrTvT6IGuuF1Jl5qCrlBNFEJA0WQJKV8z8bEHmrJxQunP6FpCEIAoZ9qIWlwoY9nCCaiCTCAkiyUVVswYUDlWjH4/9IYp7hrhj8bhhOfFGG8z8bpI5DRDLEAkiykb310vQvUQk8/o+k1/VBX7Qb7YkdM3Jh0nEomIhaFwsgycb5n40I6KmBh5bTv5D0BEHAsI+0MOtt2PtMgdRxiEhmWABJFkSbiKxfDLz6BzkUr3Yq3PR2KI59UnppDzURUSthASRZuPhHFSoLrTz+jxzODTP8EB7vgZTpuTAbOBRMRK2DBZBkIesXA1y9FAi9idO/kGMRFAKGfxqOqiIL9s+7IHUcIpIJFkCShaxfjIgY4QGlK6d/IcfjHaPCwNdC8Nf7JSjYVyF1HCKSARZAcnpmvRUX9leg3WgO/5Lj6vnPAAT3c8OOGbmwmm1SxyEiJ8cCSE4vN7UcNgsQeauH1FGIrkuhFBD/iRalJ0z4460iqeMQkZNjASSnl73VCO8YV/h0UEsdhahOgb3c0Odfgfj11YsoPWGSOg4ROTGHKIDl5eWYPXs2nn/+eTz55JMwma79xrdhwwY88cQTmDlzJrZt29agZTqdDnfeeSe8vb3Rp08f7N+/v8HPSc4he6sRkaM4/Qu1DQNeDIZXO1fseDgXok2UOg4ROSvRAUydOlVct26dKIqi+MUXX4hz5sy5ap3jx4+LAwYMEG02m2i1WsU+ffqIOTk59S5LTk4WV61aJf7222/i+PHjxeDgYNFoNNb7uIbQ6XQiAFGn0zX3n4BaiP68SXwff4kZ35ZJHYWowbJTDOL7+Ev8a3mx1FGInBK336Io+R7AvLw8rF27FomJiQCAxMRELF++HAZD7etjLl68GAkJCRAEAQqFAoMGDcKyZcvqXTZy5Ejcfffd6NevH1auXInS0lIcO3as3seRc8jeaoSgAMKHcw8gtR0R8Z7o+pAf9j1bgPK8aqnjEJETkrwApqamIjAwEBqNBgAQFBQElUqF9PT0WuulpKQgKiqq5uvY2FikpaXVu2z48OE193t7e8Pb2xsRERH1Pu5aTCYT9Hp9rRs5tuyt5Qjq7waNn1LqKESNctPboVC6KbDz8XypoxCRE5K8AObm5sLf37/WfV5eXsjLy6tzvSvXqWvZlU6dOoVhw4YhLCysUY+7bMGCBfDx8am5RUZGNuKVUmsTbSJythnRjsf/URuk8VPi5v+G4ew6Pc5+zw+bRGRfkhdAQRBq9v5dZjab4erqWud6V65T17IrLV26FG+//XaDnvNakpOTodPpam7Z2dkNfJUkhYt/VKGq2IrIW1kAqW3qeKc3osd7IW12HkxlvEwcEdlPkwtgRkYGtm/fDgDIzMxEfn7Thim0Wi10Ol2t+4xGI7RabZ3rGQyGmnXqWnbZ5s2bMXz4cMTExDToOa9FrVbXDCNfvpHjyt5qhIuHAiE3ukkdhahJBEHA0KVhqDbYsG9ugdRxiMiJNLoAGgwGTJ48GZ07d8aMGTMAAKGhoXjuuecwb948iGLjpi0YNmwYcnJyYDabAaBmCDYuLq7WeiNGjMCpU6dqvs7IyEB8fHy9ywDg6NGjyMrKwoQJExr8nNT2ZW81IiLeA0qV5Du6iZrMK1KFQQtDcPTDUuTtKpc6DhE5iUZvGR9//HH88ssvuP322+HpeWlozc3NDUuXLsV7772H+fPnN+r5tFotEhISak6+2LJlC2bNmgWNRoNFixbh5MmTAFBrnj6LxYL09PSaAlrXsjNnzuDjjz/GrbfeinPnzuHPP/+sOdO3rsdR21ZdYUP+7gpEcPiXnED3R/0ROsgNqY/k8TJxRGQXgtjIXXZhYWHYvn07unXrhvj4eOzYsaNmWXR0NEwmU6OHg4uKijB37lxER0ejpKQECxcuhEqlQr9+/ZCcnIxJkyYBAD7//HMcPXoUZrMZSUlJtfbWXWtZfn4+4uLikJOTU+v7rV69GlOmTKn3Oeuj1+vh4+MDnU7H4WAHc/4XAzYlnsfdxzrCv6um/gcQObiiw1X4pm8G4l4JRv9/B0sdh6hN4/a7CQVwxIgRNcf+DR8+HCkpKQCA0tJSBAUFQa1Wo7xcHsMU/AVyXLufzkfGGh3uz+4MQRCkjkNkF3ufLcDhJcW4+0hHXtqQqBm4/W7CEHBISAhOnz5d6z5RFPHEE0/AZrNh8ODBdgtH1FTZWy5d/o3lj5zJgJeC4R7igp2z8xt9vDUR0ZUaXQBffvllJCYmIjk5Gfn5+ViwYAEGDRqElStXwsvLC4sWLWqJnEQNVp5fjZIjJk7/Qk7H1UOBoe+HIWuzERnfcG5AImq6RhfATp06YevWrSgoKIDBYMBLL72EM2fO4M4778T+/fvRs2fPlshJ1GA524wAgIgRLIDkfKLHeaP97d7Y/WQ+5wYkoiZzaewDLBYLYmJi8Pnnn1+1rLS01C6hiJoja4sRgX00cA9u9K83UZtw83/CsKrraeyfdwG3LL3+3KVERNfT6D2Ao0aNuu6yCxcu4H//+19z8hA1iyiKyNlWzuFfcmqeEa4Y+GowjiwrwYX0CqnjEFEbZNcZcn18fPDBBx/Y8ymJGqXkiAkVBRYWQHJ6PR4LQGBvDVIfyYPNwhNCiKhxGlQAX3rpJQQFBUGpVCItLQ1KpfKat4iICFRVVbV0ZqLryt5mhFIjIGyIu9RRiFqUwkXAsA+1KPqzCoeXFEsdh4jamAbPA5ifn4877rgDmZmZSEhIuPqJBAEBAQF47LHHEBUVZfegjojzCDmeH8efh6XShgnbYupfmcgJ7Hw8D8c/L8M9xzvCK1IldRyiNoHb70acBBIWFoatW7fiiSeewCeffNKSmYiaxGYRkZtWjr5zA6WOQtRqBr4WgjPf6bHrn/kY8708PnwTUfM16hhADw+POsvf2bNncfjw4WaHImqKwt8qUW2wIWI4j/8j+VD7KHHzf8KQ+YMBmRs4NyARNUyT5skoLCxEZmYmzGZzrdnoKyoqsGLFCqxatcpuAYkaKifFCFcvBYL7u0kdhahVdZjkjXYJntj5WD7Ch3tA5amUOhIRObhGF8AVK1Zg+vTpsFqvPQFpQEBAs0MRNUVuSjm0Qz2gcOHl30heBEHA0KVafH3Dafz6ciEGLwqTOhIRObhGF8AFCxZg3LhxGD58OHbt2oUBAwYgKCgIALBjxw6MGzfO7iGJ6mOpsiF/TwVufCNE6ihEkvBpr8KAF4Nx4IUL6DzND4E9NVJHIiIH1uh5ADUaDdatW4fHHnsML7/8MkwmE+6//37cf//9WL58Of7888+WyElUp4J9FbBWiYgY7iF1FCLJ9H46AL6xauycnYcGTvBARDLV6ALo7u6O8vJyAEDXrl2xf/9+5OXlAQDUajV2795t34REDZCbUg5NgBIBPbjXg+RLqVJg6AdhyN9dgZMryqSOQ0QOrEmXggsPD0f//v1x9uxZPP7447jxxhvx3HPPYdSoUTh69GhL5CSqU05KOcLjPSAoePwfyVtEvCdi7/bB3n8VoKr02sdqExE1+hjAF198EWq1GgcPHoRKpcKoUaPw0EMP4dVXX4WLiwvef//9lshJdF1mgxWF6RW4eYlW6ihEDmHwO6H4qvNpHJh3Abd8wL8LIrpag68EUh+dTge1Wo1jx46hb9++9nhKh8eZxB3DuZ8M+HHsedx7Mha+ndRSxyFyCH/+pwi75xTgzvQOnBqJ6G+4/W7CEPD1+Pj4QKPR4LbbbrPXUxI1SG6KER7hLvCJ5WWwiC7rMTsAAT01SJuVB5uVJ4QQUW12K4AAsGbNGuTn59vzKYnqlZNSjojhnhAEHv9HdJnCRcAtH2hR+Gsljn1cKnUcInIwDSqAFosFq1evxuLFi3Hw4MFrrrNp0yY89NBDdg1HVJ+qYguKDlUhnNO/EF0l7CZ3dHnQF/v/fQGVFy1SxyEiB1JvAdTpdBg4cCDuvfdePP3004iLi8M777xTs1wURbz44otISkpCZWUlnnvuuRYNTHSl3NRyQATn/yO6jpveDAUA7H2uQOIkRORI6i2Ab7zxBv744w/06tULEydORFBQEJKTk5GZmYmCggIMHz4cr7/+OkJCQrB161a88cYbrZGbCMCl4V+fjip4tePxf0TX4hbkghsXhODE52XI31MudRwichD1ngV8ww034L777kNycjKAS3sEx44di5CQEOzZsweFhYUYP348PvvsM9ldB5hnEUlvVdfTCBvqjvgPw6WOQuSwbFYR3w06C6tJxOTfO/B62SR73H43YA+gKIo15Q+4dLbvO++8g++//x56vR5LlizB+vXrZVf+SHrledUoPWFCxHBPqaMQOTSF8tIJIcV/VeGv94uljkNEDqDeAhgefvWelbi4OAQGBiI9PR2zZ8+utYzTwFBryUm5NJwVHs/j/4jqE9zfDd0f9ceBFwtRnlctdRwikli9BfDixYsQRRE2m63mJooiunTpgg4dOqCiogIVFRUwGo04ePAgdu3a1Rq5iZCTYkRADzXcgxt9QRsiWRr4WghcNAL2PMMTQojkrt4t5+HDh+Hicu3VPD059EbSEEUROdvL0T5JnsduEDWFxk+Jm94OxfYHctFtuh8PnyCSsXoLoEKhQI8ePeDr61vnelarFZmZmcjLy7NXNqLr0mdWw5hVzelfiBqp8zRfHPukFDtn52PKnx2gVNn1egBE1EbUWwDnzZuH+fPnN+jJTCYTYmNjmx2KqD65KUYICkA7lAWQqDEE4dIJIWv6ZODQu8XoNzdI6khEJIF6P/qNGzeuwU+mVqsxZ86cZgUiaoic7eUI6u8Gta9S6ihEbU5ADw16PhGA314thCHLLHUcIpJAvQVwwIABjXpCFkBqaaIoIifFyOFfomaIezkYal8ldj3B67cTyREP/qA2p+SYCZWFVh7ATtQMKi8lBr8bhswfDDj/s0HqOETUylgAqc3JTSmHQiUgdLC71FGI2rSOk70RMcIDOx/Ph6XKJnUcImpFLIDU5uRsNyJ0kBtc3fnrS9QcgiBg6PtaGLOq8cdbRVLHIaJWxC0otSk2q4jc1HIO/xLZiV8XNXo/HYDfF1yE7ixPCCGSi0YXwJkzZ153mdlsxgsvvIBx48Zh8eLFsNk4pED2VfRHFcw6G8J5AgiR3fR/PhhuQS7YzRNCiGSj0QXwjz/+wLJly/DYY4/hnXfeQWlpac2yhQsX4o033sD999+PpKQk/Oc//7FrWKKcFCNc3AWExLlJHYXIabh6KDBkcSjObTIgc6Ne6jhE1AoEURTFxjxAoVBAEARcflhYWBh+++03hIWFYciQIdi3bx/0ej08PDyQlZWFtLQ0TJ06tUXCS02v18PHxwc6nQ7e3rwkWWvYMPocBAEY/0u01FGInIooitiUeB6lJ024+2gsj7Elp8btdxOPAfznP/+JrKwsGI1GPPfcc3jxxRcBXPoHBQAPj0vDc+3atUNaWpqdopLcWc025O8qR/gIDv8S2ZsgCLh5SRjK8yw4uPCi1HGIqIU1ugB27doV7733HiIiIuDu7o5//vOfOHfuHADAYrHAxaX21eWys7PtEpTowoFKWCpFngBC1EJ8Y9Xo+2wgDr5ZhLLTJqnjEFELanQBDAqqfd3I8+fPw2q1ArhUAN3da8/NVlBQ0Ix4RP9fTko51L4KBPbWSB2FyGn1TQ6CR5gLdj2ej0YeIUREbUijC2D//v1x22234cMPP8TLL7+MQYMGwd3dHTt37kReXh7c3d1rCuHRo0ftHpjkKzfFCO0wDyiUgtRRiJyWq7sCN/83DFmbjcj8gVcIIXJWjS6A8+fPhyiKePTRR/HKK68gODgYn3zyCbZt24bKykrcfvvtSEpKwqJFi3D//ffjrrvuaoncJDPV5TYU7KtExAgO/xK1tOjxXoga64VdT+SjupzTeRE5o0afBXzZuXPnoNPp0LNnTwjCpT0yBoMBXl5eyMvLw8yZM1FaWoqffvoJXl5edg3tKHgWUevJ2mLAxtHncffRjvDvxiFgopamO2vG191Oo9dTARj0RqjUcYjsittvwKX+Va4tOjr6qvu8vLzwzTffYPLkydiwYUODn6u8vBzPPvss/Pz8YDQa8eabb0KtVl+13oYNG7B9+3aYTCZMmjQJI0eObNCyoqIivPXWWygvL8fSpUuv+t6RkZE18xmuW7cOSUlJDc5OrSM3pRxuIS7w63r17wUR2Z9PexX6Jgfh99cvoss0P/h14d8ekTNp8h7AoqIiVFZW1jpIOC8vDxMmTMCFCxca9VzTpk1DUlISkpKSsGLFChw6dAjvvvturXVOnDiBadOm4cCBAxBFEf3798fGjRsRHh5e5zIAOHjwIObPnw8/Pz/873//q/W8S5YsQWhoKPz8/AAA8fHxUCqVDcrNTxCtZ23cGfh0VGHUqkipoxDJhqXKhq9vOA3v9irctiW6ZrSHqK3j9rsJxwBu27YNUVFRCAkJQXR0NGJiYmpugwcPRlFR4y4onpeXh7Vr1yIxMREAkJiYiOXLl8NgqH3w8eLFi5GQkABBEKBQKDBo0CAsW7as3mUA0LdvX/Tq1euq7221WrFp0yb07NkTI0eOxMiRIxtc/qj1mMqsuPh7JSJ4+TeiVuWiUeDmJVrkbCtHxlpeIYTImTR6CPihhx5CTEwM7rrrrqumfCkuLsaHH37YqOdLTU1FYGAgNJpLx3UFBQVBpVIhPT0dI0aMqFkvJSUFzz33XM3XsbGx+O677+pddplCcXXX3bJlC/bs2YMuXbpg1KhR+PLLLxEcHHzdrCaTCSbT/58b6/LE19SyctPKIdrAE0CIJBA9xgsxE72wZ04+ohI9ofLih2QiZ9DoAqjVapGamnrd5Y2d9y83Nxf+/v617rt8Ikld6125Tl3L6pKYmAiDwYBdu3Zh1qxZGD9+PPbt23fNsggACxYswPz58xv82sg+clPK4RXtCu8YldRRiGRpyOIwfN31NH595SIGv80TQoicQaOHgEeOHFnn5KAPPvhgo55PEISavX+Xmc1muLq61rnelevUtawh33/o0KHYsWMHMjIysH///uuum5ycDJ1OV3PjVU5aR06KkcO/RBLyjlKh3/NBOLy4CMVHq6SOQ0R20OgC+Nhjj9U6vu5KFy5cwMyZMxv1fFqtFjqdrtZ9RqMRWq22zvUMBkPNOnUta6igoCBMnjy5zlKnVqvh7e1d60Ytq+KCBSVHTAjn5d+IJNXn6UB4t1dh5+w8XiGEyAk0ugB27NgRjz/+OJRK5VU3rVaLnJycRj3fsGHDkJOTA7PZDAA1Q7dxcXG11hsxYgROnTpV83VGRgbi4+PrXdYYLi4u6N27d6MfRy0nd4cRALgHkEhiSrUCNy8JQ15aBU5/rav/AUTk0BpdAIcNG4aEhATcd999mDZtWs1t6tSpuOWWWxodQKvVIiEhAWlpaQAunZgxa9YsaDQaLFq0CCdPngQAzJw5E9u2bQNw6ZrD6enpmDFjRr3LLhNF8apPrevXr8fx48cBACdPnoS3tzc6d+7c6NdALScnpRx+XdXwCGvYkD4RtZx2o7zQYZI39jxdAJPOKnUcImqGRp8EMn36dEyYMOG6y5tSApcvX465c+fiwIEDKCkpwcKFCwEAX3/9NaKjo9G5c2f06tULDz74IJ555hmYzWa89957CA29dDByXcsA4Pfff8eOHTtw8eJF7Ny5E0OHDgUApKenY+rUqRg6dCji4+Px6quvNjo7tayclHJEJXD4l8hRDH43FKu6ZuDXlwsx5L0wqeMQURM1eiJoURTrnAz0t99+Q//+/ZsdrC3gRJItS3/ejC+jTyHhu0h0uN1H6jhE9H8OvnUR+/99AZMPdkRgT16akdoebr/rGQKuqqpCdXV1rfvqKn+FhYWYNGmSfZKR7OWmlAMCEH4Lj/8jciS9ngyAbyc1TwghasPqLIA33HADbrrpplr3+fn5XfMEEKVSibCwME6NQnaTk2JEUB8NNAFNvmQ1EbUApUqBoe+HIX93BU5+WSZ1HCJqgjq3rElJSfD19a113z333IO0tDT07t37qrn28vPzsXXrVruHJPkRRRG5KeWIvcdX6ihEdA0Rwz0Re5cP9v6rADG3eUPtyyuEELUljT4G8NChQygtLb3uNCvjxo3Dpk2b7BLO0fEYgpZTesKEVV1PY9xPUYhK9JI6DhFdQ3leNb7qfBpdHvDF0CWNm3uVSErcfjfhLOD65sm78pq8RE2Vk2KEwgUIu9m9/pWJSBIeWlfEzQ/G3n8VoOuDfgjq6yZ1JCJqoEbPA1iXgoIC3HffffZ8SpKp3JRyhAx0h8qTw0pEjqzH4wHw66ZG2uw8iDaeEELUVjS6AB4+fBhDhw6Fp6fnVSeBhIeHN/pKIER/J9pE5OwoRziv/kHk8JSuAm5ZqsWF/ZU4/nmp1HGIqIEaPQQ8ZcoUVFdXY+zYsXB3rz08V1xcjB9//NFu4Uieiv6sgqnEysu/EbUR2qEe6DzVF/ueu4D2Sd7Q+PPMfSJH1+i/UldXV/z1119wcbn2QwcNGtTsUCRvOSnlUGoEhA7i8X9EbcWgt0KQuV6P/f++gGHLw6WOQ0T1aPQQ8OjRo69b/gDggw8+aFYgotwUI8KGuEOptushqkTUgjxCXTHwtRAc/agUF36tkDoOEdWj0VvYmTNnYtWqVddcZjabMWXKlGaHIvmyVovI21mBiOG8/i9RW9P9UX8E9NRg56x82Kw8IYTIkTV6CHjs2LEoKCjA888/f9Wy0tJS6PV6uwQjeSr8tRLVRhsiRvD4P6K2RuEi4JYPtFg3+CyOfVKK7o/4Sx2JiK6j0QUwLCwMSqUSgYGBUCj+/w5EURQhCAILIDVLbooRKm8F5xMjaqPCbnJHlwd9sT/5Ajrc7g23IJ4QQuSIGv2XOWvWLNx5553XXCaKIrp3797sUCRfOSnl0N7iAYWLIHUUImqiQQtDkfm9HvuSL2D4JzwhhMgRNfoYwFOnTuHDDz+85jJBEJCent7sUCRPlkobCvZWcPiXqI1zD3bBjW+E4PinpSjYzxNCiBxRowvgm2++ib179153uYcHN97UNPl7K2A1iTwBhMgJdHvYH0H9NEiblccTQogcUKML4OOPP17nmb533HFHswKRfOWmlMMtSAn/G9RSRyGiZlIoL50QUnSoCkeXl0gdh4j+ptHHAHbq1AkffPABDh8+DK1WW3O/KIo4ceIE1q9fb9eAJB85KUaEx3tAUPD4PyJnEBLnjm7T/bB/3gV0mOQD9xCeEELkKARRFBu1b7579+44duzY9Z9QEGC1WpsdrC3Q6/Xw8fGBTqeDt7e31HHaNLPeik/8j+OWD7S44WFOHUHkLKqKLVjZ6TRixnthxP8ipI5DBIDbb6AJewDvvfdeGI1GjBgxAkqlstaygoICzJgxw27hSD7ydpZDtALhvP4vkVPRBLhg0MIQpD6ch67T/aAdwr9xIkfQpAJoMpkQGxt71TK9Xo9///vfdglG8pKTUg7PSFf4dFBJHYWI7KzbQ3449kkpds7Kw+SDHTnNE5EDaPRJIO3atbtm+QOA6upqxMTENDsUyU/O9nJEjPCAIHDDQORsBMWlE0KKj5jw1/vFUschIjRhD2BVVRU++OADnD59GmazGVceQlhRUYFDhw7xesDUKJUXLSg+XIU+zwRIHYWIWkhwPzd0f9QfB14sRMcpPvAIc5U6EpGsNboAPvbYY/jss8+uu3zgwIHNCkTyk5taDgAIj+f8f0TObOBrIcj4Roc9zxRg1FeRUschkrVGDwH/8ssv+P7776HX6/HRRx/hwIEDsNlssNls2LRpE1555ZWWyElOLCelHL6dVPCM4B4BImem8VPiprdCcXqVDrmpRqnjEMlaowtgbGwsJkyYAE9PT0yePBnvv/9+zbIxY8bghx9+sGc+koGc7UZEjODePyI56HK/L0Jvckfa7HxYq3mFECKpNLoAiqKIL7/8En/99Re8vb3h7u6OFStWAADOnj2LH3/80e4hyXkZss3QnTZz+hcimRAUAoYuDUPZCRMO/4cnhBBJpdETQe/YsQPjxo1DVVUVdu7ciZiYGPTo0QMuLi4oLS1Fjx498Pvvv7dUXofCiSSb78SKUmy/Pxf/uNgFboG8SgCRXOx6Ih/HPi3FvSdiefgHtTpuv5uwBzA+Ph5Hjx7Fxo0bMXDgQGi1Wmzfvh1DhgzB3XffjTVr1rRETnJSOdvLEdhbw/JHJDNxrwTD1VOBPU/nSx2FSJYavQeQ/j9+gmgeURSxot1JdJjsgyHvhEkdh4ha2cmVZdg2NQe3bY1G5EgeB0yth9vvJuwBBICffvoJQ4YMQd++fQEAR44cwcMPP4wjR47YNRw5N12GGcYcCyKG842fSI463esD7VB37HwsD1aTTeo4RLLS6AL46aefYty4cdi7dy/KysoAAN27d8fs2bMRFxeH1NRUO0ckZ5W91QiFC6Ad6i51FCKSgCAIGLpUC12GGYfe4wkhRK2p0QXw9ddfx/z586HX6xEVFVVzf69evRAUFIRnnnnGrgHJeeVsK0fIje5QeSmljkJEEgnorkHPJwLw26uFMGSZpY5DJBuNLoD+/v544YUX4OnpWeu6raIoori4mMPA1CA2i4icFCMib+XwL5Hcxb0cDLWvErvnFEgdhUg2Gl0AAwMDYbFYAKDWdYBXrlyJiooKtGvXzn7pyGkV/l4Js86GCB74TSR7Ki8lBr8TirPr9Dj/i0HqOESy0OgCOG3aNEyePBknT54EABQUFOD999/HrFmzIAgCnnrqKbuHJOeTs80IVy8FQuLcpI5CRA6g4xQfhMd7YOdj+bBU8YQQopbW6MnX7rnnHly8eBEDBgyA0WhEeHg4RFGEWq3G/PnzMXPmzJbISU4me6sR4fEeULgI9a9MRE7v0gkhYVjTMwN/vF2EAS8ESx2JyKk1eR5Ao9GI/fv3o7CwED4+Prjpppvg5+dn73wOjfMINU11uQ2f+B3H4HdD0fOxAKnjEJED2Te3AH8uLsZdRzrCt6Na6jjkpLj9buI8gADg6emJkSNH4p577kG3bt1gNBrtmYucWN7OctiqRU78SkRX6f9iMNzDXJA2Mw+8TgFRy2lyAbxSVFQUUlJSMHr0aLz11lv2eEpyYtnbjPAId4FvZ5XUUYjIwbi6K3DLMi1ytpfj1MoyqeMQOS27Xgpu165dGDZsGKxWq72e0qFxF3LTrO55GkH93DDi8wipoxCRg9pydzaytxlxz/FYXiuc7I7bbzvtAbzs5ptvRocOHez5lORkyguqUfyXidO/EFGdhiwOg2gRsfdZzg1I1BLsWgABIDiYZ27R9eWmlAMAj/8jojq5h7hg0FuhOPF5GXJTeYw5kb3VWwDPnz/fqCdUqXhcF11f9lYjAnqo4R7CIR0iqlu3h/wQNsQdqY/kcW5AIjurtwDeeeedjXpCnrVF1yOKInK2GRHBy78RUQMICgHDPtRCn1mN3xdclDoOkVOpdzfM4cOHsXfvXgwYMABKpbLOdS9evIizZ882OkR5eTmeffZZ+Pn5wWg04s0334RaffX8Txs2bMD27dthMpkwadIkjBw5skHLioqK8NZbb6G8vBxLly5t8HOSfZWdNMOYY+HwLxE1mH83Dfo+F4iDC4oQe5cP/LtqpI5E5BTqLYBmsxk333xzi4Z49NFHkZSUhKSkJKxYsQLJycl49913a61z4sQJvPbaazhw4ABEUUT//v2xceNGhIeH17kMALKysnDy5MmrJqqu73FkX9nbjFC4CtAO9ZA6ChG1If3mBeH0Gh1SH8lDUmoMBAWvIETUXA06CUSr1WLw4MEYOnRonbc+ffrAxaVxx3bl5eVh7dq1SExMBAAkJiZi+fLlMBhqXxB88eLFSEhIgCAIUCgUGDRoEJYtW1bvMgDo27cvevXqddX3ru9xZF8524wIvckNrh52P/eIiJyYi0aBYcu1yN9VgeOfl0odh8gp1NvW2rVrh7Nnz0KhaNhG++977uqTmpqKwMBAaDSXdusHBQVBpVIhPT0dI0aMqFkvJSUFzz33XM3XsbGx+O677+pddtm18jfkcVcymUwwmUw1X+v1+oa+TNmzWUTk7ihHn38FSh2FiNqgiOGe6HK/L/b+6wKix3vDPZgnkhE1R72t7oknnmhw+QOAhISERgXIzc2Fv79/rfu8vLyQl5dX53pXrlPXssZ87/oet2DBAvj4+NTcIiMj6/0edEnhr5Uw6208AYSImuymRaEQFMDuOflSRyFq8+ptdnPmzGnUE3br1q1R6wuCULP37zKz2QxXV9c617tynbqWNeZ71/e45ORk6HS6mlt2dna934Muyd5qhMpHgeB+blJHIaI2yi3QBYPfDcXpVTpkbTHU/wAiui7JD8bSarXQ6XS17jMajdBqtXWuZzAYatapa1ljvnd9j1Or1fD29q51o4bJ3mpExHBPKFx48DYRNV3nqb4IH+6BtEfzUF3BuQGJmkryAjhs2DDk5OTAbDYDQM0QbFxcXK31RowYgVOnTtV8nZGRgfj4+HqX1aWpj6PGMemsKNhXgcjRHP4louYRBAHDlmtRnmdB+kuFUscharMkL4BarRYJCQlIS0sDAGzZsgWzZs2CRqPBokWLcPLkSQDAzJkzsW3bNgCAxWJBeno6ZsyYUe+yy0RRvGqS6oY8jpovZ7sRohVoxwJIRHbgG6tG3Pxg/PluES78WiF1HKI2SRAd4NIdRUVFmDt3LqKjo1FSUoKFCxdCpVKhX79+SE5OxqRJkwAAn3/+OY4ePQqz2YykpKRae+vqWvb7779jzpw5uHjxIj788EMMHTq0QY+rj16vh4+PD3Q6HYeD67Dj4Vzk7SzHvSc6SR2FiJyEzSLi24FnYKsWcedvHaBUSb4/g9oQbr8dpAC2VfwFqp8oilgRfQrtJ3rj5v+ESR2HiJxI0Z+VWNv/DPq/GIwBLwRLHYfaEG6/HWAImJxb2UkzjFnVaJfA4V8isq/AXm7o81wQfnv1IkqOVUkdh6hNYQGkFpX1iwFKtQDtLbz8GxHZX//ng+DTQYWUf+TCZuWAFlFDsQBSi8rabETYze5wdeevGhHZn4tGgfhPw3EhvRJ/LSmWOg5Rm8GtMrUYS6UNuanlaJfgJXUUInJiYTe5o+fjAdg/7wJ0Z81SxyFqE1gAqcXk7SqHtUrk9C9E1OIGvh4MtyAXpD6ce9WUX0R0NRZAajFZvxjhEe4C/xvUUkchIien8lRi2Eda5Gwvx/HPy6SOQ+TwWACpxWRtNqLdaE8IAi//RkQtr90oL3R5wBd7nspHeV611HGIHBoLILUIQ7YZpcdMiBzN4/+IqPUMficUSo0CabPzOBRMVAcWQGoR2ZuNEBRA5EhO/0JErUfj74JbloYh8wcDMr7RSx2HyGGxAFKLyNpsRHCcGzT+LlJHISKZ6XCHDzrc6Y2ds/NQccEidRwih8QCSHZns4jI3mrk9C9EJJlblmohKIC0RzkUTHQtLIBkdxfSK2HW2Tj9CxFJxi3IBbcs1+Ls93qcWqWTOg6Rw2EBJLs7/5MBan8lgge4SR2FiGSsw+0+iL3HB7se51nBRH/HAkh2d/5HA6ISPaFQcvoXIpLW0P+GQakWkPoIh4KJrsQCSHZlzKlG0aEqRI3l8X9EJD1NgAuGfajFuU0GnFxRJnUcIofBAkh2df4nAwQFePwfETmMmNu80XmaL3Y9kQ9jDoeCiQAWQLKzcz8aEHqTO6d/ISKHMmRxGFw9FNgxg9cKJgJYAMmOLFU25GwzcviXiByOxk+J+E/CkfWLEcc/LZU6DpHkWADJbvLSymGpEBHNAkhEDigq0QtdH/LD7qcKoD9vljoOkaRYAMluzv1ogGekK/y7q6WOQkR0TYPfCYXaV4mUf+RCtHEomOSLBZDsQhRFnN9kQPQ4LwgCp38hIsek9lFi+OfhyE0px5//KZY6DpFkWADJLkpPmKDPrObxf0Tk8CJHeKLXnADsm3sBxX9VSR2HSBIsgGQX5380QqkREB7vIXUUIqJ63fhGCHw7qbD13mxYTTap4xC1OhZAsovzPxoQMdwDru78lSIix+eiUeDWlREoPWnG/ucLpY5D1Oq4taZmM+msyN9dzuFfImpTAnu5YeBrwTj0ThFyU41SxyFqVSyA1GzZW4ywWcACSERtTu+nAqEd6o5t03JhKrNKHYeo1bAAUrNlbtDDv7sa3lEqqaMQETWKQilgxBcRMOus2PlYntRxiFoNCyA1i7X60vQv7Sd6Sx2FiKhJvKNUGLpUi1Nf6XB6jU7qOEStggWQmiV/VzlMZTbEsAASURvW6V4fdJzsjbSZuTDmVEsdh6jFsQBSs2SuN8AzwgVBfTVSRyEiajJBEHDLMi1cPBTYel82bFZeJYScGwsgNZkoijj7gx4xE7x59Q8iavM0/i64dWUE8nZW4Pc3Lkodh6hFsQBSkxX9WQVjVjViJnD4l4icQ/gwT/R/Pgi/vlyIvN3lUschajEsgNRkmesNUPkooL3FXeooRER2M+DFYITe5I6t9+SgqsQidRyiFsECSE2W+YMeUWO8oFTx14iInIfCRcCtX0Wg2mjDjhl5EEUeD0jOh1tuahL9eTOKDlXx7F8ickpe7VQY/mk4zq7T4+iHpVLHIbI7FkBqksz1BihcBUQleEodhYioRbRP8kb3R/2xe04+io9USR2HyK5YAKlJMtfrETHCAypvpdRRiIhazOB3QuEbq8LmKdmorrBJHYfIblgAqdGqSq3ISyvn2b9E5PRc3BQYtToShkwzdj+ZL3UcIrthAaRGO7dBD9EGxNzmJXUUIqIW599Ng5v/G4ZjH5fi5MoyqeMQ2QULIDVaxlo9wga7w0PrKnUUIqJW0fUhP3Se5ovUR3JRfJTHA1LbxwJIjWLSWZG9xYgOd/pIHYWIqNUIgoBbPtDCu70Kmydlw2y0Sh2JqFlYAKlRzm3Qw1YtosPtPP6PiOTF1UOBhG/bwZhTjVTOD0htHAsgNUrGWj1Cb3KHZwSHf4lIfvw6qxH/SThOr9bhyLISqeMQNRkLIDWYWW9F1mYjOkzi3j8ikq/YKT7o8bg/dj9ZgAu/Vkgdh6hJWACpwTI3GmAzi+hwBwsgEcnb4EWhCOqrweY7s3m9YGqTXKQOAADl5eV49tln4efnB6PRiDfffBNqtfqq9TZs2IDt27fDZDJh0qRJGDlyZLOXlZeXIzIyEqWlly71s27dOiQlJbXgq227znyrQ8hAN3i1U0kdhYhIUkqVAqO/icSaPmewbWoOxm6MgqAQpI5F1GCC6ABHsU6bNg1JSUlISkrCihUrcOjQIbz77ru11jlx4gSmTZuGAwcOQBRF9O/fHxs3bkR4eHiTlwHAkiVLEBoaCj8/PwBAfHw8lMqGXd1Cr9fDx8cHOp0O3t7OvVfMbLDis6ATGPh6CPo8HSh1HCIih3D+FwM2jTmP/s8HYeArIVLHoQaS0/b7eiQfAs7Ly8PatWuRmJgIAEhMTMTy5cthMBhqrbd48WIkJCRAEAQoFAoMGjQIy5Yta9Yyq9WKTZs2oWfPnhg5ciRGjhzZ4PInN+d/NMBq4vAvEdGVohK8cOMbIfjt1Ys4s04ndRyiBpO8AKampiIwMBAajQYAEBQUBJVKhfT09FrrpaSkICoqqubr2NhYpKWlNWvZli1bsGfPHnTp0gWjR49GYWFhnVlNJhP0en2tm1xkfKNHcH83eEdz+JeI6Ep9nwtEhzu9sW1aLoqPcJJoahskL4C5ubnw9/evdZ+Xlxfy8vLqXO/KdZq6LDExEQaDAWlpacjNzcX48eNhs13/Yt8LFiyAj49PzS0yMrKJr7ptqSq14tyPBsTew8mfiYj+ThAEjPg8Aj7tXfHTxCxUlXKSaHJ8khdAQRBq9v5dZjab4erqWud6V67T1GWXlw8dOhQ7duxARkYG9u/ff92sycnJ0Ol0Nbfs7OwmvOK25+w6HWzVImKnsAASEV2Lq4cCiT9EwVRixdZ7smGzSn54PVGdJC+AWq0WOl3t4yaMRiO0Wm2d6xkMhpp1mrrsSkFBQZg8eXKdpU6tVsPb27vWTQ5OfaVDxHAPXvuXiKgOPu1VGL0mEtlbjDjw/AWp4xDVSfICOGzYMOTk5MBsNgNAzfBsXFxcrfVGjBiBU6dO1XydkZGB+Pj4Zi37OxcXF/Tu3bv5L8qJGHOrkZtajth7fKWOQkTk8CJv9cSgt0JxcGERTq/hSSHkuCQvgFqtFgkJCbVOzJg1axY0Gg0WLVqEkydPAgBmzpyJbdu2AQAsFgvS09MxY8aMZi1bv349jh8/DgA4efIkvL290blz51Z65W1DxhodlCqBZ/8SETVQ76cC0OleH2x/IAcX0nmlEHJMDjEPYFFREebOnYvo6GiUlJRg4cKFUKlU6NevH5KTkzFp0iQAwOeff46jR4/CbDYjKSmp1p68piybN28elixZgqFDhyI+Ph5z5syBQtHwTiyHeYS+6ZcBr2gVEr9rJ3UUIqI2w1Jlw/rhmdCdrcad6e05gb6DkcP2uz4OUQDbKmf/BSo9YcKqrqeR8F0kOtzOE0CIiBqjotCCbweegcpbidt3x0DlxXlmHYWzb78bQvIhYHJcp1aVQeWjQNQYL6mjEBG1Oe7BLhi7KQr6TDO23J3DM4PJobAA0jWJNhGnvipDhzu84aLhrwkRUVME3KDB6G8ikfWzAXufKZA6DlENbtnpmvJ2VUB/thpd7veTOgoRUZsWleCFm/8bhj8XF+PI8hKp4xABAFykDkCO6fhnpfDpqELYze5SRyEiavN6zA5A6UkTdj6WB89IV0SP5aE1JC3uAaSrmA1WnPlWhy4P+EIQBKnjEBE5hSHvhSF6nBc2T85CwQFOD0PSYgGkq2R8o4OlUuTwLxGRHSmUAkZ9HYnA3m74cex5lJ0ySR2JZIwFkK5y/LMytBvlCc8IXvqNiMieXNwUGLuxHdyCXbBh9DmUF1RLHYlkigWQaik9aULB3gp0eZB7/4iIWoLG3wXjf4mCzSxi05jzMOutUkciGWIBpFpO/K8Uaj8lYibwAGUiopbi1U6F8b9EQ3/WjJ9vz4LVbJM6EskMCyDVsFaLOPG/MnS6x4dz/xERtbCAHhqMWR+FvF0V2HpvDmwWThRNrYdbeaqR+YMeFQUW3DDTX+ooRESyEH6LBxLWRuLs93qkPJQL0cYSSK2DBZBqHPmgBGE3uyOgu0bqKEREshFzmzduXRmBk1+WYedj+RBFlkBqeZwImgAAJcerkJtajltXRUgdhYhIdmLv8kV1uQ07pufB1VOBQW+GcB5WalEsgAQAOLKsBG5BSnS43VvqKEREstTtIX9UG23Y/WQBXL0UGPBCsNSRyImxABKqy204+UUZus/2h1LNowKIiKTS64lAVBttOPB8IZQqAX2fC5I6EjkpFkDCqVVlMBtsuOFhnvxBRCS1fv8OgtUsYt/cC7BZRPSfxz2BZH8sgDIniiIO/7cY0WO94B2tkjoOEZHsCYKAgfNDoFAKOPB8IUQrMOBFlkCyLxZAmcveakTJERNu/m+Y1FGIiOgKA14MhqAEDjxfCJtFRNz8YJ4YQnbDAihzf75bjMA+GoQP85A6ChER/U3/ecFQuAjYN/cCRCsw8DWWQLIPFkAZKz5ShazNRoxcGcE3FCIiB9X3uSAILgL2PlOAaqMNQ94LhaDgezY1DwugjB16twge4S7oONlH6ihERFSHPk8HwtVDgbRZeagqtmD45xFQurIEUtOxAMpUeUE1Tn2lw8BXg/kmQkTUBnSf6Q+1vxLb7stBVcl5JHzbDq7unLqLmoa/OTL157vFUKoFTv1CRNSGxE72wbgfo5C3swIbbj2HqhKL1JGojWIBlKHKixb8tbQYPf8ZALWvUuo4RETUCJG3emJiSjTKTprw/dBM6M+bpY5EbRALoAwdeq8IEAT0ejJA6ihERNQEIXHuuH13e1SX2/DtwLO48GuF1JGojWEBlJmqEgv+WlKCHrP94RbIQ0CJiNoqvy5qTDrQAd4xrvjhlkycWaeTOhK1ISyAMvPnf4ohWkX0fjpQ6ihERNRM7sEumJgSg+jxXvjljmwcfOsiRFGUOha1AdwFJCOVFy34891idJ/lD/dg/uiJiJyBi5sCo76OhE9sIfY9dwGlx0245QMtXNy4j4euj78dMvLrq4UQlEC/5CCpoxARkR0JCgE3vhaCkSvCcXq1DutuPsuTQ6hOLIAyoTtjwtHlpeg7NwiaAO79IyJyRp2n+uGOve1RVWzF2n5nkL3NKHUkclAsgDKxf14h3IKV6PUEz/wlInJmQX3cMPm3Dgju74aNo8/h4JsXIdp4XCDVxgIoAwX7KpCxRoe4+cE8JoSISAY0AS4Y+2MU+iYHYd/cC9iYeB7lBdVSxyIHwjbg5GxWEWmz8hDUT4MuD/hJHYeIiFqJQnnpuMDxm6NQ9GcV1vTKwPmfDVLHIgfBAujkji4vQdGfVbjlAy0USl7zl4hIbtqN8sJdhzsiuL8bNo05j91P5cNSZZM6FkmMBdCJVVywYP+8C+g23Q8hce5SxyEiIom4B7tg7KYoDFkcir+WluCbvmdQsJ9XD5EzFkAnJYoids7Og8JFwI1vhEgdh4iIJCYIAno9EYjJBzvA1VOBdYPPYs8z+aiu4N5AOWIBdFKnV+tw5js9blmm5SXfiIioRsANGtyxtz1uXBCCv94vwZpeGcjZweli5IYF0AmV51dj5+x8dJzig453+kgdh4iIHIzCRUDfZ4Mw5VBHuIe4YP3wc9h8VzaMOTxTWC5YAJ2MzSpi6305UKgE3LI0TOo4RETkwPy6qJG0MwYjvghHbmo5vupyGgffvAirmcPCzo4F0Mmkv1SIvNRyjF4dwSt+EBFRvQSFgC7T/HDvyVjcMMMP++ddwKoup3HyqzJOIO3EWACdyLlNevz++kUMfD0E4cM8pY5DRERtiNpHiSHvhWHKnx0R0FODbfflYE2fMzj3kwGiyCLobFgAnUThb5XYPCUbMRO90PfZQKnjEBFRGxVwgwZjfojCHXvbQ+2rwI9jz+O7m87i7Ho99wg6ERZAJ6A7Y8KmsecQ0EODW7+KhKDghM9ERNQ8oYPcMTE1BuN+joLCVcDPE7PwdfcMHP9fKY8RdAIsgG1c2SkTvr8lEyofJcZujIKrO3+kRERkH4IgICrBC7fvbI/b97SHT6wKKQ/m4ot2p7Dv3wXQZ5qljkhNJIgc2G8yvV4PHx8f6HQ6eHt7t/r3v/hHJTaNOQ+1nxITtkfDI8y11TMQEZG8lByrwpFlJTi5ogxmgw3tRnuiy4N+iB7n1WZ2Qki9/XYEDlEAy8vL8eyzz8LPzw9GoxFvvvkm1Gr1Vett2LAB27dvh8lkwqRJkzBy5MgWXVYfKX+BTq0qw47pufC/QYOxP0bBPZhn/BIRUeupLrfh9OoyHP2oFIXplXDxUCDmNi90nOKDyFs9HboMsgA6SAGcNm0akpKSkJSUhBUrVuDQoUN49913a61z4sQJTJs2DQcOHIAoiujfvz82btyI8PDwFlnWEFL8AlVetGDXP/NxerUOne7zQfxH4XBxc9w/MiIicn5lGSZkrNEhY40OxX+ZoFQL0A51R+RoL7Qb5Qn/G9QOdXw6C6ADHAOYl5eHtWvXIjExEQCQmJiI5cuXw2Aw1Fpv8eLFSEhIgCAIUCgUGDRoEJYtW9ZiyxxNeUE1Drx0ASs7nkLWFiNGfhmBkSsiWP6IiEhyvh3V6D8vGHcdjsU9J2Ix6M0QCEoBB56/gNU9M/CJ33FsGJWJAy9ewNn1epSdMsFmkXz/k6xJPm6YmpqKwMBAaDQaAEBQUBBUKhXS09MxYsSImvVSUlLw3HPP1XwdGxuL7777rsWWXYvJZILJZKr5Wq/XN+k118dssCIvrRxVJVboMszI31OBvNRyKFQCesz2R59ngzjkS0REDsmvsxp+ndXo9UQgLJU2FOyrQMH+ClzYX4kjy0pQVWQFAChcBfh0VMGrnSvcw1wu3UJc4OqhgNJNARc3AQpXAeHxHlB5KiV+Vc5H8haRm5sLf3//Wvd5eXkhLy+vzvWuXKclll3LggULMH/+/Ma+xEarKLDgx/FZAAC3ICVCBrrjlmVadJjkDY2/5D8yIiKiBnFxUyBiuCcihl+6OIEoiqjIt6D0hKnmVp5z6evc1HJUXrDAUll7z+C9J2Oh6sQCaG+StwlBEGr2/l1mNpvh6upa53pXrtMSy64lOTkZTz31VM3Xer0ekZGRDX6tDeUdo8IDeZ2h9lPCRcMhXiIicg6CIMBD6woPrWtNKfw70SbCahJhqbTBahbhFih5VXFKkv+rarVa6HS6WvcZjUZotdo61zMYDDXrtMSya1Gr1dc8O9neFC4Cp3QhIiJZEhQCXNwEHuPewiT/1x02bBhycnJgNl+aTPLyEGxcXFyt9UaMGIFTp07VfJ2RkYH4+PgWW0ZERETkrCQvgFqtFgkJCUhLSwMAbNmyBbNmzYJGo8GiRYtw8uRJAMDMmTOxbds2AIDFYkF6ejpmzJjRYsuIiIiInJVDzANYVFSEuXPnIjo6GiUlJVi4cCFUKhX69euH5ORkTJo0CQDw+eef4+jRozCbzUhKSqq1t64lltWH8wgRERG1Pdx+O0gBbKv4C0RERNT2cPvtAEPARERERNS6WACJiIiIZIYFkIiIiEhmWACJiIiIZIYFkIiIiEhmWACJiIiIZIYFkIiIiEhmWACJiIiIZIYFkIiIiEhmXKQO0JZdvoiKXq+XOAkRERE11OXttpwvhsYC2AwGgwEAEBkZKXESIiIiaiyDwQAfHx+pY0iC1wJuBpvNhry8PHh5eUEQBLs+t16vR2RkJLKzs2VxnUK+XufG1+vc+HqdmzO+XlEUYTAYoNVqoVDI82g47gFsBoVCgYiIiBb9Ht7e3k7zB9cQfL3Oja/XufH1Ojdne71y3fN3mTxrLxEREZGMsQASERERyQwLoINSq9V46aWXoFarpY7SKvh6nRtfr3Pj63Vucnu9csGTQIiIiIhkhnsAiYiIiGSGBZCIiIhIZlgAiYiIiGSGBZCIiIhIZlgAHVB5eTlmz56N559/Hk8++SRMJpPUkVrMunXrEBMTg4CAADzxxBOwWCxSR2oVZrMZvXr1QmpqqtRRWs3evXvxzjvv4IcffkBRUZHUcVrM8ePHMXv2bLz33nuYNWsWDh06JHWkFvHzzz8jLi4O586dq7nPmd+7rvV6nfn961qv9zI5vn85I14JxAE9+uijSEpKQlJSElasWIHk5GS8++67Useyu6ysLPzwww/49ttvcfz4ccycORORkZF45plnpI7W4t56661rvrE6q08//RRnz57F66+/LnWUFjd16lSsX78e4eHhyMrKwujRo3H8+HGpY9lVYWEhLBYLfv3111r3O+t717VerzO/f13v53uZ3N6/nJZIDiU3N1fUaDRiZWWlKIqiWFhYKLq5uYl6vV7iZPa3c+dOsbq6uubrZ599VhwzZoyEiVrH7t27xc8++0yMiooSd+zYIXWcFpeWliaOHDlStNlsUkdpFe7u7uLx48dFUbz09xsWFiZxopZhtVpFAGJmZqYois7/3vX31+vs719/f72Xye39y5lxCNjBpKamIjAwEBqNBgAQFBQElUqF9PR0iZPZ38033wwXl/+/E1qr1aJdu3YSJmp5RqMR3333HR588EGpo7Sap556Cl26dMFjjz2GxMRE7Nu3T+pILWry5MmYPn06DAYDVq5ciSVLlkgdqUUoFLU3H87+3vX31+vs719/f72APN+/nBkLoIPJzc2Fv79/rfu8vLyQl5cnUaLW8+uvv+LRRx+VOkaLevPNNzF37lypY7SaU6dO4eDBg3jooYewdOlSDB8+HKNHj0ZhYaHU0VrM+++/D5VKhQEDBsDT0xN33HGH1JFahZzfuwC+f1HbwwLoYARBqPkEfZnZbIarq6tEiVrH6dOnERISgp49e0odpcX8/PPPGDhwIIKDg6WO0mqOHDkCf39/9O7dGwDw+OOPw2az4YcffpA0V0uqqKjAXXfdhalTp2LOnDnYsWOH1JFahVzfuwC+f1HbxJNAHIxWq4VOp6t1n9FohFarlShRy7NYLPjoo4+wYMECqaO0qHfeeQe//fZbzdd6vR7jxo3D888/77Sfqi0WS60zIzUaDWJjY1FcXCxhqpZ133334euvv4a/vz9EUcSUKVOQmZkJDw8PqaO1KDm+dwF8/3Lm9y9nxz2ADmbYsGHIycmB2WwGgJrhk7i4OCljtai3334b//rXv6BSqaSO0qJWrlyJQ4cO1dy0Wi0++eQTzJw5U+poLaZnz57Q6XS1pn1xcXFB165dJUzVcoqKinD48OGaodB58+bBy8vL6c4CvhY5vncBfP9y5vcvZ8cC6GC0Wi0SEhKQlpYGANiyZQtmzZp11dCKs3jttdfQr18/VFRU4OzZs/jss8+QkZEhdawWERoaiujo6Jqbi4sLQkND4evrK3W0FtOlSxckJCRg7dq1AICysjJUVVVh7NixEidrGf7+/tBoNMjNza11X6dOnSRM1TJEUaz1X2d/7/r76wWc+/3r769Xju9fzo5DwA5o+fLlmDt3Lg4cOICSkhIsXLhQ6kgt4tVXX8WLL75Y674uXbrgH//4h0SJqCV8+eWX+Oc//4nKykpkZWVh9erVTntcmEKhwA8//IBXXnkF/fr1w4ULF7Bo0SJ4e3tLHc2ujEYjvvzySwDAF198gcceewyBgYFO+951rde7bNkyp33/ut7Pl5yLIF75cYaIiIiInB6HgImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiIiISGZYAImIiIhkhgWQiKgFlJeX4/3330dsbCxSU1OljkNEVAuvBEJEbVZ6ejp+/vlnvPrqq7BarRgzZgwEQQBw6RJWpaWl2L9/Pz777DM88MADrZrt559/xjfffOM0lwYjIufCAkhEbVZcXBzi4uKwfv16/PHHH1i/fj1cXGq/rS1ZskSSbJMmTUJRURF27dolyfcnIqoLh4CJqM2r61q706dPl+yC9RqNRpLvS0RUHxZAInJqbm5umDhxotQxiIgcCgsgETmtQ4cOITU1FTabDZs3b8b48ePh5eWF7Oxs/POf/0TXrl3RpUsXrF69utbjbDYb3nvvPYwcORJ9+/ZF586d8dZbb8Fms9VaLz8/Hw8++CDi4+PRpUsXJCUlITs7+6ocFosF77zzDkaNGgV/f3+89957Lfq6iYjqwwJIRE5rzZo1AACFQoHRo0fDxcUFRqMRGzduxH//+18cOXIEN9xwA+6++2789NNPNY+bMWMGdu/ejc2bN+PgwYNYsGABnnvuOTz88MM16xQWFmLQoEG47bbbsGPHDhw4cACbN2/GuHHjrsrx9ddfY/r06diyZQsefvhhPPPMMzh//nzL/wMQEV0HCyAROY0RI0Zg2LBhGDZsGKKiorBw4cJay318fAAAjz76KABAqVRi4cKFEAQBb731FgBg9+7d+Oyzz/D8889DqVQCAG6//XZMnDgRn376KXbv3g0AeOmllxATE4OkpKSa5544cSKqqqqu2lM4derUmu89dOhQ2Gw2/PHHHy30r0BEVD+eBUxETmP79u01ZwHbbDY8+eST11zv8lQxABAbGwutVotff/0VAPDVV18BADp16lTrMffddx9++OEHrF+/HkOGDMGGDRswZcqUWuusWrWq3oxubm4AAIPB0LAXRUTUArgHkIickkKhwIwZMxq0bkREBEwmEwDg3LlzAC5N5Hyl6OhoAEBZWRmAS0PA1dXVjc51uXxardZGP5aIyF64B5CInFaPHj0atF5paWlNwYuMjAQAnD59GsHBwTXrXJ5qpmPHjgAAf3//aw7jms1mlJWV1XosEZGj4R5AInJ6KSkp2Lt3b83XRqOx5v+Li4tx/vz5mqliLh/T9/fh3IyMDLi4uODOO+8EAAwfPhx79uzBgQMHaq33ySefQK1Wt8TLICKyGxZAImrzLhe6ysrKq5bt27cPycnJGDRoUM19S5curfn/F154AUFBQUhOTgYAJCYm4rbbbsOnn35acxWPiooKvP3223jhhRfQvn17AMCLL74IT09P3HHHHVi7di1+//13vPLKK6isrKw54SMnJwfApeHiy0pKSgBcmkKGiEgqHAImojZr586dWLVqFX7//XcAQP/+/REWFgbg0kkgpaWlOHLkCF566aVaJ35UV1dj1KhRKCgoQPv27bFz504EBATULF+zZg1efvll3HvvvWjXrh3c3Nwwbdq0WtcT7tq1K9LS0vDkk09i2rRpiIqKwpw5c/DII48AAB577DF8+OGHAICHHnoIJ06cgEajwYIFCwAAr7zyCoxGI15//fUW/TciIroWQRRFUeoQRESt4YEHHsAXX3wBvu0RkdxxCJiIZIPFj4joEhZAIpKNy8fdFRQUSJyEiEhaLIBE5PRyc3PRrVs3bN26FQDQr18/rF27VuJURETS4TGARERERDLDPYBEREREMsMCSERERCQzLIBEREREMsMCSERERCQzLIBEREREMvP/AJEKxEmW9OvpAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "id": "a7f7262c",
   "metadata": {
    "papermill": {
     "duration": 0.008412,
     "end_time": "2022-07-05T00:56:39.414917",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.406505",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 3.6 学习率调节器：OneCycleLR\n",
    "在本小节，为了能够快速收敛，实现了学习率适应性变化的Scheduler类。\n",
    "\n",
    "具体使用了OneCycleLR的方法：\n",
    "随着训练次数增加，学习率由初始值先上升至`max_lr`，再下降至低于初始值的大小。\n",
    "<img src=\"attachment:OneCycleLR.png\" width=\"400\" alt=\"图片名称\" align=center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "740a6df3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.433509Z",
     "iopub.status.busy": "2022-07-05T00:56:39.433226Z",
     "iopub.status.idle": "2022-07-05T00:56:39.442454Z",
     "shell.execute_reply": "2022-07-05T00:56:39.441583Z"
    },
    "papermill": {
     "duration": 0.02092,
     "end_time": "2022-07-05T00:56:39.444471",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.423551",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class OneCycleLR:\n",
    "    def __init__(self, optim, max_lr, epochs, steps_per_epoch=1, pct_start=0.3):\n",
    "        super(OneCycleLR, self).__init__()\n",
    "        self.step_num = 0\n",
    "        self.optim = optim\n",
    "        self.total_steps = epochs * steps_per_epoch\n",
    "        self.step_size_up = float(pct_start * self.total_steps) - 1\n",
    "        self.step_size_down = float(self.total_steps - self.step_size_up) - 1\n",
    "        self.learning_rate = max_lr\n",
    "        self.step()\n",
    "\n",
    "    # 学习率的余弦退火函数\n",
    "    @staticmethod\n",
    "    def _annealing_cos(start, end, pct):\n",
    "        cos_out = math.cos(math.pi * pct) + 1\n",
    "        return end + (start - end) / 2.0 * cos_out\n",
    "\n",
    "    # 根据设置的学习率上升与下降段的比例调整学习率\n",
    "    def step(self):\n",
    "        down_step_num = self.step_num - self.step_size_up\n",
    "        a = self._annealing_cos(self.learning_rate * 0.00001, self.learning_rate, self.step_num / self.step_size_up)\n",
    "        b = self._annealing_cos(self.learning_rate, self.learning_rate * 0.00001, down_step_num / self.step_size_down)\n",
    "        if self.step_num < self.step_size_up:\n",
    "            lr_value = a\n",
    "        else:\n",
    "            lr_value = b\n",
    "        self.optim.lr = lr_value\n",
    "        self.step_num += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a70dd88",
   "metadata": {
    "papermill": {
     "duration": 0.00848,
     "end_time": "2022-07-05T00:56:39.461964",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.453484",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 4.网络模型\n",
    "模型的主体部分，通过上一小节中的手写模块实现了LeNet、ResNet9、ResNet18和ResNet50 & WRN50_2的网络架构。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d305d658",
   "metadata": {
    "papermill": {
     "duration": 0.008788,
     "end_time": "2022-07-05T00:56:39.479312",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.470524",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 4.1 LeNet\n",
    "LeNet是最早的神经网络之一，包括C1卷积层、S2池化层（下采样）、C3卷积层、S4池化层、两个全连接层和输出全连接层。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c423dc5e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.499069Z",
     "iopub.status.busy": "2022-07-05T00:56:39.498250Z",
     "iopub.status.idle": "2022-07-05T00:56:39.509174Z",
     "shell.execute_reply": "2022-07-05T00:56:39.508354Z"
    },
    "papermill": {
     "duration": 0.02262,
     "end_time": "2022-07-05T00:56:39.511020",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.488400",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class LeNet(BaseModule):\n",
    "    def __init__(self, config):\n",
    "        super(LeNet, self).__init__(config)\n",
    "        self.conv1 = Conv2d(in_channels=3, out_channels=6, kernel_size=5)\n",
    "        self.conv2 = Conv2d(in_channels=6, out_channels=16, kernel_size=5)\n",
    "\n",
    "        fc1_size = ((32 - 5 + 1) // 2 - 5 + 1) // 2\n",
    "\n",
    "        self.max_pool = MaxPool2d(kernel_size=(2, 2), stride=2)\n",
    "        self.activation = ReLU()\n",
    "\n",
    "        self.fc1 = Linear(in_features=16 * fc1_size * fc1_size, out_features=120)\n",
    "        self.fc2 = Linear(in_features=120, out_features=84)\n",
    "        self.fc3 = Linear(in_features=84, out_features=config.class_num)\n",
    "\n",
    "        self.loss = CrossEntropyLoss()\n",
    "        self.optimizer = SGD(\n",
    "            params=self.parameters(),\n",
    "            lr=config.lr,\n",
    "            momentum=config.momentum\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        conv1_out = self.conv1(x)\n",
    "        conv1_out = self.activation(conv1_out)\n",
    "        max_pool1_out = self.max_pool(conv1_out)\n",
    "        conv2_out = self.conv2(max_pool1_out)\n",
    "        conv2_out = self.activation(conv2_out)\n",
    "        max_pool2_out = self.max_pool(conv2_out)\n",
    "        max_pool2_out = max_pool2_out.view(x.size()[0], -1)\n",
    "        fc1_out = self.fc1(max_pool2_out)\n",
    "        fc1_out = self.activation(fc1_out)\n",
    "        fc2_out = self.fc2(fc1_out)\n",
    "        fc2_out = self.activation(fc2_out)\n",
    "        fc3_out = self.fc3(fc2_out)\n",
    "        return fc3_out"
   ]
  },
  {
   "attachments": {
    "image-2.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAABaAAAABTCAYAAABpnqDYAAAgAElEQVR4nOy9eXBc13Xg/Xu9rwC6sTQ2EjtXgKS4iaRIkdRKyZZsyZI1ciQnHjtfZb5xpmaS1JSnaiY1VXb8pRJ7UkkqcWJP7NiRZUuKJMuibVEURYqiSJFaKFAUAIIk9n1roPf9fX9A3QKa3cBrdINYeH9VqEa/d9995913+95zzz33XEmW5ZMIBAKBQCAQCAQCgUAgEAgEAoFAkGMkWZblpRZCIBAIBAKBQCAQCAQCgUAgEAgEqw/VUgsgEAgEAoFAIBAIBAKBQCAQCASC1YkwQAsEAoFAIBAIBAKBQCAQCAQCgWBREAZogUAgEAgEAoFAIBAIBAKBQCAQLAqapRZAIMiErq4uBgYGCAQCAKhUKsLhMGq1mmg0CoBWqyUcDifOx2KxxPUz08XRaDREIpFZx5KvA9DpdIRCoRtkSnX9XOnT5T+XjEryhNnPvpDz6Z4lExmU3EdpPgCSJDFfqHol98vknrlON9c7XWhapc+81HkqqVOZ3j+T8lR6/1zfW0m9zSRdLp8jV8+Qq9/mfPnMJe98zzLX/dOdm0ueVPdL156nyj/5+uQ2ZOb3mdfHj8ePabVaJEmiqKiI8vJybDZbSnkFgpVMMBikp6eHwcFBwuEwsVhs1m8g/vuY+X9yW5nqd5j8G0/XjqRrd9O1EenSL1YbBnPrk0ryV6LHKOl/lPQrucoHlOtfSvvOXOsKStMp1QEyyTMTHUlpOWZCJnnOV3/jZKKfKi3TXL/LXNe1XI4xcqUb5mKcON995qsT2bSJmbb16fJS2tanuj5Z/pl5perX4uWpUqnQarWUlZWxdu1a9Hp9ymcUCJY7YhNCwYpgaGiI559/Hq1WS0lJCVqtFphWMqLR6KwGfmbjn6yEpOoIlBoU0nVamR5Pl7+Sc/N12vOdn09xUKIIKlFQcpUGlCmSSpU5pffMdTqlCnYmaTMZYCjNMxNjca7LKJO0S1meSvPLtQE6l+Wdq2dQInsu5Floeznf/dOdy9QAnS59qrTJx+b6PvP/+G8zfkytViNJEqOjowQCARobGzlw4EBKmQWClUhLSwtvvPEGRqORkpISAGRZnvUbSPVbSf5NperXlOiGqfJKd/18+SxWG6bk+mzzV5KH0nxy2T/lus/OtZ6U68nqTPLMREfKREdTymLop5nkmWu9KtfvPNfPfDN/e7m413z3ycYpYb78c9XWp0ufnH+q65Pr03x9WTy9SqVClmVGRkbw+/3cc889bNq0KeVzCgTLGWGAFix7xsfH+cd//EceffRRMdsnEAgEAkES586do7CwkAcffHCpRREIsuby5cucPHmSBx54YKlFEQgEAoFg2fG73/2Ow4cP09jYuNSiCAQZIWJAC5Y1LpeLH/3oRzz55JPC+CwQ3EQ6OjqYmJjI+Lrx8XEmJycXQSKBQJCOvXv34nQ6efPNN5daFIEgK9rb2zl16pQwPgsEN5FYLEZrayterzej69xuNx9++CFut3uRJBMIBKl44IEHOHXqFO3t7UstikCQEcIALVjWvPzyyxw+fHipxRAIbjkCgUDGhmRZlunv76erq2uRpBIIBOm4/fbbuX79Ov39/UstikCwIGKxGEePHuXIkSNLLYpAcEuhUqkYHx/POB50JBKhrKyM5ubmRZJMIBCk48iRIxw9elRxyB2BYDkgNiEULFvim8/ccccdSy2KQHBLEg6HefPNN9FoNJSUlHDlyhUOHDhAR0cHRqMRSZK4evUqsixjNpvZtm0bFoslsUmoQCC4uZhMJrq7u6moqFhqUQSCjOnt7UWn0y21GALBLcvw8DDt7e0MDQ2xdetWLl++zL59+xgcHCQajRKJRHC5XAwODlJSUsLu3bvp6emhqKhoqUUXCG5J9Ho9PT09VFdXL7UoAoEihAe0YNnS398vBtECwRKi0WgoLy9nbGwMr9fL5s2baW5uprKykvHxcRwOB/fccw9FRUXs3LmT4uLipRZZILilKS8vFx7QghXLwMAADodjqcUQCG5Z8vPz0el0jIyMkJ+fj9FopLe3F4fDgcvloqamhj179lBcXMwdd9yB1Wpl69atdHR0EA6Hl1p8geCWw+FwCL1PsKIQBmjBskXsjykQLA2yLOP1epmYmOCDDz5AlmXGxsZYu3YtPp+PvLw8fD4fr7/+Oh0dHbjdbpxOJ7IsMzExgdvtFr9fgWAJULJDvUCwXJFlGUmSlloMgeCWIxaL4fV66ezspLu7m1AohNPppKmpiYmJCWw2G93d3Zw+fZr+/n7GxsbweDx0dXUxMjJCbW0tWq12qR9DILjlUKlUIgSHYEUhQnAIli2xWIxIJLLUYggEtxySJLFr1y6AxGcsFmNycpJNmzZhMplmxehsampK/L9z586bK6xAIEggy3LGMTwFguVCJBIRA2mBYAlQqVTcf//9s46FQqFEKA61Ws3v/d7vJc7V1dUBiNAbAsESI+wlgpWG8IAWLFskSUKtVi+1GAKBgGkFx+l0irA4AsEyR/SbgpWKSqUSHtACwTIhGAwCYLPZllgSgUCQDmEvEaw0hAe0YNkiy7JYSiwQLBM0Gk3C40UgECxfRL8pWKnEYjERvkkgWCZYrVasVutSiyEQCOZA2EsEKw3hAS0QCAQCgUAgEAgEAoFAIBAIBIJFQRigBcsWSZJQqUQVFQgEAoFACbIsi6WYghWLCMEhEAgEAoFyJEkS/aZgRSGse4JliyzLYjMagUAgEAgUIkmSWIopWLGIEBwCgUAgEChHlmXRbwpWFCIGtEAgEGSJLMuMjY0xOTmZOCZJ0iyFIPk7THt7pZpkUavVcxqR0l2n9PpMme9+C713JvlC6jLM9T0Wep+bJdtyvWaxyywTeZSmVXr/xfo9pjqeLFM87/hxu92OzWYTq4MEAoFgCZFlmYGBAQKBQKL9n9l+Z9qHztXPKOmrlOhei9GPKpVvIfnCwvTZ1ab33awyyPSa5ZReaVqlZTlffvOdT1c3Ul2XfCz+XaVSIcsyJSUl5OXlzSuzQLBSWHEG6Gg0yvDwMGNjY8Bnsz7pOn0lx1N9T3Usk8HiTFIdS5VXNseU3FeJASw577nKLNW5+PXxRlOSJIxGI2vWrMFgMJAJYkmJYCXw4Ycf8sEHH2A0GrHZbITDYQC0Wm3if0j9G9VoNEQikRvyTL42mfmU3vmuz5R0cqYiE0U5U6V6Icr+QspiIddkUkbZ3Ge5XrPYA7FMyldpWqUyz1cW89XjdNenkjM5r3ia+C7nExMTRKNRdu/ezebNm1PeT/SbAoFAsHicOnWKy5cvU1xcjMlkSrTZM/u0TPWbufoZJXkp6bMzkSmTPj2TtJnqSgvRrRZyzc0yDN8sXVGn0xEKhTK6ZrHfTabllckzKC0jpTLPJ+t8+aSrT6nkTD4W/65Wq1Gr1Zw+fRq9Xs/BgweprKy8IU8RslSw0lhRBugLFy5w4sQJjEYjxcXFxGIxotEosizP+qHPbBTSHVdiOE5Ok6rRmM+Am3zfXBxT0nilkkWJbMn5zFVmM/OLn4tfr9FoErN3fr+foaEh1q9fzxNPPIFSxJISwXLn9ddfJxaLcf/996PRrKjmVCAQrFAikQgXL15kcnKSO+6444bzot8UCASC3CPLMj/5yU9Yt24djz766FKLIxAIbhGCwSBvvfUWe/fupba2dtY5EbJUsNJYMRaTY8eO0d/fz+OPP77UoggWSFdXF//7f/9v/vzP/1zM1AlWPMeOHQNgy5YtSyyJYKXR0dFBcXExVquVwcFBJEmitLQ0Zdqenh7y8vIoKCi4yVIKlisajYZdu3bx/vvv89Zbb3Hw4MGlFkkgEAhWPc888wy33XYbDodjqUURrCDC4TDXrl1j/fr1qFQqOjs7sdlsafU6ofcJktHr9Rw+fJg33niDcDjM+vXrl1okgWDBrAgr4LFjxxgYGODOO+9calEEWVBdXc3nP/95vve97y21KAJBVly5coVwOExTU9NSiyJYgdTW1mK1WoHpgYnb7U6bNhgM4nK5bpZoghXEzp07mZiYoLOzc6lFEQgEglXNmTNnqK6uFsZnQcZotVo2btyYcL6KRCJz6nVC7xOk45577uH8+fNzjhsEguXOsveA7urqoquri7vvvnupRRHkgIKCApqamnj11Vd56KGHllocgWBBXLlyhbKysqUWQ7AMiUQinDhxApPJRFlZGRqNhsuXL7NhwwYkSSIQCDA8PMzmzZsZHh4mGAxiMpk4efIkTU1NfPTRRzQ0NDAxMcHExARr165d6kcSLGNKSkpob2+npqZmqUURCASCVUtrayt33XXXUoshWIYEg0Fee+01ioqKsNlsaLVaenp6cDgciRjh3d3d7Nixg+HhYYaHh1m7du0svc9qtaJSqbBarajV6qV+JMEyxm6309HRwdatW5daFIFgQSx7D+iWlpZVN9scCAT4xS9+wWuvvcYzzzzDSy+9xPHjx2lra+PatWs8//zzvPPOOxw9epQf/ehHdHV1LbXIOWXt2rVcunRp3nQqlQqdTncTJBIIMmNkZISSkpKlFkOwDNFoNNjtdrZs2ZLYjPXgwYPY7XbeeeedxACju7sbtVqNw+FAp9Oh1WrRarUYDAbsdjuxWIzJycmcbiYpWH04HA5GR0dnHdPr9UskjUCQHfFNlwSC5cTk5CR6vR6j0bjUogiWIXq9nsLCQnbu3Inb7aaoqIi9e/ei0Wh4++23ycvLw2Aw0NbWhslkory8HLVaPUvvKy8vx+/3MzIystSPI1jmFBcXMzQ0lPiuUqnEPkSCFcWyr60DAwOrLsaqwWCguLiY/fv38/rrr2MymdixYwejo6PIssz999/P6OgoNpuNTZs2MTw8THV19VKLnTP0ej35+fnzGvFisVjGO/gKBItNfIMvrVa7xJIIliOxWAyPx8PU1BQ6nY4TJ07Q2NiIzWZj3759jI2NEQqFKC0t5cqVK5SUlCDLMqFQiJaWFsbHx7l27Rrj4+O4XC4mJiZEzHxBWnQ63Q2TFMFgUPH1/f39yLKc2Eg4vrHyfBs1J39PTp/uWKprF5o+3fG58srmulyfy7WMmZRrqmNKNuOe+X1m+vhxSZIAsNlsmEymlPLPRTQavWGzbIFgqQmFQhgMhkXLPxaL3dC+in5/5RCJRHA6nXi9XgKBAKdPn2bjxo2Ew2Fuv/12JiYm8Hg8rFu3jvb2dtRqNcXFxYTDYVpaWhgbG2NiYgKTyYTH48FisYh2UJAWnU43yz4Si8WIRCKKro1EIgnj9cx+e+ZnnPm+p9IZstVD5js3l250s3W7dCw0v0x143TnMim7+XT5dO88uc6oVCpKSkoUT4QsewN0KBRalbM6Ho+HwcFB7HY7LpcLp9OJTqfj6NGjbN++HbvdzvDwMDabjaqqqqUWN+eo1Wrh2SdYkUiSlBhkCwTJqFQqDh8+nPj++7//+7PO19fXJ/5ft27dTZNrpRGLxQgGg4m/SCQya5Aei8USSpFOp0Ov12MwGBL/30pk2h6Fw2FefPFFWltbqaioIBgMJgY08c9kpTN5wGMwGAgEAonvGo3mhgGQWq1OOYhOzmuuPOZKn+44TE90pzPEz3VOq9Wm1U3mOjeXLOmea65BRXL5KrkunQyp3kOq/FMdS7525r1npo+XjUajQa/XMzQ0RF5eHocPH2bz5s0pn0MguBUIBoNMTU0xNTWFx+PB6/Xi8/mIxWK43e5E26DT6YhEIsiyjNFoxOfzodFoMJvNSJKEwWBAr9djMpkwGAwYjUbMZjMmk2lBkz2C3KHRaBJhJdNtCrxx40Zgeg8QQWoikQihUGiW3ifLckLfi3+qVCr0ej16vR6dTofBYLilnIIWMg4dHh7m6NGj9PX1UVlZic/nS/Tv8c9kHSdZd0n+nkqXykS/m0sHSqc3zaUbzaWHpdNH55NjIQbouQzhc8mY7tkylT2TsptPl0++d/IYIf5pNpvp7e2lsrKSz33uc5SWlqaUN3GfOc8uA4LB4Ko09lgsFsrKym6I2/jNb34TgI6ODhwOB5s2bVoK8RadeKciEKw0ZFnG5/Mt+Pr45iJutxu/359QtkKhUOJ/mA7Vk9z2xQchsVgMjUaTWL4X/9PpdIm/uGIW/9TpdKuyLRWsHGYalP1+P4FAgEAgkPg/Eong8XgIBoNEo9FZhmW9Xp9QduJ/8WWHXq93Vt7hcBi9Xo/FYsFms836W43L+/1+v+K0LpeL73//+9xzzz3s2LFDtAmCRcPlcnH27FnGxsbSGmUEgpWALMuKV5aMj48zOjrKyMgIbrcbn89Hfn4++fn55OXlYbfbE8ZjjUaDRqNJ2S/FYjHC4TDhcDjRt8X7TI/Hw+joKNFoFKfTic/nw2w2Y7FYMBgMmEymxP9GoxGDwYDZbBZe1YKbSrIjwUx9L/5/fCImGAwiy3LCsGwwGFCr1bP0vZmfPp9vVt7RaBSLxYLRaMRut1NQUJD4XG3EV00qpaOjg5dffpk777xT9MWCRcXpdPLzn/+chx9+mIaGhrTplr0BejUSCAQoKysjHA6nXdIVj30ci8WEwiAQLDOUxAH0er1MTk7icrlm/cViMfLy8igsLEStVqPT6bBarTcYjNPNusqynBiUxL0FZv7vcrkSxuyZxm0gMcOd6i/uNRpX8OJ/M7+r1Wo0Gs2smd34UhxJkhLt1czPeJ7zLUGfb9lPJsdytYxJybm5ZrqVonSGXem9Mpmxz1R+SZKIRCJEo1FisVjic+b/kiTh9XpnebKEQqFZA4t4PM14zOv4/zO9WhZK3Fjg8XhwOp1MTEzQ0dHBxMQEFosFu91OUVERpaWl5OfnL/g+ywWlcUknJyf527/9W55++ulFlmhl0tHRkRi0KiUSidDW1obVamXt2rXCoD+DuAf0qVOnCAQC3H///UstkkCwICRJSruyxufz0d/fz8DAABMTExgMBkpKSqiurqa4uHjBcaNnenlaLJZ508c9q30+H16vF4/Hw9DQ0KwJ3pkTu/HPuAE8rgfGPUxnTvICNyzRnxmyKfkzXaieVM84V2gnUKYbznU81/pgLs4rTZNJukx0uUz1PlmWb9D1Uul+gUBgls4Xr2/xyRaDwZCYFLFarYn/4/U8mxXv0WiUcDjM5OQkTqeToaEhWltbmZqaSjggFBcXU1ZWtuJXDEiSpNjju7W1laNHj/LII48sslQrD6U6X19fH2q1msLCQtra2hgZGWHz5s2UlZXdJElXDjabjS984Qu88sor3H///TQ2NqZMt+wN0PGlR6sJg8Ew75LEysrKmyTN0qAkllomDaxAcDNJ5XHodDoTXi+jo6MJA1t+fj52u52amprERiRLRdxIHYlECIfDs4zXM42K8aVu4XA4oWDGBx9qtTqhuMaV0riBOhKJJJbvzFzGky6+2EwleL6lXpB6GVKqY+mWH6U7nixLJufmWhqlFKWDgblCAMxkrudMJlP51Wo1siyjVqsT712r1c6atJg5qTFzoHuziK8UMBgMFBUVzTo3NTWF0+lkamqK06dPE41GqaiooLy8nIqKipsmYy6Z2R7NZSh55plnePjhh2+WWCuOQCDA5ORkRgZot9tNdXU1b775JjqdTgxIUnDo0CFOnTrF1atX5/SIARLtikCwnEj2gHa73XR1ddHb20swGKS8vJy6ujoOHDiwZPXXbDZjNpvnTDPTmzqu/83UAeO6YXK4g/g4PK7HxfXCuK6RvJQ/nW6XrAMq0fuSSWeQzVTvW2i8WCVyKtGrlOppuU6XaVogUafj7XPceSZZD0xegZmNI0GmxDewLS0tnbX8X5ZlJiYmcDqdjI+P09zcnNj0sby8fEVuKB93RIqTzl7i8/n49a9/zZe+9KWbKd6KQanOF7fJtbW1YbFYWL9+/S0X6i9TvvCFL/CrX/2K6urqlBOoy94A7fV6RaiGVUi6+D0zSW5gBYLlQiwWY3h4eJbB2Wq1UlxczJo1a9i+ffuynGGPL/cUCG5V4kuhAbZu3YrH42FgYID29nZOnTpFeXk5ZWVllJeXk5eXt8TSZk66peJdXV0AWK3Wmy3SiiIcDvPmm2+i0WgoKSnhypUrHDhwgI6ODoxGI5IkcfXq1UTMu23btmGz2XA4HKvCm36xqK6u5r333pvXAB2fBBUIlhuhUIi2tjY6OzsJh8PU1NSwd+9ebDbbUoummPjksBKPaoFgNSBJEoWFhRQWFgKwe/duJiYmGBwc5KOPPmJycjJhjC4rK1vwioWlJJ295OLFi6xZs2YJJFq+vPfee2i1WoaHh1mzZs0snc/hcODxeNDpdGg0GiRJSqwsLi0txe12o9fruXDhAuXl5fh8PsLhMC6Xi8HBQUpKStizZ8+8E4G3ChUVFVy8eJEDBw7ccE5YIgQCgUABsixz/fp1rl+/Tnt7e2I518aNG7nzzjuFYVcgWIFYLBbWrVuX2BByYGCAgYEBTp8+jVqtpq6ujrq6umUdO1rJgKmnp4fi4uKbIM3KRqPRUF5eTktLC2azmc2bN9Pc3MzGjRtpb29n8+bNVFVVcfHiRTZv3ozNZmNoaIiqqqplOem4XCgrK+P9999fajEEgowZHR3lzJkzXLlyhZKSEnbv3p0wZgkEgpWH3W7HbrezefNmwuEwAwMD9Pf3J/r92tpaqqqqllrMtChdId7T0zPvZnC3EsFgkK6uLh566CG2bdtGS0vLLJ3PZDLh8/lobm7mvvvu45VXXmH//v3Y7XZUKhVmsxmDwYBGoyEUCjE1NcXmzZsxGo2cOXOGO+64Y0lXOS83ysrK6O7uFgZogUAgyJSuri7a29vp6uqirq6OhoYGNmzYwD333DPnddFoNLGpRhylMYzToTQW3nz5L2RX31yQy7h4ubp3fDlprj2CvF5vxrEKM023VO9xISy0Hmb6jOniis/1LlQqVUJpjHvCwPRmUh0dHbzwwgvU1NTQ2Ni4LD0blGxC6PF4lqXsywVZlvF6vXi9Xtrb2zEYDIyNjdHU1MSVK1fIy8vD5/Px+uuv09jYiNvtxul04na7aWtrQ6PRIMuyCMGRBr1enwjztJwncwSCOH19fVy8eJFQKERlZSU+n48dO3bMeU0gEEgbeuFm9vu5ji+cSxa731eajyRJOZ80dLvdiTBlc5HLcl9oXjfj3efqHtmMk2Yem/lpMBgSoUOqqqoSBueBgQGuX7/Ohx9+SH19PY2NjcsuFKzSFeIul4va2tqbINHKQK/XY7VaOXr0KHV1dUQikVk63+DgIBUVFWzdupWBgQHuuusuRkdHMRqNRCKRWaGLtFot3d3djI+P09jYyNjYGB6PRxigZ2A2m3G73SnPzWmAHh8fz2jAGD8ny3LOPG2UePZEIpEbHlBpB67RaG7qktTJyckbYqEqbVhv5jKvaDTK1NTUvI1uKtnVavW8S5dzGRdqampqQUs2ZVnGZrPdtI4lFovhdDoXFB9OpVLd1OW9mZZpvB7EYjFsNttNi4EnyzJOp3NB9XS+Mm1paaG5uRmHw0F9fT333Xdf4p5zce3aNa5evcrk5CQmk4loNIpGoyEYDGIwGAgEArNiz+l0ulm7Gc+UNZ4+TvL35Gvhxthuer3+hiX5N+OYEtlSxeBLfsZUaZLzSVdmM9PNlCd+XKVSodPpUKlUuN1uysvL2bFjR1aGunPnztHX14fRaCQajRIKhW6IizhTxplll/zs85VXchknX58qTXJ5pcoXlMfbVhqDW2m9SZZlvt/AXGU281y8HOLH4veOG8M0Gg1VVVVs3bo1kVd82eauXbu4evUqx48fp6SkhG3btq04b9e4biZIjSRJ7Nq1CyDxGYvFmJycZNOmTZhMJo4cOZJI39TUlPh/7dq1N1fYFUo8pqxAsJwJhUK89dZbSJLE9u3bqaioYHh4mPb29rTXNDc309XVRTQaTWzUG+9j4p8z++KZfVPy3g7JfVpyP5n8PVUfrFQXSNVXZ3Msuf9OpXsr0QWSZU1+xrnKLJV+Fb8+LnNc7wsGg1RWVrJnzx4Wisfj4f3332doaAiLxTJr/5SZcqfSQZOfPbm8Ur3b+XSkVGmU6mSp8lJ6TGl+873bVGnm+k0kl9nMa5N1boPBQCQSQZIkfD4fNpuNdevWUVdXl7g+7oQQiURoaWnhF7/4BY2NjWzZsoWVhuhvb2SmHhcnrvOlYuPGjWnPbdu2LfH/zDokmGaucUdKA/Srr77K9evXKS4uTuxmm/xDNxqNCa+beEMQd123WCx0dXVRVVXFoUOHsnL/n8uz59q1a5x/9yMmJiax2x0EAwG0Oj3hUHBW4xRveGd2WJIEWp2eSDRELBpk3bpa9h/Yu2A55+P06eOMjFylpMQMRAGZUCjyqXxqIpEoBoOOQCCunEz/bzTq8fuDmM1Grl0bpaRkDdu2HcThcCyKnF1dXbS89hucQwNUGXT4QiEMGg2BSASjVov/0/LTqdWEotHEJ4BKktCpNYQkiUmdgYqtt7H3nvtS3idZCUrHXIbFEydOcf1aDyZTHsQgBkQ+lU+lVhOLRtHqdIQ/vZdOpycUCqLXGwgGAxhNJkZG+igvd3DH/t0Jr7dc09PTQ0vLGcbHh6itLcTrDSTe78x3rtNpCYXC6HSaRN1Qq9VoNCqiUYmxsRDl5evYt+/wosgJcPb1Y/R//BGF4SCaWJRILEbkUyVGq1YTTnrnBq2WQDicqBsmvZYuX5jCigoaH3ho0WJPDQwMcPzds3QPD2IpLyXo96PT6wkFg4k2AEi8/5kKjlqtQVJJqIGY28eWmlruO/hZmV65coV33nmH+vp6HnzwwRuM1JIkpTVQvvXWW8iyTH19fUYbWQmWnlAoxOTkJEePHuXQoUMZt7E+n48XXniB2267jb17967IOHK3MlNTU4yNjfHrX/+ahx566Ia+p6GhgYaGBjo6Ojh27BhVVVVs3759iaSdTXJdE5u45Yb4pLGIoXjzkCRp2XmbCW4dmpub+fDDD7nzzjtnGRTSbe4ai8V49TnO+mgAACAASURBVNVXKS8vZ8uWLSty34BbGZ/Ph9Pp5Gc/+xmPPfZYxhPLg4ODnD59mqamJtavX39TN90TZM/ExATd3d0MDAzcECZAo9GwZcsWtmzZQktLC88++yw7duxg/fr1SyTtZ0iSNKuuiX5TsNKYZYD2er38/d//PV/+8pe59957s3Ijl2WZoaEhXnvtNfbv3099fX3Wws7k7DsX6O9z0bjpIHZb0fwXzIHX62FopJef/usvefqrX87p4C0QCPDyy//Ef/gPW6iouAOTKbtdM4eGnBw9+iqBwAGqqnLbCH5y+hS+t17n0RI7pQ2VWeXlDobouvQuL33czMN//N8WHB831cxJKBTiRz98htu2HOTQnbswGrLzRJtwjnHi+Dlu276OxqYNWeWVzJUrHzA5+TGPPbae0tLsjBUej5+enjFefPEHPPzwNxTFf1JKOBzmV9/7Sz5vhIfteVh02SnRwx4vJ376z/jvfoB1e+/IkZTTvN/awonWZipv38G2wkNZ5RX0+ekeHuH/+5d/5k+f/ho6nQ5ZlnnqqafSKpLxpdrJnDt3DpVKxaZNm7KSSbA06HQ6SkpKuOeeezh58iT79u1TPCnl8Xh49dVXefDBB4Xxb4US35zQYrHw8ssv8+ijj6ZMV1tbS21tLS0tLbzwwgvs2LFjyZc4Jk/Up1q9kumu94LpMhNeLbnDbDbPO0gWnvqCpSAcDvPb3/6WkpISvva1r91wPt3mri+//DKNjY0UFWU3DhUsDSaTCZPJRGlpKb/61a/44he/qNgI3d/fz7lz5+YNxydYvsRjQV++fJnz589z++23p0y3adMmNm7cyPvvv8/vfvc7duzYQUlJyU2W9jNkWZ7lyJeu3xThIARLSbqJW4DESNnj8fDDH/6Qb33rW9TW1mZdaSVJoqysjD/4gz/g7NmztLS0ZJXfTM6ceZehfh97dt+dtfEZwGy2UFezka2Nh/nxvzybAwmnCQQC/PrXP+B//I97aWgoz9r4DFBaauMb39jPwMAp2ttzV6aXTp4geuo1Hq9dQ6kl+ziRVr2OJkcxv29R88Jf/kUOJJwmEonwg3/4KUfu/T3WrKnN2vgMYLcVcejOh/jog+s0f3Q5B1JOc/nyeQKBj/m939tDaWn24VMsFiObNq3hm9/cy/PP/10OJPyMX377z/nPZflsLC7CkoMZfIfFzFdqKnD97le0vP1WDiSc5v2Pmzl25WM2Pngv1sLsPYz1JiOOmirWPfZ5vvvjHxKJRNiwYUPGXgzt7e0Eg8Fbzvjs8/mYmJhYajFyzuHDhzl58qRig93x48fZt2/fLWV89nq9tLW1pY3vtVJxOBxUV1dz7ty5OdNt2rSJL33pSwwNDXHixAlCoRA/+MEPlu1Ga5FIZEGhqgSCXOH1eoVxWbDs6O7u5mc/+xl79+5l717lK2HPnTtHXV3dLWd8npqaWnX9vlqtZt++fRw/flxR+nA4zMmTJzl8ePFWpC5HJicnaW1tvSGsxkqnsbExEQs4HfEwXQcOHKC5uZnz588D8O1vfxun03mzRM2I1faeBCuLdBO3MMMA/cILL/D0008vigBf/epXOX369II6rOSZyM7OTvp7XezadTBX4iUoLCxhXd0u3njjZE7ye/PNX/Ef/+MeNJrcb7jy9NMH6eo6w+TkZNZ59fb24jv7Fg/X5X7H10KTkUetGk7/+3OzjqebEZmPo6/+jv37Prcoy5wOHnyQix+2MzY2lnVeg4ODTE628KUvLTyuWDrMZgNPP72DN9/8dU7ye+MXP+dpRz6mHHpUx3lsfS2jb59gaGgo67xGR0d5s+0yTUfuzoFks9Hq9dQeOcwLv/vNgq6/dOnSkntBLgUul4vOzs6lFmNRqKqq4vLl+Sek+vr60Ov1Od/EcLnj8/moqKjg3XffXXWetVVVVVy/fn3eUFEqlYp9+/axadMmnn32WV555RX+6Z/+KSf9skAgEAgWlwsXLtDa2srXv/71jDwag8Eg169fv+XC88iyzMjICIODg0stSs6xWq1oNBr6+/vnTXv58mVqampuglTLh/gkdmFhIRcuXFhqcXJOXV2dIp3fYrFw7733Yrfb+Zu/+RveeOMNvvOd79wECQWC1YMKpmd/DQbDos7ibtu2jQ8++CDj63w+36zvH35wmdqaxfMyrKlZR1tLp+IYxekYGhrCavXkxPM1HXv2VNDa+lHW+bSfepNdeYsXr3R9kZ3xlsuzwhakmxGZi/HxcSbGA5SULN4u89VVG/no4sdZ59PWdpE9exZPMa2tLcXv78PlcmWVz9TUFMGrrVQXLN4Gh7eb9bSeeyfrfN691Ez+htyG8plJQamDLs+UIo/emZ6uPp+PcDic1cZ1y5FgMMixY8e4cOEC7e3tdHZ2cubMGdra2uju7mZsbAyz2bzg8DrLncLCQoaHh+dNNzQ0tCrjfbe1tXHq1ClaWlro7u7m9OnTXLt2jc7OTlpbWykuLkaj0VBeXr4q64DNZmNkZERRWkmSOHnyJH6/n6tXr/IP//APiyzdjfcXCAQCgXJee+21GzYXTUdyjNXR0dGbujH8zWJwcJDXXnuNy5cvc+3aNdra2rh48SKtra1cu3YNn8+HyWTKaQjA5YTNZlOk9w0PD1NYWHgTJLq5xPW+5uZment7eeONN+jq6qKrq4vu7m7sdjtarZaKioqlFjXnWCwWAoGAYq/hkZERTp6cdlh8//33ef755xdTPIFgVaGCaQ/YxZ7FXbNmDb29vVnnMzAwRHHRwjc1VILd7sjaY3NoaIjq6sXdjKKyspDx8Z6s8xnt6aIyb3G996rUctZlOjg4iK1g8YzPAMVFpfT1z698zIfT2UNl5eIqJ/X11qy9EAYHB6mNZTfZMh+VeRbG21uzzqdzqJ8Cx+LG3FKX2BXV05lL2T0ez6r0ftXr9dhsNrZu3YrX68Vut7Njxw40Gg3nz59ftQOQOFarFY/HM2+61fr+rVYrVVVVFBYWEolE2L59O2VlZbS2tuJ2u4nFYgwPD6/aGLlWq1Xxqq2ioiIefPBB9u3bh8Ph4K233uLVV19dZAk/Q0lYA61Wi1o9/2osESIh94gyncZisSz5ZIl4F7lnJZbp888/z4YNG2hsbFSUPjnGqsfjwWq1LpZ4S4bFYqGsrIyGhga8Xi8VFRVs3LiRQCBAS0uLoj5kJWOxWBTpfV6vd1XrfTabDVmW2bNnD/n5+bz33nvEYjGi0SiTk5NUV1cvtaiLgsViUezYtWXLFh5++GGampooLS3lueeeo7m5OWsZctmeKt0MfSW24ZmwFM8nynSadCGdNQButzutAfrjjz+msrIy65ne/Pz8GzbKmY+pqalZ3/1+P2aTadE7QKMhX1EHNJOpqSny8z/zIvX7/eTnz+8R2dk5hEqloqoqc8NaQYGZUMg3f8IZJBtLQqEQWrUGfYYebOM+P8NeH5uKlRlZi4jhzLBMXS7XLCOX1+vFYl1co77ZbCXgzyxmktfrxWg0JrxiZVkmEolgsczd8MdiMS5e7GD9+op506bCbrfQ3Z1ZWJvJyUkKCgpmyV6uzyycSUyW+Xh4jDp7vqJ40RadjkgkiizLigefPp8PnU43y7PSHwhgsGan8Hknp3CPT1Bal3rpnCkvL+NQQUs9oF4s4oqm3+8nEAjw7rvv0tDQQDgc5rbbbsPtdiNJEl6vl1gsturiH2dSX1ejohEOhxkbG6O+vp4LFy6wYcMGCgsLaWpqYmBggEuXLiHLMqOjo2zdunXV7f4ei8UUv3+NRsMTTzzBE088weDgINeuXaO9vZ2WlhYCQ614nNOTWm5vAI1aIhKVsZj0eHxBtBoV4cj0hJbVbMDtDWA0aPEHwljMepDB7KijasNOiouL55Ul3aYf4XB4zhjQb59+h97eIQL+IB6vG7PFitfjRq83EAxO94nxYwAqlZpYLIrJZCYUCqHXG7BYjKytKmfv3tQb+eSC69ev09d3GadzGEkCl8uLRqMmEolitZpwu32o1Sqi0elnzcsz43J5MRh0BAIhrNZpvSw/v5KGhi1UVma34fJcnD/9FhOXP8Lv8eLxB7CajLh9frRaDeHwdNia+DEg8RwWkwF/MIRRr8eYn0/hpiZ27z+waHJ2d3fzflsLPaPDIIPP40Gj0xIJhTFZzPg8szfdNVks+DwetHod4WAIk9UCMqx1lLJz/UaqqlKHc/N4PPO2lRqNZk4d/+zZd+npHsDnDeDz+zAajXi9HjQaLZFIGACLxYrn03qq1eoIh0NYzFa8Pu/0pmNGI2ury7jjDuXxfjOlr6+Pq1cvMTnZi0ol4XJ50et1BIPTddDtni5TlUpFLBZLHIvXjel6KmO3l1JZ2bSoIb7ee+cM459cwj81hT8UwqDV4vEHZv2OZtZTvU5LMBROHMszm9AZjdg3bGLP3fcumpwDAwO898nHXB8aQELC5/GgMxgIBQKYrBZ87ukxhlqrIRqOYDSb8Xu9qDRqYpHodD0FVMEw+7dtz8qIJknSqoypH+/3JUliYGCAyclJqqqqMJvNNDQ04HK5cLlcRKOZ6fQrhUyeZzXrfeXl5bzzzjts2LCBgoICdu3axfj4OB999BFqtZrLly/T1NS06vR+SZIUP5PVauUb3/gGkUiE/v5+Ll++zJUrV9ASxTfchs89QTQmE43GCIam+3udVkMoHMFi1OPxT68Ej+uCVrMetzeI1WJAUmkxlzaw+bZ9ikKWqlSqlCsR57K7eTwe3n3/LbqGm5FjMh63D5PFiM/jT3wCGIx6Av4gOoOWUGC6j9VqNWh1GiSVClt+Eesr97JlyxZF5bYQmpubGe3sx+NyE41GCUfChMLTsuh1eoKhICaDEV9gWmar2YLb60l85lvzkCQJW0UJ23fvXLQVyz6fj3dPv0vvlV6i4ShurxuLyYLH58FsNOP1T/f7JoMJX8CHVqsl/Olz6HQ61Co1arWafHs+9dvr2bZt26LICdNhhDr6rjHpniQWixEKhhLhFA0GPYFAEIPRkLCJWSxmPB4vZosZr8eL1WoBSaK8uILd229POyGbbkWBBqYHWuka0qamJsUPE4lE6OnpSakoybKcKGSlfP/73+f555/H6XRy6NAhioqKiESiGeWxEGQ5M6Wir6+Pb33rWzQ0NPDwww9TX1+f6JznIxSK4PeHFmSAlmUyjr354x//mPPnz/Pwww9z++23U15eTiS2sDIdcHkUG6BjGfbTfX19fPOb30SlUmEymWhoaJj2QMg0owWQaZn+27/9G8ePH+fxxx9n9+7d1NTUEArNn4dKpWJiwk0wGF6QAToazawsRkZG+JM/+RPKy8v53Oc+x9atW5FlmUxVaJUkMer1UZlnwaLQ5hTOUFn9+c9/zrFjx3jggQc4dOgQdXV1RMORnCi8UyOjaQ3QssIBxUyFQJblVTkQUavV3HfffQBzbsyzGpfigfL3uhrfPUB1dXVigP7kk0/OOnerxb3MhLKyMsrKyrCo/JSN/o4N5VrWbrOg16qB5Am0VArb7DQuX4i+sSucvdCCt/oOqjffuK/ATOP/XJt+pMLlcvHTf32exk372LyhAbt9fiN3OkbHhpmaGuNf/u+zPPX0lxa810M6zp59g7y8MQ4cKGbt2hoMhoVNeng8fvr6xvnggzcZH69n69Z9OZXT4/Fw9G+/zwMGKDGbKKuY6cBRkOKKVMemGXB7GDl/gn9/5zSf+y9/otijSSnHz57h8uQo1pq1rNndhM64sA3Igz4/E8PDvPjJhzT193Lvvv0LyicSiRCN3qiT+nw+/u1nL7CubifrG/ZRVOhYUP4A4+MjOCfH+acf/CtPPf1Yzj0ZL148QyTSxYEDpaxZcztm88LK1O8P0tMzSkvLec6d62Tv3tzufxEIBPjt3/0f7tTJ7NNrqShPXw+nSV93hzxeRj46xy/OnObzf/Lfc+4dfObiB5zvvk7euloqt2/AsEADQjgYxDkwxNt9QwydPMHnDisrU0mSZhl4VqPxEcBut3P33dNl8sADD6RMk0ms7JVGJnrfajO+w2y978tf/vIN51Y7qfqe+dBoNFRVVbFmzRrOvfZLNgZPsaZWx9piCyrVXHXEmuL/6c9xV4DBiY/41Uvv0Xj4SYpKZ+vckiTNmqiNxWIZ2S46e67ym7f/Lw17NGzdYcZaYATiuoUp6RNu1F2niUZjjA4Mcm3wl7T/5j0e+9zXFcuglNd/9Rs2F1axbcPtrC2tWPCkx8SUk8GxEV7/5a/Zde8BKtbm1vmgp7OHo/96lE0Fm9hu2U6eaWGhTWNyjJGpEYbeGOKFiy/w2B88lvO25tVjr2CtNFO/txpHRcmCwyi6Jl2MDo3x4rHnObjrLmqqlMfF18C0oSHVw4XDYV566SXuuOMOjh49Sk1NDRMTE0QiEfLz82lra+PgwYNoNBpcLhc6nY6RkREqKytv8IaSJAm3283Ro0dvuI9Op0sZc7mrq4srV67w7W9/mx//+MdUVFRQV7cx5YNEo1E++PAdDAYTsViMWCyC3VZMZ9dVwuEQdnsxY2ND1Naup7eviy1NuwiHQ5SX3TiQVqnUXLp0SVEcKJgOYzA+Ps758+d56aWXqK2tJT8/n7//+8cUXX/pUhdvv/0Jhw418c47rdTUOHC5fDz++HwKvIzXG0jEIJrJzFmVmfT29nLhwgUuXLhARUUFlZWV1FuMUJV60BmORnnu8hWseh01tnw0KhXnege4u3YtMVnm6JXrbC93cKqzl0c21mNMsyxfJcEnn3ySiK/7ySefkJeXR2FhYcIDZCaDg4OMjIzQ0dHBRx99RF1dHVarlaee/NOU+Q8N9TE6NsTgYC8b1m/B6/MwMTGK2ZJHf38XJqOZ8vIqrly5xL59dzM2NkxtzYaUXnvBYOoy1Wg0KRv4rq4uLl26xKVLl6isrKSyspLqamUrBvz+EKdPf0JX1zDbttUyNubC7w9x6FATa9fObQhQqSTa2tpSeuymKtPx8XFGRkY4f/48x44do76+HrPZzF9uXJv2HtcmnJy43sPGYjtmnZZQNIb/U8+tIY+XDwaGKTRND652lKcPjROJRjh16tQN7Uy6enr9+nWam5tpbm7mmWeeobKyEkNRIZvTNMJjPX1cPf8+sixTWl9LNBTCWlTISGc3BaUOdEYD/W1Xqd+9Pa2MAJJKQsX8nVuygWc1xsAVoCjMiEajWZUDkVud5JifmdB2/nUa+YDD28uzliPPpGPTWh2b1sJzbx+nX6ulYt2OWWkWumfF1NQUv/zFUR568PdRq7Nvw4qLHBQXOSgtXcuP/+WXfP0bT+bMM/7ixXcoKRnjyJHsPUIsFiMbNlSyYUMlL710nitX9Kxfv2P+CxXg8Xh442/+iv+2tjjjlWWpKLdaKLdaqA+F+OFf/wUP/vf/mXZJY6acufgBF13jbLgre+9qvcmIo6YaR001H506g/HiB+y/LTdl6vf7+befvshdhx7HYMjeAF9YWEJhYQlr19TyzDMv8NRTj+TMCN3W9gFabQePP74767yMRj3r11eyfn0lv/3tRS5depctW3KzsXUwGOS3f/UXfKOykLwcTBSVWsyUWsxsjEb5h7/9Hnf9lz8lLy83KxbPf/Ixb/d1svnIXVnnpdXrKampoqSmirbz7yOdOc2D+++c97r4ysY4KpVK9PurkGTDXjpWewi6W5VsxnKnX/kJTzX5qXZkH36zMM9AYZ6Bxmr4pzd+Bnu+StEMe5UsywsylsO08fn1D/+Ze76avX6qVqsoXZNH6RroudrJs7/6e77yxT/OOt84v3nuZe7deDvrq7Pf/8meb8Oeb2Nz3Xr+5ehzsJ+cGaF7Ont4/SfH+dz6z2edl0pSUVpQSmlBKf3j/Tzzj8/w9H9+OgdSTvPSb/6dtbdV0LAp+/CJeQV55BXkUbehlt/+4nUkSaJ6rbKJKg2Q1ltXq9Vit9sxGo2UlZWxc+dOzpw5g8lkYvv27ajVavx+PwUFBdhsNgKBABaLJeWAQ5Zl1Gp1yl1jJUlKef+4p4fNZqO0tJR9+/bh96aemfT5PIyND/Pgkcdxu6d459wJtt+2j/6BbgwGIw0NmwgG/VRW1NDb18XU1ARr16Qu/FgsSlFRUcqlmalknelBXl1dTUNDA+Xl5Sh1jNuypRqjUYffH6KszMbOnfWcOdOi4EoJtVpKKWcqAySAyTQ9o1VYWEhpaSn79+9H7rqe9g5atZpSi5nbykoY9HiRZZlHNzYQicVQSRIGjQa9Wo1FN3dnHJOnZ9bjsnZ2drJmzRocDkfKMp3ZCdTU1FBXV0dFRUVar4Ou7mtUlK+lqXEnF947TVlpJRazlfGJURwlFThKygiGgmzZsptr11qpq01tfAZAyqxM4/U0XqYHDhwgEBiYszw+u1bH9u11WCwGTCYdDQ3THYLfP78HWywmY7PZFNfTmQpTTU0NtbW102XqH0t7D5vBQJ29gF0VpbzTM8BtZSWYtBre6RlAp1ajkiQsOh3+eWZeYzGZysrKGxR2tVqdshON19N423PgwAGujw6n9YA25lmxV5ZTuXE9n5x8G0mSaNizi+GOLgbarrLjoSNMjY4Rmie8ihyTiWXsE56517xgZaBk1U4kElm13lC3MskxP5VytfVj1oQ+4vC27JX7ZJ44sJafvn2CQWs5ZWWZ7YeQymvkuV+8wuGDX8qJ8XkmFrOVPbvv58UXj/Lkk49mnd+VK59gMnVz5MjcE4gL4dFHb+e5596lu7sobeiITHj9hz/g6+X2nBifZ2LR6Xiq1MZzP/4RD/2/2Q/wWq+2c2Gknw2H7siBdLNZf2g/50+eoehaPhvqPxs46vX6BRnt/v2FX3PH3gdzYnyeiU6n5+AdD/PC86/ytf/45PwXzENnZyd+/yc5MT4n8+CDt/Hqq+9z9Wo+DQ2pHXEy4Y2f/IivOPJzYnyeiVat5g8rCvnHf/w7Hv/W/8w6v97eXk5ebWHzkdx6fwM03L6TK+feo/zyx2xrVL7aF+ZeOSxYuSg17GW6oluwMljoWO6jt4/ySIObakfuNyb9o3sq+c6rz7Lvsf+W8YS+Vqu9IXb9b97+UU6Mz8msbbATDg5w6swxDu2/P+v8zp56m/2123JifE7m659/gv/zwv/l/ie/mLA3LJRAIMDRnx7NifE5mYrCCgJDAd44eoJ7Pp99H3jm3NuUbS7OifE5mQefvI+Xf/xrHi348qzJ53STOpr4yVSDk1gshtvtxuVyMTU1hdvtxuv1IkkSfX196PV6KioqOHv2LHq9HrvdTmdnJ06n84aY0ZIkYbPZ2Lx5s+KH2bJlC0eOHOFrX/sa27ZtIxgM8stnX0mZ1mSyEI1GOX7iFRrqNqJRa+jsasdgMOL3+/G4XQQCfmJyjMqKKnw+b9ofskqloqKigoaGBkVy2u12du7cyZ/92Z+xdetWiouLOXfuHCrVxLzXajRqJibc5OWZqKws5NKlLtxuP15vkEgkikYz10ysTF6eVbGcABs2bOAP//APZ4UK+c3ffj9t+pgsMxkIMubzo1eree5yG/vWVFBjy2PCH6DIZORc7wC9U248ofAcHtAS5eXlCVmvXbvGunXr0sZfLCsro6mpifvuu4+nnnoKh8PB+fPncTlTy1lZWc3bZ16ntmY9JcVldHS2U1JShtlspa+vk/x8G8Ggn4b6RlpaL7LVmnqAIEkSVktmZbp582a+8pWv8MQTT9DQ0IBGo+HFF3+g6FpJkhgenkSv16LX6xgfd2MwaBUN0tRqFaWlpYpldTgcbNu2jf/0n/4Te/fupbS0lIsXLyK99mLaa4LRKL0uNypJ4tqEk2Gvj53lDtyhEMFIlE7nFO5QCLNWS0yWUaWRO880HT9O6eBzy5YtiTKtr69Hp9Px1z/50ZzXu8fGcY2OsaZxEwNXrjJ49TqmPCtqrZbBa9eRkFBr1PinXGljFksqlSIPaIFAIEhGlmV6L73J0/cvPIzFfBxpzONfL56krOwrGV03vTLss8m1jz/+mNLSeoyG7JTvdBQVOlC1W+jq6spq6a4sy3zyyVn++I9zb9SLc/fdm3j22bezNkC3trbS6J8i37E4YYlKzCaqeqZjjNfXZzcoO3nhPJVH5vf8XCiVO7dx8o0zswzQwWAwY6Pd1atXMRmLyc+351pEACyWPAryymltbWXjxuwMu5988jZf/Wr2xuF03HVXIz/4wdmsDdDXr1+ncnyY0urcGyEAzDot2wny8ccfZxTGMRWvnT/Lmr235UiyG6nauY1TL/2OrZsbhUezQCDIGKfTiTTRzuZt2Xs+p+OhzVrevniObbcfzOi6cDg8q107+94brNu7ePu11DUWc+rnZ9jpuSOrVUUul4vJ3hG2fSF7Q3Y67rttPx9eeJ/9h7LTg86/fZ6NeYvX79eV1nH84nGmDszeay5TfD4f1/raeeSBh3Io3Wy2H9zGu++d5b67jySOpZvU0UD6zZZUKhWPPjrtvRL3XK6urub48eNUVlaydetWAEUGsIUsV/qjP/ojJEmivr4+4X2q1qQ2DqnVah763H9IfK+uXjct96efABUV04OL9evmVogkSZWRklxYWMh3vvOdG47PHftnmrq6MurqPvNk+upXp5eYVVfPH99uIWX65S9/edbyonA4jEad3uCmkiQe2/xZGf7Pg5/Fgq2xzRcv7jPUqtkx0+Zb5mK32/nrv/5rTp48icMxXRZzbQ5QWVHNk0/8P5/JVvOZzA31m2alffDI43PeO9P4Qo888giPPfZZuJVoNKp4o8x77ln4cuJMdeW8vDz+6q/+atYxeQ6jMUwv//3abdO7hP/Rrs9kXV80PRhsdBQpurdG8+nvV2G5PPTQQ3zxi1+cJad6njpjLSqkrGF6Vq+s4dM49A03zvLZK9IPuiRJAgU//VwtgRYIBKuHyclJCnUBNGrlfWOmOGxG3GO9s/S2hbRHvb1DFNqUx2tbCAUFxQwODmdlgHa73RiNQYzGBcQN4QAAIABJREFU3HpqzqSoKA+PZ5RwOJzV0uqJtk/YYlzczTgrNBLXr7VDFgZoj8fDWDhA5SJtxANgtFoYC/jw+XxZeRj19Q1RkL+4MWdtBcX0941kZYAOhUIEAk7s9tzGPp6J2WxAp/PhdruzirE80dNFlWZxja3lBj0XrrRAFgboaDTK0PgYWwsXZ/IBQKPV4tOqcTqd2O3p7yNJ0qrbaFcgEGTP+Pg4tXnpN/vLBaU2E57mTuAzA/RCdJXuocvctnvx+n0AQ5GHkZGRrAzQIyMjrFnkfr+syMGbrReyzqe7tYetlq05kCg91oiF4eHhrAzQIyMjFDgWfr0SihyFfDB2SVFaFUBRUREDA8pCBoTDYRwOR8bLFAYHBzOOB6ZWqwkGg4lBltlsxu/3Ewgs7g99yjU8pyKihPz8fAYH07jr5oihIScGQ+ZlOhOj0UgkGsOVwaZFC6EnGKWw8LPZQaX1Z6aB3WazMTY+knPZZuJ0jmE2ZzZYSi5TtVqNVqtnbMyVS9FuoLfXOatMF4LNZmNAQbiPbBjz+VHrDYqNz3BjmUqShNVgxD2eelVBLBrFWmgnGs4uFIZrdIzR0VGGhobmTDdzV1cxMFm9KDHsabXKViwIVhYqlSrjyUiXy0WJLTPlPhyJ8V77KKHw9LLfziE33SOeOa+psqmZnJxMfE/eZTpZ7lSho7weDwUFi+exA5Cfb8c5kVk/mBzP2ul0Ulo6vz4WDkd47712QqGFLY2urCxI7FGhhFROCh6vj2JTZmEivKEwHwwo228Epr2g3ZOZ6ZbJZTo5OUl+sbLJ47kY6ezGM5FeFmtxIU5nZrImx153udzYChbPAAmQbyvE5c6unk5MTFBePv/gLtt6WlpamFE9TYVndHTR62mx2Yjb683oHsl7a0xMTGC8CfU0v6ho3noqy/Ksd76Q/kGw/EnebDIdud5gV7A8WMhYzuPxUFKQWX3wBsK81z6a+P5Jt5PRqfS2reJ8AxH/7D5qvjAwye3p9Aa/MSx52dVdlzNAz9X0fVB+kWaWfqqE5LA3U1NTlNhv1E+Hx0d56Y3fcOr9sxnln4rCAhv+QCAjh9Pkfl+WZfw+P/lJGw7G5BjXh67j8rto628jJme3Wb3NbM9Yl0q2s01OTmIruVFH8Xv9tDa3zZnXQM8gQ/3z9/9mixmVJCnal0YDUF5eztWrV+dNDNOD7S1btihKO5Ph4WEqKrJflri2qpzxiVEqytNvnJYt/oCb4uLsltA6HA4+/nhhGwMpZWjISWFh9jELi9ZvZKirNeex4GYyrDexfQFlOrNhKCkpIRhUNrOyUCac46xZk1lszVTYbFUMDTkpKsrNJiypGB4Os25ddvW0uLiYFsPizoYOe7wUbt6VdT515RVcGZ8gr+jGTslaaMeaAy8ZvctHaUMV58+fZ3JykpqaGmpqaliz5sbNSuMkD0yS6evrY2x0ErfbC6iJhMMJg5BWpyUcCic+AfQGA8FAAJ1eTygYRKfXo1bHsNnzWLeuISNDfqZcvdqK3z+BWh0mFApiMhnw+QIYDDoCgelnNBj0BALBWfG7dTotkgSRiAqNxorJVLioO2YPDg7i6u7E55zAEAoQDIWJfdpW6LUaguEIWo2G8KcdsFGvwx8MYdDpCIRCGHVa/GYrJkc5dRs3pvUkSDbspSIcDs+pwLS1tTPpdBEOxwgFwxgMBgKBwKzNd/V6A8FgAEklIcem89LqdMhyDJVKwmw2YrNbUu6hkCuGhoZwuYbw+aYwGORZcejj71+jUROJTL/zeN3Q63UEgyGMRj0+n4q8vEIqK+sWdZXAlStXCPX3gnuKcDCYeL86rZbQp0p5/NhM9Dot0WgMWW9Al5eHrmItdXXp9oKIpTTczkcsllmIAa1GxcCEj8aq6bBloUgMfyhCVUl675FINDrnpEey3M3NzXz729/mrrvuwm6343A4iMly1krxfMhyDEVLSj4lEonwF3/xFxQUFHD//fdTX1+PJElEo/PLqdVqGBiYoLFxYTpROBzNyKB09epVvvvd7/Lkk0+ydetWSktLicmZ7yCgkqDf5WZH+fyr3mB6P41M6lgsFuO73/0uWq2WL37xizPKdGEbGM0kHAzim3JhsaeOexmLxjKanLt69Sr/9b/+V26//XYcDgfl5eUgyxn/pjJFjsbIdArxL//yL5EkiUceeSSxSlNJmWZbTyORSEb1tKenh//1v/4XDz/8MHv27Pl0HCYvej2V5dSTNHPxve99j0gkklSm2e+vMW89jc3dnqa+Zu7+oaOjA+eEG583QCQqo9FoCAWDIJFoEuO6HnymC8aPaXUa9HotBTYz69evz0i2TPD7/fT3d+ByjWMyxfD7g+j1WoLBMEajAf+n+6bE+/64LiBJ0+/YaNTj94PZXEB+fllixepi0NnZSbC3i6DbhRQMoJZUBD/t71WSREz+/9l78/i2ruve94t5JgmCBOd5HjRQ1CzZoi1Zkh3JYxzLQ5yhaRI3vY3TtPe2TV+bJq9NX1/u66c3N2mb1Dd162tn8GxXtuXIlq2JmgdKpESKkziCxEAAxDy9PyBQBAiQIAG6zufy+4/Eg3MONjY2zl577bV+K4RCKsPlDdsuEVswYguI5TJQZ6LMK6A6gRxobLHJRMQ692bj8/no7r6BxWIjFBTN2PFejyfqOxdLxPh9/ihbEIEAmUxKiAAZGhV5+Tnk5ycu8J4qAwMDOJ1G/H4HIlEAgQA8nlt9eivzPPwdhz9v5PuP2H8SiZRgUIZCoU2LNn0iAoEANzqv4hwfQzZtx+2bZc/PsvWkYglevy/KFhQKhUjFYjwyOcosLZqyivDcEoelFHQOhUJJ1/yKIBQIGDU7Z/72BYKY7V5yM+NvDIZCIXxJ2EGzef7553nuued48skn2b59OxqNZsZ+TxWTwUFpTfw1dzDIop6nFouF73znO7S2ttLW1jZjkwdiOtXtcXPm6kV2bbqTw6ePRr1mm7bjcDkJhoJkaTJRKZILJPTHSJQsxJtvvsnRo0d5+OGHWb16NVqtNu7zQigQMuWYIjcjh/qi+qTvn4jF2upWq5U/+7M/o6Wlhba2tnltaYFQyMToJA1rErezsDTsFxsfHkebo0UmT+wv9Pv8SfXpjANao9Fw5coVmpubF7xosTidTs6ePcsf/MEfLPpapVIZ9UHuuGMLL/zb6xQVPpnOJs5w9txRtmxdl7KTR6fTIZWWcPFiH2vXVqapdbfx+fy8++4Ajz9+f8r32rjnXt7663a+nZN+8XyAD2+OUXnn7qgd46U4JzIzM6msKqSz6xKNDelPdwiFQnRcPco3n/1ayvfatOkOXnnlR0teZCzExx93otevRpViCq1arSZ/650cP/0x2wqXJ93lTbODR+/elfJ97t52B0d/8j8oqkten3sxDF6+yrqSctatW8e6detwuVz09/dz+fJl3nvvvRnN8sU4AS9dvIJErKOkcA16fdHt58qtR5pQEHYoRIxnILw4EYAAASFC4YnMasZkmuCdg0fYsrUFXZpTUu12O52dH/HAA/Xk59ej12dFLfoTFYqd/TqEF2U3b05y86aFCxdO0di4Ka3tBLhxup18wzAbZUJKszOQiGINjVkrvETtBSwuDyND13jn4hkq77mX7IL06mE6HA6OHT1LU8MmSovyyNbmEAwFZ1oX1cqY7zzSRgg/lyYmxjCbxznVfp5Nm9NfiK2//wo5OW42b9ZTWlqHNKaobOT7jyw6E2G1OhgeNnLo0Ifk568hNze9fepyueh881V2amQUKGUUFGff/t0Q3X/xiIzTUDDIsM3GzesXONfbzard96atjcEFnC5dQ1MYplwYLC4aSjIRi4RMTrlxuH2cu2HC6fGjVc8fhePxB7l+/ToajYZAIMDY2BgXL15EKBRiMBgYHx+nvb195vwLFy5w6tQpTp06xZtvvklVVRUqpYbG2vh6gpc7ziIWizGZJsjNzUcuU3Cjt4ucnDymrGa8HjfFJZX0dF/hrrbPYDJPUF4WX+N/cnKSixcvzjkerwCtz+ejr6+P9vZ2fvWrX1FRUUFNTQ1tbckFL1itTt566zQ2mxOJRExhYTbDwyaefHLHnDEdi9fr5+rVq2RlRcunJCqUe+3aNY4fP87x48dpbm6moqKCXJmUB8oSR5WfHBrlhnmKiqwMMuUyrkwYWV8Ydi5cGp/EHwxi93gp0KhmZK7iYbRY4vapWCyesxgKBAL09PRw+vRp3n77bSorK6mtrUVUGX9TNeDzc+Hd91Go1Uz0D7Ju314mB29CKIRco8ZltaHJ0THRP0h28fy/74DfT2dnJxMT4cy18fFxTp8+jUQiidunHR0dnD9/nvPnz3P06FEqKirQqLUUfSa+VFk6x6nJbE56nEJY7/vs2bO8/PLLNDc3U1dXx6pVyaUcpzJOg8EQnZ2dmEymqOOJimT39PRw4sQJTpw4QVVVFY2NjeTIpNxRmDg44obZwuHem5RmaSjO0ESN01H7NFcMRnTKsA3fWhjfORYCLFbbovr06tWrnDlzhldffZWamhrWrl2LKzdxVPnAxctYxgxMjRmo3tSKx+la0jidd1JbJKFQiJPHz1KQX0t1ZRM5OXkIBUIQhOa8zez5KmILRuwCoUCIyTSBeWqC37x/jO13rE/7pq7BMIzReJVdu6opLl5DZqaaRLZTZO6PtQUFAgFer4/BwQkuXRqkv99IRUXytZ6S5cqhd1nnm6ZELqFYr0Yg0BDf1pt9LPz/iL0lEggYszsYvdnF+x0XaLr/4ZQLkMViMExy8fw1mps2U1ORj0aTSYjQzHc9t3VxPkUI/AEfExNjjI71MTnZxapV6XfuXrvWzurVasrLCykt1d+2j259v7ftPsGcYxFEIgHj41MYDBbefPM/aGzckZL0Qjws42NcP/QO+3VqCjPU6LJ10Xbf7PbN9PNsWzr8uXzBADenJug5O0J3Xgm1m7bMfbMlEArNZ3WGeffcMCKhgEylhGAIVHIxU9NeJqZc3Jx0MDgxPROEkIhpp5vOzk4gnB0yOjrKxYsXEYlE9PT0YDAYUChuO7DPnz/P2bNnuXr1KvX19dTW1iJQWYH4z9Suc+NMjtqZHJ1m8z3lGMccZOUqGe6zkFugRqaQ0N9lZPXmheyxEIODgwmlfWPnqcnJSXp7e3n//fd54YUXqK2tJS8vj6/t/FzUeS6PG+u0DX/Qj0gkoqu/hwnTJOOmSRorazHbppiyWVlT10Tf8CCBYJCrvdcJhYLk6/QMT4zxxL0PIZXctq/tTsei5qiOjg4OHjzIwYMHaWlpoba2FuwQKg/NjLPZWJ1WzvSepb6onksDl1BKFZTklOIP+vEH/DjcDgLBAGKRCAFCRi2jlOvLWV0WHegbIjRTey+WeOtzk8lET08Phw4d4l/+5V9YvXo1+fn5tD22dc71EQyjE0yOTWKaMFNcXsT4iAGxWIRMIcfjcpNbmEvPlRtUNVQS8AdxOV2s3TQ3IDkQCEQ9LxIFeM3kmezbt4/nnnsOmUy2qAJsC+F2u3n++ed5+umn03I/tVrNZ/bfxW/ee4VdOx9Jyz0jXLh0kuwcMatXp8cJv3XrLt555wWkUjGNjemL2PZ6ffzsZx9x773p6VO5XM4dX/0GP/vpj/nd+vRGLR4bHMFQu4rN27ZHHV9q9ejtd2zhlZffpueGZI62cyqEQiHeO/QrnnjyswufnARisZidOw/wT//0Gl//+t1puWeEU6eu0d8vYtu2zWm5X+uOu/h4oB/pyCAbitK70/7PXX3s/r1nk0ppS4avPfYEP335TdY8sj+tsgdDXdfIGDfTdv9DM8cUCgWNjY00Njbi9/sZGBjgxo0bvP/++wwNDdHf309JSQnt7e28/PLLrFq1Kkqf6fKla2RmlLN2derfU768iPy8IpoaW3j74Is0rRKh1aZHy8lutzM21s73vrdvwcVvMjQ3l9HcXEZRUR9vv32UxsY70tDKMH1nT9E0PsDu8vjFSxeDWiqlJFPD5hL48QfvIrrnM2Tq0xO943A4OHXyCg/s+zJy+eJSneNRXl5DeXkNgzcLOXLkA9ra0mM4A9y8eY2aGj/796d+T41GQXFxDps31/OTn/wGs1lMdnZ6NrZcLhe9r/+Kb1eXoJGlLnnTkCujAagxT/Gvr73MhocW/+x3OBxzNgEXei7laxWMmpyMGB1Ypj18YWcNPaM2rt6cQioWUaZX4/IuEIEVCuHz+fB6vfj9/hm5MpFIhNfrxefzRUXvz07VnJqawmazIRCI49YR8Ho9DI/0s/Ou/VRXNfLBkbe5a8d9GE0TSKUyiovCNoJMKiM3N5+R0ZuUllTG/9whCAQDcaPFEjmgI7aBw+HA6/ViNBoJBpPbyMjMVLJ7dwuHDl1AJBKybl0VTqcnqQhqgSAc/RTb1kQLkdl9arFYyMnJQRFQICCxAzpbIWdVXg6ZMhlml5u91RWIhUK6Jk3IxSJc/hBZC2pdhwgEQ3H7NJxiG93WYDAYt0/1FfEd0CKJGE2OjuKGOgQiIcOdXai1WnLKSug9ewHPtIOazRsw3hwm4PXDAj6x2X3qdrtxuVwEAoGk+1QkjP9bT+s4FQgILmKcwm0bNpIBZTQaEQiSc7ykMk5BiM/nm9PWRJvEsX1qt9tRBhUISOyA1srlVGVnUZWdNWecykQihAIBaqkU1zyRogLCUVvx+lQikcybPu50OhEKhRiNRpTZie0cZWYmErmcovpaek6fo7i+dsnjdCGSlVo7/JujbNtyPwX5qdsoRUVlFBWVUVnewDuHXmL7HS1pc0KbTAaglz//831puJuc7GwNLS1VvPHGaW7e7Ka4uHbhy5Lk/Buv8kSGhJo0RAJXZWdRBazL9fE/3niZqgcfjXLapYLJZOZG9ziPPfpMWu6XocmiuqqBs+eO0tXZQ0Nj+nwzV658xEMP1aQlSKqyMp/KynxaWqr4h3/4DUVFW1MOjopgm5hg+vA7/PXadIwnCdp8OWuAw4M3udIepHLztkXfJa7dt8A1lfkart60MG5x4vOHeHxHJX3jds70GKkuyKCqIAlN/9DtCO3I83M+uy/i6HW5XHg8HqxWKwpR4sh9jVaOTCGmsimXkT4LtikPa7YVM3TDQl+nibseqsU84cDlnF/6IxQK2yLxnv3xHNCzo85dLhdWqxW1Wj3HPpVLZWiUahSy8DMwX5fL2KSBCbOR+vJqMtUZ+Hw+NEo11wdu0FK/iuqSctovn6OlfhVOd9j2YPYyNxQ/myHeZj5ES4V4vV6sVisyvzyu8xlALpEjl8jRyNXkZeahz9Rjd9txeVzUFFQjFcsYtYxyY6yHTGUWRdlFc5zPEN5EWWyfzrYHpqensdvt8xYP6+3qo7CsAHWmGovRisftQa7NQF+Qi906jVAoRK6Qo9PrOPTab8gvzotbQ1AgFERtZiWa66O8Qr/zO7/DL37xC0ZHR8nLy6O4uHgm1SvR7tfsY5F/hUIh4+PjGAwG2tvb+eIXvzgnqiRZdDod4+PjURW/S0qKaNu5gbcP/htVlWvIycknQ5O5YITW7Nci/w8E/IwbRrBYxsjSSdjRlj5nCcC99z7Fhx++xuhoJ4WFakpLcxP2afxdxvDeqEgkZHzcgsEwxfvv32Tv3s8vuU/joS8pZdVXnuFvn/tn7lZJKMxQk61URLVroZ1bQTiED18gwM0pG4NBIdbVG9m2b26U9tDQEMXFSzPQHvnsPt5+6xAXLk+hzchFry9EJBLG/d6j+zR6B18oFGC2mDCbJ7nWfZonn3qEnDjyDkslJ6eA1asf5K/+6hfs2VNJUZGOnJzMhGl7icYBQCAQZHBwgqEhKwaDhu3b0xexB3DnF77Mx6/8ClNvF8USIWVZGYgSjVMSx5cKBQImHU7G7Q7ec/i595lvoitMXXonQr4uh9/du5+f/esvyFq3ioxcHZrsbEKh4O1+Ewghkq4S+X/Mj18QCk/OlrFx3EYz+Z4gn5vlfI5FLBZTXV1NdXU1oVCI73//+4yOjnL69Gna29u5efMm3/ve9/jyl79MU1MTPd19qJUFaXE+x7Lvvif4xa9/ws5dW1IqmBWhs/OjtDmfZ7NmTSU+X4APP7xEdXXqGQsD3depG+5jd4LIvVT4xqpa/vbQfyB95PGZxch8RU9nE8+hcfzYOfZ/5otpcT7Ppqy0ikDAz4ULF2hpSX2jdHh4gIoKF/v3b0xD66L5vd/bxQ9/eBCZLD2LkWtvv8Y3q4rT4nyeTWV2Fo8HTLz18Yc03HlX0teZzWZ+8IMfEAwGqa2tpaioCLlcTplv/hTOa0NWxi1Opt1+cjMVnOqe5ObENLUthRy/OkFxropgMMTq8uyEhYylEhHVzc1kZWXh9/sZGRlh06ZwtsHIyAiDg4O0tbXNnK9Wq2loaGDnzp089NBDlJaW8tJLL8e/t1SGWpXBBx++TVlZNRkZWfQPdN/SxpRgs1mQSmUICBf7vdF7jYb6+LJsAoGAPH3+TNsWwufz8eabb/LUU0/xwAMPUF1djdlsZnz8o6SuDwTCGRh6fSYGgxWLZRqHw4PX61+wiKFYLKapqSVp+bWMjAwaGxt54okn2LJlC6WlpRz8t58TchoTXuP2Bxi22ikoUvH8xV62lBRQn5PNlNuDSCDg8vgkMrGI0szEzkGBQIA+W5t0nwaDQV577TVKS0t57LHHqK6uxm6389LpYwmvCfh8WA0TiCVS9JVl9LSfRSKXkVtWyuj1HsZ6epEq5Ph9XjxOZ8L7CEUi1q5dO5Pu3Nvby44dOxJuSOfl5VFdXc3WrVv5/Oc/T3l5Oa+9+lbcc9M5TgF0upyk+xTCBdlLS0vZt28fa9euxe1209X1ZlLXpjJOBYIQa9asmVcWbDZ6vZ6mpib279/PXXfdRWVlJR/88kWYGEx4jScQYMhmp0mv463r0eM0GArSb7Fi93pRSSQEExSyDgHZGZmL6tOKinBq/P79+1mzZg2BQICf/ebgvNdYJyZRZWVRvmYVo9d7ljROQ4Sj8IqKihJq+8ZKrcVbC589c4mNrXvS4nyejUqlZs+uz3Ho8C/YuWvxDrNY7HY7NlsHf/iH6V1HADzwwEZee+0UIyODFBWl7ty8dvQIn1UJqdGlN0NXJZXwX2qK+eGbr9L6WHRGdTLBLbHnOJ1OLl3o4eEHfyet7QRY33oHp04foa9vgMrK1IPEursvsm9fRdozdOVyKd/85i7+6q8OsnHjZ1K+n8fjYeDdN/nO2tQlDGLZWVaEf+Am16/pqKi/ff/YGgSxDA4O8jd/8zfk5ORQXl5OTU0NRqORLfMkxgaDIU50GpBJREw7/SjkYs73mhiadLCjOZ8LvUZEIiFFOiWJopMBVAopa9eGs4EMBgP9/f0zz1exWIxOp4uy+86dO0dTUxNf+MIX2LZtGzqdjh8//915+8U07iAjW055vY7Oc+MMXjehyZQhkQoZ7Dbfml+FTE95CAZDcW1UgQAqq6poaWmZ971m3tNk4rXXXmPv3r3s3buX6upqLl++TCimFIrX58Nsm8I6bWfKbqOjp4sxowGL3Yo/4OfawA0y1Rlc7buO3THNax8cZH3jWpxuFxb7FA6XE6/Ph2LWukypUCxqjurs7GTHjh3s27ePLVu2kJeXx4++96M55wVDQZxeJ9NuBw73NC6vG7vLRqYyA7/fz7BpGIvDQoW+gqtDV5BJ5JimTRAKEQgGEAnnqjBUVFSwcWNyazWLxUJ5eTltbW0zkladnZ2Ymavj7Pf5sFvtNKytp7ezj7wiPVm6TMaHxwCQyqQ4HU4CAT9ul5vujh423NHKUP8wfp8fyRJ9B3OswAMHDnDhwgWuXbvGyZMn8fl8CIVCXK6wOHokGkCtVjM9HR4dCoUCl8uFRqPBbrejUqlQqVQUFRXx7LPPLqlhEaqqquju7p5zvKKijC9/5QBnz1zgRt9JPJ4ALpcTuVyB2+1CqVThdIYLYEikUnxe74z2JoR3DGQyOWKxkNxcLWtbK5ZNX/Ouux6iq6uLU6f6+eCDKwQC4d0Atzu8kxHZaZmttxX5v0qlxOFwolQqEIkyyM4u5MCBB5alnfll5Tzyf32PS8ePcq6vD8HUFNMuNwq5DJfbg0qhwHFrHES0lyL/AoiEwrAWrFxJRs1qCptW01w5V35kcnIynLawAEKhMGHEwb79u7l+/ToDA0OcvdiB3xcgGAzh84X7VCQWE/D7Z8YDMGdsKBRK1GoZhUX6tMhuxEOvL+Txx5/l/PnTnDs3SCjUh8PhRKVS4HC4Zv6F2995RFMVwmNDKhUTDIrJyNCTn9/C9u3LI0Fx5yOfo6enhwtdV/loZATcTnx+P/6Izu8tTS25TIr7VvuUcjlOtxulXIbT7UEplyPKyiF7zTae2rptWYqzFeTk8hfP/BdOnD7FwJVext1XcDmdKFRKXA4ncoUc963fkVyhwO1yzWitQVhrT3BLE6wwS0d9aWVS43E2Wq2Wbdu2MTk5yZtvhhed/f39/N3f/R333HMPhflVPPxg6vI4idi6aQ8dl9tZ15qaY/f69as88khT2p3PEdavr+HkycPYbLZFF6GdTSAQwHL2FHub0y9nFOHRYj2/Onua5jvCsgRhXbeFI9JiF6HXr3fTWL8RhTy9qZ0RKivq6O29jNlsTqlYbjAYxGC4zjPPpH8RGuHRR9fz859fZs2a1KKru7u7uVMhJnMe3bFUaMjV0d47xOTk5IIOyEjUw/j4OBaLhZ6eHs6ePYtMJqOgoIA/PzD/b3JLg54tDdFR4W2rwvpqDSXJLbJFovjp9hFinSgtLS388Ic/JBgMUloazsYKb67Efz7feceepNoBUJA/jyNsgcVcLBKJhO9+97tRadEmkylpzduHH56bXlhdnVz0tFgsXFRmVl1dHW+88UbUMYEgURxMmDX5uazJD4+v7951u61Prw0vOquTcLIIECTcmIiHUCic06dWqxWRKHFWkkgiITNPT+HquI/pAAAgAElEQVQtuasND9w381pBTeWtf+Nrp0e/tyiqT4VC4bzjoaGhgZ///OcYDIaZGgLzXZOucSqAuE7U+fiLv/iLqD51uVyIxclJ96UyThf67cdSUVHBq6++GvUbWmicFmrUfOnWBme8cfq76xeuAyQEhKLFFen7zne+E9Wnk5OTCOcZpwCZ+lyK6sP2W1H9bdt4MeNUJBJx5coVurq6EIlEaLVasrOz0el0ZGVlzZlng8Eghw4diqprNDExgUioobR04fdbChpNJlXlq+np6Uk5S3lw8Apf+ML6NLVsLvv2tfK9771HYWFpSmsAk8lEltFAU1X6glhmo5ZK2amRcbbzKtWNt2VDkpkHYs+5cuUaWzbvTnsbI2zcsIOXX/snyspKUpIInZqaQqWy09KSel2eeMjlUh58sIEjR7qorU1NNqTv3BkeLV4eaUiAPeUlnLlwBn919cymaCgUivv9u1wupqamGBgYwGKxcO3aNY4dO4ZGo6GiooKn5tkYEAoFfPGeuWvMbY3hjMtVFcnZ8fN970KhcM7G7je+8Q3UajVr165Fp9OFs+UWmKN0+SoqG8OFX/XF4TVbWRwJ+rySeTbJZ9WySQadTsePf/zjOXarKGb+yNRk8MX7HwPg6f2PzrlPa2N827u1MTxfVZfM9fGJxeK4UbyJeOKJJ/jSl74083coFIq7oS4UCLmjIRzQWpEXft+8rNsZtqvLb8+hVflJzBmCxSkHaLVafvSjH83JmIn3KdUZavYdCNt4VfXheXN8eJxV65vJL87jekcPLZvDfdu49vZvurIuvs9ULBbP2CjzbejEndlbWlqS3rlYblpaWrhw4QJjY2MUFEQXh5PJZGzbnv4Iw+WgoaEBWD6B/nQhkUhY33Y3tKVXNmI2R44c4fd///cXPC8YDM5bEKCurm5Zi3OkC6FQyPr1m4FP/1itqalJqwTPcrJ14yYSqxl9MohEIpqamsjPz6ekpAShUMjk5CT1tenXPp5NdnYuU+cXV2E+Hl6vmby85THuI1RVZXL9ujklB7TFYiFPsrwV5/PVKtw3hlK+j8Vso6R5+QrxAGi1hVgsxpQc0DabDb1emrTTZCnk52vxeEwLn7gAvtFhCpTLV9QQoFIEZ8zmGQd0MBjEarXS19eH3W7HbDZjtVrxer1kZmaSlZU1o3XY0NDA5z//eerr6xk99tyythNgZCpEfWbiSJl4MhKxCyu1Wo3NZiEzc3lqPwC37p9EauksYjU5MzMz6eqyprNZcRkft7NxY/JZZfGc4hqVGpNxiAJNejUwZzPpdKKuWFz2W2yfZmVlYTPHr2If8PnR6LIJpqFIoc1sisrUmy0HkohgMBiV5qrRqLHZptDnpl4cOhFWmwW1JrVxmpWVhcGw/ON0cnKK2trF/WZjx6pSl4t54BrLt50LRpcbjXZxmS/x+tRrnkp4vvhWgEowGFxUYcZYbGYzBx58EJ1Oh8PhwGKx3Mq8GA9HjZnN+Hw+BgcH0Wq1BINBjh49SjAYZHR0lPvuuw+LxUK2Nv3ZWbPRZuu50deX8n2mp40UFCzfc18iEZObK8Fms0XJ0i0Ws9nMWkF6iqYlolApxzk+Co2p6VZbTHaytakVhZ8PgUCAWqXFarWmZPdNTU1RX790WzwZ8vKy8Ho7U76Pe+QmeWXLa0sXisPjTK8PO7r9fj8mkwmLxYLVamVqaoqpqSkEAgFZWVlIJBJ0Oh02m43GxkaeeuopNBoNxpFfLWs7zdMe5PMU1gsGg3MkI+RyedR8KxaLEYtEOO1elJq5wX0SqRAQJ4xsTha70U9mxeJ+97HO58zMTIyj40tuQzJYbFMoFYpFbZLFtlMgEKBUKbG7bGQo0yOLGQ+b00Z55uIyFmKdzxkZGQyO9iZ1bXZuNje6erFN2ahpTH5T1TntJMTtfkq0oQMJHNCfNr74xS/yd3/3d2zcuHEmemeF3z4CgQC//vWv+epXv4pmkcb+Cit8WhAIBDMLpezsbL7yla9w8uRJtm8P65z39PQgFy/vQkStzrhV/MWblC4hhH9/sTvowaB7wYVIMBjkwoU+6uqKUKsXLylRVKTj9OnUnJA2m4010sUtMIOhEB0GI1XZmaiT6COVVIJcJsXlciWtCRgvEs3l8qFLk+5xInJ0egaG+1O6h9VqpaQkZ8HzJietGAxTS0rXlMkkFBRo4mrmJSKeM8HhcpGvW1xEucPr45rRTGthcguYArWayeudMwuP/v5+CgoKqK+vR6vVUldXR2ZmZtTnMJlMbN++nfvuu2/mmXBhWkggEJwTvZEuxi1O1Nn5M7/lUCi0JA3LoqJ8bg4YKClZPjeUxTJBTX1qG5pZWVnY7UJcLs+C8gRLxWi0IZdrk36WJkJb18BI1wXSX8r7NkPeADk1qaUkq9VqMkVi3A4H8pjfpUgiprC2OsGVyeOyT6OVyFKW3ykuKeDShWGqq5YvgMNsmaChKTVHh0wmQyzOwmy2k529PPatw+HG6ZSmtJkLkFNewdDJIyxfDCyMOt1k16dWp0UikaDXarGbzGjiFF7OLkp9UyLg8xOcdqLTheX3Ihm8sRKBfX19vPvuu8jlcjo7w/OE1+vl17/+Ne3t7TQ0NPHUgT9MuT3zka0NO8gXQ+x8arfbKSrKRCKZf/nv8/m5eLGPNWsqlpQhV1ubw7VrUyk5oKenpynQLO75seh5X6MiNJRYNikesXaf0+lEqVQhky3vJnmevpypqYmUHNB2u53CwoWv7+8fRygUUla2eFs2vKZwL3jebGLXJx6PB5FQtOisN5PThcHhpDE3OTnNOoWEX3d0IJfLsdlsdHeHZZyKi4vRarWUlJSQlZUV5Xi8dOkSTz755Myab2pqip5OOXsX1dLFMWpyoci7HRiarCZ9LOUFzZgM5+M6oPOK07Mx4ZxUkrc5tfk0Ly+Pj46fT0t7EjEyMY6uKPW1WkldCcZO07I6oK1C65wg3MWSn5/PR2ctSZ0rlUmjop2TZdJgpCgnyazDRd/9PwGhUMif/Mmf8Pzzz9Pf349GoyEvL29mEpivSut81VtnM99rs9sRO/Ekc918utlLObbUdiTTtth7L7bPItfP1gO32+1MTEwwOTnJ008/PbPTuMIKv42EQiGc8+gJCgQCgqHkU2SXSiAYWFTUT3t7Ox0dHezevZuSkhIkEklSMhNCoRCz2Y7H41uSA3ox6U0QLlT1s5/9jObmZlatWkVOTthJuoiMLuC2Hnlxhhp1knbaYlKbAY4fP85bb73Frl270Ol0t4zUcEGr5SRI4l3leHi9Xn7605/S2NhIc3Mzen244nkyRa+EQgFjY+Yl6wWGFvlbGBgY4J133mH37t2UlpaG+zQUiqp6ngxCAYzY7EkvREOhEIQElJaWkpGRQWFhIfn5+dTXJ3b27dq1a84ioHDV3Ry99iFtTQs795fCR102Stfsn/lbIBDMSKTNR2xRlbVrV3P82HOsalqPWJx+CR6r1YzXb0mLtFlDw2aOHbvOPfcsnPq/FD7+uJvGxtQzlJqamvjVG3Lu9PlQpEGfP5Ypt5s+RQYPVafuIL5r3Ubev9BBzTJlEQ5d7GDPuug0b5VKteiU/NraWo5+fIZphx21Kv2OXZfbidE4SEPDjpTv1di4mY8/PseDD7amoWVzOXbsOvX1qWdXVVVV8YZai9nlIjtNBdhm4w0EOBUQ8/iqVSnfa1frRl65eJG6nXemoWVzuXmxgz3rF+5TlUqFRqOhsbERk8mERqNBLpdTVFRESUkJHk9gycXVkyUYglDCCizxefHFF1EqlWzcuJHi4uJbKdELz8kSiZjR0aXP+4tV3rBYLDz33HO0tbVRW1s7s8kSWOZ5PxgKIUyq+OdtXn/9dc6ePUtJSQk1NTW3bKkFCgengVAouCipIKvVyksvvURrayu1tbVkZmbesvkX7lOv14/L5V2SAzpcX2txfXr48GGsViubNm2ipKQkafs0HqO26aQd0MFQCIlEQlVVFZmZmajVarZv3z6z7ojHM888EyW7kJWVhV9TRc/ITWqKlscJeei6hzX7buv/xmrSJyI2CnZjSxsvvX+CktRNiLgMXDNRlb9hJjtwqWg0GjKL9FztvU5T1fJkun/UcZptD96T8n02bNvAi6depDJveYI5BicHKWosTGkzD8IZRuX5FXRf6aG2eXky3S+fuMq9W28XthUIBAnrKvxWOKAjfOELX+DGjRsMDw9jNIZ3LH0+H8FgMKpStVQqnflhJjoe6zyNV+k61hkrk8nmpLbGnhOvcubs912uY/HaFnss3meMvVfsNbP/ju2z2ZWsI/0QOV8qlc44opVKJZs2baKxsTGlNLkVVvg0Ei7KcPtRGjbw4jsgx8eHmTSOMzY2RH3dahzOaczmSVTqDEZGBlAqVJSUVDI83D8TmViQX4xSOXcydzodDA8Pz9GfSrRp1N3dzaFDhzh16hRFRUWsX78eqXSahes3g8vl5eOPrzIwYGDt2kqMRhsul5e2tlWUls6fdhgMhpiammJ4eHjOa/Haarfb6erq4sMPP6Sqqori4mKKi4upCyV26rp8Pv79UicFGjV5aiVSkYjrRjM6hQJ/MMjLV7vZWlrIxwPDHFiV2KHo8ngZHR1FJpPhdruZnp5mcHAwYZ92dHRw/vx5rly5Qk1NDevWrcPjJmGXXu44i1gsxmg0oFZryNbmcnOoj+zsHIymCbweN8UllfR0X+Guts9gMk9QXlYzx2kTuiUPEa9P421QOp1Orl69ygcffEBxcTGVlZWUlpaye3dyGUV9feP89KfvUldXRFfXMBUVedhsTh59dPuC1zqdbjye0TlRuon6tK+vj8OHD3P8+HFKSkrYtGkTAusU6BK39eTQKDfMU1RkZZApl3Flwsj6wnwALo1P4g8GsXu8FGhU1OXEj/4JEkKnzZwpmCYUChecr+JFoDSuXkfHR73Irw2wuT69m61vnB7FXbCDqiUU8I23EfS5x+7n1Zdf4b69B9LVRAB8Pi/HTvwHn//CI2m5X3PzWk6dGuPYsU62b08tqjKWd945TyBQTmXl4uoAJGLX17/Bc3//t/x+fXprigSCQX4+YmbXt/5rWu63urGR0SkTV0+dpWZTemNhu9vPsEalZVVD9He11ECJRz+3nxdfeIN79zyRdhvygyOv87nH9i18YhJUV9dz/ryRd9+9yN69a9Nyzwgff3wFk0nHxo2pF/QF2PmVr/H8//f/8PvlUiQpaMrG41+uD3DPt/4kLfeqrKxko8nI6Y+OU7cj9QJ8sxm4dAW9y8/6OxYnOblp0ya+/e1v43Q6WbduHRAuQJioPPdsu0+vLyQrM5sbvV1UVzVytes8SoWKwsIyrl+/zNatOzEaDVRW1M+ZXwTA9LQj6Xkfwhl5HR0dvPXWW9TU1NDc3ExpaXKOPavVyVtvncZmcyKRiCkszGZ42MSTT+5YMCra7w8wOTmZtF7xyMgIJ06c4MSJE9TV1VFVVYVWqyWUkfj3fsNs4XDvTRpys1FJJXgDQaZc4fXqqH2aKwYjulvSXa237IFYQiGwOZ0zfWowGHA4HPPafX19fbS3t9PZ2Ul5eTnr16/H543/OQOBAOfOH0cuVxIMBgkG/WRrc+kf6MHn85KdnYvROE5lZR1DwwOsXrUBn89LYcHcLMpgMIBpcjJu8fF4bR0fH+fq1ascOnSIiooK6uvrycnJIRhcl7BPZ3P58gBHj16lrW0Vx493JW33hUIh7HZn0jY/hGvovPfeexw8eJCCggK2bduGxGoF4meT+gIBfnnlOhqZlAptJmKhkJNDo+ysLCUYCvH29V7WFeZxpH+IhxqqE24IB4JQUFBAfn54fCRT2D2e5u+Gux/kf//vH/IHGR6yNenN1PqXD8co3fToHImiZIi1+7RaLXetfZqP3/hn7nwgvQ7TiRE7I2cz+J3HH0zL/e7Y1car//oSWeoMivLSK8H1v995lbqta1N26kLYWb7zsZ0c+eWHtFUnX8g8GUx2E13uLr722a+m5X533bmTF17+d1QaFUVlyUUqJ8uhV39Da/3GmWwimN/m+61yQANUV1dTnYbojxU+/QgEgqQmgxVW+KSZbVSHQqGoTadwhEl8Y3Rg8AZFhaWsal7P6TMfU5BfjFqlwWSeJE9fRJ6+AI/Xw6rm9XRcOUtNdVNc5/Otd2J8fHzOgjzeJhgwE7VttVoJhUK43W6amkpItGiajUIhZd26KtRqOUqllJqa8MTlcnkWuBKEwrBTeXR0dM5r8TbFZqeY9vb24vf7MRgMPN6aOB1IIZGQr1axq7KUjweHaSnIozo7i/ahMcRCIVKREJlIhHKBtFMIMT4+jkQiwePxMDExwdDQUNRm22wihXi9Xi9DQ0O43W6KiyuIV+LJ6/UwPNLPzrv2U1JcwfGTh1nXspWR0UEUchXFReEiJjKpjNzcfEZGb1JaUhk3YlBwq5+S7dNI8V1gZtPCaDRy993J6X9XVuazbl0V58/3UlCgZf36ao4dS07jTygMj9PYXfBE49Rqtc60eWJigvb2dir0OQgEiR3Q2Qo5q/JyyJTJMLvc7K2uQCwU0jVpQi4W4fKHyFpAvkEgCBfSjRBeLC4tAmfVjkc5/cEr2DqGaMiXUJK79GgQnz/IkHGa030OnPrt1K6eq3wfW2AsXsRBXl4eFy9ejCq4mpubyyOP7uGlF5+jde1daLNzU4oytdosmE2TdFw9xhe/fCBl+YXZbNp0L0eOvIXD0UFDQ/6CG1/zEQgEGBoycuHCEC5XERs3LryRkizZ2dls+8M/5ft////y2WwlBRoVWfKlp2ZbXG5G7dO8PuXmgT/+05Qji2azd+sd+D84TM+xdrSVZWTl5y3ZwRsMBpkaM2DpG6BepOSerdF9arFYwhqUcRbvsxGJRHPakJGRwZOff5D/9dxzrFvbhi5bj0aTSmq/DaNpgnMXDvPFLx2I0qlOlXXrtnPy5GHeeOMca9YUU1KSs+SiYaFQiKEhI1evjjE6qmTHjl1pa6dareaeP/xv/PCHf8v9mVIKNOqUoqGtbg9j09O8YnbxmT/4I7LmiSBcLDs2bMR/8gTnDn9Ebl0NWfl6RAuMo/mYGjdgHRxG7/Bx4L7kNh8EAsHMuJRKpeh0uiibYL6CixG7r652NR8ceZuG+jUYTROIJeIou2/16o3cuNFFVeVc5zOELTW/3xt33k9ko0TmWKvVysDAAA6HA40mudTqzEwlu3e3cOjQBUQiIevWVeF0epKKTBWJhBiNxqSzdScnJ2f+PzAwgM/nIysrC8GmxNrMWrmcquwsNhTlc/zmKC0FegTAsZsjyEQihAIBaqkUVxw743Z7IBAIzfSp2WzGbrfPa/dFgrJsNhtjY2OcPHmSutr42TlO5zRGk4H79j6K3W6NsvvkcgU1NY14PC6KiyoYGh7AajVTWhJfc1UgEGCxxE+hj9evRqNxZsyOjo4iFAoxGAwIhcllaKxeXY5CIcXl8i7K7otEL8cbp4nsvkifWq1WpFIpJ0+eZE2CYAEAiUhEvlpFS4GesWkHoVCIhxtq8AfDUeJysRiZSIR6gY0SoYAoGzve950sbY89yz+99Rz3lE9RqpORp13689Th9jFmdvL25WmKNz9OfpwCerF2Xzx/SXZ2NhMT0bItDbWrEAqf4civ/o3aTXKy81TI5Et/nponHJhG3EzdyONLjy1c32sxPPD5z/HKr99gc1kTpXlF5OcsPajD5XYxMjHOby4cp3bLWiqq0+eAr66vhgNw6MX3aNI1o8vQIZcs3e4zT5sxOY2MScf4yh//TtraCfDEw0/y8tu/xtJoIb8kj5y8pc/VHreHidFJLp24Qmv9Buqro+eW2LE3m986B/QK/+cQCoVSmgxWWGG5iHXwzSYUChEi/uvFxeUcPXaIyoo69LkF9PV3o9cXoFJpGB7uJzNTi8fjQqutw+/3o1AmduCIxWJaW1uT3qQxGAzU1NSwefNmWltbKSkp4cyZgyQTAS0QCDAYppDJJMhkUkwmO3K5JKl06mAQiouLWbMmucgtu93Ohx9+SG1tLVu3bqWmpgaLxYLo0smE1/iDQQwOJzaPF4fXz+tdPawrzMPu9TLt9eENBDk1PMb4tAPXPOnxYpGYNa2tyOVyHA4Hfr9/RuctHpOTk+j1ejZv3kxbWxslJSV8cPhE3DRZqVSGWpXBBx++TUlJJWKRmP6BbhQKJQKhAOuUGalUhgCoqKjlRu81GurjL2pChKM2ItFXC+FyuXjnnXeora1l27Zt1NTUYLPZkMvjFyKbTVhGycXYmIXKyjyOH7+G3e7C4fDg9wcWLGIoFApZt25d0s5IuVxOWVkZGzZsYOvWrRQXF3Pq4NvMFzjp9gcYttopKFLx/MVetpQUUJ+TzZTbg0gg4PL4JDKxiNLMxBp3oVA4yjRdrLv7EW5cu0JH90WmzxhRyeU43V6EAgHBUAiFXIrL7UUsEuK/taBXKmQ4XR5kUjEerx+lXIp12km2voTC5q3UFsXfMJi9yA+FQnOyoSA8Xk6cODHneE5ONs/83hc58uFRutrbkcpkOB0OlAolTpcTiUSGz+e51b7wMbgdcadQKPH5fEglEoIEKS8r4hv/Jb0Gc4S2tv1cu3aN//iPi0xNXUatVuFwOGcWtSqVAofDFRUNqFIpcTicyGRSPB4vKpUSq9WOTldAVdVWVq1Kf9qkVqvlqb/6a04ceo/JyxeQiv04PB6UMhlOjweJSITv1hyikstwuMP9GxkLSpkMt9eLXCrFhwB940ae3LMn7e0E2Hf3Trq7uznVcZXuD0+iVKtwu1yIxBICfh9yhQJ3jMSLTKHE43Iilkjw+3zIFUoc03aKcvPY19AUt5CxyWRKSo4lEAjE3fjRaDR889mvcuTDo5w+dw6RUIzb40Yml+NyOhGLJfj9YXtx9jiNjA2lQjlzfsDvp7y8iGe/9fWldNmCbNmyk97eXt555wIm02UyMzU4HE6kUgler29mTMLt31Hk2O2xrMRun0arzaGkZC07dqSm+x0PtVrN49/9vzlx6BC/6exAMj2Nx+dDKhbj9Nx+VgGoZDIct54rErEYn9+PWi5n2u1GJZfj8XnJWbWOA/fsWbLDfT52btlK1cAAxzsucv3ICVSaDDwuJxKpFJ/Xi1yhxB15NolEBAMBZHIFHrcLoVBEMBhArlTinHaQo9Wyo66R1TuSz6ZYSK4sOI9Tdrbdp1So6B8I68xKxFLs9qkZu6+mupnOrgus0WyMex8B4fofGzfGfz0ex48fp7m5mTvvvJPm5mY0Gg0Gw9x5IB6BQJCbNyfR6zMxGKxYLNM4HB68Xv+CevyBQJCGhgbKypKT8BgdHeXdd99l9erV7Nixg+LiYq5du0bIfDPhNZ5AgCGbHaFAwA2zBYPDScOteT8YCtJvsWL3elFJJGGpjTj2aigEKoWM1lt9OjIygtlsntfu6+rqQq/Xs3//flpaWtBqtRw7ei7uuUqlmkAgwPuH36CmqmHG7pPLFbhcLqbtNtxuF8FQkOKiMpxOR0Jt31AoRE1NDZWVyc1ZBoOB9vZ2qqur2bVrF6Wlpdy4cSMpCQ6xWITZbCcjQ0lxsY7LlweStvtCoRBKpZwNG5Ifp/39/dTU1LBt2zbWr19Pbm4u53/9i4TnB0MhptwejE4XMpGIX165xtaSIiq0GZhdbnKUCk4OjTJktTPtTWzzB0OkTTpHLBZz50Nf48yF0xy5dJlQwEEoBKFgCK8/PN/PzPFyCU53eL6K2IIqhRSHK/yv0+VFU1BD3d7tCWtmxdp98fwl5eXldHR0zDleV91Ejva/cvLsYToPd6LWKHBMu1EqZTidHpQqGU5H+Hkvk0vwuH1IpWK83vDmgVgiRCKR4PcHUMiyqC1pZf/Dc4MjUkUkErH/wMOcO3OO02ffJ+gNgDAseePzhdsSyeRXyOW4bgXbqJRKHE4nKqUKh9OBWqnE7pgmtyifDZ9pS0lHPRHVddXontVx8sN2Ll+/jEqpwulyoJApcXmcyGUK3J6wLSWTyvF43YhFEvyB8PcmEUsRioSEQiFkSilVm6rYtTV9m84RhEIhn7v/Mc5dOMe59zpwe12IxWJ8Xh/+W+M0Yi/L5FI87rBKglKlwOlwoVQqcDpdqNRK7LZpivOL2bP5vriyNRaLhYaG+BueKw7oFVZYYYU0IxDEj4YpLirn8cdup9JUVNyORqypjl4M7dp5/7zvIRaJF6WnuXv3bvbs2RNl3C5UiGamLbtSSydeTDs1Gg1/9Ed/FFVoyWKxMF9RZrFQyO+2hp21Dzfedn5EZBjKspIrriFaZPRfpPhcYWHhjHSDSChKGFR+5x23nUjNTWHncUV5/NT/gvz5ClkuTmBRoVDwp3/6p1F9arfbk/petFo1D88yLKuqwp+zvDw5jUWxWLSo77+hoYEf/OAHUdp1IpFo3k+8Jj+XNfnhiNjv3nW7rU+vDUdJVuvmL7QJ4UioxegrJkN1fTPV9c1x9foiEUvz1X5YaqGZeOTn51NSUsLVq1dpaoqOKhOLxey6J5w66HK5bmVxzG1foui1yPFYvcHloL6+nvr6+pk+TdSHsW2b/W8iTbp0IhAI2LZnL+zZi9PpRCgULljnI/b1pRaYXCy1tbUzkfEej2fRdU2S6dMTJ07wl3/5lym3te2uO4Db4zRem+YbD59Un1ZVVVFVFY5kjPTpfL/5CLG//cVqZi+Frbt3w+7duN3uBX/jsW2MOGaXkh6+WMrLyykvD2cKxY7T+b7/T6JP58sciLX7ZpOXF50Gfd/eR+d9H9F8hlAcvvrVr0Zt/k5PTy+4aRzh4TgOperq5NK2RaLFtbOwsJAf/OAHUc42gSBeLtmsazRqvtQSLvn69Q23bdR1t/Sff3f9wjUDBIBYuLgNkwMHwlkTW7ZsQaEIO5IT9alIJGL/Z25LXJXfsvdm231FRWEnfV3t/JrpkTkkWfLy8vjjP/7jOX0qTEEFXwoAABksSURBVGIMVVUVUFV1W/bg6afvvtX+he2+sCzh4vp03759PProozO/I5/Ph3ierAKhQMBnm2734Z/v2DLz/wpt8hkty/F4Xd2yEVo2zkjELmXeT5cttWbNGk6cOMHk5CS5udFZYzqdjn17PgfMnU/jEe9ziESiTyRTvXVDK2xoxev1Likz8ZOao7RaLfc9fC/AjN23WIRCYdrs/vlobWmltaUVv9+P3+9P2oaKsJAtZTabMZvNCQOlVhzQK6ywwgopMDs1E8LOPotpalnf0+sNR3gslM48m/gOAjE2m5Pc3GWs3mt1LNqYmu0ohXBU7GhgeRfiwVCIabd7pq2x32s8ZDLZXGkJiQCncxqFYvmMHYfDvmgnSrw+tdkcCc5OHyMjdsrKkm+rRCKZY9DKpFKsHi9ZiuVzcFrdXkI5tx3V6XRSfFJOz4W4//77+cd//EdEIlHC4oqfhHMuHXxa+jQZPomFT7pId58Gg0FeffVVnnnmmbRGxv62jFNIf58uF5/EBlK6+LT1qUQqZNoxvazv4XROI5Ut7juKzTxSKBSMjS1vOwFu3pxGoVicbmtspKdcLsfmWbjQWipYPR5CMbJGC839Go0GoVA446BRKBTY7fZla2MEq81IYfHiZK1i+1Qmk2G1Ji6ing6sVgcCweKckrHjVCKR4PP78QYCSJchoyKCyeOLmkuWWqcgHp8WCdEvf/nL/Pf//t/ZtWtXwsjf35b59JNwzKaL3xa7LxlptMViNps5fPgw3/rWtxK/b1rfcYUVVljh/wBmOyZjUzN1Oh3Xuy4t6/tPTo6TrU3daSyT6RgeNi6rA/r69UlycppTuodOp6MzCb3pVBi22cnQ347uWCjlNkLsOdnZWUxMjqPTpbcA3WwmJkcoLU+uyncidDodFy60p6lF8RkdNZGRkZWyM1eZX8DozWtJR7IvhV6Hm8zc299ZuhYhnwSL6d9nnnmGF154gePHj6PT6Si6JesRWXhFJAESRREm+3eiY+k8nqjw1nzXpHJdul/7pO6X7HeTzLFEUTKzo0yFQiEmkwmDwcCVK1f4+te/TkFBeosIrbDCJ0kkKyQROp2WCcNwQtmsdDBpHEerTW0OFIlEyOUaxsct5OcvnBm0VAYGjLS2bln4xHnQ6XT0dbhJf2L/bUZs06gKoiUtkpn7I3NkhOzsTCYnx8nNjV/sMB1MTI6wak3qfdrdfY22tjQ1Kg7Dwybk8tTsUwCNPo9hm53KRUQ0L5brbh/Vs6QD4hVr/rSSbDtlMhnf/va3+cd//Ef0ej06nY68vLy4WTmxttFy233ptps+adsuEelu42Lbl8p3kOzf8caOwWDAaDRiMBj41re+Ne+m9ooDeoVPLQsZfCus8J9F7AQ9e6dboVCQkSlnZPQmRYWJC6elQveNS5SWp76gLyws59ixi7S0xC96kioGg4WJiRCrVqWmtyWTyZCWV3HdaKZunsIkqXB6corsjXdGHUtmt10iidbCLq8o4dzZC8u2ELXbrUw7DOj1qX1nIpEIpbKEjo4BVq0qT1Projl1qg+dLnWN3ZKqaj66cpHUll6JsXk89IhkrJvlJBOJFicd8p9JPINyPp566imuXr1Kd3c37e3tOJ1OZDIZHo8HuVweNxVfqVTOFDIFZlKPI8QrLpSoiFPsvSJEtPySPT/S5ngkugaY+YzxmO+eS30tUT/Mt6iI7d/ZxCs0Cok/V7zz4/VPvGOx187+e3YbI59RKpUiFotRKpVUVlZy4MABFsOK3bfCp5FQKBT1WxUKhVGBCIWFhXR1HcflcqBQpK/46my6ey6y7Y7U5NAAdLoKTp3q5YEH1qehVXO5dKkPtbpsyQVNI+Tl5XFepmHK7U6pkOt8HLU6KYrRV08mGjDWNiwrL6T7xuVlc0APDPSg12emHFWbk5PDpUs+JietyxZ4cvx4H4WFyRU6nA9t02pOtn+0bA7oHpMZUWlllJMs3ZGgy8li7D6ZTMazzz7L6dOn6evro6+vD6fTOceJGGvHxNousX/Hs1MS2SHx7KD5bKDF2o/zvTcktpsWasdSHNDzOZmXYrcm6otE7UtU7DPe/Rey5WPfO3aNEPlXpVKRmZlJVVUVTzzxRNy2RrVxwTNWWOE/iViDb4UVPi3MNljijdMNG9fyH2+/xoP7v4xKFb+AxFI5duJ9FMoA+fmpG7pZWVnYbKW88MIRnnoqvSERgUCAH/3oA5qb01M8q2n7nfz7i//GNxVyclXpTW16+8Ygo2U1VBQXzxybnWI5H7HnqNVqKir1vPf+K+y555G0thPg4LsvcseO1I17gMbGFn75y0NkZqooLc1d+IJFcPjwRXp6BNTUpL4Jo1Qq0W29k38+cYSvNVWnoXXR/PhqL7WfjTaYko2A/zQQm46ezLhtamqaowW9wgr/2azYfSt8Wpnt/Is3RjduXM3b77zEow9/Je3vffCdX1DfWJYWmZTi4nK6ui6g0Vzi7ruTKw6dLAMDBl555RotLfek5X519+zhf778En/ekv4inP/rSjfqrXdFyT8k62wKBoNRDvaSkhKuTF3j1JmP2LRhR1rbaTZPcvrs++zee+fCJydBff1W/uEfDvH97z+U9s2+f//3I0D5HMm3pVBYWEh/WTVv9lzn/pr0BklYXG7+bdjExs9H19oRCAS/NXZf7GZEMuN248aNiypiusIKy0Vq25MrrLDCCv8HMnuHNTMzE4vFMuecvfe2cfC9FxgY6MHlTl1zzTJl4qOjB5HJ3dQ31Cx8QZKUllbidJbxs599yMRE6trVbreX3t4xvvOd12lu3pNWrcb1jz3J/xwx02GYZDpOpORimXA4+XXPIAMV9VS0RhtlVqs1KQ0vjUaD1WqNOlZaWkJZeTYH3/sFZosx5XZ6vV7Gxob49xf/njvb1qdVW2zdunv46U/PcPZsL3Z76uPUaLTx+utnOX8+QE1N+ha3+WXlyHbs5sedvYxPp65d7Q0EGJyy8ucXu6l75ADqGB1Iq9VKVtbypX6mk0TRFCussMIKK6SH2VFgGRkZc+Z9jUbDtu1reemXP8ZgGMHvjx+xthhM5knePvgCNfX5FBWlT8amrq6FM2e8vP76OYxGW8r3s9mcnD59g+eeO8/atbvS0MIwKpWK2s89yZ9duM6AxYonTlTfYhmfdvCTzj4EO3ZTWF4R9ZrVak3KeapWq5mairaXm1fVEwiZOH7yfaamzCm30+mcpre3iw8+foVdu7enfL8IMpmM5ubd/Lf/9jL9/eO43anb0gaDhZ/+9EM8nnJKSioWviBJKlo3MFzVyK96BplwpG6fOrw+OieM/MOAgdYnnp7zusVi+a2x+xJFxK6wwm8DKxHQK3xqSaYI2Aor/GcwO/1OJpOh0WhwOBxRkRQikYh7dt/BpYtnOX3ufTRqLWKJlIDfP7PDHkl1EYslM4sVqUyG1+OZSUeXyeRMTIyiVMmorimhqCj9sh65ucUYjXJ+8pNzuFwWamrycLncKBQyXC4PcrkEt9t36/NK8Hh8iERCAoHw55BIwlOJUCigv38KuTyPTZvuT/h+S0UkErH+swd499wZrL095AmCKIVCPH4/kc1/mViExx9AKhbh9d9KFZeKcXn9M68ppGJuTrsQZmVTsH4bZWVzoyumpqaSijLX6/V0dHTMOZ5fkItMLubkqTcxm20UFpTh8biRSmV4vZ6otKbIMYFASCgU6VPJrfQmIUbTKHq9lv33pyeqaDYCgYDW1ns4fLiDV1/tRK+Xo1ZLcbluL0pkMikejxexWIjfH25fZGxIpRK8Xh9yuYzhYTPBoAy9vprq6vTLeugKCxHtfYDnTp3A3n2NugwVTq8PhUSMy+dHIhLhu5ViJ5dKcHujDXSZWEwgFEQsENLn8aEqLWf9k3vjFkebnp5Gp0tdx/CTIDYd+LcpjXSFFWYjFApXJDhW+NQRK7WWnZ3N9PTcYn5KpZx79mzj/LnfYDRO8f+3d3Y/bVxpGH9mbI9tKGUIhvBh85FGsKFl6UJJ0yaqEkGVknSlrZpKVdtEinrXP6ZSr6q9rRR1paxaNWobRVWVSJtdbZpoq4pElICyoRDIBiKZGLANtmcvYBx7POOZsWdgnD6/G8zM+XjPmTlz3jlzznNaIzFkcxn4fD5sbm5CEPDUV5GCSG9ufzxUfcFgMIR0OoVgMIjFpTlEmmUMDvWhsdF5uYS+vpfx22//xWef/ROimEYstg/JZDrv44VCUn5w0u/3IZPJ5n1BdaZwOBxEIpHCysomGhs7MDzs3OCzSjgcxqtnz+Nv/7qOxK/3cUDyI6Pk4BMEpHd8PLVeVV8AeOoL1kkBbGxuoV4K4Ncn62ho70Ds5J/R1FSqgZ1IJPDCC+byZm1tbYjH49i/f3/R8YGBPszPL+DqP/6OdCqDlkgb0uk0ApKErc3NvK8HAD6fH9lsRrPEXdjx/ZNY21hFe1sLxsedG3xWCQaDeO21v+DChVtYX/83entl5HLbN+bWTv2pMgKF94HqC6rHwuEg7t79H+rq9qGzcwDNzRHDPCsl9vIwFiKt+Ot/biJz9zf0NGz7fcGAD+mtLMJSAMkdXy/gE7GVzRX5gqIoQPL5kFIUPMooqH+hDyPjpbOAE4kEGhsba2KjO0EQivw8URQd3eSXELfhWwrxLLW0BJr8vtDqlI6MjGByclJ3adPQy4MAtp0bK/ezVldJURQM/anX9UGlSCSCSOQN5HI5PHnyBA0N24MA6ibahau9jPwzQRBw+LB7G8Wp/GFkFBgZxdraGnK5HIKaDRJC6u7kBXGkgnOKouCP9fWGenpbW1u4f/8+PvzwQ1NbOjs7MTk5ieXlZbS0FMtYNDU14eixw1AUBfF4PD+4ol5XdaDFbGOI0cZ+Uzuqpa9vEMAg1tfXkc1m4fcX2yNJKLHZ71d2dOO2jw0MhF133mVZhnzyFIDtjwSNO5qxkk79BXWWJKplOFLmhX5qagoDAwM18SIClD6P9LTfCKkFcrkcJTiI51AUpWjGYTAYRH9/P6anp9HfX9w/+/1+HH51WyZLO0vaTn4jo32VG2yRWKwXsVgvNjc3kUwm0dCw3XdKkrIz6F7aFrVuU0uLHz097uheq4iiiBePvQHgDTx58iT/jNATJFF7bdXfEwQBgR2f62iZ2a2PHj2CKIqWNkwdHBzEhQsXcODAgRI/MhaLIhaLYmtrq0RvVc/X09v4zu/3F01ocYsXX9zWAS+qU02lau8DSXp6TFEUHDlyxPWPhtFoFNFoFJlMBmtra2jcyU811cxTEwQBdaKILs1Kt0Lu3LmD4eFhZwx2GUVRivy8XC5nqG9MiBfhADQhhNhEu1w/Go0ikUjgp59+MtTXamhwVgvaLURRrJklaNrr4ASKouDatWs4ffq05Tjj4+O4ePEiXn/9dV2bBEHQnW3jRXbjpccp3LhPZ2ZmkE6nceyY87OO3EAQBEclWQghhJSi1V8eHh7GDz/8gNnZWRw8qL83gRszl91AkqSa+eDqhL6wlkQigZ9//tnWpqmnT5/GlStXMD6uP+s7EAjUzPV3o07dwO/3u+L33bhxAwcOHEBnZ6fjabtFrbRXQvTgADTxNFxSQryGIAgIh8N4/Phx0RL9Q4cOYWNjAzdu3EBnZyeam5vzLyzaWc128lI0s3udmB1mNuN2L6nWjkrqGNjWfovH45icnMQ777xjy8n1+/04c+YMvvnmG0SjUTQ1NSESiVjaVdno9+8BK+U1C1Pp7tWF5zKZDFZWVvD48WNks1mMjY1ZLMHes7y8XPLyaDSznxCv4/P5KMFBPMf+/fuxuLhYcvzNN9/Ejz/+iDt37mDfvn2IRCL51WpO9E2VnLca5lmiEp9KEAQsLy9jdXUVCwsLeO+992zJPsqyjJMnT+Lrr7/G4OAgZFnOTzSoxs+3i1eutZvvFdW+C+m9g6VSKaysrODBgwdob2/HoUOHqrZzt1hZWSl6/xRFkZKlpKbgADTxNFxSQrxILBZDPB4v0YgdGRnB0tISZmZmMDs7i0wmg2w2i0AggGQyibq6OmxsbOS11QDktZ5VCh2kcDiMZDKZP6fGVwmFQkUbIgJPtQRVgsFgySZl2nT08jJK3+oxvXyLte5QVA9GtumF0ZbRqM4K8yu0UY0viiJCoRBEUUQgEEBbWxvOnz+PSggEAjhz5gx++eUX3Lt3D7dv30Y2m0Uqlcrn5/P5kM1mi8pYaK/2GmjvDTW+iraO9a6h3nXQOu9W6tjOMa3dRnbo3YfaMmrDaMtYrs4Kz6n5q8fUv36/H8FgELIso7u723Amm1dZXV1FLBYrOsbNaUitks1mPTGYQoiW7u7ukokHADA2Noa7d+9ifn4e09PTSKfTyGQy+T5G9T0K+0AjP0V7Dij1r7R9orbP1AsDWPPJgOr6fq0voedb6NmhLbM2jLaM5eqs8LdaD2p8NV4oFILP50N9fT3a29vx7rvvohJkWcb58+dx8+ZNTE1N5X3+VCqFXC5XVEeF17HQ/kJ7tfVlpY71rrU2jFWfTM+HrOaYlftQr4xm93y5Oissq3pOjR8Oh5HJZHZkZiS0tLRgeHjYkuyKl9jY2CjyVXO5HCVLSU3BAWjiWbQi+4R4haNHj+Lzzz9HV1dXyWzD9vb2mnNmfu/oafBVytDQEIaGhhxJi5ByrK+v4/79+yUfTbg0k9Qqfr+fM6CJJzl+/Di++uorvPXWWyXn+vr60NfnvmYz8Sajo6MAnPUlCdFjbm4OkiSht7c3f4ybEJJag/P1iWcRRZEzYYhn+eijj3DlypWSr/uk9uALA6k1VldXcfXq1RLNylwux6WYpGbhs5h4FVmWcerUKXz//fd7bQrxKHx+ETeZn5/H7OwsJiYmio7T7yO1BqeXEs8SjUaxsLCw12YQoktDQwPef/99XLx4Ea2trWhqaoIsy/llUNplYXb00arVAHRaE84tjblqNbHdiqO3bNSNOM+SbW5fGzfCOhWu0vNW8levnSAIEAQhr1Uej8dx7ty5ko2xFhcX0dPTY1IiQrxJR0cHbt26tddmEKJLW1sbxsbG8N1336G9vR0tLS2or6/P96/V7OmwG1rAdnwBt/pnu/7Ibvl9u+WPPUu2uR3ejfvVi36f0fui+kFjeXkZGxsbUBQFZ8+eLUnz4cOHeOWVV0zLRIhX4AA08SySJKGnp0dXc40QL/D888/j448/xtTUFBYXFzEzM5PXMjPT7wX09fcAfU3lQsycMrP4dtHTjzPCqEzVhgUqc6grqQs75VXR09MzoxLbKslnN2yr5AVJr00YYacMVsNavZ/M0jMrh1F8veNGGu6CIMDn80GWZRw8eNBww5z19fWipZmE1BJdXV22n72E7Ca9vb04d+4cpqensbS0hIcPH+bv2cK+wE7/BpTvc/V0cu3EV7Hjc9nxt+yEteuPVOK/VOLD2b1elcapxLZKfMXd8H3thneyTVQa1ur9ZNZWzNIxarNW9sxRwwQCAYiiiObmZnR1dRnuUZJKpdDd3W1WJEI8g6BQ44B4mLW1NXz66af44IMP9toUQgghxLNcv34dsVgMY2Nje20KIRUzPT2Ny5cv4+23395rUwghhBDP8u2332JiYgL9/f17bQohlqFgDPE0zz33HD755BN88cUXJbvrEkIIIWR78DkSiXDwmdQ8/f39OHHiBC5durTXphBCCCGe5NKlSzhx4gQHn0nNwRnQpCZYWlrCl19+CZ/Ph46ODoRCobw+kqqPqd7KhUt8tMvC9ZaJ6R2zotFU6XGj9K2cM1u+ZLYMrtr4VsNYWWZldcmelaX9Vpd1WQ1n1Tanw9kJa2cpm1V5BDfStCPN4PT1sRPW6XBWy70X6TmVp1Oak9XkYxa3XP5G91u5NK32F0bpa+OX+7/wt5qW+lcURQiCgAcPHiCVSmFoaAjHjx/XtZmQWuT27du4fPkywuEwYrFYvp0VtgFt+wBK25ReO9S2WaM2bPT8cCq82Tkj++1QrV9oJY3dDmMnnNN+hdP+phv6yW5qMjudptO+kp009+pa7oW/6VTbc+KdzCyfap+JlTxv7R63+qzXi6+136wv0+Y1Pz+PZDKJiYkJvPTSS7rlJMTLcACa1BRzc3NYWFhAOp2GoigQRRGZTKbo4VxOt0mv07IySFAuXaOOsJwdlQyKmKUJmOvEVXveig1OhgGsOTtW7Aas65VZtc1qODsvkFbD2tHys5qm1Xq0k6adslvN3400nb6WTg/QO3lfOFUGK7ZbsceNQWSVctffyLZqPgKapa+Nrw1T+H+h7ervwr+CIKC1tRUdHR2QZdmSTYTUEltbW5ibm8Pi4iI2NzehKEq+jRS2j8J2o21jes8AbRs3atd2P1IZPW/KPTfMBmPM+rBqB2us+EVW+lErz0Yn+yerfbZVH8DpcE76uHbTtDNYa3cvDqfTtGqrnTSt1qnT95DTvraTfqlTbdhKGLNnSjUDyIB5WSrxGe1+6KsmHa39hXH0fED1mCAIkCQJHR0d6O7uRiAQsGwvIV6CA9CEEEIIIYQQQgghhBBCXIEa0IQQQgghhBBCCCGEEEJcgQPQhBBCCCGEEEIIIYQQQlyBA9CEEEIIIYQQQgghhBBCXMEP4NpeG0EIIYQQQgghhBBCCCHk2eP/kTneEDRL+UwAAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "id": "dbc9e46b",
   "metadata": {
    "papermill": {
     "duration": 0.008422,
     "end_time": "2022-07-05T00:56:39.528588",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.520166",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 4.2 ResNet9\n",
    "构造9层的ResNet，参照[github](https://github.com/davidcpage/cifar10-fast)：\n",
    "<img src=\"attachment:image-2.png\" alt=\"图片名称\" align=center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0bea6cc2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.547114Z",
     "iopub.status.busy": "2022-07-05T00:56:39.546870Z",
     "iopub.status.idle": "2022-07-05T00:56:39.562475Z",
     "shell.execute_reply": "2022-07-05T00:56:39.561583Z"
    },
    "papermill": {
     "duration": 0.027266,
     "end_time": "2022-07-05T00:56:39.564406",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.537140",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class ResNet9Block(Sequential):\n",
    "    def __init__(self, in_channels, out_channels, max_pooling=True):\n",
    "        layers = [\n",
    "            Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=3, padding=1),\n",
    "            BatchNorm2d(num_features=out_channels),\n",
    "            ReLU()\n",
    "        ]\n",
    "        if max_pooling is True:\n",
    "            layers.append(MaxPool2d(kernel_size=(2, 2), stride=2))\n",
    "        super(ResNet9Block, self).__init__(*layers)\n",
    "\n",
    "        \n",
    "class ResNet9(BaseModule):\n",
    "    def __init__(self, config):\n",
    "        super(ResNet9, self).__init__(config)\n",
    "        self.conv1 = ResNet9Block(in_channels=3, out_channels=32, max_pooling=False)\n",
    "        self.conv2 = ResNet9Block(in_channels=32, out_channels=64, max_pooling=True)\n",
    "        self.res1 = Sequential(\n",
    "            ResNet9Block(in_channels=64, out_channels=64, max_pooling=False),\n",
    "            ResNet9Block(in_channels=64, out_channels=64, max_pooling=False)\n",
    "        )\n",
    "        self.conv3 = ResNet9Block(in_channels=64, out_channels=128, max_pooling=False)\n",
    "        self.conv4 = ResNet9Block(in_channels=128, out_channels=256, max_pooling=True)\n",
    "        self.res2 = Sequential(\n",
    "            ResNet9Block(in_channels=256, out_channels=256, max_pooling=False),\n",
    "            ResNet9Block(in_channels=256, out_channels=256, max_pooling=False)\n",
    "        )\n",
    "        self.conv5 = ResNet9Block(in_channels=256, out_channels=512, max_pooling=False)\n",
    "        self.conv6 = ResNet9Block(in_channels=512, out_channels=1024, max_pooling=True)\n",
    "        self.res3 = Sequential(\n",
    "            ResNet9Block(in_channels=1024, out_channels=1024, max_pooling=False),\n",
    "            ResNet9Block(in_channels=1024, out_channels=1024, max_pooling=False)\n",
    "        )\n",
    "        self.fc = Sequential(\n",
    "            MaxPool2d(kernel_size=(4, 4), stride=4),\n",
    "            Flatten(),\n",
    "            Dropout(dropout_ratio=config.dropout),\n",
    "            Linear(in_features=config.conv6_channels, out_features=config.class_num)\n",
    "        )\n",
    "\n",
    "        self.loss = CrossEntropyLoss()\n",
    "        self.optimizer = SGD(\n",
    "            params=self.parameters(),\n",
    "            lr=config.lr,\n",
    "            weight_decay=config.weight_decay\n",
    "        )\n",
    "        self.scheduler = OneCycleLR(\n",
    "            optim=self.optimizer,\n",
    "            max_lr=config.lr,\n",
    "            epochs=config.epoch_num,\n",
    "            steps_per_epoch=config.train_batch_num\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        conv1_out = self.conv1(x)\n",
    "        conv2_out = self.conv2(conv1_out)\n",
    "        res1_out = self.res1(conv2_out) + conv2_out\n",
    "        conv3_out = self.conv3(res1_out)\n",
    "        conv4_out = self.conv4(conv3_out)\n",
    "        res2_out = self.res2(conv4_out) + conv4_out\n",
    "        conv5_out = self.conv5(res2_out)\n",
    "        conv6_out = self.conv6(conv5_out)\n",
    "        res3_out = self.res3(conv6_out) + conv6_out\n",
    "        scores = self.fc(res3_out)\n",
    "        return scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dbcef91",
   "metadata": {
    "papermill": {
     "duration": 0.008898,
     "end_time": "2022-07-05T00:56:39.582202",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.573304",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 4.3 常规ResNet的基本模块\n",
    "实现了ResNet的基本模块——残差块，供后续叠加使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2e89218e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.600890Z",
     "iopub.status.busy": "2022-07-05T00:56:39.600582Z",
     "iopub.status.idle": "2022-07-05T00:56:39.620830Z",
     "shell.execute_reply": "2022-07-05T00:56:39.619704Z"
    },
    "papermill": {
     "duration": 0.032518,
     "end_time": "2022-07-05T00:56:39.623232",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.590714",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class BasicBlock(BaseModule):\n",
    "    def __init__(self, in_channels, downsample=False):\n",
    "        super(BasicBlock, self).__init__()\n",
    "\n",
    "        if downsample is True:\n",
    "            out_channels = 2 * in_channels\n",
    "        else:\n",
    "            out_channels = in_channels\n",
    "\n",
    "        self.conv1 = Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=3, padding=1, bias=False)\n",
    "        self.bn1 = BatchNorm2d(num_features=out_channels)\n",
    "        self.conv2 = Conv2d(in_channels=out_channels, out_channels=out_channels, kernel_size=3, padding=1, bias=False)\n",
    "        self.bn2 = BatchNorm2d(num_features=out_channels)\n",
    "        if downsample is True:\n",
    "            self.downsample = Sequential(\n",
    "                Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=1, bias=False),\n",
    "                BatchNorm2d(num_features=out_channels)\n",
    "            )\n",
    "        else:\n",
    "            self.downsample = None\n",
    "        self.relu = ReLU()\n",
    "\n",
    "    def __call__(self, x):\n",
    "        identity = x\n",
    "\n",
    "        out = self.conv1(x)\n",
    "        out = self.bn1(out)\n",
    "        out = self.relu(out)\n",
    "\n",
    "        out = self.conv2(out)\n",
    "        out = self.bn2(out)\n",
    "\n",
    "        if self.downsample is not None:\n",
    "            identity = self.downsample(x)\n",
    "\n",
    "        out += identity\n",
    "        out = self.relu(out)\n",
    "\n",
    "        return out\n",
    "\n",
    "\n",
    "class Bottleneck(BaseModule):\n",
    "    def __init__(self, in_channels, out_channels, stride=1, downsample=False, width=1):\n",
    "        super(Bottleneck, self).__init__()\n",
    "        bottleneck_channels = width * out_channels // 4\n",
    "        self.conv1 = Conv2d(in_channels=in_channels, out_channels=bottleneck_channels, kernel_size=1, bias=False)\n",
    "        self.bn1 = BatchNorm2d(num_features=bottleneck_channels)\n",
    "        self.conv2 = Conv2d(in_channels=bottleneck_channels, out_channels=bottleneck_channels, kernel_size=3, stride=stride, padding=1, bias=False)\n",
    "        self.bn2 = BatchNorm2d(num_features=bottleneck_channels)\n",
    "        self.conv3 = Conv2d(in_channels=bottleneck_channels, out_channels=out_channels, kernel_size=1, bias=False)\n",
    "        self.bn3 = BatchNorm2d(num_features=out_channels)\n",
    "        self.relu = ReLU()\n",
    "        if downsample is True:\n",
    "            self.downsample = Sequential(\n",
    "                Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=1, stride=stride, bias=False),\n",
    "                BatchNorm2d(num_features=out_channels)\n",
    "            )\n",
    "        else:\n",
    "            self.downsample = None\n",
    "        self.stride = stride\n",
    "\n",
    "    def __call__(self, x):\n",
    "        identity = x\n",
    "\n",
    "        out = self.conv1(x)\n",
    "        out = self.bn1(out)\n",
    "        out = self.relu(out)\n",
    "\n",
    "        out = self.conv2(out)\n",
    "        out = self.bn2(out)\n",
    "        out = self.relu(out)\n",
    "\n",
    "        out = self.conv3(out)\n",
    "        out = self.bn3(out)\n",
    "\n",
    "        if self.downsample is not None:\n",
    "            identity = self.downsample(x)\n",
    "\n",
    "        out += identity\n",
    "        out = self.relu(out)\n",
    "\n",
    "        return out\n",
    "\n",
    "\n",
    "class ResNetLayer(Sequential):\n",
    "    def __init__(self, in_channels, blocks, stride=None, out_channels=None, fisrt_layer_downsample=True, width=1):\n",
    "        if out_channels is None:\n",
    "            layers = [BasicBlock(in_channels=in_channels, downsample=fisrt_layer_downsample)]\n",
    "            for _ in range(1, blocks):\n",
    "                layers.append(BasicBlock(in_channels=in_channels if not fisrt_layer_downsample else in_channels * 2))\n",
    "        else:\n",
    "            layers = [Bottleneck(in_channels=in_channels, out_channels=out_channels, stride=stride, downsample=True, width=width)]\n",
    "            for _ in range(1, blocks):\n",
    "                layers.append(Bottleneck(in_channels=out_channels, out_channels=out_channels, width=width))\n",
    "        super(ResNetLayer, self).__init__(*layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d69e2ea",
   "metadata": {
    "papermill": {
     "duration": 0.008516,
     "end_time": "2022-07-05T00:56:39.640601",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.632085",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 4.4 ResNet18\n",
    "18层的ResNet，由17个卷积层（conv）和1个全连接层（fc）构成。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6406db98",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.659698Z",
     "iopub.status.busy": "2022-07-05T00:56:39.659388Z",
     "iopub.status.idle": "2022-07-05T00:56:39.672335Z",
     "shell.execute_reply": "2022-07-05T00:56:39.671283Z"
    },
    "papermill": {
     "duration": 0.024755,
     "end_time": "2022-07-05T00:56:39.674404",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.649649",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class ResNet18(BaseModule):\n",
    "    def __init__(self, config):\n",
    "        super(ResNet18, self).__init__(config)\n",
    "        self.conv1 = Conv2d(in_channels=3, out_channels=64, kernel_size=7, stride=2, padding=3, bias=False)\n",
    "        self.bn1 = BatchNorm2d(num_features=64)\n",
    "        self.relu = ReLU()\n",
    "        self.maxpool = MaxPool2d(kernel_size=3, stride=2, padding=1)\n",
    "        self.layer1 = ResNetLayer(in_channels=64, blocks=2, fisrt_layer_downsample=False)\n",
    "        self.layer2 = ResNetLayer(in_channels=64, blocks=2)\n",
    "        self.layer3 = ResNetLayer(in_channels=128, blocks=2)\n",
    "        self.layer4 = ResNetLayer(in_channels=256, blocks=2)\n",
    "        self.avgpool = AdaptiveAvgPool2d(output_size=(1, 1))\n",
    "        self.flatten = Flatten()\n",
    "        self.fc = Sequential(\n",
    "            Linear(in_features=512, out_features=128),\n",
    "            ReLU(),\n",
    "            Linear(in_features=128, out_features=10)\n",
    "        )\n",
    "\n",
    "        self.loss = CrossEntropyLoss()\n",
    "        self.optimizer = SGD(\n",
    "            params=self.parameters(),\n",
    "            lr=config.lr,\n",
    "            momentum=config.momentum\n",
    "        )\n",
    "        self.scheduler = OneCycleLR(\n",
    "            optim=self.optimizer,\n",
    "            max_lr=config.lr,\n",
    "            epochs=config.epoch_num,\n",
    "            steps_per_epoch=config.train_batch_num\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.bn1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.maxpool(x)\n",
    "\n",
    "        x = self.layer1(x)\n",
    "        x = self.layer2(x)\n",
    "        x = self.layer3(x)\n",
    "        x = self.layer4(x)\n",
    "\n",
    "        x = self.avgpool(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.fc(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edfe6444",
   "metadata": {
    "papermill": {
     "duration": 0.010109,
     "end_time": "2022-07-05T00:56:39.693073",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.682964",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 4.5 ResNet50 & WRN50_2\n",
    "该模块可被ResNet50、WRN50_2所通用，使用时只需调整`config`类，通过width属性控制bottleneck的通道数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "68f4d30d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.711845Z",
     "iopub.status.busy": "2022-07-05T00:56:39.711580Z",
     "iopub.status.idle": "2022-07-05T00:56:39.724387Z",
     "shell.execute_reply": "2022-07-05T00:56:39.723368Z"
    },
    "papermill": {
     "duration": 0.024917,
     "end_time": "2022-07-05T00:56:39.726564",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.701647",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class ResNet50(BaseModule):\n",
    "    def __init__(self, config):\n",
    "        super(ResNet50, self).__init__(config)\n",
    "        self.conv1 = Conv2d(in_channels=3, out_channels=64, kernel_size=7, stride=2, padding=3, bias=False)\n",
    "        self.bn1 = BatchNorm2d(num_features=64)\n",
    "        self.relu = ReLU()\n",
    "        self.maxpool = MaxPool2d(kernel_size=3, stride=2, padding=1)\n",
    "        self.layer1 = ResNetLayer(in_channels=64, out_channels=256, blocks=3, stride=1, width=config.width)\n",
    "        self.layer2 = ResNetLayer(in_channels=256, out_channels=512, blocks=4, stride=2, width=config.width)\n",
    "        self.layer3 = ResNetLayer(in_channels=512, out_channels=1024, blocks=6, stride=2, width=config.width)\n",
    "        self.layer4 = ResNetLayer(in_channels=1024, out_channels=2048, blocks=3, stride=2, width=config.width)\n",
    "        self.avgpool = AdaptiveAvgPool2d(output_size=(1, 1))\n",
    "        self.flatten = Flatten()\n",
    "        self.fc = Sequential(\n",
    "            Linear(in_features=2048, out_features=512),\n",
    "            ReLU(),\n",
    "            Linear(in_features=512, out_features=128),\n",
    "            ReLU(),\n",
    "            Linear(in_features=128, out_features=10)\n",
    "        )\n",
    "        self.loss = CrossEntropyLoss()\n",
    "        self.optimizer = SGD(\n",
    "            params=self.parameters(),\n",
    "            lr=config.lr,\n",
    "            momentum=config.momentum,\n",
    "            weight_decay=config.weight_decay\n",
    "        )\n",
    "        self.scheduler = OneCycleLR(\n",
    "            optim=self.optimizer,\n",
    "            max_lr=config.lr,\n",
    "            epochs=config.epoch_num,\n",
    "            steps_per_epoch=config.train_batch_num\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.bn1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.maxpool(x)\n",
    "\n",
    "        x = self.layer1(x)\n",
    "        x = self.layer2(x)\n",
    "        x = self.layer3(x)\n",
    "        x = self.layer4(x)\n",
    "\n",
    "        x = self.avgpool(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.fc(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec3f8b8d",
   "metadata": {
    "papermill": {
     "duration": 0.00894,
     "end_time": "2022-07-05T00:56:39.744660",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.735720",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 5.神经网络训练工具类：Trainer\n",
    "集成了模型训练、测试、模型存储、训练日志存储、训练过程的准确率和损失值可视化等功能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "813f04f0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.763774Z",
     "iopub.status.busy": "2022-07-05T00:56:39.763502Z",
     "iopub.status.idle": "2022-07-05T00:56:39.797394Z",
     "shell.execute_reply": "2022-07-05T00:56:39.796594Z"
    },
    "papermill": {
     "duration": 0.045499,
     "end_time": "2022-07-05T00:56:39.799291",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.753792",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Trainer:\n",
    "    def __init__(self, config, network, datamanager):\n",
    "        # 用时信息\n",
    "        self.start_time = config.start_time   # 获取程序开始时间\n",
    "        self.end_time = None                  # 定义程序结束时间\n",
    "\n",
    "        # 保存传入参数\n",
    "        self.config = config                  # 获取配置类实例的入口\n",
    "        self.network = network                # 获取神经网络实例的入口\n",
    "        self.datamanager = datamanager        # 获取数据预处理类实例的入口\n",
    "\n",
    "        # 训练过程中需要记录的信息\n",
    "        self.best_model = None                # 暂存最优模型参数\n",
    "        self.best_acc = 0.0                   # 记录最高准确率\n",
    "        self.training_log = []                # 日志文件信息列表\n",
    "        self.train_acc = []                   # 训练全程训练集准确率列表\n",
    "        self.test_acc = []                    # 训练全程测试集准确率列表\n",
    "        self.loss = []                        # 训练全程损失函数值列表\n",
    "\n",
    "    # 训练函数，包含训练集、测试集上准确率的计算\n",
    "    def train(self):\n",
    "        self.network.train()            # 将神经网络设置为训练状态\n",
    "        print('Epoc\\tTrain\\tTest\\tLoss')\n",
    "        \n",
    "        dataloader = self.datamanager.get_train_dataloader()\n",
    "        for i in range(self.config.epoch_num):\n",
    "            loss = 0\n",
    "            start = time.time()\n",
    "            for j, data in enumerate(dataloader):\n",
    "                data[0] = data[0].to(self.config.device)\n",
    "                data[1] = data[1].to(self.config.device)\n",
    "                loss += self.network.step(data)\n",
    "                end = time.time()\n",
    "                info = '\\rEpoch ' + str(i + 1) + ' completed: ' + str(\n",
    "                    round(j * 100 / len(dataloader))) + ' % ' + str(round(end - start, 2)) + 's'\n",
    "                print(info, end='')      # 打印进度信息\n",
    "            loss /= len(dataloader)      # 计算每个batch上的平均损失值\n",
    "            self.loss.append(loss)\n",
    "            acc_info = '{:<8d}'.format(i + 1)\n",
    "            train_acc = self.test(mode='Train')        # 计算训练集准确率\n",
    "            acc_info = acc_info + '{:<8.1f}'.format(train_acc * 100)\n",
    "            test_acc = self.test(mode='Test')          # 计算测试集准确率\n",
    "            acc_info = acc_info + '{:<8.1f}'.format(test_acc * 100)\n",
    "            acc_info = acc_info + '{:<8.4f}'.format(loss)\n",
    "            print('\\r' + acc_info + '                                          ')   # 打印一个epoch上的准确率和损失值信息\n",
    "\n",
    "        self.end_time = time.time()\n",
    "        print('==================================================')\n",
    "        print('Time consumed:', round(self.end_time - self.start_time, 1), 's')     # 打印全程用时\n",
    "\n",
    "    # 测试函数，包括Train和Test两种模式，分别计算训练集和测试集的准确率\n",
    "    def test(self, mode='Train'):\n",
    "        self.network.eval()              # 将神经网络设置为测试状态\n",
    "        if mode == 'Train':              # 获取对应模式的DataLoader\n",
    "            dataloader = self.datamanager.get_train_dataloader()\n",
    "        else:\n",
    "            dataloader = self.datamanager.get_test_dataloader()\n",
    "\n",
    "        correct_num = 0\n",
    "        overall_num = 0\n",
    "        length = len(dataloader)\n",
    "        for i, data in enumerate(dataloader):\n",
    "            data[0] = data[0].to(self.config.device)\n",
    "            data[1] = data[1].to(self.config.device)\n",
    "            c, o = self.network.evaluate(data)\n",
    "            correct_num += c\n",
    "            overall_num += o\n",
    "            if mode == 'Train':\n",
    "                info = 'Calculating accuracy on training set: ' + str(round(100 * i / length)) + ' %'\n",
    "            else:\n",
    "                info = 'Calculating accuracy on testing set:' + str(round(100 * i / length)) + ' %'\n",
    "            print('\\r' + info + '        ', end='')      # 打印进度信息\n",
    "        acc = correct_num / overall_num\n",
    "\n",
    "        if mode == 'Train':              # 记录准确率并保存最优模型\n",
    "            self.train_acc.append(acc)\n",
    "        else:\n",
    "            self.test_acc.append(acc)\n",
    "            if acc > self.best_acc:\n",
    "                self.best_acc = acc\n",
    "                if self.best_model is not None:\n",
    "                    del self.best_model\n",
    "                self.best_model = copy.deepcopy(self.network.state_dict())\n",
    "        self.network.train()             # 将神经网络设置为训练状态\n",
    "        return acc\n",
    "\n",
    "    # 保存在测试集上准确率最高的模型\n",
    "    def save_model(self):\n",
    "        model = self.network.__class__.__name__\n",
    "        t = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')\n",
    "        filename = model + '_' + t + '.pt'\n",
    "        path = './Model/' + filename\n",
    "        torch.save(self.best_model, path)\n",
    "\n",
    "    # 保存训练日志\n",
    "    def save_training_log(self):\n",
    "        model = self.network.__class__.__name__\n",
    "        t = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')\n",
    "        filename = model + '_' + t + '.txt'\n",
    "        path = './TrainingLog/' + filename\n",
    "        file = open(path, 'w')\n",
    "        training_info = self.config.get_training_info()\n",
    "        for info in training_info:\n",
    "            file.write(info + '\\n')\n",
    "        file.write('==============================\\n')\n",
    "        file.write('Epoc    Train   Test    Loss    \\n')\n",
    "        for i in range(self.config.epoch_num):\n",
    "            line = '{:<8d}'.format(i + 1) + '{:<8.1f}'.format(self.train_acc[i] * 100) + \\\n",
    "                   '{:<8.1f}'.format(self.test_acc[i] * 100) + '{:<8.4f}'.format(self.loss[i]) + '\\n'\n",
    "            file.write(line)\n",
    "        file.write('==============================\\n')\n",
    "        line = 'Best acc: ' + str(round(self.best_acc * 100, 1)) + '%\\n'\n",
    "        file.write(line)\n",
    "        line = 'Time consumed: ' + str(round(self.end_time - self.start_time, 1)) + 's\\n'\n",
    "        file.write(line)\n",
    "        file.close()\n",
    "\n",
    "    # 显示并保存准确率和损失值变化曲线\n",
    "    def visualize(self):\n",
    "        model = self.network.__class__.__name__\n",
    "        t = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')\n",
    "        acc_filename = model + '_' + t + 'acc.jpg'\n",
    "        loss_filename = model + '_' + t + 'loss.jpg'\n",
    "        acc_path = './Figure/' + acc_filename\n",
    "        loss_path = './Figure/' + loss_filename\n",
    "\n",
    "        x = np.array(range(1, self.config.epoch_num + 1))\n",
    "        y1 = np.array(self.train_acc)\n",
    "        y1 = y1 * 100\n",
    "        y2 = np.array(self.test_acc)\n",
    "        y2 = y2 * 100\n",
    "        y3 = np.array(self.loss)\n",
    "\n",
    "        plt.figure()\n",
    "        plt.xlabel('Epoch', fontdict={'family': 'Times New Roman', 'size': 14})\n",
    "        plt.ylabel('Accuracy/%', fontdict={'family': 'Times New Roman', 'size': 14})\n",
    "        plt.plot(x, y1, 'dodgerblue', linewidth=1.0, label='Train')\n",
    "        plt.plot(x, y2, 'red', linewidth=1.0, label='Test')\n",
    "        plt.xticks(fontproperties='Times New Roman', size=10)\n",
    "        plt.yticks(fontproperties='Times New Roman', size=10)\n",
    "        plt.legend(prop={'family': 'Times New Roman', 'size': 10})\n",
    "        plt.savefig(acc_path)\n",
    "        plt.show()\n",
    "\n",
    "        plt.figure()\n",
    "        plt.xlabel('Epoch', fontdict={'family': 'Times New Roman', 'size': 14})\n",
    "        plt.ylabel('Loss', fontdict={'family': 'Times New Roman', 'size': 14})\n",
    "        plt.plot(x, y3, 'darkviolet', linewidth=1.0, label='Train')\n",
    "        plt.xticks(fontproperties='Times New Roman', size=10)\n",
    "        plt.yticks(fontproperties='Times New Roman', size=10)\n",
    "        plt.legend(prop={'family': 'Times New Roman', 'size': 10})\n",
    "        plt.savefig(loss_path)\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d5be281",
   "metadata": {
    "papermill": {
     "duration": 0.008497,
     "end_time": "2022-07-05T00:56:39.816413",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.807916",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 6.模型训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e042c1b6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-05T00:56:39.836137Z",
     "iopub.status.busy": "2022-07-05T00:56:39.834729Z",
     "iopub.status.idle": "2022-07-05T03:58:18.813372Z",
     "shell.execute_reply": "2022-07-05T03:58:18.803253Z"
    },
    "papermill": {
     "duration": 10898.991622,
     "end_time": "2022-07-05T03:58:18.816823",
     "exception": false,
     "start_time": "2022-07-05T00:56:39.825201",
     "status": "completed"
    },
    "scrolled": false,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n",
      "Epoc\tTrain\tTest\tLoss\n",
      "1       33.1    33.4    2.2915                                            \n",
      "2       93.6    92.5    1.1782                                            \n",
      "3       98.0    95.8    0.1744                                            \n",
      "4       98.5    96.0    0.0998                                            \n",
      "5       99.0    96.2    0.0710                                            \n",
      "6       99.6    96.8    0.0436                                            \n",
      "7       99.8    97.2    0.0269                                            \n",
      "8       99.8    97.2    0.0190                                            \n",
      "9       100.0   97.4    0.0112                                            \n",
      "10      100.0   97.6    0.0055                                            \n",
      "11      100.0   97.8    0.0030                                            \n",
      "12      100.0   97.7    0.0024                                            \n",
      "13      100.0   97.8    0.0018                                            \n",
      "14      100.0   97.7    0.0015                                            \n",
      "15      100.0   97.7    0.0015                                            \n",
      "==================================================\n",
      "Time consumed: 10898.2 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEKCAYAAADn+anLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnN0lEQVR4nO3de3xcdZ3/8denSZo0TWnaNL3QtJ2WFgQVWswi4A3osqgg8GOFH66yVVm7+FBBXX9cvP8eKyvsrot23VWrIhVQQYRFZdefUIugyKVc5FYupT1pk7ZpmqbXtM3t8/vjnNBpOpNMmpk5k5n38/E4jznzPWcy75ZyPjnne77fY+6OiIhIKmPiDiAiIoVLRUJERNJSkRARkbRUJEREJC0VCRERSas87gDZNGXKFE8kEnHHEBEZVZ588slt7l6faltRFYlEIsHq1avjjiEiMqqYWVO6bbrcJCIiaalIiIhIWioSIiKSloqEiIikpSIhIiJp5a1ImNnNZrbVzJ5PaptsZveb2avR66So3cxsmZmtNbNnzezkfOUUEZGD8nkmcQvw7gFt1wIr3X0BsDJ6D/AeYEG0LAW+k6eMIiKSJG/jJNz9ITNLDGi+ADgjWl8BPAhcE7X/2MN5zB81s1ozm+Hum/MUV6Qg9Dl090JPX7h0J7/2Qo9D/2z/r79G65603t+OH7r9sP1z+OSAw360p9922PtB9s2V4fxdFMIDFxZNh+qK7P/cuAfTTUs68G8BpkXrM4GNSfs1R22HFQkzW0p4tsHs2bNzl1QKTk8fdPXCgZ7oNVqS27oGtB1Ibkt6n/yZXj/8wHkkr31JB28n6cDeB13Rwb6n9+CB/5AiEO3X61AxBsrHRK9lSevRAmAGFr1CtJ7UTnK7HbqdgeuWo/9gB2OkfD/wew/b19Jvy5lhfFHeMqXxjbOLs0i8zt3dzIZdkN19ObAcoLGxsRAKuhyBnj7o2Acd+2H7PmjfF77fHr3fnvy+M3zt6oXKMhhbBpXl0WvZ4W1jk9qS26ui9xMrD923zA49uI7h0INw8gF1TIr2Qw7QSe3JB/eKsgEH/6gAJLdVJGURiUvcRaK1/zKSmc0AtkbtLcCspP0aojYZBQ70wM4D0bI/fE0+0LcnFYP+ZW93eLCeNA7qxsGkKpg8LlxmHQUnTgvbJ1cd3GdcDn5rEpFDxV0kfgksAW6IXu9Nav+kmf0MeCuwU/0R+XXYgX7/4Qf+dOs9feEBv7YKjqoM1+uqwwN/3TiYU3uwAEyODvwTq8LfykWksOStSJjZTwk7qaeYWTPwFcLicKeZXQ40AZdEu/838F5gLdAJfCRfOYuJO+zpgh0HYEd0oN+RvAxsH+JAP7Hq4OvRE+D4KUltSdurK3SJRKRY5PPupg+k2bQ4xb4OfCK3iUavNW3wyvaDB/udSQf8gW2VZeGBfmIV1EYH/f5lSjXMn3TwvQ70IjJQ3JebZBi2dcINf4Dfb4BTZ0YH/ui3+hPqk4pBVBAmVoUdsSKvc4euLjhwYPCluxt6eg4uA9+naxtq32SH3K6U4taq4Wzv7Q2Xnp6D66mW4Wx3h7Kyg0t5+aHvBy6Zbh8zYHjaUH/mTNevuQamTSPbVCRGgZ4+uPVZWPY4XPQG+N1lMKEy7lQyIt3dsG9fuHR2Hvo6VNvA9aEO+P1LV1e4VFRAZeXgy9ix4UEteamoyLytuvrw9vLygwe1QwY+ePq2TPfN5kG8fzEbWZFJt204f85M1yH8e88BFYkC98Qm+PIqOKoK7vhrOLYu7kQFyj08cO7Zc3DZu/fQ9+naBrb39h78mck/P11bJvv29R08qO/bF24bNy48mI4bd+h6qrb+9YkTYcaMg23jxg19sB/4fuBvsiKDUJEoUG174et/hD9uhC+8Hd53bAn1EXR3w7Zt0NoKW7ceXAa+7+g4eGDv7ISqKqipOXwZP/7wtqOPTt0+fnz4G2W/I7kkkmpfs0MP9jn6rU8k21QkCkxPH/z4z/DvT8DFJ8DKy6BmbNypRsgddu9OfaBP9X7XLqirg6lTDy7TpoWvxx578H1tLUyYEB7cq6vDSwQiklUqEgXk8Rb40qpw7MCd74cFk4/wB7mnvh6d6XXrwdaPZPveveFvzukO+m9/+8H3U6fC5Mk64IsUCBWJAtC6F77+B3i0Gb74Djh3QZpLS+7w29/Cf/4nNDUNfmdKqmvRQ3VWDty3/7Wm5tDPD9yebr3/tbo6XERk1FGRiFF3L6z4M3z7Cbj0jeGlpfGpLi3t2we33grf/Gb4G/mnPw0LFw5+kC+ZDgwRySUViZg82gxffhDqq+Gui2F+qktLmzaFZw3Ll8Opp8K3vw1nnqkCICJ5oyKRZ6174Po/wBMt8MV3wnvnpzjmP/UU3HQT3Hcf/M3fwB//CAsWxJJXREqbbpjOk+5eWP4UnHM7NEyAlX87oO+htxfuuQfe9S648EJ485vhtdfCswcVCBGJic4k8uCRjeGlpaMnwN2XwLxJSRt374abb4Zly6C+Hj7zGbjoIt1HLyIFQUUih9o74Su/h6c3w5feCecck3TmEARhYVixAhYvhttug9NOizOuiMhhVCRyaMWfobcPHrgsekCOO/zhj+FdSqtWwUc/GvY/zJkTd1QRkZRUJHJo4y44MwHj6Iaf/DzsjN6xA666Cm65JRx/ICJSwFQkcmjb1j2csurbcFvU+fylL8G552o0sYiMGioSOfSOu7/J9LYH4Ne/Dge/iYiMMroFNkd6+mDy5tcoX3KZCoSIjFoqEjnSugcSOwLK5yXijiIicsRUJHKkeTfM2h5AIhF3FBGRI6YikSObtvcwuWMTzJoVdxQRkSNWEEXCzK4ys+fN7AUz+3TUNtnM7jezV6PXSUP8mIKyY10L+yZPDWdkFREZpWIvEmb2JuBjwCnAScB5ZjYfuBZY6e4LgJXR+1HjwNqA/UdrkJyIjG6xFwngeOAxd+909x7g98BFwAXAimifFcCF8cQ7Qk1N9M1JxJ1CRGRECqFIPA+8w8zqzKwaeC8wC5jm7pujfbYA01J92MyWmtlqM1vd1taWn8QZqGwOqDgmEXcMEZERib1IuPsa4Ebgt8BvgGeA3gH7OOBpPr/c3RvdvbG+vj7HaTPjDhM2B9Qcm4g7iojIiMReJADc/Yfu/hZ3fyfQAbwCtJrZDIDodWucGYejrRNmdwRU6kxCREa5gigSZjY1ep1N2B/xE+CXwJJolyXAvfGkG76W3TBbYyREpAgUytxNvzCzOqAb+IS77zCzG4A7zexyoAm4JNaEw9DS0cubOlo0RkJERr2CKBLu/o4Ube3A4hjijFjHuk3sq51CRWVl3FFEREakIC43FZsDawP2zUzEHUNEZMRUJHLAg4BejZEQkSKgIpEDYzcGlM9NxB1DRGTEVCSyzB1qNmmMhIgUBxWJLNt1AGZuDxg3PxF3FBGREVORyLLm3eFAOktocj8RGf1UJLKspaOX+u3NMHt23FFEREZMRSLLOtZvZt/EOqiqijuKiMiIqUhk2b5XAzqPTsQdQ0QkK1QksqwvCOjRGAkRKRIqEllWsUFjJESkeKhIZFnNpoDqBYm4Y4iIZIWKRBZ1dsO0toCaBbr9VUSKg4pEFrXshtk7mxijy00iUiRUJLKoZUcfU7dv1BgJESkaKhJZtH3dZvZPmATjxsUdRUQkK1QksmjfqwF7NUZCRIqIikQW9a4P6JmdiDuGiEjWqEhkUdmGgDJ1WotIEVGRyKLxm5sYd4xufxWR4lEQRcLMPmNmL5jZ82b2UzOrMrO5ZvaYma01szvMbGzcOQfT1QtTWgOOOi4RdxQRkayJvUiY2UzgSqDR3d8ElAGXAjcCN7n7fKADuDy+lEPbsgdm7wgom5eIO4qISNbEXiQi5cA4MysHqoHNwFnAXdH2FcCF8UTLTPOOPqZv3wBzdLlJRIpH7EXC3VuAfwU2EBaHncCTwA5374l2awZmpvq8mS01s9VmtrqtrS0fkVNqX7+F/TW1GiMhIkUl9iJhZpOAC4C5wNHAeODdmX7e3Ze7e6O7N9bX1+co5dA6XwnYMyMR2/eLiOTCERUJMxtvZkdlKcNfAuvdvc3du4G7gbcBtdHlJ4AGoCVL35cTPesCumfpUpOIFJdhFQkzO87MHgd2Ax1m9qyZLRphhg3AqWZWbWYGLAZeBFYB74/2WQLcO8LvyakxG5sYk0jEHUNEJKuGeybxPeD7QA0wBfg1YafyEXP3xwg7qJ8CnosyLQeuAT5rZmuBOuCHI/meXKtuCajScyREpMgMWiTM7DYzq0tqmgHc7u6d7t4B3EF4KWhE3P0r7v4Gd3+Tu1/m7gfcfZ27n+Lu8939Ync/MNLvyZU+h8mtARM1RkJEikz5ENsfA54ws6+4+63AT4DHzew+oAL4a0Z4JlEMtu6F2R0BY49JxB1FRCSrBj2TcPd/B94FXGxmvwVuJbwMVA4Y8NloKWkaIyEixWqoMwncfSNwvpldAtxP2C/xf9y9L9fhRott61s5MP4oKqur444iIpJVGXVcm1mdu98JnAzMJ7wEdXJOk40ie14J2DNDZxEiUnyG6rhebGatQJuZNQMnuPtSwktMt5rZN8ys5IcY96xvoqshEXcMEZGsG+pM4j+AfyacT+mTwDcB3P33wCJgH/BM7uKNDmOaAkxjJESkCA1VJGYA97n7fuA3wOvzXrh7l7t/Ebgoh/lGhXEtAZXzE3HHEBHJuqE6rn8J3GVmvwTeDvz3wB3c/YVcBBst3KF2S8BRx50fdxQRkawb6kzicsK7mSYCtwGfznWg0aZjPzRsD6jWaGsRKUJDnUlcAfxXNF5CUmjZ6Ry7vUljJESkKA11JvEe4GUze9rMvmpmC/OQaVRpXddK17gaGD8+7igiIlk31Ijr9xB2Vv8TsABYZWaBmS0zs7PMrCwfIQvZnleb2KXnSIhIkRpyMJ2773H3n7v7BwkLxt8RTsnxI8LxE7eZ2fvNrCR/le5eF3BAYyREpEgNa6pwd+9x9wfc/VPuPofwgUGvAV+kROdwGtMUgMZIiEiRynRajgtTXVpy96eiab4XAjdkO9xoUNUcUKnZX0WkSGV6JnE70GJmN5rZcal2iB49WnImbg6YoOdIiEiRyrRITAe+Qjht+Itm9gcz+0ip9kP029MFM9oDJhybiDuKiEhOZFQk3H23u3/P3U8FTiR8GNHXgc1m9n0zOzWXIQtVy05nZkcTltAYCREpTsN9xnX/NBw3ET6Heizwv4GHzewxMzsxy/kKWuv6Nrorq6GmJu4oIiI5kXGRMLMKM7vEzH4DrAfOIhyRPQ2YA6whfOZ1ydj9SsBOjZEQkSI25JPpAMzs34EPAE74CNPPuvuLSbvsM7NrgU3Zj1i4utYF7NcYCREpYhkVCeAEwudJ3O3uXWn22QacOdwA0d1SyWcg84AvAz+O2hNAAFzi7h3D/fk5FQT4nETcKUREcibTjuvF7v6zQQpE/0C73w83gLu/7O4Lo7EWbwE6gXuAa4GV7r4AWBm9LyhVzQFjNUZCRIpYpoPprjezK1K0X2Fm/5jFPIuB19y9CbgAWBG1rwAuzOL3ZMVRmwMmLNCdTSJSvDLtuL4MeDpF+5PA32YvDpcCP43Wp7n75mh9C2EH+WHMbKmZrTaz1W1tbVmMMrj9PTB1WxMTNZBORIpYpkViKpDqCNxOmoP3cJnZWOB84OcDt7m7E3aaH8bdl7t7o7s31tfXp9olJzbtchq2B5TN1ZmEiBSvTIvEBuAdKdrfCTRnKct7gKfcvTV632pmMwCi161Z+p6s2Bpso2dsFRx1VNxRRERyJtMi8T3gJjP7mJkdEy1LgW8QDqrLhg9w8FIThM/XXhKtLwHuzdL3ZMXOVwJ2Tk/EHUNEJKcyugXW3b9hZlOAZYSjrAG6gG+5+z+PNEQ0B9TZwN8nNd8A3GlmlwNNwCUj/Z5sOvBawD6NkRCRIpfpOAnc/Toz+xrhmAmANe6+Jxsh3H0vUDegrZ3wbqfCtD6gT2MkRKTIZVwk4PWD+RM5yjKqVDYHVJyWctZ0EZGikXGRMLMzCfsNZnPwkhMA7n5WlnMVvJrNTdQsOCfuGCIiOZXpYLoPA/8DTADOILwddhJwMvBi2g8WqZ4+qG8LmHR8Iu4oIiI5lendTZ8DPunuHwC6gevcfRFwG5CVfonRZMvucIxExTyNkRCR4pZpkZgHPBCtHwD6H6DwbeDDWc5U8Fo3tNNbPhYmTow7iohITmVaJNoJLzUBtABvitbrgHHZDlXodr4UsENjJESkBGTacf0w8FfAc8CdwDIzO5vwFtX7c5StYO1fG9A5MxF3DBGRnMu0SHwSqIrWvw70AG8jLBhfy0GugtYXNNE3W/0RIlL8hiwSZlZOODvrfwG4ex9wY25jFbaxGwPK/2J+3DFERHJuyD4Jd+8B/gWoyH2c0aFmU8D4BYm4Y4iI5FymHdePEj41ruT1OdRtDZj0hkTcUUREci7TPonvA/9qZrMJHzS0N3mjuz+V7WCFatvecIzEuPnqkxCR4pdpkfhJ9PpvKbY5UJadOIVvy4btzCsrh9rauKOIiORcpkVibk5TjCI7XwromDbn9dGEIiLFLNPnSTTlOsho0flaE3s1RkJESkRGRcLMLhpsu7vfnZ04ha9vfUDvrETcMURE8iLTy013pWn36LVk+iTKNwSULZwXdwwRkbzI6BZYdx+TvBA+T+KthNN1vDOXAQtNzaaAao2REJESkek4iUO4e4+7PwF8HvjP7EYqXO4wuVVjJESkdAzr8aUp7ACOyUKOUWHXfmfm9oDxxybijiIikheZdlyfPLAJmAFcAzw90hBmVgv8gHAKcgc+CrwM3AEkgAC4xN07RvpdI7F5YwezzLBJtXHGEBHJm0wvN60Gnohe+9d/Sdhh/XdZyPEt4Dfu/gbgJGANcC2w0t0XACuj97HqeKmJ9mmJuGOIiOTNkQ6m6wPa3H3/SAOY2UTCzu8PA7h7F9BlZhcQPk8bYAXwIOGZS2w61wbsPToRZwQRkbwqhMF0c4E24EdmdhLh3FBXAdPcfXO0zxZgWqoPm9lSYCnA7NmzcxgTetcF9MxO5PQ7REQKSUaXm8zsejO7IkX7FWb2jyPMUA6cDHzH3RcRTh54yKUld3cOjslgwLbl7t7o7o319fUjjDK4sg0BY+YmcvodIiKFJNM+ictI3UH9JPC3I8zQDDS7+2PR+7sIi0armc0AiF63jvB7Rmz8poBx8xNxxxARyZtMi8RUwktCA7WT5jJQptx9C7DRzI6LmhYDLxJ2jC+J2pYA947ke7JhcmtArcZIiEgJybTjegPwDmDdgPZ3Ep4JjNSngNvNbGz0HR8hLGB3mtnlQBNwSRa+54h1dsOMbQHjj9NzJESkdGRaJL4H3BQdxH8XtS0Gvk4Wnnft7s8AjSk2LR7pz86Wzc07mI4zZvKkuKOIiORNpnc3fcPMpgDLCOdtAugCvuXu/5yrcIWkY01AxdQE483ijiIikjcZT8vh7teZ2deAE6KmNe6+JzexCs/eVwP2aIyEiJSYTKflmA6Uu3sz4Wjr/vYGoNvdW3OUr2D0rAvo0hgJESkxmd7ddBvwnhTt5wC3Zi9O4RrTFDAmkYg7hohIXmVaJBqBh1K0P0zqDueiU90SUHWM7mwSkdKSaZEoBypTtFelaS86ta1NTDwuEXcMEZG8yrRIPAZ8PEX7J0jqoyhWXb0wfVtA3QmJuKOIiORVpnc3fQH4nZmdyMFxEmcRTp9RMGMZcqW1eQeT+3oonzI57igiInmV6TOuHwVOI3z4z0XRsg44FajOVbhC0f5SE+1TE6AxEiJSYoYzTuLPwAfh9VtfPwLcA8whfPhQ0drzSkCFxkiISAnKtE8CMyszs4vM7D5gPXAh8F1gfo6yFYyedQEHZiXijiEikndDnklEs7P+HeGU4HuBnxCOj7jM3V/MbbzCMKYpwObp9lcRKT2DnkmY2cPAo8Ak4BJ3n+fuXyTNA4CKVVVLE1XHJOKOISKSd0OdSZwG/Aew3N1fyEOeglS7JaBaYyREpAQN1SfxF4SF5A9m9rSZfSaax6lk9DlM3RYwRWMkRKQEDVok3P1pd/8EMAP4N+B8YGP0uXPNrOgfrtC2aSdje7uomlYXdxQRkbzLdJzEfne/1d3PBI4H/gX4DLDFzP4nlwHj1ramibb6hMZIiEhJyvgW2H7uvtbdrwVmET5StCvrqQrInlcCds1IxB1DRCQWGQ+mG8jde4F7o6Voda8L6GrQ7a8iUpqGfSZRcpqa8DmJuFOIiMRCRWIIVc0BlRojISIl6ogvN2WTmQXAbqAX6HH3RjObDNwBJAgnFrzE3TvynW3iloCxGiMhIiWqkM4kznT3he7e/6S7a4GV7r4AWBm9zyt3mLo1YMrxiXx/tYhIQSikIjHQBcCKaH0F4YSCedWxdReVPfupmTkl318tIlIQCqVIOPBbM3vSzJZGbdPcfXO0vgWYluqDZrbUzFab2eq2trashtq2pomtGiMhIiWsIPokgLe7e4uZTQXuN7OXkje6u5tZykkF3X05sBygsbExqxMP7nqlicrpuv1VREpXQZxJuHtL9LqV8EFGpwCtZjYDIHrdmu9cB14L2N+QyPfXiogUjNiLhJmNN7MJ/evAXwHPA78ElkS7LSGGQXsWBPRpjISIlLBCuNw0DbjHwuv+5cBP3P03ZvYEcKeZXQ40EU4BkleVzQF2+in5/loRkYIRe5Fw93XASSna24HF+U900ITNAWOOTcQZQUQkVrEXiUJWvzUAPUdCREqYikQau9v3UNXVSdWs+rijiIjEJvaO60LVtqaJrVPmYGM0RkJESpeKRBq7Xg7YMT0RdwwRkVjpclMa+9cG2MxE3DFERGKlM4l0goBejZEQkRKnIpHG2I0BFXqOhIiUOF1uSmPC5oCeBYm4Y4iIxEpFIo0pWwO6j9fkfiJS2lQkUti/cy/jDuxh4tyUs5OLiJQM9Umk0PZiE611cxijMRIiUuJUJFLY+XJAh8ZIiIjoclMq+9YG9BydiDuGiEjsdCaRgmuMhIgIoCKR0tgNAeXzEnHHEBGJnS43pTB+cxNj5uv2VxERFYkU6rYGdB6fiDuGiORBd3c3zc3N7N+/P+4oOVdVVUVDQwMVFRUZf0ZFYoCePZ1U79tFzTyNkRApBc3NzUyYMIFEIkH0GOWi5O60t7fT3NzM3LlzM/6c+iQG2LamidbJsxlbob8akVKwf/9+6urqirpAAJgZdXV1wz5j0pFwgB0vBWyflog7hojkUbEXiH5H8ucsmCJhZmVm9rSZ/Tp6P9fMHjOztWZ2h5mNzUeOfWsDOvUcCRERoICKBHAVsCbp/Y3ATe4+H+gALs9HiN71Ab2zdGeTiORHe3s7CxcuZOHChUyfPp2ZM2e+/r6rq2vQz65evZorr7wyp/kKouPazBqAc4Hrgc9aeE50FvA30S4rgK8C38l1lvKNTZSdc36uv0ZEBIC6ujqeeeYZAL761a9SU1PD5z73ude39/T0UF6e+lDd2NhIY2NjTvMVypnEN4Grgb7ofR2ww917ovfNwMx8BKlpCRin50iISIw+/OEPc8UVV/DWt76Vq6++mscff5zTTjuNRYsWcfrpp/Pyyy8D8OCDD3LeeecBYYH56Ec/yhlnnMG8efNYtmxZVrLEfiZhZucBW939STM74wg+vxRYCjB79uwR55ncGjDmDYkR/xwRGZ3mfCv7P7PpquF/prm5mUceeYSysjJ27drFww8/THl5OQ888ACf//zn+cUvfnHYZ1566SVWrVrF7t27Oe644/j4xz8+rDERqcReJIC3Aeeb2XuBKuAo4FtArZmVR2cTDUBLqg+7+3JgOUBjY6OPJEhf5z7Gd+6gav70kfwYERnFjuSAngsXX3wxZWVlAOzcuZMlS5bw6quvYmZ0d3en/My5555LZWUllZWVTJ06ldbWVhoaGkaUI/bLTe5+nbs3uHsCuBT4nbt/EFgFvD/abQlwb66zdLzUxJbJs6mujP2vRURK3Pjx419f/9KXvsSZZ57J888/z69+9au0Yx0qKytfXy8rK6OnpyflfsNRyEfDawg7sdcS9lH8MNdf2KExEiJSgHbu3MnMmWG37C233JLX7y6oIuHuD7r7edH6Onc/xd3nu/vF7n4g19/f+WrA3hm6/VVECsvVV1/Nddddx6JFi7JydjAc5j6iy/gFpbGx0VevXn3En3/qI9exs7yGM7//hSymEpFCtmbNGo4//vi4Y+RNqj+vmT3p7invpS2oM4m4lTcFjJmbiDuGiEjBKIS7mwpG9aaA/fMTcccQESkYKhJJJm0J6NUYCRGR16lIRLxzHzV7OyifPyPuKCIiBUN9EpE9azewZdIsjhqnvxIRkX46Ikba1wRsnzqHEplWXkQkI7rcFNm7tondMxJxxxCREtPe3s7ixYsB2LJlC2VlZdTX1wPw+OOPM3bs4I/SefDBBxk7diynn356TvKpSER6XgvompWIO4aIlJihpgofyoMPPkhNTU3OioQuN0XGbAgwjZEQkQLw5JNP8q53vYu3vOUtnHPOOWzevBmAZcuWccIJJ3DiiSdy6aWXEgQB3/3ud7nppptYuHAhDz/8cNaz6EwiUt0SUHVMIu4YIhK3XHRMDmNmC3fnU5/6FPfeey/19fXccccdfOELX+Dmm2/mhhtuYP369VRWVrJjxw5qa2u54oorhn32MRwqEpHaLQH7NEZCRGKequjAgQM8//zznH322QD09vYyY0Z4a/6JJ57IBz/4QS688EIuvPDCvORRkQDYv58Je9rp1RgJEYmZu/PGN76RP/3pT4dtu++++3jooYf41a9+xfXXX89zzz2X8zzqkwD2rdvIltoG6ieUxR1FREpcZWUlbW1trxeJ7u5uXnjhBfr6+ti4cSNnnnkmN954Izt37mTPnj1MmDCB3bt35yyPigSwpaeS/zrn0xojISKxGzNmDHfddRfXXHMNJ510EgsXLuSRRx6ht7eXD33oQ7z5zW9m0aJFXHnlldTW1vK+972Pe+65J2cd15oqHGjbC49vgnMX5CCUiBQ0TRWuqcKHVD9eBUJEJBUVCRERSUtFQkRKXjFddh/Mkfw5VSREpKRVVVXR3t5e9IXC3Wlvb6eqqmpYn9M4CREpaQ0NDTQ3N9PW1hZ3lJyrqqqioaFhWJ+JvUiYWRXwEFBJmOcud/+Kmc0FfgbUAU8Cl7l7V3xJRaQYVVRUMHfu3LhjFKxCuNx0ADjL3U8CFgLvNrNTgRuBm9x9PtABXB5fRBGR0hR7kfDQnuhtRbQ4cBZwV9S+Argw/+lEREpb7EUCwMzKzOwZYCtwP/AasMPde6JdmoGZaT671MxWm9nqUrimKCKST7H3SQC4ey+w0MxqgXuANwzjs8uB5QBm1mZmTTkJeeSmANviDjEMoymvsubOaMo7mrJCYeadk25DQRSJfu6+w8xWAacBtWZWHp1NNAAtGXy+PtcZh8vMVqcb7l6IRlNeZc2d0ZR3NGWF0Zc39stNZlYfnUFgZuOAs4E1wCrg/dFuS4B7YwkoIlLCCuFMYgawwszKCIvWne7+azN7EfiZmX0NeBr4YZwhRURKUexFwt2fBRalaF8HnJL/RFm3PO4AwzSa8ipr7oymvKMpK4yyvEU1VbiIiGRX7H0SIiJSuFQkREQkLRWJHDGzWWa2ysxeNLMXzOyquDMNJRrU+LSZ/TruLEMxs1ozu8vMXjKzNWZ2WtyZ0jGzz0T/Bp43s59G85UVDDO72cy2mtnzSW2Tzex+M3s1ep0UZ8Z+abL+S/Tv4Fkzu6f/bslCkCpv0rZ/MDM3sylxZMuUikTu9AD/4O4nAKcCnzCzE2LONJSrCG8/Hg2+BfzG3d8AnESB5jazmcCVQKO7vwkoAy6NN9VhbgHePaDtWmCluy8AVkbvC8EtHJ71fuBN7n4i8ApwXb5DDeIWDs+Lmc0C/grYkO9Aw6UikSPuvtndn4rWdxMexFJOLVIIzKwBOBf4QdxZhmJmE4F3Et0W7e5d7r4j1lCDKwfGmVk5UA1sijnPIdz9IWD7gOYLCOdMgwKaOy1VVnf/bdIUPo8SDr4tCGn+bgFuAq4mnKeuoKlI5IGZJQhv830s5iiD+SbhP9q+mHNkYi7QBvwoujz2AzMbH3eoVNy9BfhXwt8YNwM73f238abKyDR33xytbwGmxRlmGD4K/E/cIQZjZhcALe7+57izZEJFIsfMrAb4BfBpd98Vd55UzOw8YKu7Pxl3lgyVAycD33H3RcBeCudyyCGia/kXEBa2o4HxZvaheFMNj4f3yRf8b7xm9gXCy7y3x50lHTOrBj4PfDnuLJlSkcghM6sgLBC3u/vdcecZxNuA880sIHzQ01lmdlu8kQbVDDS7e/+Z2V2ERaMQ/SWw3t3b3L0buBs4PeZMmWg1sxkA0evWmPMMysw+DJwHfNALe/DXMYS/MPw5+v+tAXjKzKbHmmoQKhI5YmZGeM18jbv/W9x5BuPu17l7g7snCDtVf+fuBfvbrrtvATaa2XFR02LgxRgjDWYDcKqZVUf/JhZToJ3sA/yScM40KPC508zs3YSXSs9398648wzG3Z9z96nunoj+f2sGTo7+TRckFYnceRtwGeFv5c9Ey3vjDlVEPgXcbmbPEj7R8J/ijZNadLZzF/AU8Bzh/3MFNS2Dmf0U+BNwnJk1m9nlwA3A2Wb2KuHZ0A1xZuyXJuu3gQnA/dH/Z9+NNWSSNHlHFU3LISIiaelMQkRE0lKREBGRtFQkREQkLRUJERFJS0VCRETSUpEQKWDRLKHvH3pPkdxQkRBJw8xuiQ7SA5dH484mki+xP+NapMA9QDgoMllXHEFE4qAzCZHBHXD3LQOW7fD6paBPmtl9ZtZpZk0DJ+8zszeb2QNmts/MtkdnJxMH7LPEzJ4zswNm1mpmKzjUZDP7uZntNbN1o22CQBndVCRERub/Es5ztJBwuo0fm1kjQDR9+f8D9gCnAP+LcHK/m/s/bGZ/D3wP+BFwIvBeYOBTzL5MOHfSScAdwM1mNjtnfyKRJJqWQyQNM7sF+BCwf8Cm/3D3a8zMgR+4+8eSPvMAsMXdP2RmHyN8lkRD9OApzOwMYBWwwN3XmlkzcJu7p5zqPPqOG9z9uuh9ObALWOruhTxTrxQJ9UmIDO4hYOmAth1J638asO1PhE/4AzgeeLa/QEQeIXyw0wlmtovwaYUrh8jwbP+Ku/eYWRswNaP0IiOkIiEyuE53X5uDnzucU/juFJ/VpWLJC/1DExmZU1O8739exBrgzWY2IWn76YT/361x961AC+EzJkQKks4kRAZXmeKpYb3u3hatX2RmTwAPAu8nPOC/Ndp2O2HH9o/N7MvAJMJO6ruTzk6uB24ys1bgPqAaWOzu38jVH0hkOFQkRAb3l8DmAW0thI+dBPgq8NfAMqAN+Ii7PwHg7p1mdg7wTeBxwg7we4Gr+n+Qu3/HzLqAfwBuBLYD/52jP4vIsOnuJpEjFN15dLG73xV3FpFcUZ+EiIikpSIhIiJp6XKTiIikpTMJERFJS0VCRETSUpEQEZG0VCRERCQtFQkREUnr/wN6s/DfssiruAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEKCAYAAAAW8vJGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAdV0lEQVR4nO3dfXRddZ3v8ff3PCQnT21pmjZtUijIk4BCMVJa1Nteh9EBRrxXnHG8OiBz5crCgZmlCwdcotc1c68zc+eJq0uHQQYZXTgj6tgRfELLBaXQBuS5olUKTWmTNC1t2uY53/vH3knPSZM0pznZe5+cz2uts84+e++T80lXmk/2/u0Hc3dERETGpOIOICIiyaJiEBGRAioGEREpoGIQEZECKgYRESmQiTvAbC1ZssRXrVoVdwwRkbLyxBNP7HX3psmWlX0xrFq1ivb29rhjiIiUFTN7eapl2pUkIiIFVAwiIlJAxSAiIgXKfoxBRKRYQ0NDdHR00N/fH3eUOZfL5WhtbSWbzc74PSoGEak4HR0dNDQ0sGrVKsws7jhzxt3p6emho6ODU089dcbv064kEak4/f39NDY2zutSADAzGhsbi94yUjGISEWa76Uw5kS+z4othpGBUXY/eiTuGCIiiVPBxeBsvPQlRgZH444iIhWmp6eHCy64gAsuuIDm5mZaWlrGXw8ODk773vb2dm688cY5zVexg89VC9IsPKOa7p/307ymNu44IlJBGhsbeeqppwD4zGc+Q319PR//+MfHlw8PD5PJTP7rua2tjba2tjnNV7FbDADL19WyR7uTRCQBrrnmGj7ykY+wZs0abr75ZrZs2cLatWtZvXo169at48UXXwTgoYce4oorrgCCUrn22mtZv349p512GrfffntJslTsFgNA87paXvrOQfjTuJOIiASH0T766KOk02kOHjzII488QiaT4cEHH+TWW2/lm9/85jHv+cUvfsGmTZvo7e3lrLPO4vrrry/qnIXJVHwxbP7EHty9Yo5QEJFjfcGeK/nXvMHPK/o9733ve0mn0wAcOHCAq6++ml/96leYGUNDQ5O+5/LLL6e6uprq6mqWLl1KZ2cnra2ts8pe0cWw4NQsoyPQ+8oQC06pijuOiMTkRH6Jz4W6urrx6U996lNs2LCBb3/72+zYsYP169dP+p7q6urx6XQ6zfDw8KxzVPQYg5lpnEFEEunAgQO0tLQAcPfdd0f62RVdDBDsTtqzWcUgIsly8803c8stt7B69eqSbAUUw9w90g8stba2Np/NjXp2/+wwj9y0m99rP72EqUQkybZt28brX//6uGNEZrLv18yecPdJj3ut+C2GpjfVsH/bAEOHdaKbiAioGMjkUjS+MUfXVu1OEhEBFQMQjDPoukkilaXcd6PP1Il8nyoGxs6A7os7hohEJJfL0dPTM+/LYex+DLlcrqj3VfR5DGOa19ay6bpX8VHHUjrRTWS+a21tpaOjg+7u7rijzLmxO7gVQ8UA1K3IUrUgxWu/HOSks6uP/wYRKWvZbLaoO5pVGu1KCmmcQUQkoGII6QxoEZGAiiHUrGIQEQFUDOMa35DjUMcQ/ftH4o4iIhIrFUMolTGWvrmGzse01SAilU3FkKd5rXYniYioGPLoyCQRkQiLwcxWmtkmM3vBzJ43s5smWcfM7HYz225mz5jZhVHlA2i+uIauLX2MDs/vsyFFRKYT5RbDMPAxdz8HuBi4wczOmbDO7wBnhI/rgC9GmI/c4gz1K7P0PNsf5ceKiCRKZMXg7rvd/clwuhfYBrRMWO1K4B4PPAYsMrPlUWUEHbYqIhLLGIOZrQJWA49PWNQC7Mx73cGx5YGZXWdm7WbWXuprnWicQUQqXeTFYGb1wDeBP3H3gyfyNdz9Dndvc/e2pqamkubTGdAiUukiLQYzyxKUwtfc/VuTrLILWJn3ujWcF5lFZ1YxeHCUw68ORfmxIiKJEeVRSQZ8Gdjm7n87xWobgT8Mj066GDjg7rujyghgKQvOZ9isrQYRqUxRXnb7EuCDwLNm9lQ471bgZAB3/xLwAHAZsB04AnwownzjmtfVsPvRI7zuPQvj+HgRkVhFVgzu/lNg2rvgeHA7pRuiSTS15etqeezWzrhjiIjEQmc+T2Lpm2vZ+3Q/w/2jcUcREYmcimES2boUJ51dTfeTOtFNRCqPimEKOtFNRCqVimEKKgYRqVQqhimMnQEdjIeLiFQOFcMUGk7OkkrDwZd0opuIVBYVwxTMTLuTRKQiqRimoWIQkUqkYpiGrrQqIpVIxTCNptU5DmwfZLB3JO4oIiKRUTFMI12Voml1js4tfXFHERGJjIrhODTOICKVRsVwHCoGEak0KobjaF5bS+djR/BRnegmIpVBxXActUszVDdm2LdtIO4oIiKRUDHMgO4DLSKVRMUwAxpnEJFKomKYARWDiFQSFcMMLD63miN7hunbOxx3FBGROadimIFU2li2poY9m7XVICLzn4phhrQ7SUQqhYphhprX1WqLQUQqgophhpatqaX7iX5GhnSim4jMbyqGGapemGbBqVl6nu6PO4qIyJxSMRRB92cQkUqgYihC81oNQIvI/KdiKIKOTBKRSqBiKMLC06sY7huld+dg3FFEROaMiqEIZhYetqo7uonI/KViKJKutCoi852KoUgaZxCR+U7FUKSlbTXse76foSOjcUcREZkTKoYiZWpSLD4vR3e7xhlEZH5SMZyA5bpukojMY5EVg5ndZWZdZvbcFMvXm9kBM3sqfNwWVbZi6QxoEZnPotxiuBt453HWecTdLwgfn40g0wkZOwPaXRfUE5H5J7JicPeHgX1Rfd5cqm/NkqlNcWC7TnQTkfknaWMMa83saTP7npmdG3eY6ei6SSIyXyWpGJ4ETnH384H/C/z7VCua2XVm1m5m7d3d3VHlK6BxBhGZrxJTDO5+0N0PhdMPAFkzWzLFune4e5u7tzU1NUWac4zOgBaR+SoxxWBmzWZm4fRFBNl64k01tcbzcxx8aYiB10bijiIiUlKZqD7IzO4F1gNLzKwD+DSQBXD3LwFXAdeb2TDQB7zPE3zYTzprLG3L0fn4EU5+R0PccURESiayYnD3PzjO8s8Dn48oTkmMjTOoGERkPknMrqRypHEGEZmPVAyzsOziWrq29DE6ktg9XiIiRVMxzELNkgy1yzPse34g7igiIiWjYpgl3Z9BROYbFcMsaZxBROYbFcMs6QxoEZlvVAyzdNLZ1Qz0DHOkczjuKCIiJaFimCVLGcsu1o17RGT+UDGUgAagRWQ+UTGUgIpBROYTFUMJLLuohu6f9zEyMBp3FBGRWVMxlEBVQ5pFZ1bT/fP+uKOIiMyaiqFEmtdpAFpE5odZF4OZZUsRpNxpnEFE5ouiisHMbjSz9+S9/jLQZ2YvmtlZJU9XRsbOgE7wLSRERGak2C2GG4FuADN7G/B7wPuBp4C/KWmyMtOwKouPQu8rQ3FHERGZlWJv1NMCvBRO/y7wDXf/NzN7FnikpMnKjJmN705acEpV3HFERE5YsVsMB4Gl4fSlwI/D6SEgV6pQ5UrjDCIyHxRbDD8E/snM7gROB74Xzj+Xo1sSFUtXWhWR+aDYYrgB+BnQBFzl7vvC+RcC95YyWDlqujDH/l8MMHhoJO4oIiInrKgxBnc/CPzxJPM/XbJEZSxdnWLJ+Tm6tvbRuqE+7jgiIiek2MNVz8k/LNXMLjWzr5rZLWaWLn288qNxBhEpd8XuSroLWA1gZiuB7wCLCXYx/Xlpo5Wn5ZfUsfunKgYRKV/FFsPZwJPh9FXA4+5+GfBB4A9KGaxcrXhbsMUwMqQT3USkPBVbDGlgMJx+O/BAOP1rYFmpQpWzXGOGhlVVdD/ZF3cUEZETUmwxPAdcb2ZvJSiG74fzW4C9pQxWzlrW1/HqQ4fjjiEickKKLYZPAB8GHgLudfdnw/nvAraUMFdZa1lfxy4Vg4iUqWIPV33YzJqABe6+P2/RPwIacQ2t+E+1PHh1ByNDTjprcccRESlK0ZfddvcRgiuqnmdm55pZzt13uHvXHOQrS7nFGRacVkX3ExpnEJHyU+x5DBkz+2tgP/A08Cyw38z+SvdlKKTdSSJSrordYvgr4APAR4AzgTOA6wkOV/3fpY1W3jQALSLlqtjLbr8fuNbdH8ib92sz6wbuBD5esmRlbsXbannwDzXOICLlp9gthoUE5yxM9Gtg0azTzCO5xRkWvq6K7naNM4hIeSm2GJ4muIvbRDeFyySPxhlEpBwVWww3A1eH93j+Svh4kWDcQbuRJlihYhCRMlRUMbj7wwSDzvcB9eHjG8A7mHxLYpyZ3WVmXWb23BTLzcxuN7PtZvaMmV1YTLYkWvG2OvZs1nWTRKS8FDv4jLu/Cnwyf56ZnQ+85zhvvRv4PHDPFMt/h+AopzOANcAXw+eylTspzcLTq+ja2sfydbVxxxERmZGiT3A7UeHWxr5pVrkSuMcDjwGLzGx5NOnmTjDOcCjuGCIiMxZZMcxAC7Az73VHOO8YZnadmbWbWXt3d3ck4U6UzmcQkXKTpGKYMXe/w93b3L2tqakp7jjTWv7WOvY81sfI4GjcUUREZmRGYwxmtvE4qywoQZZdwMq8163hvLKWOynNojOq6Grv1ziDiJSFmQ4+98xg+UuzzLIR+KiZfZ1g0PmAu++e5ddMhBXr69i16ZCKQUTKwoyKwd0/NNsPMrN7gfXAEjPrAD4NZMOv/yWCu8FdBmwnuIT3rD8zKVrW1/HM7T20ffL464qIxK3ow1VPlLtPe09od3fghojiRGrFW+v40fs7GBkcJV1VlsM6IlJB9FsqAtWL0iw6MzifQUQk6VQMEWnZUMeuTTpsVUSST8UQEV1QT0TKhYohIsvfUkfn432MDOh8BhFJNhVDRKoXpVl0VhWdGmcQkYRTMUSoZUO9xhlEJPFUDBHSOIOIlAMVQ4SWv6WWri0aZxCRZFMxRKh6YZqTzq6mc4vGGUQkuVQMEWvZoN1JIpJsKoaIBRfUUzGISHKpGCK24i21dG7pY7hf4wwikkwqhohVLUiz+JxqujTOICIJpWKIgQ5bFZEkUzHEQBfUE5EkUzHEYPkltXRu1TiDiCSTiiEGVQvSLD63ms7HNc4gIsmjYoiJxhlEJKlUDDFp2VDHqyoGEUkgFUNMNM4gIkmlYohJVUOaxvOq6XzsSNxRREQKqBhipHEGEUkiFUOMdEE9EUkiFUOMmi+ppau9n+E+jTOISHKoGGJUVZ+m8Q3V7NE4g4gkiIohZhpnEJGkUTHErGW9zmcQkWRRMcSs+ZJaup7QOIOIJIeKIWZV9WmWvLGaPZs1ziAiyaBiSIAVGmcQkQRRMSSABqBFJElUDAmw/JI6up/sZ+iIxhlEJH4qhgTI1qVYcn5O4wwikggqhoTQYasikhSRFoOZvdPMXjSz7Wb2Z5Msv8bMus3sqfDx36PMFyeNM4hIUmSi+iAzSwNfAC4FOoCtZrbR3V+YsOq/uvtHo8qVFM3raun+eTDOkK3VhpyIxCfK30AXAdvd/TfuPgh8Hbgyws9PtGxdiqYLcux5VOMMIhKvKIuhBdiZ97ojnDfRe8zsGTO7z8xWTvaFzOw6M2s3s/bu7u65yBoLnc8gIkmQtH0W/wGscvc3Aj8CvjLZSu5+h7u3uXtbU1NTpAHnkgagRSQJoiyGXUD+FkBrOG+cu/e4+0D48k7gTRFlS4TmdbV0P9XP0GGdzyAi8YmyGLYCZ5jZqWZWBbwP2Ji/gpktz3v5LmBbhPlil61N0bRa5zOISLwiKwZ3HwY+CvyA4Bf+v7n782b2WTN7V7jajWb2vJk9DdwIXBNVvqRoWV/Hrk3anSQi8YnscFUAd38AeGDCvNvypm8BbokyU9KsWF/Hltu64o4hIhUsaYPPFa95bS17n9Y4g4jER8WQMNnaFE0X6nwGEYmPiiGBWtbX0bHpUNwxRKRCqRgSSOcziEicVAwJ1Ly2lr3PDDB4aCTuKCJSgVQMCZSpSbH0TRpnEJF4qBgSStdNEpG4qBgSSie6iUhcVAwJ1XxxLT3PapxBRKKnYkio8XGGn2mcQUSipWJIsJYNGmcQkeipGBJM4wwiEgcVQ4Itu7iWnucGGOzVOIOIREfFkGCZXIqlbTl2a5xBRCKkYki4lg31ujyGiERKxZBwLTrRTUQipmJIuGVrajTOICKRUjEkXCaXYtmba9j9U40ziEg0VAxlQOcziEiUVAxlQOMMIhIlFUMZWHpRDftfGGDwoMYZRGTuqRjKQCaXYumba3jm9h76e4bjjiMi85yKoUys+fNldLX38S+n/ZJvveU3PPG5bnqe68fd444mIvNMJu4AMjPL19Wy/N9PYbh/lF0PHebl+3u5/3dfxkdh1RUNrLqigZYNdWRy6noRmR0r978429ravL29Pe4YsXB39r0wwMvf7WXH/b3sfaqflvV1rLqigVMub6C+JRt3RBFJKDN7wt3bJlumLYYyZmY0npuj8dwcF36iif59w7zy/UPs+G4vm2/ppOGULKsub+CUKxpY9uYaLGVxRxaRMqBimEdyizOc+f5FnPn+RYwOO3s2H2HHd3v5ybW76N87zCmXBbucVl5aT9WCdNxxRSShtCupQhx8aZAd9/fy8nd72f2zIyxbUxPscrqsgYWnV2lrQqTCTLcrScVQgQYPjdDxYDCA/cr3eznSNULdigz1K7PUr8zScHL2mOnqk9KYqTxE5guNMUiBqvo0p717Aae9ewEAw/2jHOoY4tDOIQ69EjzvfbqfHf/RG8zbOcTosNNwchX1K8MCObkqKI6VWerD8sjW6ogokflAxSBkcikWnV7NotOrp1xn4MDIeEkc2jlE7ytDvPr/Do9PH+4YIlOXKtziaM1S35qhrjWYrmtReYiUAxWDzEj1wjTVC9M0npebdLm709c9Mr7V0btziMO7huh5rp/DHUMc6hji8K7hoDxas9S1ZMLiyIbFcfS1BsZF4qVikJIwM2qXZqhdmmHpm2omXcfd6d87Euy26hgKC2OY3Q8fHp93aOcQljbqW7LUtU4sjyy1zRmqT0oHj4UpDZqLzAEVg0TGzKhpylDTlKFp9dTlMXhgdEJ5DNG1tY+Xvn2QI53DDOwfYWD/CEOHRqlakDpaFNM8cuPTwfpVC9Ok0ioVkclEWgxm9k7gH4A0cKe7f27C8mrgHuBNQA/w++6+I8qMEi8zo3pRmupFU++2GjM67AweGKE/LIrgMXp0et8IB389yMD+wnUGXxthsHeUbP3RUsnWpUjnjEwueC6cTpEJ56VzKTI1k8ybbL2cka4OptM5UxFJ2YisGMwsDXwBuBToALaa2UZ3fyFvtT8C9rv76Wb2PuAvgd+PKqOUl1TGyDVmyDUW/2M8OuIMHjxaIsOHRxnuH2Wk3xnud0YmmR46PEp/z0g4P5hXOD0arh9O9zkjA0fnp9KMl8RYkRQUUPVUhVNYPulqI1VlpKuC51T26HR6bH62cJ2p5lkaHYYsx4hyi+EiYLu7/wbAzL4OXAnkF8OVwGfC6fuAz5uZebmfbCGJk0obuXAXUxTcndGhsaKYqlgKS2ZkwAuLpt8ZeG0oeD3kjA46I4N5zycwz0c4WhYZsLRhqcmep16WShukIJUGS4VlM/7M0XEgCx/AeBeZ5U1PWD4+bZO8b+K6dsz7sML1xwtwks8oXPdoURZ05hTThetM8d45ct4Ni2k8d/ot6xMRZTG0ADvzXncAa6Zax92HzewA0AjszV/JzK4DrgM4+eST5yqvSMmYjf01DzTEneYoHw0LKywMHwUfCebnP4+OOIwGz1OtE8z3Y58dcBj/82782QvmFSz3sVU8b3qqdf3o8vx1p/icydf1guUFnzfD+fkLjvlT1ikslhKpapibw7/LcvDZ3e8A7oDgzOeY44iULUuFu7CmPoVFKlCUZxvtAlbmvW4N5026jpllgIUEg9AiIhKRKIthK3CGmZ1qZlXA+4CNE9bZCFwdTl8F/ETjCyIi0YpsV1I4ZvBR4AcEh6ve5e7Pm9lngXZ33wh8GfgXM9sO7CMoDxERiVCkYwzu/gDwwIR5t+VN9wPvjTKTiIgU0hXNRESkgIpBREQKqBhERKSAikFERAqU/a09zawbeDnuHBMsYcLZ2glXTnnLKSuUV95yygrllTeJWU9x96bJFpR9MSSRmbVPdS/VJCqnvOWUFcorbzllhfLKW05ZQbuSRERkAhWDiIgUUDHMjTviDlCkcspbTlmhvPKWU1Yor7zllFVjDCIiUkhbDCIiUkDFICIiBVQMJWRmK81sk5m9YGbPm9lNcWc6HjNLm9nPzey7cWc5HjNbZGb3mdkvzGybma2NO9NUzOxPw5+B58zsXjMr/f0XZ8HM7jKzLjN7Lm/eYjP7kZn9Knw+Kc6MY6bI+tfhz8EzZvZtM1sUY8QCk+XNW/YxM3MzWxJHtplSMZTWMPAxdz8HuBi4wczOiTnT8dwEbIs7xAz9A/B9dz8bOJ+E5jazFuBGoM3dzyO4zHzSLiF/N/DOCfP+DPixu58B/Dh8nQR3c2zWHwHnufsbgV8Ct0Qdahp3c2xezGwl8NvAK1EHKpaKoYTcfbe7PxlO9xL84mqJN9XUzKwVuBy4M+4sx2NmC4G3EdyzA3cfdPfXYg01vQxQE96JsBZ4NeY8Bdz9YYJ7nuS7EvhKOP0V4N1RZprKZFnd/YfuPhy+fIzgjpCJMMW/LcDfATdTeKfoRFIxzBEzWwWsBh6POcp0/p7gB3U05hwzcSrQDfxzuOvrTjOrizvUZNx9F/B/CP4y3A0ccPcfxptqRpa5++5weg+wLM4wRbgW+F7cIaZjZlcCu9z96bizzISKYQ6YWT3wTeBP3P1g3HkmY2ZXAF3u/kTcWWYoA1wIfNHdVwOHSc6ujgLhvvkrCcpsBVBnZh+IN1VxwlvqJv4vWzP7JMEu3K/FnWUqZlYL3Arcdrx1k0LFUGJmliUoha+5+7fizjONS4B3mdkO4OvAfzazr8YbaVodQIe7j22B3UdQFEn0W8BL7t7t7kPAt4B1MWeaiU4zWw4QPnfFnGdaZnYNcAXw3xJ+b/jXEfyR8HT4/60VeNLMmmNNNQ0VQwmZmRHsA9/m7n8bd57puPst7t7q7qsIBkZ/4u6J/avW3fcAO83srHDW24EXYow0nVeAi82sNvyZeDsJHSifYCNwdTh9NfCdGLNMy8zeSbAb9F3ufiTuPNNx92fdfam7rwr/v3UAF4Y/04mkYiitS4APEvz1/VT4uCzuUPPIHwNfM7NngAuA/xVvnMmFWzX3AU8CzxL8P0vUJRHM7F5gM3CWmXWY2R8BnwMuNbNfEWz1fC7OjGOmyPp5oAH4Ufj/7EuxhswzRd6yoktiiIhIAW0xiIhIARWDiIgUUDGIiEgBFYOIiBRQMYiISAEVg0jChFffvCruHFK5VAwieczs7vAX88THY3FnE4lKJu4AIgn0IMGJivkG4wgiEgdtMYgca8Dd90x47IPx3TwfNbP7zeyImb088QJ5ZvYGM3vQzPrMbF+4FbJwwjpXm9mzZjZgZp1m9hUKLTazb5jZYTP7TbldhE/Km4pBpHj/k+C6QhcQXOriHjNrAwgvBf4D4BBwEfBfCC6gd9fYm83sfwD/CPwz8EbgMmDi3b5uI7hW0fnAvwJ3mdnJc/YdieTRJTFE8pjZ3cAHgP4Ji77g7p8wMwfudPcP573nQWCPu3/AzD5McC+G1vBmTZjZemATcIa7bzezDuCr7j7pZcPDz/icu98Svs4AB4Hr3D3JV8CVeUJjDCLHehi4bsK81/KmN09YtpngTngArweeGSuF0KMEN0M6x8wOEtzV78fHyfDM2IS7D5tZN7B0RulFZknFIHKsI+6+fQ6+bjGb50OTvFe7fiUS+kETKd7Fk7weu9/CNuANZtaQt3wdwf+1be7eBewiuEeDSCJpi0HkWNWT3F1rxN27w+n/amZbgYeAqwh+ya8Jl32NYHD6HjO7DTiJYKD5W3lbIX8B/J2ZdQL3A7XA2939b+bqGxIphopB5Fi/BeyeMG8XwS0ZAT4DvAe4HegGPuTuWwHc/YiZvQP4e2ALwSD2d4Cbxr6Qu3/RzAaBjwF/CewDHpij70WkaDoqSaQI4RFD73X3++LOIjJXNMYgIiIFVAwiIlJAu5JERKSAthhERKSAikFERAqoGEREpICKQURECqgYRESkwP8H1xU47OEl0G8AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "config = Config(name='WRN50_2', mode='Train')\n",
    "datamanager = DataManager(config)\n",
    "network = config.get_network()\n",
    "trainer = Trainer(config, network, datamanager)\n",
    "trainer.train()\n",
    "trainer.save_model()\n",
    "trainer.save_training_log()\n",
    "trainer.visualize()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 10917.316234,
   "end_time": "2022-07-05T03:58:23.876355",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-07-05T00:56:26.560121",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}