Chapter11_Part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ce2d6e30",
   "metadata": {},
   "source": [
    "# Chapter 11. Deep Learning in Chemistry"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d09f3a0",
   "metadata": {},
   "source": [
    "## 11.2. Multilayer Perceptron (MLP)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa334069",
   "metadata": {},
   "source": [
    "In this section, we will explore the structure of a MLP, and use MLP network to solve more complex regression and classification problems."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a096bd7c",
   "metadata": {},
   "source": [
    "### 11.2.1. Structure of MLP Network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92baff2b",
   "metadata": {},
   "source": [
    "The structure of a MLP network can be described as follow:\n",
    "\n",
    "![MLP structure](images/MLP_structure.png)\n",
    "\n",
    "**Key points to remember about MLP network**\n",
    "\n",
    "- MLP can solve comlex problems with non-linearity, which a single perceptron can not.\n",
    "- MLP contains 1 or multiple hidden layers and 1 output layer. Each layer has a defined number of perceptrons (neurons) in it.\n",
    "- Each neurons contains the weights (w) for each of its inputs, and a bias (b).\n",
    "- The output of a neuron is calculated using the following equation:\n",
    "\n",
    " $$\\text{output} = \\sum_{i}(w_i \\times x_i) + b$$\n",
    "\n",
    "- The output of a neuron should be passed through an activation function. Activation function has many purposes in neural network, such as mapping the values to a new desired range, and introducing non-linearity to the network.\n",
    "- Some commonly used activation function in ANN are:\n",
    "![Activation functions](images/activation_functions.png)\n",
    "- The input and output values of the network should be scaled to small range for gradient descent to work."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5972708",
   "metadata": {},
   "source": [
    "### 11.2.2. How MLP Network Works"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26814b2b",
   "metadata": {},
   "source": [
    "The workflow of an MLP network is similar to a single-perceptron network, which includes the following steps:\n",
    "- **Step 1.** Initialize parameters\n",
    "- **Step 2.** Forward pass\n",
    "- **Step 3.** Calculate the value of the loss function\n",
    "- **Step 4.** Backward pass\n",
    "- **Step 5.** Update parameters.\n",
    "- Return to **Step 2**, repeat after a number of iterations (epochs) or until the loss function is low enough.\n",
    "\n",
    "During the backward pass, the change chain rule is applied to calculate the gradients through activation functions. For example:\n",
    "![Backpropagation - Activation functions](images/backpropagation_activation_function.png)\n",
    "\n",
    "Because the gradient flows through activation function, **the first derivative of the activation function must exist** in order for backpropagation to work."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87cf1436",
   "metadata": {},
   "source": [
    "### 11.2.3. PyTorch Implementation of MLP"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf5d20fa",
   "metadata": {},
   "source": [
    "#### 11.2.3.1. MLP network with 1 input, 1 hidden layer containing 5 neurons, 1 output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "170f1606",
   "metadata": {},
   "source": [
    "In this section, we will build an MLP network to perform regression for non-linear function y=f(x)\n",
    "\n",
    "*Network structure:*\n",
    "![MLP network](images/MLP_1_5_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc5c3d1a",
   "metadata": {},
   "source": [
    "**a. Import modules**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "429c8af5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Import modules\n",
    "import math\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import matplotlib.pyplot as plt\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e75ea2ca",
   "metadata": {},
   "source": [
    "**b. Load data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebfccc2b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the data points\n",
    "x = np.linspace(0, 10, 11)\n",
    "y = x*x*x - 10*x*x + 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2d5fb8b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define input and output scalers\n",
    "input_scaler = MinMaxScaler(feature_range=(0, 1))\n",
    "output_scaler = MinMaxScaler(feature_range=(0, 1))\n",
    "\n",
    "# Scale the data to the range from 0 to 1\n",
    "x_scaled = input_scaler.fit_transform(x.reshape(-1, 1)).reshape(-1)\n",
    "y_scaled = output_scaler.fit_transform(y.reshape(-1, 1)).reshape(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "270224da",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Convert to tensor\n",
    "x_scaled = torch.tensor(x_scaled).float()\n",
    "y_scaled = torch.tensor(y_scaled).float()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbf300a3",
   "metadata": {},
   "source": [
    "**c. Create model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f34e3598",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the regression class\n",
    "class RegressionModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(RegressionModel, self).__init__()\n",
    "        self.hidden = nn.Linear(1, 5)  # 1 input and 5 hidden\n",
    "        self.output = nn.Linear(5, 1)  # 5 hidden and 1 output\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.hidden(x)\n",
    "        x = torch.sigmoid(x)\n",
    "        return self.output(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a93267b0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Instantiate the model\n",
    "model = RegressionModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "737fcf88",
   "metadata": {},
   "source": [
    "**d. Training**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59aaafbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the loss function and optimizer\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abf487e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the train function\n",
    "def train(x, y):\n",
    "    # Forward pass\n",
    "    outputs = model(x.unsqueeze(1))\n",
    "    loss = criterion(outputs, y.unsqueeze(1))\n",
    "    \n",
    "    # Backward and optimize\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    return loss.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a1340db",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create lists of losses for visualization\n",
    "losses = []\n",
    "\n",
    "# Train the model\n",
    "num_epochs = 10000\n",
    "progress_bar = tqdm(range(num_epochs))\n",
    "for epoch in progress_bar:\n",
    "    train_loss = train(x_scaled, y_scaled)\n",
    "    \n",
    "    # Add loss to lists for visualization\n",
    "    losses.append(train_loss)\n",
    "        \n",
    "    # Print progress\n",
    "    progress_bar.set_description(f'Epoch [{epoch+1}/{num_epochs}], Train loss: {train_loss:.4f}') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ecdd638",
   "metadata": {},
   "source": [
    "**e. Visualization**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6012dd11",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Axis limits\n",
    "x_min = math.floor(min(x))\n",
    "x_max = math.ceil(max(x))\n",
    "y_min = math.floor(min(y))\n",
    "y_max = math.ceil(max(y))\n",
    "\n",
    "# Create the plot\n",
    "x_values = np.linspace(x_min, x_max, num=101)\n",
    "x_values_scaled = input_scaler.transform(x_values.reshape(-1, 1)).reshape(-1)\n",
    "y_values_scaled = model(torch.tensor(x_values_scaled).float().unsqueeze(1))\n",
    "y_values_scaled = y_values_scaled.detach().numpy()\n",
    "y_values = output_scaler.inverse_transform(y_values_scaled.reshape(-1, 1)).reshape(-1)\n",
    "\n",
    "fig = plt.figure()\n",
    "plt.scatter(x, y, color='red')\n",
    "plt.plot(x_values, y_values)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19d2ea3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize MSE loss values over time\n",
    "plt.plot(losses)\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('MSE loss')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37055600",
   "metadata": {},
   "source": [
    "**f. Make prediction**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f89d4829",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input for prediction\n",
    "x_pred = 5\n",
    "\n",
    "# Scale input to range 0 to 1\n",
    "x_pred_scaled = input_scaler.transform([[x_pred]])\n",
    "\n",
    "# Convert to tensor\n",
    "x_pred_scaled = torch.tensor(x_pred_scaled).float()\n",
    "\n",
    "# Run model forward\n",
    "y_pred_scaled = model(x_pred_scaled)\n",
    "\n",
    "# Convert back to number\n",
    "y_pred_scaled = y_pred_scaled.item()\n",
    "\n",
    "# Scale output back to original range\n",
    "y_pred = output_scaler.inverse_transform([[y_pred_scaled]])[0][0]\n",
    "\n",
    "# Show result\n",
    "print(y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dd410c1",
   "metadata": {},
   "source": [
    "**g. Save and load model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3712059d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model\n",
    "model_name = 'model_3'\n",
    "file_name = f'./{model_name}_{num_epochs}.ckpt'\n",
    "torch.save(model.state_dict(), file_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51f233a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load model\n",
    "file_name = f'./{model_name}_10000.ckpt'\n",
    "loaded_model = RegressionModel()\n",
    "loaded_model.load_state_dict(torch.load(file_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c355a6d8",
   "metadata": {},
   "source": [
    "#### 11.2.3.2. MLP network with 3 inputs, 2 hidden layers containing 5 neurons each, 2 outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f852a0e3",
   "metadata": {},
   "source": [
    "In this section, we will build an MLP network to perform regression for non-linear function with multiple inputs and outputs $(y_1, y_2) = f(x_1, x_2, x_3)$\n",
    "\n",
    "*Network structure:*\n",
    "![MLP network](images/MLP_3_5_5_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9517164e",
   "metadata": {},
   "source": [
    "**a. Import modules**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "deb11c13",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import modules\n",
    "import math\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import matplotlib.pyplot as plt\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3de7c90a",
   "metadata": {},
   "source": [
    "**b. Load data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55a6cefc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the data points\n",
    "np.random.seed(42)\n",
    "x1 = np.random.rand(10)\n",
    "x2 = np.random.rand(10)\n",
    "x3 = np.random.rand(10)\n",
    "y1 = x1*x1*x2 - 5*x2*x3 + 5\n",
    "y2 = x3*x1 + 8*x2*x1 - 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "296dd874",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine inputs into 1 array\n",
    "x = np.column_stack((x1, x2, x3))\n",
    "\n",
    "# Combine outputs into 1 array\n",
    "y = np.column_stack((y1, y2))\n",
    "\n",
    "# Define input and output scalers\n",
    "input_scaler = MinMaxScaler(feature_range=(0, 1))\n",
    "output_scaler = MinMaxScaler(feature_range=(0, 1))\n",
    "\n",
    "# Scale the data to the range from 0 to 1\n",
    "x_scaled = input_scaler.fit_transform(x)\n",
    "y_scaled = output_scaler.fit_transform(y)\n",
    "\n",
    "# Convert to tensor\n",
    "x_scaled = torch.tensor(x_scaled).float()\n",
    "y_scaled = torch.tensor(y_scaled).float()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1467b6f6",
   "metadata": {},
   "source": [
    "**c. Create model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a48cea8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define regression class\n",
    "class RegressionModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(RegressionModel, self).__init__()\n",
    "        self.hidden1 = nn.Linear(3, 5)  # 3 input and 5 hidden\n",
    "        self.hidden2 = nn.Linear(5, 5)  # 5 hidden and 5 hidden\n",
    "        self.output = nn.Linear(5, 2)  # 5 hidden and 2 output\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.hidden1(x)\n",
    "        x = torch.sigmoid(x)\n",
    "        x = self.hidden2(x)\n",
    "        x = torch.sigmoid(x)\n",
    "        return self.output(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a89af10e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate the model\n",
    "model = RegressionModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7a5086a",
   "metadata": {},
   "source": [
    "**d. Training**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91b7d264",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the loss function and optimizer\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4d1267f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the train function\n",
    "def train(x, y):\n",
    "    # Forward pass\n",
    "    outputs = model(x)\n",
    "    loss = criterion(outputs, y)\n",
    "    \n",
    "    # Backward and optimize\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    return loss.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "013a5dcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create lists of losses for visualization\n",
    "losses = []\n",
    "\n",
    "# Train the model\n",
    "num_epochs = 5000\n",
    "progress_bar = tqdm(range(num_epochs))\n",
    "for epoch in progress_bar:\n",
    "    train_loss = train(x_scaled, y_scaled)\n",
    "    \n",
    "    # Add loss to lists for visualization\n",
    "    losses.append(train_loss)\n",
    "        \n",
    "    # Print progress\n",
    "    progress_bar.set_description(f'Epoch [{epoch+1}/{num_epochs}], Train loss: {train_loss:.4f}') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3d18638",
   "metadata": {},
   "source": [
    "**e. Visualization**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33a76692",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize MSE loss values over time\n",
    "plt.plot(losses)\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('MSE loss')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcb03fb9",
   "metadata": {},
   "source": [
    "**f. Make prediction**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9daeefaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input for prediction\n",
    "x_pred = [8, 3, 4]\n",
    "\n",
    "# Scale input to range 0 to 1\n",
    "x_pred_scaled = input_scaler.transform([x_pred])\n",
    "\n",
    "# Convert to tensor\n",
    "x_pred_scaled = torch.tensor(x_pred_scaled).float()\n",
    "\n",
    "# Run model forward\n",
    "y_pred_scaled = model(x_pred_scaled)\n",
    "\n",
    "# Convert back to number\n",
    "y_pred_scaled = y_pred_scaled.detach().numpy()\n",
    "\n",
    "# Scale output back to original range\n",
    "y_pred = output_scaler.inverse_transform(y_pred_scaled)[0]\n",
    "\n",
    "# Show result\n",
    "print(y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd0ccdf1",
   "metadata": {},
   "source": [
    "**g. Save and load model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7ea803f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model\n",
    "model_name = './model_4'\n",
    "file_name = f'./{model_name}_{num_epochs}.ckpt'\n",
    "torch.save(model.state_dict(), file_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f8dfad4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load model\n",
    "file_name = f'./{model_name}_5000.ckpt'\n",
    "loaded_model = RegressionModel()\n",
    "loaded_model.load_state_dict(torch.load(file_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "047b5ad9-7606-4fd6-92b9-564b0397a970",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  },
  "toc": {
   "base_numbering": "",
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Table of Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "244.333px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}