integrations/r/Neptune-R.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "# Using Neptune in R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Before you start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Install dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "install.packages('neptune', dependencies = TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "library(neptune)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Initialize Neptune"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "init_neptune(project_name = 'shared/r-integration',\n",
    "             api_token = 'ANONYMOUS',\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Neptune gives you an option of logging data under a public folder as an anonymous user. This is great when you are just trying out the application and don't have a Neptune account yet.  \n",
    "\n",
    "---\n",
    "\n",
    "If you already have a [Neptune account](https://neptune.ai/register), you can create your own experiment and start logging to it using your personal API token. Pass your `username` to the `project_qualified_name` argument of the `neptune.init()` method: `project_qualified_name='YOUR_USERNAME/YOUR_PROJECT_NAME`. If you don't have a project yet, keep `/sandbox` at the end. The `sandbox` project is automatically created for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Quickstart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Step 1: Create an Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "This creates an experiment in Neptune.\n",
    "\n",
    "Once you have a live experiment you can log things to it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "create_experiment(name='minimal example')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Click on the link above to open this experiment in Neptune.\n",
    "\n",
    "For now it is mostly empty but keep the tab with experiment open to see what happens next. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Step 2: Log metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Log your performance metrics during or after training with the ``log_metric`` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "log_metric('accuracy', 0.92)\n",
    "\n",
    "for (i in 0:100){\n",
    "  log_metric('random_training_metric', i * 0.6)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Step 3: Explore results in the Neptune UI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Go to the experiment link and explore charts that were created from the metric values you logged. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Step 4: Stop logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "When you are finished logging you should stop your current Neptune experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "stop_experiment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Example: Full model training and versioning on a Sonar dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Let's do something more real-life:\n",
    "\n",
    "* train a model on a Sonar dataset\n",
    "* keep track of model training artifacts and training metrics "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Install packages and import the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "install.packages(c('digest', 'mlbench', 'randomForest'), dependencies = TRUE)\n",
    "\n",
    "library(digest)\n",
    "library(mlbench)\n",
    "library(randomForest)\n",
    "\n",
    "# load dataset\n",
    "data(Sonar)\n",
    "dataset <- Sonar\n",
    "x <- dataset[,1:60]   # predictors\n",
    "y <- dataset[,61]     # labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Log hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "params = list(ntree=625,\n",
    "              mtry=13,\n",
    "              maxnodes=50\n",
    "              )\n",
    "\n",
    "create_experiment(name='training on Sonar',\n",
    "                  params = params\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Organize experiments with tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "append_tag(c('random-forest','sonar'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Log data versions and other properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "set_property(property = 'data-version', value = digest(dataset))\n",
    "\n",
    "SEED=1234\n",
    "set.seed(SEED)\n",
    "set_property(property = 'seed', value = SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Train model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "model <- randomForest(x = x, y = y,\n",
    "  ntree=params$ntree, mtry = params$mtry, maxnodes = params$maxnodes,\n",
    "  importance = TRUE\n",
    "  )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Log evaluation metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "log_metric('mean OOB error', mean(model$err.rate[,1]))\n",
    "log_metric('error class M', model$confusion[1,3])\n",
    "log_metric('error class R', model$confusion[2,3])\n",
    "\n",
    "for (err in (model$err.rate[,1])) {\n",
    "  log_metric('OOB errors', err)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Log model weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "save(model, file=\"model.Rdata\")\n",
    "log_artifact('model.Rdata')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Log performance charts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "for (t in c(1,2)){\n",
    "  jpeg('importance_plot.jpeg')\n",
    "  varImpPlot(model,type=t)\n",
    "  dev.off()\n",
    "  log_image('feature_importance', 'importance_plot.jpeg')\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Stop experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "stop_experiment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Explore results in the Neptune UI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Go to the experiment link and explore charts that were created from the metric values you logged. \n",
    "\n",
    "You can also check out [this experiment dashboard](https://ui.neptune.ai/o/shared/org/r-integration/experiments?viewId=fa3b57a5-77fb-4edb-83fc-505014d3649d):\n",
    "\n",
    "![image](https://neptune.ai/wp-content/uploads/r-integration-tour.gif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Untitled1.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}