Guide_Continuous_Integration_and_Testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing and Continuous Integration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content\n",
    "\n",
    "1. [Testing](#Test)\n",
    "    1. [Basic Test Procedure](#TestBasic)\n",
    "    2. [Testing Types](#TestType)\n",
    "    3. [Unit Tests](#TestUnit)\n",
    "    4. [Integration Tests](#TestInteg)\n",
    "    5. [System Tests](#TestSystem)\n",
    "    6. [Error Messages](#TestMsg)\n",
    "    7. [Dealing with External Resources](#TestExtern)\n",
    "    8. [Test Configuration](#TestConfig)\n",
    "2. [Continuous Integration](#CI)\n",
    "    1. [Automated Tests](#CITest)\n",
    "    2. [Test Coverage](#CICover)\n",
    "    3. [Static Code Analysis](#CIStat)\n",
    "    4. [Jenkins Projects Overview](#CIProj)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  1. <a id=\"Test\"> Testing </a>\n",
    "Any programming code that is meant to be used more than once should have a test, i.e., an additional piece of programming code that is able to check whether the original code is doing what it's supposed to do.\n",
    "\n",
    "Writing tests is work. As a matter of facts, it can be a _lot_ of work, depending on the program often more than writing the original code.\\\n",
    "Luckily, it essentially follows always the same basic procedure and a there are a lot of tools and frameworks available to facilitate this work.\n",
    "\n",
    "In CLIMADA we use the Python in-built _test runner_ [unittest](https://docs.python.org/3/library/unittest.html) for execution of the tests and the [Jenkins](https://www.jenkins.io/) framework for _continuous integration_, i.e., automated test execution and code analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Why do we write test?\n",
    "- The code is most certainly __buggy__ if it's not properly tested.\n",
    "- Software without tests is __worthless__. It won't be trusted and therefore it won't be used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### When do we write test?\n",
    "- __Before implementation.__ A very good idea. It is called [Test Driven Development](#https://en.wikipedia.org/wiki/Test-driven_development).\n",
    "- __During implementation.__ Test routines can be used to run code even while it's not fully implemented. This is better than running it interactively, because the full context is set up by the test.\\\n",
    "  _By command line:_ \\\n",
    "  `python -m unittest climada.x.test_y.TestY.test_z`\\\n",
    "  _Interactively:_ \\\n",
    "  `climada.x.test_y.TestY().test_z()`\n",
    "- __Right after implementation.__ In case the coverage analysis shows that there are missing tests, see [Test Coverage](#CI.Cover).\n",
    "- __Later, when a bug was encountered.__ Whenever a bug gets fixed, also the tests need to be adapted or amended. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1.A. <a id=\"TestBasic\"> Basic Test Procedure </a>\n",
    "* __Test data setup__ \\\n",
    "  Creating suitable test data is crucial, but not always trivial. It should be extensive enough to cover all functional requirements and yet as small as possible in order to save resources, both in space and time.\n",
    "* __Code execution__ \\\n",
    "  The main goal of a test is to find bugs _before_ the user encounters them. Ultimately every single line of the program should be subject to test.\\\n",
    "  In order to achieve this, it is necessary to run the code with respect to the whole parameter space. In practice that means that even a simple method may require a lot of test code.\\\n",
    "  (Bear this in mind when designing methods or functions: <i style=\"color:darkred;\">the number of required tests increases dramatically with the number of function parameters!</i>)\n",
    "* __Result validation__ \\\n",
    "  After the code was executed the _actual_ result is compared to the _expected_ result. The expected result depends on test data, state and parametrization.\\\n",
    "  Therefore result validation can be very extensive. In most cases it won't be practical nor required to validate every single byte. Nevertheless attention should be paid to validate a range of results that is wide enough to discover as many thinkable discrepancies as possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1.B. <a id=\"TestTypes\"> Testing types </a>\n",
    "Despite the common basic procedure there are many different kinds of tests distinguished. (See [WikiPedia:Software testing](#https://en.wikipedia.org/wiki/Software_testing)). Very commonly a distinction is made based on levels: \n",
    "- __Unit Test__: tests only a small part of the code, a single function or method, essentially without interaction between modules\n",
    "- __Integration Test__: tests whether different methods and modules work well with each other\n",
    "- __System Test__: tests the whole software at once, using the exposed interface to execute a program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1.C. <a id=\"TestUnit\"> Unit Tests </a>\n",
    "Unit tests are meant to check the correctness of program units, i.e., single methods or functions, they are supposed to be fast, simple and easy to write.\\\n",
    "For each module in CL\n",
    "\n",
    "##### Developer guidelines:\n",
    "- __Each module in CLIMADA has a counter part containing unit tests.__ \\\n",
    "  _Naming suggestion:_ `climada.x.y` &rarr; `climada.x.test.test_y`\n",
    "  \n",
    "- __Write a test class for each class of the module, plus a test class for the module itself in case it contains (module) functions.__ \\\n",
    "  _Naming suggestion:_ `class X` &rarr; `class TestX(unittest.TestCase)`, module `climda.x.y` &rarr; `class TestY(unittest.TestCase)`\n",
    "  \n",
    "- __Ideally, each method or function should have at least one test method.__ \\\n",
    "  _Naming suggestion:_ `def xy()` &rarr; `def test_xy()`, `def test_xy_suffix1()`, `def test_xy_suffix2()`\\\n",
    "  _Functions that are created for the sole purpose of structuring the code do not necessarily have their own unit test._\n",
    "  \n",
    "- __Aim at having _very_ fast unit tests!__ \\\n",
    "  _There will be hundreds of unit tests and in general they are called _in corpore_ and expected to finish after a reaonable amount of time.\\\n",
    "  Less than 10 milisecond is good, 2 seconds is the maximum acceptable duration_.\n",
    "  \n",
    "- __A unit test shouldn't call more than one climada method or function.__ \\\n",
    "  _The motivation to combine more than one method in a test is usually creation of test data. Try to provide test data by other means. Define them on the spot (within the code of the test module) or create a file in a test data directory that can be read during the test. If this is too tedious, at least move the data acquisition part to the constructor of the test class._\n",
    "  \n",
    "- __Do not use external resources in unit tests.__ \\\n",
    "  _Methods depending on external resources can be skipped from unit tests. See [Dealing with External Resources](#TestExtern)._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1.D. <a id=\"TestInteg\"> Integration Tests </a>\n",
    "Integration tests are meant to check the correctness of interaction between units of a module or a package.\\\n",
    "As a general rule, more work is required to write integration tests than to write unit tests and they have longer runtime.\n",
    "\n",
    "##### Developer guidelines:\n",
    "- __Write integration tests for all intended use cases__.\n",
    "\n",
    "- __Do not expect external resources to be immutable__.\n",
    "  If calling on external resources is part of the workflow to be tested, take into account that they may change over time.\\\n",
    "  If the according API has means to indicate the precise version of the requested data, make use of it, otherwise, adapt your expectations and leave room for future changes.\\\n",
    "  _Example given_: your function is ultimately relying on the _current_ GDP retrieved from an online data provider, and you test it for Switzerland where it's in about 700 Bio CHF at the moment. Leave room for future development, try to be on a reasonably save side, tolerate a range between 70 Bio CHF and 7000 Bio CHF.\n",
    "\n",
    "- __Test location__:\n",
    "  Integration are written in modules `climada.test.test_xy` or in `climada.x.test.test_y`, like the unit tests.\\\n",
    "  For the latter it is required that they do not use external resources and that the tests do not have a runtime longer than 2 seconds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1.E. <a id=\"TestSystem\"> System Tests </a>\n",
    "Integration tests are meant to check whether the whole software package is working correctly.\n",
    "\n",
    "In CLIMADA, the system test that checks the core functionality of the package is executed by calling `make install_test` from the installation directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1.F. <a id=\"TestMsg\"> Error Messages </a>\n",
    "When a test fails, make sure the raised exception contains all information that might be helpful to identify the exact problem.\\\n",
    "If the error message is ever going to be read by someone else than you while still developing the test, you best assume it will be someone who is completely naive about CLIMADA.\n",
    "\n",
    "<i style='color:darkred'>Writing extensive failure messages will eventually save more time than it takes to write them.</i>\n",
    "\n",
    "Putting the failure information into logs is neither required nor sufficient: the automated tests are built around error messages, not logs.\\\n",
    "Anything written to `stdout` by a test method is useful mainly for the developer of the test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1.G. <a id=\"TestExtern\"> Dealing with External Resources </a>\n",
    "Methods depending on external resources (calls a url or database) are ideally atomic and doing nothing else than providing data. If this is the case they can be skipped in unit tests on safe grounds - provided they are tested at some point in higher level tests.\n",
    "\n",
    "In CLIMADA there are the utility functions `climada.util.files_handler.download_file` and `climada.util.files_handler.download_ftp`, which are assigned to exactly this task for the case of external data being available as files.\n",
    "\n",
    "Any other method that is calling such a data providing method can be made compliant to unit test rules by having an option to replace them by another method. Like this one can write a dummy method in the test module that provides data, e.g., from a file or hard coded, which be given as the optional argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import climada\n",
    "def x(download_file=climada.util.files_handler.download_file):\n",
    "    filepath = download_file('http://real_data.ch')\n",
    "    return Path(filepath).stat().st_size\n",
    "\n",
    "import unittest\n",
    "class TestX(unittest.TestCase):\n",
    "    def download_file_dummy(url):\n",
    "        return \"phony_data.ch\"\n",
    "    \n",
    "    def test_x(self):\n",
    "        self.assertEqual(44, x(download_file=self.download_file_dummy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Developer guideline:\n",
    "- When introducing a new external resource, add a test method in `test_data_api.py`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1.H. <a id=\"TestConfig\"> Test Configuration </a>\n",
    "Use the configuration file `climada.config` in the installation directory to define file pathes and external resources used during tests (see the [Constants and Configuration Guide](./Guide_Configuration.ipynb))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. <a id=\"CI\"> Continuous Integration </a>\n",
    "The CLIMADA Jenkins server used for continuous integration is at [(https://ied-wcr-jenkins.ethz.ch) ](https://ied-wcr-jenkins.ethz.ch)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  2.A. <a id=\"CITest\"> Automated Tests </a>\n",
    "On Jenkins tests are executed and analyzed automatically, in an unbiased environment. The results are stored and can be compared with previous test runs.\\\n",
    "Jenkins has a GUI for monitoring individual tests, full test runs and test result trands.\\\n",
    "Developers are requested to watch it. At first when they push commits to the code repository, but also later on, when other changes in data or sources may make it necessary to review and refactor code that once passed all tests.\n",
    "##### Developer guidelines:\n",
    "- All tests must pass before submitting a pull request.\n",
    "- Integration tests don't run on feature branches in Jenkins, therefore developers are requested to run them locally.\n",
    "- After a pull request was accepted and the changes are merged to the develop branch, integration tests may still fail there and have to be adressed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  2.B. <a id=\"CICover\"> Test Coverage </a>\n",
    "Jenkins also has an interface for exploring code coverage analysis result.\\\n",
    "This shows which part of the code has never been run in any test, by module, by function/method and even by single line of code.\n",
    "\n",
    "__Ulitmately every single line of code should be tested.__\n",
    "\n",
    "##### Developer guidelines:\n",
    "- Make sure the coverage of novel code is at 100% before submitting a pull request.\n",
    "\n",
    "Be aware that the having a code coverage alone does not grant that all required tests have been written!\\\n",
    "The following artificial exmple would have a 100% coverage and still obviously misses a test for `y(False)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ".."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "been here\n",
      "been there\n",
      "been everywhere\n",
      "been here\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "----------------------------------------------------------------------\n",
      "Ran 2 tests in 0.003s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "def x(b:bool):\n",
    "    if b:\n",
    "        print('been here')\n",
    "        return 4\n",
    "    else:\n",
    "        print('been there')\n",
    "        return 0\n",
    "\n",
    "def y(b:bool):\n",
    "    print('been everywhere')\n",
    "    return 1/x(b)\n",
    "\n",
    "\n",
    "import unittest\n",
    "class TestXY(unittest.TestCase):\n",
    "    def test_x(self):\n",
    "        self.assertEqual(x(True), 4)\n",
    "        self.assertEqual(x(False), 0)\n",
    "\n",
    "    def test_y(self):\n",
    "        self.assertEqual(y(True), 0.25)\n",
    "\n",
    "unittest.TextTestRunner().run(unittest.TestLoader().loadTestsFromTestCase(TestXY));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  2.C. <a id=\"CICover\"> Static Code Analysis </a>\n",
    "At last Jenkins provides an elaborate GUI for pylint findings which is especially useful when working in feature branches.\n",
    "\n",
    "_Observe it!_\n",
    "\n",
    "##### Developer guidelines:\n",
    "- _High Priority Warnings_ are as severe as test failures and must be addressed at once.\n",
    "- Do not introduce new _Medium Priority Warnings_.\n",
    "- Try to avoid introducing _Low Priority Warnings_, in any case their total number should not increase."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  2.D. <a id=\"CIProjects\"> Jenkins Projects Overview </a>\n",
    "* #### [climada_install_env](https://ied-wcr-jenkins.ethz.ch/job/climada_install_env/)\n",
    "  Branch: __develop__ \\\n",
    "  Runs every day at 1:30AM CET\n",
    "  - creates conda environment from scratch\n",
    "  - runs core functionality system test (`make install_test`)\n",
    "  \n",
    "* #### [climada_ci_night](https://ied-wcr-jenkins.ethz.ch/job/climada_ci_night/)\n",
    "  Branch: __develop__ \\\n",
    "  Runs when climada_install_env has finished successfully\n",
    "  - runs all test modules\n",
    "  - runs static code analysis\n",
    "\n",
    "* #### [climada_branches](https://ied-wcr-jenkins.ethz.ch/job/climada_branches/)\n",
    "  Branch: __any__ \\\n",
    "  Runs when a commit is pushed to the repository\n",
    "  - runs all test modules _outside of climada.test_\n",
    "  - runs static code analysis\n",
    "\n",
    "* #### [climada_data_api](https://ied-wcr-jenkins.ethz.ch/job/climada_data_api/)\n",
    "  Branch: __develop__ \\\n",
    "  Runs every day at 0:20AM CET\n",
    "  - tests availability of external data APIs\n",
    "  \n",
    "* #### [climada_data_api](https://ied-wcr-jenkins.ethz.ch/job/climada_data_api/)\n",
    "  Branch: __develop__ \\\n",
    "  No automated running\n",
    "  - tests executability of CLIMADA tutorial notebooks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.3 64-bit ('climada': conda)",
   "language": "python",
   "name": "python37364bitclimadaconda2bcabf5424a143ed8fd9177a65a9f185"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}