Skip to content

Latest commit

 

History

History
229 lines (160 loc) · 9.76 KB

TESTING.md

File metadata and controls

229 lines (160 loc) · 9.76 KB
Raw

Testing guide

LibreMesh has unit tests that help us add new features while keeping maintenance effort contained.

We encourage contributors to write tests when adding new functionality and also while fixing regressions.

LibreMesh unit testing is based in the powerful busted library which has a very good documentation.

Tests are run inside a x86_64 Docker image with some lua and openwrt libraries avaible.

We also have a development qemu virtual machine, this is a full libremesh image that can be used in development.

How to run the tests

Just execute ./run_tests: run_tests

This will build the testing Docker image automaticaly in the first run and then execute the tests and create the coverage report. Note: you must have Docker installed and running.

Use LUA_ENABLE_LOGGING=1 ./run_tests if you want to send the logging to stdout.

Testing directory structure

The lua code of package foo should be in the expanded files tree structure form: package/foo/files/usr/lib/lua/foo.lua

Test files live inside a tests directory with its names begining with test_:

package/foo/tests/test_foo.lua
package/foo/tests/test_utils.lua

Testing utilities, fake libraries and integration tests live inside a root tests/ directory:

tests/test_some_integration_tests.lua
tests/test_other_general_tests.lua
tests/fake/bazlib.lua
tests/tests/test_bazlib.lua

How to write tests

Here is a very simple test of a library foo:

local foo = require 'foo'

describe('foo library tests', function()
    it('very simple test of f_sum(a, b)', function()
        assert.is.equal(4, foo.f_sum(2, 2))
        assert.is.equal(2, foo.f_sum(2, 0))
        assert.is.equal(0, foo.f_sum(2, -2))
    end)
end)

Using lime.config or the uci library

If you need to test something that directly or indirectly uses the configuration uci library then you must do the following in order to have a clean and temporary uci environment for each test:

local foo = require 'foo'
local test_utils = require 'tests.utils'

local uci -- do not forget this line!

describe('foo lib tests', function()
    it('test directly using uci', function()
        uci:set('wireless', 'radio0', 'wifi-device')
        uci:set('wireless', 'radio0', 'type', 'mac80211')
        -- this updates a config file in /tmp/tmpdir.XYZ/config/wireless
        uci:commit('wireless')

        assert.is.equal('mac80211', foo.get_radio_type('radio0')
    end)

    before_each('', function()
        -- this creates a temporary and fresh uci envornment for each test. There is no initial configuration, you have to create the config somehow inside the test.
        uci = test_utils.setup_test_uci()
    end)

    after_each('', function()
        -- this cleans the temporary uci envornment
        test_utils.teardown_test_uci(uci)
    end)
end)

You also must use lime.config.get_uci_cursor() when you need a uci cursor, instead of using libuci:cursor(). This way the functions test_utils.setup_test_uci() and test_utils.teardown_test_uci(uci) can do its work providing a shared and clean uci config environment for each test. (Note: if something is not working as expected make sure all the uci cursors in all the code you use and its dependencies use the cursor provided by lime.config.get_uci_cursor())

Testing advices and hints

  • Libraries should provide a way to change *hardcoded things. For example using module variables to declare paths and then at the test code override this variable:
-- file foo.lua
foo.search_paths = {"/usr/lib/lua/lime/hwd/*.lua"}
  • Execution of commands with side effects, like os.execute('reboot'), should be put inside a library function like foo._reboot() and then using stubs or mocks in the tests. Even better is to separate the logic part of the code of the executional part so in the test you don't even have to mock this.
  • Put special atention on untrivial logic: regex, parsing, multiple ifs, nested conditions.
  • Testing trivialities is not helpful, but if you have trivial code you should have at least one test just to run through the code so you know you don't have syntax or require errors. This helps to future developers when then want to perform refactoring.
  • Use setup() / teardown() and before_each() / after_each() to refactor repeated code in the tests.
  • Look to existing tests to find inspiration.
  • Ask for help or advice in a pull request!

Coverage report

Coverage is measured using the luacov library each time the tests are run. The results statistics are merged at ./luacov.stats.out and a human friendly report is generated at luacov.report.out.

Under the hood: tools in detail

As one of the goals is that it must be easy for developers to write, modify and run the tests we created some simple tools to do this:

  • testing image -> Dockerfiles/Dockerfile.unittests
  • testing shell environment -> tools/dockertestshell
  • running the tests -> ./run_tests script

Testing shell environment

To provide an easy way to develop or test things within the docker image there is a tool that opens a bash shell inside the docker image that has some features that allows easy development:

  • /home/$USER is mounted inside the docker image so each change you do to the code from inside is maintained when you close the docker container
  • the same applies to /tmp
  • you have the same user outside and inside
  • network access is garanted
  • and some goodies like bashrc, some useful ENV variables, PS1 modification, etc.

To enter the shell environment run:

[lime-packages]$ ./tools/dockertestshell
(docker) [lime-packages]$

You can see that the prompt is changed adding (docker) in the left part so you can easily remember that you are inside the docker container.

This environment is also used by run_tests script.

run_tests script in detail

The idea behind this script is simple:

  • creates the testing docker image if it is not available
  • sets the search path of the tests for buster
  • sets the lua library paths, prepending the fake library paths and adding the paths to the libremesh packages with packages/lime-system/files/usr/lib/lua/?.lua. This doesn't work automaticaly for every package if the paths does not use the files/path/to/final/destination. So if you want to test some package without the files convention maybe it would be good to move the package to this convention. Also it does not work if the lua module we want to test does not finish with .lua, in this case the path must be explicitly added (how to do this here).
  • runs the tests using the dockertestshell

run_tests also passes the first argument as an argument to busted so you can do things like./run_tests --help to see the busted help, or ./run_tests '--tags=footag --verbose' so only the tests that have the tag #footag in the description of the test are run

Development with qemu virtual machine

This image is not a perfect firmware image, it does not have wifi for example but ethernet network LAN and WAN is supported. All the files inside the packages files/ can be copied into the rootfs, overwriting a precooked image that is a full LibreMesh x86_64 image. So don't expect that everything runs exactly as in a wireless router but most things will perform as expected:

  • initialization scripts: uci-defaults, init.d, etc
  • lime-config
  • ubus / rpcd
  • lime-app

ICMPv4 does NOT work between qemu nodes, so ping (v4) will not work as expected. Everything else (including ICMPv6 i.e. ping6) does work as expected, however.

How to start and stop the image

You will need a rootfs and ramfs LibreMesh files. To generate one you can use a libremesh buildroot and select x86_64 target and select the option to generate an initramfs.

Prebuilt development images can be downloaded from here:

Install the package qemu-system-x86_64 if you don't have already installed.

Build a mesh network

Up to 100 qemu nodes can be setup. Use the --node-id N. All the node's LAN interfaces are bridged together. You can use --enable-wan in only one of the nodes to share your internet connection to the network.

Start it

$ sudo ./tools/qemu_dev_start  path/to/openwrt-x86-64-generic-rootfs.tar.gz path/to/openwrt-x86-64-ramfs.bzImage

Stop it

$ ./tools/qemu_dev_stop

Run 12 nodes simultaneously, with arbitrary complex topology of clouds and links

### apt install ansible, if you don't have it already
$ cd tools/ansible/
$ sudo ansible-playbook qemu_cloud_start.yml
### use clusterssh to manage the nodes
$ sudo ansible-playbook qemu_cloud_stop.yml

Update with local libremesh code

If you want to update the qemu image with new LibreMesh files of a local repository you can use the option --libremesh-workdir path/to/workdir, for example:

$ sudo ./tools/qemu_dev_start  path/to/rootfs.tar.gz path/to/bzImage --libremesh-workdir .

Enable WAN and share internet to the virtual machine

Use the --enable-wan IFC, this will create a NAT and share your internet connection to the virtual machine using the specified interface IFC.

Lime-App

If you want to test a specific version of the Lime-App you can copy the build files into the lime-app package after each build:

[lime-app ]$ mkdir -p path/to/lime-packages/packages/lime-app/files/www/app/
[lime-app ]$ npm run build:dev_router  && cp -r build/* path/to/lime-packages/packages/lime-app/files/www/app/