Testing automation ?

[mstormi]

Hi Wouter, hi guys,

we've just created (well, polished) our Continuous Integration pipeline to continously test openHABian changes/PRs. We're actually using Docker builds inside Travis CI based on the Balena Docker images for various Raspi models and x86.
Here's an example: https://travis-ci.com/github/mstormi/openhabian/builds/162007330
Is that something you would benefit from or might want to use as well ?

best,
Markus

[wborn]

More automated testing would be nice. 😉 If the automated testing is generic enough we might be able to use it for any distribution method.

Then we can:

1. Create some script to download and install a VM/Docker with OH for that particular distro
2. Launch the generic test suite, e.g. to test that REST API, LSP, UIs for that distro work
3. Launch specific tests for that particular distro method

If all distro artifacts are published to a staging repo we can test those with the suite and manually. Then if nothing critical is found we publish all staged artifacts.

Regarding Balena images:
Our long term goal is to make the openHAB Docker image an official Docker image see also #83. One of the requirements for that is that your image is based on other official images. So that rules out any custom images like Balena or the multiarch images we are currently based on. When #293 is merged we will be based on official images.

[mstormi]

Well feel free to setup Travis to watch your Github repo (if you don't do that already) and have a peek at my .travis.yml Unlike the various Dockerfiles in openHABian, it's not specific to Balena. I wouldn't think people to use Docker on top of Raspis for sake of memory but you can of course use any other Docker image as well as the basis for your own tests. I used the Balena images as our primary target is to validate openHABian (incl. OS) will work on Raspis (and x86), and Balena is great there because they have images tailored to each and every HW variant, 32/64 bit, and Linux distro. Essentially see openhab/openhabian#684.
Artifacts you can transfer back to Github Pages https://docs.travis-ci.com/user/deployment/pages

Speaking of a generic test suite, does it exist ? is there anything we could take from there and add to openHABian build tests ? Today I just test for the dashboard to display no REST or other tests.

[mstormi]

Do you know of a good method to observe memory usage inside Docker ?

Ultimately I want to find out if we can run 64 bit Java and/or 64 bit OS and/or extensions like ZRAM on a 1GB Raspi. But most usual tools output figures sometimes 10 times as large as usual when running inside Docker.
EDIT: here's an example doing "free -m" and "ps" on the java process at the end of openHABian install:
https://travis-ci.com/github/mstormi/openhabian/jobs/325182744#L1751
672M vs 4.7G ...

PS: Travis now allows for native ARM builds. Alpha but it works for me. Use arch: arm64. (aarch64 only)

[wborn]

I'm not aware of any existing generic openHAB test suites. For easily testing many REST calls we could also generate a client based on the Swagger spec using swagger-codegen. It can generate clients in various languages.

I run openHAB in Docker on my RPi 3B with Raspbian without any additional issues. In my experience it performs equally well compared to a non-Dockerized instance. I only use RPi's for testing with a secondary OH instance since it's too slow for my needs and because of the 1GB memory it doesn't have a lot of head room to run anything else besides OH. I've also tested running openHAB Docker images on the same RPi 3B with an arm64 build of Debian 10. But the performance is noticeably worse than running a 32-bit OS.

Perhaps on a RPi 4 it makes more sense to run a 64-bit OS. I did see some benchmarks that showed significant performance gains when running a 64-bit OS on the RPi 4.

Since the containers are using the same kernel as the host OS, you'll see exactly the same memory stats as you'd see on the host. Perhaps the memory details inside the container make more sense if you start the container with some memory constraints. But it may not be supported by the kernels used on Travis CI.

The memory details obtained from ps all may also not be very usable. The most reliable stats are probably those supplied by the docker stats command. But you'll only have those on the host. You can also get these from inside the container if you also install a docker client in the container and mount the docker socket /var/run/docker.sock into your container.