more cleanup and restructuring

.circleci/config.yml

@@ -114,10 +114,11 @@ jobs:
             export DEBIAN_FRONTEND=noninteractive
             export FORCED_ROOT=1
             ./software/common/common.sh install
+            ./software/common/common.sh generate
       - run:
           name: Run unit tests
           command: |
-            cd software/debug
+            cd software/tools
             poetry install --no-interaction
             poetry run pytest -v
 

manufacturing/internals/electronics/README.md

@@ -67,7 +67,7 @@ Make sure the PEM nuts `[A2]` are installed properly into the support frame `[A1
 The Raspberry Pi serves as the user interface computer for the ventilator. It should be installed onto the PCB **before** attaching the PCB to the support frame.
 
 * install a heat sinks `[A6]` onto the Raspberry Pi `[A5]`
-* set up the memory card `[A7]` with the appropriate operating system and software as described in the [software deployment instructions](../../../software/utils/rpi_config).
+* set up the memory card `[A7]` with the appropriate operating system and software as described in the [software deployment instructions](../../../software/bootstrap).
 * install memory card into the RaspberryPi
 * Install the 22mm spacers `[A10]` onto the PCB `[A3]` at the location where the Raspberry Pi should be mounted.
 * Install the Raspberry Pi `[A5]` onto the PCB `[A3]` using bolts and washers `[A11-12]`.

quality-assurance/testing/README.md

@@ -125,4 +125,4 @@ scenarios. As of writing, we have used the tool for automating the collection of
 We plan to expand its use to more automated tests and apply it as part of the shared hardware-in-the-loop testing
 infrastructure described above.
 
-More information about the debug interface is available [here](../../software/utils/debug).
+More information about the debug interface is available [here](../../software/tools).

software/README.md

@@ -7,15 +7,9 @@ These are utilities for bench testing with mechanical lung simulators only. It i
 
 ## Contents
 
+* [bootstrap](bootstrap) - initial deployment scripts for all software directly via the Raspberry pi integrated in a ventilator. Can also be used to configure a linux machine for software development or integration testing.
 * [design](design) - documents on software architecture and requirements
 * [common](common) - code common to both controller and GUI executables
 * [controller](controller) - code for pneumatic system controller (stm32)
-* [debug](debug) - controller debug client for examining and manipulating low level variables, calibration, and testing
+* [debug](debug) - controller debug client and other tools for calibration, configuration and testing
 * [gui](gui) - code for the ventilator graphical interface (Qt)
-* [utils](utils) - scripts for debugging controller and mocking communications interface
-
-## Quick start
-
-If a proper enclosed unit is built as described under [manufacturing](../manufacturing), it should be possible to deploy all GUI and controller software directly via the Raspberry pi that is integrated into the ventilator unit.
-
-Scripts for doing so are available under [utils/rpi_config](utils/rpi_config).

software/utils/rpi_config/README.md → software/bootstrap/README.md

@@ -36,7 +36,7 @@ Whether by `ssh` or attached keyboard, copy and paste this command into a termin
 
 <!-- \TODO change command to point to master before merging!!! -->
 ```shell
-bash <(wget -qO- https://github.com/RespiraWorks/Ventilator/raw/master/software/utils/rpi_config/bootstrap.sh)
+bash <(wget -qO- https://github.com/RespiraWorks/Ventilator/raw/master/software/bootstrap/bootstrap.sh)
 ```
 
 Follow the onscreen directions. When the initial installation is complete, the system will reboot.
@@ -51,7 +51,7 @@ This is a good time to:
 When the machine boots, you should see an icon that says `Ventilator update` on the desktop. Either double-click or use the touch screen to run this shortcut.
 This will build the graphical interface and controller software, and deploy the latter to the STM32.
 
-**NOTE:** You may also want to check that audio is set to be piped to `HDMI` rather than the `AV Jack` by right clicking on the volume icon in the task bar.
+**NOTE:** You may also want to check that audio is set to be piped to `HDMI` rather than the `AV Jack` by right-clicking on the volume icon in the task bar.
 
 ## What you have available
 

software/utils/rpi_config/bootstrap.sh → software/bootstrap/bootstrap.sh

@@ -24,11 +24,10 @@ if [ -n "$VERBOSE" ]; then
 fi
 
 EXIT_FAILURE=1
-EXIT_SUCCESS=0
 
 # Check if Linux
 PLATFORM="$(uname -s)"
-if [ $PLATFORM != "Linux" ]; then
+if [ "$PLATFORM" != "Linux" ]; then
   echo "Error: This script only supports 'Linux'. You have $PLATFORM."
   exit $EXIT_FAILURE
 fi
@@ -56,7 +55,7 @@ if [ -z "$VERBOSE" ]; then
   echo " THIS WILL MESS WITH YOUR SYSTEM CONFIGURATION."
   echo " THIS IS FOR RASPBERRY-PI ONLY!"
   echo " "
-  read -n1 -s -r -p $'Press any key to continue...\n' key
+  read -n1 -s -r -p $'Press any key to continue...\n' _
 fi
 
 ### Install git-lfs and update the system
@@ -73,7 +72,7 @@ sudo raspi-config nonint do_blanking    1  # disable screen blanking
 sudo raspi-config nonint do_boot_splash 1  # disable splash screen
 
 ### Clone repository without LFS and go in
-cd ${HOME}
+cd "${HOME}"
 GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/RespiraWorks/Ventilator.git ventilator
 cd ventilator
 
@@ -89,24 +88,24 @@ cd ventilator
 git lfs pull -I "software/**"
 
 ### Desktop shortcuts
-/bin/cp -rf software/utils/rpi_config/user/Desktop/* ${HOME}/Desktop
+/bin/cp -rf software/bootstrap/user/Desktop/* "${HOME}/Desktop"
 
 ### On-screen keyboard, in case its needed
-sudo cp -f software/utils/rpi_config/keyboard.xml /usr/share/matchbox-keyboard/
+sudo cp -f software/bootstrap/keyboard.xml /usr/share/matchbox-keyboard/
 
 ### Execute desktop shortcuts without bitching
-mkdir -p ${HOME}/.config/libfm && cp -f software/utils/rpi_config/user/.config/libfm.conf ${HOME}/.config/libfm
+mkdir -p "${HOME}/.config/libfm" && cp -f software/bootstrap/user/.config/libfm.conf "${HOME}/.config/libfm"
 
 # Install dependencies and do initial configuration for build toolchains
 ./software/gui/gui.sh install
-./software/controller/controller.sh install
+./software/common/common.sh install
 ./software/controller/controller.sh configure
 
 if [ -z "$VERBOSE" ]; then
   echo "Installation complete. Please check that this terminated with no errors."
   echo "Upon restart, please run the 'Update' app from your desktop to complete deployment."
   echo " "
-  read -n1 -s -r -p $'Press any key to restart the machine\n' key
+  read -n1 -s -r -p $'Press any key to restart the machine\n' _
 
   sudo shutdown -r now
 fi

software/utils/rpi_config/update.sh → software/bootstrap/update.sh

@@ -68,10 +68,10 @@ cd "$(dirname "$0")"/../..
 git pull
 
 ### Set RW background - must be done in Desktop mode, thus not in boostrap.sh
-pcmanfm --set-wallpaper ${HOME}/ventilator/software/utils/rpi_config/desktop_background.jpg
+pcmanfm --set-wallpaper ${HOME}/ventilator/software/bootstrap/desktop_background.jpg
 
 ### Update Desktop shortcuts
-/bin/cp -rf ${HOME}/ventilator/software/utils/rpi_config/user/Desktop/* ${HOME}/Desktop
+/bin/cp -rf ${HOME}/ventilator/software/bootstrap/user/Desktop/* ${HOME}/Desktop
 
 if zenity --question --no-wrap --title="PIO Update" \
           --text "<span size=\"large\">Update PlatformIO and libraries?</span>"

software/utils/rpi_config/user/Desktop/Github → software/bootstrap/user/Desktop/Github

@@ -2,5 +2,5 @@
 Type=Link
 Name=Github repo
 Comment=RespiraWorks ventilator repository on Github
-Icon=/home/admin/ventilator/software/utils/icons/rw_logo_reverse.png
+Icon=/home/admin/ventilator/software/bootstrap/icons/rw_logo_reverse.png
 URL=https://github.com/RespiraWorks/Ventilator

software/common/common.sh

@@ -70,7 +70,7 @@ update_platformio() {
 
 generate_network_protocols() {
   PROTOCOLS_DIR=generated_libs/protocols
-  PYTHON_LIB_PATH=../debug/protocols
+  PYTHON_LIB_PATH=../tools/protocols
   GUI_LIB_PATH=../gui/src/protocols
   NANOPB_PLUGIN=${HOME}/.local/bin/protoc-gen-nanopb
 
@@ -186,14 +186,6 @@ elif [ "$1" == "install" ]; then
   install_common_tooling
   exit_good
 
-###########
-# INSTALL #
-###########
-elif [ "$1" == "install" ]; then
-  ensure_not_root
-  install_linux
-  exit_good
-
 ##########
 # UPDATE #
 ##########

software/controller/README.md

@@ -4,7 +4,7 @@ Here resides the code for the ventilator controller.
 
 Please also see the [Software design pages](../design/controller_architecture.md) for more information on controller architecture.
 
-If you are not actively working on the controller code but simply need it for prototype testing, you will probably want to use the [automated deployment scripts](../utils/rpi_config).
+If you are not actively working on the controller code but simply need it for prototype testing, you will probably want to use the [automated deployment scripts](../bootstrap).
 
 ## Rationale and structure
 
@@ -29,7 +29,7 @@ The controller shares the [common code](../common) with the GUI. The part of the
   * building and uploading/deployment of controller firmware
   * running unit tests, static checks, integration tests
   * generating test coverage reports locally
-  * shortcut to [debug interface](../utils/debug)
+  * shortcut to [debug interface](../tools)
   * self-documented if you run it without parameters or with `--help`
 * [platfomio.ini](platformio.ini) - the equivalent of a "make file" which governs how platformio builds targets
 * [.ycm_extra_conf.py](.ycm_extra_conf.py) - configuration for [YouCompleteMe](https://github.com/ycm-core/YouCompleteMe) (for some IDEs)

software/controller/controller.sh

@@ -181,7 +181,7 @@ generate_coverage_reports() {
 }
 
 function run_debug() {
-  ../debug/debug.sh "$@"
+  ../tools/debug.sh "$@"
 }
 
 # prints info from device manifest, if SN is defined in environment

software/controller/lib/debug/README.md

@@ -7,7 +7,7 @@ production system. **#TODO** How?
 The Python CLI interface can be accessed via the common controller utilities script at
 [software/controller/controller.sh](../../controller.sh)
 
-There is also a [brief tutorial](../../../debug) on how to use it.
+There is also a [brief tutorial](../../../tools) on how to use it.
 
 ## Controller side architecture
 [![Debug Interface Architecture diagram](images/architecture.png)](https://docs.google.com/drawings/d/18oN96yqyU3Ky_Kb82zGym1iXULxy9IW3KtIfMk-IDdI/edit?usp=sharing)

software/gui/README.md

@@ -18,7 +18,7 @@ To deploy it as intended for the ventilator you will need:
 A more complete list of hardware components is on the
 [Electronics sub-assembly page](../../manufacturing/internals/electronics).
 
-If you are not actively working on the code for this application but simply need it for prototype testing, you will probably want to use the [automated deployment scripts](../utils/rpi_config).
+If you are not actively working on the code for this application but simply need it for prototype testing, you will probably want to use the [automated deployment scripts](../bootstrap).
 
 ## Building the GUI
 

software/debug/.gitignore → software/tools/.gitignore

@@ -1,2 +1,2 @@
 # generated network protocol code
-debug/protocols/*
+protocols/*

software/debug/README.md → software/tools/README.md

@@ -1,16 +1,20 @@
 # Debug serial interface
 
-The controller debug interface should be run using [debug.sh](debug.sh) wrapper or from the common controller utilities script at [software/controller/controller.sh](../../controller/controller.sh)
+The controller debug interface should be run using [debug.sh](debug.sh) wrapper or from the common controller utilities script at [software/controller/controller.sh](../controller/controller.sh)
 
 This Python script can be used to interface with the controller via the virtual serial port that is created when you plug a USB cable into the controller's debug port. The command line interface allows you to examine and directly manipulate a number of low level variables, as well as run calibration routines and automated performance tests.
 
-To use this debug tool you will need to have Python 3.x installed on your development computer, as well as a number of python packages. Necessary dependencies will be installed by the [controller.sh](../../controller/controller.sh) script.
+To use this debug tool you will need to have Python 3.x installed on your development computer, as well as a number of python packages. Necessary dependencies will be installed by the [controller.sh](../controller/controller.sh) script.
 
 The debug tool was developed and tested using Linux, but it should be possible to use it with other operating systems. The command line interface will allow you to discover available commands using `help`. Some commands provide an "autocomplete" feature you can access with `TAB`. To get help on a specific command, enter help _command name_.  For example, to get a description of the `get` command:
 ```
 [sn:v03e2] help get
 ```
 
+## Regenerating python proto bindings
+
+Use the [common.sh](../common/common.sh) with the `generate` command.
+
 ## Connecting
 
 To use the debug tool you will first need to connect your development computer to the controller's debug port using a USB cable. This should create a virtual serial port.
@@ -120,18 +124,18 @@ To retrieve save data and view it, you may type:
 test read 2021-08-15-23-23-23_john_test.json --plot
 ```
 
-Test scenarios are defined in `json` files in the [test_scenarios](test_scenarios) directory. All files in that directory should be loaded when the script starts. You may also see additional commands provided by this interface with `help test`.
+Test scenarios are defined in `json` files in the [scenarios](scenarios) directory. All files in that directory should be loaded when the script starts. You may also see additional commands provided by this interface with `help test`.
 
 ### run
 This command takes the name of a python script and executes it in the same environment that the debug program runs in.  This allows the script to use functions defined in the debug tool to do things like get or set variables. The debug tool will search for scripts in the [scripts](scripts) subdirectory. These scripts may take additional parameters.
 
 ## Unit Tests
 ### Local Setup
-The unit tests for the debug interface were created using pytest. Necessary dependencies will be installed by the [controller.sh](../../controller/controller.sh) script.
+The unit tests for the debug interface were created using pytest. Necessary dependencies will be installed by the [controller.sh](../controller/controller.sh) script.
 
 ### Running Unit Tests Locally
-To run the unit tests, run the following command in Bash (not in the debug interface) within this working directory (`Ventilator/software/utils/debug`):
+To run the unit tests, run the following command in Bash (not in the debug interface) within this working directory (`Ventilator/software/tools`):
 ```bash
-python3 -m pytest --verbose unit_test.py
+poetry run pytest --verbose
 ```
 Omit the verbose flag if you only need to see failed tests.

software/debug/debug/debug_cli.py → software/tools/debug/debug_cli.py

@@ -45,7 +45,7 @@
 import debug.debug_funcs as debug_funcs
 
 MANIFEST_URL = "https://docs.google.com/spreadsheets/d/e/2PACX-1vRduOfterWmAy_xrc356rRhjz4QDLgOScgG1VPx2-KNeH8zYEe29SCw_DKOJG-5hqSO6BXmG1BumUul/pub?gid=0&single=true&output=tsv"
-LOCAL_DATA_PATH = "../../../local_data"
+LOCAL_DATA_PATH = "../../local_data"
 
 
 class ArgparseShowHelpError(Exception):
@@ -88,10 +88,10 @@ class CmdLine(cmd.Cmd):
     test_data_dir: Path
 
     def __init__(self, connect_to):
-        super(CmdLine, self).__init__()
+        super().__init__()
         script_path = Path(__file__).parent.resolve()
         self.scripts_directory = "scripts"
-        self.test_scenarios_dir = (script_path / "test_scenarios").resolve()
+        self.test_scenarios_dir = (script_path.parent / "scenarios").resolve()
         self.local_data_dir = (script_path / LOCAL_DATA_PATH).resolve(strict=False)
         self.test_data_dir = (self.local_data_dir / "test_data").resolve(strict=False)
         self.device_finder = DeviceScanner(self.local_data_dir / "device_manifest.tsv")

software/tools/miscellaneous/README.md

@@ -0,0 +1,4 @@
+# Miscellaneous stand-alone tools
+
+* [decoder.py](decoder.py) - decoder of serial packets sent from controller to GUI.
+* [mock_cycle_controller.py](mock_cycle_controller.py) - primitive controller emulator to test communications protocols

software/utils/decoder.py → software/tools/miscellaneous/decoder.py

@@ -1,6 +1,5 @@
 import serial  # pip install pySerial
-import struct
-from debug.lib.protocols import network_protocol_pb2
+from protocols import network_protocol_pb2
 import time
 
 p = serial.Serial("/dev/ttyACM0", 115200)

software/utils/mock_cycle_controller.py → software/tools/miscellaneous/mock_cycle_controller.py

@@ -12,8 +12,7 @@
 # and specify the other one for ventillator GUI
 
 import serial
-import struct
-from debug.lib.protocols import network_protocol_pb2
+from protocols import network_protocol_pb2
 import time
 import argparse
 import math

software/tools/poetry.toml

@@ -0,0 +1,2 @@
+[virtualenvs]
+in-project = true

software/debug/pyproject.toml → software/tools/pyproject.toml

@@ -5,7 +5,9 @@ description = "RespiraWorks ventilator debug interface"
 authors = ["Martin Shetty <1972005+martukas@users.noreply.github.com>"]
 readme = "README.md"
 packages = [
-    {include = "debug"}
+    {include = "debug"},
+    {include = "miscellaneous"},
+    {include = "protocols"}
 ]
 
 [tool.poetry.dependencies]