Best practices for backup data logging

[sgbaird]

• write backup data to SD card slot if present #95

Appending to a CSV file seems like a good option, but there are two provisos:

• accessing the same file might cause issues related to data loss (opening in append mode seems less risky, maybe as long as it's not multithreaded it's ok)
• number/order of columns gets updated in updated device code, making it difficult to read the file (use of JSON might be more flexible)

Google search: best practices for data logging

• https://www.chaossearch.io/blog/log-management-best-practices
• https://dev.splunk.com/enterprise/docs/developapps/addsupport/logging/loggingbestpractices/

  • Use clear key-value pairs
    Create events that humans can read
    Use timestamps for every event
    Use unique identifiers (IDs)
    Log in text format
    Use developer-friendly formats (NOTE: JSON is mentioned here)

Google search: microcontroller data logging sd card (examples)

• same file, CSV format
  • https://itp.nyu.edu/physcomp/labs/labs-serial-communication/lab-data-logging-with-an-sd-card-reader-using-spi-communication/
  • https://github.com/tigoe/DataloggingExamples/blob/main/Columbia_Brown_Center_2016/BatteryAndAnalogSensorLogger/BatteryAndAnalogSensorLogger.ino
• same file, TXT format
  • https://www.engineersgarage.com/arduino-sd-card-micro-sd-card-reader-sensor-data-logging/

Another consideration related to #98 is when backup data needs to be accessed directly due to malfunctioning (e.g. network connection issue). Some options:

• quickly swap it out with a new SD card and copy the files from the original SD card to a computer
• Implement some kind of hardware bypass to access the SD card while the microcontroller is in use (i.e. a bypass to GPIO 10-15, but not sure if this will cause issues if two separate microcontrollers connect to the same SD card leads. See also this discussion https://forums.tomshardware.com/threads/sd-card-splitter.674174/)
• expose two SD card ports such that one always has a micro SD card inserted that can be logged to (no interruption, if writing to one SD card fails, then try writing to the other SD card)
• Log to two files, one that is a failsafe that records all data, and another file that checks to see if there are experiments that never got logged to a backup database

https://www.google.com/search?q=backup+data+logging+micropython

• https://stackoverflow.com/questions/68323888/how-to-implement-a-real-time-data-logging-backup-system
• https://stackoverflow.com/questions/43947206/automatically-delete-old-python-log-files

See also:

• Persistent and accessible storage of data #117
• How can data be securely transferred? #84