Store monitoring outputs in a standardized format

[luccalb]

User Story

1. As a RTDIP engineer
2. I want to save statistics about my data (like mean, missing data count, etc.) to a persistant storage
3. So that I can later generate a report from it, without manipulating the original data

Additional context

• Actually having a persistant DB is not favoured by Shell, as they want the RTDIP users to decide themselves how to use the data
• We need do develop a standardized data format (either a dataframe or a log file) that all of our monitoring components can use
• Two ways this could be implemented
  • either write to a shared dataframe directly from each component
  • or first write to a log file, then have a separate component that reads through the log and creates an output dataframe

Acceptance Criteria

• Data from monitoring components is written to a separate dataframe
• The data persists outside of the component runtime, for other components to read from, or to create reports from
• The format of the monitoring output dataframe is standardized and all monitoring components use it
• The format is document

Definition of Done

• Test cases have been created and are running successfully
• Documentation for the new component was added
• Github Actions are running without errors

[luccalb]

Feedback by shell: Logging results is enough, visualization or databases is out of scope for now

[mollle]

Apparently a dataframe with the "logs" instead (or in addition) of just logging. for exmple 2 Columns, with component name and then the log message in the other column. the dataframe would be handed from on step to the next one. after all the quality checks we want to hand over a dataframe so the user can then check the data quality and display the data in their way