Explore the test data and brainstorm RTDIP component ideas

[luccalb]

Explore the test data provided by shell and brainstorm ideas for RTDIP components that ensure better data quality or identify trends/anomalies

[Timm638]

Some Brainstorming done with scitkit-learn as inspiration:

• Dimensionality Reduction (Reduce redundant data, e. g. which sources correlate strongly with each other?)
• Normalization of Data (By Z-Mean, Min-Max-Scaling, ...)
• Other Preprocessing Methods: Map scalar data into bins, One-hot encoding
• Trend Identification: Linear Regression, ARIMA

Other notes:

• When we implement these functions, in which format should be work with the data? Convert everything into a pandas Dataframe and then back to the original format?

[chris-1187]

RTDIP component ideas:

Persistent Agent with datastore (Probably pushed back as Shell does not see a need for a DB right now):

• The thought was to instantiate an InfluxDB within the pipeline creation and store monitoring and other data there. InfluxDB is a minimal timeseries DB with a python API and Grafana support (for opt. visualisations).
• Covered through Issue Store monitoring outputs in a standardized format #26.

Missing value imputation with imputeFD or MICE through Apache SystemDS (SystemML) integration:

• Apache SystemDS is a ML system for the end-to-end data science lifecycle, including data cleansing
• It runs on top of Apache Spark and can be integrated through it's python bindings
• Optimized for big data and single node operations
• Prerequisite: Flagged missing values -> defined Pattern

Opt. expansion: General SystemDS integration and ability to run any ML algorithm through it's DML (Data Manipulation Language) script language