Dimensionality Reduction

[sanalmert]

User Story

1. As a RTDIP user
2. I want to reduce the dimensionality of incoming data
3. So that i can use it for further ML processing and so overal simplify my data

Additional context

• Reducing the redundant data
• How to allocate strong correlating sources

Acceptance Criteria

• A component exists that
  • will combine two (or more) columns into one if they are exact duplicates or highly correlate
  • it receives an optional threshold parameter [0-1]
  • will analyze if two columns correlation is higher than the threshold, if so combine them into one by a given method (sum, mean, delete one,..)

Definition of Done

• Test cases have been created and are running successfully
• Documentation for the new component was added
• Github Actions are running without errors