Refactor the Run upload process

[lampajr]

Refactor the process such that runs and related datasets get created/persisted synchronously whereas the datasets processing (label values calculation and so on) is deferred to the async process by making use if the dataset-even queue

Fixes Issue

Fixes #1976

Changes proposed

This change proposes to slightly update how the Run upload is managed, moving from a completely sync process to a hybrid one.

The process can be summarized as follow:

1. Persist the Run and any related dataset in the same transaction (if everything goes fine obv).
2. Once run and datasets are flushed, send dataset events to the dataset-event queue to process label values (and change point detection and so on) asynchronously.

TODOs/Open points:

• What to do with the datastore uploads, should we follow the same process even if more than 10 runs are passed?
• Change the API to always return 202 if the process succeeded with the runId that was created.
• Adapt tests
• [ ] Adapt the UI nothing to do here

Check List (Check all the applicable boxes)

• My code follows the code style of this project.
• My change requires changes to the documentation.
• I have updated the documentation accordingly.
• All new and existing tests passed.

[lampajr]

The behavior will change as follows:

• Uploading a new Run
  • If datastore has more than 10 runs, all of them will be processed async and the api will return 202 with empty array
  • If less than 10 runs, the process will persist the runs and their datasets and it will queue up the datasets recalculation such that they will be processed async . The API will return the list of run ids.

Given that the transform() method is getting called by other processes, if isRecalculation is set to true (which is not the case for new Run upload) the datasets recalculation is getting processed sync.

[lampajr]

Given that with the new implementation, uploading a run is not fully synchronous anymore, I had to change some tests in order to explicitly recalculate the labelValues.

Moreover I create a different in memory resource for AMQ such that we can skip processing events in the AMQ broker if not actually required, especially because they will throw errors as they are not running in the same transaction of the test

[johnaohara]

Moreover I create a different in memory resource for AMQ

wdym by this? what happens in the event of a crash? we have an external amq instance to persist any messages that have not been processed so we can recover in the event of a crash. why do we need an in-memory resource?

[lampajr]

Moreover I create a different in memory resource for AMQ

wdym by this? what happens in the event of a crash? we have an external amq instance to persist any messages that have not been processed so we can recover in the event of a crash. why do we need an in-memory resource?

Sorry, I should have mentioned that is just for testing purposes, see horreum-backend/src/test/java/io/hyperfoil/tools/horreum/test/AMQPInMemoryResource.java such that all tests that do not require or that they don't check async processing can use that. This will avoid to see some exceptions during the tests that are "expected" but that can cause confusion

[johnaohara]

Sorry, I should have mentioned that is just for testing purposes, see horreum-backend/src/test/java/io/hyperfoil/tools/horreum/test/AMQPInMemoryResource.java such that all tests that do not require or that they don't check async processing can use that. This will avoid to see some exceptions during the tests that are "expected" but that can cause confusion

ok, thanks for clarifying