Tracking file provenance #3712
Open
Tracking file provenance #3712
+1,110
−92
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…le_inputs_and_outputs
initial code to handle file related monitoring messages
…t the one that used them last
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces the ability to track file provenance to Parsl. The provenance information includes:
This feature uses the existing monitoring infrastructure to capture the requisite information and store it in a databse. The file provenance information can also be accessed via the parsl-visualizer interface. An additional keyword argument for the MonitoringHub has been added
capture_file_provenanceto enable/disable the provenance tracking. By default it isFalse, turning off the provenance tracking.While this type of file information can be tracked manually for smaller workflows, larger workflows require an automated solution, like this one.
Changed Behaviour
In general, users will not change Parsl's behavior, as the provenance tracking code is turned off by default. Enabling it will only gather additional information in the background. The only change users should outwardly see is the workflow slowdown typically seen when unsing the monitoring framework.
Fixes
This fixes #3711
Type of change