You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It took @KaiqianZhang and I some time this morning to figure out a weird bug. Turns out Kaiqian has revised part of the input data, and not knowing that DSC does not track file status she did not remove cached output files. It apparently caused troubles.
Ordinary workflow systems do track input and output files explicitly. But in the context of DSC the files tracked are files of data flow, an abstracted concept users do not have to worry about. In other words we have not introduced the notion that users have to worry about any file status.
Tracking input script file status is easy and was done. We do not have mechanism tracking arbitrary input file status. Proposed solutions:
Add something like path("/path/to/file") to explicitly mark what we want to track. But it is one additional parameter and more importantly a new concept we'll have to explain to users.
For any string type of input I check if it is a file (if it exists on disk) and track its status if it does. A bit more work for the DSC parser but should not be too bad.
I'll go for solution 2. Any objections?
The text was updated successfully, but these errors were encountered:
It took @KaiqianZhang and I some time this morning to figure out a weird bug. Turns out Kaiqian has revised part of the input data, and not knowing that DSC does not track file status she did not remove cached output files. It apparently caused troubles.
Ordinary workflow systems do track input and output files explicitly. But in the context of DSC the files tracked are files of data flow, an abstracted concept users do not have to worry about. In other words we have not introduced the notion that users have to worry about any file status.
Tracking input script file status is easy and was done. We do not have mechanism tracking arbitrary input file status. Proposed solutions:
path("/path/to/file")
to explicitly mark what we want to track. But it is one additional parameter and more importantly a new concept we'll have to explain to users.I'll go for solution 2. Any objections?
The text was updated successfully, but these errors were encountered: