You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For now we either define "global" root for all checkpoints or have to specify separate root for each type of checkpoints. This is okay but sometimes we need to transfer data between locations.
Possible scenarios:
checkpoint some of the datasets locally
or publish reports to the different S3 bucket
For now to support such behavior I am overriding datasets in the catalog. But I believe we can make it much better by introducing locations and tag inputs/outputs with location tags. For example:
01_raw/base.csv will be check-pointed in the default root location.
while 01_raw/base.csv#reports will be saved to the reports location.
I am not sure whether #* suffix is the best API though. Looks like the possible interference with dataset transcoding convention should be considered.
The text was updated successfully, but these errors were encountered:
For now we either define "global"
root
for all checkpoints or have to specify separate root for each type of checkpoints. This is okay but sometimes we need to transfer data between locations.Possible scenarios:
For now to support such behavior I am overriding datasets in the catalog. But I believe we can make it much better by introducing locations and tag inputs/outputs with location tags. For example:
01_raw/base.csv
will be check-pointed in the default root location.01_raw/base.csv#reports
will be saved to thereports
location.I am not sure whether
#*
suffix is the best API though. Looks like the possible interference with dataset transcoding convention should be considered.The text was updated successfully, but these errors were encountered: