How do I open a DVC-tracked directory from Python API ?

[AlexandreBrown]

Context

Hello,
I just setup a new private git repo with DVC.
I pushed a directory containing my dataset.
eg: /mydataset/ which contains subfolders with images.
The mydataset folder is tracked by DVC.
I also created a tag for the commit that adds the mydataset.dvc
Data was pushed to the remote successfully (S3 remote).
I made sure my notebook had SSH configured and it works in the terminal I can clone.

Problem

Now I want to test loading the data from the Python API.
I am getting an error IsADirectoryError which is accurate, I indeed have a directory.

Code

pip install dvc[s3]

import dvc.api    

with dvc.api.open(
        path='datasets/mydataset',
        repo='git@gitlab.com:private_repo/my-data-registry',
        rev='v1.0'
        ) as fd:
    print("It worked!") # This never gets printed, it crashes with an IsADirectoryError exception

Question

How would I go about loading a directory ?

Thanks

[pmrowla]

This is expected, api.open can only be used to open files. If you passed it something like datasets/mydataset/some_file_inside_dataset then it would open the file.

[AlexandreBrown]

@pmrowla
I would expct some method that offers me the possibility to download the dataset folder locally to a path I specify.
Perhaps something like

target_local_path = "myLocalFolder/"
api.get_folder(path, repo, ref, target_local_path)

[AlexandreBrown]

Since it is a common use case for Deep Learning and supported by the CLI already I'd expect to be able to have the same experience via the Python API.

Note that I could also simply do a subprocess and call the CLI from Python but then that'd defeat the purpose of having a Python API IMO.

This might be a good workaround until it's added tho.

What do you think?

[pmrowla]

You could use the internal Repo API to do this. It sounds like you are looking for behavior equivalent to dvc get, so you can use Repo.get (which is what is used under the hood in dvc get):

from dvc.repo import Repo

Repo.get("git@gitlab.com:private_repo/my-data-registry", "datasets/mydataset")

This would download the mydataset directory into your current working directory. You can use out=... to specify some alternate download path (the same way you use -o in dvc get).

For more details on how dvc get flags map to Repo.get() arguments, see:

dvc/dvc/commands/get.py

Lines 39 to 45 in d5b809d

	Repo.get(
	self.args.url,
	path=self.args.path,
	out=self.args.out,
	rev=self.args.rev,
	jobs=self.args.jobs,
	)

Repo.get() itself is defined here:

dvc/dvc/repo/get.py

Line 19 in d5b809d

def get(url, path, out=None, rev=None, jobs=None):

[pmrowla]

If you have a local clone of your repo, you can also use Repo.pull to get the same behavior as dvc pull, but based on your explanation it sounds like you are trying to download the dataset without a copy of the entire repo (so you should use dvc get/Repo.get)

[AlexandreBrown]

@pmrowla I tested your suggestion using Repo.get and it works flawlessly.
That's exactly what I was looking for, thanks a lot!

[karajan1001]

Can we provide a new API something like dvc.walk to iter over a remote repo?

[skshetry]

There's also a proposal PR for traversable pathlib-like API on #6850.

[efiop]

for the record: we plan on publishing dvcfs - fsspec implementation for dvc repos, that has methods like ls/walk/open/download/etc, which would likely be much more handy than trying to fit dir handling into existing api.open.

[AlexandreBrown]

Resolution

Closing this discussion since my issue can be resolved by using Repo.get as suggested by @pmrowla

That being said I think it would be very appreciated to have good doc on how to handle this use case since such use case should be fairly common for Deep Learning cases. Lacking to do so might mean losing a good portion of the community affected by this use case since their first impression will be that this use case is not supported. I don't think it's super obvious for new users that the Repo class exists (seems internal).

I think this will be solved by @skshetry 's PR as his suggestion with root.download(to="path") makes a lot of sense.

Thanks everyone for your help, the DVC community is solid!

[dberenbaum]

@AlexandreBrown Thanks for the feedback and support!

Can you elaborate more on how you are using this?

• Is the repo where you upload the data the same as the one where you want to load the data? For the data loading repo, are you using DVC for anything else (like tracking an experiment pipeline)?
• Is it possible to use dvc get from the command line before loading the data? Is downloading through Python a convenience to mimic other data loaders?
• Is the data static, or do you want to track changes to it?

[AlexandreBrown]

Hello @dberenbaum , sure I can give more context.

Is the repo where you upload the data the same as the one where you want to load the data?

• No, we created a data-registry repository that acts as a source of truth for the datasets and their versions via git tags.
• Our datasets are mostly images (image classification and segmentation) but we could also have more ML-oriented datasets in the future (right now we mostly do Deep Learning) or also 3D Cloud Points in the future as well.

For the data loading repo, are you using DVC for anything else (like tracking an experiment pipeline)?

• No we only use DVC to version and tack our datasets for now, we have the data-registry repo, we have our own python library that wraps dvc.rep and offers a way for its clients to specify a dataset name , version and output path (to prevent repeating the use of the base gitlab path (with potentially the auth if we use an access token) in all ML repositories and to abstract DVC implementation details). Then we have one repository per model. These model repositories have a dependency on our python library. Therefore they can easily get a dataset of a specific version. These model repositories define the training pipelines (we use Kubeflow Pipelines) and notebooks (exploration and analysis) so there are two ways that we could end up downloading the model (notebook & pipelines).

Is it possible to use dvc get from the command line before loading the data? Is downloading through Python a convenience to mimic other data loaders?

• It depends, when doing data analysis via notebooks then yes we can use any way (CLI, Python etc). But when we run Kubeflow Pipelines then it's a completely different story. For instance, we might have a Kubeflow pipeline with the following steps/components :
  • Download dataset (will simply call our python library to execute dvc.rep.Repo.get and download the dataset given as input).
  • Split dataset, this step will take as input the file path produced by the previous step and split the dataset into train, valid, test sets and move the files to their respective folder structure (eg: train/x, train/y , valid/x etc).
  • Analyze dataset, this step takes the output paths from the split dataset step (eg: train_x_path, etc) then generates stats about the dataset
  • Train model, this step takes the output paths from the split dataset step (train and valid paths), this step will perform some data augmentation at runtime during training.
  • Test model, this step takes the output paths from the split dataset step (test paths)
• All these steps/components are executed in their own docker container and some of them can run in parallel (for instance we can run the Analyze dataset part in parallel to the train model step) but they each share a common PVC storage. So when the dataset is download in the first step, the split dataset does not have to redownload it, same for other components/steps.
• The execution of the pipeline is performed in an automated way meaning we must have a way to automatically download the dataset at runtime. Now it would be possible to simply call the CLI via python subprocess but if we can use a Python API then it's much nicer to work with as we can use our library that wraps the get function etc.
• Note that all steps inputs and outputs are saved automatically by Kubeflow Pipelines so for instance we can clearly see which dataset_name and dataset_version (git tag) was used for this run (there are a lot of other parameters like the seed, etc but let's keep it simple for the description). We also log metrics during training and test of the model etc. Kubeflow offers an API that we use to create metrics and log data, at the end of the pipeline Kubeflow creates a json file of the metrics in a key-value pair style and automatically stores it to s3 and the values are also sent to a mysql database so that we can do lineage tracking later. To be precise, all input and output artifacts are sent to S3 after a run so the model file is sent to s3 with a uuid of the run in the path and all metrics files associated with the run are also uploaded automatically.

It's important to note I just started using dvc today and so far by using the Repo.get method it worked for notebooks but I still need to think about the best way to authenticate the pipeline docker images. Options are ssh by injecting the ssh keys into the image or using Gitlab Project Access Key). If you have a better idea for integration let me know.

Is the data static, or do you want to track changes to it?

I'd say it is static for the same run (the data augmentation is not saved to disk because otherwise that would limit us to have the same data augmentation between epochs) but we store the data augmentation metadata so that we can see which one were applied with the seed etc.

That should cover it all, let me know if this helps or if you have questions,

Thanks

[AlexandreBrown]

We might also start using DVC to track the pipeline run that produced the latest model in production this would allow to retrieve the model metadata that we already store and produce a diff.
I have to see if this fits into our workflow because we'd need to do commits and push in an automated way during the pipeline run which is not something we currently do.

[daavoo]

Very interesting use case @AlexandreBrown , a lot of different things going on, thanks for sharing.

This is getting a little off-topic (maybe better to move to #5924) but, my high-level thought is that from the DVC side we should try to research and consolidate (or help others do) in-depth integration with these different Pipeline "orchestrators?".

For example, I like the approach of this community integration between airflow and DVC:

https://github.com/covid-genomics/airflow-dvc

Without being too familiar with KubeFlow, it seems that DVC could provide some pre-defined components for KubeFlow Pipelines in order to abstract DVC implementation details and that would be already valuable (i.e. DVC download component).

[dberenbaum]

Thanks @AlexandreBrown! That's really helpful and interesting. Your approach makes sense to me.

@mnrozhkov I know you were working on some ideas around airflow integration. Are those still in the plans, and have you considered kubeflow also?