-
Notifications
You must be signed in to change notification settings - Fork 362
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add DirFileSystem #745
Add DirFileSystem #745
Conversation
3e5f154
to
7686547
Compare
Since this is quite substantial, it will take us some time to review. Please bear with us. @isidentical , do you or any of your team have an interest here? @lucmos , am I right in thinking that this implementation wouldn't participate in URL unpacking, i.e., there would be no point in doing something like |
CC: @efiop |
From dvc perspective, we don't have a notion of |
Hello, thanks for the replies and comments! @martindurant yes, the idea is that @efiop Thanks for the review! I will work on it as soon as possible |
8517ccb
to
6afff88
Compare
I would like to increase the number of tests by reusing the tests written for other file system, and wrapping that filesystem in this I tried to do this with the |
Not really! Again, the idea in #651 might give a framework that makes such a thing easier, but it's still not super-clear. |
(I don't think #651 is making any progress, though) |
@lucmos Btw, just curious, what's the plan with |
@efiop not sure I got what you are referring to. Do you have a link to |
That would be fsspec.spec.AbstractBufferedFile , used by several backends which don't have their own native file-like object. The instances have an However, where there is a native file of file wrapper (like LocalFileOpener), probably that instance contains details of the path in terms of the wrapped filesystem and we can do nothing about it. If it's not in a |
I see, thanks for the explanation! Would something like this be enough? |
That would work, in the sense that you would be able to open the right file, but its |
I see, I think it may be OK to do this and document in the docstrings. |
That's probably the easiest thing to do, I don't have a problem with it. |
8d5e117
to
cafd2ce
Compare
Sorry for the delay, I updated the |
Please add your implementation to the API docs. |
0a0f0b6
to
2415a01
Compare
2415a01
to
a5aab99
Compare
Ok, I've added unit tests for all methods for sync/async_impl/asynchronous filesystems, to keep it lightweight and maintainable. Ideally, some func tests would be really nice too, but we don't yet have nice fixtures in fsspec (we have some in dvc, but didn't get to contributing them yet) to reuse different filesystems in a unified way (e.g. fixtures that will automatically generate temp paths in every filesystem, etc), so leaving it for later. I'm not happy with some |
a5aab99
to
8b6512c
Compare
This is used in multiple implementations (s3fs, http, etc) that allow _external_ async use and is documented in fsspec docs (but the example is http-specific), so let's add it to the spec, so that we could officially wrap it in fsspec#745
The DirFileSystem is a filesystem-wrapper. It assumes every path it is dealing with
is relative to its
path
. After performing the necessary paths operations it delegateseverything to the wrapped filesystem.
Resolves #395