Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add storage options and protocol #135

Merged
merged 8 commits into from
Aug 28, 2023

Conversation

ap--
Copy link
Collaborator

@ap-- ap-- commented Aug 27, 2023

This PR adds .storage_options and .protocol to UPath.

The "non-pathlib" interface of UPath is then:

  • UPath.protocol: str the fsspec protocol
  • UPath.storage_options: dict[str, Any] the storage_options for the fsspec class
  • UPath.path: str the fsspec compatible path for use with the fsspec class
  • UPath.fs: AbstractFileSystem convenience attribute to access a instantiated fsspec class

the first three provide a public interface to access a file via fsspec as follows:

pth = UPath(..., **sopts)

from fsspec import filesystem

fs = filesystem(pth.protocol, **pth.storage_options)
with fs.open(pth.path) as f:
    data = f.read()

This closes #129, and closes #91.

additional changes:

  • To support correct behavior for cloud filesystems .path had to be overwritten to return a path that includes the bucket (consistent with fsspec).
  • Since the webdav filesystem doesn't serialize to string well (which is why we use the webdav+http and webdav+https protocol workaround) the WebdavPath.protocol returns "webdav" and the base_url is added to the .storage_options. Which then allows to support the usecase from the code mentioned above again.
  • HTTPPath().path returns a full url, in line with fsspec behavior.

extra note regarding local filesystems

# protocol is empty string for non-uri localpaths:
p = UPath("/path/file.txt")
assert isinstance(p, PosixUPath)
assert p.protocol == ""

# and it's "file" for uri localpaths
u = UPath("file:///path/file.txt")
assert isinstance(p, LocalPath)
assert p.protocol == "file"

Both "file" and "" return LocalFileSystem when using fsspec.get_filesystem_class(protocol).

Copy link
Collaborator

@normanrz normanrz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be very handy for my use cases!

@ap-- ap-- merged commit 24e15d3 into fsspec:main Aug 28, 2023
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants