Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LS seems not to be working #224

Open
GabrielAzevedoFerreiraQB opened this issue Apr 22, 2021 · 2 comments
Open

LS seems not to be working #224

GabrielAzevedoFerreiraQB opened this issue Apr 22, 2021 · 2 comments

Comments

@GabrielAzevedoFerreiraQB
Copy link

GabrielAzevedoFerreiraQB commented Apr 22, 2021

LS is not showing the actual directories. it does not behave as expected for reading directories, but works as expected if we pass it the path to a file.
I have the following blob:
image

and am running the following

st_op = {'connection_string': connect_str}
fs = fsspec.filesystem('abfs', **st_op)
print(fs.ls('abfs://', detail=True))
print(fs.ls('abfs://demo/', detail=True))
print(fs.open('abfs://demo/crash.csv').read())
print(fs.ls('abfs://demo/dir/', detail=True))
<class 'adlfs.spec.AzureDatalakeFileSystem'>
<adlfs.spec.AzureBlobFileSystem object at 0x7fb04593f350>
containers:(OK!) [{'metadata': {}, 'deleted': None, 'last_modified': datetime.datetime(2021, 4, 21, 7, 16, 44, tzinfo=datetime.timezone.utc), 'etag': '"0x8D904956B0F24A6"', 'name': '9c6d625f-ccad-4f75-b9bb-24659ea5da2a', 'size': None, 'type': 'directory'}, {'metadata': {}, 'deleted': None, 'last_modified': datetime.datetime(2021, 4, 22, 5, 36, 35, tzinfo=datetime.timezone.utc), 'etag': '"0x8D905509818F626"', 'name': 'another', 'size': None, 'type': 'directory'}, {'metadata': {}, 'deleted': None, 'last_modified': datetime.datetime(2021, 4, 21, 7, 12, 57, tzinfo=datetime.timezone.utc), 'etag': '"0x8D90494E42673D3"', 'name': 'demo', 'size': None, 'type': 'directory'}]
single container is empty (Not OK!!!)  []
reading a file works (OK) b'driver_id,time,crash,source\nadkavy,1213037028000,0,San_Francisco\nadkavy,1213036968000,0,San_Francisc'
reading a file works (OK) []
ls on an (existing) file works []
ls on an (existing) folder does not work (Not OK!!)

Process finished with exit code 0

What happened:

What you expected to happen:

Minimal Complete Verifiable Example:

# Put your MCVE code here

Anything else we need to know?:

Environment:

  • Dask version:
  • Python version:
  • Operating System:
  • Install method (conda, pip, source):
@GabrielAzevedoFerreiraQB
Copy link
Author

pip show fsspec Name: fsspec
Version: 2021.4.0
Summary: File-system specification
Home-page: http://github.com/intake/filesystem_spec
Author: None
Author-email: None
License: BSD
Location: /opt/anaconda3/envs/lineage_viz37/lib/python3.7/site-packages
Requires: 
Required-by: s3fs, kedro, adlfs

pip show adlfs 
Name: adlfs
Version: 0.7.3
Summary: Access Azure Datalake Gen1 with fsspec and dask
Home-page: https://github.com/dask/adlfs/
Author: None
Author-email: None
License: BSD
Location: /opt/anaconda3/envs/lineage_viz37/lib/python3.7/site-packages
Requires: azure-identity, azure-datalake-store, azure-core, aiohttp, msrestazure, requests, azure-storage-blob, fsspec
Required-by: 

@hayesgb
Copy link
Collaborator

hayesgb commented May 13, 2021

What do you get with print(fs.ls('abfs://demo/dir', detail=True))?

Please also see this discussion on fsspec.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants