Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial ADLS gen2 support #453

Merged
merged 15 commits into from
Aug 28, 2024
Merged

Initial ADLS gen2 support #453

merged 15 commits into from
Aug 28, 2024

Conversation

pjbull
Copy link
Member

@pjbull pjbull commented Jul 17, 2024

Add initial ADLS gen2 support:

  • Accept or create a data_lake_service_client
  • When necessary in AzureBlobClient use the data_lake_service_client API. This does not use the ADLS API everywhere possible since ADLS Gen2 works with the normal blob API in many places. Do this since it is common to enable hierarchical namespaces when creating a blob storage account, not realizing that it needs ADLS APIs for some scenarios.
  • Add the ADLS SDK as a dependency
  • Adds a new Azure test rig for Gen2 if configured
  • Configures ADLS Gen2 testing on Github Actions

This also incorporates the fixes in:

Closes #161


  • Test the different instantiation pipelines
  • Test empty folders on ADLS Gen2
  • Fix mocks to do offline tests for ADLS Gen2 and not
  • Add additionally functionality mentioned in Support Azure Data Lake Storage Gen2 #161

Copy link
Contributor

github-actions bot commented Jul 17, 2024

@github-actions github-actions bot temporarily deployed to pull request July 17, 2024 19:12 Inactive
@pjbull pjbull changed the title Minimal ADLS gen2 support WIP: Minimal ADLS gen2 support Jul 18, 2024
@github-actions github-actions bot temporarily deployed to pull request July 28, 2024 05:26 Inactive
Copy link

codecov bot commented Jul 28, 2024

Codecov Report

Attention: Patch coverage is 97.70115% with 2 lines in your changes missing coverage. Please review.

Project coverage is 94.2%. Comparing base (f3605a6) to head (9853b6b).
Report is 1 commits behind head on master.

Files Patch % Lines
cloudpathlib/azure/azblobclient.py 97.4% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           master    #453     +/-   ##
========================================
+ Coverage    93.6%   94.2%   +0.6%     
========================================
  Files          23      23             
  Lines        1665    1728     +63     
========================================
+ Hits         1559    1629     +70     
+ Misses        106      99      -7     
Files Coverage Δ
cloudpathlib/azure/azblobpath.py 93.6% <100.0%> (+0.9%) ⬆️
cloudpathlib/cloudpath.py 94.5% <100.0%> (+0.3%) ⬆️
cloudpathlib/azure/azblobclient.py 96.8% <97.4%> (+2.0%) ⬆️

... and 2 files with indirect coverage changes

@github-actions github-actions bot temporarily deployed to pull request July 28, 2024 06:04 Inactive
@github-actions github-actions bot temporarily deployed to pull request July 28, 2024 18:14 Inactive
@github-actions github-actions bot temporarily deployed to pull request August 2, 2024 17:07 Inactive
@github-actions github-actions bot temporarily deployed to pull request August 2, 2024 17:18 Inactive
@github-actions github-actions bot temporarily deployed to pull request August 2, 2024 21:19 Inactive
@github-actions github-actions bot temporarily deployed to pull request August 11, 2024 15:15 Inactive
@pjbull pjbull changed the title WIP: Minimal ADLS gen2 support Minimal ADLS gen2 support Aug 11, 2024
Copy link
Member Author

@pjbull pjbull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of typos/tweaks

cloudpathlib/azure/azblobclient.py Outdated Show resolved Hide resolved
cloudpathlib/azure/azblobclient.py Outdated Show resolved Hide resolved
docs/docs/authentication.md Outdated Show resolved Hide resolved
tests/mock_clients/mock_azureblob.py Show resolved Hide resolved
tests/mock_clients/mock_azureblob.py Outdated Show resolved Hide resolved
@pjbull
Copy link
Member Author

pjbull commented Aug 13, 2024

@M0dEx Thanks for your patience—ended up being more here than I expected. I think this is basically ready except a few small typos/tweaks. If you've got time, it would be great if you took this branch for a test drive and took a look at the code changes.

@github-actions github-actions bot temporarily deployed to pull request August 15, 2024 00:07 Inactive
@pjbull
Copy link
Member Author

pjbull commented Aug 20, 2024

Planning to rebase on top of #461 when that is in so that tests are reliable.

@pjbull pjbull changed the title Minimal ADLS gen2 support Initial ADLS gen2 support Aug 22, 2024
@github-actions github-actions bot temporarily deployed to pull request August 22, 2024 18:16 Inactive
@pjbull
Copy link
Member Author

pjbull commented Aug 22, 2024

@jayqi Should be ready for the release. Scope/maintenance burden is a little larger, but I think this is a large standing issue with a fair amount of interest so it is worth it to get in.

Copy link
Member

@jayqi jayqi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, it feels a little messy but I think it's reasonable.

HISTORY.md Outdated Show resolved Hide resolved
@github-actions github-actions bot temporarily deployed to pull request August 23, 2024 23:56 Inactive
@pjbull
Copy link
Member Author

pjbull commented Aug 24, 2024

Okay, it feels a little messy but I think it's reasonable.

Truer words were never spoken.

@github-actions github-actions bot temporarily deployed to pull request August 24, 2024 00:18 Inactive
@github-actions github-actions bot temporarily deployed to pull request August 28, 2024 04:30 Inactive
@github-actions github-actions bot temporarily deployed to pull request August 28, 2024 04:44 Inactive
@pjbull pjbull merged commit 5656879 into master Aug 28, 2024
25 checks passed
@pjbull pjbull deleted the azure-data-lake-gen2 branch August 28, 2024 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Azure Data Lake Storage Gen2
3 participants