Offer improved customization options for user subclasses of UPath #173
Conversation
|
All current tests pass. |
|
@potiuk could you please run your tests once more? This implementation should now work without changes to ObjectStoragePath. It'll throw a ton of deprecation warnings but should in theory be completely backwards compatible. Anyhow, I'll try to add the migration docs asap and get the new release out. |
|
Sure. Running it now here: https://github.com/apache/airflow/actions/runs/7820390750?pr=36755 |
|
Rerun it again :) ? |
Thanks! But I think it's not necessary. The I do now cover a lot of different use cases for maintainable subclassing though. And thank you so much for your fast response time and help with running your test suite! |
|
Ok. I see it's merged now. If I understand correctly - the universal_pathlib will likely release a 0.2.0 (?) version with both Py3.12 some time soon - and the way we are using 0.1.4 is not going to be compatible with it ? Or maybe it will be 0.1.5 or 1.0 :)? I am asking because we are just about to release Airflow 2.8.2, and since we know that future releases of universal_pathlib will break Airflow, we would rather put some limits in our dependencies to prevent it. Currently we have >=0.1.4 - as we do for most of our dependencies, as we only upper-bind dependencies when we expect breaking changes (whch is the case). When do you expect this to happen (very roughly, I know those things are difficult to foresee) ? Do I understand correctly that we will have to change our code to adapt to the new interfaces? Can you help with that - maybe together with @bolkedebruin - you can agree on implementing the right set of changes that will be less "private" and future compatible. |
|
I went through all code todos now, and am ready to release a new version. Right now I still need to write instructions on how to migrate old patterns to the newer style, to make the transition smooth for downstream libraries that subclassed UPath. If you release a new version of Airflow without changes to your ObjectStoragePath implementation, then I recommend to guard your dependencies. I'm happy to help with a PR to port ObjectStoragePath to the new style. |
|
Thanks! Scheduled it for 2.8.2 release with apache/airflow#37311. We already have good mechanism to protect against such cases (i.e. constraints ) but not everyone is using them, so yeah, until we migrate, we will limit it. |
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in apache#36755
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in #36755
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in #36755 (cherry picked from commit 1301274)
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in #36755 (cherry picked from commit 1301274)
|
I recently ran a from upath.implementations.cloud import S3Path
assert str(S3Path("s3://bucket/prefix/to/key/")).endswith("/")After this PR, Perhaps the problem is that the way we're using S3 paths is too different from the abstract concept of a Path, and we need to stick with strings, but this is at least a regression in behavior that started with v0.2.0 (works fine in v0.1.4). |
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in apache/airflow#36755 GitOrigin-RevId: 13012744ada457883e57848f6fc45454d9c25a4c
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in apache/airflow#36755 GitOrigin-RevId: 13012744ada457883e57848f6fc45454d9c25a4c
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in apache/airflow#36755 GitOrigin-RevId: 13012744ada457883e57848f6fc45454d9c25a4c
WIP PR to fix #172
Please refer to the issue for a detailed description.
TODO
_accessorinpy3.12classes and provide warning with instructions on how to migrate.docs (2) provide instructions for how to migrate(moved to new issue)_FSSpecAccessor._format_pathto.path(with potential customization)docs (3) recommend overriding(moved to new issue)UPath.mkdir, etc... directly.storage_optionextraction classmethod(or skip?)the pathlib 3.8-3.11 tests...NOTES
__init__or by providing a custom_transform_init_argsmethod (needs more tests)