FileSystem Object Storage, OpenDal?

[mrhamburg]

I saw that the filesystem is now compatible with Hadoop and requires plugins to enable support for object storage backends. Would it be beneficial and easier to integrate OpenDal with its Java and Python bindings to enable easier integration with object storage and other filesystems (s3, azblob, gcs, ftp?, hugginface?) - just thinking out loud and might be able to contribute but not sure if this aligns with future architecture

[zhoukangcn]

We are working on the feature of fileset on S3, and will contribute to community later. cc @xiaozcy

[shaofengshi]

@mrhamburg Hello Menno, so far the community don't have the plan to integrate OpenDAL I think (we don't have experience on that project). What's the benefits it can bring, and will that dependency introduce additional maintenance effort, that are the concerns we have. If you can give more inputs that will be appreciated！

[mrhamburg]

Hi @shaofengshi, the only added benefit is the abstraction between Gravitino and a Fileset backend so that it allows for more backends to be added with less effort. Risk offcourse is that you are introducing an additional dependency. Let's say I want to add a Fileset for Backblaze B2, IPFS or Cloudflare D1, I would have to either integrate this from scratch (within Gravitino) or introduce a dependency (multiple, or in the case of OpenDal, just one). Same goes for the backend configs (one unified approach already backed into OpenDal). While OpenDal is written in Rust, it ships with Java bindings