write unsharded parquet?

[mdsumner]

Can we write unsharded parquet as the format? i.e. i'm doing

vds.virtualize.to_kerchunk('oisst.parquet', format='parquet')

and end up with a directory of parq fragments:

oisst.parquet/.zmetadata
oisst.parquet/anom/refs.0.parq
oisst.parquet/err/refs.0.parq
oisst.parquet/ice/refs.0.parq
oisst.parquet/lat/refs.0.parq
oisst.parquet/lon/refs.0.parq
oisst.parquet/sst/refs.0.parq
oisst.parquet/time/refs.0.parq

but I need a single file for hosting that in object store. Can I bundle that into one .parquet? (I very well may be missing something about how to onshare the parquet form in object store ... so very happy to take pointers there!). Thanks!!

In the bigger picture I'm having problems connecting s3 to

s3://vzarr/oisst.json  ## endpoint projects.pawsey.org.au

(it opens fine with xarray from https://projects.pawsey.org.au/vzarr/oisst.json)

which led me to try parquet, but I expect I want that instead of json ultimately for very large collections.

[keewis]

without knowing too much about the details of the format, the "parquet" kerchunk format copies the structure of zarr (the metadata is consolidated into .zmetadata), but replaces the chunk files with one or more parquet files (@norlandrhagen calls this format "zarrquet" instead of parquet).

Do you have a specific reason why you can't upload the references as multiple files to object storage? I have very little experience with object storage, but so far just writing the entire directory there appeared to work great. Otherwise I believe you can simply pack the directory into an archive (zip / tar / whatever) and access it using fsspec by modifying the url (zip::s3://bucket/path/to/refs.parquet.zip? not sure, though, I always forget the syntax of protocol chaining).

As for why the json didn't work, I believe you need this (same for kerchunk parquet, by the way):

so = {"endpoint_url": "https://projects.pawsey.org.au", "anon": True}
ds = xr.open_dataset(
    "s3://vzarr/virt_oisst.json",
    engine="kerchunk",
    storage_options={"target_options": so, "remote_options": so},
)

i.e. we need to specify how to access the reference file (target_options) and how to access the actual data (remote_options). See also the docs for the reference file system.

[mdsumner]

Thank you!! I didn't know anything about the relationship to fsspec actually, will explore everything here 🙏

[TomNicholas]

You can only write out what fsspec can interpret.

At this point though I would recommend experimenting using icechunk instead of kerchunk. AFAIK the only advantage kerchunk parquet has over icechunk's model is that icechunk is newer and therefore less stable. (I should double-check this and add it to the FAQ.)

but I need a single file for hosting that in object store.

I don't know why this would be the case though. If that's really a restriction then your only option is kerchunk json, as both kerchunk parquet ("zarrquet") and icechunk are really many files together.

[mdsumner]

ok will do! I figured out my object problem, total mistake on my part

[mdsumner]

this is working!

https://gist.github.com/mdsumner/c72ff510bf41c433662ef703a635daf8

so now for Icechunk! thanks again for the assistance it's really helpful for me getting these things tied together