Please support seekable format

[joshtriplett]

There's a seekable zstd format, which makes it possible to seek to and decompress segments independently. It includes a jump table at the end for all the segments.

https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md

I'd love to have support for this format in zstd-rs, both to compress (with an API to mark the end of a segment) and to decompress (parsing the jump table and allowing decompression of individual segments).

[NobodyXu]

There's a separate binding crate https://lib.rs/crates/zstd-seekable which can be used as a reference implementation

[x1957]

any updates?

[NobodyXu]

cc @gyscos Are you interested in having this seekable zstd in this repository/crate?

Looking at the code, there's a ZSTD_seekable and ZSTD_seekable_compressor, so it looks like just two more bindings to add.

If it is just binding, then I could give it a shot.

[kylewlacy]

I've been interested in this issue. I originally tried using zstd-seekable, but it only works with sync readers. I also hit a bug / unintuitive behavior in the crate (maybe related to https://nest.pijul.com/pmeunier/zstd-seekable/discussions/1?). While trying to figure out how to solve that issue or contribute to it, I eventually looked into the seekable format more and realized that the format itself is dead simple

...so, I just published zstd-framed, which wraps zstd with a custom Rust implementation for the seekable format parts. It includes both sync and async reader and writer types.

Just wanted to chime in because I published the code under the Unlicense, so it should be pretty easy to adapt into this crate if anyone was interested (although probably with some tweaks to the interface-- I built something with lots of levers for my specific use case)