Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codec trait and array refactor #6

Merged
merged 2 commits into from
Feb 12, 2024
Merged

Codec trait and array refactor #6

merged 2 commits into from
Feb 12, 2024

Conversation

LDeakin
Copy link
Owner

@LDeakin LDeakin commented Feb 12, 2024

This is a major refactor of codec traits and the array structure.
Previously, various codec and array methods had three variants:

  1. <method>_opt(..., parallel: bool) supporting optional parallelism
  2. <method>(...), serial
  3. par_<method>(...) , parallel

Now, there are just two variants:

  1. <method>_opt(..., options: &SomeOptionsStruct)
  2. <method>(...), default options (maximum concurrency)

This new API provides finer-grained control of the number of concurrent operations executed within an encode/decode method.
Most internal operations use the _opt API. The efficient/maximum concurrency of codecs can be queried (presently, this may not be accurate for many codecs).

This has the potential to improve performance and reduce memory usage. For example, a call of Array::retrieve_chunks which decodes multiple shards could recognise that the inner chunks can be decoded with massive parallelisation, and thus only decode one shard at a time. This behaviour is not yet implemented, but will follow shortly.

Copy link

codecov bot commented Feb 12, 2024

Codecov Report

Attention: 786 lines in your changes are missing coverage. Please review.

Comparison is base (5eab757) 77.57% compared to head (2433c2a) 75.71%.

Files Patch % Lines
src/array/array_async_readable.rs 0.00% 188 Missing ⚠️
src/array/array_async_readable_writable.rs 0.00% 135 Missing ⚠️
src/array/array_sync_readable.rs 32.00% 85 Missing ⚠️
src/array/array_async_writable.rs 0.00% 72 Missing ⚠️
src/array/array_sync_readable_writable.rs 29.41% 72 Missing ⚠️
src/array/array_sync_writable.rs 5.45% 52 Missing ⚠️
src/array/codec.rs 68.59% 38 Missing ⚠️
src/array/codec/array_to_bytes/codec_chain.rs 41.37% 34 Missing ⚠️
...rc/array/codec/array_to_bytes/bytes/bytes_codec.rs 17.39% 19 Missing ⚠️
...ay/codec/array_to_bytes/sharding/sharding_codec.rs 91.87% 13 Missing ⚠️
... and 12 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main       #6      +/-   ##
==========================================
- Coverage   77.57%   75.71%   -1.87%     
==========================================
  Files         123      124       +1     
  Lines       17606    17878     +272     
==========================================
- Hits        13658    13536     -122     
- Misses       3948     4342     +394     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@LDeakin LDeakin marked this pull request as ready for review February 12, 2024 05:51
@LDeakin LDeakin merged commit 8d5a27c into main Feb 12, 2024
16 of 18 checks passed
@LDeakin LDeakin deleted the codec_refactor2 branch February 12, 2024 05:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant