-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compaction foundations #109
Commits on Oct 22, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 07e83d6 - Browse repository at this point
Copy the full SHA 07e83d6View commit details -
Secondary index on Streams for recording timestamps of offsets
This is a building block for implementing time-based compaction of streams. When batches are published, a timestamp is recorded in a secondary index pointing to the last offset of the batch. Time-based compaction will then be able to truncate data based on the offsets recorded in the secondary index.
Configuration menu - View commit details
-
Copy full SHA for a3f6801 - Browse repository at this point
Copy the full SHA a3f6801View commit details
Commits on Oct 24, 2021
-
Update scheduler, pipeline indexing refactor & tests
This commit updates the scheduler to ensure that a Stream's generated StatefulSet targets its corresponding headless service correctly. The pipelines module has been refactored for more optimal storage pattern, which has removed the need for a parallel metadata tree for the pipeline. The utils module has been thoroughly tested which now guards against regressions in our data storage and indexing strategy. This also demonstrates the expected behavior of lexicographically ordered range scans and prefix scans.
Configuration menu - View commit details
-
Copy full SHA for e035cc1 - Browse repository at this point
Copy the full SHA e035cc1View commit details
Commits on Oct 25, 2021
-
Pipeline storage has been updated
Pipeline data indexing strategy has been updated to use a single tree and a more efficient indexing strategy based upon lexicographical ordering of encoded keys.
Configuration menu - View commit details
-
Copy full SHA for 54da45a - Browse repository at this point
Copy the full SHA 54da45aView commit details
Commits on Oct 27, 2021
-
stream: update docker rustc version & pipelines refactor
Updated to the latest rustc 1.56.0. Refactored Pipelines to simplify the delivery pass mechanism. Fixed a bug where active pipeline instances restored from disk were not being properly pruned if they were already complete. As part of the compaction story, finished pipelines will be removed from disk. Finished pipelines are now being removed.
Configuration menu - View commit details
-
Copy full SHA for 456911a - Browse repository at this point
Copy the full SHA 456911aView commit details
Commits on Nov 4, 2021
-
stream: add stream retention policy
Tracking of stream earliest timestamp records is being tracked. This provides the foundation for our stream compaction/truncation system. Updated tests to assert the proper functionality of the earliest timestamp tracking pattern. Fixed a few small docs items in the pipeline-txp example. Update pipeline spawning routine to ensure that only pipelines which are part of the parent stream are spawned.
Configuration menu - View commit details
-
Copy full SHA for ca035e0 - Browse repository at this point
Copy the full SHA ca035e0View commit details
Commits on Nov 5, 2021
-
Compaction routine is now well-tested. Woot woot! Operator has been updated to pass along retention policy config to stream. Updated deps across all components. closes #99
Configuration menu - View commit details
-
Copy full SHA for ad08d59 - Browse repository at this point
Copy the full SHA ad08d59View commit details
Commits on Nov 8, 2021
-
Compaction is now using disk-backed checkpoints
This ensures that spurious liveness profiles do not interrupt compaction from ever being run. Replaced chrono w/ time crate.
Configuration menu - View commit details
-
Copy full SHA for 00bfb0a - Browse repository at this point
Copy the full SHA 00bfb0aView commit details
Commits on Nov 9, 2021
-
Add tests for stream subscription functionality
Added tests for stream::subscriber::spawn_group_fetch. Added tests for stream::subscriber::try_record_delivery_response. Added tests for stream::subscriber::ensure_subscriber_record.
Configuration menu - View commit details
-
Copy full SHA for 75ea298 - Browse repository at this point
Copy the full SHA 75ea298View commit details
Commits on Nov 10, 2021
-
Configuration menu - View commit details
-
Copy full SHA for f431ddc - Browse repository at this point
Copy the full SHA f431ddcView commit details