Compaction foundations #109
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a building block for implementing time-based compaction of streams. When batches are published, a timestamp is recorded in a secondary index pointing to the last offset of the batch. Time-based compaction will then be able to truncate data based on the offsets recorded in the secondary index.
thedodd
force-pushed
the
99-compaction
branch
from
6a2ae19
to
8c2f0b7
Compare
This commit updates the scheduler to ensure that a Stream's generated StatefulSet targets its corresponding headless service correctly. The pipelines module has been refactored for more optimal storage pattern, which has removed the need for a parallel metadata tree for the pipeline. The utils module has been thoroughly tested which now guards against regressions in our data storage and indexing strategy. This also demonstrates the expected behavior of lexicographically ordered range scans and prefix scans.
thedodd
force-pushed
the
99-compaction
branch
4 times, most recently
from
dfbb047
to
f9403c8
Compare
Pipeline data indexing strategy has been updated to use a single tree and a more efficient indexing strategy based upon lexicographical ordering of encoded keys.
thedodd
force-pushed
the
99-compaction
branch
from
f9403c8
to
54da45a
Compare
Updated to the latest rustc 1.56.0. Refactored Pipelines to simplify the delivery pass mechanism. Fixed a bug where active pipeline instances restored from disk were not being properly pruned if they were already complete. As part of the compaction story, finished pipelines will be removed from disk. Finished pipelines are now being removed.
thedodd
force-pushed
the
99-compaction
branch
3 times, most recently
from
b2e0cf9
to
78e42c1
Compare
Tracking of stream earliest timestamp records is being tracked. This provides the foundation for our stream compaction/truncation system. Updated tests to assert the proper functionality of the earliest timestamp tracking pattern. Fixed a few small docs items in the pipeline-txp example. Update pipeline spawning routine to ensure that only pipelines which are part of the parent stream are spawned.
thedodd
force-pushed
the
99-compaction
branch
5 times, most recently
from
0c5f7e2
to
e0639d6
Compare
Compaction routine is now well-tested. Woot woot! Operator has been updated to pass along retention policy config to stream. Updated deps across all components. closes #99
thedodd
force-pushed
the
99-compaction
branch
from
e0639d6
to
ad08d59
Compare
This ensures that spurious liveness profiles do not interrupt compaction from ever being run. Replaced chrono w/ time crate.
thedodd
force-pushed
the
99-compaction
branch
2 times, most recently
from
23b8f20
to
25e2a87
Compare
Added tests for stream::subscriber::spawn_group_fetch. Added tests for stream::subscriber::try_record_delivery_response. Added tests for stream::subscriber::ensure_subscriber_record.
thedodd
force-pushed
the
99-compaction
branch
4 times, most recently
from
c81d145
to
dcd73a4
Compare
thedodd
force-pushed
the
99-compaction
branch
from
dcd73a4
to
f431ddc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Compaction has been implemented for Streams & Pipelines.
More efficient data storage pattern which also involves few trees/keyspaces.
closes #99