Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compaction foundations #109

Merged
merged 10 commits into from
Nov 10, 2021
Merged

Compaction foundations #109

merged 10 commits into from
Nov 10, 2021

Commits on Oct 22, 2021

  1. Configuration menu
    Copy the full SHA
    07e83d6 View commit details
    Browse the repository at this point in the history
  2. Secondary index on Streams for recording timestamps of offsets

    This is a building block for implementing time-based compaction of
    streams. When batches are published, a timestamp is recorded in a
    secondary index pointing to the last offset of the batch. Time-based
    compaction will then be able to truncate data based on the offsets
    recorded in the secondary index.
    thedodd committed Oct 22, 2021
    Configuration menu
    Copy the full SHA
    a3f6801 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2021

  1. Update scheduler, pipeline indexing refactor & tests

    This commit updates the scheduler to ensure that a Stream's generated
    StatefulSet targets its corresponding headless service correctly.
    
    The pipelines module has been refactored for more optimal storage
    pattern, which has removed the need for a parallel metadata tree for the
    pipeline.
    
    The utils module has been thoroughly tested which now guards against
    regressions in our data storage and indexing strategy. This also
    demonstrates the expected behavior of lexicographically ordered
    range scans and prefix scans.
    thedodd committed Oct 24, 2021
    Configuration menu
    Copy the full SHA
    e035cc1 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2021

  1. Pipeline storage has been updated

    Pipeline data indexing strategy has been updated to use a single tree and
    a more efficient indexing strategy based upon lexicographical ordering
    of encoded keys.
    thedodd committed Oct 25, 2021
    Configuration menu
    Copy the full SHA
    54da45a View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2021

  1. stream: update docker rustc version & pipelines refactor

    Updated to the latest rustc 1.56.0.
    
    Refactored Pipelines to simplify the delivery pass mechanism.
    
    Fixed a bug where active pipeline instances restored from disk were not
    being properly pruned if they were already complete. As part of the
    compaction story, finished pipelines will be removed from disk.
    
    Finished pipelines are now being removed.
    thedodd committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    456911a View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2021

  1. stream: add stream retention policy

    Tracking of stream earliest timestamp records is being tracked. This
    provides the foundation for our stream compaction/truncation system.
    
    Updated tests to assert the proper functionality of the earliest
    timestamp tracking pattern.
    
    Fixed a few small docs items in the pipeline-txp example.
    
    Update pipeline spawning routine to ensure that only pipelines which are
    part of the parent stream are spawned.
    thedodd committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    ca035e0 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2021

  1. Implement compaction

    Compaction routine is now well-tested. Woot woot!
    
    Operator has been updated to pass along retention policy config to
    stream.
    
    Updated deps across all components.
    
    closes #99
    thedodd committed Nov 5, 2021
    Configuration menu
    Copy the full SHA
    ad08d59 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2021

  1. Compaction is now using disk-backed checkpoints

    This ensures that spurious liveness profiles do not interrupt compaction
    from ever being run.
    
    Replaced chrono w/ time crate.
    thedodd committed Nov 8, 2021
    Configuration menu
    Copy the full SHA
    00bfb0a View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2021

  1. Add tests for stream subscription functionality

    Added tests for stream::subscriber::spawn_group_fetch.
    
    Added tests for stream::subscriber::try_record_delivery_response.
    
    Added tests for stream::subscriber::ensure_subscriber_record.
    thedodd committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    75ea298 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2021

  1. Configure base CI

    thedodd committed Nov 10, 2021
    Configuration menu
    Copy the full SHA
    f431ddc View commit details
    Browse the repository at this point in the history