Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Added support to write parquet files to S3 #5914

Merged
merged 19 commits into from
Aug 15, 2024
Merged

feat: Added support to write parquet files to S3 #5914

merged 19 commits into from
Aug 15, 2024

Commits on Jul 25, 2024

  1. Added support for writing to URIs instead of Files

    @malhotrashivam
    malhotrashivam committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    16885c8 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. Added a basic S3 writer using aws-s3-outputstream project

    @malhotrashivam
    malhotrashivam committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    054a0e6 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Added support to write locally and then push the file to S3

    @malhotrashivam
    malhotrashivam committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    7eb1bab View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Minor tweaks

    @malhotrashivam
    malhotrashivam committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    3c08c6c View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Added async S3 write support

    @malhotrashivam
    malhotrashivam committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    a8d35bb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into nightly/sm-uri-write

    @malhotrashivam
    malhotrashivam committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    e445c00 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Abort S3 upload in case of failure

    @malhotrashivam
    malhotrashivam committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    ebad990 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Minor tweaks for clarity

    @malhotrashivam
    malhotrashivam committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    c323bb5 View commit details
    Browse the repository at this point in the history

  2. Review comments part 1

    @malhotrashivam
    malhotrashivam committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    555f64d View commit details
    Browse the repository at this point in the history

  3. Spotless

    @malhotrashivam
    malhotrashivam committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    5f4b340 View commit details
    Browse the repository at this point in the history

  4. Review comments part 2

    @malhotrashivam
    malhotrashivam committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    bef6618 View commit details
    Browse the repository at this point in the history

  5. Rewording comments

    @malhotrashivam
    malhotrashivam committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    9748b1e View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. Added more comments

    @malhotrashivam
    malhotrashivam committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    ec78cef View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Moved shadow file logic from ParquetTools to a new output stream

    @malhotrashivam
    malhotrashivam committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    b4af4ca View commit details
    Browse the repository at this point in the history

  2. Added more tests

    @malhotrashivam
    malhotrashivam committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    4309c4e View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. Review comments continued

    @malhotrashivam
    malhotrashivam committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    bfd3f4e View commit details
    Browse the repository at this point in the history

  2. Fixed failing test

    @malhotrashivam
    malhotrashivam committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    9655e4a View commit details
    Browse the repository at this point in the history

  3. Review continued

    @malhotrashivam
    malhotrashivam committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    1c1d62a View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. Improved comments and removed some checks

    @malhotrashivam
    malhotrashivam committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    d99af9c View commit details
    Browse the repository at this point in the history