Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

minio integration: write intermediate data to minio, instead of storing #79

Draft
wants to merge 36 commits into
base: main
Choose a base branch
from
Draft

minio integration: write intermediate data to minio, instead of storing #79

wants to merge 36 commits into from

Commits on Apr 17, 2024

  1. changes for activation to support etl

    @supradeep2819
    supradeep2819 committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    894a6bb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' of github.com:valmi-io/valmi-activation into dev

    @rajvarkala
    rajvarkala committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    d845bdc View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. Merge pull request #70 from valmi-io/main 

    chore: Back merge main to dev
    @supradeep2819
    supradeep2819 authored Apr 26, 2024
    Configuration menu
    Copy the full SHA
    e06dc9e View commit details
    Browse the repository at this point in the history

  2. fix: add missing import to connector lib

    @gane5hvarma
    gane5hvarma committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    5d31e97 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2024

  1. Merge branch 'dev' of github.com:valmi-io/valmi-activation into dev

    @rajvarkala
    rajvarkala committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    ba79b23 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. feat: add normalization and dbt step into dag (#71) 

    * feat: add normalization and dbt step into dag
    @gane5hvarma
    gane5hvarma authored Apr 30, 2024
    Configuration menu
    Copy the full SHA
    80cd1fc View commit details
    Browse the repository at this point in the history

Commits on May 2, 2024

  1. feature: Added endpoint to retrive last run status

    @supradeep2819
    supradeep2819 committed May 2, 2024
    Configuration menu
    Copy the full SHA
    a4999b8 View commit details
    Browse the repository at this point in the history

  2. feature: Added endpoint for getting last run status

    @supradeep2819
    supradeep2819 committed May 2, 2024
    Configuration menu
    Copy the full SHA
    ececb3a View commit details
    Browse the repository at this point in the history

  3. Merge pull request #72 from valmi-io/latest_run_status 

    feature: Added endpoint to retrive last run status
    @supradeep2819
    supradeep2819 authored May 2, 2024
    Configuration menu
    Copy the full SHA
    f82d3b9 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. feat: Handled missed schema for last run status

    @supradeep2819
    supradeep2819 committed May 3, 2024
    Configuration menu
    Copy the full SHA
    e06bbc1 View commit details
    Browse the repository at this point in the history

  2. fix: retriving last run ID in descending order

    @supradeep2819
    supradeep2819 committed May 3, 2024
    Configuration menu
    Copy the full SHA
    d758da6 View commit details
    Browse the repository at this point in the history

  3. fix: retriving last run ID in descending order

    @supradeep2819
    supradeep2819 committed May 3, 2024
    Configuration menu
    Copy the full SHA
    a1ce573 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #73 from valmi-io/latest_run_status 

    fix: retriving last run ID in descending order
    @supradeep2819
    supradeep2819 authored May 3, 2024
    Configuration menu
    Copy the full SHA
    682460e View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. "feat: Enhance source postgres configuration to provide specific tabl… 

    …e schema
    @supradeep2819
    supradeep2819 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    1b56fcb View commit details
    Browse the repository at this point in the history

  2. Merge pull request #74 from valmi-io/latest_run_status 

    "feat: Enhance source postgres configuration to provide specific tabl…
    @supradeep2819
    supradeep2819 authored May 10, 2024
    Configuration menu
    Copy the full SHA
    a909e5c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'dev' of github.com:valmi-io/valmi-activation into dev

    @rajvarkala
    rajvarkala committed May 10, 2024
    Configuration menu
    Copy the full SHA
    fb0d957 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. feat: changes needed for destination postgres to work

    @gane5hvarma
    gane5hvarma committed May 14, 2024
    Configuration menu
    Copy the full SHA
    f69dcd8 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. feat: Adding checkpointing to etl sources and destinations

    @rajvarkala
    rajvarkala committed May 15, 2024
    Configuration menu
    Copy the full SHA
    374bf85 View commit details
    Browse the repository at this point in the history

  2. version up

    @rajvarkala
    rajvarkala committed May 15, 2024
    Configuration menu
    Copy the full SHA
    b22b4a2 View commit details
    Browse the repository at this point in the history

  3. feat: for etl, batching source records only on seeing state messages

    @rajvarkala
    rajvarkala committed May 15, 2024
    Configuration menu
    Copy the full SHA
    d307b97 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. feat: generating catalog when query is present postgres config

    @supradeep2819
    supradeep2819 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    e61e1a4 View commit details
    Browse the repository at this point in the history

  2. feat: multiple state messages stored for etl sources

    @rajvarkala
    rajvarkala committed May 16, 2024
    Configuration menu
    Copy the full SHA
    79654bb View commit details
    Browse the repository at this point in the history

  3. Merge branch 'dev' of github.com:valmi-io/valmi-activation into dev

    @rajvarkala
    rajvarkala committed May 16, 2024
    Configuration menu
    Copy the full SHA
    dd210cf View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. Fix finalizer step which got stuck because of transformation (#75)

    @gane5hvarma
    gane5hvarma authored May 17, 2024
    Configuration menu
    Copy the full SHA
    172195f View commit details
    Browse the repository at this point in the history

Commits on May 20, 2024

  1. feat: rm print statements and update version

    @gane5hvarma
    gane5hvarma committed May 20, 2024
    Configuration menu
    Copy the full SHA
    bad286b View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. feat: Add endpoint to fetch latest successful sync

    @supradeep2819
    supradeep2819 committed May 23, 2024
    Configuration menu
    Copy the full SHA
    dea7d8d View commit details
    Browse the repository at this point in the history

  2. feat: renamed timestamp with run_end_at in latest_run_status endpoint

    @supradeep2819
    supradeep2819 committed May 23, 2024
    Configuration menu
    Copy the full SHA
    5ff5f39 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. feat: Used schemas instead of dict in sync service

    @supradeep2819
    supradeep2819 committed May 28, 2024
    Configuration menu
    Copy the full SHA
    3197240 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #76 from valmi-io/feat.last_successful_sync 

    feat: Add endpoint to fetch latest successful sync
    @supradeep2819
    supradeep2819 authored May 28, 2024
    Configuration menu
    Copy the full SHA
    5db346c View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. bug: metrics fix for etl

    @rajvarkala
    rajvarkala committed May 29, 2024
    Configuration menu
    Copy the full SHA
    c3d109e View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. feat: source-postgres changes to support filtered queries

    @supradeep2819
    supradeep2819 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    3bd3a8a View commit details
    Browse the repository at this point in the history

  2. feat: changes for source-postgres to support filter queries

    @supradeep2819
    supradeep2819 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    0701a1a View commit details
    Browse the repository at this point in the history

  3. Merge pull request #77 from valmi-io/feat.source_postgres 

    feat: source-postgres changes to support filtered queries
    @supradeep2819
    supradeep2819 authored Jun 6, 2024
    Configuration menu
    Copy the full SHA
    7894aff View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. minio integration: write intermediate data to minio, instead of stori… 

    …ng locally.
    @saurav-malani
    saurav-malani committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    2f89aa9 View commit details
    Browse the repository at this point in the history

  2. minio integration: write intermediate data to minio, instead of stori… 

    …ng locally.
    @saurav-malani
    saurav-malani committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    34233fa View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. removed gzip part

    @saurav-malani
    saurav-malani committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    54f2ebc View commit details
    Browse the repository at this point in the history