Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run table and unit table #124

Merged
merged 40 commits into from
Nov 17, 2024
Merged

Run table and unit table #124

merged 40 commits into from
Nov 17, 2024

Commits on Nov 7, 2024

  1. Moved erd test function to test_file_writer and deleted test_plugin.p… 

    …y/CI file
    Vedant P Iyer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    7176760 View commit details
    Browse the repository at this point in the history

  2. created test functions for schema, yaml, toml readers

    Vedant P Iyer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    f874bd5 View commit details
    Browse the repository at this point in the history

  3. updated file writer CI test with extra package installs for erd test

    Vedant P Iyer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    8544983 View commit details
    Browse the repository at this point in the history

  4. created a unified dsi_units table for all ingested data and updated h… 

    …andling for writers
    Vedant P Iyer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    56e9d15 View commit details
    Browse the repository at this point in the history

  5. Updated other files to handle unified units table and separate back-w… 

    …rite/read in core calls
    Vedant P Iyer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    17c2734 View commit details
    Browse the repository at this point in the history

  6. Created run table to store metadata for every workflow run and handle… 

    …d unified unit table in sqlite reader/writer classes
    Vedant P Iyer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    10d0e54 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. updated backend actions and split back-write and read actions

    Vedant P Iyer committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    7eb97dd View commit details
    Browse the repository at this point in the history

  2. created set_schema2 which sets file reader data dict to metadata dict

    Vedant P Iyer committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    fdcb7a3 View commit details
    Browse the repository at this point in the history

  3. added inspect artifact handler function example workflow

    Vedant P Iyer committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    5a68c19 View commit details
    Browse the repository at this point in the history

  4. updated file writers to handle unified unit table

    Vedant P Iyer committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    bd205d9 View commit details
    Browse the repository at this point in the history

  5. updated schema, yaml, toml readers to use set_schema2 instead of old … 

    …metadata functions
    Vedant P Iyer committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    f638ee1 View commit details
    Browse the repository at this point in the history

  6. Only commit db insert if all data in workflow is stable/non repetitiv… 

    …e, and updated inspect artifact function to generate Jupyter notebook
    Vedant P Iyer committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    e4229f8 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. only execute sql statements if no insertion error else rollback all p… 

    …revious
    Vedant P Iyer committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    bd3b2f8 View commit details
    Browse the repository at this point in the history

  2. customized print error if duplicate/error data ingested to backend

    Vedant P Iyer committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    592a218 View commit details
    Browse the repository at this point in the history

  3. updated er diagram writer to generate dot as python object and no ext… 

    …ernal file generated
    Vedant P Iyer committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    4c3c8d6 View commit details
    Browse the repository at this point in the history

  4. created generic text file reader -- assumes only one table in data so… 

    …urce
    Vedant P Iyer committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    b41422d View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2024

  1. merged changes with main branch

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    e1afd9b View commit details
    Browse the repository at this point in the history

  2. combined and tested sqlite read/write class

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    0b49b3e View commit details
    Browse the repository at this point in the history

  3. Updated other files to handle unified units table and separate back-w… 

    …rite/read in core calls
    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    6d458ef View commit details
    Browse the repository at this point in the history

  4. updated primary key handling in schema reader

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    dec406c View commit details
    Browse the repository at this point in the history

  5. Created run table to store metadata for every workflow run and handle… 

    …d unified unit table in sqlite reader/writer classes
    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    750931b View commit details
    Browse the repository at this point in the history

  6. updated backend actions and split back-write and read actions

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    0411470 View commit details
    Browse the repository at this point in the history

  7. Only commit db insert if all data in workflow is stable/non repetitiv… 

    …e, and updated inspect artifact function to generate Jupyter notebook
    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    3e97340 View commit details
    Browse the repository at this point in the history

  8. changed coreterminal to merge to main

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    3536990 View commit details
    Browse the repository at this point in the history

  9. updated sqlite to merge to main

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    0a149d6 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'main' into runTable_and_unitTable

    @Vedant1
    Vedant1 authored Nov 15, 2024
    Configuration menu
    Copy the full SHA
    c5126a5 View commit details
    Browse the repository at this point in the history

  11. added graphviz to pip install in CI file

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    402477b View commit details
    Browse the repository at this point in the history

  12. updated tests to reflect units table in collections now

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    5895856 View commit details
    Browse the repository at this point in the history

  13. updated test file reader again

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    23252d1 View commit details
    Browse the repository at this point in the history

  14. moved nbc and nbf dependencies inline for only inspect artifact

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    718b439 View commit details
    Browse the repository at this point in the history

  15. only backend read can call read and only backend write can call put

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    4679e6e View commit details
    Browse the repository at this point in the history

  16. removed extra imports and updated erd writer if graphviz installed or… 

    … not(manual dot file)
    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    2543522 View commit details
    Browse the repository at this point in the history

  17. Updated csv reader to be faster and renamed yaml/toml to YAML1 TOML1

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    1e4f2e0 View commit details
    Browse the repository at this point in the history

  18. Updated with new toml and yaml reader names

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    c9e0845 View commit details
    Browse the repository at this point in the history

  19. Updated init function of csv reader

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    2047081 View commit details
    Browse the repository at this point in the history

  20. Updated name of set schema 2 function call in csv reader

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    61adf5b View commit details
    Browse the repository at this point in the history

  21. updated set_schema_2 in metadata to create nested ordered dict

    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    d9cd059 View commit details
    Browse the repository at this point in the history

  22. Extraneous columnns expected in csv dictionary in test function are r… 

    …emoved
    Vedant P Iyer committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    7fe1964 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2024

  1. Logger overwrites not appends to output file

    @Vedant1
    Vedant1 authored Nov 16, 2024
    Configuration menu
    Copy the full SHA
    ec9b5a0 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2024

  1. fixed description

    @jpulidojr
    jpulidojr committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    123c851 View commit details
    Browse the repository at this point in the history