Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add capability of copying files from lustre to s3 bucket on AWS #2873

Closed
wants to merge 86 commits into from
Closed

Add capability of copying files from lustre to s3 bucket on AWS #2873

wants to merge 86 commits into from

Commits on Jun 18, 2024

  1. compiled OK now

    @weihuang-jedi
    weihuang-jedi committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    103f2c4 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jun 19, 2024
    Configuration menu
    Copy the full SHA
    916ff6c View commit details
    Browse the repository at this point in the history

  2. re-test on aws with fewer changes

    @weihuang-jedi
    weihuang-jedi committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    b0ac406 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. make change in tasks.py to avoid error finding libiomp5.so problem

    @weihuang-jedi
    weihuang-jedi committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    3de972f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jun 21, 2024
    Configuration menu
    Copy the full SHA
    8308375 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2024

  1. add comments so the reviewers know that these changes are for AWS, an… 

    …d hope can provide a better way
    @weihuang-jedi
    weihuang-jedi committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    bc4c4a8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'aws-forecast-only' of ssh://github.com/NOAA-EPIC/global… 

    …-workflow-cloud into aws-forecast-only
    @weihuang-jedi
    weihuang-jedi committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    924aede View commit details
    Browse the repository at this point in the history

  3. add comments so the reviewers know that these changes are for AWS, an… 

    …d hope can provide a better way
    @weihuang-jedi
    weihuang-jedi committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    b724937 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. reverse config.resource changes, and memory restriction on AWS

    @weihuang-jedi
    weihuang-jedi committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    12ab29f View commit details
    Browse the repository at this point in the history

  2. sync with emc repo

    @weihuang-jedi
    weihuang-jedi committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    adff250 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. move common data to a shared place

    @weihuang-jedi
    weihuang-jedi committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    2290ea2 View commit details
    Browse the repository at this point in the history

  2. use ICs from s3-bucket

    @weihuang-jedi
    weihuang-jedi committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    cd2c8e7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jun 26, 2024
    Configuration menu
    Copy the full SHA
    4e144e5 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. change as suggested by reviewer

    @weihuang-jedi
    weihuang-jedi committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    46e3ef5 View commit details
    Browse the repository at this point in the history

  2. sync with develop

    @weihuang-jedi
    weihuang-jedi committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    32f13eb View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. sync sorc/ufs_model.fd

    @weihuang-jedi
    weihuang-jedi committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    a34a4c8 View commit details
    Browse the repository at this point in the history

  2. remove mpmd_opt from APRUN_UFS

    @weihuang-jedi
    weihuang-jedi committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    44011a3 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. mpmd_opt and switch off tracker/genesis default for AWS

    @weihuang-jedi
    weihuang-jedi committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    965ec80 View commit details
    Browse the repository at this point in the history

  2. add TODO

    @weihuang-jedi
    weihuang-jedi committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    3ce268e View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2024

  1. remove ncl version on AWS

    @weihuang-jedi
    weihuang-jedi committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    f03ac78 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    007a56b View commit details
    Browse the repository at this point in the history

  3. sync ufs_model

    @weihuang-jedi
    weihuang-jedi committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    2f6ec6e View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 10, 2024
    Configuration menu
    Copy the full SHA
    dba83a7 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 12, 2024
    Configuration menu
    Copy the full SHA
    24fe804 View commit details
    Browse the repository at this point in the history

  2. sync and remove gempak from noaacloud

    @weihuang-jedi
    weihuang-jedi committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    e8a2e0f View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 15, 2024
    Configuration menu
    Copy the full SHA
    4013eb1 View commit details
    Browse the repository at this point in the history

  2. update modules hash

    @weihuang-jedi
    weihuang-jedi committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    a548c7f View commit details
    Browse the repository at this point in the history

  3. update module hash

    @weihuang-jedi
    weihuang-jedi committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    d37e646 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. use bucket

    @weihuang-jedi
    weihuang-jedi committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    2a80162 View commit details
    Browse the repository at this point in the history

  2. remove /scratch1, but kept TODO

    @weihuang-jedi
    weihuang-jedi committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    fa44862 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 17, 2024
    Configuration menu
    Copy the full SHA
    55c7e7e View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. re-sync

    @weihuang-jedi
    weihuang-jedi committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    07851dc View commit details
    Browse the repository at this point in the history

  2. sync

    @weihuang-jedi
    weihuang-jedi committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    492808d View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. add is_exclusive to resource.AWSPW

    @weihuang-jedi
    weihuang-jedi committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    d7a262e View commit details
    Browse the repository at this point in the history

  2. sync hash with EMC repo

    @weihuang-jedi
    weihuang-jedi committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    af573af View commit details
    Browse the repository at this point in the history

  3. remove --export=ALL from native, when is_exclusive set true

    @weihuang-jedi
    weihuang-jedi committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    0929180 View commit details
    Browse the repository at this point in the history

  4. sync

    @weihuang-jedi
    weihuang-jedi committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    06fecca View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 25, 2024
    Configuration menu
    Copy the full SHA
    d8783ab View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    d22bc6d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'aws-forecast-only' of ssh://github.com/NOAA-EPIC/global… 

    …-workflow-cloud into aws-forecast-only
    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    a5c441f View commit details
    Browse the repository at this point in the history

  4. Make AWS works similar to on-prem machine

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    77e8233 View commit details
    Browse the repository at this point in the history

  5. remove --export=ALL from 'native'

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    96f73ba View commit details
    Browse the repository at this point in the history

  6. remove --export=ALL from 'native'

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    a33a3be View commit details
    Browse the repository at this point in the history

  7. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 25, 2024
    Configuration menu
    Copy the full SHA
    80b294b View commit details
    Browse the repository at this point in the history

  8. add py-f90nml to noaacloud modulefile

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    01a8928 View commit details
    Browse the repository at this point in the history

  9. remove un-necessary added lines

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    b035947 View commit details
    Browse the repository at this point in the history

  10. remove un-necessary added lines

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    bf3b460 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. remove added lines which was originally for AWS, but should be define… 

    …d in hosts/awspw.yaml
    @weihuang-jedi
    weihuang-jedi committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    47627ff View commit details
    Browse the repository at this point in the history

  2. restore as develop

    @weihuang-jedi
    weihuang-jedi committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    7bf8900 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. try to fix pynorms error

    @weihuang-jedi
    weihuang-jedi committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    0685a8f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 29, 2024
    Configuration menu
    Copy the full SHA
    381403d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'aws-forecast-only' of ssh://github.com/NOAA-EPIC/global… 

    …-workflow-cloud into aws-forecast-only
    @weihuang-jedi
    weihuang-jedi committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    0e71f7d View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 30, 2024
    Configuration menu
    Copy the full SHA
    2024835 View commit details
    Browse the repository at this point in the history

  2. sync with EMC repo

    @weihuang-jedi
    weihuang-jedi committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    2c52016 View commit details
    Browse the repository at this point in the history

  3. sync Gaea link with EMC repo, and only include blocks/packs that run … 

    …on AWS
    @weihuang-jedi
    weihuang-jedi committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    cd6c541 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'aws-forecast-only' of github.com:NOAA-EPIC/global-workf… 

    …low-cloud into aws-forecast-only
    @weihuang-jedi
    weihuang-jedi committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    1f60ed0 View commit details
    Browse the repository at this point in the history

  5. merge fro develop

    @weihuang-jedi
    weihuang-jedi committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    e1a57b4 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Remove ACCOUNT_SERVICE

    @weihuang-jedi
    weihuang-jedi committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    fe9a457 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Aug 1, 2024
    Configuration menu
    Copy the full SHA
    5c6e052 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Merge branch 'develop' of ssh://github.com/NOAA-EPIC/global-workflow-… 

    …cloud into aws-atmc768
    @weihuang-jedi
    weihuang-jedi committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    1a273ae View commit details
    Browse the repository at this point in the history

  2. update AWSPW.env for coupled model

    @weihuang-jedi
    weihuang-jedi committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    cc0ae02 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Merge remote-tracking branch 'origin/develop' into aws-coupled-atm-oc… 

    …n-ic3-c192
    @weihuang-jedi
    weihuang-jedi committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    7bf88dd View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. Merge remote-tracking branch 'origin/develop' into aws-coupled-atm-oc… 

    …n-ic3-c192
    @weihuang-jedi
    weihuang-jedi committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    c429225 View commit details
    Browse the repository at this point in the history

  2. add C768

    @weihuang-jedi
    weihuang-jedi committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    c2b765d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'NOAA-EMC:develop' into aws-coupled-atm-ocn-ic3-c192

    @weihuang-jedi
    weihuang-jedi authored Aug 8, 2024
    Configuration menu
    Copy the full SHA
    199e8be View commit details
    Browse the repository at this point in the history

  4. make c48s2sw run, but have issue with gfswavepostpnt

    @weihuang-jedi
    weihuang-jedi committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    e9734a9 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. sync

    @weihuang-jedi
    weihuang-jedi committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    a96dcca View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. synce with develop

    @weihuang-jedi
    weihuang-jedi committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    1920240 View commit details
    Browse the repository at this point in the history

  2. turn off wave on CSPs

    @weihuang-jedi
    weihuang-jedi committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    0fafa6b View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. sync with develop

    @weihuang-jedi
    weihuang-jedi committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    3256945 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'NOAA-EMC:develop' into aws-coupled-atm-ocn-ic3-c192

    @weihuang-jedi
    weihuang-jedi authored Aug 23, 2024
    Configuration menu
    Copy the full SHA
    aa43bcb View commit details
    Browse the repository at this point in the history

  3. Merge branch 'NOAA-EMC:develop' into aws-coupled-atm-ocn-ic3-c192

    @weihuang-jedi
    weihuang-jedi authored Aug 23, 2024
    Configuration menu
    Copy the full SHA
    398d40d View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. sync with develop

    @weihuang-jedi
    weihuang-jedi committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    1ca7d1f View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. revert change to develop

    @weihuang-jedi
    weihuang-jedi committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    fd90543 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'NOAA-EMC:develop' into aws-coupled-atm-ocn-ic3-c192

    @weihuang-jedi
    weihuang-jedi authored Aug 27, 2024
    Configuration menu
    Copy the full SHA
    219117f View commit details
    Browse the repository at this point in the history

  3. revert and sync to develop

    @weihuang-jedi
    weihuang-jedi committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    3911fd2 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'aws-coupled-atm-ocn-ic3-c192' of github.com:NOAA-EPIC/g… 

    …lobal-workflow-cloud into aws-coupled-atm-ocn-ic3-c192
    @weihuang-jedi
    weihuang-jedi committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    fb1b45d View commit details
    Browse the repository at this point in the history

  5. create branch aws-copy2bucket

    @weihuang-jedi
    weihuang-jedi committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    c9a8a6a View commit details
    Browse the repository at this point in the history

  6. testing scripts/exglobal_atmos_products.sh

    @weihuang-jedi
    weihuang-jedi committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    4e347d0 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. trying to copy over ocean-ice products

    @weihuang-jedi
    weihuang-jedi committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    3c4d1f9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'develop' of github.com:NOAA-EPIC/global-workflow-cloud … 

    …into aws-copy2bucket
    @weihuang-jedi
    weihuang-jedi committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    c322afa View commit details
    Browse the repository at this point in the history

  3. need to mkdir for ocean/ice first

    @weihuang-jedi
    weihuang-jedi committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    5dd063a View commit details
    Browse the repository at this point in the history

  4. Merge branch 'develop' of github.com:NOAA-EPIC/global-workflow-cloud … 

    …into aws-copy2bucket
    @weihuang-jedi
    weihuang-jedi committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    650ce7b View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. Use [[]] instead of [] as bash prefers the double square bracket

    @weihuang-jedi
    weihuang-jedi committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    a06f8d2 View commit details
    Browse the repository at this point in the history

  2. remove commented print

    @weihuang-jedi
    weihuang-jedi committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    c87bde0 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. remove changes from rocoto_viewer.py

    @weihuang-jedi
    weihuang-jedi committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    c63a193 View commit details
    Browse the repository at this point in the history