Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test kubernetes in CI #3482

Draft
wants to merge 76 commits into
base: master
Choose a base branch
from
Draft

Test kubernetes in CI #3482

wants to merge 76 commits into from
+184 −5

Commits on Jun 7, 2024

  1. Added options to set annotations and a service account in the Kuberne… 

    …tes worker pod configuration
    @shishichen
    shishichen committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    8e3d7cf View commit details
    Browse the repository at this point in the history

  2. Correct punctuation in debug message. hack out tests that won't fail-… 

    …as-expected as root
    @benclifford
    benclifford committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    45269ed View commit details
    Browse the repository at this point in the history

  3. Fix a couple of docstrings

    @benclifford
    benclifford committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    7ceec7a View commit details
    Browse the repository at this point in the history

  4. a bit of name sanitization for default pod names

    @benclifford
    benclifford committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    0cdece2 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2024

  1. fiddle with markings to deal with no shared fs and no staging 

    many of these newly marked "no shared fs" tests do not need to be shared fs, but instead could be staging-based?

it would be good to test the tests that aren't marked as staging required with no staging providers configured, even in normal CI, to validate that test mark is applied properly.

should also rewrite the mark name as shared_fs_required to match the X_required format?
    @benclifford
    benclifford committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    87d3454 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. add config file i've been using

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    954bad7 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'shishichen/add-k8s-pod-options'

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    d3e3828 View commit details
    Browse the repository at this point in the history

  3. add the dockerfile i've been using

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    17a00dd View commit details
    Browse the repository at this point in the history

  4. beginning of kubernetes-in-CI

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    62e0e36 View commit details
    Browse the repository at this point in the history

  5. push docker image? upgrade ubuntu

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    9086e19 View commit details
    Browse the repository at this point in the history

  6. fiddle with default name

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    26869e6 View commit details
    Browse the repository at this point in the history

  7. Add kubernetes, needed for submitting from inside a cluster

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    16c0a49 View commit details
    Browse the repository at this point in the history

  8. Add more bits for running everything in a kubernetes cluster

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    b03615a View commit details
    Browse the repository at this point in the history

  9. fix syntax error in github workflow definition

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    e122d19 View commit details
    Browse the repository at this point in the history

  10. Tighten timeout, add some debugging info at the end

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    ee14f6e View commit details
    Browse the repository at this point in the history

  11. Correct pod name from my test

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    120cf78 View commit details
    Browse the repository at this point in the history

  12. try to stop Job from recreating pod on failure, but instead abort fast

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    5c55fe6 View commit details
    Browse the repository at this point in the history

  13. Randomise test order to see if a test failure is specific to a partic… 

    …ular test, or something else
    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    b56dfd9 View commit details
    Browse the repository at this point in the history

  14. Merge branch 'master' into benc-k8s-kind-ci

    @benclifford
    benclifford authored Jun 10, 2024
    Configuration menu
    Copy the full SHA
    dd0f66c View commit details
    Browse the repository at this point in the history

  15. Add some memory logging

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    f8f5a27 View commit details
    Browse the repository at this point in the history

  16. Allocate more memory to workers

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    f4a7300 View commit details
    Browse the repository at this point in the history

  17. Add a staging_required marker that apparently wasn't breaking things … 

    …always in k8s
    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    ffdb021 View commit details
    Browse the repository at this point in the history

  18. messing with backoff limits and restart policy

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    21711e9 View commit details
    Browse the repository at this point in the history

  19. remove apparently invalid restart policy

    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    fb1733e View commit details
    Browse the repository at this point in the history

  20. Flush out some more staging_required tests (by setting storage_access… 

    …=[] in local htex test...)
    @benclifford
    benclifford committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    73b3e1d View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Switch the Kubernetes client call to read_namespaced_pod_status() to … 

    …read_namespaced_pod(), which is functionally the same but requires fewer permissions.
    @shishichen
    shishichen committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    31bc958 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. Fixed Kubernetes worker container launch command to remove trailing s… 

    …emicolon.
    @shishichen
    shishichen committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    8b39024 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Merge remote-tracking branch 'origin/master' into benc-k8s-kind-ci

    @benclifford
    benclifford committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    4538763 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'shishichen/fix-k8s-launch-cmd' into ben… 

    …c-k8s-kind-ci
    @benclifford
    benclifford committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    5f43aeb View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'shishichen/swap-k8s-pod-status' into be… 

    …nc-k8s-kind-ci
    @benclifford
    benclifford committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    299de99 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. Merge branch 'master' into benc-k8s-kind-ci

    @benclifford
    benclifford authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    68e3a5d View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. Merge branch 'master' into benc-k8s-kind-ci

    @benclifford
    benclifford authored Jun 24, 2024
    Configuration menu
    Copy the full SHA
    fe3c55e View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. Merge remote-tracking branch 'origin/master' into benc-k8s-kind-ci

    @benclifford
    benclifford committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    064b833 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/benc-k8s-kind-ci' into benc-k8s-… 

    …kind-ci
    @benclifford
    benclifford committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    9c6a04e View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into benc-k8s-kind-ci

    @benclifford
    benclifford authored Jul 2, 2024
    Configuration menu
    Copy the full SHA
    ba5f047 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2024

  1. Merge branch 'master' into benc-k8s-kind-ci

    @benclifford
    benclifford authored Jul 7, 2024
    Configuration menu
    Copy the full SHA
    69fbf03 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Merge remote-tracking branch 'origin/master' into benc-k8s-kind-ci

    @benclifford
    benclifford committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    75b7c02 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/benc-k8s-kind-ci' into benc-k8s-… 

    …kind-ci
    @benclifford
    benclifford committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    780fbb0 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Merge branch 'master' into benc-k8s-kind-ci

    @benclifford
    benclifford authored Aug 5, 2024
    Configuration menu
    Copy the full SHA
    2324744 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. function data in temp

    @colinthomas-z80
    colinthomas-z80 committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    b75a3ae View commit details
    Browse the repository at this point in the history

  2. use getpass for username

    @colinthomas-z80
    colinthomas-z80 committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    2c18d6c View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. use tempfile module

    @colinthomas-z80
    colinthomas-z80 committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    c201ec1 View commit details
    Browse the repository at this point in the history

  2. flake etc

    @colinthomas-z80
    colinthomas-z80 committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    9f6b037 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. Merge branch 'master' into tmp_function_data

    @benclifford
    benclifford authored Aug 21, 2024
    Configuration menu
    Copy the full SHA
    5ec7cdb View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Merge branch 'master' into benc-k8s-kind-ci

    @benclifford
    benclifford authored Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c3f6d45 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Merge branch 'master' into benc-k8s-kind-ci

    @benclifford
    benclifford authored Aug 26, 2024
    Configuration menu
    Copy the full SHA
    edf870f View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Merge branch 'master' into benc-k8s-kind-ci

    @benclifford
    benclifford authored Sep 3, 2024
    Configuration menu
    Copy the full SHA
    811b8e5 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Merge remote-tracking branch 'refs/remotes/origin/master' into benc-k… 

    …8s-kind-ci
    @benclifford
    benclifford committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    7347f64 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into tmp_function_data

    @benclifford
    benclifford authored Sep 4, 2024
    Configuration menu
    Copy the full SHA
    cd7229f View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Merge remote-tracking branch 'origin/master' into benc-k8s-kind-ci

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    08f8ce9 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'refs/remotes/origin/benc-k8s-kind-ci' i… 

    …nto benc-k8s-kind-ci
    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    5967f01 View commit details
    Browse the repository at this point in the history

  3. Build cctools and run a probably-broken taskvine vs kubernetes test c… 

    …onfig
    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    4938dbf View commit details
    Browse the repository at this point in the history

  4. fix repr in taskvine

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    98d7693 View commit details
    Browse the repository at this point in the history

  5. install cloudpickle explicitly for taskvine

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    dfc94a8 View commit details
    Browse the repository at this point in the history

  6. Add more time onto job timeout, because more is happening in job with… 

    … taskvine now
    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    47378f3 View commit details
    Browse the repository at this point in the history

  7. revert to 180s test time

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    43af8ef View commit details
    Browse the repository at this point in the history

  8. Log more to the console, kubernetes style

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    6a32f0f View commit details
    Browse the repository at this point in the history

  9. Note a (documentation?) bug in taskvine address selection

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    d4fab6a View commit details
    Browse the repository at this point in the history

  10. force hostname based address config, in line with comment in previous… 

    … commit
    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    21dcae6 View commit details
    Browse the repository at this point in the history

  11. now we're starting taskvine test successfully, give it time to complete

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    e1cce03 View commit details
    Browse the repository at this point in the history

  12. Make taskvine shutdown scale-in more like htex shutdown scale-in

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    4d4b4ba View commit details
    Browse the repository at this point in the history

  13. enable staging_required tests in taskvine, because taskvine might be … 

    …able to deal with staging those files properly (or it might be good to flush out more test problems)
    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    2e42e5c View commit details
    Browse the repository at this point in the history

  14. Output timestamps in kubernetes log to help diagnose hangs

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    3ba7e12 View commit details
    Browse the repository at this point in the history

  15. failed to get non-staging tests working, made a note in comments

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    084d797 View commit details
    Browse the repository at this point in the history

  16. correct duplicated 'and' in pytest -k option

    @benclifford
    benclifford committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    f34f2b8 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Merge remote-tracking branch 'colinthomas-z80/tmp_function_data' into… 

    … benc-k8s-kind-ci
    @benclifford
    benclifford committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    60a8611 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Add utils to sanitize strings for DNS compliance 

    These convert any string to a valid RFC 1123 DNS subdomain or label.
    @rjmello
    rjmello committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    1f09e5c View commit details
    Browse the repository at this point in the history

  2. Ensure k8s pod names/labels are RFC 1123 compliant 

    - Modified Kubernetes pod names and labels to conform to RFC 1123 for
DNS subdomain names and labels, ensuring compliance with Kubernetes
naming conventions.

- Replaced the trailing timestamp in the job name with an eight-character
hex string (job ID) to improve collision avoidance.

- Replaced `app` pod label with `parsl-job-id`.

- Updated container name to use job ID.
    @rjmello
    rjmello committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    83278c2 View commit details
    Browse the repository at this point in the history

  3. Use hex value for k8s job ID instead of pod name

    @rjmello
    rjmello committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    86ade32 View commit details
    Browse the repository at this point in the history

  4. Add tests for KubernetesProvider submit

    @rjmello
    rjmello committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    0c4d541 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into benc-k8s-kind-ci

    @benclifford
    benclifford committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    08693ab View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/rjmello-kube-pod-names' into ben… 

    …c-k8s-kind-ci, with conflicts
    @benclifford
    benclifford committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    415f780 View commit details
    Browse the repository at this point in the history

  7. Fix some bad merge

    @benclifford
    benclifford committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    c78defa View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'origin/master' into benc-k8s-kind-ci

    @benclifford
    benclifford committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    54ea143 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2024

  1. Merge remote-tracking branch 'origin/master' into benc-k8s-kind-ci

    @benclifford
    benclifford committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    535289f View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. Merge branch 'master' into benc-k8s-kind-ci

    @benclifford
    benclifford authored Nov 1, 2024
    Configuration menu
    Copy the full SHA
    fd26ddd View commit details
    Browse the repository at this point in the history