Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[not for merge] tidying threads and file descriptors at shutdown #3397

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from
Draft

[not for merge] tidying threads and file descriptors at shutdown #3397

wants to merge 7 commits into from
+78 −19

Commits on Nov 29, 2024

  1. log fd counts

    @benclifford
    benclifford committed Nov 29, 2024
    Configuration menu
    Copy the full SHA
    9d3d3cc View commit details
    Browse the repository at this point in the history

  2. tmp: log threads

    @benclifford
    benclifford committed Nov 29, 2024
    Configuration menu
    Copy the full SHA
    0ac0775 View commit details
    Browse the repository at this point in the history

  3. hack tv diskspace for hang

    @benclifford
    benclifford committed Nov 29, 2024
    Configuration menu
    Copy the full SHA
    b857453 View commit details
    Browse the repository at this point in the history

  4. Prototype htex result queue shutdown 

    Before this PR, this thread stays running forever, until it is terminated
by Python process exit.

After this PR, this thread polls an flag once per second, then properly
closes the ZMQ socket that it is using.

pytest parsl/tests/test_monitoring/ --config local
Before this PR, at end of test: 32 threads, 451 fds open.

After this PR, at end of test: 1 thread, 48 fds open.
    @benclifford
    benclifford committed Nov 29, 2024
    Configuration menu
    Copy the full SHA
    0964f5a View commit details
    Browse the repository at this point in the history

  5. assert on number of threads left running after each test in config lo… 

    …cal test
    @benclifford
    benclifford committed Nov 29, 2024
    Configuration menu
    Copy the full SHA
    a839e03 View commit details
    Browse the repository at this point in the history

  6. radical mpi test leaves lots of threads: 

    parsl/tests/test_radical/test_mpi_funcs.py

...


            logger.error(f"BENC: end open fds: {end_fds} (vs start {start_fds}")
            logger.error(f"BENC: end threads: {threading.active_count()}")
    
>           assert threading.active_count() == 1, "test left threads running: " + repr(threading.enumerate())
E           AssertionError: test left threads running: [<_MainThread(MainThread, started 140532448974656)>, <Thread(Thread-18 (
_work), started daemon 140531866793664)>, <Thread(Thread-19 (_run_proxy), started daemon 140531858400960)>, <Thread(Thread-20 (
_monitor), started daemon 140531850008256)>, <Thread(Thread-21 (_work), started daemon 140531841615552)>, <Thread(Thread-23 (_w
atch), started daemon 140531229243072)>, <Thread(Thread-24 (_listener), started daemon 140531220850368)>, <Thread(Thread-25 (_l
istener), started daemon 140530985985728)>, <Thread(Thread-26 (_listener), started daemon 140530977593024)>, <Thread(Thread-27 
(_listener), started daemon 140530969200320)>, <Thread(Thread-28 (_listener), started daemon 140530960807616)>, <Thread(Thread-
29 (_listener), started daemon 140530952414912)>, <Thread(Thread-31 (_listener), started daemon 140530390398656)>, <Thread(Thre
ad-32 (_listener), started daemon 140530365220544)>, <Thread(Thread-33 (_listener), started daemon 140530113570496)>, <Thread(T
hread-35 (_listener), started daemon 140530096785088)>, <Thread(Thread-36 (_listener), started daemon 140530071606976)>, <Threa
d(Thread-38 (_listener), started daemon 140529627023040)>, <Thread(Thread-39 (_listener), started daemon 140529618630336)>, <Th
read(Thread-40 (_bulk_collector), started daemon 140529610237632)>]
E           assert 20 == 1
E            +  where 20 = <function active_count at 0x7fd041fcfec0>()
E            +    where <function active_count at 0x7fd041fcfec0> = threading.active_count
    @benclifford
    benclifford committed Nov 29, 2024
    Configuration menu
    Copy the full SHA
    72c8dd5 View commit details
    Browse the repository at this point in the history

  7. count fds

    @benclifford
    benclifford committed Nov 29, 2024
    Configuration menu
    Copy the full SHA
    254d8ba View commit details
    Browse the repository at this point in the history