Failure in exception handling in current_thread_executor.py _WorkItem.run()

[ShaheedHaque]

I'm not sure of the exact role that asgiref plays in the Django stack, and specifically how that might vary between a "debug" setup where Django itself - IIUC - serves the incoming requests, versus a "deployment" setup where, in my case, Django is fronted by nginx and gunicorn/uvicorn, so please forgive any inaccuracies in the following description.

I have a Django view whose Jinja template ends up expanding a link to a static URL, like this:

     const html = `{{ static(some-directory) }}/a-subdirectory/foo.html`;

In my "debug" setup, this works as expected, but it seems in my "deployment" setup the static(some-directory) ends up throwing the following exception:

   ValueError("Missing staticfiles manifest entry for 'some-directory'")

That in itself might be a minor issue, but unfortunately, when that bubbles up to the exception handling starting at

asgiref/asgiref/current_thread_executor.py

Line 42 in 8cf847a

self.future.set_exception(exc)

we end up spinning on some Linux futex (according to strace) consuming 100% of the CPU until forcibly SIGKILLd. The stack at this point looks like this:

Traceback (most recent call last):,
  File "/home/ubuntu/venv/lib/python3.10/site-packages/asgiref/sync.py", line 534, in thread_handler,
    raise exc_info[1],
  File "/home/ubuntu/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 42, in inner,
    response = await get_response(request),
  File "/home/ubuntu/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async,
    response = await wrapped_callback(,
  File "/home/ubuntu/venv/lib/python3.10/site-packages/asgiref/sync.py", line 479, in __call__,
    ret: _R = await loop.run_in_executor(,
  File "/home/ubuntu/venv/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run,
    result = self.fn(*self.args, **self.kwargs),
  File "/home/ubuntu/venv/lib/python3.10/site-packages/asgiref/sync.py", line 538, in thread_handler,
    return func(*args, **kwargs),
  File "/home/ubuntu/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 104, in view,
    return self.dispatch(request, *args, **kwargs),
  File "... our code ...", line 255, in dispatch,
    raise RuntimeError("about to return: generic_views.py[255]"),
RuntimeError: about to return: generic_views.py[255],

I'm inclined to described this as a serious failure because:

• short of inserting a print("exception=", exc) there was no way to see what the problem was.
• the 100% CPU spin is indefinite in duration.

I'm filing this against asgiref without really knowing if the actual error is in one of the other related parties at this point. I am of course, happy to help diagnose the problem further as needed.

Encls: pip freeze output

$ pip freeze
aiohttp==3.8.6
aiohttp-retry==2.8.3
aiosignal==1.3.1
alembic==1.12.0
amqp==5.1.1
anyio==4.0.0
asgiref==3.7.2
asttokens==2.4.1
async-timeout==4.0.3
attrs==23.1.0
Authlib==1.2.1
autobahn==23.6.2
Automat==22.10.0
azure-common==1.1.28
azure-core==1.29.4
azure-identity==1.13.0
azure-mgmt-core==1.4.0
azure-mgmt-rdbms==10.1.0
azure-mgmt-resource==23.0.1
azure-mgmt-subscription==3.1.1
Babel==2.13.0
backcall==0.2.0
bcrypt==4.0.1
bidict==0.22.1
billiard==4.1.0
blinker==1.6.3
boto3==1.28.62
botocore==1.31.62
Brotli==1.1.0
cachetools==5.3.1
calendra==7.7.1
carehare==1.0.2
celery==5.3.4
certifi==2023.7.22
cffi==1.16.0
channels==4.0.0
channels-rabbitmq==4.0.1
chardet==5.2.0
charset-normalizer==3.3.0
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
colorama==0.4.6
constantly==15.1.0
convertdate==2.4.0
coverage==7.3.2
cryptography==41.0.4
cssselect==1.2.0
daphne==4.0.0
datadiff==2.2.0
decorator==5.1.1
deepdiff==6.6.0
defusedxml==0.8.0rc2
Deprecated==1.2.14
diff-match-patch==20230430
Django==4.2.6
django-annoying==0.10.6
django-axes==6.1.1
django-csp==3.7
django-excel==0.0.10
django-extra-views==0.14.0
django-filter==23.3
django-formtools==2.4.1
django-import-export==3.3.1
django-ipware==5.0.1
django-jinja==2.11.0
django-jinja-bootstrap-form==4.5.0
django-jsonstore==0.5.1
django-model-utils==4.3.1
django-oauth-toolkit==2.3.0
django-phonenumber-field==7.2.0
django-polymorphic==3.1.0
django-queryable-properties==1.8.4
django-viewflow==1.11.0
djangorestframework==3.14.0
dnspython==2.4.2
drf-spectacular==0.26.5
drf-spectacular-sidecar==2023.10.1
ecdsa==0.18.0
email-validator==2.0.0.post2
et-xmlfile==1.1.0
eventlet==0.33.3
exceptiongroup==1.1.3
executing==2.0.0
Flask==2.2.5
flask-babel==3.1.0
Flask-Compress==1.14
Flask-Gravatar==0.5.0
Flask-Login==0.6.2
Flask-Mail==0.9.1
Flask-Migrate==4.0.5
Flask-Paranoid==0.3.0
Flask-Principal==0.4.0
Flask-Security-Too==5.1.2
Flask-SocketIO==5.3.6
Flask-SQLAlchemy==3.0.5
Flask-WTF==1.1.1
frozenlist==1.4.0
google-api-core==2.12.0
google-api-python-client==2.102.0
google-auth==2.23.3
google-auth-httplib2==0.1.1
google-auth-oauthlib==1.0.0
googleapis-common-protos==1.60.0
greenlet==1.1.2
gunicorn==21.2.0
h11==0.14.0
httpagentparser==1.9.5
httplib2==0.22.0
httptools==0.6.0
hyperlink==21.0.0
idna==3.4
importlib-metadata==6.8.0
incremental==22.10.0
inflection==0.5.1
ipython==8.16.1
isodate==0.6.1
itsdangerous==2.1.2
jaraco.classes==3.3.0
jedi==0.19.1
jeepney==0.8.0
Jinja2==3.1.2
jmespath==1.0.1
jpy @ https://github.com/jpy-consortium/jpy/archive/refs/heads/master.tar.gz#sha256=213ef9a4371136013205390d604aa51b299fce406f5645de7ef077f41803f80d
JPype1==1.4.1
jsonschema==4.19.1
jsonschema-specifications==2023.7.1
jwcrypto==1.5.0
keyring==23.13.1
kombu==5.3.2
ldap3==2.9.1
lml==0.1.0
lunardate==0.2.0
lxml==4.9.3
Mako==1.2.4
MarkupPy==1.14
MarkupSafe==2.1.3
matplotlib-inline==0.1.6
more-itertools==10.1.0
msal==1.24.1
msal-extensions==1.0.0
msgpack==1.0.7
msrest==0.7.1
multicorn @ file:///home/ubuntu/multicorn2-main
multidict==6.0.4
netifaces==0.11.0
networkx==3.1
oauthlib==3.2.2
odfpy==1.4.1
openpyxl==3.0.10
ordered-set==4.1.0
packaging==23.2
paiyroll @ file:///home/ubuntu/repo/source
paiyroll-fdw @ file:///home/ubuntu/repo/source/paiyroll/fdw
pamqp==3.2.1
paramiko==3.3.1
parso==0.8.3
passlib==1.7.4
pdfkit==1.0.0
pexpect==4.8.0
pgadmin4==7.7
phonenumbers==8.13.22
pickleshare==0.7.5
pika==1.3.2
Pillow==9.5.0
platformdirs==3.11.0
portalocker==2.8.2
prompt-toolkit==3.0.39
protobuf==4.24.4
psutil==5.9.5
psycopg==3.1.9
psycopg-binary==3.1.9
psycopg2-binary==2.9.9
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pyexcel==0.7.0
pyexcel-io==0.6.6
pyexcel-ods==0.6.0
pyexcel-webio==0.1.4
pyexcel-xls==0.7.0
pyexcel-xlsx==0.6.0
pyexcel-xlsxw==0.6.1
Pygments==2.16.1
PyJWT==2.8.0
pyluach==2.2.0
PyMeeus==0.5.12
PyNaCl==1.5.0
pyOpenSSL==23.2.0
pyotp==2.9.0
pyparsing==3.1.1
pypng==0.20220715.0
pyquery==2.0.0
pyrabbit==1.1.0
python-consul2==0.1.5
python-dateutil==2.8.2
python-dotenv==1.0.0
python-engineio==4.7.1
python-jose==3.3.0
python-socketio==5.9.0
python3-openid==3.2.0
python3-saml==1.16.0
pytz==2023.3.post1
PyYAML==6.0.1
pyzstd==0.15.9
qrcode==7.4.2
referencing==0.30.2
requests==2.31.0
requests-file==1.5.1
requests-oauthlib==1.3.1
requests-toolbelt==1.0.0
rpds-py==0.10.4
rsa==4.9
s3transfer==0.7.0
SecretStorage==3.3.3
service-identity==23.1.0
setproctitle==1.3.3
simple-websocket==1.0.0
six==1.16.0
sniffio==1.3.0
social-auth-app-django==5.3.0
social-auth-core==4.4.2
speaklater3==1.4
SQLAlchemy==2.0.21
sqlparse==0.4.4
sshtunnel==0.4.0
stack-data==0.6.3
tablib==3.5.0
tblib==2.0.0
terminaltables==3.1.10
texttable==1.7.0
traitlets==5.12.0
twilio==8.9.1
Twisted==23.8.0
txaio==23.1.1
typing_extensions==4.8.0
tzdata==2023.3
ua-parser==0.18.0
uk-postcode-utils==1.1
ukmodulus==0.1.2
uritemplate==4.1.1
urllib3==1.26.17
user-agents==2.2.0
uvicorn==0.23.2
uvloop==0.17.0
vine==5.0.0
watchfiles==0.20.0
wcwidth==0.2.8
websockets==11.0.3
Werkzeug==2.2.3
workalendar==17.0.0
wrapt==1.15.0
wsproto==1.2.0
WTForms==3.0.1
xdg==6.0.0
xlrd==2.0.1
XlsxWriter==3.1.7
xlwt==1.3.0
xmlsec==1.3.13
xmltodict==0.13.0
yarl==1.9.2
zeep==4.2.1
zipp==3.17.0
zope.interface==6.1

[ShaheedHaque]

I just realised that this only seems to happen for exceptions thrown during template rendering. AFAIK from using the IDE debugger, template rendering does not happen on the same thread that "normal" Django view code code uses. I therefore suspect that there is some unexpected interaction between the thread-sensitive code in asgiref around this point, and the thread that does template rendering.