Production deployment crashes on source 879 and up

[jgonggrijp]

When viewing a source in the interface on the production server, one of two things can happen:

1. (as intended) the source is displayed and highlights are loaded;
2. (bug) source download stalls, after which the login popup appears. Attempting to reload the page first results in a 502 bad gateway and then in a 503 service unavailable.

As far as I was able to determine from limited (binary search) testing, behavior 1 occurs with all sources up to 878 and behavior 2 occurs with all sources from 879 upwards. This is a sizable chunk of all sources, as the sample sources list may produce source numbers at least as high as 1041.

The following traceback, which frequently appears in the logs, suggests that the /source/*/fulltext endpoint fails to load the text from Elasticsearch:

--> 2024-09-25 14:39:52,900 ERROR in /hum/web/gw-c-dh-read-it.im.hum.uu.nl/data/env/lib64/python3.8/site-packages/django/utils/log.py:224
Internal Server Error: /source/945/fulltext
Traceback (most recent call last):
  File "/hum/web/gw-c-dh-read-it.im.hum.uu.nl/data/env/lib64/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/hum/web/gw-c-dh-read-it.im.hum.uu.nl/data/env/lib64/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/hum/web/gw-c-dh-read-it.im.hum.uu.nl/data/source/backend/sources/views.py", line 283, in source_fulltext
    f = result['hits']['hits'][0]['_source']['text']
IndexError: list index out of range

which means it might have something to do with the recent migration to ES8. However, I have not yet checked whether these sources are present in the new index or not.

As @JeltevanBoheemen determined, the backend can be brought back to life by running python deploy.py --stop followed by python deploy.py --start. However, this appears to be stateful; I need to run --start twice in order to get it working again. This may or may not be a separate issue.

[jgonggrijp]

CC @BeritJanssen