From 11aef56ee930ddf9b6b7ba7e943bea14276f6892 Mon Sep 17 00:00:00 2001 From: Tomer Nosrati Date: Wed, 4 Sep 2024 19:08:21 +0300 Subject: [PATCH] Added docs regarding the relevancy of soft shutdown and ETA tasks (#9238) --- docs/userguide/configuration.rst | 8 ++++++++ docs/userguide/workers.rst | 8 ++++++++ 2 files changed, 16 insertions(+) diff --git a/docs/userguide/configuration.rst b/docs/userguide/configuration.rst index eedd3d19d29..23b2974f34a 100644 --- a/docs/userguide/configuration.rst +++ b/docs/userguide/configuration.rst @@ -3311,6 +3311,14 @@ If the :setting:`worker_soft_shutdown_timeout` is set to a value greater than 0. the :ref:`soft shutdown ` anyways if there are no tasks running. This setting will enable the soft shutdown even if there are no tasks running. +.. tip:: + + When the worker received ETA tasks, but the ETA has not been reached yet, and a shutdown is initiated, + the worker will **skip** the soft shutdown and initiate the cold shutdown immediately if there are no + tasks running. This may lead to failure in re-queueing the ETA tasks during worker teardown. To mitigate + this, enable this configuration to ensure the worker waits regadless, which gives enough time for a + graceful shutdown and successful re-queueing of the ETA tasks. + .. _conf-events: Events diff --git a/docs/userguide/workers.rst b/docs/userguide/workers.rst index b613f97d50b..1f2cef97c83 100644 --- a/docs/userguide/workers.rst +++ b/docs/userguide/workers.rst @@ -166,6 +166,14 @@ behavior. To enable the soft shutdown, set :setting:`worker_soft_shutdown_timeou The soft shutdown will be skipped if there are no tasks running. To force the soft shutdown, *also* enable the :setting:`worker_enable_soft_shutdown_on_idle` setting. +.. warning:: + + If the worker is not running any task but has ETA tasks reserved, the soft shutdown will not be initiated + unless the :setting:`worker_enable_soft_shutdown_on_idle` setting is enabled, which may lead to task loss + during the cold shutdown. When using ETA tasks, it is recommended to enable the soft shutdown on idle. + Experiment which :setting:`worker_soft_shutdown_timeout` value works best for your setup to reduce the risk + of task loss to a minimum. + For example, when setting ``worker_soft_shutdown_timeout=3``, the worker will allow 3 seconds for all currently executing tasks to finish before it terminates. If the time limit is reached, the worker will initiate a cold shutdown and cancel all currently executing tasks.