Skip to content

Commit

Permalink
Added docs regarding the relevancy of soft shutdown and ETA tasks (ce…
Browse files Browse the repository at this point in the history
  • Loading branch information
Nusnus authored Sep 4, 2024
1 parent ae4a47f commit 11aef56
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 0 deletions.
8 changes: 8 additions & 0 deletions docs/userguide/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3311,6 +3311,14 @@ If the :setting:`worker_soft_shutdown_timeout` is set to a value greater than 0.
the :ref:`soft shutdown <worker-soft-shutdown>` anyways if there are no tasks running. This setting will
enable the soft shutdown even if there are no tasks running.

.. tip::

When the worker received ETA tasks, but the ETA has not been reached yet, and a shutdown is initiated,
the worker will **skip** the soft shutdown and initiate the cold shutdown immediately if there are no
tasks running. This may lead to failure in re-queueing the ETA tasks during worker teardown. To mitigate
this, enable this configuration to ensure the worker waits regadless, which gives enough time for a
graceful shutdown and successful re-queueing of the ETA tasks.

.. _conf-events:

Events
Expand Down
8 changes: 8 additions & 0 deletions docs/userguide/workers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,14 @@ behavior. To enable the soft shutdown, set :setting:`worker_soft_shutdown_timeou
The soft shutdown will be skipped if there are no tasks running. To force the soft shutdown, *also* enable the
:setting:`worker_enable_soft_shutdown_on_idle` setting.

.. warning::

If the worker is not running any task but has ETA tasks reserved, the soft shutdown will not be initiated
unless the :setting:`worker_enable_soft_shutdown_on_idle` setting is enabled, which may lead to task loss
during the cold shutdown. When using ETA tasks, it is recommended to enable the soft shutdown on idle.
Experiment which :setting:`worker_soft_shutdown_timeout` value works best for your setup to reduce the risk
of task loss to a minimum.

For example, when setting ``worker_soft_shutdown_timeout=3``, the worker will allow 3 seconds for all currently
executing tasks to finish before it terminates. If the time limit is reached, the worker will initiate a cold shutdown
and cancel all currently executing tasks.
Expand Down

0 comments on commit 11aef56

Please sign in to comment.