Dynomite/Redis nodes always run out of memory after some time, resulting in PoolOfflineException

[arketec]

We are having the hardest time with this issue. Here is our setup:

conductor-community server (3.12) and UI run inside docker containers, using docker compose
3-node Dynomite/Redis cluster hosted on 3 separate machines
Elasticsearch 7 (cloud hosted) using the conductor-community es7-persistance library

The VMs running Dynomite each have 16gb RAM to use, and after around 4-5 weeks, this RAM runs out and Dynomite stops responding to conductor:

PoolOfflineException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down, hashtag=null, password=null], latency=0(0), attempts=0]host pool is offline and no Racks available for fallback

This does not recover upon reload, even after restarting/clearing the Dynomite RAM. We have to fully bring everything down (Dynomite/Redis, Conductor server) and then back up again to get things working again.

This is not ideal, as when this occurs, all the historical workflow data is lost.

attempts to solve this with configs:
maxmemory-policy allkeys-lru (redis)
conductor.workflow-status-listener.type=archive (conductor)

here are the configs:

Conductor
docker-compose.yaml
config-hosted-production.properties

Dynomite/Redis
node1.yml
node2.yml
node3.yml

redis.conf

maxmemory 7gb
maxmemory-policy allkeys-lru
replica-ignore-maxmemory no