PostgresMetadataDAO taskDefCache considerations and distributed inconsistency

[bfrggit]

PostgresMetadataDAO: https://github.com/Netflix/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresMetadataDAO.java

Note that Ln 41 creates a concurrent hash map of in-memory cache of task definitions.

We are trying to build a stateless microservice on top of Netflix Conductor. The existence of this cache breaks statelessness of the service and causes inconsistency on distributed cluster -- Consider that we have 2 instances of our service using the same PostgreSQL DB as the persistence. IF a delete call is sent to instance 1, then there is no way that the deleted task def can be ever deleted from instance 2.

Note that Ln 100, the deletion from cache only happens if the task def is successfully deleted from DB. Also, even if this is not the issue, it is still possible that when I delete the task def from instance 1 and DB, a read operation of the same task def can still return the deleted entity from instance 2 cache.

Any thoughts? Are other DB persistence metadata DAOs doing the same?

We'd like to understand the considerations behind having this cache. Why is it necessary and what is the potential impact of disabling/circumventing it. Thanks!

[aravindanr]

@rickfish @mactaggart if you have any thoughts about the cache in PostgresMetadataDAO, please share.

@bfrggit Other implementations also have a cache. Netflix uses the CassandraMetadataDAO which also has a cache. The cache exists so Conductor does not overload Cassandra with read queries. The cache works for our usage pattern. TaskDef updates are rare and our users don't expect TaskDef changes to take effect immediately.

For your use-case, I see 2 options,

1. Extend PostgresMetadataDAO and supply your implementation for getTaskDef() method that does not use the cache.
2. Modify PostgresMetadataDAO and add a configuration property that disables the cache.

[bfrggit]

I ended up with something similar to option 1. Instead of overriding getTaskDef() I used reflection to replace taskDefCache to a dummy implementation so nothing can be stored in the cache.

Do you think Conductor itself can add some flexibility in the code to

• Allow option 2? OR
• Add another constructor to allow construction of the PostgresMetadataDAO with a custom map (i.e., ? extends ConcurrentHashMap<String, TaskDef>) as the cache? OR
• Add a protected setter for the cache member?

Any other thoughts than overriding the cache or the DAO?