Proposal for Multi-Tenancy support using namespaces.

[venkag]

Background

There were two discussions (here and here) on this topic. While the general idea is the same, I would like to add more details in this discussion.

Proposal

Support for multi-tenancy is planned as part of the Roadmap and this proposal is a step towards realizing the plan.

There are two main problems to solve w.r.t multi-tenancy. They are:

1. Tenant Isolation.
2. Fairness (variations: Noisy Neighbor, Resource limits etc.)

This proposal focuses only on Tenant Isolation and does not discuss Fairness.

In general, this is a complicated topic and it touches a lot of modules. Things to consider:

• The API requests need to be identified to which tenant they belong to (REST). Information can be carried in:
  • Header field (e.g. x-tenant-id:<tenant-id>).
  • Path parameter (/<tenant-id>/<rest-of-the-path>).
  • Query parameter (/path?x-tenant-id=<tenant-id>)
• Objects/Entities stored in the database need to be identified to which tenant they belong to (DAO).
  • Can use additional column to identify each object/entity within the database - Column based isolation.
  • Can use different connections to different databases based on tenant - Database based isolation.
  • Can use different schemas within a database based on tenant - Schema based isolation.
• All actions that happens within conductor should also be aware of the tenant context (Conductor Internal).
  • For example, when a workflow definition is registered, check for task definition constraint check should also be performed within the tenant context.

Workers can be designed to be tenant aware or unaware. This is where fairness comes into picture. In this proposal all scheduled tasks are treated equal irrespective of which tenant they belong to.

To begin with most entities within conductor will have an additional field called namespace. These entities are: Task, TaskDef, WorkflowDef, WorkflowTask, Workflow. The namespace within the entities will help with the isolation. namespace is generic term and can be used for Tenant Isolation.

Just adding namespace to the classes is not enough, they have to be populated (set) appropriately and have to be looked up with the namespace context (get). The flow typically, is this:

Client --> REST --> Conductor Services --> DAO --> Database

• Conductor Services such as MetadataService, WorkflowService must be enhanced to include additional APIs with additional parameter namespace. To maintain backward compatibility, new interface methods with additional parameter will have default implementation to call the base method ignoring the namespace parameter.
• ExecutionDAO (and ExecutionDAOFacade), MetadataDAO must be enhanced to include additional APIs with additional parameter namespace. To maintain backward compatibility, new interface methods with additional parameter will have default implementation to call the base method ignoring the namespace parameter.
• PollDAO and QueueDAO will not be modified as they are not required to be tenant aware. Workers will not be tenant aware. All tasks will be treated equal.
• Some request classes such as RerunWorkflowRequest, StartWorkflowRequest will include additional field namespace.
• TaskMappers need to be enhanced to populate namespace when new tasks are created.

Implementation

We have an implementation that we are using internally that makes all the above changes excluding the changes required for REST module. Our implementation also has Postgres related enhancements and uses Column based isolation. For other database implementations, backward compatibility is maintained by ignoring the namespace field.

If you are OK with the above proposal, I can raise a PR with these changes.

@kishorebanala @apanicker-nflx @aravindanr
cc: @taojwmware

[venkag]

@apanicker-nflx @kishorebanala @aravindanr Any feedback on this proposal?

[aravindanr]

hello @venkag , we are reviewing it.

[venkag]

Thank you @aravindanr

[aravindanr]

@venkag Thanks for taking the time to write this proposal. Sorry for the delay in the response. We had several discussions regarding this proposal. As you have mentioned, this proposal mainly deals with data isolation between tenants and not fairness across tenants or within tenants. We do see that namespace could be useful when we consider fairness.

The roadmap item about tenant isolation is mainly geared towards fairness, avoiding noisy neighbors and providing the ability to set resource limits so any one or few workflows/tenants can use up all available resources.

We believe data isolation can be achieved by using a layer on top of Conductor and does not need to be part of the core offering. Other users have done similar things, an example is this talk https://www.youtube.com/watch?v=noVJ1owfTR0 from the recent meetup.

[venkag]

Thank you @aravindanr for the feedback!

[amoroziuk]

@aravindanr Hello, just looking for a way to implement multi-tenancy with a conductor, are there any recommendations on how to implement it? In the video you've mentioned I don't see any designs and/or solutions for this, just a generic mention of this feature.