Should filters be part of spec?

[globalflea]

After some thinking, It feels strange to me that filters (state or action) expression is part of the specs as it means that business logic is divided between external functions (via REST, GraphQL, etc) and within the specs. Imagine if i modify a filter expression in the specs, i need to know the impact to called external functions since the inputs to these functions are now different?

Also, there's a need to work through the overall mutation to the messages as it flows into a state (state filters) to actions (action filters) then to external functions, subsequently again to action (action filters) and state (state filters) which couples the spec tightly with the biz logic - may be better if the spec just focus on control flow

I think it would be better if the spec were simpler and leave any filtering to the external functions if necessary. If there's a need to modify/filter a data before being processed by a function (biz logic), this can be achieved by adding another function (filter data) prior to calling the final function (biz logic)

[cdavernas]

I tend to disagree with you here, @globalflea. IMO, providing filtering at worklow level allows the exact opposite of what you are implying: it decouples business from the workflow. External services should return whatever data they deem fit in order to fulfill their goals, independently, and the workflow should just orchestrates, filters, aggregates, transforms, etc. data from different sources in order to achieve its own goals. Obviously, if data returned by external services changes, the orchestration part (aka the workflow) should change as well, or should at least be written in such a way it wouldn't be too much impacted by model changes.

[mswiderski]

fully agree with this statement - workflow definition without data definition is not really complete.

[tsurdilo]

@globalflea @cdavernas
Think i see both sides, let me know what you think:
For use cases where dsl is exposed to end users/customers i see the point of wanting to reduce the amount of expressions they can define to a minimum (or in some cases disable it at all). Expressions mistakes, especially as powerful as jq is can potentially lead to problems.
On the other hand, being able to manipulate workflow state (data) is really important in many other cases and many business problems could not be solved without it.

I'd like to look into a possible way to explicitly disable data filtering. Runtimes can do that on their own but having a way to define that in dsl might be good to have. I don't know if this should also include expression type functions, as really they can be used for filtering or even add/remove/restructure data as well. WDYT?

[ricardozanini]

I don't think we need to add a way to disable filtering or data transformation. It's a core part of the spec and the control flow.

[tsurdilo]

Imo control-flow (that's not based on data logic) can be expressed without filters (think just pure orchestration of steps to take, a pipeline for example).
I still think tho that the main request for this is really about exposing expression properties to end-user/customers that write the json/yaml. Being able to create a sub-set of our spec schema has been requested a number of times already so definitely we should take a look at this in future.

[mswiderski]

I agree with @ricardozanini . Control flow logic without data is not really useful outside of pure modeling use cases for documentation purpose and I think splitting the spec in such a way will be really damaging for the execution semantic. It's sort of lesson learned from other standards like BPMN where execution was an after thought while in DMN it was from the very beginning. So I believe having spec that allows to define executable workflows is a must for real interoperability between runtimes and that includes data and data manipulation.