Separate the specs into 2 - control & data

[globalflea]

I was thinking that maybe the specs should be broken into 2 separate layers. The first is the Control layer (or Core layer) which deals strictly with control flow only, no data specification nor manipulation - specifically, no need json, just any type that can be serialized and sent over the wire; also no filters, no jq (or other) expressions etc. This layer will deal with various states flows, errors and transition. Implementations can be certified for control layer alone.
On top of this Control layer, there can be the additional Data manipulation layer. Here json is required, also openapi, etc support. Here filter jq (& other) expression is supported. Implementations can also be certified at this 'higher' level
This may make adopting the specs easier as it's a stage process - conform to Control layer first, then later add on Data layer. Or for simpler engines just the control layer.
Not sure if it's possible, just a thought.

[tsurdilo]

Yes completely agree that we do need to come up with conformance levels.
Typically you see things like "Modeling Conformance" and "Execution Conformance"
but it seems that for our case we really want to do this in much more details and have a better plan.

It's not only about simplicity of adoption but as DSLs are often used as means for customers/end-users to define orchestrations, I think we should look into much deeper subsets (even maybe on a state level) as far as comformance goes.

Ontop of my head i can think of subset of things like "events", "functions" and "state-level" conformances. wdyt?

We could structure our json schemas that define the dsl in ways that can fit the different conformance levels.

[tsurdilo]

I am all for setting up dedicated community meetings for this topic as its really something that we have on the radar for a long time, but were never able to get started.
@cdavernas @mswiderski @ricardozanini @antmendoza
are you guys willing to help out on this ?

[globalflea]

Imho, I am happy with the clear segregation of the control flow layer. Think there needs to be a balance - the spec shouldn't be to complex nor convoluted.

[tsurdilo]

again agree :) the issue is that the dsl the spec describes has to also be usable be able to address most business problems.
i think atm we are tending to be on the more "complex" side of things, and being able to get to a point where people that only have the need to use a subset of it, should be able to do it and still claim conformance. So yeah let's put this as a high priority for our project and start on it.

[tsurdilo]

@globalflea as of right now there is nothing that would exclude any runtime that implements the spec partially to like not be able to receive help or for some reason be "excluded" from claiming it uses serverless workflow specification. One thing we are doing atm is focusing on fixing existing issues people raise over new feature development. This will allow runtimes to "catch up" as well as us to start focusing on things like TCK and this.

[mswiderski]

are you guys willing to help out on this ?

Sign me in!

In general I would not split the spec for the sake of conformance. I'd prefer the spec to be as complete as possible when it comes to workflow as such. In the end I don't see option to run workflows without data manipulation being described.

Conformance levels are a different thing where saying given implementation conforms to certain part of it is completely fine. But to make it proper we would have to have TCK to actually check the conformance level which would be of huge value on its own.
But yeah, we need to start somewhere and starting on control flow logic as a first part of it might be a good place.

[ricardozanini]

I tend to agree with @mswiderski. Data manipulation is too important. I won't split either. But I agree that conformance levels are a must. As an example, we have the Operators SDK:

You can be in conformance in an early stage (let's say no data filtering or transformation), and a limited set of states, or no timeouts, retries, etc.

I kinda "broke" the sections of the spec in this doc to show the conformance stage of our implementation. We can start thinking about something like this, maybe?

That would be easier to explain the spec for newcomers. :)