Correlation

[dmarrazzo]

Correlation requires some extra care; in fact there is a common use case that cannot managed by the current state of the spec.
When a workflow starts a conversation with an external system, it happens that the external system define its own correlation id and it expects to use it for all the following interactions; in fact, those external systems are not designed to accept a correlation id provided by the workflow (e.g. workflow instance id).

Furthermore, those interactions are asynchronous: the external system emits a message with that specific correlation id, so that message must be routed to workflow instance who initiated the communication.

sequenceDiagram
    autonumber
    workflow ->>external system: new interaction
    external system-->>workflow: new correlation id
    activate external system
    external system->>workflow: deferred response (new correlation id)

Loading

The previous use case cannot be modeled with current spec capabilities, because it's not possible to define a new correlation id (context attribute) based on the function response.
A key aspect is that the correlation ID value can't be calculated at the beginning of the workflow, it's the result of a service call that happens at any time during the workflow lifecycle.
A possible solution could be the introduction of a new tag to define a new correlation id that can be used to resume the workflow instance.
E.g.

- name: New interaction
  type: operation
  actions:
    - name: New interaction
      functionRef:
        refName: call
      correlationDataFilter:
        results: '${ .sessionId }'
        toContextAttributeName: externalSystemID

Another important aspect is related to the communication protocol: the external system need to know also the address where to send the deferred response. For example in case of an http protocol along with the message 1, it should be included also the endpoint where to call back the workflow. Since, the workflow designer has no knowledge of underlying infrastructure, those information should be added by the engine, but spec users need to know where and how this information will be filled by the engine to rely on it.

[cdavernas]

Firstly, it seems like you're using a DSL version earlier than 1.0.0. If that's the case, please consider upgrading to version 1.0.0-alpha1 and later, as older versions are being discontinued.

---

Secondly, I'm not entirely sure I understand the challenges you're facing, as the specification is designed to support dynamic and externally defined correlation IDs.

Here's how your example can be modeled:

1. Storing a Dynamic Correlation ID

After a task completes, you can store the response in the workflow context using the export feature:

- newInteraction:
    call: http
    with:
      method: get
      endpoint: https://someendpoint.com
    export:
      as: '$context + { externalSystemId: .result.correlationId }'

2. Correlating Future Events:

When listening for events, you can correlate them based on this stored value:

- consumeEvent:
    listen:
      to:
        one:
          with:
            type: com.example.type.v1
            source: https://example.source.com
          correlate:
            - myDynamicKey:
                from: '$context.externalSystemId'

---

Another important aspect is related to the communication protocol: the external system need to know also the address where to send the deferred response

Currently, there is no built-in way to support webhook-style callbacks, as it would require workflow runtimes to handle per-instance servers for all protocols.

To implement similar functionality, you should rely on CloudEvents, which facilitate asynchronous processes effectively.

[dmarrazzo]

Thank you @cdavernas this addresses my concern.
As you notices, I'm currently using version 0.8 of the spec. I gave a quick look to the latest spec 1.0.0 which looks quite revolutionary compared to the previous version, but, since the spec document does not emphasize the correlation concept very much, I thought that it was still a challenge.

[cdavernas]

@dmarrazzo You are welcome, I'm glad I could help 😉

I gave a quick look to the latest spec 1.0.0 which looks quite revolutionary compared to the previous version

Yes, the 1.0.0 is indeed a leap, and addresses many of the limitations that were hindering the spec, amongst which (but not limited to) fault handling, portable extensions, custom functions, external catalogs, container processes, ...

the spec document does not emphasize the correlation concept very much

You are right, we probably should emphasize/document correlation more in the spec document, possibly by adding a dedicated section and related samples. I'll work on that in the upcoming days, thanks for the input!