Metadata storage on workflows

[NickTomlin]

Since we are moving in the direction of schemas for workflows with #2722 (specifically ui/src/codegen/JSONSchemaWorkflow.js) it might be useful to explore a pattern that would allow defining additional metadata on workflows and tasks.

{
  "name": "ntomlin.example",
  "tasks": [ /* ... /*],
  "metadata": {
     "private": false,
     "tags": ["data_warehouse", "some_team", "long_running"],
     "schema": { ref: "http://path-to-json/schema" } 
  }
 }
}

Use cases

• Providing input / output schema (e.g. the infra for Providing a schema language for workflows and tasks #2500)
• Tagging / categorizing workflows
  • Functionality (e.g. the use case)
  • Permissions
  • Visibility (e.g. is this a generic workflow/task that anyone can use?)
• Providing custom links/debugging information for tooling

A lot of these could be implemented as strongly typed first class fields within the framework, but I think metadata would allow an implementation in user space first that could then be consolidated into the core implementation once we find good patterns.

Potential implementations

• A metadata field on workflows, tasks, and maybe events would be a Map<String, Object> capable of holding this.
• A metadata pointer to storage that can be queried via a separate API (or a flag)

Potential concerns

Size

Metadata could bloat the size of workflow definitions if we always housed it in-line.

Complexity

If conductor isn't strictly managing this "user space" blob it could introduce potential breakages if conductor were to alter the shape.

Since this is purely a "user land" object, users might think a bug caused by metadata changes is the fault of conductor when its not.

[aravindanr]

In principle, conductor team is open to this idea of supporting a "metadata" or more specifically user-provided opaque key value pairs that are only meaningful to workflow authors. However, the implementation could be easily lead to troubles if proper structure is not enforced.

To reduce the complexity and mitigate the concerns about size of the metadata, we think that "metadata" should be a Map<String, String> that has restrictions on the number of pairs and length of the key and value. For e.g: At most 5 pairs, key can be up-to 256 chars and value up-to 512 chars long. These are not the final numbers, just an example to illustrate the point. Discussions need to happen about how this metadata is presented to users.

1. Does TaskDef inherit the metadata from WorkflowDef?
2. Do workers get the metadata when a task is polled automatically?
3. Should there be a separate api to read metadata?
   ...

Also, we think that tags should be a top level field in the definition that Conductor understands. To begin with, tags could be useful for grouping workflow executions and searching for them in the UI.

[NickTomlin]

👍 on tags being a higher level concept. I think that makes a ton of sense!

In terms of limiting what metadata should be, are there any tests we could perform to see what the impact on conductor would be?

Map<String, String> with a limit may be pretty restrictive in terms of tooling. Anything more than a simple key / value pair would probably require a JSON blob to be stuffed into a key and de-serialized or having values simply point to external stores and leaving everything up to clients. I understand that we wouldn't want metadata to impact performance though, so I wonder if metrics could help find a happy medium.

In terms of the questions you outlined:

Does TaskDef inherit the metadata from WorkflowDef?

At the moment, I think that there shouldn't be inheritance here. I'd imagine tasks have their own metadata. Since tasks could potentially be used by multiple workflows, inheritance may not actually be ideal. Perhaps tasks could just look up the metadata of their parent workflow if they really need it manually? Or there could be a convenience function to do that call?

Do workers get the metadata when a task is polled automatically?

I personally don't have a use case for this, and i'd prefer to have the workflow be able to reference it's metadata and pass that as inputParameters to the task. Having a reference that the task can look up would help provide a bridge to the workflow metadata if it was needed.

Should there be a separate api to read metadata?

Probably :\ I hate splitting APIs up but otherwise workflow responses are going to be very| large if they include workflowDef, tasks, and metadata. Some users won't need it, others will always need it.

If I was thinking about this like a graphql API, i'd provide the ability to query all of this in a single call, but probably only provide some of it to workers / search and then look up the rest in a detailed view.

[marosmars]

A little bit of input:

+1 for tags being top level concept

Should there be a separate api to read metadata?

There is a similar case in the API here where workflow executions can be retrieved with or without additional information : includeTasks or includeClosed. So metadata could be handled in a similar way, being optionally added to the response based on a query param (excluded by default).

[aravindanr]

In terms of limiting what metadata should be, are there any tests we could perform to see what the impact on conductor would be?

There are currently no tests. But an ideal test is to find the increase in memory footprint when a workflow definition with metadata is loaded into memory.

Map<String, String> with a limit may be pretty restrictive in terms of tooling.

Conductor must have some constraints on the size of the metadata field. Size constraints are easier to validate if it's Map<String,String>. If it's a Map<String,Object>, what do you propose as the constraints on that field?

[NickTomlin]

what do you propose as the constraints on that field?

This is a great question. I am not an expert in this area but my naive implementation would simplify stringify metadata and assert on byte size:

public static void processMetadata () throws JsonProcessingException {
     // This value should probably be driven by a property
    int MAX_METADATA_BYTES = 4000;
    ObjectMapper mapper = new ObjectMapper();

    // some JSON structure we've unpacked from metadata
    Map<String, Object> metadata = Map.of(
            "someKey", Map.of(
                    "lots", "of",
                    "key", "value",
                    "pairs", List.of("one", "two", "three")
            )
    );
  
    // this could also be done at per key/value basis if we wanted more fine-grained validation messages
    String s = mapper.writeValueAsString(metadata);
    byte[] bytes = s.getBytes();

    if (bytes.length > MAX_METADATA_BYTES) {
        throw new RuntimeException("Metadata exceeds max allowed value of " + MAX_METADATA_BYTES + " please see metadata docs https://some/link for more information");
    }
    
    // metadata is within limit
}

Since this would happen during the ingestion of metadata not at task time, I think the performance hit of re-serializing wouldn't actually be all that bad. It may be possible to do this via a customization of the de-serializer for metadata as well.

As for MAX_METADATA_BYTES we'd need to figure out how to actually calculate this.

Is there an upper bounds for tasks in workflowDefinition? That is probably the most similar collection in that it can theoretically have a very large set of deeply nested objects in it.

[boney9]

We just had a situation where metadata storage for workflows as well as workflow executions would have been super helpful. An example is to track the context on why the workflow instance was started.

Any thoughts on prioritizing this? We can help with the implementation if required.

[NickTomlin]

After some discussion between Orkes and Netflix folks we came to the conclusion that this is possible by using inputTemplate (which exists on both tasks and workflows and is available when retrieving a workflow definition) as a first iteration.

This would probably take the form of something like

{
  "taskDefinitions": [{}],
  "inputTemplate": {
     "$metadata": {
         "schema": { "$ref": "https://my-schema.com" },
         "private": true
     }
   }
}

If this needs to be a first class property we can re-open the proposal.