DAG Enhancements

[Leahh02]

Brainstorm of ways to improve the beeflow dag command.

• Be able to make multiple versions of the workflow dag without overriding older versions. This should be similar to how beeflow config new works.

• Make an argument or option for beeflow dag to specify where to save the dags to. Maybe this could be a dags directory in the user's workdir.

• Some kind of watch command to get a live view (discussion) Added discussion Dags giphy possiblity #944

• Save the final dag in the workflow archive

• Make the "depends on" arrows bold

• "beeflow core pull-deps" should pull neo4j and add the plugin - both ci and pull-deps should use a dockerfile in beeflow/data/dockerfiles/. maybe Dockerfile.neo4j also modify configuration for new dockerfile see issue Fix "beeflow core pull-deps" to use a Dockerfile to pull the neo4j container and add the apoc file #943

• Figure out what to do with the graphmls directory

• If when archiving the workflow and a dag does not exist, make one (this will happen for failed or if we decide to archive cancelled workflows)

• Should we use the long wf_id?

• Check to make sure there's a png in the archive

[pagrubel]

Do we want to use the entire wf_id for the png file, or maybe the wf_id-dag as a directory and store the multiple files in there?

[pagrubel]

Also, beeflow core pull-deps should build neo4j with the plugin using a dockerfile

[pagrubel]

Moving my recommendations up to the list.

[Leahh02]

Right now I'm making a directory that just has the graphmls after they've been exported by neo4j and I'm trying to figure out where they should go. Right now I have the graphmls directory inside the dags directory in .beeflow.

I'm going to move the dags to a user specified location, but I'm not sure that the graphmls should go there too because they don't really serve a big purpose for the person looking at them. They can be helpful in trouble shooting if something is wrong with making the graph, but that's it. I could maybe move them to the workflow archive.

[pagrubel]

Closing this as all the enhancements have either been done or a separate issue has been made.