Cancel Workflows not working

[pagrubel]

Cancelling workflow is not working. At one point we let running or schedule tasks complete and just did not schedule any new ones but now all tasks continue. When testing this, I ran beeflow-dag several times and it only archived the final one, so this will also need to be tested when cancel is fixed.

The workflow is not actually being cancelled, instead it actually runs to completion:
beeflow query 0a
Archived/Cancelled
clamr--RUNNING
ffmpeg--WAITING

$ beeflow query 0a
Archived/Cancelled
clamr--COMPLETED
ffmpeg--PENDING

$ beeflow query 0a
Archived
clamr--COMPLETED
ffmpeg--COMPLETED

[Leahh02]

I realized that whenever I cancel a workflow I always pause it beforehand, so I think maybe that's why I've only seen his be a problem once.

I looked into it, beeflow cancel uses conn.delete(_resource(long_wf_id), json={'option': 'cancel'}, timeout=60). In beeflow/wf_manager/resources/wf_actions.py the docstring for the delete method says that "For cancel, current tasks finish running." but that shouldn't mean that tasks that haven't started yet should start.

[pagrubel]

So, currently scheduled jobs should be allowed to complete, but no others should be scheduled. We have discussed an option to cancel all jobs too, but for sure tasks that are waiting should not run.

[Leahh02]

So, currently scheduled jobs should be allowed to complete, but no others should be scheduled. We have discussed an option to cancel all jobs too, but for sure tasks that are waiting should not run.

That makes sense

[pagrubel]

Oh and the Archived/Cancelled state should not change to Archived, and of course all dags that were done should be in the archive.