Checkpoint/Restart Milestone

[rstyd]

Our next milestone is checkpoint/restart support.

Here is a summary:

1. Task manager (TM) detects task is interrupted before complete and reports to the workflow manager (WFM)
2. The workflow manager instructs the TM to restart the workflow with the correct restart inputs
3. If a task restart tries to use the same checkpoint file multiple times, report to user as possible error

Here is my current understanding on how this will work

• TM submits job to a WLM system (Slurm/LSF)
  • The submitted task will have a hint in CWL that enables checkpointing and includes a pattern for the checkpoint file(s)
• After the job finishes, the task manager will check it's state from the WLM. If the WLM reported an error, then the TM will look for the checkpoint files
• If no checkpoint files are found, it will report to the WFM that an unrecoverable error has occurred
• If it does find a checkpoint file, it will verify the checkpoint file is valid using any mechanism the developer provides, then either report as a correctable error if the checkpoint file is good or uncorrectable if not
• If an uncorrectable error occurs, WFM will inform the user
• The WFM will then have the TM submit a new job with the checkpoint file and checkpoint parameters
• The TM will keep track of checkpoint files. It will use some sort of hashing algorithm to ensure the same checkpoint file isn't used to rerun multiple times and that the most recent checkpoint file is always used.

[trandles-lanl]

NOTES

We need to have discussions with code teams about how they do checkpoints and restarts today.

• If TaskManager goes away (crash, DST, ?), pause workflow until TaskManager returns
  • when TaskManager returns, automatically restart the workflow
• TaskManager should monitor for checkpoint files and report the most recent checkpoint back to the WorkflowManager
  • /path/to/checkpoint/file
  • what other metadata?
    • timestamp?
    • file size?
• How to determine if a checkpoint file is good?
• Hints:
  • checkpoint Y/N
  • checkpoint file path
  • checkpoint file regex
  • (see "How to determine if a checkpoint file is good?")
  • restart parameters
• Restarts are a new task added to the DAG as they happen.