Task resiliency if cancelled before any time has elapsed

[pagrubel]

There was a problem with a node failure as a job was being submitted. This caused an automatic cancellation of the job and task. So it was a system problem not a BEE problem. Should we check time elapsed and if it is zero resubmit the task.

Here is the example status of the job:

--------------- --------------- --------- --------------- ---------- ---------- ------------------- ------------------- ---------- 
       15926486 cat-5793969dbb+     kvats           cn213    general CANCELLED+ 2024-07-24T09:30:44 2024-07-24T09:30:44   00:00:00 
 15926486.batch           batch                     cn213             CANCELLED 2024-07-24T09:30:44 2024-07-24T09:30:44   00:00:00 
15926486.extern          extern                     cn213             CANCELLED 2024-07-24T09:30:44 2024-07-24T09:30:44   00:00:00 

FI I was able to get this information using the following commands:

export SACCT_FORMAT=jobid%15,jobname%15,user,nodelist,partition,state,start,end,elapsed
sacct -j 15925486

[pagrubel]

If the state is Cancelled and no time has elapsed, we should consider resubmitting the job with a limit as to how many times this is attempted

[pagrubel]

We are going to talk to the slurm experts about this.