Update percentage completion or any metadata for a task

[ChaitanyaTessell]

I am using the conductor version v3.0.6.

I wanted to update the percentage complete for a task and some other data (in the output). If I call conductor's update task api from my worker with output as percentage value during the execution of the task, the conductor actually puts the task again in the queue to be polled. I just wanted to updated some output in the update task call and nothing else, the worker should just update the task and continue its execution.
Do we have any such functionality in conductor?

[kishorebanala]

@ChaitanyaTessell Task logs is a better place to do this. This is append only, compared to over-writing task output.

Updating the task releases the worker lease, and goes through another cycle of poll and execute on a different worker. Renewing the task lease on same worker and continuing the execution is not supported today. Should you be interested, feel free to open a discussion with your proposal.

[naik91]

Agreed on the part that non contexual stuff can be and should be put in the logs.
I have one more usecase for this. As I mentioned that I have long running tasks, where I will do multiple steps inside a single task, I want to store some data at the certain checkpoints inside a task, so that in case of retries I should be able to resume by accessing the same data again. Is that something that is possible or any way in which we can implement it?

[kishorebanala]

@naik91 In the long running task case, Is the task using callBackAfterSeconds to update and poll later? For eg., if the worker thinks it'll need to execute later, it can update Conductor with IN_PROGRESS state and some duration of callBackAfterSeconds. The task will then be available for poll after this duration, where another worker (say worker-2) will continue from where initial worker (say worker-1) left off. In this case, you can put any additional data in task output while updating and the subsequent execution can use it, if available.

However, if the worker is long running, and does not update Conductor with callBackAfterSeconds, thereby not giving up the lease, I think it's best to store any task specific state elsewhere than in Conductor.

[naik91]

Thanks for your response.
The worker itself is long running and I will need that data in case of cleanup workflow (failure workflow scenarios) and possibly in retries for the failed tasks. So, storing the context of workflow execution in the workflow itself makes sense for me.
Let me know your thoughts on this.

[kishorebanala]

@naik91 State updates outside of task lifecycle could get trickier. For eg., if the worker dies before the state update reaches Conductor, Conductor wouldn't be able to guarantee correctness of this state.

As I mentioned that I have long running tasks, where I will do multiple steps inside a single task..

Would it be possible to split each of these steps into a separate task, consider using DO WHILE?

[naik91]

Got it.
Do while will not work as the steps are not the same and I want to keep them in a single task as it is logically a single task.
I think I got your point on data inconsistency as it would be something like a 2 phase commit (step execution + task data save), so saved data in task might not be always present.
Will think on the solve where it can be stored at the execution end.
Thanks for your responses.