[Bug Report] Frequent freezes during training

[rfurko-tt]

Describe the bug
Running training with GPT-2S results in frequent freezes after ~300k samples.
Training restart also requires tt-smi -r 0
Switch blocking from false to true removes the issue: tt::tt_metal::EnqueueProgram(queue, program, false);

To Reproduce

1. Branch: rfurko/kahan_summation
2. Run tt-train/nano-gpt example
3. Wait for 500-2000 steps
4. Freeze

Expected behavior
No freezes and crashes

Screenshots
One thread callstack here:

All other threads callstack:

Please complete the following environment information:

• OS: Ubuntu 20.04
• Version of software: rfurko/kahan_summation

[dmakoviichuk-tt]

@davorchap we are not convinced that it is related to the di/dt. My expectation didt would happen anyway if we have blocking EnqueueProgram.So this is pretty annoying issue.
@tt-asaigal feel free to ping Roman if you need any help to reproduce this issue.

[rfurko-tt]

In the shared branch we forced matmul to use sublocks (1, 1) to minimize possibility di/dt. It didn't change frequency of freezes.

[tt-asaigal]

I took a look at the branch, and its fairly out of sync with latest main. A fairly non-deterministic hang was exposed by this commit pushed on Nov 8, for which a workaround went in yesterday.
The branch has the hanging commit but not the workaround. Would it be possible to rebase on latest main and try the workload again?

[dmakoviichuk-tt]

@tt-asaigal could you show a link to the workaround?

[tt-asaigal]

please try cherry-picking these 2 in order:
9fff3ce
3036495

[rfurko-tt]

I've rebased to latest main and can confirm that freezing is not an issue now. Thanks @tt-asaigal!