Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TTNN eltwise_logical_xor sweep test execution hangs (GS & WH) #10144

Closed
Tracked by #13795
dveizovicTT opened this issue Jul 11, 2024 · 2 comments
Closed
Tracked by #13795

TTNN eltwise_logical_xor sweep test execution hangs (GS & WH) #10144

dveizovicTT opened this issue Jul 11, 2024 · 2 comments
Assignees
Labels
bug Something isn't working GS op_cat: eltwise WH

Comments

@dveizovicTT
Copy link

Execution of eltwise_logical_xor sweep test hangs after first sweep combination.

Steps to reproduce the behavior:
Run sweep test using the following command:

pytest tests/tt_eager/python_api_testing/sweep_tests/run_sweep_test.py --input-path tests/tt_eager/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_working/wormhole/pytorch_eltwise_logical_xor.yaml --input-method cli --cli-input eltwise_logical_xor_output

Expected behavior
Test should run for the combination of parameters:

Running with shape: [[4, 7, 32, 96], [4, 7, 32, 96]] and seed: 17155532
2024-07-11 09:07:02.730 | DEBUG | tests.tt_eager.python_api_testing.sweep_tests.common:run_tt_lib_test:42 - Running with args: {'dtype': [<DataType.BFLOAT16: 0>, <DataType.BFLOAT16: 0>], 'layout': [<Layout.TILE: 1>, <Layout.TILE: 1>], 'input_mem_config': [MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::DRAM,shard_spec=std::nullopt), MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::DRAM,shard_spec=std::nullopt)], 'output_mem_config': MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::DRAM,shard_spec=std::nullopt)}

And it should be passing.
After the first sweep combination which is passed, test is running for:

tests.tt_eager.python_api_testing.sweep_tests.run_pytorch_test:run_sweep_test:231 - Running with shape: [[4, 7, 32, 96], [4, 7, 32, 96]] and seed: 16305027
2024-07-11 09:07:06.323 | DEBUG | tests.tt_eager.python_api_testing.sweep_tests.common:run_tt_lib_test:42 - Running with args: {'dtype': [<DataType.BFLOAT16: 0>, <DataType.BFLOAT16: 0>], 'layout': [<Layout.TILE: 1>, <Layout.TILE: 1>], 'input_mem_config': [MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::DRAM,shard_spec=std::nullopt), MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::L1,shard_spec=std::nullopt)], 'output_mem_config': MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::DRAM,shard_spec=std::nullopt)}

At this point, the test execution hangs.
The output result file should be available in the output directory listed in command (in this case eltwise_logical_xor_output)
The resulted csv file will contain only 1 line (the first sweep combination)

Additional context
There is unit test available for the sweep combination which hangs the execution.
Checkout branch dveizovic/sweep-tests-unit-set9
Run unit test:
pytest tests/ttnn/python_api_testing/non_working_unit_tests/wormhole/test_logical_xor.py
However the unit test is not able to reproduce the hang.

@KalaivaniMCW
Copy link
Contributor

KalaivaniMCW commented Oct 25, 2024

As mentioned the unit test is passing without the error
image

The sweep test for the same parameters mentioned also pass - in the new sweep framework
image

the test files are in branch kalaivani/bug_test_files

@KalaivaniMCW
Copy link
Contributor

Closing since tests are passing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working GS op_cat: eltwise WH
Projects
None yet
Development

No branches or pull requests

3 participants