Skip to content

Nightly fast dispatch tests #6121

Nightly fast dispatch tests

Nightly fast dispatch tests #6121

Triggered via schedule November 28, 2024 20:00
Status Failure
Total duration 57m 23s
Artifacts 33
build-artifact  /  ...  /  build-docker-image
1m 12s
build-artifact / build-docker-image / build-docker-image
Matrix: build-artifact / build-artifact
Matrix: fd-nightly / fd-nightly
Matrix: fd-nightly / nightly-wh-models
Matrix: fd-nightly / nightly-wh-unstable-models
Fit to window
Zoom out
Zoom in

Annotations

14 errors, 2 warnings, and 192 notices
fd-nightly / [Unstable] Nightly N150 mamba 1
Process completed with exit code 1.
fd-nightly / [Unstable] Nightly N150 mamba 4
Process completed with exit code 1.
fd-nightly / FD WH N300 pgm dispatch nightly wormhole_b0
Process completed with exit code 1.
fd-nightly / Nightly N150 qwen
Process completed with exit code 1.
fd-nightly / [Unstable] Nightly N300 mamba 3
Process completed with exit code 1.
fd-nightly / [Unstable] Nightly N150 mamba 3
Process completed with exit code 1.
fd-nightly / [Unstable] Nightly N300 mamba 2
The self-hosted runner: tt-metal-ci-vm-87 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
fd-nightly / Nightly N150 functional_unet
The self-hosted runner: tt-metal-ci-vm-79 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
pcie-cards-are-being-used-cleanup
Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.
fd-nightly / Nightly N150 functional_unet
The action 'Run frequent reg tests scripts' has timed out after 30 minutes.
fd-nightly / Nightly N300 functional_unet
The self-hosted runner: tt-metal-ci-vm-43 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
pcie-cards-are-being-used-cleanup
Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.
fd-nightly / Nightly N300 functional_unet
The action 'Run frequent reg tests scripts' has timed out after 30 minutes.
fd-nightly / [Unstable] Nightly N150 mamba 2
Process completed with exit code 1.
fd-nightly / FD WH N300 pgm dispatch nightly wormhole_b0
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
fd-nightly / Nightly N150 qwen
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
printing-out-smi-info-cleanup
Touching and printing out SMI info
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 77 %
disk-usage-after-startup
Disk usage is 77 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 59 %
disk-usage-after-startup
Disk usage is 59 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 85 %
disk-usage-after-startup
Disk usage is 85 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 58 %
disk-usage-after-startup
Disk usage is 58 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 73 %
disk-usage-after-startup
Disk usage is 73 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 74 %
disk-usage-after-startup
Disk usage is 74 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 80 %
disk-usage-after-startup
Disk usage is 80 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 84 %
disk-usage-after-startup
Disk usage is 84 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 85 %
disk-usage-after-startup
Disk usage is 85 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 84 %
disk-usage-after-startup
Disk usage is 84 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 80 %
disk-usage-after-startup
Disk usage is 80 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 86 %
disk-usage-after-startup
Disk usage is 86 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 63 %
disk-usage-after-startup
Disk usage is 63 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 84 %
disk-usage-after-startup
Disk usage is 84 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 65 %
disk-usage-after-startup
Disk usage is 65 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 85 %
disk-usage-after-startup
Disk usage is 85 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 75 %
disk-usage-after-startup
Disk usage is 75 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 75 %
disk-usage-after-startup
Disk usage is 75 %
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 56 %
disk-usage-after-startup
Disk usage is 56 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 88 %
disk-usage-after-startup
Disk usage is 88 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 76 %
disk-usage-after-startup
Disk usage is 76 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 88 %
disk-usage-after-startup
Disk usage is 88 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 83 %
disk-usage-after-startup
Disk usage is 83 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 80 %
disk-usage-after-startup
Disk usage is 80 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 92 %
disk-usage-after-startup
Disk usage is 92 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 71 %
disk-usage-after-startup
Disk usage is 71 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
disk-usage-before-startup
Disk usage is 86 %
disk-usage-after-startup
Disk usage is 86 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
disk-usage-before-startup
Disk usage is 84 %
disk-usage-after-startup
Disk usage is 84 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 52 %
disk-usage-after-startup
Disk usage is 52 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful

Artifacts

Produced during runtime
Name Size
TTMetal_build_grayskull
301 MB
TTMetal_build_wormhole_b0
302 MB
test_reports_05b28bac-b99e-4188-b42e-2445c444b564
710 Bytes
test_reports_069b8b51-b57c-416f-8ba8-17b3c6c3a131
656 Bytes
test_reports_0940ae6a-767f-47c7-a5e9-10eb4c76cbdc
484 Bytes
test_reports_112efc38-c640-4608-8696-c543dddb9f29
838 Bytes
test_reports_1c2da2fa-5cc9-485f-ae11-ca0665199ed0
2.67 KB
test_reports_1cf46525-7d90-461a-85b9-be837051d00f
6.17 KB
test_reports_1d8e7b09-6dfd-42b1-8190-57fa1f7e5f9f
741 Bytes
test_reports_3796a5c2-ff08-40f8-b1c9-48ff07c1d0b3
1.32 KB
test_reports_3d2e8620-bd58-4311-a4de-fce0b628ab78
548 Bytes
test_reports_3f79d21a-7833-4b73-a327-2d694ab5517d
4.59 KB
test_reports_4bc544f9-064a-49aa-889e-e2e562602927
430 Bytes
test_reports_5d00e1d4-25fc-4ec2-bf56-ddfad2e5bc5f
833 Bytes
test_reports_62c081e8-369b-4ace-b457-4acd1b98cbdb
431 Bytes
test_reports_65660834-0ac2-4d8c-8f7b-2920583aab32
2.66 KB
test_reports_7b316c9f-ea75-4572-af8d-542b8d1e0297
653 Bytes
test_reports_7b9ba3b8-f82f-4c85-a1e2-70e567addf3b
720 Bytes
test_reports_84069b5f-3c35-437b-b458-bdad863815de
626 Bytes
test_reports_84270a7b-f7de-4276-8df6-283190b403ac
3.5 KB
test_reports_a189e267-3f70-42e5-8b85-2d64cc1f1e3b
6.18 KB
test_reports_a4a64afa-d3fa-4321-9623-775826060a67
480 Bytes
test_reports_abdeec1b-16e7-4732-9cd8-468cf7812657
1.37 KB
test_reports_b47dd4af-3e54-4314-be3b-3acae66b57ab
550 Bytes
test_reports_b4bd80a2-f314-450f-8b64-b366bde7c3f2
6.66 KB
test_reports_b54b7935-2409-4313-b0a4-f0815a29d882
624 Bytes
test_reports_c7750ff5-6df9-4d43-a53f-dd7e006afc57
623 Bytes
test_reports_ce2b3e2e-8403-4400-8f92-fd7b82172299
609 Bytes
test_reports_d1fd416b-82aa-4013-a5a5-b587ed4c4f31
609 Bytes
test_reports_e35d9934-36ba-4562-bf62-9bb97a634ed0
1.31 KB
test_reports_ef0fcea3-b22c-464a-b6e5-5a75c5396b9f
629 Bytes
test_reports_f6c1cb21-bfb3-4d34-89dd-cda192f123dd
4.55 KB
test_reports_f6ee80f8-2c55-4410-931a-8f7c2f208e33
477 Bytes