{2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825

trz42 · 2024-11-21T20:26:38Z

Builds

magma/2.7.2-foss-2023a-CUDA-12.1.1
PyTorch/2.1.2-foss-2023a-CUDA-12.1.1

Superseedes #718

eessi-bot · 2024-11-21T20:26:42Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

riscv-eessi-io-bot · 2024-11-21T20:26:42Z

Instance eessi-bot-riscv is configured to build for:

architectures: riscv64/generic
repositories: riscv.eessi.io-20240402

riscv-eessi-io-bot · 2024-11-21T20:26:42Z

Instance eessi-bot-riscv is configured to build for:

architectures: riscv64/generic
repositories: riscv.eessi.io-20240402

eessi-bot · 2024-11-21T20:26:43Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-compat, eessi.io-2023.06-software

trz42 · 2024-11-21T20:27:02Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-21T20:27:05Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

riscv-eessi-io-bot · 2024-11-21T20:27:05Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-21T20:27:05Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 30162, for details & status see {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment)

eessi-bot · 2024-11-21T20:27:05Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-21T20:27:08Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30162

date	job status	comment
Nov 21 20:27:08 UTC 2024	submitted	job id `30162` awaits release by job manager
Nov 21 20:27:13 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 21 20:28:17 UTC 2024	running	job `30162` is running
Nov 21 22:32:31 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-30162.out` ✅ no message matching `FATAL:` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732226990.tar.gz` size: 302 MiB (316730812 bytes) entries: 113 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `magma/2.7.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 no other files in tarball
Nov 21 22:32:31 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 433.113 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 443.086 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.85 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.52 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 9.31 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.44 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.31 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7756.73 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7744.15 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30162.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

some /tmp/eb-m470fsqz/eb-r9zeygi0/tmpb1cr2ofk/rpath_wrappers/gxx_wrapper/g++ run failed with

/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld: warning: libcupti.so.12, needed by lib/libtorch_cpu.so, not found (try using -rpath or -rpath-link)
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld: lib/libtorch_cpu.so: undefined reference to `cuptiGetResultString@libcupti.so.12'
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld: lib/libtorch_cpu.so: undefined reference to `cuptiMetricEnumEvents@libcupti.so.12'
...
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld: lib/libtorch_cpu.so: undefined reference to `cuptiSubscribe@libcupti.so.12'
collect2: error: ld returned 1 exit status

we should be able to fix this by adding the directory that contains libcupti to $LIBRARY_PATH in a pre_configure hook (see {2023.06}[foss/2023a] PyTorch v2.1.2 with CUDA/12.1.1 NorESSI/software-layer#369)

…PATH

trz42 · 2024-11-22T08:13:13Z

Build again after applying fix to find libcupti...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-22T08:13:16Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-22T08:13:16Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 30339, for details & status see {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment)

eessi-bot · 2024-11-22T08:13:16Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-22T08:13:19Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30339

date	job status	comment
Nov 22 08:13:18 UTC 2024	submitted	job id `30339` awaits release by job manager
Nov 22 08:13:44 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 22 08:19:50 UTC 2024	running	job `30339` is running
Nov 22 18:13:32 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-30339.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732297886.tar.gz` size: 509 MiB (534700273 bytes) entries: 12854 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `magma/2.7.2-foss-2023a-CUDA-12.1.1` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 22 18:13:32 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 436.522 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 446.803 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.57 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.69 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.67 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 9.33 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.28 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.3 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7868.44 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7743.76 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30339.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2024-11-22T18:21:18Z

Also build for zen3…

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-22T18:21:21Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-22T18:21:21Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 accelerator:nvidia/cc80 resulted in:
- submitted job 30343, for details & status see {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment)

riscv-eessi-io-bot · 2024-11-22T18:21:21Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-22T18:21:21Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-22T18:21:25Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen3 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30343

date	job status	comment
Nov 22 18:21:24 UTC 2024	submitted	job id `30343` awaits release by job manager
Nov 22 18:21:35 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 22 18:27:37 UTC 2024	running	job `30343` is running
Nov 23 02:24:10 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-30343.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1732327591.tar.gz` size: 509 MiB (534708669 bytes) entries: 12854 modules under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/software `magma/2.7.2-foss-2023a-CUDA-12.1.1` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 23 02:24:10 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen3-node+default P: perf: 522.848 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen3-node+default P: perf: 532.462 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 2.42 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 2.32 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 5.5 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 5.43 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 0.24 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 0.22 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen3-node+default P: bandwidth: 14283.69 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen3-node+default P: bandwidth: 14294.01 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30343.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

boegel · 2024-11-22T20:23:07Z

eb_hooks.py

+    if self.name == 'PyTorch' and self.version == '2.1.2':
+        if 'cudaver' in self.cfg.template_values and self.cfg.template_values['cudaver'] == '12.1.1':


I don't see a reason to make this specific to a particular PyTorch version or CUDA version?

We only know that the failure happens in this specific case. If we apply it to other cases, we will not know whether it was necessary or not.

boegel · 2024-11-22T20:24:08Z

eb_hooks.py

+            _cupti_lib_dir = os.path.join(_eessi_software_path, 'software', 'CUDA', _cudaver, 'extras', 'CUPTI', 'lib64')
+            print_msg("pre_configure_hook_pytorch_add_cupti_libdir: cupti_lib_dir: '%s'", _cupti_lib_dir)
+            if _library_path:
+                env.setvar('LIBRARY_PATH', ':'.join([_library_path, _cupti_lib_dir]))


This seems like a bug in our CUDA installation/module, no?

I'm fine with proceeding like this for now, even if we also fix it somewhere else this won't cause trouble, but there's probably a more general fix for this?

Right, we might find a better solution by changing the CUDA module, eg, by adding the directory to LIBRARY_PATH through the module.

It could be a worthwhile effort to try.

@boegel could it be that lib_path (at least parts of it) is missing in https://github.com/easybuilders/easybuild-easyblocks/blob/57c0eaed8dc29e223fe68a75f7bf195cca0c2d04/easybuild/easyblocks/c/cuda.py#L362

A little before that line lib_path is constructed as list ['lib64', 'extras/CUPTI/lib64', 'nvvm/lib64'], but in line 362 only ['lib64', 'stubs/lib64'] is used.

trz42 · 2024-11-23T10:37:09Z

Try a different approach where we rebuild the CUDA module such that it prepends the directory containing the libcupti library to LIBRARY_PATH and then not using the hook used in the previous builds...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

eessi-bot · 2024-11-23T12:22:50Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

riscv-eessi-io-bot · 2024-11-23T12:22:50Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-23T12:22:53Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30525

date	job status	comment
Nov 23 12:22:52 UTC 2024	submitted	job id `30525` awaits release by job manager
Nov 23 12:23:54 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 23 12:25:01 UTC 2024	running	job `30525` is running
Nov 23 15:07:20 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-30525.out` ✅ no message matching `FATAL:` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732371817.tar.gz` size: 302 MiB (316742437 bytes) entries: 114 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `magma/2.7.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 23 15:07:20 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 436.385 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 446.555 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.65 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.6 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.8 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.56 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.31 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7790.17 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7770.53 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30525.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2024-11-23T19:07:07Z

New attempt to add prepend_path... to module file

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-23T19:07:11Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-23T19:07:11Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 30526, for details & status see {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment)

riscv-eessi-io-bot · 2024-11-23T19:07:11Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-23T19:07:11Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-23T19:07:14Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30526

date	job status	comment
Nov 23 19:07:13 UTC 2024	submitted	job id `30526` awaits release by job manager
Nov 23 19:07:39 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 23 19:12:42 UTC 2024	running	job `30526` is running
Nov 23 21:59:43 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-30526.out` ✅ no message matching `FATAL:` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732396533.tar.gz` size: 302 MiB (316747535 bytes) entries: 114 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `magma/2.7.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 23 21:59:43 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 424.221 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 439.051 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 6.32 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.7 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.8 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 9.04 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.34 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.34 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7750.78 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7786.2 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30526.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2024-11-23T23:42:38Z

Add missing double quotes...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

eessi-bot · 2024-11-23T23:42:40Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 30527, for details & status see {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment)

riscv-eessi-io-bot · 2024-11-23T23:42:40Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

riscv-eessi-io-bot · 2024-11-23T23:42:40Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-23T23:42:40Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-23T23:42:43Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30527

date	job status	comment
Nov 23 23:42:42 UTC 2024	submitted	job id `30527` awaits release by job manager
Nov 23 23:42:54 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 23 23:48:57 UTC 2024	running	job `30527` is running
Nov 24 09:52:50 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-30527.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732439991.tar.gz` size: 2577 MiB (2702411388 bytes) entries: 18370 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `CUDA/12.1.1.lua` `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `CUDA/12.1.1` `magma/2.7.2-foss-2023a-CUDA-12.1.1` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 24 09:52:50 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 432.645 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 433.129 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.58 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.71 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 9.06 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.56 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7791.5 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7872.26 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30527.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2024-11-24T10:23:19Z

Repeat build with updated easyblock (see easybuilders/easybuild-easyblocks#3516) instead of parse_hook...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-24T10:23:22Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-24T10:23:22Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 30704, for details & status see {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment)

eessi-bot · 2024-11-24T10:23:23Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-24T10:23:26Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30704

date	job status	comment
Nov 24 10:23:25 UTC 2024	submitted	job id `30704` awaits release by job manager
Nov 24 10:23:55 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 24 10:24:58 UTC 2024	running	job `30704` is running
Nov 24 20:26:31 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-30704.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732477966.tar.gz` size: 2577 MiB (2702394556 bytes) entries: 18370 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `CUDA/12.1.1.lua` `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `CUDA/12.1.1` `magma/2.7.2-foss-2023a-CUDA-12.1.1` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 24 20:26:31 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 435.332 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 433.519 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 5.09 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.3 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.84 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.48 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.31 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7812.55 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7795.21 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30704.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2024-11-25T13:58:52Z

Repeat build using the latest commit in easybuilders/easybuild-easyblocks#3516...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

eessi-bot · 2024-11-25T13:58:54Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 30882, for details & status see {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment)

riscv-eessi-io-bot · 2024-11-25T13:58:54Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

riscv-eessi-io-bot · 2024-11-25T13:58:54Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-25T13:58:55Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-25T13:58:58Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30882

date	job status	comment
Nov 25 13:58:57 UTC 2024	submitted	job id `30882` awaits release by job manager
Nov 25 13:59:46 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 25 14:05:49 UTC 2024	running	job `30882` is running
Nov 26 00:14:40 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-30882.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732578090.tar.gz` size: 2577 MiB (2702396428 bytes) entries: 18370 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `CUDA/12.1.1.lua` `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `CUDA/12.1.1` `magma/2.7.2-foss-2023a-CUDA-12.1.1` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 26 00:14:40 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 441.298 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 438.418 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.69 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.63 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.83 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.44 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.31 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7899.95 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7899.8 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30882.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

DEBUG {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1

e4f37fe

trz42 added 2023.06-software.eessi.io 2023.06 version of software.eessi.io accel:nvidia labels Nov 21, 2024

add pre_configure hook for PyTorch to add CUPTI lib64 dir to LIBRARY_…

4504675

…PATH

trz42 changed the title ~~DEBUG {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1~~ {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 Nov 22, 2024

boegel requested changes Nov 22, 2024

View reviewed changes

truib added 2 commits November 23, 2024 11:30

add cupti lib dir to LIBRARY_PATH in CUDA module

6140430

rebuild CUDA with added CUPTI lib dir

eff0de6

add CUPTI lib dir in parse_hook, use pathJoin and root

d3345ea

quote path in module file

8df8a21

use updated easyblock for CUDA, disable parse_hook

32e1c8e

use latest commit for cuda.py easyblock

4ac3218

		if self.name == 'PyTorch' and self.version == '2.1.2':
		if 'cudaver' in self.cfg.template_values and self.cfg.template_values['cudaver'] == '12.1.1':

{2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825

Are you sure you want to change the base?

{2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825

Conversation

trz42 commented Nov 21, 2024 • edited Loading

eessi-bot bot commented Nov 21, 2024

riscv-eessi-io-bot bot commented Nov 21, 2024

riscv-eessi-io-bot bot commented Nov 21, 2024

eessi-bot bot commented Nov 21, 2024

trz42 commented Nov 21, 2024

riscv-eessi-io-bot bot commented Nov 21, 2024

riscv-eessi-io-bot bot commented Nov 21, 2024

eessi-bot bot commented Nov 21, 2024 • edited Loading

eessi-bot bot commented Nov 21, 2024 • edited Loading

eessi-bot bot commented Nov 21, 2024 • edited by trz42 Loading

trz42 commented Nov 22, 2024

riscv-eessi-io-bot bot commented Nov 22, 2024

eessi-bot bot commented Nov 22, 2024 • edited Loading

eessi-bot bot commented Nov 22, 2024 • edited Loading

eessi-bot bot commented Nov 22, 2024 • edited Loading

trz42 commented Nov 22, 2024

riscv-eessi-io-bot bot commented Nov 22, 2024

eessi-bot bot commented Nov 22, 2024 • edited Loading

riscv-eessi-io-bot bot commented Nov 22, 2024

eessi-bot bot commented Nov 22, 2024 • edited Loading

eessi-bot bot commented Nov 22, 2024 • edited Loading

boegel Nov 22, 2024

Choose a reason for hiding this comment

trz42 Nov 22, 2024

Choose a reason for hiding this comment

boegel Nov 22, 2024

Choose a reason for hiding this comment

trz42 Nov 22, 2024

Choose a reason for hiding this comment

trz42 Nov 23, 2024 • edited Loading

Choose a reason for hiding this comment

trz42 commented Nov 23, 2024

eessi-bot bot commented Nov 23, 2024 • edited Loading

riscv-eessi-io-bot bot commented Nov 23, 2024

eessi-bot bot commented Nov 23, 2024 • edited Loading

trz42 commented Nov 23, 2024

riscv-eessi-io-bot bot commented Nov 23, 2024

eessi-bot bot commented Nov 23, 2024 • edited Loading

riscv-eessi-io-bot bot commented Nov 23, 2024

eessi-bot bot commented Nov 23, 2024 • edited Loading

eessi-bot bot commented Nov 23, 2024 • edited Loading

trz42 commented Nov 23, 2024

eessi-bot bot commented Nov 23, 2024 • edited Loading

riscv-eessi-io-bot bot commented Nov 23, 2024

riscv-eessi-io-bot bot commented Nov 23, 2024

eessi-bot bot commented Nov 23, 2024 • edited Loading

eessi-bot bot commented Nov 23, 2024 • edited Loading

trz42 commented Nov 24, 2024

riscv-eessi-io-bot bot commented Nov 24, 2024

eessi-bot bot commented Nov 24, 2024 • edited Loading

eessi-bot bot commented Nov 24, 2024 • edited Loading

eessi-bot bot commented Nov 24, 2024 • edited Loading

trz42 commented Nov 25, 2024

eessi-bot bot commented Nov 25, 2024 • edited Loading

riscv-eessi-io-bot bot commented Nov 25, 2024

riscv-eessi-io-bot bot commented Nov 25, 2024

eessi-bot bot commented Nov 25, 2024 • edited Loading

eessi-bot bot commented Nov 25, 2024 • edited Loading

trz42 commented Nov 21, 2024 •

edited

Loading

eessi-bot bot commented Nov 21, 2024 •

edited

Loading

eessi-bot bot commented Nov 21, 2024 •

edited

Loading

eessi-bot bot commented Nov 21, 2024 •

edited by trz42

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

trz42 Nov 23, 2024 •

edited

Loading

eessi-bot bot commented Nov 23, 2024 •

edited

Loading

eessi-bot bot commented Nov 23, 2024 •

edited

Loading

eessi-bot bot commented Nov 23, 2024 •

edited

Loading

eessi-bot bot commented Nov 23, 2024 •

edited

Loading

eessi-bot bot commented Nov 23, 2024 •

edited

Loading

eessi-bot bot commented Nov 23, 2024 •

edited

Loading

eessi-bot bot commented Nov 23, 2024 •

edited

Loading

eessi-bot bot commented Nov 23, 2024 •

edited

Loading

eessi-bot bot commented Nov 24, 2024 •

edited

Loading

eessi-bot bot commented Nov 24, 2024 •

edited

Loading

eessi-bot bot commented Nov 24, 2024 •

edited

Loading

eessi-bot bot commented Nov 25, 2024 •

edited

Loading

eessi-bot bot commented Nov 25, 2024 •

edited

Loading

eessi-bot bot commented Nov 25, 2024 •

edited

Loading