Running ufs on hosts with MOFED v5.6 and GPFS v5.1.3.1

[rickgrubin]

Interested to know if anyone is running ufs on hosts with network fabric MOFED v5.6 and GPFS v5.1.3.1 (what IBM is calling Spectrum Scale) or similar. Thanks.

[mark-a-potts]

I am not sure if it has been run on that specific combination, but there is no reason that it wouldn't run. As long as your system is set up with a correctly configured version of MPI (intelmpi is the most tested, but other flavors will work as well), the model won't care about the fabric or the file system.

[rickgrubin]

Thanks for your reply.

The model most certainly runs; however, it runs at a significant performance deficit that isn't IO bound, but rather MPI bound. The results are consistent with various versions of Intel MPI, hence the question, because the variable changing in this equation is the version / implementation of the fabric and GPFS.

[mark-a-potts]

Hmm. If you are using slurm, you might try experimenting with srun vs. mpirun/mpiexec. I have seen cases where srun is significantly faster. It often requires the --mpi=pmi2 flag to be used, though.

[rickgrubin]

Thanks for your reply, will give this a try.