![]() ![]() $ /uufs//sys/spack/linux-rocky8-nehalem/nvhpc-21.7/ucx-1.11.2-jvwucdhwoqpn2xsttr55wgb5kzzbo32v/bin/ucx_info -d | grep mlx $ /uufs//sys/spack/linux-rocky8-nehalem/gcc-8.5.0/ucx-1.11.2-ujc57b4cyrldztpxujdb7v3kaaww54tt/bin/ucx_info -d | grep verbs Transport: rc_verbs Transport: ud_verbs $ /uufs//sys/spack/linux-rocky8-nehalem/nvhpc-21.7/ucx-1.11.2-jvwucdhwoqpn2xsttr55wgb5kzzbo32v/bin/ucx_info -d | grep verbs $ucx_info -v UCT version=1.11.2 revision ef2bbcf configured with: -disable-logging -disable-debug -disable-assertions -disable-params-check -prefix=/uufs//sys/spack/linux-rocky8-nehalem/gcc-8.5.0/ucx-1.11.2-etdhrh4gzoj2nroomy7bipr2p2e3ly4l -with-verbs=/usr -enable-mt -enable-cma -disable-params-check -without-avx -enable-optimizations -disable-assertions -disable-logging -with-pic -with-rc -with-ud -with-dc -with-mlx5-dv -without-ib-hw-tm -without-dm -with-cm -without-rocm -without-java -without-cuda -without-gdrcopy -without-knem -without-xpmem $ ucx_info -v UCT version=1.11.2 revision ef2bbcf configured with: -disable-logging -disable-debug -disable-assertions -disable-params-check -prefix=/uufs//sys/spack/linux-rocky8-nehalem/nvhpc-21.5/ucx-1.11.2-asrhvd26hyucdhokcp6l5ufukmgxync7 -with-verbs=/usr -enable-mt -enable-cma -disable-params-check -without-avx -enable-optimizations -disable-assertions -disable-logging -with-pic -with-rc -with-ud -with-dc -with-mlx5-dv -without-ib-hw-tm -without-dm -with-cm -without-rocm -without-java -without-cuda -without-gdrcopy -without-knem -without-xpmem with-verbs=/usr -disable-mt -enable-cma -disable-params-check -without-avx -enable-optimizations -disable-assertions -disable-logging -with-pic -with-rc -with-ud -with-dc -with-mlx5-dv -without-ib-hw-tm -without-dm -with-cm -without-rocm -without-java -without-cuda -without-gdrcopy -without-knem -without-xpmem ![]() ![]() Spack install target=nehalem fabrics=ucx +internal-hwloc+thread_multiple schedulers=slurm +legacylaunchers ^ucx +mlx5-dv+verbs+cm+ud+dc+rc+cma the GNU (on Rocky Linux 8), replace with results in the following configure arguments: I'll be happy to provide more info but first I'd be curious to hear if you had similar reports in the past, or, if someone has successfully built UCX with NVHPC for IB. So, I don't think it's OS/driver stack related. I have also did the same build on our older CentOS 7 system and the issue is present there too. OpenMPI which is built atop of this UCX, in the NVHPC build case, runs TCP over IB resulting in ~20 us latencies, while the gcc/intel build runs over IB with ~1.7 us latencies.Īny thoughts on this would be appreciated. I have verified in the Spack build logs that the libibverbs and libmlx5 are being found by configure and linked in both the GNU/Intel and NVHPC builds, which makes me perplexed as to why the ucx_info is not reporting these as active in the NVHPC build. When building with GNU or Intel compilers, ucx_info correctly shows these, but with the NVHPC it does not. We are building OpenMPI with UCX as fabric using Spack package manager on our clusters that have various generations of Mellanox InfiniBand. ![]()
0 Comments
Leave a Reply. |