Verbs Provider on Multi‐NIC Systems

On systems with multiple RDMA NICs, if these NICs are configured in the same IP subnet, the default ARP settings may prevent RDMA-CM from connecting to any NIC other than the first one. The observed error is "connection rejected" if FI_LOG_LEVEL is set or hang otherwise. To solve this problem, change the ARP settings by running the following script as root (assuming 8 IB NICs with netdev name ib0~ib7):

#!/bin/sh

# set ARP rules
for i in 0 1 2 3 4 5 6 7 ; do
        /sbin/sysctl -w net.ipv4.conf.ib$i.accept_local=1
        /sbin/sysctl -w net.ipv4.conf.ib$i.arp_ignore=2
        /sbin/sysctl -w net.ipv4.conf.ib$i.rp_filter=2
done

# clean the ARP cache
ip -s -s neigh flush all