额外注意:IB交换机也分可不可以管理,比如SX6036就是可以管理的IB交换机,
两台可以管理的IB交换机可以在IB交换机上配置高可用来实现其中一台IB交换机意外离线而导致的IB网络失效的问题。
若是不可管理的IB交换机,就只能在服务器端使用opensm来实现IB网络的高可用性。
介绍:
三台pcserver,安装RHEL7.2,每台pcserver上安装有两块HCA卡,每块HCA卡2个端口(port),
只不过每块HCA卡的第二个端口(port)没有插入IB线缆,并且每块HCA卡的第二个端口(port)在nmtui中被disable掉了。
三台主机的主机名为
rac1
rac2
node1
两台IB交换机,SX6036。
在三台pcserver上,安装mellanox的ofed驱动,ofed驱动使用opensmd服务来管理IB的子网。如下所示:
[root@rac1 ~]# systemctl status opensmd ● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager Loaded: loaded (/etc/rc.d/init.d/opensmd) Active: active (running) since Mon 2017-06-05 18:39:07 CST; 14h ago Docs: man:systemd-sysv-generator(8) Process: 2762 ExecStop=/etc/rc.d/init.d/opensmd stop (code=exited, status=0/SUCCESS) Process: 2786 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS) Main PID: 2793 (opensm) CGroup: /system.slice/opensmd.service └─2793 /usr/sbin/opensm --daemon Jun 05 18:39:06 rac1 OpenSM[2791]: Loading Cached Option:guid = 0x7cfe900300917f11 Jun 05 18:39:06 rac1 OpenSM[2791]: Loading Cached Option:guid = 0x7cfe900300917ed1 Jun 05 18:39:06 rac1 OpenSM[2793]: /var/log/opensm.log log file opened Jun 05 18:39:06 rac1 OpenSM[2793]: OpenSM 4.8.1.MLNX20170118.1a8ad26 Jun 05 18:39:06 rac1 OpenSM[2793]: Entering DISCOVERING state Jun 05 18:39:06 rac1 OpenSM[2793]: Entering STANDBY state Jun 05 18:39:07 rac1 opensmd[2786]: Starting IB Subnet Manager.[ OK ] Jun 05 18:39:07 rac1 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager. Jun 05 18:43:06 rac1 OpenSM[2793]: Entering DISCOVERING state Jun 05 18:43:06 rac1 OpenSM[2793]: Entering STANDBY state [root@rac1 ~]#
[root@rac2 ~]# systemctl status opensmd ● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager Loaded: loaded (/etc/rc.d/init.d/opensmd) Active: active (running) since Mon 2017-06-05 18:48:07 CST; 14h ago Docs: man:systemd-sysv-generator(8) Process: 1679 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS) Main PID: 1711 (opensm) CGroup: /system.slice/opensmd.service └─1711 /usr/sbin/opensm --daemon Jun 05 18:48:06 rac2 OpenSM[1695]: Loading Cached Option:guid = 0x248a0703005d07f1 Jun 05 18:48:06 rac2 OpenSM[1695]: Loading Cached Option:guid = 0x248a0703005d0841 Jun 05 18:48:06 rac2 OpenSM[1711]: /var/log/opensm.log log file opened Jun 05 18:48:06 rac2 OpenSM[1711]: OpenSM 4.8.1.MLNX20170118.1a8ad26 Jun 05 18:48:06 rac2 OpenSM[1711]: Entering DISCOVERING state Jun 05 18:48:06 rac2 OpenSM[1711]: SM port is down Jun 05 18:48:07 rac2 opensmd[1679]: Starting IB Subnet Manager.[ OK ] Jun 05 18:48:07 rac2 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager. Jun 05 18:48:46 rac2 OpenSM[1711]: SM port is up Jun 05 18:48:46 rac2 OpenSM[1711]: Entering STANDBY state [root@rac2 ~]#
[root@host1 ~]# systemctl status opensmd ● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager Loaded: loaded (/etc/rc.d/init.d/opensmd) Active: active (running) since Mon 2017-06-05 18:33:50 CST; 14h ago Docs: man:systemd-sysv-generator(8) Process: 2935 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS) Main PID: 2941 (opensm) CGroup: /system.slice/opensmd.service └─2941 /usr/sbin/opensm --daemon Jun 05 18:33:49 host1 systemd[1]: Starting LSB: Activates/Deactivates InfiniBand Subnet Manager... Jun 05 18:33:49 host1 OpenSM[2941]: /var/log/opensm.log log file opened Jun 05 18:33:49 host1 OpenSM[2941]: OpenSM 4.8.1.MLNX20170118.1a8ad26 Jun 05 18:33:49 host1 OpenSM[2941]: Entering DISCOVERING state Jun 05 18:33:49 host1 OpenSM[2941]: Entering STANDBY state Jun 05 18:33:50 host1 opensmd[2935]: Starting IB Subnet Manager.[ OK ] Jun 05 18:33:50 host1 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager. Jun 05 18:43:09 host1 OpenSM[2941]: Entering DISCOVERING state Jun 05 18:43:09 host1 OpenSM[2941]: Entering MASTER state [root@host1 ~]#
三台主机的/etc/opensm/opensm.conf文件的内容如下(只贴自行修改的部分):
rac1上的/etc/opensm/opensm.conf文件的内容: # The port GUID on which the OpenSM is running guid 0x7cfe900300917f11 guid 0x7cfe900300917ed1
rac2上的/etc/opensm/opensm.conf文件的内容: # The port GUID on which the OpenSM is running guid 0x248a0703005d07f1 guid 0x248a0703005d0841
rac2上的/etc/opensm/opensm.conf文件的内容: # The port GUID on which the OpenSM is running guid 0x248a0703005d07f1 guid 0x248a0703005d0841
rac1的ip地址如下:
[root@rac1 ~]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 0c:c4:7a:ac:3c:c8 brd ff:ff:ff:ff:ff:ff inet 192.168.100.101/24 brd 192.168.100.255 scope global enp5s0f0 valid_lft forever preferred_lft forever inet6 fe80::ec4:7aff:feac:3cc8/64 scope link valid_lft forever preferred_lft forever 3: enp5s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 0c:c4:7a:ac:3c:c9 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/24 brd 172.17.0.255 scope global enp5s0f1 valid_lft forever preferred_lft forever inet6 fe80::ec4:7aff:feac:3cc9/64 scope link valid_lft forever preferred_lft forever 4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024 link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:11 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 10.10.10.101/24 brd 10.10.10.255 scope global ib0 valid_lft forever preferred_lft forever inet6 fe80::7efe:9003:91:7f11/64 scope link valid_lft forever preferred_lft forever 5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024 link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:12 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024 link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 10.10.11.102/24 brd 10.10.11.255 scope global ib2 valid_lft forever preferred_lft forever inet6 fe80::7efe:9003:91:7ed1/64 scope link valid_lft forever preferred_lft forever 7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024 link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500 link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff [root@rac1 ~]#
rac2的ip地址如下:
[root@rac2 ~]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 0c:c4:7a:ab:d3:4c brd ff:ff:ff:ff:ff:ff inet 192.168.100.102/24 brd 192.168.100.255 scope global enp5s0f0 valid_lft forever preferred_lft forever inet6 fe80::ec4:7aff:feab:d34c/64 scope link valid_lft forever preferred_lft forever 3: enp5s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 0c:c4:7a:ab:d3:4d brd ff:ff:ff:ff:ff:ff 4: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024 link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:07:f1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 10.10.10.105/24 brd 10.10.10.255 scope global ib0 valid_lft forever preferred_lft forever 5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024 link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:07:f2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024 link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:08:41 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 10.10.11.107/24 brd 10.10.11.255 scope global ib2 valid_lft forever preferred_lft forever inet6 fe80::268a:703:5d:841/64 scope link valid_lft forever preferred_lft forever 7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024 link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:08:42 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 52:54:00:74:ff:7b brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500 link/ether 52:54:00:74:ff:7b brd ff:ff:ff:ff:ff:ff [root@rac2 ~]#
node1(第三台pcsever)的ip地址如下:
[root@host1 ~]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 0c:c4:7a:ab:e5:ce brd ff:ff:ff:ff:ff:ff inet 192.168.100.103/24 brd 192.168.100.255 scope global enp5s0f0 valid_lft forever preferred_lft forever inet6 fe80::ec4:7aff:feab:e5ce/64 scope link valid_lft forever preferred_lft forever 3: enp5s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 0c:c4:7a:ab:e5:cf brd ff:ff:ff:ff:ff:ff 4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024 link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:16:c2:31 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 10.10.10.109/24 brd 10.10.10.255 scope global ib0 valid_lft forever preferred_lft forever inet6 fe80::7efe:9003:16:c231/64 scope link valid_lft forever preferred_lft forever 5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024 link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:16:c2:32 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024 link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7d:a1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 10.10.11.111/24 brd 10.10.11.255 scope global ib2 valid_lft forever preferred_lft forever inet6 fe80::7efe:9003:91:7da1/64 scope link valid_lft forever preferred_lft forever 7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024 link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7d:a2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 52:54:00:53:65:c8 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500 link/ether 52:54:00:53:65:c8 brd ff:ff:ff:ff:ff:ff [root@host1 ~]#
rac1:
[root@rac1 ~]# ibhosts Ca : 0x248a0703005d07f0 ports 2 "rac2 HCA-1" Ca : 0x7cfe90030016c230 ports 2 "host1 HCA-1" Ca : 0x7cfe900300917f10 ports 2 "rac1 HCA-1" [root@rac1 ~]# ibaddr GID fe80::7cfe:9003:91:7f11 LID start 0x2 end 0x2 [root@rac1 ~]# ibstat CA 'mlx4_0' CA type: MT4099 Number of ports: 2 Firmware version: 2.40.7000 Hardware version: 1 Node GUID: 0x7cfe900300917f10 System image GUID: 0x7cfe900300917f13 Port 1: State: Active Physical state: LinkUp Rate: 56 Base lid: 2 LMC: 0 SM lid: 4 Capability mask: 0x02514868 Port GUID: 0x7cfe900300917f11 Link layer: InfiniBand Port 2: State: Down Physical state: Polling Rate: 10 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02514868 Port GUID: 0x7cfe900300917f12 Link layer: InfiniBand CA 'mlx4_1' CA type: MT4099 Number of ports: 2 Firmware version: 2.40.7000 Hardware version: 1 Node GUID: 0x7cfe900300917ed0 System image GUID: 0x7cfe900300917ed3 Port 1: State: Active Physical state: LinkUp Rate: 56 Base lid: 5 LMC: 0 SM lid: 8 Capability mask: 0x0251486a Port GUID: 0x7cfe900300917ed1 Link layer: InfiniBand Port 2: State: Down Physical state: Polling Rate: 10 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02514868 Port GUID: 0x7cfe900300917ed2 Link layer: InfiniBand [root@rac1 ~]# ibstatus Infiniband device 'mlx4_0' port 1 status: default gid: fe80:0000:0000:0000:7cfe:9003:0091:7f11 base lid: 0x2 sm lid: 0x4 state: 4: ACTIVE phys state: 5: LinkUp rate: 56 Gb/sec (4X FDR) link_layer: InfiniBand Infiniband device 'mlx4_0' port 2 status: default gid: fe80:0000:0000:0000:7cfe:9003:0091:7f12 base lid: 0x0 sm lid: 0x0 state: 1: DOWN phys state: 2: Polling rate: 10 Gb/sec (4X) link_layer: InfiniBand Infiniband device 'mlx4_1' port 1 status: default gid: fe80:0000:0000:0000:7cfe:9003:0091:7ed1 base lid: 0x5 sm lid: 0x8 state: 4: ACTIVE phys state: 5: LinkUp rate: 56 Gb/sec (4X FDR) link_layer: InfiniBand Infiniband device 'mlx4_1' port 2 status: default gid: fe80:0000:0000:0000:7cfe:9003:0091:7ed2 base lid: 0x0 sm lid: 0x0 state: 1: DOWN phys state: 2: Polling rate: 10 Gb/sec (4X) link_layer: InfiniBand [root@rac1 ~]# ibv_ ibv_asyncwatch ibv_dcini ibv_devices ibv_intf ibv_rc_pingpong ibv_srq_pingpong ibv_uc_pingpong ibv_umr ibv_cc_pingpong ibv_dctgt ibv_devinfo ibv_polldcinfo ibv_shared_mr ibv_task_pingpong ibv_ud_pingpong ibv_xsrq_pingpong [root@rac1 ~]# ibv_devinfo hca_id: mlx4_1 transport: InfiniBand (0) fw_ver: 2.40.7000 node_guid: 7cfe:9003:0091:7ed0 sys_image_guid: 7cfe:9003:0091:7ed3 vendor_id: 0x02c9 vendor_part_id: 4099 hw_ver: 0x1 board_id: MT_1090120019 phys_port_cnt: 2 Device ports: port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 8 port_lid: 5 port_lmc: 0x00 link_layer: InfiniBand port: 2 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx4_0 transport: InfiniBand (0) fw_ver: 2.40.7000 node_guid: 7cfe:9003:0091:7f10 sys_image_guid: 7cfe:9003:0091:7f13 vendor_id: 0x02c9 vendor_part_id: 4099 hw_ver: 0x1 board_id: MT_1090120019 phys_port_cnt: 2 Device ports: port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 4 port_lid: 2 port_lmc: 0x00 link_layer: InfiniBand port: 2 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: InfiniBand [root@rac1 ~]# ibnetdiscover # # Topology file: generated on Tue Jun 6 08:37:33 2017 # # Initiated from node 7cfe900300917f10 port 7cfe900300917f11 vendid=0x2c9 devid=0xc738 sysimgguid=0x7cfe900300f17940 switchguid=0x7cfe900300f17940(7cfe900300f17940) Switch 36 "S-7cfe900300f17940" # "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0 [31] "H-7cfe90030016c230"[1](7cfe90030016c231) # "host1 HCA-1" lid 3 4xFDR [33] "H-248a0703005d07f0"[1](248a0703005d07f1) # "rac2 HCA-1" lid 0 4xFDR [35] "H-7cfe900300917f10"[1](7cfe900300917f11) # "rac1 HCA-1" lid 2 4xFDR vendid=0x2c9 devid=0x1003 sysimgguid=0x248a0703005d07f3 caguid=0x248a0703005d07f0 Ca 2 "H-248a0703005d07f0" # "rac2 HCA-1" [1](248a0703005d07f1) "S-7cfe900300f17940"[33] # lid 0 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR vendid=0x2c9 devid=0x1003 sysimgguid=0x7cfe90030016c233 caguid=0x7cfe90030016c230 Ca 2 "H-7cfe90030016c230" # "host1 HCA-1" [1](7cfe90030016c231) "S-7cfe900300f17940"[31] # lid 3 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR vendid=0x2c9 devid=0x1003 sysimgguid=0x7cfe900300917f13 caguid=0x7cfe900300917f10 Ca 2 "H-7cfe900300917f10" # "rac1 HCA-1" [1](7cfe900300917f11) "S-7cfe900300f17940"[35] # lid 2 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR [root@rac1 ~]# iblinkinfo CA: rac2 HCA-1: 0x248a0703005d07f1 0 1[ ] ==( 4X 14.0625 Gbps Initialize/ LinkUp)==> 1 33[ ] "MF0;switch-138276:SX6036/U1" ( ) CA: host1 HCA-1: 0x7cfe90030016c231 3 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 31[ ] "MF0;switch-138276:SX6036/U1" ( ) Switch: 0x7cfe900300f17940 MF0;switch-138276:SX6036/U1: 1 1[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 2[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 3[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 4[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 5[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 6[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 7[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 8[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 9[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 10[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 11[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 12[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 13[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 14[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 15[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 16[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 17[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 18[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 19[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 20[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 21[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 22[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 23[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 24[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 25[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 26[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 27[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 28[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 29[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 30[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 31[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 3 1[ ] "host1 HCA-1" ( ) 1 32[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 33[ ] ==( 4X 14.0625 Gbps Initialize/ LinkUp)==> 0 1[ ] "rac2 HCA-1" ( ) 1 34[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 35[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 2 1[ ] "rac1 HCA-1" ( ) 1 36[ ] ==( Down/ Polling)==> [ ] "" ( ) CA: rac1 HCA-1: 0x7cfe900300917f11 2 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 35[ ] "MF0;switch-138276:SX6036/U1" ( ) [root@rac1 ~]# ibswitches Switch : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0 [root@rac1 ~]# ibnodes Ca : 0x248a0703005d07f0 ports 2 "rac2 HCA-1" Ca : 0x7cfe90030016c230 ports 2 "host1 HCA-1" Ca : 0x7cfe900300917f10 ports 2 "rac1 HCA-1" Switch : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0 [root@rac1 ~]#
rac2:
[root@rac2 ~]# ibhosts Ca : 0x7cfe900300917ed0 ports 2 "rac1 HCA-2" Ca : 0x7cfe900300917da0 ports 2 "host1 HCA-2" Ca : 0x248a0703005d0840 ports 2 "rac2 HCA-2" [root@rac2 ~]# ibaddr GID fe80::248a:703:5d:841 LID start 0x1 end 0x1 [root@rac2 ~]# ibstat CA 'mlx4_0' CA type: MT4099 Number of ports: 2 Firmware version: 2.40.7000 Hardware version: 1 Node GUID: 0x248a0703005d07f0 System image GUID: 0x248a0703005d07f3 Port 1: State: Initializing Physical state: LinkUp Rate: 56 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02514868 Port GUID: 0x248a0703005d07f1 Link layer: InfiniBand Port 2: State: Down Physical state: Polling Rate: 10 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02514868 Port GUID: 0x248a0703005d07f2 Link layer: InfiniBand CA 'mlx4_1' CA type: MT4099 Number of ports: 2 Firmware version: 2.40.7000 Hardware version: 1 Node GUID: 0x248a0703005d0840 System image GUID: 0x248a0703005d0843 Port 1: State: Active Physical state: LinkUp Rate: 56 Base lid: 1 LMC: 0 SM lid: 8 Capability mask: 0x0251486a Port GUID: 0x248a0703005d0841 Link layer: InfiniBand Port 2: State: Down Physical state: Polling Rate: 10 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02514868 Port GUID: 0x248a0703005d0842 Link layer: InfiniBand [root@rac2 ~]# ibstatus Infiniband device 'mlx4_0' port 1 status: default gid: fe80:0000:0000:0000:248a:0703:005d:07f1 base lid: 0x0 sm lid: 0x0 state: 2: INIT phys state: 5: LinkUp rate: 56 Gb/sec (4X FDR) link_layer: InfiniBand Infiniband device 'mlx4_0' port 2 status: default gid: fe80:0000:0000:0000:248a:0703:005d:07f2 base lid: 0x0 sm lid: 0x0 state: 1: DOWN phys state: 2: Polling rate: 10 Gb/sec (4X) link_layer: InfiniBand Infiniband device 'mlx4_1' port 1 status: default gid: fe80:0000:0000:0000:248a:0703:005d:0841 base lid: 0x1 sm lid: 0x8 state: 4: ACTIVE phys state: 5: LinkUp rate: 56 Gb/sec (4X FDR) link_layer: InfiniBand Infiniband device 'mlx4_1' port 2 status: default gid: fe80:0000:0000:0000:248a:0703:005d:0842 base lid: 0x0 sm lid: 0x0 state: 1: DOWN phys state: 2: Polling rate: 10 Gb/sec (4X) link_layer: InfiniBand [root@rac2 ~]# ibv_devinfo hca_id: mlx4_1 transport: InfiniBand (0) fw_ver: 2.40.7000 node_guid: 248a:0703:005d:0840 sys_image_guid: 248a:0703:005d:0843 vendor_id: 0x02c9 vendor_part_id: 4099 hw_ver: 0x1 board_id: MT_1090120019 phys_port_cnt: 2 Device ports: port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 8 port_lid: 1 port_lmc: 0x00 link_layer: InfiniBand port: 2 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx4_0 transport: InfiniBand (0) fw_ver: 2.40.7000 node_guid: 248a:0703:005d:07f0 sys_image_guid: 248a:0703:005d:07f3 vendor_id: 0x02c9 vendor_part_id: 4099 hw_ver: 0x1 board_id: MT_1090120019 phys_port_cnt: 2 Device ports: port: 1 state: PORT_INIT (2) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: InfiniBand port: 2 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: InfiniBand [root@rac2 ~]# ibnetdiscover # # Topology file: generated on Tue Jun 6 08:36:34 2017 # # Initiated from node 248a0703005d0840 port 248a0703005d0841 vendid=0x2c9 devid=0xc738 sysimgguid=0x7cfe900300ef40c0 switchguid=0x7cfe900300ef40c0(7cfe900300ef40c0) Switch 36 "S-7cfe900300ef40c0" # "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0 [31] "H-7cfe900300917da0"[1](7cfe900300917da1) # "host1 HCA-2" lid 8 4xFDR [33] "H-248a0703005d0840"[1](248a0703005d0841) # "rac2 HCA-2" lid 1 4xFDR [35] "H-7cfe900300917ed0"[1](7cfe900300917ed1) # "rac1 HCA-2" lid 5 4xFDR vendid=0x2c9 devid=0x1003 sysimgguid=0x7cfe900300917ed3 caguid=0x7cfe900300917ed0 Ca 2 "H-7cfe900300917ed0" # "rac1 HCA-2" [1](7cfe900300917ed1) "S-7cfe900300ef40c0"[35] # lid 5 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDR vendid=0x2c9 devid=0x1003 sysimgguid=0x7cfe900300917da3 caguid=0x7cfe900300917da0 Ca 2 "H-7cfe900300917da0" # "host1 HCA-2" [1](7cfe900300917da1) "S-7cfe900300ef40c0"[31] # lid 8 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDR vendid=0x2c9 devid=0x1003 sysimgguid=0x248a0703005d0843 caguid=0x248a0703005d0840 Ca 2 "H-248a0703005d0840" # "rac2 HCA-2" [1](248a0703005d0841) "S-7cfe900300ef40c0"[33] # lid 1 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDR [root@rac2 ~]# iblinkinfo CA: rac1 HCA-2: 0x7cfe900300917ed1 5 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 7 35[ ] "MF0;switch-cad412:SX6036/U1" ( ) CA: host1 HCA-2: 0x7cfe900300917da1 8 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 7 31[ ] "MF0;switch-cad412:SX6036/U1" ( ) Switch: 0x7cfe900300ef40c0 MF0;switch-cad412:SX6036/U1: 7 1[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 2[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 3[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 4[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 5[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 6[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 7[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 8[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 9[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 10[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 11[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 12[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 13[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 14[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 15[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 16[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 17[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 18[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 19[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 20[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 21[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 22[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 23[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 24[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 25[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 26[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 27[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 28[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 29[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 30[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 31[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 8 1[ ] "host1 HCA-2" ( ) 7 32[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 33[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 1[ ] "rac2 HCA-2" ( ) 7 34[ ] ==( Down/ Polling)==> [ ] "" ( ) 7 35[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 5 1[ ] "rac1 HCA-2" ( ) 7 36[ ] ==( Down/ Polling)==> [ ] "" ( ) CA: rac2 HCA-2: 0x248a0703005d0841 1 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 7 33[ ] "MF0;switch-cad412:SX6036/U1" ( ) [root@rac2 ~]# ibswitches Switch : 0x7cfe900300ef40c0 ports 36 "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0 [root@rac2 ~]# ibnodes Ca : 0x7cfe900300917ed0 ports 2 "rac1 HCA-2" Ca : 0x7cfe900300917da0 ports 2 "host1 HCA-2" Ca : 0x248a0703005d0840 ports 2 "rac2 HCA-2" Switch : 0x7cfe900300ef40c0 ports 36 "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0 [root@rac2 ~]#
node1:
[root@host1 ~]# ibhosts Ca : 0x7cfe900300917f10 ports 2 "rac1 HCA-1" Ca : 0x248a0703005d07f0 ports 2 "rac2 HCA-1" Ca : 0x7cfe90030016c230 ports 2 "host1 HCA-1" [root@host1 ~]# ibaddr GID fe80::7cfe:9003:16:c231 LID start 0x3 end 0x3 [root@host1 ~]# ibstat CA 'mlx4_0' CA type: MT4099 Number of ports: 2 Firmware version: 2.40.7000 Hardware version: 1 Node GUID: 0x7cfe90030016c230 System image GUID: 0x7cfe90030016c233 Port 1: State: Active Physical state: LinkUp Rate: 56 Base lid: 3 LMC: 0 SM lid: 4 Capability mask: 0x02514868 Port GUID: 0x7cfe90030016c231 Link layer: InfiniBand Port 2: State: Down Physical state: Polling Rate: 10 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02514868 Port GUID: 0x7cfe90030016c232 Link layer: InfiniBand CA 'mlx4_1' CA type: MT4099 Number of ports: 2 Firmware version: 2.40.7000 Hardware version: 1 Node GUID: 0x7cfe900300917da0 System image GUID: 0x7cfe900300917da3 Port 1: State: Active Physical state: LinkUp Rate: 56 Base lid: 8 LMC: 0 SM lid: 8 Capability mask: 0x0251486a Port GUID: 0x7cfe900300917da1 Link layer: InfiniBand Port 2: State: Down Physical state: Polling Rate: 10 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02514868 Port GUID: 0x7cfe900300917da2 Link layer: InfiniBand [root@host1 ~]# ibstatus Infiniband device 'mlx4_0' port 1 status: default gid: fe80:0000:0000:0000:7cfe:9003:0016:c231 base lid: 0x3 sm lid: 0x4 state: 4: ACTIVE phys state: 5: LinkUp rate: 56 Gb/sec (4X FDR) link_layer: InfiniBand Infiniband device 'mlx4_0' port 2 status: default gid: fe80:0000:0000:0000:7cfe:9003:0016:c232 base lid: 0x0 sm lid: 0x0 state: 1: DOWN phys state: 2: Polling rate: 10 Gb/sec (4X) link_layer: InfiniBand Infiniband device 'mlx4_1' port 1 status: default gid: fe80:0000:0000:0000:7cfe:9003:0091:7da1 base lid: 0x8 sm lid: 0x8 state: 4: ACTIVE phys state: 5: LinkUp rate: 56 Gb/sec (4X FDR) link_layer: InfiniBand Infiniband device 'mlx4_1' port 2 status: default gid: fe80:0000:0000:0000:7cfe:9003:0091:7da2 base lid: 0x0 sm lid: 0x0 state: 1: DOWN phys state: 2: Polling rate: 10 Gb/sec (4X) link_layer: InfiniBand [root@host1 ~]# ibv_devinfo hca_id: mlx4_1 transport: InfiniBand (0) fw_ver: 2.40.7000 node_guid: 7cfe:9003:0091:7da0 sys_image_guid: 7cfe:9003:0091:7da3 vendor_id: 0x02c9 vendor_part_id: 4099 hw_ver: 0x1 board_id: MT_1090120019 phys_port_cnt: 2 Device ports: port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 8 port_lid: 8 port_lmc: 0x00 link_layer: InfiniBand port: 2 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx4_0 transport: InfiniBand (0) fw_ver: 2.40.7000 node_guid: 7cfe:9003:0016:c230 sys_image_guid: 7cfe:9003:0016:c233 vendor_id: 0x02c9 vendor_part_id: 4099 hw_ver: 0x1 board_id: MT_1090120019 phys_port_cnt: 2 Device ports: port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 4 port_lid: 3 port_lmc: 0x00 link_layer: InfiniBand port: 2 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: InfiniBand [root@host1 ~]# ibnetdiscover # # Topology file: generated on Tue Jun 6 08:38:13 2017 # # Initiated from node 7cfe90030016c230 port 7cfe90030016c231 vendid=0x2c9 devid=0xc738 sysimgguid=0x7cfe900300f17940 switchguid=0x7cfe900300f17940(7cfe900300f17940) Switch 36 "S-7cfe900300f17940" # "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0 [31] "H-7cfe90030016c230"[1](7cfe90030016c231) # "host1 HCA-1" lid 3 4xFDR [33] "H-248a0703005d07f0"[1](248a0703005d07f1) # "rac2 HCA-1" lid 0 4xFDR [35] "H-7cfe900300917f10"[1](7cfe900300917f11) # "rac1 HCA-1" lid 2 4xFDR vendid=0x2c9 devid=0x1003 sysimgguid=0x7cfe900300917f13 caguid=0x7cfe900300917f10 Ca 2 "H-7cfe900300917f10" # "rac1 HCA-1" [1](7cfe900300917f11) "S-7cfe900300f17940"[35] # lid 2 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR vendid=0x2c9 devid=0x1003 sysimgguid=0x248a0703005d07f3 caguid=0x248a0703005d07f0 Ca 2 "H-248a0703005d07f0" # "rac2 HCA-1" [1](248a0703005d07f1) "S-7cfe900300f17940"[33] # lid 0 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR vendid=0x2c9 devid=0x1003 sysimgguid=0x7cfe90030016c233 caguid=0x7cfe90030016c230 Ca 2 "H-7cfe90030016c230" # "host1 HCA-1" [1](7cfe90030016c231) "S-7cfe900300f17940"[31] # lid 3 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR [root@host1 ~]# iblinkinfo CA: rac1 HCA-1: 0x7cfe900300917f11 2 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 35[ ] "MF0;switch-138276:SX6036/U1" ( ) CA: rac2 HCA-1: 0x248a0703005d07f1 0 1[ ] ==( 4X 14.0625 Gbps Initialize/ LinkUp)==> 1 33[ ] "MF0;switch-138276:SX6036/U1" ( ) Switch: 0x7cfe900300f17940 MF0;switch-138276:SX6036/U1: 1 1[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 2[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 3[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 4[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 5[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 6[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 7[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 8[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 9[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 10[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 11[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 12[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 13[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 14[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 15[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 16[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 17[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 18[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 19[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 20[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 21[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 22[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 23[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 24[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 25[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 26[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 27[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 28[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 29[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 30[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 31[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 3 1[ ] "host1 HCA-1" ( ) 1 32[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 33[ ] ==( 4X 14.0625 Gbps Initialize/ LinkUp)==> 0 1[ ] "rac2 HCA-1" ( ) 1 34[ ] ==( Down/ Polling)==> [ ] "" ( ) 1 35[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 2 1[ ] "rac1 HCA-1" ( ) 1 36[ ] ==( Down/ Polling)==> [ ] "" ( ) CA: host1 HCA-1: 0x7cfe90030016c231 3 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 31[ ] "MF0;switch-138276:SX6036/U1" ( ) [root@host1 ~]# ibswitches Switch : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0 [root@host1 ~]# ibnodes Ca : 0x7cfe900300917f10 ports 2 "rac1 HCA-1" Ca : 0x248a0703005d07f0 ports 2 "rac2 HCA-1" Ca : 0x7cfe90030016c230 ports 2 "host1 HCA-1" Switch : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0 [root@host1 ~]#