两台IB交换机使用server side方式做高可用的记录

时间:2021-05-16 22:46:03
两台IB交换机使用server side方式做高可用的记录

额外注意:IB交换机也分可不可以管理,比如SX6036就是可以管理的IB交换机,
两台可以管理的IB交换机可以在IB交换机上配置高可用来实现其中一台IB交换机意外离线而导致的IB网络失效的问题。

若是不可管理的IB交换机,就只能在服务器端使用opensm来实现IB网络的高可用性。

介绍:
三台pcserver,安装RHEL7.2,每台pcserver上安装有两块HCA卡,每块HCA卡2个端口(port),
只不过每块HCA卡的第二个端口(port)没有插入IB线缆,并且每块HCA卡的第二个端口(port)在nmtui中被disable掉了。
三台主机的主机名为
rac1
rac2
node1


两台IB交换机,SX6036。

在三台pcserver上,安装mellanox的ofed驱动,ofed驱动使用opensmd服务来管理IB的子网。如下所示:

[root@rac1 ~]# systemctl status opensmd
● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager
   Loaded: loaded (/etc/rc.d/init.d/opensmd)
   Active: active (running) since Mon 2017-06-05 18:39:07 CST; 14h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2762 ExecStop=/etc/rc.d/init.d/opensmd stop (code=exited, status=0/SUCCESS)
  Process: 2786 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS)
 Main PID: 2793 (opensm)
   CGroup: /system.slice/opensmd.service
           └─2793 /usr/sbin/opensm --daemon

Jun 05 18:39:06 rac1 OpenSM[2791]:  Loading Cached Option:guid = 0x7cfe900300917f11
Jun 05 18:39:06 rac1 OpenSM[2791]:  Loading Cached Option:guid = 0x7cfe900300917ed1
Jun 05 18:39:06 rac1 OpenSM[2793]: /var/log/opensm.log log file opened
Jun 05 18:39:06 rac1 OpenSM[2793]: OpenSM 4.8.1.MLNX20170118.1a8ad26
Jun 05 18:39:06 rac1 OpenSM[2793]: Entering DISCOVERING state
Jun 05 18:39:06 rac1 OpenSM[2793]: Entering STANDBY state
Jun 05 18:39:07 rac1 opensmd[2786]: Starting IB Subnet Manager.[  OK  ]
Jun 05 18:39:07 rac1 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager.
Jun 05 18:43:06 rac1 OpenSM[2793]: Entering DISCOVERING state
Jun 05 18:43:06 rac1 OpenSM[2793]: Entering STANDBY state
[root@rac1 ~]#

[root@rac2 ~]# systemctl status opensmd
● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager
   Loaded: loaded (/etc/rc.d/init.d/opensmd)
   Active: active (running) since Mon 2017-06-05 18:48:07 CST; 14h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1679 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS)
 Main PID: 1711 (opensm)
   CGroup: /system.slice/opensmd.service
           └─1711 /usr/sbin/opensm --daemon

Jun 05 18:48:06 rac2 OpenSM[1695]:  Loading Cached Option:guid = 0x248a0703005d07f1
Jun 05 18:48:06 rac2 OpenSM[1695]:  Loading Cached Option:guid = 0x248a0703005d0841
Jun 05 18:48:06 rac2 OpenSM[1711]: /var/log/opensm.log log file opened
Jun 05 18:48:06 rac2 OpenSM[1711]: OpenSM 4.8.1.MLNX20170118.1a8ad26
Jun 05 18:48:06 rac2 OpenSM[1711]: Entering DISCOVERING state
Jun 05 18:48:06 rac2 OpenSM[1711]: SM port is down
Jun 05 18:48:07 rac2 opensmd[1679]: Starting IB Subnet Manager.[  OK  ]
Jun 05 18:48:07 rac2 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager.
Jun 05 18:48:46 rac2 OpenSM[1711]: SM port is up
Jun 05 18:48:46 rac2 OpenSM[1711]: Entering STANDBY state
[root@rac2 ~]# 



[root@host1 ~]# systemctl status opensmd
● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager
   Loaded: loaded (/etc/rc.d/init.d/opensmd)
   Active: active (running) since Mon 2017-06-05 18:33:50 CST; 14h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2935 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS)
 Main PID: 2941 (opensm)
   CGroup: /system.slice/opensmd.service
           └─2941 /usr/sbin/opensm --daemon

Jun 05 18:33:49 host1 systemd[1]: Starting LSB: Activates/Deactivates InfiniBand Subnet Manager...
Jun 05 18:33:49 host1 OpenSM[2941]: /var/log/opensm.log log file opened
Jun 05 18:33:49 host1 OpenSM[2941]: OpenSM 4.8.1.MLNX20170118.1a8ad26
Jun 05 18:33:49 host1 OpenSM[2941]: Entering DISCOVERING state
Jun 05 18:33:49 host1 OpenSM[2941]: Entering STANDBY state
Jun 05 18:33:50 host1 opensmd[2935]: Starting IB Subnet Manager.[  OK  ]
Jun 05 18:33:50 host1 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager.
Jun 05 18:43:09 host1 OpenSM[2941]: Entering DISCOVERING state
Jun 05 18:43:09 host1 OpenSM[2941]: Entering MASTER state
[root@host1 ~]#

三台主机的/etc/opensm/opensm.conf文件的内容如下(只贴自行修改的部分):
rac1上的/etc/opensm/opensm.conf文件的内容:
# The port GUID on which the OpenSM is running
guid 0x7cfe900300917f11
guid 0x7cfe900300917ed1

rac2上的/etc/opensm/opensm.conf文件的内容:
# The port GUID on which the OpenSM is running
guid 0x248a0703005d07f1
guid 0x248a0703005d0841

rac2上的/etc/opensm/opensm.conf文件的内容:
# The port GUID on which the OpenSM is running
guid 0x248a0703005d07f1
guid 0x248a0703005d0841

rac1的ip地址如下:
[root@rac1 ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 0c:c4:7a:ac:3c:c8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.101/24 brd 192.168.100.255 scope global enp5s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:feac:3cc8/64 scope link
       valid_lft forever preferred_lft forever
3: enp5s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 0c:c4:7a:ac:3c:c9 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/24 brd 172.17.0.255 scope global enp5s0f1
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:feac:3cc9/64 scope link
       valid_lft forever preferred_lft forever
4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024
    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:11 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.10.10.101/24 brd 10.10.10.255 scope global ib0
       valid_lft forever preferred_lft forever
    inet6 fe80::7efe:9003:91:7f11/64 scope link
       valid_lft forever preferred_lft forever
5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024
    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:12 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024
    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.10.11.102/24 brd 10.10.11.255 scope global ib2
       valid_lft forever preferred_lft forever
    inet6 fe80::7efe:9003:91:7ed1/64 scope link
       valid_lft forever preferred_lft forever
7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024
    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff
[root@rac1 ~]#


rac2的ip地址如下:
[root@rac2 ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 0c:c4:7a:ab:d3:4c brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.102/24 brd 192.168.100.255 scope global enp5s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:feab:d34c/64 scope link
       valid_lft forever preferred_lft forever
3: enp5s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:ab:d3:4d brd ff:ff:ff:ff:ff:ff
4: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024
    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:07:f1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.10.10.105/24 brd 10.10.10.255 scope global ib0
       valid_lft forever preferred_lft forever
5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024
    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:07:f2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024
    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:08:41 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.10.11.107/24 brd 10.10.11.255 scope global ib2
       valid_lft forever preferred_lft forever
    inet6 fe80::268a:703:5d:841/64 scope link
       valid_lft forever preferred_lft forever
7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024
    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:08:42 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 52:54:00:74:ff:7b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:74:ff:7b brd ff:ff:ff:ff:ff:ff
[root@rac2 ~]# 



node1(第三台pcsever)的ip地址如下:
[root@host1 ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 0c:c4:7a:ab:e5:ce brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.103/24 brd 192.168.100.255 scope global enp5s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:feab:e5ce/64 scope link
       valid_lft forever preferred_lft forever
3: enp5s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:ab:e5:cf brd ff:ff:ff:ff:ff:ff
4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024
    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:16:c2:31 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.10.10.109/24 brd 10.10.10.255 scope global ib0
       valid_lft forever preferred_lft forever
    inet6 fe80::7efe:9003:16:c231/64 scope link
       valid_lft forever preferred_lft forever
5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024
    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:16:c2:32 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024
    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7d:a1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.10.11.111/24 brd 10.10.11.255 scope global ib2
       valid_lft forever preferred_lft forever
    inet6 fe80::7efe:9003:91:7da1/64 scope link
       valid_lft forever preferred_lft forever
7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024
    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7d:a2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 52:54:00:53:65:c8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:53:65:c8 brd ff:ff:ff:ff:ff:ff
[root@host1 ~]# 

rac1:
[root@rac1 ~]# ibhosts
Ca    : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"
Ca    : 0x7cfe90030016c230 ports 2 "host1 HCA-1"
Ca    : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"
[root@rac1 ~]# ibaddr
GID fe80::7cfe:9003:91:7f11 LID start 0x2 end 0x2
[root@rac1 ~]# ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.40.7000
    Hardware version: 1
    Node GUID: 0x7cfe900300917f10
    System image GUID: 0x7cfe900300917f13
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 56
        Base lid: 2
        LMC: 0
        SM lid: 4
        Capability mask: 0x02514868
        Port GUID: 0x7cfe900300917f11
        Link layer: InfiniBand
    Port 2:
        State: Down
        Physical state: Polling
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02514868
        Port GUID: 0x7cfe900300917f12
        Link layer: InfiniBand
CA 'mlx4_1'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.40.7000
    Hardware version: 1
    Node GUID: 0x7cfe900300917ed0
    System image GUID: 0x7cfe900300917ed3
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 56
        Base lid: 5
        LMC: 0
        SM lid: 8
        Capability mask: 0x0251486a
        Port GUID: 0x7cfe900300917ed1
        Link layer: InfiniBand
    Port 2:
        State: Down
        Physical state: Polling
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02514868
        Port GUID: 0x7cfe900300917ed2
        Link layer: InfiniBand
[root@rac1 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7f11
    base lid:     0x2
    sm lid:         0x4
    state:         4: ACTIVE
    phys state:     5: LinkUp
    rate:         56 Gb/sec (4X FDR)
    link_layer:     InfiniBand

Infiniband device 'mlx4_0' port 2 status:
    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7f12
    base lid:     0x0
    sm lid:         0x0
    state:         1: DOWN
    phys state:     2: Polling
    rate:         10 Gb/sec (4X)
    link_layer:     InfiniBand

Infiniband device 'mlx4_1' port 1 status:
    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7ed1
    base lid:     0x5
    sm lid:         0x8
    state:         4: ACTIVE
    phys state:     5: LinkUp
    rate:         56 Gb/sec (4X FDR)
    link_layer:     InfiniBand

Infiniband device 'mlx4_1' port 2 status:
    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7ed2
    base lid:     0x0
    sm lid:         0x0
    state:         1: DOWN
    phys state:     2: Polling
    rate:         10 Gb/sec (4X)
    link_layer:     InfiniBand

[root@rac1 ~]# ibv_
ibv_asyncwatch     ibv_dcini          ibv_devices        ibv_intf           ibv_rc_pingpong    ibv_srq_pingpong   ibv_uc_pingpong    ibv_umr
ibv_cc_pingpong    ibv_dctgt          ibv_devinfo        ibv_polldcinfo     ibv_shared_mr      ibv_task_pingpong  ibv_ud_pingpong    ibv_xsrq_pingpong
[root@rac1 ~]# ibv_devinfo
hca_id:    mlx4_1
    transport:            InfiniBand (0)
    fw_ver:                2.40.7000
    node_guid:            7cfe:9003:0091:7ed0
    sys_image_guid:            7cfe:9003:0091:7ed3
    vendor_id:            0x02c9
    vendor_part_id:            4099
    hw_ver:                0x1
    board_id:            MT_1090120019
    phys_port_cnt:            2
    Device ports:
        port:    1
            state:            PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            8
            port_lid:        5
            port_lmc:        0x00
            link_layer:        InfiniBand

        port:    2
            state:            PORT_DOWN (1)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            0
            port_lid:        0
            port_lmc:        0x00
            link_layer:        InfiniBand

hca_id:    mlx4_0
    transport:            InfiniBand (0)
    fw_ver:                2.40.7000
    node_guid:            7cfe:9003:0091:7f10
    sys_image_guid:            7cfe:9003:0091:7f13
    vendor_id:            0x02c9
    vendor_part_id:            4099
    hw_ver:                0x1
    board_id:            MT_1090120019
    phys_port_cnt:            2
    Device ports:
        port:    1
            state:            PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            4
            port_lid:        2
            port_lmc:        0x00
            link_layer:        InfiniBand

        port:    2
            state:            PORT_DOWN (1)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            0
            port_lid:        0
            port_lmc:        0x00
            link_layer:        InfiniBand

[root@rac1 ~]# ibnetdiscover
#
# Topology file: generated on Tue Jun  6 08:37:33 2017
#
# Initiated from node 7cfe900300917f10 port 7cfe900300917f11

vendid=0x2c9
devid=0xc738
sysimgguid=0x7cfe900300f17940
switchguid=0x7cfe900300f17940(7cfe900300f17940)
Switch    36 "S-7cfe900300f17940"        # "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[31]    "H-7cfe90030016c230"[1](7cfe90030016c231)         # "host1 HCA-1" lid 3 4xFDR
[33]    "H-248a0703005d07f0"[1](248a0703005d07f1)         # "rac2 HCA-1" lid 0 4xFDR
[35]    "H-7cfe900300917f10"[1](7cfe900300917f11)         # "rac1 HCA-1" lid 2 4xFDR

vendid=0x2c9
devid=0x1003
sysimgguid=0x248a0703005d07f3
caguid=0x248a0703005d07f0
Ca    2 "H-248a0703005d07f0"        # "rac2 HCA-1"
[1](248a0703005d07f1)     "S-7cfe900300f17940"[33]        # lid 0 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR

vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe90030016c233
caguid=0x7cfe90030016c230
Ca    2 "H-7cfe90030016c230"        # "host1 HCA-1"
[1](7cfe90030016c231)     "S-7cfe900300f17940"[31]        # lid 3 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR

vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe900300917f13
caguid=0x7cfe900300917f10
Ca    2 "H-7cfe900300917f10"        # "rac1 HCA-1"
[1](7cfe900300917f11)     "S-7cfe900300f17940"[35]        # lid 2 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR
[root@rac1 ~]# iblinkinfo
CA: rac2 HCA-1:
      0x248a0703005d07f1      0    1[  ] ==( 4X       14.0625 Gbps Initialize/  LinkUp)==>       1   33[  ] "MF0;switch-138276:SX6036/U1" ( )
CA: host1 HCA-1:
      0x7cfe90030016c231      3    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1   31[  ] "MF0;switch-138276:SX6036/U1" ( )
Switch: 0x7cfe900300f17940 MF0;switch-138276:SX6036/U1:
           1    1[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    2[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    3[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    9[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   10[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   11[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   12[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   13[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   14[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   15[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   16[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   17[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   18[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   19[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   20[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   21[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   22[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   23[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   24[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   25[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   26[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   27[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   28[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   29[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   30[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   31[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       3    1[  ] "host1 HCA-1" ( )
           1   32[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   33[  ] ==( 4X       14.0625 Gbps Initialize/  LinkUp)==>       0    1[  ] "rac2 HCA-1" ( )
           1   34[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   35[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       2    1[  ] "rac1 HCA-1" ( )
           1   36[  ] ==(                Down/ Polling)==>             [  ] "" ( )
CA: rac1 HCA-1:
      0x7cfe900300917f11      2    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1   35[  ] "MF0;switch-138276:SX6036/U1" ( )
[root@rac1 ~]# ibswitches
Switch    : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[root@rac1 ~]# ibnodes
Ca    : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"
Ca    : 0x7cfe90030016c230 ports 2 "host1 HCA-1"
Ca    : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"
Switch    : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[root@rac1 ~]# 


rac2:
[root@rac2 ~]# ibhosts
Ca    : 0x7cfe900300917ed0 ports 2 "rac1 HCA-2"
Ca    : 0x7cfe900300917da0 ports 2 "host1 HCA-2"
Ca    : 0x248a0703005d0840 ports 2 "rac2 HCA-2"
[root@rac2 ~]# ibaddr
GID fe80::248a:703:5d:841 LID start 0x1 end 0x1
[root@rac2 ~]# ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.40.7000
    Hardware version: 1
    Node GUID: 0x248a0703005d07f0
    System image GUID: 0x248a0703005d07f3
    Port 1:
        State: Initializing
        Physical state: LinkUp
        Rate: 56
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02514868
        Port GUID: 0x248a0703005d07f1
        Link layer: InfiniBand
    Port 2:
        State: Down
        Physical state: Polling
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02514868
        Port GUID: 0x248a0703005d07f2
        Link layer: InfiniBand
CA 'mlx4_1'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.40.7000
    Hardware version: 1
    Node GUID: 0x248a0703005d0840
    System image GUID: 0x248a0703005d0843
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 56
        Base lid: 1
        LMC: 0
        SM lid: 8
        Capability mask: 0x0251486a
        Port GUID: 0x248a0703005d0841
        Link layer: InfiniBand
    Port 2:
        State: Down
        Physical state: Polling
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02514868
        Port GUID: 0x248a0703005d0842
        Link layer: InfiniBand
[root@rac2 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
    default gid:     fe80:0000:0000:0000:248a:0703:005d:07f1
    base lid:     0x0
    sm lid:         0x0
    state:         2: INIT
    phys state:     5: LinkUp
    rate:         56 Gb/sec (4X FDR)
    link_layer:     InfiniBand

Infiniband device 'mlx4_0' port 2 status:
    default gid:     fe80:0000:0000:0000:248a:0703:005d:07f2
    base lid:     0x0
    sm lid:         0x0
    state:         1: DOWN
    phys state:     2: Polling
    rate:         10 Gb/sec (4X)
    link_layer:     InfiniBand

Infiniband device 'mlx4_1' port 1 status:
    default gid:     fe80:0000:0000:0000:248a:0703:005d:0841
    base lid:     0x1
    sm lid:         0x8
    state:         4: ACTIVE
    phys state:     5: LinkUp
    rate:         56 Gb/sec (4X FDR)
    link_layer:     InfiniBand

Infiniband device 'mlx4_1' port 2 status:
    default gid:     fe80:0000:0000:0000:248a:0703:005d:0842
    base lid:     0x0
    sm lid:         0x0
    state:         1: DOWN
    phys state:     2: Polling
    rate:         10 Gb/sec (4X)
    link_layer:     InfiniBand

[root@rac2 ~]# ibv_devinfo
hca_id:    mlx4_1
    transport:            InfiniBand (0)
    fw_ver:                2.40.7000
    node_guid:            248a:0703:005d:0840
    sys_image_guid:            248a:0703:005d:0843
    vendor_id:            0x02c9
    vendor_part_id:            4099
    hw_ver:                0x1
    board_id:            MT_1090120019
    phys_port_cnt:            2
    Device ports:
        port:    1
            state:            PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            8
            port_lid:        1
            port_lmc:        0x00
            link_layer:        InfiniBand

        port:    2
            state:            PORT_DOWN (1)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            0
            port_lid:        0
            port_lmc:        0x00
            link_layer:        InfiniBand

hca_id:    mlx4_0
    transport:            InfiniBand (0)
    fw_ver:                2.40.7000
    node_guid:            248a:0703:005d:07f0
    sys_image_guid:            248a:0703:005d:07f3
    vendor_id:            0x02c9
    vendor_part_id:            4099
    hw_ver:                0x1
    board_id:            MT_1090120019
    phys_port_cnt:            2
    Device ports:
        port:    1
            state:            PORT_INIT (2)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            0
            port_lid:        0
            port_lmc:        0x00
            link_layer:        InfiniBand

        port:    2
            state:            PORT_DOWN (1)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            0
            port_lid:        0
            port_lmc:        0x00
            link_layer:        InfiniBand

[root@rac2 ~]# ibnetdiscover
#
# Topology file: generated on Tue Jun  6 08:36:34 2017
#
# Initiated from node 248a0703005d0840 port 248a0703005d0841

vendid=0x2c9
devid=0xc738
sysimgguid=0x7cfe900300ef40c0
switchguid=0x7cfe900300ef40c0(7cfe900300ef40c0)
Switch    36 "S-7cfe900300ef40c0"        # "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0
[31]    "H-7cfe900300917da0"[1](7cfe900300917da1)         # "host1 HCA-2" lid 8 4xFDR
[33]    "H-248a0703005d0840"[1](248a0703005d0841)         # "rac2 HCA-2" lid 1 4xFDR
[35]    "H-7cfe900300917ed0"[1](7cfe900300917ed1)         # "rac1 HCA-2" lid 5 4xFDR

vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe900300917ed3
caguid=0x7cfe900300917ed0
Ca    2 "H-7cfe900300917ed0"        # "rac1 HCA-2"
[1](7cfe900300917ed1)     "S-7cfe900300ef40c0"[35]        # lid 5 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDR

vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe900300917da3
caguid=0x7cfe900300917da0
Ca    2 "H-7cfe900300917da0"        # "host1 HCA-2"
[1](7cfe900300917da1)     "S-7cfe900300ef40c0"[31]        # lid 8 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDR

vendid=0x2c9
devid=0x1003
sysimgguid=0x248a0703005d0843
caguid=0x248a0703005d0840
Ca    2 "H-248a0703005d0840"        # "rac2 HCA-2"
[1](248a0703005d0841)     "S-7cfe900300ef40c0"[33]        # lid 1 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDR
[root@rac2 ~]# iblinkinfo
CA: rac1 HCA-2:
      0x7cfe900300917ed1      5    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       7   35[  ] "MF0;switch-cad412:SX6036/U1" ( )
CA: host1 HCA-2:
      0x7cfe900300917da1      8    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       7   31[  ] "MF0;switch-cad412:SX6036/U1" ( )
Switch: 0x7cfe900300ef40c0 MF0;switch-cad412:SX6036/U1:
           7    1[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7    2[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7    3[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7    9[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   10[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   11[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   12[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   13[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   14[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   15[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   16[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   17[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   18[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   19[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   20[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   21[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   22[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   23[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   24[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   25[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   26[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   27[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   28[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   29[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   30[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   31[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       8    1[  ] "host1 HCA-2" ( )
           7   32[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   33[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1    1[  ] "rac2 HCA-2" ( )
           7   34[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           7   35[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       5    1[  ] "rac1 HCA-2" ( )
           7   36[  ] ==(                Down/ Polling)==>             [  ] "" ( )
CA: rac2 HCA-2:
      0x248a0703005d0841      1    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       7   33[  ] "MF0;switch-cad412:SX6036/U1" ( )
[root@rac2 ~]# ibswitches
Switch    : 0x7cfe900300ef40c0 ports 36 "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0
[root@rac2 ~]# ibnodes
Ca    : 0x7cfe900300917ed0 ports 2 "rac1 HCA-2"
Ca    : 0x7cfe900300917da0 ports 2 "host1 HCA-2"
Ca    : 0x248a0703005d0840 ports 2 "rac2 HCA-2"
Switch    : 0x7cfe900300ef40c0 ports 36 "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0
[root@rac2 ~]# 

node1:
[root@host1 ~]# ibhosts
Ca    : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"
Ca    : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"
Ca    : 0x7cfe90030016c230 ports 2 "host1 HCA-1"
[root@host1 ~]# ibaddr
GID fe80::7cfe:9003:16:c231 LID start 0x3 end 0x3
[root@host1 ~]# ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.40.7000
    Hardware version: 1
    Node GUID: 0x7cfe90030016c230
    System image GUID: 0x7cfe90030016c233
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 56
        Base lid: 3
        LMC: 0
        SM lid: 4
        Capability mask: 0x02514868
        Port GUID: 0x7cfe90030016c231
        Link layer: InfiniBand
    Port 2:
        State: Down
        Physical state: Polling
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02514868
        Port GUID: 0x7cfe90030016c232
        Link layer: InfiniBand
CA 'mlx4_1'
    CA type: MT4099
    Number of ports: 2
    Firmware version: 2.40.7000
    Hardware version: 1
    Node GUID: 0x7cfe900300917da0
    System image GUID: 0x7cfe900300917da3
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 56
        Base lid: 8
        LMC: 0
        SM lid: 8
        Capability mask: 0x0251486a
        Port GUID: 0x7cfe900300917da1
        Link layer: InfiniBand
    Port 2:
        State: Down
        Physical state: Polling
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02514868
        Port GUID: 0x7cfe900300917da2
        Link layer: InfiniBand
[root@host1 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
    default gid:     fe80:0000:0000:0000:7cfe:9003:0016:c231
    base lid:     0x3
    sm lid:         0x4
    state:         4: ACTIVE
    phys state:     5: LinkUp
    rate:         56 Gb/sec (4X FDR)
    link_layer:     InfiniBand

Infiniband device 'mlx4_0' port 2 status:
    default gid:     fe80:0000:0000:0000:7cfe:9003:0016:c232
    base lid:     0x0
    sm lid:         0x0
    state:         1: DOWN
    phys state:     2: Polling
    rate:         10 Gb/sec (4X)
    link_layer:     InfiniBand

Infiniband device 'mlx4_1' port 1 status:
    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7da1
    base lid:     0x8
    sm lid:         0x8
    state:         4: ACTIVE
    phys state:     5: LinkUp
    rate:         56 Gb/sec (4X FDR)
    link_layer:     InfiniBand

Infiniband device 'mlx4_1' port 2 status:
    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7da2
    base lid:     0x0
    sm lid:         0x0
    state:         1: DOWN
    phys state:     2: Polling
    rate:         10 Gb/sec (4X)
    link_layer:     InfiniBand

[root@host1 ~]# ibv_devinfo
hca_id:    mlx4_1
    transport:            InfiniBand (0)
    fw_ver:                2.40.7000
    node_guid:            7cfe:9003:0091:7da0
    sys_image_guid:            7cfe:9003:0091:7da3
    vendor_id:            0x02c9
    vendor_part_id:            4099
    hw_ver:                0x1
    board_id:            MT_1090120019
    phys_port_cnt:            2
    Device ports:
        port:    1
            state:            PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            8
            port_lid:        8
            port_lmc:        0x00
            link_layer:        InfiniBand

        port:    2
            state:            PORT_DOWN (1)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            0
            port_lid:        0
            port_lmc:        0x00
            link_layer:        InfiniBand

hca_id:    mlx4_0
    transport:            InfiniBand (0)
    fw_ver:                2.40.7000
    node_guid:            7cfe:9003:0016:c230
    sys_image_guid:            7cfe:9003:0016:c233
    vendor_id:            0x02c9
    vendor_part_id:            4099
    hw_ver:                0x1
    board_id:            MT_1090120019
    phys_port_cnt:            2
    Device ports:
        port:    1
            state:            PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            4
            port_lid:        3
            port_lmc:        0x00
            link_layer:        InfiniBand

        port:    2
            state:            PORT_DOWN (1)
            max_mtu:        4096 (5)
            active_mtu:        4096 (5)
            sm_lid:            0
            port_lid:        0
            port_lmc:        0x00
            link_layer:        InfiniBand

[root@host1 ~]# ibnetdiscover
#
# Topology file: generated on Tue Jun  6 08:38:13 2017
#
# Initiated from node 7cfe90030016c230 port 7cfe90030016c231

vendid=0x2c9
devid=0xc738
sysimgguid=0x7cfe900300f17940
switchguid=0x7cfe900300f17940(7cfe900300f17940)
Switch    36 "S-7cfe900300f17940"        # "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[31]    "H-7cfe90030016c230"[1](7cfe90030016c231)         # "host1 HCA-1" lid 3 4xFDR
[33]    "H-248a0703005d07f0"[1](248a0703005d07f1)         # "rac2 HCA-1" lid 0 4xFDR
[35]    "H-7cfe900300917f10"[1](7cfe900300917f11)         # "rac1 HCA-1" lid 2 4xFDR

vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe900300917f13
caguid=0x7cfe900300917f10
Ca    2 "H-7cfe900300917f10"        # "rac1 HCA-1"
[1](7cfe900300917f11)     "S-7cfe900300f17940"[35]        # lid 2 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR

vendid=0x2c9
devid=0x1003
sysimgguid=0x248a0703005d07f3
caguid=0x248a0703005d07f0
Ca    2 "H-248a0703005d07f0"        # "rac2 HCA-1"
[1](248a0703005d07f1)     "S-7cfe900300f17940"[33]        # lid 0 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR

vendid=0x2c9
devid=0x1003
sysimgguid=0x7cfe90030016c233
caguid=0x7cfe90030016c230
Ca    2 "H-7cfe90030016c230"        # "host1 HCA-1"
[1](7cfe90030016c231)     "S-7cfe900300f17940"[31]        # lid 3 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR
[root@host1 ~]# iblinkinfo
CA: rac1 HCA-1:
      0x7cfe900300917f11      2    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1   35[  ] "MF0;switch-138276:SX6036/U1" ( )
CA: rac2 HCA-1:
      0x248a0703005d07f1      0    1[  ] ==( 4X       14.0625 Gbps Initialize/  LinkUp)==>       1   33[  ] "MF0;switch-138276:SX6036/U1" ( )
Switch: 0x7cfe900300f17940 MF0;switch-138276:SX6036/U1:
           1    1[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    2[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    3[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    9[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   10[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   11[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   12[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   13[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   14[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   15[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   16[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   17[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   18[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   19[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   20[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   21[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   22[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   23[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   24[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   25[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   26[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   27[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   28[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   29[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   30[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   31[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       3    1[  ] "host1 HCA-1" ( )
           1   32[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   33[  ] ==( 4X       14.0625 Gbps Initialize/  LinkUp)==>       0    1[  ] "rac2 HCA-1" ( )
           1   34[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   35[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       2    1[  ] "rac1 HCA-1" ( )
           1   36[  ] ==(                Down/ Polling)==>             [  ] "" ( )
CA: host1 HCA-1:
      0x7cfe90030016c231      3    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1   31[  ] "MF0;switch-138276:SX6036/U1" ( )
[root@host1 ~]# ibswitches
Switch    : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[root@host1 ~]# ibnodes
Ca    : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"
Ca    : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"
Ca    : 0x7cfe90030016c230 ports 2 "host1 HCA-1"
Switch    : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0
[root@host1 ~]#