infinband 6036 交换机配置ip步骤
Connecting to COM3...
Connected.
Mellanox MLNX-OS Switch Management
switch-ecdd50 login: admin
Password:
Mellanox Switch
Mellanox configuration wizard
Do you want to use the wizard for initial configuration?
Please answer \'yes\' or \'no\'.
Do you want to use the wizard for initial configuration? yes
Step 1: Hostname? [switch-ecdd50]
Step 2: Use DHCP on mgmt0 interface? [yes] no
Step 3: Use zeroconf on mgmt0 interface? [no] no
Step 4: Primary IPv4 address and masklen? [0.0.0.0/0] 10.10.90.210
Step 5: Netmask or mask length? [255.255.255.0] 255.255.255.0
Step 6: Default gateway? 10.10.90.254
Step 7: Primary DNS server?
Step 8: Domain name?
Step 9: Enable IPv6? [yes] no
Step 10: Admin password (Enter to leave unchanged)?
You have entered the following information:
1. Hostname: switch-ecdd50
2. Use DHCP on mgmt0 interface: no
3. Use zeroconf on mgmt0 interface: no
4. Primary IPv4 address: 10.10.90.210
5. Netmask or mask length: 255.255.255.0
6. Default gateway: 10.10.90.254
7. Primary DNS server:
8. Domain name:
9. Enable IPv6: no
10. Admin password (Enter to leave unchanged): (unchanged)
To change an answer, enter the step number to return to.
Otherwise hit <enter> to save changes and exit.
Choice:
Configuration changes saved.
switch-ecdd50 [standalone: master] > exit
确认生效
switch-ecdd50 [standalone: master] # show interfaces mgmt0
Interface mgmt0 status:
Comment:
Admin up: yes
Link up: yes
DHCP running: no
IP address: 10.10.90.210
Netmask: 255.255.255.0
IPv6 enabled: no
Speed: 1000Mb/s (auto)
Duplex: full (auto)
Interface type: ethernet
Interface source: physical
MTU: 1500
HW address: F4:52:14:EC:DD:50
RX bytes: 1177381 TX bytes: 37262213
RX packets: 16017 TX packets: 151851
RX mcast packets: 0 TX discards: 0
RX discards: 0 TX errors: 0
RX errors: 0 TX overruns: 0
RX overruns: 0 TX carrier: 0
RX frame: 0 TX collisions: 0
TX queue len: 1000
switch-ecdd50 [standalone: master] #
重新配置
switch-b9568c [standalone: master] >
switch-b9568c [standalone: master] > enable
switch-b9568c [standalone: master] # configure terminal
switch-b9568c [standalone: master] (config) # configuration jump-start
确认交换机opensm状态
switch (config)#ib sm
switch (config)#show ib sm
enable
保存
configuration write
#mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
[root@com34 /root]
#flint -d /dev/mst/mt26428_pciconf0 -qq q
Image type: FS2
FW Version: 2.7.0000
Device ID: 26428
Description: Node Port1 Port2 Sys image
GUIDs: 0002c9030009f3f0 0002c9030009f3f1 0002c9030009f3f2 0002c9030009f3f3
MACs: 000000000000 000000000001
VSD:
PSID: HP_0180000009
mellanox firmware 更新
http://www.mellanox.com/page/firmware_download
http://www.mellanox.com/page/firmware_HCA_FW_update
SNMP 配置
[root@neo ~]# ssh admin@10.10.90.211
The authenticity of host \'10.10.90.211 (10.10.90.211)\' can\'t be established.
RSA key fingerprint is f6:91:17:71:0f:95:10:7e:9d:e1:bb:2b:9e:2d:96:c7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added \'10.10.90.211\' (RSA) to the list of known hosts.
Mellanox MLNX-OS Switch Management
Password:
Last login: Sat Nov 16 22:27:11 2013 from 192.168.1.38
switch-ecdd50 [standalone: master] > enable
switch-ecdd50 [standalone: master] # show snmp
SNMP enabled: yes
SNMP port: 161
System contact:
System location:
Read-only communities:
public
Read-write communities:
(none)
Interface listen enabled: yes
No Listen Interfaces.
switch-ecdd50 [standalone: master] # configure terminal
switch-ecdd50 [standalone: master] (config) # snmp-server community private_community rw
switch-ecdd50 [standalone: master] (config) # show snmp
SNMP enabled: yes
SNMP port: 161
System contact:
System location:
Read-only communities:
public
Read-write communities:
private_community
Interface listen enabled: yes
No Listen Interfaces.
switch-ecdd50 [standalone: master] (config) # configuration write
switch-ecdd50 [standalone: master] (config) # interface ib 1/36（36号端口，如果是12号端口就是1/12)
switch-ecdd50 [standalone: master] (config interface ib 1/36) #
switch-ecdd50 [standalone: master] (config interface ib 1/36) # ?
clear clear port counters
description Set ib interface description
exit Leave "interface ib 1/36" mode
help View description of the interactive help system
mtu Set ib port MTU
no Negate ib port settings
op-vls Set ib port operational VLs
phy-profile Manage interface physical profile
power-management Set ib interface power management parameters
show Display system configuration or status
shutdown Turn off selected port
speed Set ib port speed
width Set ib port width
switch-ecdd50 [standalone: master] (config interface ib 1/36) #
shutdown 为disable某个端口，no shutdown 为enable某个端口
保存配置
switch-ecdd50 [standalone: master] (config) # configuration write
switch-ecdd50 [standalone: master] (config) # interface ib
1/1 1/11 1/13 1/15 1/17 1/19 1/20 1/22 1/24 1/26 1/28 1/3 1/31 1/33 1/35 1/4 1/6 1/8 clear
1/10 1/12 1/14 1/16 1/18 1/2 1/21 1/23 1/25 1/27 1/29 1/30 1/32 1/34 1/36 1/5 1/7 1/9
修改ib端口为eth
switch-ecdd50 [standalone: master] (config) # system profile vpi-single-switch
Warning! The switch configuration is going to be deleted and the system will be reloaded.
Type \'yes\' to confirm profile change: yes
disable 端口
interface ib 1/3
shutdown
exit
获取ib之间的rdma延迟
服务端
[root@qdata-com31-demo /root]
#ib_read_lat
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Read Latency Test
Dual-port : OFF Device : mlx4_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
Mtu : 4096[B]
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x24 QPN 0x021f PSN 0xda7bd OUT 0x10 RKey 0x48056ebd VAddr 0x007feaed01b000
remote address: LID 0x22 QPN 0x021e PSN 0xc84c98 OUT 0x10 RKey 0x056ebd VAddr 0x007fdeb6a15000
---------------------------------------------------------------------------------------
客户端
[root@qdata-com32-demo /root]
#ib_read_lat 172.16.128.31
---------------------------------------------------------------------------------------
RDMA_Read Latency Test
Dual-port : OFF Device : mlx4_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 1
Mtu : 4096[B]
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x22 QPN 0x021e PSN 0xc84c98 OUT 0x10 RKey 0x056ebd VAddr 0x007fdeb6a15000
remote address: LID 0x24 QPN 0x021f PSN 0xda7bd OUT 0x10 RKey 0x48056ebd VAddr 0x007feaed01b000
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec]
2 1000 2.51 28.47 2.56
---------------------------------------------------------------------------------------
一公里增加5us的延迟，因此实际测出来400us就是80公里，200us就是40公里。
升级ib交换机os
获取os，一个os一个os升级
[root@cobbler virtbox]# ssh admin@10.10.90.243
Mellanox MLNX-OS Switch Management
Password:
Last login: Wed May 6 11:59:16 2015 from 10.10.90.250
Mellanox Switch
6280R [standalone: master] > enable
6280R [standalone: master] # configure terminal
switch-ecdd50 [standalone: master] (config) # image fetch scp://root:letsg0@10.10.90.250/virtbox/image-PPC_M460EX-SX_3.4.0012.img
switch-ecdd50 [standalone: master] (config) # image install image-PPC_M460EX-3.4.2008.img
switch-ecdd50 [standalone: master] (config) # image boot next
switch-ecdd50 [standalone: master] (config) # configuration write
switch-ecdd50 [standalone: master] (config) # configuration write
6280R [standalone: master] (config) # show images
Installed images:
Partition 1:
PPC_M460EX 3.6.3004 2017-02-05 17:31:50 ppc
Partition 2:
PPC_M460EX 3.5.1006 2016-03-09 11:29:49 ppc
Last boot partition: 1
Next boot partition: 1
Images available to be installed:
image-PPC_M460EX-3.6.3004.img
PPC_M460EX 3.6.3004 2017-02-05 17:31:50 ppc
Serve image files via HTTP/HTTPS: no
No image install currently in progress.
Boot manager password is set.
Image signing: trusted signature always required
Admin require signed images: yes
Settings for next boot only:
Fallback reboot on configuration failure: yes (default)
switch-ecdd50 [standalone: master] (config) # reload
然后重复如上功能：
image fetch scp://root:letsg0@10.10.90.250/virtbox/image-PPC_M460EX-3.4.2008.img
image fetch scp://root:letsg0@10.10.90.250/virtbox/image-PPC_M460EX-3.5.1006.img
image fetch scp://root:letsg0@10.10.90.250/virtbox/image-PPC_M460EX-3.6.3004.img
switch-ecdd50 [standalone: master] (config) # show files system
switch-ecdd50 [standalone: master] (config) # show images
switch-ecdd50 [standalone: master] (config) # images delete xxx
查看版本等信息
switch-ecdd50 [standalone: master] > enable
switch-ecdd50 [standalone: master] # show version
Product name: MLNX-OS
Product release: 3.6.3004
Build ID: #1-dev
Build date: 2017-02-05 17:31:50
Target arch: ppc
Target hw: m460ex
Built by: jenkins@16fa41bd487e
Version summary: PPC_M460EX 3.6.3004 2017-02-05 17:31:50 ppc
Product model: ppc
Host ID: F45214ECDD50
Uptime: 5d 19h 14m 54.208s
CPU load averages: 1.48 / 1.32 / 1.36
Number of CPUs: 1
System memory: 821 MB used / 1206 MB free / 2027 MB total
Swap: 0 MB used / 0 MB free / 0 MB total
switch-ecdd50 [standalone: master] # show asic-version
===================================================
Module Device Version
===================================================
MGMT SX 9.4.2160
查看ib port conter信息：
switch-ecdd50 [standalone: master] # show interfaces
<PastedGraphic-2.png>