一、简介
iSCSI（internet SCSI）技术由IBM公司研究开发，是一个供硬件设备使用的、可以在IP协议的上层运行的SCSI指令集，这种指令集合可以实现在IP网络上运行SCSI协议，使其能够在诸如高速千兆以太网上进行路由选择。iSCSI技术是一种新储存技术，该技术是将现有SCSI接口与以太网络(Ethernet)技术结合，使服务器可与使用IP网络的储存装置互相交换资料。
iSCSI是一种基于TCP/IP 的协议，用来建立和管理IP存储设备、主机和客户机等之间的相互连接，并创建存储区域网络（SAN）。SAN 使得SCSI 协议应用于高速数据传输网络成为可能，这种传输以数据块级别（block-level）在多个数据存储网络间进行。SCSI 结构基于C/S模式，其通常应用环境是：设备互相靠近，并且这些设备由SCSI 总线连接。
iSCSI 的主要功能是在TCP/IP 网络上的主机系统（启动器 initiator）和存储设备（目标器 target）之间进行大量数据的封装和可靠传输过程。iSCSI可分享的设备类型有很多，包括镜像文件（*.img）、分区（partition）、物理硬盘、raid设备、逻辑卷等
完整的iSCSI系统的拓扑结构如下：
Linux环境挂载iSCSI存储
1. 检查系统是否安装iSCSI initiator
[[email protected] ~]# rpm -qa|grep iscsi [[email protected] ~]# [[email protected] ~]# mount /dev/cdrom /mnt/ mount: block device /dev/sr0 is write-protected, mounting read-only [[email protected] ~]# cd /mnt/ [[email protected] Packages]# find . -name *iscsi* ./iscsi-initiator-utils-6.2.0.872-41.el6.i686.rpm [[email protected] Packages]# yum list|grep iscsi Unable to read consumer identity iscsi-initiator-utils.i686 6.2.0.872-41.el6 dvd [[email protected] Packages]# rpm -ivh iscsi-initiator-utils-6.2.0.872-41.el6.i686.rpm ^C [[email protected] Packages]# yum -y install iscsi-initiator-utils iscsi-initiator-utils-devel [[email protected] ~]# rpm -qa|grep iscsi iscsi-initiator-utils-6.2.0.873-14.el6.x86_64 iscsi-initiator-utils-devel-6.2.0.873-14.el6.x86_64 [[email protected] ~]#
这个安装将iscsid、iscsiadm安装到/sbin目录下，它还将把默认的配置文件安装到/etc/iscsi目录下：
[[email protected] ~]# cat /sbin/iscsi iscsiadm iscsid iscsi-iname iscsistart iscsiuio [[email protected] ~]# cat /etc/iscsi/i initiatorname.iscsi iscsid.conf [[email protected] ~]# cat /etc/iscsi/i
/etc/iscsi/iscsid.conf 所有刚发起的iSCSI session默认都将使用这个文件中的参数设定。
/etc/iscsi/initiatorname.iscsi 软件iSCSI initiator的intiator名称配置文件。
在iscsi启动的时候，iscsid和iscsiadm会读取这两个配置文件。
2. 查看Initiator IQNname
[[email protected] Packages]# more /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:19bd3b75485f [[email protected] Packages]#
3.启动iscsi的服务，并将iscsi写到开机自行加载。
[[email protected] Packages]# chkconfig --list|grep iscsi iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off iscsid 0:off 1:off 2:off 3:on 4:on 5:on 6:off [[email protected] Packages]# service iscsi start [[email protected] Packages]# service iscsid start [[email protected] Packages]# /sbin/iscsistart iscsistart: InitiatorName not set. Exiting iscsistart [[email protected] src]#
安装完成后使用service iscsi start 和service iscsid start无法启动尝试使用/sbin/iscsistart ，启动报：InitiatorName not set. Exiting iscsistart，解决：直接使用iscsiadm通过下面的第4步查询target，自动启动iscsi和iscsid，如果出现“iscsid dead but pid file exist”,输入命令：rm –fr /var/run/iscsid.pid
4. 查询ISCSI设备target的iqn号码
首先要在存储设备上划分磁盘阵列的磁盘空间（即Virtual disk），设置好iSCSI 目标方（target），
iscsiadm是基于命令行的iscsi管理工具，提供了对iscsi节点、会话、连接以及发现记录的操作。默认情况下，iscsi发起方和目标方之间通过端口3260连接
iscsiadm的使用说明可以查看/usr/share/doc/iscsi-initiator-utils-6.2.0.742/README，也可以运行man iscsiadm或iscsiadm --help
Discover targets at a given IP address:
iscsiadm --mode discoverydb --type sendtargets --portal 192.168.1.10 --discover
[[email protected] ~]# iscsiadm -m discovery -t st -p 10.0.0.10 Starting iscsid: [ OK ] iscsiadm: No portals found [[email protected] ~]# ps -ef|grep -i iscsi root 2259 2 0 07:03 ? 00:00:00 [iscsi_eh] root 2303 1 0 07:03 ? 00:00:00 iscsiuio root 2308 1 0 07:03 ? 00:00:00 iscsid root 2309 1 0 07:03 ? 00:00:00 iscsid root 2314 2154 0 07:04 pts/0 00:00:00 grep -i iscsi [[email protected] ~]#
-m discovery //侦测target
-t sendtargets //通过iscsi协议
-p IP:port //指定target的IP和port，不写port的话，默认为3260
iscsiadm: No portals found
这时我们需要到存储上去允许我们接入，这里需要用到Initiator IQNname
[[email protected] ~]# cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:19bd3b75485f [[email protected] ~]#
[[email protected] ~]# iscsiadm -m discovery -t st -p 10.0.0.10 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1 [[email protected] ~]#
此时会找到目标target iqn，类似10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1 ，被发现的目标也叫做节点
iscsiadm 侦测到的结果会写入/var/lib/iscsi/nodes/ 中，因此只需启动/etc/init.d/iscsi 就能够在下次开机时，自动连接到正确的target了。
[[email protected] ~]# cd /var/lib/iscsi/nodes/ [[email protected] nodes]# ls iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1 [[email protected] nodes]#
侦测信息都写入了/var/lib/iscsi/nodes/iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1/10.0.0.10,3260,1/default 文件中了。
5. 登录target
Login, must use a node record id found by the discovery:
iscsiadm --mode node --targetname iqn.2001-05.com.doe:test --portal 192.168.1.1:3260 --login
#登录全部扫描到的服务器 [[email protected] Packages]# iscsiadm -m node --login #登录某一个，targetname为检测到的存储target name [[email protected] 10.0.0.10,3260,1]# iscsiadm -m node --targetname iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1 -p 10.0.0.10 -l Logging in to [iface: default, target: iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1, portal: 10.0.0.10,3260] (multiple) Login to [iface: default, target: iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1, portal: 10.0.0.10,3260] successful. #查看核对登录信息 [[email protected] 10.0.0.10,3260,1]# iscsiadm -m session -o show tcp: [1] 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1 (non-flash) [[email protected] 10.0.0.10,3260,1]# fdisk -l Disk /dev/sda: 53.7 GB, 53687091200 bytes 255 heads, 63 sectors/track, 6527 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0008019d Device Boot Start End Blocks Id System /dev/sda1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 26 1070 8388608 82 Linux swap / Solaris /dev/sda3 1070 6528 43834368 83 Linux Disk /dev/sdb: 859.0 GB, 859003944960 bytes 255 heads, 63 sectors/track, 104434 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xbc7a8d15 Device Boot Start End Blocks Id System /dev/sdb1 1 104434 838866073+ 7 HPFS/NTFS [[email protected] 10.0.0.10,3260,1]#
可以看到一个800多G的盘，因为这里挂载的是镜像，挂载lun步骤一样
也可简写
[[email protected] Packages]# iscsiadm -m node -p 10.0.0.10 -l
默认重启后也会自动挂载的，如果要在系统启动时自动登入可以如下操作：
[[email protected] Packages]# iscsiadm -m node -targetname iqn.1994-05.com.redhat:4ef28aa338b5 -p 192.168.100.252 --op update -n node.startup -v automatic [[email protected] Packages]# iscsiadm -m node -T iqn.1994-05.com.redhat:4ef28aa338b5 -p 192.168.100.252 --op update -n node.startup -v automatic
iqn.1994-05.com.redhat:4ef28aa338b5是目标名 iqn号可以在/etc/iscsi/initiatorname.iscsi中查看
Logout:
iscsiadm --mode node --targetname iqn.2001-05.com.doe:test --portal 192.168.1.1:3260 --logout
[[email protected] Packages]# umount /mnt/data [[email protected] ~]# iscsiadm -m node --targetname iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1 -p 10.0.0.10 -u Logging out of session [sid: 1, target: iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1, portal: 10.0.0.10,3260] Logout of [sid: 1, target: iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1, portal: 10.0.0.10,3260] successful. [[email protected] ~]# fdisk -l Disk /dev/sda: 53.7 GB, 53687091200 bytes 255 heads, 63 sectors/track, 6527 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0008019d Device Boot Start End Blocks Id System /dev/sda1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 26 1070 8388608 82 Linux swap / Solaris /dev/sda3 1070 6528 43834368 83 Linux [[email protected] ~]#
6.显示已经映射结果磁盘阵列：iscsiadm –mode node
[[email protected] Packages]# iscsiadm -m node -o show [[email protected] Packages]# iscsiadm -m node 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-1cb196-d89a91b07-3e549b4f79c51ee0-vol-98.115 [[email protected] ~]#
7.登入节点后以看到目标方的存储设备信息
在服务器上运行# fdisk –l可以看到目标方的存储设备信息
[[email protected] Packages]# fdisk -l Disk /dev/sdb: 3489.8 GB, 3489862254592 bytes 255 heads, 63 sectors/track, 424284 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdb doesn't contain a valid partition table
/dev/sdb即为连接的iscsi存储设备，PS：如果SAN硬盘通过iscsiadm扫描后能看到三个，挂载其中的mapper映射即可
8.对iscsi设备分区、格式化、挂载
[[email protected] Packages]# fdisk /dev/sdb [[email protected] Packages]# mkfs.ext4 /dev/sdb1
对iscsi存储设备分区，在开机自动挂载需要注意挂载选项不能使用defaults，应为_netdev，UUID要顶格写（Linux系统重启后，磁盘设备的名称可能会发生变化，从而引起文件系统不能挂载上来或者不能正确挂载，使用UUID的方式进行挂载可以解决这个问题，使用命令“tune2fs -l /dev/sdb”可以查到文件系统的UUID，也可以使用给文件系统设置卷标的方式来解决），例如：
[[email protected] Packages]# vi /etc/fstab UUID=9b71318a-6609-4679-8a69-420c1fffea63 /mnt/data ext4 _netdev 0 0
将/etc/fstab中所有的档案系统都挂载上：
[[email protected] Packages]# mount -a
用df查看文件系统是否已经挂载成功
[[email protected] Packages]# df -Th
存储容量大于2T，不能用不同分区 fdisk ，只能用GPT分区.
[[email protected] Packages]# parted /dev/sdb #对/dev/sdb进行分区 print #打印信息，从中可以看出这个磁盘大小和分区格式 mklabel gpt mkpart primary 0% 100% print #从中可以看出这个磁盘分区格式已经变了成GPT了
备注：如果重新安装OS后，就需要重新获取IQN name。
多路径功能配置
环境
存储：Dell Equallogic
存储IP：10.0.0.10
服务器：eth1 10.0.0.97 eth2 10.0.0.98
1．使用ping命令确保两块网卡到EQ的Group IP地址能通
宕掉eth1后eth2立马就可以通，此时再启动eth1后eth1就不通了，eth1、eth2反过来测试结果每次只有一个IP会通，Linux默认启用了反向路由检查，我们只需要关闭反向路由检查即可。
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter
[[email protected] ~]# cat /proc/sys/net/ipv4/conf/all/rp_filter 0 [[email protected] ~]# cat /proc/sys/net/ipv4/conf/eth0/rp_filter 1 [[email protected] ~]# cat /proc/sys/net/ipv4/conf/eth1/rp_filter 1 [[email protected] ~]# cat /proc/sys/net/ipv4/conf/eth2/rp_filter 1 [[email protected] ~]# echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter [[email protected] ~]# echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter [[email protected] ~]# ping 10.0.0.10 -I eth1 PING 10.0.0.10 (10.0.0.10) from 10.0.0.97 eth1: 56(84) bytes of data. 64 bytes from 10.0.0.10: icmp_seq=1 ttl=255 time=0.126 ms ^C --- 10.0.0.10 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 772ms rtt min/avg/max/mdev = 0.126/0.126/0.126/0.000 ms [[email protected] ~]# ping 10.0.0.10 -I eth2 PING 10.0.0.10 (10.0.0.10) from 10.0.0.98 eth2: 56(84) bytes of data. 64 bytes from 10.0.0.10: icmp_seq=1 ttl=255 time=0.136 ms ^C --- 10.0.0.10 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 752ms rtt min/avg/max/mdev = 0.136/0.136/0.136/0.000 ms [[email protected] ~]#
这个是临时的重启就丢失了，我们需要写入开机启动文件/etc/rc.d/rc.local里
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter
[[email protected] ~]# vim /etc/rc.d/rc.local #!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter [[email protected] ~]#
或者直接修改内核文件/etc/sysctl.conf
[[email protected] ~]# vim /etc/sysctl.conf # Controls source route verification net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.all.rp_filter = 0 net.ipv4.conf.eth1.rp_filter = 0 net.ipv4.conf.eth2.rp_filter = 0 [[email protected] ~]# sysctl -p
此时我们通过iscsiadm -m discovery -t st -p 10.0.0.10 还是只能看到一条路径
[email protected] ~]# iscsiadm -m discovery -t st -p 10.0.0.10 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-413c82a0a-36e6675ea9b58bc4-vol-98.115-2017-03-06-01:00:03.112094.5 [[email protected] ~]#
此时我们需要建立iscsi端口，
[[email protected] ~]# iscsiadm -m iface -I iscsi_eth1 --op=new New interface iscsi_eth1 added [[email protected] ~]# iscsiadm -m iface -I iscsi_eth2 --op=new New interface iscsi_eth2 added [[email protected] ~]#
绑定iscsi端口，建立iscsi_eth1和eht1、iscsi_eth2和eth2的映射关系.
[[email protected] ~]# iscsiadm -m iface -I iscsi_eth1 --op=update -n iface.net_ifacename -v eth1 iscsi_eth1 updated. [[email protected] ~]# iscsiadm -m iface -I iscsi_eth2 --op=update -n iface.net_ifacename -v eth2 iscsi_eth2 updated. [[email protected] ~]#
也可以直接编辑iscsi_eth2和iscsi_eth1文件，把里面的iface.net_ifacename=default改成iface.net_ifacename=eth2和iface.net_ifacename=eth1即可。
[[email protected] ~]# cat /var/lib/iscsi/ifaces/iscsi_eth1 # BEGIN RECORD 6.2.0-873.22.el6 iface.iscsi_ifacename = iscsi_eth1 iface.net_ifacename = eth1 iface.transport_name = tcp iface.vlan_id = 0 iface.vlan_priority = 0 iface.iface_num = 0 iface.mtu = 0 iface.port = 0 iface.tos = 0 iface.ttl = 0 iface.tcp_wsf = 0 iface.tcp_timer_scale = 0 iface.def_task_mgmt_timeout = 0 iface.erl = 0 iface.max_receive_data_len = 0 iface.first_burst_len = 0 iface.max_outstanding_r2t = 0 iface.max_burst_len = 0 # END RECORD [[email protected] ~]#
确认iface配置
[[email protected] ~]# iscsiadm -m iface default tcp,<empty>,<empty>,<empty>,<empty> iser iser,<empty>,<empty>,<empty>,<empty> iscsi_eth2 tcp,<empty>,<empty>,eth2,<empty> iscsi_eth1 tcp,<empty>,<empty>,eth1,<empty> [[email protected] ~]#
重启iscsi服务后就可以发现两条路径了
[[email protected] ~]# service iscsi restart Stopping iscsi: [ OK ] Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2001-05.com.equallogic:0-8a0906-413c82a0a-36e6675ea9b58bc4-vol-98.115-2017-03-06-01:00:03.112094.5, portal: 10.0.0.10,3260]. iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure) iscsiadm: Could not log into all portals [ OK ] [[email protected] ~]# iscsiadm -m discovery -t st -p 10.0.0.10 iscsiadm: This command will remove the record [iface: iscsi_eth2, target: iqn.2001-05.com.equallogic:0-8a0906-d20c82a0a-dfb6675fa0158cff-115test-2017-03-21-00:02:41.113343.1, portal: 10.0.0.10,3260], but a session is using it. Logout session then rerun command to remove record. iscsiadm: This command will remove the record [iface: iscsi_eth1, target: iqn.2001-05.com.equallogic:0-8a0906-d20c82a0a-dfb6675fa0158cff-115test-2017-03-21-00:02:41.113343.1, portal: 10.0.0.10,3260], but a session is using it. Logout session then rerun command to remove record. 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test [[email protected] ~]# iscsiadm -m node -l Logging in to [iface: iscsi_eth2, target: iqn.2001-05.com.equallogic:0-8a0906-413c82a0a-36e6675ea9b58bc4-vol-98.115-2017-03-06-01:00:03.112094.5, portal: 10.0.0.10,3260] (multiple) Logging in to [iface: iscsi_eth1, target: iqn.2001-05.com.equallogic:0-8a0906-413c82a0a-36e6675ea9b58bc4-vol-98.115-2017-03-06-01:00:03.112094.5, portal: 10.0.0.10,3260] (multiple) iscsiadm: Could not login to [iface: iscsi_eth2, target: iqn.2001-05.com.equallogic:0-8a0906-413c82a0a-36e6675ea9b58bc4-vol-98.115-2017-03-06-01:00:03.112094.5, portal: 10.0.0.10,3260]. iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure) iscsiadm: Could not login to [iface: iscsi_eth1, target: iqn.2001-05.com.equallogic:0-8a0906-413c82a0a-36e6675ea9b58bc4-vol-98.115-2017-03-06-01:00:03.112094.5, portal: 10.0.0.10,3260]. iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure) iscsiadm: Could not log into all portals [[email protected] ~]# netstat -anp|grep 3260 tcp 0 0 10.0.0.98:50480 10.0.0.14:3260 ESTABLISHED 1501/iscsid tcp 0 0 10.0.0.97:55681 10.0.0.11:3260 ESTABLISHED 1501/iscsid [[email protected] ~]#
认证失败！因为之前挂在了其他机器上，这里需要现在原来机器上退出登录，然后再重新登录就可以了
[[email protected] ~]# iscsiadm -m node --login Logging in to [iface: iscsi_eth2, target: iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test, portal: 10.0.0.10,3260] (multiple) Logging in to [iface: iscsi_eth1, target: iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test, portal: 10.0.0.10,3260] (multiple) Login to [iface: iscsi_eth2, target: iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test, portal: 10.0.0.10,3260] successful. Login to [iface: iscsi_eth1, target: iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test, portal: 10.0.0.10,3260] successful. [[email protected] ~]# iscsiadm -m session -o show tcp: [1] 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-d20c82a0a-dfb6675fa0158cff-115test-2017-03-21-00:02:41.113343.1 (non-flash) tcp: [2] 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-d20c82a0a-dfb6675fa0158cff-115test-2017-03-21-00:02:41.113343.1 (non-flash) tcp: [5] 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test (non-flash) tcp: [6] 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test (non-flash) [[email protected] ~]#
此时用fdisk可以看到2块盘
[[email protected] ~]# fdisk -l Disk /dev/sda: 644.2 GB, 644245094400 bytes 255 heads, 63 sectors/track, 78325 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000a75d6 Device Boot Start End Blocks Id System /dev/sda1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 26 2115 16777216 82 Linux swap / Solaris /dev/sda3 2115 78326 612162560 83 Linux Disk /dev/sdd: 10.7 GB, 10742661120 bytes 64 heads, 32 sectors/track, 10245 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sde: 10.7 GB, 10742661120 bytes 64 heads, 32 sectors/track, 10245 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/mpathb: 10.7 GB, 10742661120 bytes 255 heads, 63 sectors/track, 1306 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 [[email protected] ~]#
我这里查看那的时候是3块，因为前面没配置iscsi端口前登陆了次，可以先删除连接在重新获取（见文末删除失效的iscsi 连接）
输入iscsiadm -m session可以看到当前正在连接的session.
[[email protected] ~]# iscsiadm -m node 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test [[email protected] ~]# iscsiadm -m session tcp: [3] 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test (non-flash) tcp: [4] 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test (non-flash) [[email protected] ~]#
安装DM-multipath 软件包
由于需要实现存储设备的多路径访问及故障切换，故需要运行multipath服务。Multipath服务原理为iSCSI磁阵中的一个真实的virtual disk通过与2个冗余阵列控制器连接的端口对外提供映射，故服务器上可以看到两个物理磁盘，但此磁盘对应一个真实的一个真实的virtual disk，故这两条路径间可以实现故障切换和负载均衡。
[[email protected] ~]# yum -y install device-mapper multipath-tools device-mapper-multipath device-mapper-event #device-mapper* [[email protected] ~]# rpm -qa|grep -i mapper device-mapper-multipath-libs-0.4.9-93.el6.x86_64 device-mapper-persistent-data-0.6.2-0.1.rc7.el6.x86_64 device-mapper-libs-1.02.117-7.el6_8.1.x86_64 device-mapper-event-1.02.117-7.el6_8.1.x86_64 device-mapper-event-libs-1.02.117-7.el6_8.1.x86_64 device-mapper-multipath-0.4.9-93.el6.x86_64 device-mapper-1.02.117-7.el6_8.1.x86_64 [[email protected] ~]# lsmod | grep dm_multipath [[email protected] ~]# modprobe dm_multipath [[email protected] ~]# modprobe dm_round_robin [[email protected] ~]# lsmod | grep dm_multipath dm_multipath 17916 1 dm_round_robin dm_mod 102467 3 dm_multipath,dm_mirror,dm_log [[email protected] ~]# chkconfig multipathd on [[email protected] ~]# /etc/init.d/multipathd start Starting multipathd daemon: [ OK ] [[email protected] ~]# multipath -ll Mar 18 01:09:19 | /etc/multipath.conf does not exist, blacklisting all devices. Mar 18 01:09:19 | A sample multipath.conf file is located at Mar 18 01:09:19 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf Mar 18 01:09:19 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf [[email protected] ~]#
说明：
（1）device-mapper-multipath
提供multipathd和multipath等工具和multipath.conf等配置文件。这些工具通过device mapper的ioctr的接口创建和配置multipath设备（调用device-mapper的用户空间库。创建的多路径设备会在/dev/mapper中）
（2）device-mapper
device-mapper包括两大部分：内核部分和用户部分。
内核部分由device-mapper核心（multipath.ko）和一些target driver（dm-multipath.ko）构成。dm-mod.ko是实现multipath的基础，dm-multipath其实是dm的一个target驱动。核心完成设备的映射，而target根据映射关系和自身特点具体处理从mappered device 下来的i/o。同时，在核心部分，提供了一个接口，用户通过ioctr可和内核部分通信，以指导内核驱动的行为，比如如何创建mappered device，这些device的属性等。
用户空间部分包括device-mapper这个包。其中包括dmsetup工具和一些帮助创建和配置mappered device的库。这些库主要抽象，封装了与ioctr通信的接口，以便方便创建和配置mappered device。device-mapper-multipath的程序中就需要调用这些库。
查看多路径提示多路径软件的配置文件/etc/multipath.conf，可以使用/sbin/mpathconf创建多路径配置文件。
或者 cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf
[[email protected] ~]# /sbin/mpathconf --enable --find_multipaths y --with_multipathd y --with_chkconfig y [[email protected] ~]# /etc/init.d/multipathd restart [[email protected] ~]# multipath -F #删除现有路径 [[email protected] ~]# multipath -v2 # #格式化路径,重新扫描设备，检测路径，合并路径 [[email protected] ~]# multipath -ll #查看多路径 mpatha (36090a0a8a0823c41c48bb5a95e67e636) dm-0 EQLOGIC,100E-00 size=1000G features='0' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=1 status=active ----多路径下的两个盘符sdb和sdc. | `- 21:0:0:0 sdb 8:16 active ready running `-+- policy='round-robin 0' prio=1 status=enabled `- 20:0:0:0 sdc 8:32 active ready running [[email protected] ~]# ll /dev/mapper/ total 0 crw-rw---- 1 root root 10, 58 Mar 21 14:44 control lrwxrwxrwx 1 root root 7 Mar 21 14:57 mpatha -> ../dm-0 [[email protected] ~]#
命令查看到两条活跃路径，多路径下的两个盘符sdb和sdc.，他们之间互为A/A 关系。断掉其中一根线路，那么系统自动切换到另外一条。如果配置正确的话就会在/dev/mapper/目录下多出mpath*之类设备。
使用multipath -F删除现有路径 multipath -v2重新扫描设备，multipath -ll 查看多路径，这时会在/dev/mapper/目录下生成和别名对应的设备文件。
①/dev/mapper/mpathn 是软件虚拟出来的多路径设备，这个可以被我们用来挂载使用。
如果想生成mpath设备，则需要打开user_friendly_names选项，将配置文件中下面3行的注释取消（去掉行首的#号）
defaults {
user_friendly_names yes
}
②/dev/mpath/mpathn 这个是udev设备管理器创建的，不能用来挂载。
③/dev/dm-n 这个是软件自身使用的，不能被软件以外使用。不可挂载。
如果还是没有查看到多路径需要检查多路径配置文件 /etc/multipath.conf 中
blacklist {
devnode "*"
}
这3行注释掉，或者改成以下，sda为本地盘符
blacklist {
devnode "^sda"
}
重启multipathd服务就可以查看到多路径了。
继续编辑/etc/multipath.conf,我们给这个10G的LUN起个可以自己命名的名字lun-98.115
[[email protected] ~]# vim /etc/multipath.conf #blacklist { # wwid 26353900f02796769 # devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" # devnode "^hd[a-z]" #} multipaths { multipath { wwid 36090a0a8a0820cd2ff8c15a05f67b6df #此值multipath -v3可以看到 alias lun-98.115 #映射后的别名,可以随便取 path_grouping_policy multibus #路径组策略 path_checker readsector0 #决定路径状态的方法 path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法 failback manual #故障恢复的模式 rr_weight priorities no_path_retry 5 #在disable queue之前系统尝试使用失效路径的次数的数值 } # multipath { # wwid 1DEC_____321816758474 # alias red # } } [[email protected] ~]# /etc/init.d/multipathd restart ok Stopping multipathd daemon: [ OK ] Starting multipathd daemon: [ OK ] [[email protected] ~]# multipath -ll Mar 21 15:22:37 | /etc/multipath.conf line 72, invalid keyword: path_checker lun-98.115 (36090a0a8a0820cd2ff8c15a05f67b6df) dm-0 EQLOGIC,100E-00 size=10G features='1 queue_if_no_path' hwhandler='0' wp=ro `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:0 sdb 8:16 active ready running `- 4:0:0:0 sdc 8:32 active ready running [[email protected] ~]# ll /dev/mapper/ total 0 crw-rw---- 1 root root 10, 58 Mar 21 14:44 control lrwxrwxrwx 1 root root 7 Mar 21 15:22 lun-98.115 -> ../dm-0 [[email protected] ~]#
用 multipath –ll 命令查看到两条活跃路径，他们之间互为A/A 关系。断掉其中一根线路，那么系统自动切换到另外一条。
※ 千万不要写错path_checker（可能值有：readsector0, tur, emc_clariion, hp_sw, directio）。 不清楚的，可从存储的官方资料获得。
一个target 对应一个multipath. 如果有多个target 就写多个multipath 选项。
格式化LUN、挂载
在对多路径软件生成的磁盘进行分区之前最好运行一下pvcreate命令，将其标记为物理卷
[[email protected] ~]# fdisk -l Disk /dev/sdc: 10.7 GB, 10742661120 bytes Disk /dev/sdc: 10.7 GB, 10742661120 bytes Disk /dev/mapper/lun-98.115: 10.7 GB, 10742661120 bytes [[email protected] ~]# pvcreate /dev/mapper/lun-98.115 [[email protected] ~]# fdisk /dev/mapper/lun-98.115
用fdisk对多路径软件生成的磁盘进行分区保存时会有一个报错,此报错可以忽略。
如果分了多分区，进入/dev/mapper/只能看到完整的磁盘/dev/mapper/mpathn，此时可以以下任意操作：
1、重启IPSAN或者FCSAN的驱动，如果是用iscsi-initiator来连接IPSAN的重启ISCSI服务就可以发现所生成的磁盘分区了；
2、执行kpartx -a /dev/mapper/mpathn来添加device-mapper分区,此时可以看到第一个分区mpathnp1，第二个分区mpathnp2...
3、使用multipath –F 命令清除多路径设备缓存后，再用multipath –v3 命令重新加载。
[[email protected] ~]# partprobe /dev/mapper/lun-98.115 #重新扫描分区 [[email protected] ~]# mkfs.ext4 /dev/mapper/lun-98.115 [[email protected] ~]# mkdir /app [[email protected] ~]# mount /dev/mapper/lun-98.115 /app/ [[email protected] ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/sda3 ext4 575G 4.3G 542G 1% / tmpfs tmpfs 3.9G 72K 3.9G 1% /dev/shm /dev/sda1 ext4 190M 39M 142M 22% /boot /dev/mapper/lun-98.115 ext4 9.8G 23M 9.3G 1% /app [[email protected] ~]#
删除失效的iscsi 连接
如果因为错误配置等原因创建了iscsi 对应，系统不会自动删除，需手动删除
[[email protected] ~]# iscsiadm -m node 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test [[email protected] ~]# iscsiadm -m node --targetname iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test -u [[email protected] ~]# iscsiadm -m node -o delete -T iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test -p 10.0.0.10:3260 [[email protected] ~]# iscsiadm -m node iscsiadm: No records found [[email protected] ~]#
如果其中一条链路出现故障，那么在日志里面我们可以看到如下类似的信息：
[[email protected] ~]# tail /var/log/messages Dec 12 14:03:15 bhnfs1 kernel: iscsi: cmd 0x28 is not queued (8) Dec 12 14:03:15 bhnfs1 kernel: iscsi: cmd 0x28 is not queued (8) Dec 12 14:03:15 bhnfs1 multipathd: sdf: readsector0 checker reports path is down Dec 12 14:03:15 bhnfs1 multipathd: sdg: readsector0 checker reports path is down Dec 12 14:03:20 bhnfs1 kernel: iscsi: cmd 0x28 is not queued (8) Dec 12 14:03:20 bhnfs1 multipathd: sdc: readsector0 checker reports path is down Dec 12 14:03:20 bhnfs1 kernel: iscsi: cmd 0x28 is not queued (8) Dec 12 14:03:20 bhnfs1 multipathd: sdf: readsector0 checker reports path is down Dec 12 14:03:20 bhnfs1 kernel: iscsi: cmd 0x28 is not queued (8) Dec 12 14:03:20 bhnfs1 multipathd: sdg: readsector0 checker reports path is down
报错汇总：
[[email protected] ~]# service multipathd restart 正在关闭multipathd 端口监控程序： [失败] 正在启动守护进程multipathd：/sbin/multipathd: error while loading shared libraries:libsysfs.so.2: cannot open shared object file: No such file or directory[失败] [[email protected] ifaces]# 在/etc/ld.so.conf中加一行/usr/local/lib，.然后运行/sbin/ldconfig [[email protected] ~]# cat /etc/ld.so.conf include ld.so.conf.d/*.conf /usr/local/lib [[email protected] ~]# ldconfig [[email protected] ~]# find / -namelibsysfs.so.2 [[email protected] ~]# yum install -ylibsysfs* [[email protected] ~]# service multipathd restart 正在关闭multipathd 端口监控程序： [失败] 正在启动守护进程multipathd：/sbin/multipathd: error while loading shared libraries:libreadline.so.5: cannot open shared object file: No such file or directory[失败] [[email protected] ~]# tar-xvf readline-5.2.tar.gz [[email protected] ~]# cd cdreadline-5.2 [[email protected] cdreadline-5.2]# ./configure [[email protected] cdreadline-5.2]# make && make install [[email protected] cdreadline-5.2]# yum install -y libreadline* [[email protected] cdreadline-5.2]# service multipathd restart
[[email protected] ~]# multipath -F
Mar 21 15:34:05 | /etc/multipath.conf line 72, invalid keyword: path_checker
[[email protected] ~]#
转载于:https://blog.51cto.com/ityunwei2017/1421544