大纲
前言
一、环境准备
二、iscsi 安装与配置
三、cman与rgmanager 集群安装与配置
四、cLVM 安装与配置
五、gfs2 安装与配置
六、测试
前言
前面的几篇博客中我们一直在讲解高可用集群，但一直没有讲解共享存储，我们又说共享存储是十分重要的，在这一篇博文中我们重点来讲一下共享存储，与前面的博文有所不同的是在这一篇博文中我们重点讲解的是具体操作，理论知识不做重点说明，至于什么是RHCS集群套件、什么是iscsi，大家可以到网上去搜索一下，全都是我在这里就不重点说明。但有一点我得说明一下，有博友会说了现在共享存储不是NAS就是SAN，你这个应用范围也太小了吧，但我想说是，企业级的NAS或都SAN设备动不动就是几十万或都上百万（我这里说少了），不是什么公司都能承受的（毕竟大公司也就那几个），对于中小型企业来说，我们不想用NAS或者SAN但我们又想用共享存储，性能上不是要求太高，我想这种方案还个不错的选择。好了，引子就说到这里，下面我们来具体演示一下，这种共享存储的实现。
一、环境准备
1.操作系统
CentOS 6.4 X86_64
2.软件版本
scsi-target-utils-1.0.24-3.el6_4.x86_64
iscsi-initiator-utils-6.2.0.873-2.el6.x86_64
cman-3.0.12.1-49.el6_4.1.x86_64
rgmanager-3.0.12.1-17.el6.x86_64
gfs2-utils-3.0.12.1-49.el6_4.1.x86_64
lvm2-cluster-2.02.98-9.el6.x86_64
3.实验拓扑
简单说明：RHCS集群套件，要求节点最少得3个。所以这里有3个集群节点和一个共享存储。（注，shared storage不但是共享存储，还是跳板机，且shared storage的主机名是target.test.com）
4.集群环境
(1).配置各节点名称
node1:
[[email protected] ~]# uname -n node1.test.com [[email protected] ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.18.201 node1.test.com node1 192.168.18.202 node2.test.com node2 192.168.18.203 node3.test.com node3 192.168.18.208 target.test.com target
node2:
[[email protected] ~]# uname -n node2.test.com [[email protected] ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.18.201 node1.test.com node1 192.168.18.202 node2.test.com node2 192.168.18.203 node3.test.com node3 192.168.18.208 target.test.com target
node3:
[[email protected] ~]# uname -n node3.test.com [[email protected] ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.18.201 node1.test.com node1 192.168.18.202 node2.test.com node2 192.168.18.203 node3.test.com node3 192.168.18.208 target.test.com target
shared storage:
[[email protected] ~]# uname -n target.test.com [[email protected] ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.18.201 node1.test.com node1 192.168.18.202 node2.test.com node2 192.168.18.203 node3.test.com node3 192.168.18.208 target.test.com target
(2).配置各节点与跳板机ssh互信
node1:
[[email protected] ~]# ssh-****** -t rsa -f ~/.ssh/id_rsa -P '' [[email protected] ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected]
node2:
[[email protected] ~]# ssh-****** -t rsa -f ~/.ssh/id_rsa -P '' [[email protected] ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected]
node3:
[[email protected] ~]# ssh-****** -t rsa -f ~/.ssh/id_rsa -P '' [[email protected] ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected]
shared storage:
[[email protected] ~]# ssh-****** -t rsa -f ~/.ssh/id_rsa -P '' [[email protected] ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected] [[email protected] ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected] [[email protected] ~]# ssh-copy-id -i .ssh/id_rsa.pub [email protected]
(3).配置各节点时间同步
node1:
[[email protected] ~]# ntpdate 202.120.2.101
node2:
[[email protected] ~]# ntpdate 202.120.2.101
node3:
[[email protected] ~]# ntpdate 202.120.2.101
shared storage:
[[email protected] ~]# ntpdate 202.120.2.101
大家有没有发现我们时间同步的操作包括下面的很操作都是相同的有没有一种方法，只要执行一次，方法有很多，我们来说一下最常用的方法，ssh。在上面的操作中我们已经配置了ssh互信，下面我们就在跳板机上操作一下。
[[email protected] ~]# alias ha='for I in {1..3}; do' #设置一个别名，因为每次都得用到时 [[email protected] ~]# ha ssh node$I 'ntpdate 202.120.2.101'; done #各节点都在时间同步 20 Aug 14:32:40 ntpdate[14752]: adjust time server 202.120.2.101 offset -0.019162 sec 20 Aug 14:32:41 ntpdate[11994]: adjust time server 202.120.2.101 offset 0.058863 sec 20 Aug 14:32:43 ntpdate[1578]: adjust time server 202.120.2.101 offset 0.062831 sec
注：大家看到了吧，这就是配置跳板机的好处，配置只要执行一次。
(5).安装yum源
[[email protected] ~]# ha ssh node$I 'rpm -ivh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm'; done [[email protected] ~]# ha ssh node$I 'rpm -ivh http://elrepo.org/elrepo-release-6-5.el6.elrepo.noarch.rpm'; done
(6).关闭防火墙与SELinux
[[email protected] ~]# ha ssh node$I 'service iptables stop'; done
node1:
[[email protected] ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of these two values: # targeted - Targeted processes are protected, # mls - Multi Level Security protection. SELINUXTYPE=targeted
node2:
[[email protected] ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of these two values: # targeted - Targeted processes are protected, # mls - Multi Level Security protection. SELINUXTYPE=targeted
node3:
[[email protected] ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of these two values: # targeted - Targeted processes are protected, # mls - Multi Level Security protection. SELINUXTYPE=targeted
好了，环境准备全部完成，下面我们来配置一下iscsi。
二、iscsi 安装与配置
1.安装target
[[email protected] ~]# yum install -y scsi-target-utils
2.配置target
[[email protected] ~]# vim /etc/tgt/targets.conf #<target iqn.2008-09.com.example:server.target2> # direct-store /dev/sdd # incominguser someuser secretpass12 #</target> <target iqn.2013-08.com.test:teststore.sdb> #配置target名称 <backing-store /dev/sdb> #配置共享磁盘 vendor_id test #配置发行商（任意） lun 6 #配置LUN号 </backing-store> incominguser iscsiuser iscsiuser #配置认证的用户名和密码 initiator-address 192.168.18.0/24 #配置允许的网段 </target>
3.启动target并设置为开机自启动
[[email protected] ~]# service tgtd start [[email protected] ~]# chkconfig tgtd on [[email protected] ~]# chkconfig tgtd --list tgtd 0:关闭 1:关闭 2:启用 3:启用 4:启用 5:启用 6:关闭
4.查看配置的target
[[email protected] ~]# tgtadm --lld iscsi --mode target --op show Target 1: iqn.2013-08.com.test:teststore.sdb System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: null Backing store path: None Backing store flags: Account information: iscsiuser ACL information: 192.168.18.0/24
5.在各节点上安装initiator
[[email protected] ~]# ha ssh node$I 'yum install -y initiator'; done
6.配置initiator
node1:
[[email protected] ~]# vim /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.2013-08.com.test:node1 [[email protected] ~]# vim /etc/iscsi/iscsid.conf #修改下面三项 node.session.auth.authmethod = CHAP #开启CHAP认证 node.session.auth.username = iscsiuser #配置认证用户名 node.session.auth.password = iscsiuser #配置认证密码
node2:
[[email protected] ~]# vim /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.2013-08.com.test:node2 [[email protected]~]# vim /etc/iscsi/iscsid.conf #修改下面三项 node.session.auth.authmethod = CHAP #开启CHAP认证 node.session.auth.username = iscsiuser #配置认证用户名 node.session.auth.password = iscsiuser #配置认证密码
node3:
[[email protected] ~]# vim /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.2013-08.com.test:node3 [[email protected] ~]# vim /etc/iscsi/iscsid.conf #修改下面三项 node.session.auth.authmethod = CHAP #开启CHAP认证 node.session.auth.username = iscsiuser #配置认证用户名 node.session.auth.password = iscsiuser #配置认证密码
7.各节点启动initiator并设置为开机自启动
[[email protected] ~]# ha ssh node$I 'service iscsi start'; done [[email protected] ~]# ha ssh node$I 'chkconfig iscsi on'; done [[email protected] ~]# ha ssh node$I 'chkconfig iscsi --list'; done iscsi 0:关闭 1:关闭 2:启用 3:启用 4:启用 5:启用 6:关闭 iscsi 0:关闭 1:关闭 2:启用 3:启用 4:启用 5:启用 6:关闭 iscsi 0:关闭 1:关闭 2:启用 3:启用 4:启用 5:启用 6:关闭
8.在各节点上发现一下target
[[email protected] ~]# ha ssh node$I 'iscsiadm -m discovery -t st -p 192.168.18.208:3260'; done 192.168.18.208:3260,1 iqn.2013-08.com.test:teststore.sdb 192.168.18.208:3260,1 iqn.2013-08.com.test:teststore.sdb 192.168.18.208:3260,1 iqn.2013-08.com.test:teststore.sdb
9.各节点登录一下target并查看一下磁盘
[[email protected] ~]# ha ssh node$I 'iscsiadm -m node -T iqn.2013-08.com.test:teststore.sdb -p 192.168.18.208 -l'; done [[email protected] ~]# ha ssh node$I 'fdisk -l'; done Disk /dev/sda: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000dfceb Device Boot Start End Blocks Id System /dev/sda1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 26 1301 10240000 83 Linux /dev/sda3 1301 1938 5120000 83 Linux /dev/sda4 1938 2611 5405696 5 Extended /dev/sda5 1939 2066 1024000 82 Linux swap / Solaris Disk /dev/sdb: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x5f3b697c Device Boot Start End Blocks Id System Disk /dev/sdd: 21.5 GB, 21474836480 bytes 64 heads, 32 sectors/track, 20480 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0c68b5e3 Device Boot Start End Blocks Id System Disk /dev/sda: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000dfceb Device Boot Start End Blocks Id System /dev/sda1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 26 1301 10240000 83 Linux /dev/sda3 1301 1938 5120000 83 Linux /dev/sda4 1938 2611 5405696 5 Extended /dev/sda5 1939 2066 1024000 82 Linux swap / Solaris Disk /dev/sdb: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sdd: 21.5 GB, 21474836480 bytes 64 heads, 32 sectors/track, 20480 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0c68b5e3 Device Boot Start End Blocks Id System Disk /dev/sda: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000dfceb Device Boot Start End Blocks Id System /dev/sda1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 26 1301 10240000 83 Linux /dev/sda3 1301 1938 5120000 83 Linux /dev/sda4 1938 2611 5405696 5 Extended /dev/sda5 1939 2066 1024000 82 Linux swap / Solaris Disk /dev/sdb: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sdd: 21.5 GB, 21474836480 bytes 64 heads, 32 sectors/track, 20480 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0c68b5e3 Device Boot Start End Blocks Id System
好了，到这里iscsi配置全部完成，下面我们来配置一下，集群。
三、cman与rgmanager 集群安装与配置
1.各节点安装cman与rgmanager
[[email protected] ~]# ha ssh node$I 'yum install -y cman rgmanager'; done
2.配置集群
(1).配置集群名称
[[email protected] ~]# ccs_tool create testcluster
(2).配置fencing设备
[[email protected] ~]# ccs_tool addfence meatware fence_manual [[email protected] ~]# ccs_tool lsfence Name Agent meatware fence_manual
(3).配置集群节点
[[email protected] ~]# ccs_tool addnode -n 1 -f meatware node1.test.com [[email protected] ~]# ccs_tool addnode -n 2 -f meatware node2.test.com [[email protected] ~]# ccs_tool addnode -n 3 -f meatware node3.test.com [[email protected] ~]# ccs_tool lsnode Cluster name: testcluster, config_version: 5 Nodename Votes Nodeid Fencetype node1.test.com 1 1 meatware node2.test.com 1 2 meatware node3.test.com 1 3 meatware
3.同步配置文件到各节点
[[email protected] cluster]# scp cluster.conf [email protected]:/etc/cluster/ [[email protected] cluster]# scp cluster.conf [email protected]:/etc/cluster/
4.启动各节点集群
node1:
[[email protected] cluster]# service cman start Starting cluster: Checking if cluster has been disabled at boot... [确定] Checking Network Manager... [确定] Global setup... [确定] Loading kernel modules... [确定] Mounting configfs... [确定] Starting cman... [确定] Waiting for quorum... [确定] Starting fenced... [确定] Starting dlm_controld... [确定] Tuning DLM kernel config... [确定] Starting gfs_controld... [确定] Unfencing self... [确定] Joining fence domain... [确定]
node2:
[[email protected] cluster]# service cman start Starting cluster: Checking if cluster has been disabled at boot... [确定] Checking Network Manager... [确定] Global setup... [确定] Loading kernel modules... [确定] Mounting configfs... [确定] Starting cman... [确定] Waiting for quorum... [确定] Starting fenced... [确定] Starting dlm_controld... [确定] Tuning DLM kernel config... [确定] Starting gfs_controld... [确定] Unfencing self... [确定] Joining fence domain... [确定]
node3:
[[email protected] cluster]# service cman start Starting cluster: Checking if cluster has been disabled at boot... [确定] Checking Network Manager... [确定] Global setup... [确定] Loading kernel modules... [确定] Mounting configfs... [确定] Starting cman... [确定] Waiting for quorum... [确定] Starting fenced... [确定] Starting dlm_controld... [确定] Tuning DLM kernel config... [确定] Starting gfs_controld... [确定] Unfencing self... [确定] Joining fence domain... [确定]
5.查看各节点启动端口
node1:
[[email protected] cluster]# netstat -ntulp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1082/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1158/master tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 14610/sshd tcp 0 0 :::22 :::* LISTEN 1082/sshd tcp 0 0 ::1:25 :::* LISTEN 1158/master tcp 0 0 ::1:6010 :::* LISTEN 14610/sshd udp 0 0 192.168.18.201:5404 0.0.0.0:* 15583/corosync udp 0 0 192.168.18.201:5405 0.0.0.0:* 15583/corosync udp 0 0 239.192.47.48:5405 0.0.0.0:* 15583/corosync
node2:
[[email protected] cluster]# netstat -ntulp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1082/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1158/master tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 14610/sshd tcp 0 0 :::22 :::* LISTEN 1082/sshd tcp 0 0 ::1:25 :::* LISTEN 1158/master tcp 0 0 ::1:6010 :::* LISTEN 14610/sshd udp 0 0 192.168.18.201:5404 0.0.0.0:* 15583/corosync udp 0 0 192.168.18.201:5405 0.0.0.0:* 15583/corosync udp 0 0 239.192.47.48:5405 0.0.0.0:* 15583/corosync
node3:
[[email protected] cluster]# netstat -ntulp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1082/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1158/master tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 14610/sshd tcp 0 0 :::22 :::* LISTEN 1082/sshd tcp 0 0 ::1:25 :::* LISTEN 1158/master tcp 0 0 ::1:6010 :::* LISTEN 14610/sshd udp 0 0 192.168.18.201:5404 0.0.0.0:* 15583/corosync udp 0 0 192.168.18.201:5405 0.0.0.0:* 15583/corosync udp 0 0 239.192.47.48:5405 0.0.0.0:* 15583/corosync
好了，到此集群配置完成，下面我们来配置cLVM。
四、cLVM 安装与配置
1.安装cLVM
[[email protected] ~]# ha ssh node$I 'yum install -y lvm2-cluster'; done
2.启用集群LVM
[[email protected] ~]# ha ssh node$I 'lvmconf --enable-cluster'; done
3.查看一下启用的集群LVM
[[email protected] ~]# ha ssh node$I 'grep "locking_type = 3" /etc/lvm/lvm.conf'; done locking_type = 3 locking_type = 3 locking_type = 3
注：所有节点启用完成。
4.启动cLVM服务
[[email protected] ~]# ha ssh node$I 'service clvmd start'; done Starting clvmd: Activating VG(s): No volume groups found [确定] Starting clvmd: Activating VG(s): No volume groups found [确定] Starting clvmd: Activating VG(s): No volume groups found [确定]
5.将各节点的cman rgmanger clvmd设置为开机自启动
[[email protected] ~]# ha ssh node$I 'chkconfig clvmd on'; done [[email protected] ~]# ha ssh node$I 'chkconfig cman on'; done [[email protected] ~]# ha ssh node$I 'chkconfig rgmanager on'; done
6.在集群节点上创建lvm
node1:
(1).查看一下共享存储
[[email protected] ~]# fdisk -l #查看一下共享存储 Disk /dev/sda: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000dfceb Device Boot Start End Blocks Id System /dev/sda1 * 1 26 204800 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 26 1301 10240000 83 Linux /dev/sda3 1301 1938 5120000 83 Linux /dev/sda4 1938 2611 5405696 5 Extended /dev/sda5 1939 2066 1024000 82 Linux swap / Solaris Disk /dev/sdb: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x5f3b697c Device Boot Start End Blocks Id System Disk /dev/sdd: 21.5 GB, 21474836480 bytes 64 heads, 32 sectors/track, 20480 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0c68b5e3 Device Boot Start End Blocks Id System
(2).创建集群逻辑卷
[[email protected] ~]# pvcreate /dev/sdd #创建物理卷 Physical volume "/dev/sdd" successfully created [[email protected] ~]# pvs PV VG Fmt Attr PSize PFree /dev/sdd lvm2 a-- 20.00g 20.00g [[email protected] ~]# vgcreate clustervg /dev/sdd #创建卷组 Clustered volume group "clustervg" successfully created [[email protected] ~]# vgs VG #PV #LV #SN Attr VSize VFree clustervg 1 0 0 wz--nc 20.00g 20.00g [[email protected] ~]# lvcreate -L 10G -n clusterlv clustervg #创建逻辑卷 Logical volume "clusterlv" created [[email protected] ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert clusterlv clustervg -wi-a---- 10.00g 7.在node2与node3上查看一下创建的逻辑卷 node2: [[email protected] ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert clusterlv clustervg -wi-a---- 10.00g node3: [[email protected] ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert clusterlv clustervg -wi-a---- 10.00g
好了，clvm到这里配置全部完成，下面我们将生成的逻辑卷格式成集群文件系统（gfs）。
五、gfs2 安装与配置
1.安装gfs2
[[email protected] ~]# ha ssh node$I 'yum install -y gfs2-utils'; done
2.查看一下帮助文件
[[email protected] ~]# mkfs.gfs2 -h Usage: mkfs.gfs2 [options] <device> [ block-count ] Options: -b <bytes> Filesystem block size -c <MB> Size of quota change file -D Enable debugging code -h Print this help, then exit -J <MB> Size of journals -j <num> Number of journals -K Don't try to discard unused blocks -O Don't ask for confirmation -p <name> Name of the locking protocol -q Don't print anything -r <MB> Resource Group Size -t <name> Name of the lock table -u <MB> Size of unlinked file -V Print program version information, then exit
注：对于我们用到的参数进行说明
-j # 指定日志区域的个数，有几个就能够被几个节点挂载
-J # 指定日志区域的大小，默认为128MB
-p {lock_dlm|lock_nolock}
-t <name>: 锁表的名称，格式为clustername:locktablename
3.格式化为集群文件系统
[[email protected] ~]# mkfs.gfs2 -j 2 -p lock_dlm -t testcluster:sharedstorage /dev/clustervg/clusterlv This will destroy any data on /dev/clustervg/clusterlv. It appears to contain: symbolic link to `../dm-0' Are you sure you want to proceed? [y/n] y Device: /dev/clustervg/clusterlv Blocksize: 4096 Device Size 10.00 GB (2621440 blocks) Filesystem Size: 10.00 GB (2621438 blocks) Journals: 2 Resource Groups: 40 Locking Protocol: "lock_dlm" Lock Table: "testcluster:sharedstorage" UUID: 60825032-b995-1970-2547-e95420bd1c7c
注：testcluster是集群名称，sharedstorage为锁表名称
4.创建挂载目录并挂载
[[email protected] ~]# mkdir /mydata [[email protected] ~]# mount -t gfs2 /dev/clustervg/clusterlv /mydata [[email protected] ~]# cd /mydata/ [[email protected] mydata]# ll 总用量 0
5.将node2与node3进行挂载
node2:
[[email protected] ~]# mkdir /mydata [[email protected] ~]# mount -t gfs2 /dev/clustervg/clusterlv /mydata [[email protected] ~]# cd /mydata/ [[email protected] mydata]# ll 总用量 0
node3:
[[email protected] ~]# mkdir /mydata [[email protected] ~]# mount -t gfs2 /dev/clustervg/clusterlv /mydata Too many nodes mounting filesystem, no free journals
注：大家可以看到，node2成功挂载而node3没有功功挂载，Too many nodes mounting filesystem, no free journals，没有多于的日志空间。因为我们在格式化时只创建了2个日志文件，所以node1与node2可以挂载，而node3无法挂载，至于怎么解决我们下面会说明。现在我们来测试一下集群文件系统。
六、测试
1.查看是否能快速同步文件
node1:
[[email protected] mydata]# touch 123.txt [[email protected] mydata]# ll 总用量 4 -rw-r--r-- 1 root root 0 8月 20 16:13 123.txt [[email protected] mydata]# ll 总用量 8 -rw-r--r-- 1 root root 0 8月 20 16:13 123.txt -rw-r--r-- 1 root root 0 8月 20 16:14 456.txt
node2:
[[email protected] mydata]# ll 总用量 4 -rw-r--r-- 1 root root 0 8月 20 16:13 123.txt [[email protected] mydata]# touch 456.txt [[email protected] mydata]# ll 总用量 8 -rw-r--r-- 1 root root 0 8月 20 16:13 123.txt -rw-r--r-- 1 root root 0 8月 20 16:14 456.txt
注：我们可以看到文件可以快速同步，直面我们来看一下挂载目录属性
2.查看挂载目录的属性
[[email protected] mydata]# gfs2_tool gettune /mydata incore_log_blocks = 8192 log_flush_secs = 60 quota_warn_period = 10 quota_quantum = 60 max_readahead = 262144 complain_secs = 10 statfs_slow = 0 quota_simul_sync = 64 statfs_quantum = 30 quota_scale = 1.0000 (1, 1) new_files_jdata = 0 #最常用，设置是否立刻同步到磁盘的，一般设置为1，下面我们就来设置一下 [[email protected] mydata]# gfs2_tool settune /mydata new_files_jdata 1 [[email protected] mydata]# gfs2_tool gettune /mydata incore_log_blocks = 8192 log_flush_secs = 60 quota_warn_period = 10 quota_quantum = 60 max_readahead = 262144 complain_secs = 10 statfs_slow = 0 quota_simul_sync = 64 statfs_quantum = 30 quota_scale = 1.0000 (1, 1) new_files_jdata = 1
3.查看一下日志文件
[[email protected] mydata]# gfs2_tool journals /mydata journal1 - 128MB journal0 - 128MB 2 journal(s) found.
注，大家可以看到只有两个日志文件，默认为128MB，下面我们来新增一个日志文件，并将node3挂载上
4.新增日志文件并挂载
[[email protected] ~]# gfs2_jadd -j 1 /dev/clustervg/clusterlv Filesystem: /mydata Old Journals 2 New Journals 3 [[email protected] ~]# gfs2_tool journals /mydata journal2 - 128MB journal1 - 128MB journal0 - 128MB 3 journal(s) found. [[email protected] ~]# mount -t gfs2 /dev/clustervg/clusterlv /mydata [[email protected] ~]# cd /mydata/ [[email protected] mydata]# ll 总用量 8 -rw-r--r-- 1 root root 0 8月 20 16:13 123.txt -rw-r--r-- 1 root root 0 8月 20 16:14 456.txt
注，大家可以看到node3顺利挂载上
5.最后和大家说一下怎么扩展集群逻辑卷
(1).先查看一下大小
[[email protected] ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert clusterlv clustervg -wi-ao--- 10.00g
注，现在是10个G，下面我们将其扩展到15G。
(2).扩展物理边界
[[email protected] ~]# lvextend -L 15G /dev/clustervg/clusterlv Extending logical volume clusterlv to 15.00 GiB Logical volume clusterlv successfully resized [[email protected] ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert clusterlv clustervg -wi-ao--- 15.00g
(3).扩展逻辑边界
[[email protected] ~]# gfs2_grow /dev/clustervg/clusterlv FS: Mount Point: /mydata FS: Device: /dev/dm-0 FS: Size: 2621438 (0x27fffe) FS: RG size: 65533 (0xfffd) DEV: Size: 3932160 (0x3c0000) The file system grew by 5120MB. gfs2_grow complete. [[email protected] ~]# [[email protected] ~]# df -h 文件系统 容量 已用 可用 已用%% 挂载点 /dev/sda2 9.7G 1.5G 7.7G 17% / tmpfs 116M 29M 88M 25% /dev/shm /dev/sda1 194M 26M 159M 14% /boot /dev/sda3 4.9G 138M 4.5G 3% /data /dev/sdc1 5.0G 138M 4.6G 3% /mnt /dev/mapper/clustervg-clusterlv 15G 388M 15G 3% /mydata
注，大家可以看到，现在是15G了，嘿嘿。好了，CentOS6.4+cman+rgmanager+iscsi+gfs2+cLVM 实现廉价的可扩展的集群共享存储的所有演示全部完成，希望大家有所收获。^_^……
转载于:https://blog.51cto.com/freeloda/1279063