author：skate
tiome：2013/12/06
lvs+ldirectord+pacemaker+corosync+mysql实现高可用负载均衡
环境说明
vip：192.168.213.97
rip1：192.168.213.91
rip2：192.168.213.92
client：192.168.213.93
os：centos6.3
kernel：2.6.32-279.el6.x86_64
mysql version：5.5.28
1. ldirectord安装与配置
ldirectord下载
下载地址：http://horms.net/projects/ldirectord/download.shtml
# unzip resource-agents-master.zip
# cd resource-agents-master
# ./autogen.sh
# ./configure
# make && make install
此时ldirectord还是不能使用，比如查看状态，如下：
[root@localhost resource-agents-master]# service ldirectord status
Can't locate Socket6.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/sbin/ldirectord line 838.
BEGIN failed--compilation aborted at /usr/sbin/ldirectord line 838.
[root@localhost resource-agents-master]#
需要安装一些perl的包
[root@localhost resource-agents-master]# yum install perl-Socket6 perl-IO-Socket-INET6 perl-Email-Date-Format perl-TimeDate perl-Pod-Escapes perl-Pod-Simple perl-Test-Pod perl-MailTools perl-libwww-perl
....
....
Dependencies Resolved
====================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================
Installing:
perl-IO-Socket-INET6 noarch 2.56-4.el6 base 17 k
perl-Socket6 x86_64 0.23-4.el6 base 27 k
perl-Test-Pod noarch 1.40-1.el6 base 14 k
Updating:
perl-Pod-Escapes x86_64 1:1.04-136.el6 base 32 k
perl-Pod-Simple x86_64 1:3.13-136.el6 base 212 k
Installing for dependencies:
db4-cxx x86_64 4.7.25-18.el6_4 base 588 k
db4-devel x86_64 4.7.25-18.el6_4 base 6.6 M
gdbm-devel x86_64 1.8.0-36.el6 base 25 k
perl-CGI x86_64 3.51-136.el6 base 209 k
Updating for dependencies:
db4 x86_64 4.7.25-18.el6_4 base 563 k
db4-utils x86_64 4.7.25-18.el6_4 base 130 k
perl x86_64 4:5.10.1-136.el6 base 10 M
perl-Archive-Extract x86_64 1:0.38-136.el6 base 40 k
perl-Archive-Tar x86_64 1.58-136.el6 base 73 k
perl-CPAN x86_64 1.9402-136.el6 base 246 k
perl-CPANPLUS x86_64 0.88-136.el6 base 307 k
perl-Compress-Raw-Bzip2 x86_64 2.021-136.el6 base 45 k
perl-Compress-Raw-Zlib x86_64 1:2.021-136.el6 base 69 k
perl-Compress-Zlib x86_64 2.021-136.el6 base 45 k
perl-Digest-SHA x86_64 1:5.47-136.el6 base 64 k
perl-ExtUtils-CBuilder x86_64 1:0.27-136.el6 base 48 k
perl-ExtUtils-Embed x86_64 1.28-136.el6 base 31 k
perl-ExtUtils-MakeMaker x86_64 6.55-136.el6 base 293 k
perl-ExtUtils-ParseXS x86_64 1:2.2003.0-136.el6 base 45 k
perl-File-Fetch x86_64 0.26-136.el6 base 41 k
perl-IO-Compress-Base x86_64 2.021-136.el6 base 69 k
perl-IO-Compress-Bzip2 x86_64 2.021-136.el6 base 48 k
perl-IO-Compress-Zlib x86_64 2.021-136.el6 base 135 k
perl-IO-Zlib x86_64 1:1.09-136.el6 base 33 k
perl-IPC-Cmd x86_64 1:0.56-136.el6 base 46 k
perl-Locale-Maketext-Simple x86_64 1:0.18-136.el6 base 31 k
perl-Log-Message x86_64 1:0.02-136.el6 base 46 k
perl-Log-Message-Simple x86_64 0.04-136.el6 base 28 k
perl-Module-Build x86_64 1:0.3500-136.el6 base 229 k
perl-Module-CoreList x86_64 2.18-136.el6 base 65 k
perl-Module-Load x86_64 1:0.16-136.el6 base 28 k
perl-Module-Load-Conditional x86_64 0.30-136.el6 base 34 k
perl-Module-Loaded x86_64 1:0.02-136.el6 base 27 k
perl-Module-Pluggable x86_64 1:3.90-136.el6 base 40 k
perl-Object-Accessor x86_64 1:0.34-136.el6 base 37 k
perl-Package-Constants x86_64 1:0.02-136.el6 base 26 k
perl-Params-Check x86_64 1:0.26-136.el6 base 35 k
perl-Parse-CPAN-Meta x86_64 1:1.40-136.el6 base 29 k
perl-Term-UI x86_64 0.20-136.el6 base 39 k
perl-Test-Harness x86_64 3.17-136.el6 base 231 k
perl-Test-Simple x86_64 0.92-136.el6 base 112 k
perl-Time-HiRes x86_64 4:1.9721-136.el6 base 48 k
perl-Time-Piece x86_64 1.15-136.el6 base 46 k
perl-core x86_64 5.10.1-136.el6 base 23 k
perl-devel x86_64 4:5.10.1-136.el6 base 423 k
perl-libs x86_64 4:5.10.1-136.el6 base 578 k
perl-parent x86_64 1:0.221-136.el6 base 27 k
perl-suidperl x86_64 4:5.10.1-136.el6 base 50 k
perl-version x86_64 3:0.77-136.el6 base 51 k
zlib x86_64 1.2.3-29.el6 base 73 k
zlib-devel x86_64 1.2.3-29.el6 base 44 k
Transaction Summary
====================================================================================================================================
Install 7 Package(s)
Upgrade 49 Package(s)
Total download size: 23 M
....
....
继续看看ldirectord是否可用
[root@localhost resource-agents-master]# service ldirectord status
Can not find ipvsadm at /usr/sbin/ldirectord line 887.
[root@localhost resource-agents-master]#
发现没有安装ipvsadm，安装如下：
# yum install ipvsadm
....
Dependencies Resolved
====================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================
Installing:
ipvsadm x86_64 1.26-2.el6 base 41 k
Transaction Summary
====================================================================================================================================
Install 1 Package(s)
Total download size: 41 k
....
....
[root@localhost resource-agents-master]# service ldirectord status
Config file ldirectord.cf not found
想查看配置文件放在何处，可以查看文件“/etc/init.d/ldirectord”和“/usr/sbin/ldirectord”；默认是放在“/etc/ha.d/ldirectord.cf”。把在安装目录的样本配置复制到“/etc/ha.d/”下，然后再修改。这时ldirectord服务才可以正常工作
这回可用了
[root@localhost resource-agents-master]# service ldirectord status
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[root@localhost resource-agents-master]# service ldirectord status
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[root@localhost resource-agents-master]# service ldirectord start
Starting ldirectord... success
[root@localhost resource-agents-master]# service ldirectord status
ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 28911
[root@localhost resource-agents-master]#
安装参考：
http://blog.chinaunix.net/uid-17268883-id-3145428.html
2. lvs安装
LVS软件包括二部分：
A.IPVS模块，LVS已经是Linux标准内核的一部分,直接被编译在内核中!
B.IPVS管理工具ipvsadm
检测IPVS模块是否真的编译到内核中去了
[root@master mysqlinstall]# modprobe -l |grep ipvs
kernel/net/netfilter/ipvs/ip_vs.ko
kernel/net/netfilter/ipvs/ip_vs_rr.ko
kernel/net/netfilter/ipvs/ip_vs_wrr.ko
kernel/net/netfilter/ipvs/ip_vs_lc.ko
kernel/net/netfilter/ipvs/ip_vs_wlc.ko
kernel/net/netfilter/ipvs/ip_vs_lblc.ko
kernel/net/netfilter/ipvs/ip_vs_lblcr.ko
kernel/net/netfilter/ipvs/ip_vs_dh.ko
kernel/net/netfilter/ipvs/ip_vs_sh.ko
kernel/net/netfilter/ipvs/ip_vs_sed.ko
kernel/net/netfilter/ipvs/ip_vs_nq.ko
kernel/net/netfilter/ipvs/ip_vs_ftp.ko
ipvsadm管理工具安装上面介绍了，通过yum安装比较方便，可以解决很多依赖关系，如下：
# yum install ipvsadm
安装参考：
http://beyondhdf.blog.51cto.com/229452/1331874
http://soulboy.blog.51cto.com/4007306/1299896
http://zh.linuxvirtualserver.org/node/98
http://itnihao.blog.51cto.com/1741976/744237
Directorer的lvsdr脚本
[root@localhost mysqlinstall]# more lvsdr.sh
###############################################################
#Director script:
################################################################
#!/bin/bash
#
# LVS script for VS/DR
#
# Startup script handle the initialisation of LVS
# chkconfig: - 28 72
# description: Initialise director of the Linux Virtual Server
#
# Author: zxg
# CreateTime: 2013/12/06
#
# Modificator:
# zxg: repair bug
#
#
# Source function library
. /etc/rc.d/init.d/functions
#
VIP=192.168.213.97
RIP1=192.168.213.91
RIP2=192.168.213.92
ETHx=eth0:3
PORT=3306
#
start() {
/sbin/ifconfig $ETHx $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev $ETHx
# Since this is the Director we must be able to forward packets
echo 1 > /proc/sys/net/ipv4/ip_forward
# Clear all iptables rules.
/sbin/iptables -F
# Reset iptables counters.
/sbin/iptables -Z
# Clear all ipvsadm rules/services.
/sbin/ipvsadm -C
# Add an IP virtual service for VIP 192.168.0.219 port $PORT
# In this recipe, we will use the round-robin scheduling method.
# In production, however, you should use a weighted, dynamic scheduling method.
/sbin/ipvsadm -A -t $VIP:$PORT -s wrr
# Now direct packets for this VIP to
# the real server IP (RIP) inside the cluster
/sbin/ipvsadm -a -t $VIP:$PORT -r $RIP1 -g -w 1
/sbin/ipvsadm -a -t $VIP:$PORT -r $RIP2 -g -w 3
/bin/touch /var/lock/subsys/ipvsadm &> /dev/null
}
stop() {
# Stop forwarding packets
echo 0 > /proc/sys/net/ipv4/ip_forward
# Reset ipvsadm
/sbin/ipvsadm -C
# Bring down the VIP interface
/sbin/ifconfig $ETHx down
/sbin/route del $VIP
/bin/rm -f /var/lock/subsys/ipvsadm
echo "ipvs stopped..."
}
status() {
if [ ! -e /var/lock/subsys/ipvsadm ]; then
echo "ipvsadm stopped ..."
else
echo "ipvs is running ..."
ipvsadm -L -n
fi
}
restart() {
stop
start
}
case "$1" in
start)
[ -f "/var/lock/subsys/ipvsadm" ] && exit 0
start
;;
stop)
stop
;;
restart)
restart
;;
status)
status
;;
*)
echo "Usage: $0 {start|stop|status|restart}"
;;
esac
realserver的运行脚本
[root@master mysqlinstall]# more lvsrl.sh
#############################################################
#RealServer脚本:
##############################################################
#!/bin/bash
#
# Script to start LVS DR real server.
# description: LVS DR real server
#
# Author: zxg
# CreateTime: 2013/12/06
#
# Modificator:
# zxg: repair bug
#
#
. /etc/rc.d/init.d/functions
VIP=192.168.213.97
host=`/bin/hostname`
case "$1" in
start)
# Start LVS-DR real server on this machine.
/sbin/ifconfig lo down
/sbin/ifconfig lo up
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
;;
stop)
# Stop LVS-DR real server loopback device(s).
/sbin/ifconfig lo:0 down
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
;;
status)
# Status of LVS-DR real server.
islothere=`/sbin/ifconfig lo:0 | grep $VIP`
isrothere=`netstat -rn | grep "lo:0" | grep $VIP`
if [ ! "$islothere" -o ! "isrothere" ];then
# Either the route or the lo:0 device
# not found.
echo "LVS-DR real server Stopped."
else
echo "LVS-DR real server Running."
fi
;;
*)
# Invalid entry.
echo "$0: Usage: $0 {start|status|stop}"
exit 1
;;
esac
先把脚本lvsrl.sh部署到所有的realserver上，并启动
然后把脚本lvsdr.sh部署到director上，并启动
可以用如下两个命令查看lvs的运行状态
# ip addr show dev eth0 //查看ip的漂移情况
# ipvsadm -ln //查看转发配置表
例如：
[root@localhost mysqlinstall]# ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:b8:7a:78 brd ff:ff:ff:ff:ff:ff
inet 192.168.213.90/24 brd 192.168.213.255 scope global eth0
inet 192.168.213.97/32 brd 192.168.213.97 scope global eth0:3
inet6 fe80::250:56ff:feb8:7a78/64 scope link
valid_lft forever preferred_lft forever
[root@localhost mysqlinstall]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.213.97:3306 rr persistent 2
-> 192.168.213.91:3306 Route 1 0 0
-> 192.168.213.92:3306 Route 1 0 40
[root@localhost mysqlinstall]#
在客户端写个循环连接的脚本
[root@slave93 mysqlinstall]# more load_mysql.sh
while (true)
do
mysql -uroot -proot -h192.168.213.97 -e "select count(*) from test.a;"
sleep 3
done
在客户端执行脚本
[root@slave93 mysqlinstall]# sh load_mysql.sh
在director上查看连接情况
# ipvsadm -lcn
上面的lvs是采用的持久连接的方式，轮询的均衡算法；对于使用数据库大都会使用持久连接（300s），可以采用wrr的均衡算法
上面的director是独立的主机，为了节省成本，可以把director和某一台realserver部署同一台上，比如部署到91上
[root@master mysqlinstall]# sh lvsdr.sh status
ipvs is running ...
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.213.97:3306 wrr
-> 192.168.213.91:3306 Local 1 0 0
-> 192.168.213.92:3306 Route 3 0 0
到现在，lvs可以正常工作，可以实现负载均衡，但后端某个realserver有问题时，lvs把请求分配器上，就会有问题，这时就需要配置
前面我们安装的ldirectord服务，其可以自动监控后端realserver的健康状态，发现有问题时就更新director的虚拟服务器和RS链表。即自动从集群中摘除，待以后故障恢复，ldirectord服务会自动把其加入集群中
按如下修改ldirectord的配置文件
[root@master mysqlinstall]# more /etc/ha.d/ldirectord.cf
#
# Sample ldirectord configuration file to configure various virtual services.
#
# Ldirectord will connect to each real server once per second and request
# /index.html. If the data returned by the server does not contain the
# string "Test Message" then the test fails and the real server will be
# taken out of the available pool. The real server will be added back into
# the pool once the test succeeds. If all real servers are removed from the
# pool then localhost:80 is added to the pool as a fallback measure.
# Global Directives
checktimeout=3 //超时
checkinterval=1 //check的间隔时间
#fallback=127.0.0.1:80
#fallback6=[::1]:80
autoreload=yes //实时读取配置文件
#logfile="/var/log/ldirectord.log" //定义日志文件位置
#logfile="local0"
#emailalert="admin@x.y.z"
#emailalertfreq=3600
#emailalertstatus=all
quiescent=no //真正的把故障realserver从lvs映射表中删除，以保证集群对外的不间断服务
#Sample configuration for a MySQL virtual service.
virtual = 192.168.213.97:3306
#real=master->slave92:3306 gate 10
real=192.168.213.91 gate 10
real=192.168.213.92 gate 10
fallback=127.0.0.1:3306
service=mysql
scheduler=wrr
#persistent=600
#netmask=255.255.255.255
protocol=tcp
checktype=negotiate
login="root"
passwd="root"
database="test"
request="SELECT * FROM a"
然后重启ldirectord服务
[root@master mysqlinstall]# service ldirectord restart
[root@master mysqlinstall]# service ldirectord status
然后查看log看ldirectord的工作状况
[root@master mysqlinstall]# tail -f /var/log/ldirectord.log
测试ldirectord是否可以对realserver做健康检查，为了便于测试，我把均衡算法改为rr，并去掉持久连接
测试前：
[root@master mysqlinstall]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.213.97:3306 rr
-> 192.168.213.91:3306 Local 10 0 59
-> 192.168.213.92:3306 Route 10 0 59
[root@master mysqlinstall]#
启动客户端测试脚本，每秒执行一次数据库查询操作
# sh load_mysql.sh
停止92的mysql
查看结果
[root@master mysqlinstall]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.213.97:3306 rr
-> 192.168.213.91:3306 Local 10 0 62
[root@master mysqlinstall]#
查看ldirectord的log
[root@master mysqlinstall]# tail -f /var/log/ldirectord.log
[Sat Dec 7 01:52:22 2013|ldirectord.cf|7445] Added real server: 192.168.213.92:0 (192.168.213.97:3306) (Weight set to 10)
[Sat Dec 7 01:52:22 2013|ldirectord|26081] system(/sbin/ipvsadm -a -t 192.168.213.97:3306 -r 192.168.213.92:0 -g -w 10) failed:
[Sat Dec 7 01:52:22 2013|ldirectord|26081] Added real server: 192.168.213.92:0 (192.168.213.97:3306) (Weight set to 10)
[Sat Dec 7 02:02:53 2013|ldirectord.cf|7445] Deleted real server: 192.168.213.92:3306 mapped from 192.168.213.92:0 (192.168.213.97:3306)
[Sat Dec 7 02:02:53 2013|ldirectord|26081] system(/sbin/ipvsadm -d -t 192.168.213.97:3306 -r 192.168.213.92:3306) failed:
[Sat Dec 7 02:02:53 2013|ldirectord|26081] Deleted real server: 192.168.213.92:3306 mapped from 192.168.213.92:0 (192.168.213.97:3306)
前端的服务有短暂1s的影响
.....
+----------+
+----------+
| count(*) |
+----------+
| 2 |
+----------+
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.213.97' (111)
+----------+
| count(*) |
+----------+
| 2 |
+----------+
......
到目前为止，lvs可以负载均衡前端的请求，并可以通过ldirectord服务实现对后端的realserver的健康检查，但是如过lvs和ldirectord所在的机器故障怎么办呢，就没办法高可用了，这时就需要corosync和pacemaker来做HA。
会继续下面会继续........
-------续---------