LVS+Keepalived+MySQL高可用配置
本文所有配置前提是已实现MySQL双主备份(MySQL双主)
安装前的准备:
VIP:192.168.0.201
Keepalived:
Keepalived-Master:192.168.0.241
Keepalived-Backup:192.168.0.277
RealServer:
MySQL1:192.168.0.225
MySQL2:192.168.0.226
kernel-devel各版本下载地址:http://rpmfind.net/linux/rpm2html/search.php?query=kernel-devel
keepalived下载地址:http://www.keepalived.org/download.html
kernel-devel.2.6.32-431.el6.x86_64.rpm (根据系统实际内核版本)
keepalived-1.2.7.tar.gz
ipvsadm-1.26.tar.gz
依赖包 :
yum install openssl-devle popt-devel popt-static libnl*
一、软件安装
1、安装kernel-devel:
rpm -ivh kernel-devel-2.6.32-431.29.2.el6.x86_64.rpm
2、安装lvs管理工具ipsadm:
tar zxvf ipvsadm-1.26.tar.gz
cd ipvsadm
make
make install
3、安装keepalived:
tar zxvf keepalived-1.2.7.tar.gz cd keepalived ./configure --prefix=/usr/local/keepalived make make install
4、将keepalived添加到系统服务:
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/ mkdir /etc/keepalived cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
二、配置
1、MySQL服务器中的配置 :
将realserver脚本放在/etc/init.d/目录下，执行即可:
#!/bin/sh VIP=192.168.0.201 sh /etc/rc.d/init.d/functions case "$1" in # 禁用本地的ARP请求、绑定本地回环地址 start) /sbin/ifconfig lo down /sbin/ifconfig lo up echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce /sbin/sysctl -p >/dev/null 2>&1 /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up #在回环地址上绑定VIP，设定掩码，与Direct Server（自身）上的IP保持通信 /sbin/route add -host $VIP dev lo:0 echo "LVS-DR real server starts successfully.\n" ;; stop) /sbin/ifconfig lo:0 down /sbin/route del $VIP >/dev/null 2>&1 echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce echo "LVS-DR real server stopped.\n" ;; status) isLoOn=`/sbin/ifconfig lo:0 | grep "$VIP"` isRoOn=`/bin/netstat -rn | grep "$VIP"` if [ "$isLoON" == "" -a "$isRoOn" == "" ]; then echo "LVS-DR real server has run yet." else echo "LVS-DR real server is running." fi exit 3 ;; *) echo "Usage: $0 {start|stop|status}" exit 1 esac exit 0
然后使用 ip addr 命令查看VIP是否绑定:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet 192.168.0.201/32 brd 192.168.0.201 scope global lo:0 inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:59:f1:3f brd ff:ff:ff:ff:ff:ff inet 192.168.0.225/24 brd 192.168.0.255 scope global eth0 inet6 fe80::5054:ff:fe59:f13f/64 scope link valid_lft forever preferred_lft forever
可以看到vip已经绑定在了lo上。两台msyql服务器执行同样的操作。
2、Keepalived的配置:
2.1、首先关闭系统防火墙iptables和selinux
2.2、开启路由转发及非本机ip绑定支持功能:
echo 1 >/proc/sys/net/ipv4/ip_forward
echo 1 >/proc/sys/net/ipv4/ip_nonlocal_bind
2.3、修改Keepalived的配置文件/etc/keepalived/keepalived.conf:
! Configuration File for keepalived global_defs { notification_email { 123@sina.com } notification_email_from 321@sina.com smtp_server 192.168.0.241 smtp_connect_timeout 30 router_id LVS1 } vrrp_instance VI_1 { state BACKUP ＃角色,若都为BACKUP，则表示当其中一台挂掉时，以另一台为MASTER，(即不会主动抢VIP) interface eth0 ＃主备之间健康检查的网卡 virtual_router_id 51 ＃主备id相同 priority 100 advert_int 1 authentication { auth_type PASS ＃主备之间通信认证 auth_pass 0711 } virtual_ipaddress { 192.168.0.201 } } virtual_server 192.168.0.201 3306 { delay_loop 6 lb_algo rr lb_kind DR nat_mask 255.255.255.0 persistence_timeout 50 protocol TCP real_server 192.168.0.225 3306 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_prot 3306 } } real_server 192.168.0.226 3306 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_prot 3306 } } }
keepalived的Master和Backup配置文件不同的是，
Master: state为MASTER，priority的值比BACKUP大。
BACKUP:state为BACKUP，priority的值比MASTER小。
2.4、启动Keepalived:
service keepalived start
然后使用ipvsadm工具查看集群列表:
主、备结果一样。
[root@localhost ~]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.201:3306 rr persistent 50 -> 192.168.0.225:3306 Route 1 0 0 -> 192.168.0.226:3306 Route 1 0 0
查看ip地址情况:
MASTER:
[root@localhost ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 64:00:6a:34:c3:e2 brd ff:ff:ff:ff:ff:ff inet 192.168.0.241/24 brd 192.168.0.255 scope global p4p1 inet 192.168.0.201/32 scope global p4p1 inet6 fe80::6600:6aff:fe34:c3e2/64 scope link valid_lft forever preferred_lft forever
BACKUP：
[root@localhost ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:b3:34:e6 brd ff:ff:ff:ff:ff:ff inet 192.168.0.227/24 brd 192.168.0.255 scope global eth0 inet6 fe80::5054:ff:feb3:34e6/64 scope link valid_lft forever preferred_lft forever
可以看到，MASTER上绑定了VIP，而BACKUP没有。
3、测试:
3.1keepalived主从测试:
把MASTER的keepalived服务停掉，service keepalived stop ，然后再查看ip:
[root@localhost ~]# service keepalived stop Stopping keepalived: [ OK ] [root@localhost ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 64:00:6a:34:c3:e2 brd ff:ff:ff:ff:ff:ff inet 192.168.0.241/24 brd 192.168.0.255 scope global p4p1 inet6 fe80::6600:6aff:fe34:c3e2/64 scope link valid_lft forever preferred_lft forever
发现VIP没了。此时再查看BACKUP的ip:
[root@localhost ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:b3:34:e6 brd ff:ff:ff:ff:ff:ff inet 192.168.0.227/24 brd 192.168.0.255 scope global eth0 inet 192.168.0.201/32 scope global eth0 inet6 fe80::5054:ff:feb3:34e6/64 scope link valid_lft forever preferred_lft forever
VIP成功漂移到了BACKUP上。
同时，通过keepalived的日志也可以看出，当master停掉后，backup主动成为master:
cat /var/log/messages
Jan 25 16:18:28 localhost Keepalived_healthcheckers[1199]: SMTP connection ERROR to [192.168.0.227]:25. Jan 25 16:42:41 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Transition to MASTER STATE Jan 25 16:42:42 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Entering MASTER STATE Jan 25 16:42:42 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) setting protocol VIPs. Jan 25 16:42:42 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.201 Jan 25 16:42:42 localhost Keepalived_healthcheckers[1199]: Netlink reflector reports IP 192.168.0.201 added Jan 25 16:42:47 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.201
3.2MySQL高可用测试:
通过VIP连接mysql:
localhost:~ ahaii$ mysql -h192.168.0.201 -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 643 Server version: 5.5.54-log Source distribution Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql>
连接成功。
同时在keepalived-master上查看lvs分配情况:
[root@localhost ~]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.201:3306 rr persistent 50 -> 192.168.0.225:3306 Route 1 0 0 -> 192.168.0.226:3306 Route 1 1 0 [root@localhost ~]#
显示现在连接的是226的那台mysql服务器。
此时，将226服务器上的mysql服务停掉，然后再通过vip连接mysql，并查看lvs分配情况:
依然能通过vip连接mysql:
localhost:~ ahaii$ mysql -h192.168.0.201 -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 37014 Server version: 5.5.54-log Source distribution Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql>
此时，226已被keepalived集群剔除掉，并且客户端连接的是225服务器 ：
[root@localhost ~]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.201:3306 rr persistent 50 -> 192.168.0.225:3306 Route 1 1 0
当把226服务器的mysql服务重新启动时，keepalived又将其加入到集群中:
[root@localhost ~]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.201:3306 rr persistent 50 -> 192.168.0.225:3306 Route 1 1 0 -> 192.168.0.226:3306 Route 1 0 0
keepalived剔除、添加226服务器的日志记录如下:
Jan 25 17:00:27 localhost Keepalived_healthcheckers[1429]: TCP connection to [192.168.0.226]:3306 failed !!! Jan 25 17:00:27 localhost Keepalived_healthcheckers[1429]: Removing service [192.168.0.226]:3306 from VS [192.168.0.201]:3306 Jan 25 17:03:27 localhost Keepalived_healthcheckers[1429]: TCP connection to [192.168.0.226]:3306 success. Jan 25 17:03:27 localhost Keepalived_healthcheckers[1429]: Adding service [192.168.0.226]:3306 to VS [192.168.0.201]:3306
疑惑:
1、客户点请求VIP时，由于调度器和realserver都有VIP，如何保证是调度器接收请求，而不是realserver？
为了保证是调度器接收客户端的请求，在realserver上都做了如下配置:
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
用arp_ignore参数设定当请求的目标地址是本机IP时，对其ARP的查询作出不同级别的响应。
1:表示只回答目标IP地址是本地网卡的arp请求
2:表示只回答目标IP地址是本地网卡的arp请求，并且来访IP与本机IP在同一网段内。
用arp_announce参数设定本机向外宣告自身IP-MAC时的级别。
1:表示尽量避免响应ARP请求中MAC不是本网卡的。如一个主机有多块网卡，其中一块网卡接收到了ARP请求，发现所请求的MAC是本机另一块网卡的，这个时候接收到ARP请求的这块网卡就尽量避免响应。
2:表示总是使用最合适的网卡来响应。一个主机有多块网卡，其中一块网卡接收到了ARP请求，发现所请求的MAC是本机另一块网卡的，这个时候接收到ARP请求的这块网卡就一定不响应，只有发现请求的MAC是自己的才给与响应。
以上设置，就是为了接收客户点请求的是调度器的VIP，而不是realserver的VIP。
2、为什么负载均衡器和realserver上都有VIP呢？
当均衡器收到客户端请求后，根据配置的算法选取一台realserver，将客户端请求报文的目标MAC地址修改为选取的realserver的MAC地址后重新封装该数据帧并发送给选取的realserver。realserver收到该数据帧后检查IP报文发现该报文的目标IP(VIP)是自己，然后对该请求报文做出响应，将响应结果直接发送给客户端(整个过程中源地址不变)。
以上是整个安装配置和测试的过程。