Nginx高可用部署

Nginx高可用部署

前言

目前是通过Nginx做Web服务器和负载均衡，如果Nginx服务宕掉的话，会导致所有服务都无法进行访问，需要对Nginx来做高可用，目前是通过keepalived的虚拟IP（VIP）的漂移来实现的，当一台Nginx服务器宕掉后VIP会自动漂到另一台服务器继续提供访问。

部署环境及版本

系统版本：CentOS Linux release 7.9.2009
Nginx版本：1.19.6
Keepalived版本：v1.3.5

准备

主机ip	安装程序
192.168.102.212	Nginx、Keepalived
192.168.102.213	Nginx、Keepalived

1、关闭自带防火墙，用iptables,添加hosts文件解析

# 关闭SELinux
setenforce 0 
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

# 关闭Firewalld并禁止自启动
systemctl stop firewalld
systemctl disable firewalld

# 安装iptables
yum install -y iptables-services

# 启动iptables服务，并设置开机自启动（修改规则在/etc/sysconfig/iptables）
systemctl start iptables
systemctl enable iptables.service

2、时间同步

yum -y install ntp
systemctl enable ntpd
systemctl start ntpd
timedatectl set-timezone Asia/Shanghai
ntpdate -u time.nist.gov
ntpdate -u time.nist.gov
date

一、Nginx部署

1、基础环境部署脚本

vim Deployment_preparation.sh
#!/bin/bash
#部署前准备
yum update -y
yum install ntpdate -y
ntpdate time.windows.com
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
systemctl stop firewalld.service
systemctl disable firewalld.service
firewall-cmd --state
systemctl list-unit-files | grep firewalld
yum install -y iptables-services
systemctl enable iptables.service
systemctl start iptables
systemctl restart iptables

2、Nginx部署脚本

      
#!/bin/bash
read -p "please input your Intranet address : " IP
package_PATH_="/server/tools"
yum install -y pcre pcre-devel
yum install -y zlib zlib-devel
yum install gcc-c++ -y
yum install -y openssl openssl-devel
yum install lua-devel -y    
yum install -y gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel wget unzip
if [ -d ${package_PATH_} ];then
    echo dir ${package_PATH_} exist!
else
    mkdir -p /server/tools
fi
cd $package_PATH_
if [ -e nginx-1.19.6.tar.gz ];then
   echo "nginx-1.19.6.tar.gz is exist !"
else
    echo "Please An installation package nginx-1.19.6.tar.gz is available"
    exit 24
fi
tar -zvxf nginx-1.19.6.tar.gz
useradd work
cd  nginx-1.19.6
./configure --user=work --group=work --prefix=/data/nginx --with-http_v2_module --with-http_ssl_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre
make  && make install
sed -i '2c user work work;' /data/nginx/conf/nginx.conf
sed -i '3c worker_processes  auto;' /data/nginx/conf/nginx.conf
sed -i '9c pid        logs/nginx.pid;' /data/nginx/conf/nginx.conf
sed -i '13c worker_connections  10240;' /data/nginx/conf/nginx.conf
sed -i '116c include /data/nginx/conf/vhost/*.conf;' /data/nginx/conf/nginx.conf 
mkdir -pv /data/nginx/conf/vhost
cd /usr/lib/systemd/system
cat > nginx.service << EOF
[Unit]
Description=nginx
After=network.target

[Service]
Type=forking
ExecStart=/data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf
ExecReload=/data/nginx/sbin/nginx -s reload
ExecStop=/data/nginx/sbin/nginx -s quit
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF
systemctl start nginx
systemctl status nginx
systemctl enable nginx
systemctl daemon-reload
echo  "Cluster_Nginx is Deployment complete!"

二、Keepalived部署Nginx高可用

1、安装keepalived和killall

yum install psmisc keepalived -y

2、将原来配置备份，参考下面进行配置

Nginx的master配置

mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
 
::
vrrp_script chk_nginx  {
script "/usr/local/src/check_nginx_pid.sh"
interval 2 #脚本检测频率
weight -5 #脚本执行成功与否，权重怎么计算
fall 2 #如果连续两次检测失败，认为节点服务不可用
rise 1 #如果连续2次检查成功则认为节点正常
}
 

vrrp_instance VI_1 {
state MASTER
interface ens160 #节点IP的网卡
virtual_router_id 200 #同一个instance相同
priority 212 # 优先级，数值越大，优先级越高
advert_int 1
authentication { #节点间的认证，所有的必须一致
auth_type PASS
auth_pass Hirain_ha_215
}
virtual_ipaddress { #VIP，自定的，和外网的IP要一个网段
192.168.102.99/24
}
 
track_script { #指定前面脚本的名字
chk_nginx
} 
}

Nginx的backup配置

mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

#指定proxysql服务检测脚本
vrrp_script chk_nginx  {
script "/usr/local/src/check_nginx_pid.sh"
interval 2 #脚本检测频率
weight -5 #脚本执行成功与否，权重怎么计算
fall 2 #如果连续两次检测失败，认为节点服务不可用
rise 1 #如果连续2次检查成功则认为节点正常
}


vrrp_instance VI_1 {
state BACKUP
interface ens160 #节点IP的网卡
virtual_router_id 200 #同一个instance相同
priority 211 # 优先级，数值越大，优先级越高
advert_int 1
authentication { #节点间的认证，所有的必须一致
auth_type PASS
auth_pass Hirain_ha_215
}
virtual_ipaddress { #VIP，自定的，和外网的IP要一个网段
192.168.102.99/24
}

track_script { #指定前面脚本的名字
chk_nginx
}
}

3、Nginx进程检测脚本

vim /usr/local/src/check_nginx_pid.sh
#!/bin/bash
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
      systemctl start nginx
      sleep 5
      if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then
              killall keepalived
      fi
fi

脚本授权

chmod 755 /usr/local/src/check_nginx_pid.sh

4、启动keepalived并设置开机自启动

systemctl start keepalived
systemctl enable keepalived

三、测试

测试场景一

Nginx服务宕掉，测试Nginx是否自动重启

[root@mgr01 vhost]# ps axu|grep nginx
root      3817  0.0  0.0  46104  1160 ?        Ss   10:10   0:00 nginx: master process /data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf
work      3818  0.0  0.1  50276  5612 ?        S    10:10   0:00 nginx: worker process
work      3819  0.0  0.1  50276  5612 ?        S    10:10   0:00 nginx: worker process
root      4143  0.0  0.0 112812   980 pts/0    S+   10:12   0:00 grep --color=auto nginx

systemctl stop nginx
[root@mgr01 vhost]# ps axu|grep nginx
root      4302  0.0  0.0 115408  1444 ?        S    10:13   0:00 /bin/bash /usr/local/src/check_nginx_pid.sh
root      4308  0.0  0.0  46104  1156 ?        Ss   10:13   0:00 nginx: master process /data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf
work      4309  0.0  0.1  50276  5612 ?        S    10:13   0:00 nginx: worker process
work      4310  0.0  0.1  50276  5612 ?        S    10:13   0:00 nginx: worker process
root      4313  0.0  0.0 112812   980 pts/0    S+   10:13   0:00 grep --color=auto nginx
[root@mgr01 vhost]# ps axu|grep nginx
root      4308  0.0  0.0  46104  1156 ?        Ss   10:13   0:00 nginx: master process /data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf
work      4309  0.0  0.1  50276  5612 ?        S    10:13   0:00 nginx: worker process
work      4310  0.0  0.1  50276  5612 ?        S    10:13   0:00 nginx: worker process
root      4340  0.0  0.0 112812   980 pts/0    S+   10:13   0:00 grep --color=auto nginx

测试结果：Nginx服务宕掉后，会自动重新启动

测试场景二

Nginx服务器宕掉后，或者是Nginx重启失败后，VIP是否会漂到另一台服务器

[root@mgr01 vhost]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 12:34:56:78:96:17 brd ff:ff:ff:ff:ff:ff
    inet 192.168.102.212/22 brd 192.168.103.255 scope global noprefixroute dynamic ens160
       valid_lft 123607sec preferred_lft 123607sec
    inet 192.168.102.99/24 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::ef48:def0:4499:80ef/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::37bf:4ad7:b805:198a/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::30ad:fb50:af40:6b75/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever

[root@mgr01 vhost]# shutdown
[root@mgr02 src]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 12:34:56:78:96:18 brd ff:ff:ff:ff:ff:ff
    inet 192.168.102.213/22 brd 192.168.103.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet 192.168.102.99/24 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::ef48:def0:4499:80ef/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::37bf:4ad7:b805:198a/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::30ad:fb50:af40:6b75/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever

Nginx服务器宕掉后，或者是Nginx重启失败后，VIP会漂到另一台服务器

四、总结

Nginx+Keepalived,如果Nginx服务宕掉后，会先进行重启，重启失败后，会通过VIP的漂移到正常的Nginx服务器上，从而继续提供服务,恢复后，会漂到priority数值最大的Nginx服务上面。