Docker compose部署redis哨兵集群

Docker compose部署redis哨兵集群

• 安装Docker和docker-compose
• 准备docker-compose文件
• redis exporter本地部署
• 准备Redis配置文件
• ACL用户权限配置
• Linux内核参数优化
• 启停Redis实例
• 主从复制配置

环境准备：

IP	版本	角色
172.x.x.11	RHEL 7.9	master
172.x.x.12	RHEL 7.9	replica
172.x.x.13	RHEL 7.9	replica

安装Docker和docker-compose

安装docker-ce：

# 移除旧版本
sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine
		  
# 配置仓库
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
#官方源
#sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
#阿里云源
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo yum makecache fast

# 检查可安装的版本
yum list docker-ce --showduplicates | sort -r

# 安装指定版本
#sudo yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io docker-buildx-plugin docker-compose-plugin
yum install -y docker-ce-25.0.1 docker-ce-cli-25.0.1 containerd.io docker-buildx-plugin docker-compose-plugin

# 启动
sudo systemctl enable docker
sudo systemctl start docker		  

配置阿里云镜像源:

cat > /etc/docker/daemon.json << EOF
{
 "registry-mirrors" : ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF

sudo systemctl restart docker

????Docker社区版部署参见：

• https://docs.docker.com/engine/install/centos/
• https://gottdeskrieges.blog.csdn.net/article/details/113242029

安装docker-compose：

curl -SL https://github.com/docker/compose/releases/download/v2.24.6/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose

chmod 755 /usr/local/bin/docker-compose

????参考：https://docs.docker.com/compose/install/standalone/

准备docker-compose文件

• Redis版本：7.2.4
• Redis安装路径：/opt/docker-compose/redis

准备单台服务的compose文件，每台服务器上部署一个redis实例、一个sentinel实例和一个redis exporter。容器网络采用仅主机模式，避免NAT转发的复杂性。

version: '3'

services:
  redis-server:
    image: redis:7.2.4
    network_mode: host
    container_name: redis-server
    restart: unless-stopped
    #ports:
    #  - 6379:6379
    environment:
      TZ: "Asia/Shanghai"
    command: ["/conf/redis.conf"]
    volumes:
      - ./conf/:/conf/
      - ./data/:/data/
    
  redis-sentinel:
    image: redis:7.2.4
    network_mode: host
    container_name: redis-sentinel
    restart: unless-stopped
    environment:
      TZ: "Asia/Shanghai"
    #ports:
    #  - 26379:26379
    command: ["/conf/sentinel.conf","--sentinel"]
    volumes:
      - ./conf/:/conf/
    
  redis-exporter:
    image: oliver006/redis_exporter
    network_mode: host
    restart: unless-stopped
    environment:
      TZ: Asia/Shanghai
      REDIS_ADDR: redis://localhost:6379
      REDIS_USER: redis_monitor
      REDIS_PASSWORD: Monpass_xxxx
    # ports:
    #   - 9121:9121

redis exporter本地部署

如果没有现成的redis exporter镜像，可以采用本地部署，并注释掉docker compose文件中的对应内容。

直接下载二进制安装包：

cd /opt
wget https://github.com/oliver006/redis_exporter/releases/download/v1.58.0/redis_exporter-v1.58.0.linux-amd64.tar.gz

tar -xvf redis_exporter-v1.58.0.linux-amd64.tar.gz
cd redis_exporter
./redis_exporter --redis.addr=redis://localhost:6379 --redis.user=redis_monitor --redis.password="Monpass_xxxx"

检查redis exporter采集到的数据：

curl -X GET http://localhost:9121/metrics

????参考：https://github.com/oliver006/redis_exporter

准备Redis配置文件

哨兵节点配置文件/opt/docker-compose/redis/conf/sentinel.conf：

#port 26379
loglevel verbose
sentinel announce-ip "<REDIS_SENTINEL_IP>"
sentinel announce-port 26379
dir "/data"

sentinel monitor <SVC_NAME> <REDIS_SERVER_IP> 6379 2
sentinel auth-user <SVC_NAME> sentinel_user
sentinel auth-pass <SVC_NAME> xxxxxx
sentinel down-after-milliseconds <SVC_NAME> 15000

其中，<SVC_NAME>是自定义的哨兵集群服务名。由于数据库实例和哨兵实例都部署在同一台服务器上，<REDIS_SENTINEL_IP>和<REDIS_SERVER_IP>可以都填写本地宿主机IP。

数据库节点配置文件/opt/docker-compose/redis/conf/redis.conf：

bind 0.0.0.0
#port 6379
timeout 3600
tcp-keepalive 300
loglevel notice
databases 16

save 600 100
save 120 10000
save 7200 1

dir /data
masteruser <username>  # 主从复制用户
masterauth <password>

replica-read-only yes
#min-replicas-to-write 3
#min-replicas-max-lag 10

replica-announce-ip "<REDIS_SERVER_IP>"
replica-announce-port 6379
aclfile /conf/users.acl

maxclients 10000
maxmemory 10gb
maxmemory-policy noeviction

appendonly yes
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

ACL用户权限配置

配置好管理员用户、监控用户、sentinel用户、主从复制用户、以及应用用户。

在redis.conf中配置aclfile参数，ACL文件（conf/users.acl）示例如下：

user default off nopass &* -@all
user root on >Rootpass_xxxx allkeys allchannels +@all
user redis_monitor on >Monpass_xxxx +client +ping +info +config|get +cluster|info +slowlog +latency +memory +select +get +scan +xinfo +type +pfcount +strlen +llen +scard +zcard +hlen +xlen +eval allkeys
user sentinel_user on >Senpass_xxxx allchannels +multi +slaveof +ping +exec +subscribe +config|rewrite +role +publish +info +client|setname +client|kill +script|kill
user replica_user on >Replpass_xxxx +psync +replconf +ping
user appuser on >Apppass_xxxx allkeys allchannels +@all -@admin -@dangerous

导入ACL文件配置：

# 列出已有的用户权限清单
docker exec -it redis-server redis-cli acl list

# 导入ACL文件中的用户权限配置
docker exec -it redis-server redis-cli acl load

docker exec -it redis-server redis-cli acl list

????参考：https://redis.io/docs/management/security/acl/#create-and-edit-user-acls-with-the-acl-setuser-command

Linux内核参数优化

修改Linux内核参数：

sed -i '/vm.overcommit_memory/d' /etc/sysctl.conf
echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf
echo "net.core.somaxconn=2048" >> /etc/sysctl.conf
sysctl -p

禁用透明大页：

cat <<EOF >> /etc/rc.d/rc.local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
  echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
EOF

chmod +x /etc/rc.d/rc.local

编辑/etc/security/limits.conf和/etc/security/limits.d/90-nproc.conf，添加以下配置：

*       soft    core    unlimited
*       hard    core    unlimited
*       soft    nofile  65536
*       hard    nofile  65536
*       soft    nproc   65536
*       hard    nproc   65536

最后重启服务器生效。

启停Redis实例

修改文件权限：

cd /opt/docker-compose/
chown -R polkitd:root redis/

在docker-compose.yml文件所在路径下执行以下命令来启停单个节点上的REDIS实例。

#启动容器
cd /opt/docker-compose/redis && docker-compose up -d

#停止容器
cd /opt/docker-compose/redis && docker-compose stop

主从复制配置

将当前节点redis实例配置为172.x.x.11上的实例的从实例：

docker exec -it redis-server redis-cli --user xx -a xxxxxx replicaof 172.x.x.11 6379

也可以把replicaof 172.x.x.11 6379配置到从实例的redis.conf文件中。

检查各节点角色和同步状态：

docker exec -it redis-server redis-cli --user xx -a xxxxxx -h 172.x.x.11 -p 6379 info replication
docker exec -it redis-server redis-cli --user xx -a xxxxxx -h 172.x.x.12 -p 6379 info replication
docker exec -it redis-server redis-cli --user xx -a xxxxxx -h 172.x.x.13 -p 6379 info replication

测试主从切换：

# 停止172.x.x.11上的主实例
docker stop redis-server

# 检查哨兵节点日志输出
docker logs -fn100 redis-sentinel

# 输出以下日志内容表示发生了故障转移
...
1:X 01 Jun 2020 17:31:50.435 # +sdown master mymaster 172.x.x.11 6379    # ->主实例客观宕机
1:X 01 Jun 2020 17:31:50.458 # +new-epoch 4
1:X 01 Jun 2020 17:31:50.460 # +vote-for-leader 79c1f8276a8fedd78a9f987da8c85fce0ea7f751 4
1:X 01 Jun 2020 17:31:50.527 # +odown master mymaster 172.x.x.11 6379 #quorum 3/2          # ->主实例主观宕机
1:X 01 Jun 2020 17:31:50.527 # Next failover delay: I will not start a failover before  Mon Jun  1 17:37:50 2020
1:X 01 Jun 2020 17:31:51.708 # +config-update-from sentinel 79c1f8276a8fedd78a9f987da8c85fce0ea7f751 172.x.x.12 26379 @ mymaster 172.x.x.11 6379
1:X 01 Jun 2020 17:31:51.708 # +switch-master mymaster 172.x.x.11 6379 172.x.x.12 6379     # ->主实例地址发生切换
1:X 01 Jun 2020 17:31:51.708 * +slave slave 172.x.x.13:6379 172.x.x.13 6379 @ mymaster 172.x.x.12 6379
1:X 01 Jun 2020 17:31:51.708 * +slave slave 172.x.x.11:6379 172.x.x.11 6379 @ mymaster 172.x.x.12 6379
...

上面的信息表示主实例从172.x.x.11切换到了172.x.x.12。

主从切换后，主从节点的sentinel.conf中的replicaof参数会发生变化，相应的epoch参数也会自动+1。

References：
【1】https://www.codenong.com/cs106478941/
【2】https://lipeng1667.github.io/2021/06/21/set-up-redis-with-replication-and-sentinel-using-docker-compose/