Codis3.2集群HA高可用方案
Codis高可用方案官方推荐使用Sentinel
Redis 本身就是最终一致性的。Master 挂了，Promote Slave 成为新的 Master 需要时间（测试15秒内）。其实 Sentinel 就是这个逻辑。Codis3.2 自己没有实现 HA，而是直接依赖 Sentinel 的。
注意事项：
Sentinel需要使用原生的Redis-server，版本要等于或高于Codis3.2里面的3.2.8版本， 这里是在Redis3.2.9的下配置测试的，另外Protected-mode no这个保护模式需要关闭，Sentinel.conf默认配置不需要写主从关系，Codif-fe界面SYNC会读取主从关系。
1，每个 Group 前面的 SYNC 是刷 group 所涉及到的 Slot 的路由表给 Proxy 的，包含主从关系。
2，而 Sentinel 前面的 SYNC 是给所有 Sentinel 刷配置的，同时把 Sentinel 列表刷给每个 Proxy。所以只要在集群搭建完成之后，或者每次添加、删除 Redis server 之后（稳定之后） ，在 Sentinel 下面点一次 SYNC 即可。
3，Codis3.2 的只解决了并发迁移＋异步迁移问题。不会去做 HA 有关的工作。所以在 3.2 下解决问题的话， Sentinel 就足够了，而且只能用 Sentinel。Sentinel 负责的是 Group 内的主从切换。
4，内核优化
echo never > /sys/kernel/mm/transparent_hugepage/enabled
sysctl vm.overcommit_memory=1
一，Codis集群架构
192.168.188.120 codis120 codis-server zookeeper sentinel192.168.188.121 codis121 codis-server zookeeper sentinel
192.168.188.122 codis122 codis-server zookeeper sentinel
192.168.188.123 codis123 codis-server codis-proxy sentinellvs
192.168.188.124 codis124 codis-server codis-proxy sentinellvs
192.168.188.125 codis125 codis-server codis-dashboard sentinelcodis-fe
二，配置Codis-server
每台虚拟机部署3个实例：6379、6380、6381，共18个实例
192.168.188.120:6379、6380、6381192.168.188.121:6379、6380、6381192.168.188.122:6379、6380、6381192.168.188.123:6379、6380、6381192.168.188.124:6379、6380、6381192.168.188.125:6379、6380、6381
三，配置Group
设置好1主2从关系，共6组，注意主从从3个实例需要来自3台虚拟机
Group1（主从从）192.168.188.120:6379 192.168.188.121:6380 192.168.188.123:6381Group2（主从从）192.168.188.121:6379 192.168.188.120:6380 192.168.188.124:6381Group3（主从从）192.168.188.122:6379 192.168.188.123:6380 192.168.188.125:6381Group4（主从从）192.168.188.123:6379 192.168.188.122:6380 192.168.188.120:6381Group5（主从从）192.168.188.124:6379 192.168.188.125:6380 192.168.188.121:6381Group6（主从从）192.168.188.125:6379 192.168.188.124:6380 192.168.188.122:6381
四，配置Sentinel
需要配置6个Sentinel，没台虚拟机上1个
注意这个原生是在sentinel 的配置文件里配置，3.2.9的原生的sentinel配置下，另外protected-mode no这个保护模式需要关闭，sentinel.conf默认配置不需要写主从关系，Codif-fe界面SYNC会读取主从关系。
1，配置信息
[root@codis120 config]# cat sentinel.conf |grep -Ev "^#|^$"
protected-mode noport 26379dir "/tmp"
2，启动sentinel，多个Sentinel便于选举和保证高可用
192.168.188.120:26379192.168.188.120:26379192.168.188.120:26379192.168.188.120:26379192.168.188.120:26379192.168.188.120:26379
3，SYNC后通过日志查看哨兵记录
20980:X 19 Sep 10:53:28.965 # +monitor master codis-easypass-2 192.168.188.121 6379 quorum 220980:X 19 Sep 10:53:28.965 # +set master codis-easypass-2 192.168.188.121 6379 parallel-syncs 120980:X 19 Sep 10:53:28.965 # +set master codis-easypass-2 192.168.188.121 6379 down-after-milliseconds 3000020980:X 19 Sep 10:53:28.965 # +set master codis-easypass-2 192.168.188.121 6379 failover-timeout 30000020980:X 19 Sep 10:53:28.967 * +slave slave 192.168.188.120:6380 192.168.188.120 6380 @ codis-easypass-2 192.168.188.121 637920980:X 19 Sep 10:53:28.969 * +slave slave 192.168.188.124:6381 192.168.188.124 6381 @ codis-easypass-2 192.168.188.121 637920980:X 19 Sep 10:53:28.973 # +monitor master codis-easypass-3 192.168.188.122 6379 quorum 220980:X 19 Sep 10:53:28.973 # +set master codis-easypass-3 192.168.188.122 6379 parallel-syncs 120980:X 19 Sep 10:53:28.973 # +set master codis-easypass-3 192.168.188.122 6379 down-after-milliseconds 3000020980:X 19 Sep 10:53:28.973 # +set master codis-easypass-3 192.168.188.122 6379 failover-timeout 30000020980:X 19 Sep 10:53:28.997 # +monitor master codis-easypass-4 192.168.188.123 6379 quorum 220980:X 19 Sep 10:53:28.997 # +set master codis-easypass-4 192.168.188.123 6379 parallel-syncs 120980:X 19 Sep 10:53:28.997 # +set master codis-easypass-4 192.168.188.123 6379 down-after-milliseconds 3000020980:X 19 Sep 10:53:28.997 # +set master codis-easypass-4 192.168.188.123 6379 failover-timeout 30000020980:X 19 Sep 10:53:29.001 # +monitor master codis-easypass-5 192.168.188.124 6379 quorum 220980:X 19 Sep 10:53:29.002 # +set master codis-easypass-5 192.168.188.124 6379 parallel-syncs 120980:X 19 Sep 10:53:29.002 # +set master codis-easypass-5 192.168.188.124 6379 down-after-milliseconds 3000020980:X 19 Sep 10:53:29.002 # +set master codis-easypass-5 192.168.188.124 6379 failover-timeout 30000020980:X 19 Sep 10:53:29.006 # +monitor master codis-easypass-6 192.168.188.125 6379 quorum 220980:X 19 Sep 10:53:29.006 # +set master codis-easypass-6 192.168.188.125 6379 parallel-syncs 120980:X 19 Sep 10:53:29.006 # +set master codis-easypass-6 192.168.188.125 6379 down-after-milliseconds 3000020980:X 19 Sep 10:53:29.006 # +set master codis-easypass-6 192.168.188.125 6379 failover-timeout 30000020980:X 19 Sep 10:53:29.010 # +monitor master codis-easypass-1 192.168.188.120 6379 quorum 220980:X 19 Sep 10:53:29.010 # +set master codis-easypass-1 192.168.188.120 6379 parallel-syncs 120980:X 19 Sep 10:53:29.010 # +set master codis-easypass-1 192.168.188.120 6379 down-after-milliseconds 3000020980:X 19 Sep 10:53:29.010 # +set master codis-easypass-1 192.168.188.120 6379 failover-timeout 30000020980:X 19 Sep 10:53:29.029 * +slave slave 192.168.188.121:6380 192.168.188.121 6380 @ codis-easypass-1 192.168.188.120 637920980:X 19 Sep 10:53:29.031 * +slave slave 192.168.188.123:6381 192.168.188.123 6381 @ codis-easypass-1 192.168.188.120 637920980:X 19 Sep 10:53:29.033 * +slave slave 192.168.188.124:6380 192.168.188.124 6380 @ codis-easypass-6 192.168.188.125 637920980:X 19 Sep 10:53:29.036 * +slave slave 192.168.188.122:6381 192.168.188.122 6381 @ codis-easypass-6 192.168.188.125 637920980:X 19 Sep 10:53:29.038 * +slave slave 192.168.188.123:6380 192.168.188.123 6380 @ codis-easypass-3 192.168.188.122 637920980:X 19 Sep 10:53:29.040 * +slave slave 192.168.188.125:6381 192.168.188.125 6381 @ codis-easypass-3 192.168.188.122 637920980:X 19 Sep 10:53:29.042 * +slave slave 192.168.188.122:6380 192.168.188.122 6380 @ codis-easypass-4 192.168.188.123 637920980:X 19 Sep 10:53:29.044 * +slave slave 192.168.188.120:6381 192.168.188.120 6381 @ codis-easypass-4 192.168.188.123 637920980:X 19 Sep 10:53:29.047 * +slave slave 192.168.188.125:6380 192.168.188.125 6380 @ codis-easypass-5 192.168.188.124 637920980:X 19 Sep 10:53:29.049 * +slave slave 192.168.188.121:6381 192.168.188.121 6381 @ codis-easypass-5 192.168.188.124 637920980:X 19 Sep 10:53:30.958 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-4 192.168.188.123 637920980:X 19 Sep 10:53:30.960 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-3 192.168.188.122 637920980:X 19 Sep 10:53:30.962 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-2 192.168.188.121 637920980:X 19 Sep 10:53:30.965 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-6 192.168.188.125 637920980:X 19 Sep 10:53:30.976 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-2 192.168.188.121 637920980:X 19 Sep 10:53:30.978 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-5 192.168.188.124 637920980:X 19 Sep 10:53:30.981 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-3 192.168.188.122 637920980:X 19 Sep 10:53:30.989 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-2 192.168.188.121 637920980:X 19 Sep 10:53:30.991 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-4 192.168.188.123 637920980:X 19 Sep 10:53:30.993 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-3 192.168.188.122 637920980:X 19 Sep 10:53:31.001 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-6 192.168.188.125 637920980:X 19 Sep 10:53:31.003 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-1 192.168.188.120 637920980:X 19 Sep 10:53:31.005 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-4 192.168.188.123 637920980:X 19 Sep 10:53:31.008 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-3 192.168.188.122 637920980:X 19 Sep 10:53:31.010 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-2 192.168.188.121 637920980:X 19 Sep 10:53:31.012 * +sentinel sentinel d0268512448981b56e0eb76318af009b4575404c 192.168.188.125 26379 @ codis-easypass-5 192.168.188.124 637920980:X 19 Sep 10:53:31.015 * +sentinel sentinel 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54 192.168.188.123 26379 @ codis-easypass-1 192.168.188.120 637920980:X 19 Sep 10:53:31.052 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-6 192.168.188.125 637920980:X 19 Sep 10:53:31.055 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-1 192.168.188.120 637920980:X 19 Sep 10:53:31.058 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-4 192.168.188.123 637920980:X 19 Sep 10:53:31.059 * +sentinel sentinel 28758dd53514376486fdec2acdb73f273fa90645 192.168.188.121 26379 @ codis-easypass-5 192.168.188.124 637920980:X 19 Sep 10:53:31.062 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-6 192.168.188.125 637920980:X 19 Sep 10:53:31.069 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-5 192.168.188.124 637920980:X 19 Sep 10:53:31.071 * +sentinel sentinel 2204a1a2c3c5387bb33ab34de52c4eeda59257d4 192.168.188.122 26379 @ codis-easypass-1 192.168.188.120 637920980:X 19 Sep 10:53:31.276 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-4 192.168.188.123 637920980:X 19 Sep 10:53:31.292 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-6 192.168.188.125 637920980:X 19 Sep 10:53:31.303 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-3 192.168.188.122 637920980:X 19 Sep 10:53:31.314 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-2 192.168.188.121 637920980:X 19 Sep 10:53:31.322 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-5 192.168.188.124 637920980:X 19 Sep 10:53:31.325 * +sentinel sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-1 192.168.188.120 6379
4，也可以通过图形界面查看
http://192.168.188.125:18080/api/topom/sentinels/info/192.168.188.120:26379/monitored
5，最后配置文件核对
新增内容已经保存到sentinel.conf里面了
[root@codis120 config]# cat sentinel.conf |grep -Ev "^#|^$"
protected-mode noport 26379dir "/tmp"sentinel myid 620b655b311656aa6016e6b864b0f2a8fb03d453sentinel monitor codis-easypass-6 192.168.188.125 6379 2sentinel failover-timeout codis-easypass-6 300000sentinel config-epoch codis-easypass-6 0sentinel leader-epoch codis-easypass-6 0sentinel known-slave codis-easypass-6 192.168.188.124 6380sentinel known-slave codis-easypass-6 192.168.188.122 6381sentinel known-sentinel codis-easypass-6 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404csentinel known-sentinel codis-easypass-6 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645sentinel known-sentinel codis-easypass-6 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54sentinel known-sentinel codis-easypass-6 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4sentinel known-sentinel codis-easypass-6 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9sentinel monitor codis-easypass-2 192.168.188.121 6379 2sentinel failover-timeout codis-easypass-2 300000sentinel config-epoch codis-easypass-2 0sentinel leader-epoch codis-easypass-2 0sentinel known-slave codis-easypass-2 192.168.188.120 6380sentinel known-slave codis-easypass-2 192.168.188.124 6381sentinel known-sentinel codis-easypass-2 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404csentinel known-sentinel codis-easypass-2 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645sentinel known-sentinel codis-easypass-2 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54sentinel known-sentinel codis-easypass-2 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4sentinel known-sentinel codis-easypass-2 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9sentinel monitor codis-easypass-1 192.168.188.120 6379 2sentinel failover-timeout codis-easypass-1 300000sentinel config-epoch codis-easypass-1 0sentinel leader-epoch codis-easypass-1 0sentinel known-slave codis-easypass-1 192.168.188.123 6381sentinel known-slave codis-easypass-1 192.168.188.121 6380sentinel known-sentinel codis-easypass-1 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404csentinel known-sentinel codis-easypass-1 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645sentinel known-sentinel codis-easypass-1 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54sentinel known-sentinel codis-easypass-1 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4sentinel known-sentinel codis-easypass-1 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9sentinel monitor codis-easypass-3 192.168.188.122 6379 2sentinel failover-timeout codis-easypass-3 300000sentinel config-epoch codis-easypass-3 0sentinel leader-epoch codis-easypass-3 0sentinel known-slave codis-easypass-3 192.168.188.123 6380sentinel known-slave codis-easypass-3 192.168.188.125 6381sentinel known-sentinel codis-easypass-3 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404csentinel known-sentinel codis-easypass-3 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645sentinel known-sentinel codis-easypass-3 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54sentinel known-sentinel codis-easypass-3 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4sentinel known-sentinel codis-easypass-3 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9sentinel monitor codis-easypass-4 192.168.188.123 6379 2sentinel failover-timeout codis-easypass-4 300000sentinel config-epoch codis-easypass-4 0sentinel leader-epoch codis-easypass-4 0sentinel known-slave codis-easypass-4 192.168.188.120 6381sentinel known-slave codis-easypass-4 192.168.188.122 6380sentinel known-sentinel codis-easypass-4 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404csentinel known-sentinel codis-easypass-4 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645sentinel known-sentinel codis-easypass-4 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54sentinel known-sentinel codis-easypass-4 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4sentinel known-sentinel codis-easypass-4 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9sentinel monitor codis-easypass-5 192.168.188.124 6379 2sentinel failover-timeout codis-easypass-5 300000sentinel config-epoch codis-easypass-5 0sentinel leader-epoch codis-easypass-5 0sentinel known-slave codis-easypass-5 192.168.188.125 6380sentinel known-slave codis-easypass-5 192.168.188.121 6381sentinel known-sentinel codis-easypass-5 192.168.188.125 26379 d0268512448981b56e0eb76318af009b4575404csentinel known-sentinel codis-easypass-5 192.168.188.121 26379 28758dd53514376486fdec2acdb73f273fa90645sentinel known-sentinel codis-easypass-5 192.168.188.123 26379 9c1ccd7b64d0eb0fa0be4db4b4b86e4b8aa79c54sentinel known-sentinel codis-easypass-5 192.168.188.122 26379 2204a1a2c3c5387bb33ab34de52c4eeda59257d4sentinel known-sentinel codis-easypass-5 192.168.188.124 26379 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9sentinel current-epoch 0
五，Group内主从切换测试
Group3测试
Group3（主从从）
192.168.188.122:6379 192.168.188.123:6380 192.168.188.125:6381
1，首先，停止192.168.188.122:6379
[root@codis122 admin]# /usr/local/codis/admin/codis-server-admin-6379.sh stop
/usr/local/codis/admin/../config/redis-6379.conf
stopping codis-server ...
STOPPED
2，检测发现192.168.188.125:6381被选举为主了，切换过程大概15秒内
8980:X 19 Sep 11:51:17.416 # +sdown master codis-easypass-3 192.168.188.122 63798980:X 19 Sep 11:51:17.461 # +new-epoch 18980:X 19 Sep 11:51:17.462 # +vote-for-leader 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 18980:X 19 Sep 11:51:17.488 # +odown master codis-easypass-3 192.168.188.122 6379 #quorum 6/28980:X 19 Sep 11:51:17.488 # Next failover delay: I will not start a failover before Tue Sep 19 12:01:17 20178980:X 19 Sep 11:51:18.283 # +config-update-from sentinel 6ee9b78f205908be1bbbe5dc9cc5a55cb6d1f3f9 192.168.188.124 26379 @ codis-easypass-3 192.168.188.122 63798980:X 19 Sep 11:51:18.283 # +switch-master codis-easypass-3 192.168.188.122 6379 192.168.188.125 63818980:X 19 Sep 11:51:18.284 * +slave slave 192.168.188.123:6380 192.168.188.123 6380 @ codis-easypass-3 192.168.188.125 63818980:X 19 Sep 11:51:18.284 * +slave slave 192.168.188.122:6379 192.168.188.122 6379 @ codis-easypass-3 192.168.188.125 63818980:X 19 Sep 11:51:48.285 # +sdown slave 192.168.188.122:6379 192.168.188.122 6379 @ codis-easypass-3 192.168.188.125 63818980:X 19 Sep 11:52:07.256 # -sdown slave 192.168.188.122:6379 192.168.188.122 6379 @ codis-easypass-3 192.168.188.125 6381
3，最后启动192.168.188.122:6379，变成了Group3的一个从
[root@codis122 admin]# /usr/local/codis/admin/codis-server-admin-6379.sh start
/usr/local/codis/admin/../config/redis-6379.conf
starting codis-server ...
18701:X 19 Sep 11:52:07.331 # -sdown slave 192.168.188.122:6379 192.168.188.122 6379 @ codis-easypass-3 192.168.188.125 6381
本文出自 “jerrymin” 博客，请务必保留此出处http://jerrymin.blog.51cto.com/3002256/1966926