利用sysbench工具测试MHA
1. sysbench准备数据
sysbench /usr/share/sysbench/oltp_read_write.lua \
--mysql-host=192.168.56.111 \
--mysql-user=wanbin \
--mysql-password=mysql \
--mysql-db=sbtest \
--db-driver=mysql \
--tables=10 \
--table-size=500000 \
--report-interval=10 \
--threads=128 \
--time=120 \
prepare
prepare/run/cleanup
prepare:准备数据，
run：执行压测，
cleanup:清除数据
sysbench 1.0.15 (using bundled LuaJIT 2.1.0-beta2)
Initializing worker threads...
Creating table 'sbtest10'...
Creating table 'sbtest6'...
Creating table 'sbtest1'...
Creating table 'sbtest3'...
Creating table 'sbtest2'...
Creating table 'sbtest7'...
Creating table 'sbtest5'...
Creating table 'sbtest8'...
Creating table 'sbtest4'...
Creating table 'sbtest9'...
Inserting 500000 records into 'sbtest10'
Inserting 500000 records into 'sbtest9'
Inserting 500000 records into 'sbtest6'
Inserting 500000 records into 'sbtest5'
Inserting 500000 records into 'sbtest8'
Inserting 500000 records into 'sbtest4'
Inserting 500000 records into 'sbtest7'
Inserting 500000 records into 'sbtest3'
Inserting 500000 records into 'sbtest2'
Inserting 500000 records into 'sbtest1'
Creating a secondary index on 'sbtest5'...
Creating a secondary index on 'sbtest6'...
Creating a secondary index on 'sbtest9'...
Creating a secondary index on 'sbtest10'...
Creating a secondary index on 'sbtest7'...
Creating a secondary index on 'sbtest3'...
Creating a secondary index on 'sbtest8'...
Creating a secondary index on 'sbtest2'...
Creating a secondary index on 'sbtest4'...
Creating a secondary index on 'sbtest1'...
2. sysbench开始压测
sysbench /usr/share/sysbench/oltp_read_write.lua \
--mysql-host=192.168.56.111 \
--mysql-user=wanbin \
--mysql-password=mysql \
--mysql-db=sbtest \
--db-driver=mysql \
--tables=10 \
--table-size=500000 \
--report-interval=10 \
--threads=128 \
--time=120 \
run
3. master模拟意外宕机
# ps -ef|grep mysqld
avahi 741 1 0 08:50 ? 00:00:00 avahi-daemon: running [mysqldb1.local]
mysql 4741 4228 2 11:03 pts/3 00:06:30 mysqld --defaults-file=/etc/my3306.cnf
root 8346 8288 0 15:34 pts/0 00:00:00 grep --color=auto mysqld
# kill -9 4741
4. mysqldb2 上观察mha状态
# tail -f /var/log/masterha/app1/manager.log
Mon Oct 29 15:34:40 2018 - [warning] Got error on MySQL select ping: 2013 (Lost connection to MySQL server during query)
Mon Oct 29 15:34:40 2018 - [info] Executing secondary network check script: /usr/local/bin/masterha_secondary_check -s mysqldb2 -s mysqldb1 --user=root --master_host=192.168.56.100 --master_ip=192.168.56.100 --master_port=3306 --master_user=mha_rep --master_password=123456 --ping_type=SELECT
Mon Oct 29 15:34:40 2018 - [info] Executing SSH check script: exit 0
Mon Oct 29 15:34:40 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.56.100' (111))
Mon Oct 29 15:34:40 2018 - [warning] Connection failed 2 time(s)..
Mon Oct 29 15:34:41 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.56.100' (111))
Mon Oct 29 15:34:41 2018 - [warning] Connection failed 3 time(s)..
Mon Oct 29 15:34:42 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.56.100' (111))
Mon Oct 29 15:34:42 2018 - [warning] Connection failed 4 time(s)..
Mon Oct 29 15:34:45 2018 - [warning] HealthCheck: Got timeout on checking SSH connection to 192.168.56.100! at /usr/local/share/perl5/MHA/HealthCheck.pm line 342.
Monitoring server mysqldb2 is reachable, Master is not reachable from mysqldb2. OK.
Monitoring server mysqldb1 is reachable, Master is not reachable from mysqldb1. OK.
Mon Oct 29 15:34:53 2018 - [info] Master is not reachable from all other monitoring servers. Failover should start.
Mon Oct 29 15:34:53 2018 - [warning] Master is not reachable from health checker!
Mon Oct 29 15:34:53 2018 - [warning] Master 192.168.56.100(192.168.56.100:3306) is not reachable!
Mon Oct 29 15:34:53 2018 - [warning] SSH is NOT reachable.
Mon Oct 29 15:34:53 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status..
Mon Oct 29 15:34:53 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Oct 29 15:34:53 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Mon Oct 29 15:34:53 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Mon Oct 29 15:34:55 2018 - [info] GTID failover mode = 1
Mon Oct 29 15:34:55 2018 - [info] Dead Servers:
Mon Oct 29 15:34:55 2018 - [info] 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:34:55 2018 - [info] Alive Servers:
Mon Oct 29 15:34:55 2018 - [info] 192.168.56.200(192.168.56.200:3306)
Mon Oct 29 15:34:55 2018 - [info] 192.168.56.210(192.168.56.210:3306)
Mon Oct 29 15:34:55 2018 - [info] Alive Slaves:
Mon Oct 29 15:34:55 2018 - [info] 192.168.56.200(192.168.56.200:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Mon Oct 29 15:34:55 2018 - [info] GTID ON
Mon Oct 29 15:34:55 2018 - [info] Replicating from 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:34:55 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Oct 29 15:34:55 2018 - [info] 192.168.56.210(192.168.56.210:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Mon Oct 29 15:34:55 2018 - [info] GTID ON
Mon Oct 29 15:34:55 2018 - [info] Replicating from 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:34:55 2018 - [info] Not candidate for the new Master (no_master is set)
Mon Oct 29 15:34:55 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Oct 29 15:34:55 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Mon Oct 29 15:34:55 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Mon Oct 29 15:34:55 2018 - [info] Checking slave configurations..
Mon Oct 29 15:34:55 2018 - [info] Checking replication filtering settings..
Mon Oct 29 15:34:55 2018 - [info] Replication filtering check ok.
Mon Oct 29 15:34:55 2018 - [info] Master is down!
Mon Oct 29 15:34:55 2018 - [info] Terminating monitoring script.
Mon Oct 29 15:34:55 2018 - [info] Got exit code 20 (Master dead).
Mon Oct 29 15:34:55 2018 - [info] MHA::MasterFailover version 0.57.
Mon Oct 29 15:34:55 2018 - [info] Starting master failover.
Mon Oct 29 15:34:55 2018 - [info]
Mon Oct 29 15:34:55 2018 - [info] * Phase 1: Configuration Check Phase..
Mon Oct 29 15:34:55 2018 - [info]
Mon Oct 29 15:34:56 2018 - [info] GTID failover mode = 1
Mon Oct 29 15:34:56 2018 - [info] Dead Servers:
Mon Oct 29 15:34:56 2018 - [info] 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:34:56 2018 - [info] Checking master reachability via MySQL(double check)...
Mon Oct 29 15:34:56 2018 - [info] ok.
Mon Oct 29 15:34:56 2018 - [info] Alive Servers:
Mon Oct 29 15:34:56 2018 - [info] 192.168.56.200(192.168.56.200:3306)
Mon Oct 29 15:34:56 2018 - [info] 192.168.56.210(192.168.56.210:3306)
Mon Oct 29 15:34:56 2018 - [info] Alive Slaves:
Mon Oct 29 15:34:56 2018 - [info] 192.168.56.200(192.168.56.200:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Mon Oct 29 15:34:56 2018 - [info] GTID ON
Mon Oct 29 15:34:56 2018 - [info] Replicating from 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:34:56 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Oct 29 15:34:56 2018 - [info] 192.168.56.210(192.168.56.210:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Mon Oct 29 15:34:56 2018 - [info] GTID ON
Mon Oct 29 15:34:56 2018 - [info] Replicating from 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:34:56 2018 - [info] Not candidate for the new Master (no_master is set)
Mon Oct 29 15:34:56 2018 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln309] Last failover was done at 2018/10/29 10:09:03. Current time is too early to do failover again. If you want to do failover, manually remove /var/log/masterha/app1/app1.failover.complete and run this script again.
Mon Oct 29 15:34:56 2018 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/local/bin/masterha_manager line 65.
可以看到自动切换失败，需要删除/var/log/masterha/app1/app1.failover.complete
rm -rf /var/log/masterha/app1/app1.failover.complete
5. 手工failover切换master
# masterha_master_switch --conf=/etc/masterha/app1.cnf --dead_master_host=192.168.56.100 --master_state=dead --new_master_host=192.168.56.200 --ignore_last_failover
--ignore_last_failover：在缺省情况下，如果MHA检测到连续发生宕机，且两次宕机间隔不足8小时的话，
则不会进行Failover，之所以这样限制是为了避免ping-pong效应。
该参数代表忽略上次MHA触发切换产生的文件，默认情况下，MHA发生切换后会在日志目录，
也就是 /var/log/masterha/app1产生app1.failover.complete文件，
下次再次切换的时候如果发现该目录下存在该文件将不允许触发切换，除非在第一次切换后收到删除该文件
# masterha_master_switch --conf=/etc/masterha/app1.cnf --dead_master_host=192.168.56.100 --master_state=dead --new_master_host=192.168.56.200 --ignore_last_failover
--dead_master_ip=<dead_master_ip> is not set. Using 192.168.56.100.
--dead_master_port=<dead_master_port> is not set. Using 3306.
Mon Oct 29 15:49:04 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Oct 29 15:49:04 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Mon Oct 29 15:49:04 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Mon Oct 29 15:49:04 2018 - [info] MHA::MasterFailover version 0.57.
Mon Oct 29 15:49:04 2018 - [info] Starting master failover.
Mon Oct 29 15:49:04 2018 - [info]
Mon Oct 29 15:49:04 2018 - [info] * Phase 1: Configuration Check Phase..
Mon Oct 29 15:49:04 2018 - [info]
Mon Oct 29 15:49:05 2018 - [info] GTID failover mode = 1
Mon Oct 29 15:49:05 2018 - [info] Dead Servers:
Mon Oct 29 15:49:05 2018 - [info] 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:49:05 2018 - [info] Checking master reachability via MySQL(double check)...
Mon Oct 29 15:49:05 2018 - [info] ok.
Mon Oct 29 15:49:05 2018 - [info] Alive Servers:
Mon Oct 29 15:49:05 2018 - [info] 192.168.56.200(192.168.56.200:3306)
Mon Oct 29 15:49:05 2018 - [info] 192.168.56.210(192.168.56.210:3306)
Mon Oct 29 15:49:05 2018 - [info] Alive Slaves:
Mon Oct 29 15:49:05 2018 - [info] 192.168.56.200(192.168.56.200:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Mon Oct 29 15:49:05 2018 - [info] GTID ON
Mon Oct 29 15:49:05 2018 - [info] Replicating from 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:49:05 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Oct 29 15:49:05 2018 - [info] 192.168.56.210(192.168.56.210:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Mon Oct 29 15:49:05 2018 - [info] GTID ON
Mon Oct 29 15:49:05 2018 - [info] Replicating from 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:49:05 2018 - [info] Not candidate for the new Master (no_master is set)
Master 192.168.56.100(192.168.56.100:3306) is dead. Proceed? (yes/NO): yes
Mon Oct 29 15:49:06 2018 - [info] Starting GTID based failover.
Mon Oct 29 15:49:06 2018 - [info]
Mon Oct 29 15:49:06 2018 - [info] ** Phase 1: Configuration Check Phase completed.
Mon Oct 29 15:49:06 2018 - [info]
Mon Oct 29 15:49:06 2018 - [info] * Phase 2: Dead Master Shutdown Phase..
Mon Oct 29 15:49:06 2018 - [info]
Mon Oct 29 15:49:07 2018 - [info] HealthCheck: SSH to 192.168.56.100 is reachable.
Mon Oct 29 15:49:07 2018 - [info] Forcing shutdown so that applications never connect to the current master..
Mon Oct 29 15:49:07 2018 - [info] Executing master IP deactivation script:
Mon Oct 29 15:49:07 2018 - [info] /usr/local/bin/master_ip_failover --orig_master_host=192.168.56.100 --orig_master_ip=192.168.56.100 --orig_master_port=3306 --command=stopssh --ssh_user=root
IN SCRIPT TEST====/sbin/ifconfig eth0:0 down==/sbin/ifconfig eth0:0 192.168.56.111/24 up===
Disabling the VIP on old master: 192.168.56.100
Mon Oct 29 15:49:08 2018 - [info] done.
Mon Oct 29 15:49:08 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Mon Oct 29 15:49:08 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Mon Oct 29 15:49:08 2018 - [info]
Mon Oct 29 15:49:08 2018 - [info] * Phase 3: Master Recovery Phase..
Mon Oct 29 15:49:08 2018 - [info]
Mon Oct 29 15:49:08 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Mon Oct 29 15:49:08 2018 - [info]
Mon Oct 29 15:49:08 2018 - [info] The latest binary log file/position on all slaves is my3306_binlog.000027:860190114
Mon Oct 29 15:49:08 2018 - [info] Retrieved Gtid Set: 7390a401-b705-11e8-9ed9-080027b0b461:140923-149275
Mon Oct 29 15:49:08 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):
Mon Oct 29 15:49:08 2018 - [info] 192.168.56.200(192.168.56.200:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Mon Oct 29 15:49:08 2018 - [info] GTID ON
Mon Oct 29 15:49:08 2018 - [info] Replicating from 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:49:08 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Oct 29 15:49:08 2018 - [info] 192.168.56.210(192.168.56.210:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Mon Oct 29 15:49:08 2018 - [info] GTID ON
Mon Oct 29 15:49:08 2018 - [info] Replicating from 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:49:08 2018 - [info] Not candidate for the new Master (no_master is set)
Mon Oct 29 15:49:08 2018 - [info] The oldest binary log file/position on all slaves is my3306_binlog.000027:860190114
Mon Oct 29 15:49:08 2018 - [info] Retrieved Gtid Set: 7390a401-b705-11e8-9ed9-080027b0b461:140923-149275
Mon Oct 29 15:49:08 2018 - [info] Oldest slaves:
Mon Oct 29 15:49:08 2018 - [info] 192.168.56.200(192.168.56.200:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Mon Oct 29 15:49:08 2018 - [info] GTID ON
Mon Oct 29 15:49:08 2018 - [info] Replicating from 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:49:08 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Oct 29 15:49:08 2018 - [info] 192.168.56.210(192.168.56.210:3306) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Mon Oct 29 15:49:08 2018 - [info] GTID ON
Mon Oct 29 15:49:08 2018 - [info] Replicating from 192.168.56.100(192.168.56.100:3306)
Mon Oct 29 15:49:08 2018 - [info] Not candidate for the new Master (no_master is set)
Mon Oct 29 15:49:08 2018 - [info]
Mon Oct 29 15:49:08 2018 - [info] * Phase 3.3: Determining New Master Phase..
Mon Oct 29 15:49:08 2018 - [info]
Mon Oct 29 15:49:08 2018 - [info] 192.168.56.200 can be new master.
Mon Oct 29 15:49:08 2018 - [info] New master is 192.168.56.200(192.168.56.200:3306)
Mon Oct 29 15:49:08 2018 - [info] Starting master failover..
Mon Oct 29 15:49:08 2018 - [info]
From:
192.168.56.100(192.168.56.100:3306) (current master)
+--192.168.56.200(192.168.56.200:3306)
+--192.168.56.210(192.168.56.210:3306)
To:
192.168.56.200(192.168.56.200:3306) (new master)
+--192.168.56.210(192.168.56.210:3306)
Starting master switch from 192.168.56.100(192.168.56.100:3306) to 192.168.56.200(192.168.56.200:3306)? (yes/NO): yes
Mon Oct 29 15:49:09 2018 - [info] New master decided manually is 192.168.56.200(192.168.56.200:3306)
Mon Oct 29 15:49:09 2018 - [info]
Mon Oct 29 15:49:09 2018 - [info] * Phase 3.3: New Master Recovery Phase..
Mon Oct 29 15:49:09 2018 - [info]
Mon Oct 29 15:49:09 2018 - [info] Waiting all logs to be applied..
Mon Oct 29 15:49:09 2018 - [info] done.
Mon Oct 29 15:49:09 2018 - [info] Getting new master's binlog name and position..
Mon Oct 29 15:49:09 2018 - [info] my3306_binlog.000016:965341200
Mon Oct 29 15:49:09 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.56.200', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Mon Oct 29 15:49:09 2018 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: my3306_binlog.000016, 965341200, 7390a401-b705-11e8-9ed9-080027b0b461:1-149275,
a4774e9e-bba9-11e8-bf3e-08002712513f:1-2
Mon Oct 29 15:49:09 2018 - [info] Executing master IP activate script:
Mon Oct 29 15:49:09 2018 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.56.100 --orig_master_ip=192.168.56.100 --orig_master_port=3306 --new_master_host=192.168.56.200 --new_master_ip=192.168.56.200 --new_master_port=3306 --new_master_user='mha_rep' --new_master_password=xxx
Unknown option: new_master_user
Unknown option: new_master_password
IN SCRIPT TEST====/sbin/ifconfig eth0:0 down==/sbin/ifconfig eth0:0 192.168.56.111/24 up===
Enabling the VIP - 192.168.56.111/24 on the new master - 192.168.56.200
bash: /usr/bin/arping: No such file or directory
Mon Oct 29 15:49:10 2018 - [info] OK.
Mon Oct 29 15:49:10 2018 - [info] Setting read_only=0 on 192.168.56.200(192.168.56.200:3306)..
Mon Oct 29 15:49:10 2018 - [info] ok.
Mon Oct 29 15:49:10 2018 - [info] ** Finished master recovery successfully.
Mon Oct 29 15:49:10 2018 - [info] * Phase 3: Master Recovery Phase completed.
Mon Oct 29 15:49:10 2018 - [info]
Mon Oct 29 15:49:10 2018 - [info] * Phase 4: Slaves Recovery Phase..
Mon Oct 29 15:49:10 2018 - [info]
Mon Oct 29 15:49:10 2018 - [info]
Mon Oct 29 15:49:10 2018 - [info] * Phase 4.1: Starting Slaves in parallel..
Mon Oct 29 15:49:10 2018 - [info]
Mon Oct 29 15:49:10 2018 - [info] -- Slave recovery on host 192.168.56.210(192.168.56.210:3306) started, pid: 16740. Check tmp log /var/log/masterha/app1/192.168.56.210_3306_20181029154904.log if it takes time..
Mon Oct 29 15:49:12 2018 - [info]
Mon Oct 29 15:49:12 2018 - [info] Log messages from 192.168.56.210 ...
Mon Oct 29 15:49:12 2018 - [info]
Mon Oct 29 15:49:10 2018 - [info] Resetting slave 192.168.56.210(192.168.56.210:3306) and starting replication from the new master 192.168.56.200(192.168.56.200:3306)..
Mon Oct 29 15:49:10 2018 - [info] Executed CHANGE MASTER.
Mon Oct 29 15:49:11 2018 - [info] Slave started.
Mon Oct 29 15:49:11 2018 - [info] gtid_wait(7390a401-b705-11e8-9ed9-080027b0b461:1-149275,
a4774e9e-bba9-11e8-bf3e-08002712513f:1-2) completed on 192.168.56.210(192.168.56.210:3306). Executed 0 events.
Mon Oct 29 15:49:12 2018 - [info] End of log messages from 192.168.56.210.
Mon Oct 29 15:49:12 2018 - [info] -- Slave on host 192.168.56.210(192.168.56.210:3306) started.
Mon Oct 29 15:49:12 2018 - [info] All new slave servers recovered successfully.
Mon Oct 29 15:49:12 2018 - [info]
Mon Oct 29 15:49:12 2018 - [info] * Phase 5: New master cleanup phase..
Mon Oct 29 15:49:12 2018 - [info]
Mon Oct 29 15:49:12 2018 - [info] Resetting slave info on the new master..
Mon Oct 29 15:49:13 2018 - [info] 192.168.56.200: Resetting slave info succeeded.
Mon Oct 29 15:49:13 2018 - [info] Master failover to 192.168.56.200(192.168.56.200:3306) completed successfully.
Mon Oct 29 15:49:13 2018 - [info]
----- Failover Report -----
app1: MySQL Master failover 192.168.56.100(192.168.56.100:3306) to 192.168.56.200(192.168.56.200:3306) succeeded
Master 192.168.56.100(192.168.56.100:3306) is down!
Check MHA Manager logs at mysqldb2 for details.
Started manual(interactive) failover.
Invalidated master IP address on 192.168.56.100(192.168.56.100:3306)
Selected 192.168.56.200(192.168.56.200:3306) as a new master.
192.168.56.200(192.168.56.200:3306): OK: Applying all logs succeeded.
192.168.56.200(192.168.56.200:3306): OK: Activated master IP address.
192.168.56.210(192.168.56.210:3306): OK: Slave started, replicating from 192.168.56.200(192.168.56.200:3306)
192.168.56.200(192.168.56.200:3306): Resetting slave info succeeded.
Master failover to 192.168.56.200(192.168.56.200:3306) completed successfully.
6. 原mysqldb1手工加入集群
mysql> CHANGE MASTER TO MASTER_HOST='192.168.56.200', \
MASTER_PORT=3306, \
MASTER_AUTO_POSITION=1, \
MASTER_USER='repl', \
MASTER_PASSWORD='wanbin';
mysql> start slave;
7. 手工在线切换
masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=alive --new_master_host=192.168.56.100 --orig_master_is_new_slave