Preface
Yesterday I implemented a three-nodes PXC,but there were some errors when proceeding it.So I decide to research what's wrong with it.The operations and evidences shows below.
Procedure
Fisrt of all,I find out that yesterday's error is due to lack of "socat" package on first node zlm2.
//Here's the error log of it.
--08T08::18.198487+: [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21')
--08T08::18.200272+: [Note] WSREP: DONOR thread signaled with
which: no socat in (/usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.-rel22-29.26..Linux.x86_64.ssl102/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin)
--08T06::.782344Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************
--08T06::.784043Z WSREP_SST: [ERROR] socat not found in path: /usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.-rel22-29.26..Linux.x86_64.ssl102/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin
--08T06::.785926Z WSREP_SST: [ERROR] ******************************************************
--08T08::18.786496+: [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21': (No such file or directory)
--08T08::18.786587+: [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21'
--08T08::18.787610+: [Warning] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) failed: - (No such file or directory) [root@zlm2 :: /data/mysql/mysql3308/data]
#rpm -qa|grep socat [root@zlm2 :: /data/mysql/mysql3308/data]
#yum install socat
... //Omitted. Installed:
socat.x86_64 :1.7.3.2-.el7 Complete!
Restart the PXC mysqld on zlm2.
[root@zlm2 :: /data/mysql/mysql3308/data]
#mysqladmin -P3308 shutdown [root@zlm2 :: /data/mysql/mysql3308/data]
#ps aux|grep mysqld
mysql 0.1 22.4 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld [root@zlm2 :: /data/mysql/mysql3308/data]
#ps aux|grep mysqld
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld [root@zlm2 :: /data/mysql/mysql3308/data]
#mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster &
[] [root@zlm2 :: /data/mysql/mysql3308/data]
#ps aux|grep mysqld
mysql 0.4 22.1 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld
It still show the ERROR output when node zlm3 startup.
--08T09::14.789691+: [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') connection established to 2bcb8dba tcp://192.168.1.102:4567
--08T09::14.790669+: [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
--08T09::15.289238+: [Note] WSREP: declaring 2bcb8dba at tcp://192.168.1.102:4567 stable
--08T09::15.290161+: [Note] WSREP: Node e783b5a9 state primary
--08T09::15.291214+: [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,2bcb8dba,)
memb {
2bcb8dba,
e783b5a9,
}
joined {
}
left {
}
partitioned {
}
)
--08T09::15.291254+: [Note] WSREP: Save the discovered primary-component to disk
--08T09::15.291707+: [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =
--08T09::15.291726+: [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
--08T09::15.791844+: [Note] WSREP: STATE EXCHANGE: sent state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b
--08T09::15.793169+: [Note] WSREP: STATE EXCHANGE: got state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b from (zlm3)
--08T09::15.793198+: [Note] WSREP: STATE EXCHANGE: got state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b from (zlm2)
--08T09::15.793205+: [Note] WSREP: Quorum results:
version = ,
component = PRIMARY,
conf_id = ,
members = / (primary/total),
act_id = ,
last_appl. = ,
protocols = // (gcs/repl/appl),
group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7
--08T09::15.793211+: [Note] WSREP: Flow-control interval: [, ]
--08T09::15.793213+: [Note] WSREP: Trying to continue unpaused monitor
--08T09::15.793314+: [Note] WSREP: REPL Protocols: (, )
--08T09::15.793324+: [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version
--08T09::15.793327+: [Note] WSREP: Setting wsrep_ready to true
--08T09::15.793330+: [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: -> ) (Increment: -> )
--08T09::15.793332+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--08T09::15.793337+: [Note] WSREP: Assign initial position for certification: , protocol version:
--08T09::15.793351+: [Note] WSREP: Service thread queue flushed.
--08T09::16.392574+: [Note] WSREP: Member 0.0 (zlm3) requested state transfer from '*any*'. Selected 1.0 (zlm2)(SYNCED) as donor.
--08T09::16.392599+: [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: )
--08T09::16.392659+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--08T09::16.392879+: [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21')
--08T09::16.396838+: [Note] WSREP: DONOR thread signaled with
--08T09::17.917352+: [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') turning message relay requesting off
--08T07::.003790Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102
--08T09::27.124206+: [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES)
--08T09::27.138281+: [Note] Aborted connection to db: 'unconnected' user: 'sst' host: 'localhost' (Got an error reading communication packets)
--08T07::.139336Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************
--08T07::.140923Z WSREP_SST: [ERROR] xtrabackup finished with error: . Check /data/mysql/mysql3308/data//innobackup.backup.log
--------------- innobackup.backup.log (START) --------------------
xtrabackup: recognized server arguments: --datadir=/data/mysql/mysql3308/data --server-id= --open_files_limit= --log_bin=/data/mysql/mysql3308/logs/mysql-bin --innodb_data_file_path=ibdata1:100M:autoextend --innodb_buffer_pool_filename=ib_buffer_pool --innodb_log_group_home_dir=./ --innodb_log_files_in_group= --innodb_log_file_size=100M --innodb_file_per_table= --innodb_open_files= --innodb_page_size=16k --innodb_read_io_threads= --innodb_write_io_threads= --innodb_io_capacity= --innodb_doublewrite= --innodb_log_buffer_size=128M --innodb_flush_log_at_trx_commit= --innodb_buffer_pool_size=100M --innodb_flush_method=O_DIRECT --defaults_group=mysqld --parallel=
xtrabackup: recognized client arguments: --datadir=/data/mysql/mysql3308/data --server-id= --open_files_limit= --log_bin=/data/mysql/mysql3308/logs/mysql-bin --innodb_data_file_path=ibdata1:100M:autoextend --innodb_buffer_pool_filename=ib_buffer_pool --innodb_log_group_home_dir=./ --innodb_log_files_in_group= --innodb_log_file_size=100M --innodb_file_per_table= --innodb_open_files= --innodb_page_size=16k --innodb_read_io_threads= --innodb_write_io_threads= --innodb_io_capacity= --innodb_doublewrite= --innodb_log_buffer_size=128M --innodb_flush_log_at_trx_commit= --innodb_buffer_pool_size=100M --innodb_flush_method=O_DIRECT --defaults_group=mysqld --parallel= --port= --user=sst --password=* --socket=/tmp/mysql3308.sock --lock-ddl= --backup= --galera-info= --binlog-info=ON --stream=xbstream --target-dir=/tmp/pxc_sst_sn2p/donor_xb_qh5x
:: version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;port=3308;mysql_socket=/tmp/mysql3308.sock' as 'sst' (using password: YES).
Failed to connect to MySQL server: DBI connect(';mysql_read_default_group=xtrabackup;port=3308;mysql_socket=/tmp/mysql3308.sock','sst',...) failed: Access denied for user 'sst'@'192.168.1.101' (using password: YES) at - line .
:: Connecting to MySQL server host: localhost, user: sst, password: set, port: , socket: /tmp/mysql3308.sock
Using server version 5.7.--29.26-log
:: Executing LOCK TABLES FOR BACKUP...
Error: failed to execute query SHOW ENGINE INNODB STATUS: Access denied; you need (at least one of) the PROCESS privilege(s) for this operation
--------------- innobackup.backup.log (END) ----------------------
--08T07::.143691Z WSREP_SST: [ERROR] ******************************************************
--08T07::.145779Z WSREP_SST: [ERROR] Cleanup after exit with status:
--08T09::27.158932+: [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21': (Invalid argument)
--08T09::27.159041+: [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21'
--08T09::27.160430+: [Warning] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) failed: - (Invalid argument) //Error: failed to execute query SHOW ENGINE INNODB STATUS: Access denied; you need (at least one of) the PROCESS privilege(s) for this operation.
//It seams the user sst@localhost is lack of privilege of "PROCESS".
Modify the privilege of user sst@localhost.
(zlm@192.168.1.101 )[(none)]>select user,host from mysql.user;
+---------------+-------------+
| user | host |
+---------------+-------------+
| zlm | 192.168..% |
| mysql.session | localhost |
| mysql.sys | localhost |
| root | localhost |
| sst | localhost |
+---------------+-------------+
rows in set (0.02 sec) (zlm@192.168.1.101 )[(none)]>show grants for sst@localhost;
+---------------------------------------------------------------------------+
| Grants for sst@localhost |
+---------------------------------------------------------------------------+
| GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sst'@'localhost' |
+---------------------------------------------------------------------------+
row in set (0.00 sec) (zlm@192.168.1.101 )[(none)]>grant process on *.* to 'sst'@'localhost';
ERROR (): Access denied for user 'zlm'@'192.168.1.%' (using password: YES)
(zlm@192.168.1.101 )[(none)]>exit
Bye [root@zlm2 :: ~]
#mysql -uroot -pPassw0rd -hlocalhost -S /tmp/mysql3308.sock
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.--29.26-log Percona XtraDB Cluster binary (GPL) 5.7.-29.26, Revision da86071, wsrep_29. Copyright (c) - Percona LLC and/or its affiliates
Copyright (c) , , Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. (root@localhost mysql3308.sock)[(none)]>grant process on *.* to sst@localhost;
Query OK, rows affected, warning (0.01 sec) (root@localhost mysql3308.sock)[(none)]>show grants for sst@localhost;
+------------------------------------------------------------------------------------+
| Grants for sst@localhost |
+------------------------------------------------------------------------------------+
| GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sst'@'localhost' |
+------------------------------------------------------------------------------------+
row in set (0.00 sec)
Restart PXC again on zlm2.
[root@zlm2 :: ~]
#ps aux|grep mysqld
mysql 0.1 22.2 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld [root@zlm2 :: ~]
#mysqladmin -uzlm -pzlmzlm -h192.168.1. -P3308 shutdown
mysqladmin: [Warning] Using a password on the command line interface can be insecure. [root@zlm2 :: ~]
#ps aux|grep mysqld
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld [root@zlm2 :: /data/mysql/mysql3308/data]
#mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster &
[]
[] Done mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster [root@zlm2 :: /data/mysql/mysql3308/data]
#ps aux|grep mysqld
mysql 5.8 20.6 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld //No ERROR messages were found in error log.
Restart PXC again on zlm3 and check the output of error log.
[root@zlm3 :: ~]
#ps aux|grep mysqld
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld
[]+ Exit mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf [root@zlm3 :: /data/mysql/mysql3308/data]
#cat error.log |grep ERROR
--08T08::.819918Z WSREP_SST: [ERROR] Cleanup after exit with status:
--08T10::19.826527+: [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.102' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' : (Operation not permitted)
--08T10::19.826557+: [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
--08T10::19.826565+: [ERROR] WSREP: SST script aborted with error (Operation not permitted)
--08T10::19.826592+: [ERROR] WSREP: SST failed: (Operation not permitted)
--08T10::19.826595+: [ERROR] Aborting
Check the error log on donor node zlm2 begin with the time of starting mysqld on joiner zlm3.
--08T10::47.539174+: [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') connection established to b3de093b tcp://192.168.1.102:4567
--08T10::47.539620+: [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
--08T10::48.038167+: [Note] WSREP: declaring b3de093b at tcp://192.168.1.102:4567 stable
--08T10::48.038656+: [Note] WSREP: Node d4006db4 state primary
--08T10::48.039842+: [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,b3de093b,)
memb {
b3de093b,
d4006db4,
}
joined {
}
left {
}
partitioned {
}
)
--08T10::48.039890+: [Note] WSREP: Save the discovered primary-component to disk
--08T10::48.040138+: [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =
--08T10::48.040154+: [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
--08T10::48.540351+: [Note] WSREP: STATE EXCHANGE: sent state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067
--08T10::48.540644+: [Note] WSREP: STATE EXCHANGE: got state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067 from (zlm3)
--08T10::48.541101+: [Note] WSREP: STATE EXCHANGE: got state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067 from (zlm2)
--08T10::48.541140+: [Note] WSREP: Quorum results:
version = ,
component = PRIMARY,
conf_id = ,
members = / (primary/total),
act_id = ,
last_appl. = ,
protocols = // (gcs/repl/appl),
group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7
--08T10::48.541150+: [Note] WSREP: Flow-control interval: [, ]
--08T10::48.541155+: [Note] WSREP: Trying to continue unpaused monitor
--08T10::48.541224+: [Note] WSREP: REPL Protocols: (, )
--08T10::48.541234+: [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version
--08T10::48.541238+: [Note] WSREP: Setting wsrep_ready to true
--08T10::48.541241+: [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: -> ) (Increment: -> )
--08T10::48.541245+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--08T10::48.541252+: [Note] WSREP: Assign initial position for certification: , protocol version:
--08T10::48.541264+: [Note] WSREP: Service thread queue flushed.
--08T10::49.007421+: [Note] WSREP: Member 0.0 (zlm3) requested state transfer from '*any*'. Selected 1.0 (zlm2)(SYNCED) as donor.
--08T10::49.007441+: [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: )
--08T10::49.007480+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--08T10::49.007626+: [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:22')
--08T10::49.012384+: [Note] WSREP: DONOR thread signaled with
--08T10::50.911071+: [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') turning message relay requesting off
--08T08::.535108Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102
--08T10::59.626864+: [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES)
--08T10::05.537365+: [Note] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) complete.
--08T10::05.537399+: [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: )
--08T10::05.542864+: [Note] WSREP: Member 1.0 (zlm2) synced with group.
--08T10::05.542949+: [Note] WSREP: Shifting JOINED -> SYNCED (TO: )
--08T10::05.543015+: [Note] WSREP: Synchronized with group, ready for connections
--08T10::05.543022+: [Note] WSREP: Setting wsrep_ready to true
--08T10::05.543027+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--08T10::22.477724+: [Note] WSREP: forgetting b3de093b (tcp://192.168.1.102:4567)
--08T10::22.479542+: [Note] WSREP: Node d4006db4 state primary
--08T10::22.479612+: [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,d4006db4,)
memb {
d4006db4,
}
joined {
}
left {
}
partitioned {
b3de093b,
}
)
--08T10::22.479621+: [Note] WSREP: Save the discovered primary-component to disk
--08T10::22.479763+: [Note] WSREP: forgetting b3de093b (tcp://192.168.1.102:4567)
--08T10::22.479875+: [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =
--08T10::22.479976+: [Note] WSREP: STATE_EXCHANGE: sent state UUID: c8aef280-9ae8-11e8-80c9-d74e514aa266
--08T10::22.479976+: [Note] WSREP: STATE EXCHANGE: sent state msg: c8aef280-9ae8-11e8-80c9-d74e514aa266
--08T10::22.479976+: [Note] WSREP: STATE EXCHANGE: got state msg: c8aef280-9ae8-11e8-80c9-d74e514aa266 from (zlm2)
--08T10::22.479978+: [Note] WSREP: Quorum results:
version = ,
component = PRIMARY,
conf_id = ,
members = / (primary/total),
act_id = ,
last_appl. = ,
protocols = // (gcs/repl/appl),
group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7
--08T10::22.479983+: [Note] WSREP: Flow-control interval: [, ]
--08T10::22.480000+: [Note] WSREP: Trying to continue unpaused monitor
--08T10::22.480032+: [Note] WSREP: REPL Protocols: (, )
--08T10::22.480037+: [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version
--08T10::22.480040+: [Note] WSREP: Setting wsrep_ready to true
--08T10::22.480043+: [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: -> ) (Increment: -> )
--08T10::22.480046+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--08T10::22.480051+: [Note] WSREP: Assign initial position for certification: , protocol version:
--08T10::22.480060+: [Note] WSREP: Service thread queue flushed.
--08T10::26.085271+: [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') connection established to b3de093b tcp://192.168.1.102:4567
--08T10::26.085306+: [Warning] WSREP: discarding established (time wait) b3de093b (tcp://192.168.1.102:4567)
--08T10::27.948973+: [Note] WSREP: cleaning up b3de093b (tcp://192.168.1.102:4567) //There's no ERROR messages at all this time.
Suplemented on August 9.
The error above was due to the parameter of "innodb_log_group_home_dir",after I omitted it,there was no error message any more.
Evidence on node zlm2.
//Start the PCX on node zlm2.
[root@zlm2 :: ~]
#mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster &
[] //The error log of node zlm2 when node zlm3 is joining the cluster.
--09T04::43.185049+: [Note] WSREP: (13eae368, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567
--09T04::43.185703+: [Note] WSREP: (13eae368, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
--09T04::43.683941+: [Note] WSREP: declaring 23fb8f7a at tcp://192.168.1.102:4567 stable
--09T04::43.684321+: [Note] WSREP: Node 13eae368 state primary
--09T04::43.684954+: [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,13eae368,)
memb {
13eae368,
23fb8f7a,
}
joined {
}
left {
}
partitioned {
}
)
--09T04::43.684992+: [Note] WSREP: Save the discovered primary-component to disk
--09T04::43.685286+: [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =
--09T04::43.685750+: [Note] WSREP: STATE_EXCHANGE: sent state UUID: -9b79-11e8-b078-6a52474ad103
--09T04::43.688438+: [Note] WSREP: STATE EXCHANGE: sent state msg: -9b79-11e8-b078-6a52474ad103
--09T04::43.688976+: [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from (zlm2)
--09T04::44.194166+: [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from (zlm3)
--09T04::44.194203+: [Note] WSREP: Quorum results:
version = ,
component = PRIMARY,
conf_id = ,
members = / (primary/total),
act_id = ,
last_appl. = ,
protocols = // (gcs/repl/appl),
group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7
--09T04::44.194210+: [Note] WSREP: Flow-control interval: [, ]
--09T04::44.194213+: [Note] WSREP: Trying to continue unpaused monitor
--09T04::44.194278+: [Note] WSREP: REPL Protocols: (, )
--09T04::44.194285+: [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version
--09T04::44.194288+: [Note] WSREP: Setting wsrep_ready to true
--09T04::44.194315+: [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: -> ) (Increment: -> )
--09T04::44.194320+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--09T04::44.194327+: [Note] WSREP: Assign initial position for certification: , protocol version:
--09T04::44.194337+: [Note] WSREP: Service thread queue flushed.
--09T04::44.870138+: [Note] WSREP: Member 1.0 (zlm3) requested state transfer from '*any*'. Selected 0.0 (zlm2)(SYNCED) as donor.
--09T04::44.870160+: [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: )
--09T04::44.870209+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--09T04::44.870406+: [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:22')
--09T04::44.872063+: [Note] WSREP: DONOR thread signaled with
--09T04::46.270023+: [Note] WSREP: (13eae368, 'tcp://0.0.0.0:4567') turning message relay requesting off
--09T02::.527214Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102
--09T04::55.751008+: [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES)
--09T04::03.636264+: [Note] WSREP: 0.0 (zlm2): State transfer to 1.0 (zlm3) complete.
--09T04::03.636295+: [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: )
--09T04::03.647735+: [Note] WSREP: Member 0.0 (zlm2) synced with group.
--09T04::03.647805+: [Note] WSREP: Shifting JOINED -> SYNCED (TO: )
--09T04::03.647828+: [Note] WSREP: Synchronized with group, ready for connections
--09T04::03.647833+: [Note] WSREP: Setting wsrep_ready to true
--09T04::03.647837+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--09T04::23.739120+: [Note] WSREP: 1.0 (zlm3): State transfer from 0.0 (zlm2) complete.
--09T04::23.739901+: [Note] WSREP: Member 1.0 (zlm3) synced with group.
Evidence on node zlm3(using xtrabackup-v2 mothed of sst).
//Start the PCX on node zlm3.
[root@zlm3 :: /data/mysql/mysql3308]
#mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf &
[] [root@zlm3 :: /data/mysql/mysql3308]
#ps aux|grep mysqld
mysql 3.1 21.1 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld //The intact error log of node zlm3 when joining the cluster.
[root@zlm3 :: /data/mysql/mysql3308/data]
#tail -f error.log --09T04::43.084230+: [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
--09T04::43.089848+: [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.
--09T04::43.089906+: [Note] mysqld (mysqld 5.7.--29.26-log) starting as process ...
--09T04::43.109751+: [Note] WSREP: Setting wsrep_ready to false
--09T04::43.109768+: [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.
--09T04::43.109772+: [Note] WSREP: wsrep_load(): loading provider library '/usr/local/mysql/lib/libgalera_smm.so'
--09T04::43.129981+: [Note] WSREP: wsrep_load(): Galera 3.26(r) by Codership Oy <info@codership.com> loaded successfully.
--09T04::43.130124+: [Note] WSREP: CRC-32C: using hardware acceleration.
--09T04::43.130414+: [Note] WSREP: Found saved state: ----:-, safe_to_bootstrap:
--09T04::43.141875+: [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/mysql3308/data/; base_host = 192.168.1.102; base_port = ; cert.log_conflicts = no; debug = no; evs.auto_evict = ; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = ; evs.send_window = ; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = ; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/mysql3308/data/; gcache.freeze_purge_at_seqno = -; gcache.keep_pages_count = ; gcache.keep_pages_size = ; gcache.mem_size = ; gcache.name = /data/mysql/mysql3308/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 8; socket.checksum = 2; socket.recv_buf_size = 212992;
--09T04::43.179573+: [Note] WSREP: GCache history reset: bd5525ab-9a15-11e8-aa0f-4b830c783fc7: -> ----:-
--09T04::43.183306+: [Note] WSREP: Assign initial position for certification: -, protocol version: -
--09T04::43.183338+: [Note] WSREP: Preparing to initiate SST/IST
--09T04::43.183342+: [Note] WSREP: Starting replication
--09T04::43.183355+: [Note] WSREP: Setting initial position to ----:-
--09T04::43.183507+: [Note] WSREP: Using CRC-32C for message checksums.
--09T04::43.183568+: [Note] WSREP: gcomm thread scheduling priority set to other:
--09T04::43.183674+: [Warning] WSREP: Fail to access the file (/data/mysql/mysql3308/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
--09T04::43.183679+: [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
--09T04::43.184132+: [Note] WSREP: GMCast version
--09T04::43.184337+: [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
--09T04::43.184344+: [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') multicast: , ttl:
--09T04::43.184483+: [Note] WSREP: EVS version
--09T04::43.184957+: [Note] WSREP: gcomm: connecting to group 'pxc_wubx', peer '192.168.1.101:,192.168.1.102:,192.168.103:'
--09T04::43.186863+: [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567
--09T04::43.186895+: [Warning] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.102:4567' points to own listening address, blacklisting
--09T04::43.188042+: [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') connection established to 13eae368 tcp://192.168.1.101:4567
--09T04::43.188190+: [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
--09T04::43.686789+: [Note] WSREP: declaring 13eae368 at tcp://192.168.1.101:4567 stable
--09T04::43.687321+: [Note] WSREP: Node 13eae368 state primary
--09T04::43.690123+: [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,13eae368,)
memb {
13eae368,
23fb8f7a,
}
joined {
}
left {
}
partitioned {
}
)
--09T04::43.690159+: [Note] WSREP: Save the discovered primary-component to disk
--09T04::43.690256+: [Note] WSREP: discarding pending addr without UUID: tcp://192.168.0.103:4567
--09T04::43.690263+: [Note] WSREP: discarding pending addr proto entry 0x28b0b70
--09T04::44.185955+: [Note] WSREP: gcomm: connected
--09T04::44.186036+: [Note] WSREP: Shifting CLOSED -> OPEN (TO: )
--09T04::44.186098+: [Note] WSREP: Waiting for SST/IST to complete.
--09T04::44.186339+: [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =
--09T04::44.186355+: [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
--09T04::44.186383+: [Note] WSREP: STATE EXCHANGE: sent state msg: -9b79-11e8-b078-6a52474ad103
--09T04::44.186388+: [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from (zlm2)
--09T04::44.189234+: [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from (zlm3)
--09T04::44.189263+: [Note] WSREP: Quorum results:
version = ,
component = PRIMARY,
conf_id = ,
members = / (primary/total),
act_id = ,
last_appl. = -,
protocols = // (gcs/repl/appl),
group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7
--09T04::44.189270+: [Note] WSREP: Flow-control interval: [, ]
--09T04::44.189273+: [Note] WSREP: Trying to continue unpaused monitor
--09T04::44.189277+: [Note] WSREP: Shifting OPEN -> PRIMARY (TO: )
--09T04::44.189468+: [Note] WSREP: State transfer required:
Group state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:
Local state: ----:-
--09T04::44.189478+: [Note] WSREP: REPL Protocols: (, )
--09T04::44.189485+: [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version
--09T04::44.189489+: [Note] WSREP: Setting wsrep_ready to true
--09T04::44.189491+: [Warning] WSREP: Gap in state sequence. Need state transfer.
--09T04::44.189493+: [Note] WSREP: Setting wsrep_ready to false
--09T04::44.192795+: [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.102' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' )
--09T02::.755132Z WSREP_SST: [WARNING] Found a stale sst_in_progress file: /data/mysql/mysql3308/data//sst_in_progress
--09T04::44.870937+: [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|192.168.1.102:/xtrabackup_sst//
--09T04::44.871047+: [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: -> ) (Increment: -> )
--09T04::44.871090+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--09T04::44.871140+: [Note] WSREP: Assign initial position for certification: , protocol version:
--09T04::44.871282+: [Note] WSREP: Service thread queue flushed.
--09T04::44.871758+: [Note] WSREP: Check if state gap can be serviced using IST
--09T04::44.871782+: [Note] WSREP: Local UUID: ---- != Group UUID: bd5525ab-9a15-11e8-aa0f-4b830c783fc7
--09T04::44.871834+: [Note] WSREP: State gap can't be serviced using IST. Switching to SST
--09T04::44.871842+: [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (----) does not match group state UUID (bd5525ab-9a15-11e8-aa0f-4b830c783fc7): (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():. IST will be unavailable.
--09T04::44.872523+: [Note] WSREP: Member 1.0 (zlm3) requested state transfer from '*any*'. Selected 0.0 (zlm2)(SYNCED) as donor.
--09T04::44.872545+: [Note] WSREP: Shifting PRIMARY -> JOINER (TO: )
--09T04::44.872679+: [Note] WSREP: Requesting state transfer: success, donor:
--09T04::44.872720+: [Note] WSREP: GCache history reset: ----: -> bd5525ab-9a15-11e8-aa0f-4b830c783fc7:
--09T02::.589512Z WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /data/mysql/mysql3308/data//.sst from previous state transfer. Removing
--09T02::.608370Z WSREP_SST: [INFO] Proceeding with SST.........
removed ‘/data/mysql/mysql3308/logs/mysql-bin.’
--09T02::.637863Z WSREP_SST: [INFO] ............Waiting for SST streaming to complete!
--09T04::46.189014+: [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') turning message relay requesting off
--09T02::.636108Z WSREP_SST: [INFO] Preparing the backup at /data/mysql/mysql3308/data//.sst
--09T04::03.649250+: [Note] WSREP: 0.0 (zlm2): State transfer to 1.0 (zlm3) complete.
--09T04::03.659780+: [Note] WSREP: Member 0.0 (zlm2) synced with group.
--09T02::.887135Z WSREP_SST: [INFO] Moving the backup to /data/mysql/mysql3308/data/
--09T02::.003073Z WSREP_SST: [INFO] Galera co-ords from recovery: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:
--09T04::22.015642+: [Note] WSREP: SST complete, seqno:
--09T04::22.020459+: [Warning] InnoDB: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead; Please refer to http://dev.mysql.com/doc/refman/5.7/en/set-transaction.html
--09T04::22.020620+: [Note] InnoDB: PUNCH HOLE support available
--09T04::22.020665+: [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
--09T04::22.020674+: [Note] InnoDB: Uses event mutexes
--09T04::22.020679+: [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
--09T04::22.020684+: [Note] InnoDB: Compressed tables use zlib 1.2.
--09T04::22.020688+: [Note] InnoDB: Using Linux native AIO
--09T04::22.020708+: [Note] InnoDB: Adjusting innodb_buffer_pool_instances from to since innodb_buffer_pool_size is less than MiB
--09T04::22.021332+: [Note] InnoDB: Number of pools:
--09T04::22.021561+: [Note] InnoDB: Using CPU crc32 instructions
--09T04::22.024636+: [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = , chunk size = 100M
--09T04::22.027579+: [Note] InnoDB: Completed initialization of buffer pool
--09T04::22.031734+: [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
--09T04::22.056548+: [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite
--09T04::22.058938+: [Note] InnoDB: Highest supported file format is Barracuda.
--09T04::22.176951+: [Note] InnoDB: Created parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite, size bytes
--09T04::22.311846+: [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
--09T04::22.311867+: [Note] InnoDB: Creating shared tablespace for temporary tables
--09T04::22.311914+: [Note] InnoDB: Setting file './ibtmp1' size to MB. Physically writing the file full; Please wait ...
--09T04::22.753228+: [Note] InnoDB: File './ibtmp1' size is now MB.
--09T04::22.753891+: [Note] InnoDB: redo rollback segment(s) found. redo rollback segment(s) are active.
--09T04::22.753902+: [Note] InnoDB: non-redo rollback segment(s) are active.
--09T04::22.754607+: [Note] InnoDB: Waiting for purge to start
--09T04::22.805815+: [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.22-rel22 started; log sequence number 2605608
--09T04::22.806243+: [Note] Plugin 'FEDERATED' is disabled.
--09T04::22.808992+: [Note] InnoDB: Loading buffer pool(s) from /data/mysql/mysql3308/data/ib_buffer_pool
--09T04::22.854750+: [Note] InnoDB: Buffer pool(s) load completed at ::
--09T04::22.865970+: [Note] Recovering after a crash using /data/mysql/mysql3308/logs/mysql-bin
--09T04::22.866002+: [Note] WSREP: Before binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)
--09T04::22.866021+: [Note] Starting crash recovery...
--09T04::22.866046+: [Note] Crash recovery finished.
--09T04::22.866057+: [Note] WSREP: After binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)
--09T04::22.875798+: [Note] Salting uuid generator variables, current_pid: , server_start_time: , bytes_sent: ,
--09T04::22.875871+: [Note] Generated uuid: '3ba41aa7-9b79-11e8-ad75-080027de0e0e', server_start_time: , bytes_sent:
--09T04::22.875894+: [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 3ba41aa7-9b79-11e8-ad75-080027de0e0e.
--09T04::23.579815+: [Note] Auto generated SSL certificates are placed in data directory.
--09T04::23.580319+: [Warning] CA certificate ca.pem is self signed.
--09T04::23.708603+: [Note] Auto generated RSA key files are placed in data directory.
--09T04::23.708726+: [Note] Server hostname (bind-address): '*'; port:
--09T04::23.708750+: [Note] IPv6 is available.
--09T04::23.708759+: [Note] - '::' resolves to '::';
--09T04::23.708775+: [Note] Server socket created on IP: '::'.
--09T04::23.715377+: [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
--09T04::23.715477+: [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
--09T04::23.715490+: [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
--09T04::23.715499+: [Warning] 'user' entry 'sst@localhost' ignored in --skip-name-resolve mode.
--09T04::23.715539+: [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
--09T04::23.715548+: [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
--09T04::23.715559+: [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
--09T04::23.716731+: [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
--09T04::23.716749+: [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
--09T04::23.739950+: [Note] Event Scheduler: Loaded events
--09T04::23.740507+: [Note] WSREP: Signalling provider to continue on SST completion.
--09T04::23.740531+: [Note] WSREP: Initialized wsrep sidno
--09T04::23.740553+: [Note] WSREP: SST received: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:
--09T04::23.740631+: [Note] mysqld: ready for connections.
Version: '5.7.22-22-29.26-log' socket: '/tmp/mysql3308.sock' port: Percona XtraDB Cluster binary (GPL) 5.7.-29.26, Revision da86071, wsrep_29.
--09T04::23.741168+: [Note] WSREP: 1.0 (zlm3): State transfer from 0.0 (zlm2) complete.
--09T04::23.741191+: [Note] WSREP: SST leaving flow control
--09T04::23.741196+: [Note] WSREP: Shifting JOINER -> JOINED (TO: )
--09T04::23.742029+: [Note] WSREP: Member 1.0 (zlm3) synced with group.
--09T04::23.742059+: [Note] WSREP: Shifting JOINED -> SYNCED (TO: )
--09T04::23.742077+: [Note] WSREP: Synchronized with group, ready for connections
--09T04::23.742082+: [Note] WSREP: Setting wsrep_ready to true
--09T04::23.742087+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Evidence on node zlm4(using rsync mothed of sst).
//Change the parameter "wsrep_sst_method" into "rysnc" in my3308.cnf file.
#loose-wsrep_sst_method=xtrabackup-v2
loose-wsrep_sst_method=rsync [root@zlm4 :: /data/mysql/mysql3308/data]
#tail -f error.log
--09T07::21.038606+: [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
--09T07::21.041387+: [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.
--09T07::21.041490+: [Note] mysqld (mysqld 5.7.--29.26-log) starting as process ...
--09T07::21.044740+: [Note] WSREP: Setting wsrep_ready to false
--09T07::21.044761+: [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.
--09T07::21.044767+: [Note] WSREP: wsrep_load(): loading provider library '/usr/local/mysql/lib/libgalera_smm.so'
--09T07::21.050035+: [Note] WSREP: wsrep_load(): Galera 3.26(r) by Codership Oy <info@codership.com> loaded successfully.
--09T07::21.050186+: [Note] WSREP: CRC-32C: using hardware acceleration.
--09T07::21.050565+: [Warning] WSREP: Could not open state file for reading: '/data/mysql/mysql3308/data//grastate.dat'
--09T07::21.050585+: [Warning] WSREP: No persistent state found. Bootstraping with default state
--09T07::21.050637+: [Note] WSREP: Found saved state: ----:-, safe_to_bootstrap:
--09T07::21.055209+: [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/mysql3308/data/; base_host = 192.168.1.103; base_port = ; cert.log_conflicts = no; debug = no; evs.auto_evict = ; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = ; evs.send_window = ; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = ; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/mysql3308/data/; gcache.freeze_purge_at_seqno = -; gcache.keep_pages_count = ; gcache.keep_pages_size = ; gcache.mem_size = ; gcache.name = /data/mysql/mysql3308/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 8; socket.checksum = 2; socket.recv_buf_size = 212992;
--09T07::21.069257+: [Note] WSREP: GCache history reset: ----: -> ----:-
--09T07::21.071550+: [Note] WSREP: Assign initial position for certification: -, protocol version: -
--09T07::21.071584+: [Note] WSREP: Preparing to initiate SST/IST
--09T07::21.071589+: [Note] WSREP: Starting replication
--09T07::21.071606+: [Note] WSREP: Setting initial position to ----:-
--09T07::21.071812+: [Note] WSREP: Using CRC-32C for message checksums.
--09T07::21.071888+: [Note] WSREP: gcomm thread scheduling priority set to other:
--09T07::21.072021+: [Warning] WSREP: Fail to access the file (/data/mysql/mysql3308/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
--09T07::21.072030+: [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
--09T07::21.072800+: [Note] WSREP: GMCast version
--09T07::21.072942+: [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
--09T07::21.072942+: [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') multicast: , ttl:
--09T07::21.073357+: [Note] WSREP: EVS version
--09T07::21.073926+: [Note] WSREP: gcomm: connecting to group 'pxc_wubx', peer '192.168.1.101:,192.168.1.102:,192.168.1.103:'
--09T07::21.076189+: [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') connection established to 61aad7a7 tcp://192.168.1.103:4567
--09T07::21.076220+: [Warning] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.103:4567' points to own listening address, blacklisting
--09T07::21.078146+: [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') connection established to 13eae368 tcp://192.168.1.101:4567
--09T07::21.078312+: [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
--09T07::21.080311+: [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567
--09T07::21.583579+: [Note] WSREP: declaring 13eae368 at tcp://192.168.1.101:4567 stable
--09T07::21.583614+: [Note] WSREP: declaring 23fb8f7a at tcp://192.168.1.102:4567 stable
--09T07::21.584818+: [Note] WSREP: Node 13eae368 state primary
--09T07::21.589478+: [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,13eae368,)
memb {
13eae368,
23fb8f7a,
61aad7a7,
}
joined {
}
left {
}
partitioned {
}
)
--09T07::21.589519+: [Note] WSREP: Save the discovered primary-component to disk
--09T07::22.075191+: [Note] WSREP: gcomm: connected
--09T07::22.075272+: [Note] WSREP: Shifting CLOSED -> OPEN (TO: )
--09T07::22.075348+: [Note] WSREP: Waiting for SST/IST to complete.
--09T07::22.075642+: [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =
--09T07::22.075658+: [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
--09T07::22.075689+: [Note] WSREP: STATE EXCHANGE: sent state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05
--09T07::22.075697+: [Note] WSREP: STATE EXCHANGE: got state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05 from (zlm2)
--09T07::22.075704+: [Note] WSREP: STATE EXCHANGE: got state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05 from (zlm3)
--09T07::22.077812+: [Note] WSREP: STATE EXCHANGE: got state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05 from (zlm4)
--09T07::22.077845+: [Note] WSREP: Quorum results:
version = ,
component = PRIMARY,
conf_id = ,
members = / (primary/total),
act_id = ,
last_appl. = -,
protocols = // (gcs/repl/appl),
group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7
--09T07::22.077853+: [Note] WSREP: Flow-control interval: [, ]
--09T07::22.077858+: [Note] WSREP: Trying to continue unpaused monitor
--09T07::22.077864+: [Note] WSREP: Shifting OPEN -> PRIMARY (TO: )
--09T07::22.077926+: [Note] WSREP: State transfer required:
Group state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:
Local state: ----:-
--09T07::22.077935+: [Note] WSREP: REPL Protocols: (, )
--09T07::22.077943+: [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version
--09T07::22.077948+: [Note] WSREP: Setting wsrep_ready to true
--09T07::22.077951+: [Warning] WSREP: Gap in state sequence. Need state transfer.
--09T07::22.077954+: [Note] WSREP: Setting wsrep_ready to false
--09T07::22.078033+: [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' )
which: no lsof in (/usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.-rel22-29.26..Linux.x86_64.ssl102/bin:/root/perl5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin)
--09T07::22.254544+: [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin'
Read: ''lsof' not found in PATH' //Here's the important message of error but has not "ERROR" keyword in the same line.So I've almost missed it.
--09T07::22.254571+: [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' : (No such file or directory)
--09T07::22.254631+: [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
--09T07::22.254639+: [ERROR] Aborting --09T07::22.254643+: [Note] Giving client threads a chance to die gracefully
--09T07::24.254907+: [Note] WSREP: Closing send monitor...
--09T07::24.254992+: [Note] WSREP: Closed send monitor.
--09T07::24.255028+: [Note] WSREP: gcomm: terminating thread
--09T07::24.255042+: [Note] WSREP: gcomm: joining thread
--09T07::24.255061+: [Note] WSREP: rollbacker thread exiting
--09T07::24.255377+: [Note] WSREP: gcomm: closing backend
--09T07::24.260692+: [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,13eae368,)
memb {
61aad7a7,
}
joined {
}
left {
}
partitioned {
13eae368,
23fb8f7a,
}
)
--09T07::24.260770+: [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') turning message relay requesting off
--09T07::24.260814+: [Note] WSREP: Current view of cluster as seen by this node
view ((empty))
--09T07::24.261439+: [Note] WSREP: gcomm: closed
--09T07::24.261690+: [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = , memb_num =
--09T07::24.261741+: [Note] WSREP: Flow-control interval: [, ]
--09T07::24.261746+: [Note] WSREP: Trying to continue unpaused monitor
--09T07::24.261753+: [Note] WSREP: Received NON-PRIMARY.
--09T07::24.261758+: [Note] WSREP: Shifting PRIMARY -> OPEN (TO: )
--09T07::24.261767+: [Note] WSREP: Received self-leave message.
--09T07::24.261774+: [Note] WSREP: Flow-control interval: [, ]
--09T07::24.261777+: [Note] WSREP: Trying to continue unpaused monitor
--09T07::24.261780+: [Note] WSREP: Received SELF-LEAVE. Closing connection.
--09T07::24.261784+: [Note] WSREP: Shifting OPEN -> CLOSED (TO: )
--09T07::24.261794+: [Note] WSREP: RECV thread exiting : Success
--09T07::24.262101+: [Note] WSREP: recv_thread() joined.
--09T07::24.262113+: [Note] WSREP: Closing replication queue.
--09T07::24.262117+: [Note] WSREP: Closing slave action queue.
--09T07::24.262162+: [Note] WSREP: Waiting for active wsrep applier to exit
--09T07::24.262170+: [Note] WSREP: Service disconnected.
--09T07::24.262173+: [Note] WSREP: Waiting to close threads......
--09T07::29.262290+: [Note] WSREP: Some threads may fail to exit.
--09T07::29.262337+: [Note] Binlog end
--09T07::29.262596+: [Note] mysqld: Shutdown complete //Install lsof tool.
[root@zlm4 :: /data/mysql/mysql3308]
#yum -y install lsof
... //Omitted. Installed:
lsof.x86_64 :4.87-.el7 Complete! //Start the node again.
[root@zlm4 :: /data/mysql/mysql3308]
#mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf &
[] [root@zlm4 :: /data/mysql/mysql3308]
#jobs
[]+ Running mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf & //Check the error log again.
[root@zlm4 :: /data/mysql/mysql3308/data]
#tail -f error.log
--09T08::02.899495+: [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
--09T08::02.900253+: [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.
--09T08::02.900326+: [Note] mysqld (mysqld 5.7.--29.26-log) starting as process ...
--09T08::02.902758+: [Note] WSREP: Setting wsrep_ready to false
--09T08::02.902772+: [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.
--09T08::02.902776+: [Note] WSREP: wsrep_load(): loading provider library '/usr/local/mysql/lib/libgalera_smm.so'
--09T08::02.906503+: [Note] WSREP: wsrep_load(): Galera 3.26(r) by Codership Oy <info@codership.com> loaded successfully.
--09T08::02.906567+: [Note] WSREP: CRC-32C: using hardware acceleration.
--09T08::02.906904+: [Note] WSREP: Found saved state: ----:-, safe_to_bootstrap:
--09T08::02.911237+: [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/mysql3308/data/; base_host = 192.168.1.103; base_port = ; cert.log_conflicts = no; debug = no; evs.auto_evict = ; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = ; evs.send_window = ; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = ; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/mysql3308/data/; gcache.freeze_purge_at_seqno = -; gcache.keep_pages_count = ; gcache.keep_pages_size = ; gcache.mem_size = ; gcache.name = /data/mysql/mysql3308/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 8; socket.checksum = 2; socket.recv_buf_size = 212992;
--09T08::02.921480+: [Note] WSREP: GCache history reset: ----: -> ----:-
--09T08::02.923600+: [Note] WSREP: Assign initial position for certification: -, protocol version: -
--09T08::02.923636+: [Note] WSREP: Preparing to initiate SST/IST
--09T08::02.923639+: [Note] WSREP: Starting replication
--09T08::02.923652+: [Note] WSREP: Setting initial position to ----:-
--09T08::02.923845+: [Note] WSREP: Using CRC-32C for message checksums.
--09T08::02.923912+: [Note] WSREP: gcomm thread scheduling priority set to other:
--09T08::02.923991+: [Warning] WSREP: Fail to access the file (/data/mysql/mysql3308/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
--09T08::02.923996+: [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
--09T08::02.924673+: [Note] WSREP: GMCast version
--09T08::02.924779+: [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
--09T08::02.924779+: [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') multicast: , ttl:
--09T08::02.925300+: [Note] WSREP: EVS version
--09T08::02.925433+: [Note] WSREP: gcomm: connecting to group 'pxc_wubx', peer '192.168.1.101:,192.168.1.102:,192.168.1.103:'
--09T08::02.928124+: [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') connection established to 74f3db69 tcp://192.168.1.103:4567
--09T08::02.928158+: [Warning] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.103:4567' points to own listening address, blacklisting
--09T08::02.928505+: [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') connection established to 13eae368 tcp://192.168.1.101:4567
--09T08::02.929670+: [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
--09T08::02.929755+: [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567
--09T08::03.429892+: [Note] WSREP: declaring 13eae368 at tcp://192.168.1.101:4567 stable
--09T08::03.429916+: [Note] WSREP: declaring 23fb8f7a at tcp://192.168.1.102:4567 stable
--09T08::03.431850+: [Note] WSREP: Node 13eae368 state primary
--09T08::03.434752+: [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,13eae368,)
memb {
13eae368,
23fb8f7a,
74f3db69,
}
joined {
}
left {
}
partitioned {
}
)
--09T08::03.434779+: [Note] WSREP: Save the discovered primary-component to disk
--09T08::03.927902+: [Note] WSREP: gcomm: connected
--09T08::03.927980+: [Note] WSREP: Shifting CLOSED -> OPEN (TO: )
--09T08::03.928042+: [Note] WSREP: Waiting for SST/IST to complete.
--09T08::03.928251+: [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =
--09T08::03.928261+: [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
--09T08::03.928285+: [Note] WSREP: STATE EXCHANGE: sent state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8
--09T08::03.928290+: [Note] WSREP: STATE EXCHANGE: got state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8 from (zlm2)
--09T08::03.928294+: [Note] WSREP: STATE EXCHANGE: got state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8 from (zlm3)
--09T08::03.929335+: [Note] WSREP: STATE EXCHANGE: got state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8 from (zlm4)
--09T08::03.929361+: [Note] WSREP: Quorum results:
version = ,
component = PRIMARY,
conf_id = ,
members = / (primary/total),
act_id = ,
last_appl. = -,
protocols = // (gcs/repl/appl),
group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7
--09T08::03.929366+: [Note] WSREP: Flow-control interval: [, ]
--09T08::03.929369+: [Note] WSREP: Trying to continue unpaused monitor
--09T08::03.929372+: [Note] WSREP: Shifting OPEN -> PRIMARY (TO: )
--09T08::03.929413+: [Note] WSREP: State transfer required:
Group state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:
Local state: ----:-
--09T08::03.929419+: [Note] WSREP: REPL Protocols: (, )
--09T08::03.929425+: [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version
--09T08::03.929428+: [Note] WSREP: Setting wsrep_ready to true
--09T08::03.929430+: [Warning] WSREP: Gap in state sequence. Need state transfer.
--09T08::03.929432+: [Note] WSREP: Setting wsrep_ready to false
--09T08::03.929496+: [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' )
--09T06::.047845Z WSREP_SST: [INFO] Waiting for data-dir through rsync................ //Begin to transfer data using rsync mode.
--09T08::04.319479+: [Note] WSREP: Prepared SST/IST request: rsync|192.168.1.103:/rsync_sst
--09T08::04.319504+: [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: -> ) (Increment: -> )
--09T08::04.319510+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
--09T08::04.319527+: [Note] WSREP: Assign initial position for certification: , protocol version:
--09T08::04.319546+: [Note] WSREP: Service thread queue flushed.
--09T08::04.319558+: [Note] WSREP: Check if state gap can be serviced using IST
--09T08::04.319566+: [Note] WSREP: Local UUID: ---- != Group UUID: bd5525ab-9a15-11e8-aa0f-4b830c783fc7
--09T08::04.319598+: [Note] WSREP: State gap can't be serviced using IST. Switching to SST
--09T08::04.319602+: [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (----) does not match group state UUID (bd5525ab-9a15-11e8-aa0f-4b830c783fc7): (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():. IST will be unavailable.
--09T08::04.321186+: [Note] WSREP: Member 2.0 (zlm4) requested state transfer from '*any*'. Selected 0.0 (zlm2)(SYNCED) as donor.
--09T08::04.321208+: [Note] WSREP: Shifting PRIMARY -> JOINER (TO: )
--09T08::04.321275+: [Note] WSREP: Requesting state transfer: success, donor:
--09T08::04.321291+: [Note] WSREP: GCache history reset: ----: -> bd5525ab-9a15-11e8-aa0f-4b830c783fc7:
--09T08::05.939711+: [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') turning message relay requesting off
--09T08::26.343649+: [Note] WSREP: 0.0 (zlm2): State transfer to 2.0 (zlm4) complete.
--09T08::26.344408+: [Note] WSREP: Member 0.0 (zlm2) synced with group.
--09T06::.399740Z WSREP_SST: [INFO] ..............rsync completed //Finished rsync transfer.
--09T08::26.929160+: [Note] WSREP: SST complete, seqno:
--09T08::27.003561+: [Warning] InnoDB: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead; Please refer to http://dev.mysql.com/doc/refman/5.7/en/set-transaction.html
--09T08::27.003682+: [Note] InnoDB: PUNCH HOLE support available
--09T08::27.003696+: [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
--09T08::27.003701+: [Note] InnoDB: Uses event mutexes
--09T08::27.003705+: [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
--09T08::27.003710+: [Note] InnoDB: Compressed tables use zlib 1.2.
--09T08::27.003715+: [Note] InnoDB: Using Linux native AIO
--09T08::27.003734+: [Note] InnoDB: Adjusting innodb_buffer_pool_instances from to since innodb_buffer_pool_size is less than MiB
--09T08::27.004212+: [Note] InnoDB: Number of pools:
--09T08::27.004325+: [Note] InnoDB: Using CPU crc32 instructions
--09T08::27.010477+: [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = , chunk size = 100M
--09T08::27.019767+: [Note] InnoDB: Completed initialization of buffer pool
--09T08::27.023312+: [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
--09T08::27.066815+: [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite
--09T08::27.070460+: [Note] InnoDB: Highest supported file format is Barracuda.
--09T08::27.071136+: [Note] InnoDB: Log scan progressed past the checkpoint lsn
--09T08::27.071168+: [Note] InnoDB: Doing recovery: scanned up to log sequence number
--09T08::27.071185+: [Note] InnoDB: Database was not shutdown normally!
--09T08::27.071190+: [Note] InnoDB: Starting crash recovery.
--09T08::27.306683+: [Note] InnoDB: Created parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite, size bytes
--09T08::27.558968+: [Note] InnoDB: Last MySQL binlog file position , file name mysql-bin.
--09T08::27.575777+: [Note] InnoDB: Creating shared tablespace for temporary tables
--09T08::27.575872+: [Note] InnoDB: Setting file './ibtmp1' size to MB. Physically writing the file full; Please wait ...
--09T08::28.063310+: [Note] InnoDB: File './ibtmp1' size is now MB.
--09T08::28.064352+: [Note] InnoDB: redo rollback segment(s) found. redo rollback segment(s) are active.
--09T08::28.064370+: [Note] InnoDB: non-redo rollback segment(s) are active.
--09T08::28.065740+: [Note] InnoDB: Waiting for purge to start
--09T08::28.116905+: [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.22-rel22 started; log sequence number 2605219
--09T08::28.117131+: [Note] Plugin 'FEDERATED' is disabled.
--09T08::28.123701+: [Note] InnoDB: Loading buffer pool(s) from /data/mysql/mysql3308/data/ib_buffer_pool
--09T08::28.123749+: [ERROR] InnoDB: Cannot open '/data/mysql/mysql3308/data/ib_buffer_pool' for reading: No such file or directory
--09T08::28.126137+: [Note] Recovering after a crash using /data/mysql/mysql3308/logs/mysql-bin
--09T08::28.126167+: [Note] WSREP: Before binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)
--09T08::28.128885+: [Note] Starting crash recovery...
--09T08::28.128936+: [Note] Crash recovery finished.
--09T08::28.128955+: [Note] WSREP: After binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)
--09T08::28.135097+: [Note] Salting uuid generator variables, current_pid: , server_start_time: , bytes_sent: ,
--09T08::28.135163+: [Note] Generated uuid: '83fac5b7-9b99-11e8-9f3a-080027de0e0e', server_start_time: , bytes_sent:
--09T08::28.135180+: [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 83fac5b7-9b99-11e8-9f3a-080027de0e0e.
--09T08::28.649143+: [Note] Auto generated SSL certificates are placed in data directory.
--09T08::28.651737+: [Warning] CA certificate ca.pem is self signed.
--09T08::28.812659+: [Note] Auto generated RSA key files are placed in data directory.
--09T08::28.812836+: [Note] Server hostname (bind-address): '*'; port:
--09T08::28.812866+: [Note] IPv6 is available.
--09T08::28.816928+: [Note] - '::' resolves to '::';
--09T08::28.821691+: [Note] Server socket created on IP: '::'.
--09T08::28.846925+: [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
--09T08::28.846986+: [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
--09T08::28.847001+: [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
--09T08::28.847028+: [Warning] 'user' entry 'sst@localhost' ignored in --skip-name-resolve mode.
--09T08::28.847089+: [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
--09T08::28.847097+: [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
--09T08::28.847124+: [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
--09T08::28.858902+: [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
--09T08::28.858929+: [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
--09T08::28.873861+: [Note] Event Scheduler: Loaded events
--09T08::28.874109+: [Note] WSREP: Signalling provider to continue on SST completion.
--09T08::28.874133+: [Note] WSREP: Initialized wsrep sidno
--09T08::28.874156+: [Note] WSREP: SST received: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:
--09T08::28.874266+: [Note] mysqld: ready for connections.
Version: '5.7.22-22-29.26-log' socket: '/tmp/mysql3308.sock' port: Percona XtraDB Cluster binary (GPL) 5.7.-29.26, Revision da86071, wsrep_29.
--09T08::28.875367+: [Note] WSREP: 2.0 (zlm4): State transfer from 0.0 (zlm2) complete.
--09T08::28.875409+: [Note] WSREP: SST leaving flow control
--09T08::28.875418+: [Note] WSREP: Shifting JOINER -> JOINED (TO: )
--09T08::28.876587+: [Note] WSREP: Member 2.0 (zlm4) synced with group.
--09T08::28.876618+: [Note] WSREP: Shifting JOINED -> SYNCED (TO: )
--09T08::28.876639+: [Note] WSREP: Synchronized with group, ready for connections
--09T08::28.876645+: [Note] WSREP: Setting wsrep_ready to true
--09T08::28.876649+: [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. //It's okay now.No error is found this time.