一个MySQL数据库服务不能正常启动的故障处理过程

时间:2022-10-02 22:48:38


今天上午进行了一些my.cnf配置参数的修改,修改后,重启数据库,结果报错:
# service mysqld start
Starting MySQL...The server quit without updating PID file (/data/mysql/v02c09.idc.huobi.com.pid).[FAILED]


查看错误日志文件:
# tail -50 errors.log 

140823 12:03:16 mysqld_safe mysqld from pid file /data/mysql/v02c09.idc.huobi.com.pid ended
140823 12:06:16 mysqld_safe Starting mysqld daemon with databases from /data/mysql
140823 12:06:16 [Note] Plugin 'FEDERATED' is disabled.
140823 12:06:16 InnoDB: The InnoDB memory heap is disabled
140823 12:06:16 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140823 12:06:16 InnoDB: Compressed tables use zlib 1.2.3
140823 12:06:16 InnoDB: Using Linux native AIO
140823 12:06:16 InnoDB: Initializing buffer pool, size = 12.0G
140823 12:06:18 InnoDB: Completed initialization of buffer pool
140823 12:06:18 InnoDB: Error: combined size of log files must be < 4 GB
140823 12:06:18 [ERROR] Plugin 'InnoDB' init function returned error.
140823 12:06:18 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140823 12:06:18 [ERROR] Unknown/unsupported storage engine: InnoDB
140823 12:06:18 [ERROR] Aborting

140823 12:06:18 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete

140823 12:06:18 mysqld_safe mysqld from pid file /data/mysql/v02c09.idc.huobi.com.pid ended


通过查询错误日志中“[ERROR] Plugin 'InnoDB' init function returned error.”一部分,在网络上找到一个解决方案:
只要删除MySQL目录下的ib_logfile0和ib_logfile1两个文件,就可以解决问题了。


于是按照这个步骤删除文件:
# pwd
/data/mysql

# ls -l ib*
-rw-rw---- 1 mysql mysql  136314880 Aug 23 12:03 ibdata1
-rw-rw---- 1 mysql mysql 2097152000 Aug 23 12:03 ib_logfile0
-rw-rw---- 1 mysql mysql  134217728 Aug 22 12:11 ib_logfile0.bak
-rw-rw---- 1 mysql mysql 2097152000 Aug 22 12:12 ib_logfile1
-rw-rw---- 1 mysql mysql  134217728 Aug 22 12:01 ib_logfile1.bak

# rm ib_logfile0 ib_logfile1
rm: remove regular file `ib_logfile0'? yes
rm: remove regular file `ib_logfile1'? yes

# ls -l ib*
-rw-rw---- 1 mysql mysql 136314880 Aug 23 12:03 ibdata1
-rw-rw---- 1 mysql mysql 134217728 Aug 22 12:11 ib_logfile0.bak
-rw-rw---- 1 mysql mysql 134217728 Aug 22 12:01 ib_logfile1.bak


日志文件正常移除后,再次重启数据库:
# service mysqld start
Starting MySQL...The server quit without updating PID file (/data/mysql/v02c09.idc.huobi.com.pid).[FAILED]

仍然有报错,查看error.log日志文件,内容如下:

140823 12:06:18 mysqld_safe mysqld from pid file /data/mysql/v02c09.idc.huobi.com.pid ended
140823 12:19:19 mysqld_safe Starting mysqld daemon with databases from /data/mysql
140823 12:19:19 [Note] Plugin 'FEDERATED' is disabled.
140823 12:19:19 InnoDB: The InnoDB memory heap is disabled
140823 12:19:19 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140823 12:19:19 InnoDB: Compressed tables use zlib 1.2.3
140823 12:19:19 InnoDB: Using Linux native AIO
140823 12:19:19 InnoDB: Initializing buffer pool, size = 12.0G
140823 12:19:21 InnoDB: Completed initialization of buffer pool
140823 12:19:21 InnoDB: Error: combined size of log files must be < 4 GB
140823 12:19:21 [ERROR] Plugin 'InnoDB' init function returned error.
140823 12:19:21 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140823 12:19:21 [ERROR] Unknown/unsupported storage engine: InnoDB
140823 12:19:21 [ERROR] Aborting

140823 12:19:21 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete

140823 12:19:21 mysqld_safe mysqld from pid file /data/mysql/v02c09.idc.huobi.com.pid ended


查看第一条报错情况"InnoDB: Error: combined size of log files must be < 4 GB",说的是MySQL的InnoDB日志文件总大小必须小于4GB,一般设置为1300G,组数为3,这样总大小为3.9G;但该配置文件中昨天刚设置为2000M,没有设定组数;今天进行了这样的设置:
innodb_file_per_table
innodb_data_home_dir = /data/mysql
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /data/mysql
innodb_buffer_pool_size = 12G
innodb_log_file_size = 2000M
innodb_log_buffer_size = 8M
innodb_log_files_in_group = 3
innodb_file_format = Barracuda


innodb_log的大小为2000M,组数为3,总大小超过4GB,所以会报错。
将 innodb_log_file_size = 2000M 参数修改为:
innodb_log_file_size = 1300M


重新启动MySQL数据库服务:
[root@v02c09 mysql]# service mysqld start
Starting MySQL.....................................[  OK  ]

至此,各方面确认正常。




通过这个故障处理,可以得到如下结论:
1.对于MySQL数据库的innodb_log参数的相关修改,需要重启MySQL服务,并将原来的ib_logfile*转移成其他的名称,才能在新服务启动后生效;
2.对于出现类似“The server quit without updating PID file (/data/mysql/v02c09.idc.huobi.com.pid).”都是体现的msyql服务没有启动成功,一般MySQL不能启动成功的原因都是my.cnf中的配置存在问题,才导致了服务不能正常启动,此时需要立即查找error.log来确认MySQL启动过程和报错信息;
3.在error.log中可能会有多个报错信息,一般后面的错误就是结果,查找错误原因,要看前面的,尤其要关注第一个出现的错误,如本次故障处理过程中“InnoDB: Error: combined size of log files must be < 4 GB”,虽然没有[ERROR]标记,但是第一个出现故障的问题,也明确了错误的原因;
4.明确了错误的原因后,有时需要修改my.cnf文件,或者需要重启MySQL数据库服务,这时都要先确认好状态,然后再进行操作;
5.对于MySQL数据库的常用参数,有些是有限制条件和参数范围的,需要确认好后再设定,否则设定也会无效或报错。