redis报Cannot allocate memory错误

时间：2021-01-05 03:18:00

昨天16:27开始将dp的日志使用ELK处理，当时redis使用内存的量不是很大，100M多点，结果今天早上到了一看xshell被关掉了，赶紧将各服务启动起来，elasticsearch启动没有问题，logstash indexer端启动时报错，redis插件连接不到，建议查看下redis的日志：

 A plugin had an unrecoverable error. Will restart this plugin.

   Plugin: <LogStash::Inputs::Redis host=>"10.132.24.166", type=>"redis-input", data_type=>"list", key=>"logstash", name=>"default">

   Error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. {:level=>:error}

以为是redis挂掉了，查看端口，6379还在的，查看日志：

redis 频繁报错日志：

 [7409] 12 Mar 09:57:11.069 * 1 changes in 900 seconds. Saving...

 [7409] 12 Mar 09:57:11.070 # Can't save in background: fork: Cannot allocate memory

 [7409] 12 Mar 09:57:17.086 * 1 changes in 900 seconds. Saving...

 [7409] 12 Mar 09:57:17.086 # Can't save in background: fork: Cannot allocate memory

 [7409] 12 Mar 09:57:23.100 * 1 changes in 900 seconds. Saving...

top后显示VIRT：3758M，RES：3.5G

参考：http://segmentfault.net/blog/koy/1190000000767465

内核参数调整参考：https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

https://www.kernel.org/doc/Documentation/sysctl/vm.txt

http://blog.sina.com.cn/s/blog_7429b9c801012evk.html

相关文章