在/var/log/messages日志文件中找到的出错日志如下:
2018-05-22 01:06:21 localhost kernel: tx_0_n_1 invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0
2018-05-22 01:06:21 localhost kernel: tx_0_n_1 cpuset=/ mems_allowed=0-12018-05-22 01:06:21 localhost kernel: CPU: 31 PID: 32582 Comm: tx_0_n_1 Tainted: G OE ------------ 3.10.0-327.el7.x86_64 #1
2018-05-22 01:06:21 localhost kernel: Hardware name: Dell Inc. PowerEdge R730xd/0WCJNT, BIOS 2.4.3 01/17/2017
2018-05-22 01:06:21 localhost kernel: ffff883fcb883980 00000000ea82cf2a ffff883fc8e977c0 ffffffff816351f1
2018-05-22 01:06:21 localhost kernel: ffff883fc8e97850 ffffffff81630191 ffff881fccf99590 ffff881fccf995a8
2018-05-22 01:06:21 localhost kernel: 0000000000000206 ffff883fcb883980 ffff883fc8e97838 ffffffff8112882f
2018-05-22 01:06:21 localhost kernel: Call Trace:
2018-05-22 01:06:21 localhost kernel: [<ffffffff816351f1>] dump_stack+0x19/0x1b
2018-05-22 01:06:21 localhost kernel: [<ffffffff81630191>] dump_header+0x8e/0x214
2018-05-22 01:06:21 localhost kernel: [<ffffffff8112882f>] ? delayacct_end+0x8f/0xb0
2018-05-22 01:06:21 localhost kernel: [<ffffffff8116cdee>] oom_kill_process+0x24e/0x3b0
2018-05-22 01:06:21 localhost kernel: [<ffffffff8116c956>] ? find_lock_task_mm+0x56/0xc0
2018-05-22 01:06:21 localhost kernel: [<ffffffff81088dae>] ? has_capability_noaudit+0x1e/0x30
2018-05-22 01:06:21 localhost kernel: [<ffffffff8116d616>] out_of_memory+0x4b6/0x4f0
2018-05-22 01:06:21 localhost kernel: [<ffffffff811737f5>] __alloc_pages_nodemask+0xa95/0xb90
2018-05-22 01:06:21 localhost kernel: [<ffffffff811b43f9>] alloc_pages_current+0xa9/0x170
2018-05-22 01:06:21 localhost kernel: [<ffffffff81514ad0>] sk_page_frag_refill+0x70/0x160
2018-05-22 01:06:21 localhost kernel: [<ffffffff81576b73>] tcp_sendmsg+0x263/0xc20
2018-05-22 01:06:21 localhost kernel: [<ffffffff815a0f44>] inet_sendmsg+0x64/0xb0
2018-05-22 01:06:21 localhost kernel: [<ffffffff81510d10>] sock_sendmsg+0xb0/0xf0
2018-05-22 01:06:21 localhost kernel: [<ffffffff81639f78>] ? io_schedule_timeout+0xe8/0x130
2018-05-22 01:06:21 localhost kernel: [<ffffffff811688d0>] ? sleep_on_page+0x20/0x20
2018-05-22 01:06:21 localhost kernel: [<ffffffff8163879e>] ? __wait_on_bit+0x7e/0x90
2018-05-22 01:06:21 localhost kernel: [<ffffffff81511149>] ___sys_sendmsg+0x3a9/0x3c0
2018-05-22 01:06:21 localhost kernel: [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
2018-05-22 01:06:21 localhost kernel: [<ffffffff8163a2b8>] ? __schedule+0x2d8/0x900
2018-05-22 01:06:21 localhost kernel: [<ffffffff8122870e>] ? ep_poll+0x31e/0x360
2018-05-22 01:06:21 localhost kernel: [<ffffffff81639887>] ? do_nanosleep+0xa7/0xf0
2018-05-22 01:06:21 localhost kernel: [<ffffffff810aa82d>] ? hrtimer_nanosleep+0xad/0x170
2018-05-22 01:06:21 localhost kernel: [<ffffffff81512031>] __sys_sendmsg+0x51/0x90
2018-05-22 01:06:21 localhost kernel: [<ffffffff81512082>] SyS_sendmsg+0x12/0x20
2018-05-22 01:06:21 localhost kernel: [<ffffffff81645909>] system_call_fastpath+0x16/0x1b
2018-05-22 01:06:21 localhost kernel: Mem-Info:
2018-05-22 01:06:21 localhost kernel: Node 0 DMA per-cpu:
2018-05-22 01:06:21 localhost kernel: CPU 0: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 1: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 2: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 3: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 4: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 5: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 6: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 7: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 8: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 9: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 10: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 11: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 12: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 13: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 14: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 15: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 16: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 17: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 18: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 19: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 20: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 21: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 22: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 23: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 24: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 25: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 26: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 27: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 28: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 29: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 30: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: CPU 31: hi: 0, btch: 1 usd: 0
2018-05-22 01:06:21 localhost kernel: active_anon:12112625 inactive_anon:883684 isolated_anon:0#012 active_file:0 inactive_file:0 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 free:50640 slab_reclaimable:8648 slab_unreclaimable:20722#012 mapped:923 shmem:3535 pagetables:28411 bounce:0#012 free_cma:0
2018-05-22 01:06:21 localhost kernel: Node 0 DMA free:14728kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
2018-05-22 01:06:21 localhost kernel: lowmem_reserve[]: 0 1555 26119 26119
2018-05-22 01:06:21 localhost kernel: Node 0 DMA32 free:100848kB min:2656kB low:3320kB high:3984kB active_anon:1102268kB inactive_anon:375408kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1854168kB managed:1594440kB mlocked:0kB dirty:0kB writeback:0kB mapped:388kB shmem:384kB slab_reclaimable:1204kB slab_unreclaimable:2012kB kernel_stack:256kB pagetables:3640kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
2018-05-22 01:06:21 localhost kernel: lowmem_reserve[]: 0 0 24563 24563
2018-05-22 01:06:21 localhost kernel: Node 0 Normal free:41540kB min:41964kB low:52452kB high:62944kB active_anon:22659164kB inactive_anon:1514036kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:132120576kB managed:25153392kB mlocked:0kB dirty:0kB writeback:0kB mapped:260kB shmem:4904kB slab_reclaimable:20044kB slab_unreclaimable:41696kB kernel_stack:12784kB pagetables:53696kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:460 all_unreclaimable? yes
2018-05-22 01:06:21 localhost kernel: lowmem_reserve[]: 0 0 0 0
2018-05-22 01:06:21 localhost kernel: Node 1 Normal free:45444kB min:45460kB low:56824kB high:68188kB active_anon:24689068kB inactive_anon:1645292kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:134217728kB managed:27250304kB mlocked:0kB dirty:0kB writeback:0kB mapped:3044kB shmem:8852kB slab_reclaimable:13344kB slab_unreclaimable:39180kB kernel_stack:2272kB pagetables:56308kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:82 all_unreclaimable? no
2018-05-22 01:06:21 localhost kernel: lowmem_reserve[]: 0 0 0 0
2018-05-22 01:06:21 localhost kernel: Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 0*128kB 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (R) 3*4096kB (M) = 14728kB
2018-05-22 01:06:21 localhost kernel: Node 0 DMA32: 104*4kB (UE) 68*8kB (UEM) 29*16kB (UEM) 1171*32kB (UEM) 518*64kB (UEM) 99*128kB (EM) 23*256kB (UEM) 8*512kB (UEM) 6*1024kB (UEM) 0*2048kB 0*4096kB = 100848kB
2018-05-22 01:06:21 localhost kernel: Node 0 Normal: 276*4kB (UEM) 88*8kB (UEM) 55*16kB (UE) 72*32kB (UEM) 87*64kB (UEM) 71*128kB (UEM) 30*256kB (UEM) 8*512kB (UM) 10*1024kB (UM) 0*2048kB 0*4096kB = 41664kB
2018-05-22 01:06:21 localhost kernel: Node 1 Normal: 83*4kB (UEM) 88*8kB (UEM) 77*16kB (UEM) 71*32kB (UEM) 52*64kB (UEM) 35*128kB (UE) 27*256kB (UE) 13*512kB (UE) 17*1024kB (UM) 0*2048kB 0*4096kB = 43324kB
2018-05-22 01:06:21 localhost kernel: Node 0 hugepages_total=100 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
2018-05-22 01:06:21 localhost kernel: Node 1 hugepages_total=100 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
2018-05-22 01:06:21 localhost kernel: 39455 total pagecache pages
2018-05-22 01:06:21 localhost kernel: 35686 pages in swap cache
2018-05-22 01:06:21 localhost kernel: Swap cache stats: add 203844566, delete 203808880, find 44917268/61656884
2018-05-22 01:06:21 localhost kernel: Free swap = 0kB
2018-05-22 01:06:21 localhost kernel: Total swap = 4194300kB
2018-05-22 01:06:21 localhost kernel: 67052113 pages RAM
2018-05-22 01:06:21 localhost kernel: 0 pages HighMem/MovableOnly
2018-05-22 01:06:21 localhost kernel: 53548605 pages reserved
2018-05-22 01:06:21 localhost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
2018-05-22 01:06:21 localhost kernel: [ 1103] 0 1103 28624 377 60 50 0 systemd-journal
2018-05-22 01:06:21 localhost kernel: [ 1124] 0 1124 31668 0 29 96 0 lvmetad
2018-05-22 01:06:21 localhost kernel: [ 1131] 0 1131 10891 2 21 247 -1000 systemd-udevd
2018-05-22 01:06:21 localhost kernel: [ 1306] 0 1306 29185 43 26 78 -1000 auditd
2018-05-22 01:06:22 localhost kernel: [ 1328] 0 1328 6598 40 16 44 0 systemd-logind
2018-05-22 01:06:22 localhost kernel: [ 1330] 81 1330 6698 78 18 72 -900 dbus-daemon
2018-05-22 01:06:22 localhost kernel: [ 1347] 0 1347 31589 24 19 137 0 crond
2018-05-22 01:06:22 localhost kernel: [ 1354] 0 1354 23199 1 49 162 0 login
2018-05-22 01:06:22 localhost kernel: [ 1458] 997 1458 130880 103 52 2246 0 polkitd
2018-05-22 01:06:22 localhost kernel: [ 1461] 0 1461 13264 0 30 148 0 wpa_supplicant
2018-05-22 01:06:22 localhost kernel: [ 2006] 0 2006 138260 129 89 2535 0 tuned
2018-05-22 01:06:22 localhost kernel: [ 2007] 0 2007 20636 0 43 220 -1000 sshd
2018-05-22 01:06:22 localhost kernel: [ 3099] 0 3099 22781 19 43 238 0 master
2018-05-22 01:06:22 localhost kernel: [ 3151] 89 3151 22824 15 43 238 0 qmgr
2018-05-22 01:06:22 localhost kernel: [101525] 38 101525 7349 35 18 117 0 ntpd
2018-05-22 01:06:22 localhost kernel: [101526] 0 101526 7349 27 17 121 0 ntpd
2018-05-22 01:06:22 localhost kernel: [183605] 0 183605 28870 2 14 119 0 bash
2018-05-22 01:06:22 localhost kernel: [113888] 0 113888 182469 75 162 163 0 rsyslogd
2018-05-22 01:06:22 localhost kernel: [71337] 996 71337 21115 1 40 206 0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [71338] 996 71338 21116 425 41 188 0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [71339] 996 71339 21157 365 42 216 0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [71340] 996 71340 21157 310 42 214 0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [71341] 996 71341 21157 345 42 210 0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [71342] 996 71342 21149 40 40 203 0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [ 8474] 0 8474 8037 28 20 131 0 ByNodemanagerWa
2018-05-22 01:06:22 localhost kernel: [32279] 0 32279 52532746 606 67 2927 0 ByNodemanagerSe
2018-05-22 01:06:22 localhost kernel: [32463] 0 32463 66557759 12936045 27027 832056 0 ByNodemanagerCl
2018-05-22 01:06:22 localhost kernel: [32565] 0 32565 52648096 19074 117 1063 0 ByNodemanagerXd
2018-05-22 01:06:22 localhost kernel: [63588] 89 63588 22807 251 44 0 0 pickup
2018-05-22 01:06:22 localhost kernel: [71804] 996 71804 28809 52 15 0 0 ntp_check.sh
2018-05-22 01:06:22 localhost kernel: [71805] 996 71805 28809 53 14 0 0 ntp_check.sh
2018-05-22 01:06:22 localhost kernel: [71806] 996 71806 28809 52 12 0 0 ntp_check.sh
2018-05-22 01:06:22 localhost kernel: [71807] 996 71807 28809 53 12 0 0 ntp_check.sh
2018-05-22 01:06:22 localhost kernel: [71809] 996 71809 6409 105 17 0 0 ntpq
2018-05-22 01:06:22 localhost kernel: [71810] 996 71810 28370 39 14 0 0 awk
2018-05-22 01:06:22 localhost kernel: [71811] 996 71811 6409 103 19 0 0 ntpq
2018-05-22 01:06:22 localhost kernel: [71812] 996 71812 28370 39 13 0 0 awk
2018-05-22 01:06:22 localhost kernel: [71813] 996 71813 2348 26 8 0 0 sh
2018-05-22 01:06:22 localhost kernel: Out of memory: Kill process 32463 (ByNodemanagerCl) score 199 or sacrifice child
2018-05-22 01:06:22 localhost kernel: Killed process 32463 (ByNodemanagerCl) total-vm:266231036kB, anon-rss:51743276kB, file-rss:888kB