公司定位oom-killer问题

时间:2021-05-30 20:59:13

在/var/log/messages日志文件中找到的出错日志如下:

2018-05-22 01:06:21 localhost kernel: tx_0_n_1 invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0

2018-05-22 01:06:21 localhost kernel: tx_0_n_1 cpuset=/ mems_allowed=0-1
2018-05-22 01:06:21 localhost kernel: CPU: 31 PID: 32582 Comm: tx_0_n_1 Tainted: G           OE  ------------   3.10.0-327.el7.x86_64 #1
2018-05-22 01:06:21 localhost kernel: Hardware name: Dell Inc. PowerEdge R730xd/0WCJNT, BIOS 2.4.3 01/17/2017
2018-05-22 01:06:21 localhost kernel:  ffff883fcb883980 00000000ea82cf2a ffff883fc8e977c0 ffffffff816351f1
2018-05-22 01:06:21 localhost kernel:  ffff883fc8e97850 ffffffff81630191 ffff881fccf99590 ffff881fccf995a8
2018-05-22 01:06:21 localhost kernel:  0000000000000206 ffff883fcb883980 ffff883fc8e97838 ffffffff8112882f
2018-05-22 01:06:21 localhost kernel: Call Trace:
2018-05-22 01:06:21 localhost kernel:  [<ffffffff816351f1>] dump_stack+0x19/0x1b
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81630191>] dump_header+0x8e/0x214
2018-05-22 01:06:21 localhost kernel:  [<ffffffff8112882f>] ? delayacct_end+0x8f/0xb0
2018-05-22 01:06:21 localhost kernel:  [<ffffffff8116cdee>] oom_kill_process+0x24e/0x3b0
2018-05-22 01:06:21 localhost kernel:  [<ffffffff8116c956>] ? find_lock_task_mm+0x56/0xc0
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81088dae>] ? has_capability_noaudit+0x1e/0x30
2018-05-22 01:06:21 localhost kernel:  [<ffffffff8116d616>] out_of_memory+0x4b6/0x4f0
2018-05-22 01:06:21 localhost kernel:  [<ffffffff811737f5>] __alloc_pages_nodemask+0xa95/0xb90
2018-05-22 01:06:21 localhost kernel:  [<ffffffff811b43f9>] alloc_pages_current+0xa9/0x170
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81514ad0>] sk_page_frag_refill+0x70/0x160
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81576b73>] tcp_sendmsg+0x263/0xc20
2018-05-22 01:06:21 localhost kernel:  [<ffffffff815a0f44>] inet_sendmsg+0x64/0xb0
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81510d10>] sock_sendmsg+0xb0/0xf0
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81639f78>] ? io_schedule_timeout+0xe8/0x130
2018-05-22 01:06:21 localhost kernel:  [<ffffffff811688d0>] ? sleep_on_page+0x20/0x20
2018-05-22 01:06:21 localhost kernel:  [<ffffffff8163879e>] ? __wait_on_bit+0x7e/0x90
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81511149>] ___sys_sendmsg+0x3a9/0x3c0
2018-05-22 01:06:21 localhost kernel:  [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
2018-05-22 01:06:21 localhost kernel:  [<ffffffff8163a2b8>] ? __schedule+0x2d8/0x900
2018-05-22 01:06:21 localhost kernel:  [<ffffffff8122870e>] ? ep_poll+0x31e/0x360
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81639887>] ? do_nanosleep+0xa7/0xf0
2018-05-22 01:06:21 localhost kernel:  [<ffffffff810aa82d>] ? hrtimer_nanosleep+0xad/0x170
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81512031>] __sys_sendmsg+0x51/0x90
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81512082>] SyS_sendmsg+0x12/0x20
2018-05-22 01:06:21 localhost kernel:  [<ffffffff81645909>] system_call_fastpath+0x16/0x1b
2018-05-22 01:06:21 localhost kernel: Mem-Info:
2018-05-22 01:06:21 localhost kernel: Node 0 DMA per-cpu:
2018-05-22 01:06:21 localhost kernel: CPU    0: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU    1: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU    2: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU    3: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU    4: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU    5: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU    6: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU    7: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU    8: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU    9: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   10: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   11: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   12: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   13: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   14: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   15: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   16: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   17: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   18: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   19: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   20: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   21: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   22: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   23: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   24: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   25: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   26: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   27: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   28: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   29: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   30: hi:    0, btch:   1 usd:   0
2018-05-22 01:06:21 localhost kernel: CPU   31: hi:    0, btch:   1 usd:   0


2018-05-22 01:06:21 localhost kernel: active_anon:12112625 inactive_anon:883684 isolated_anon:0#012 active_file:0 inactive_file:0 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 free:50640 slab_reclaimable:8648 slab_unreclaimable:20722#012 mapped:923 shmem:3535 pagetables:28411 bounce:0#012 free_cma:0
2018-05-22 01:06:21 localhost kernel: Node 0 DMA free:14728kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
2018-05-22 01:06:21 localhost kernel: lowmem_reserve[]: 0 1555 26119 26119
2018-05-22 01:06:21 localhost kernel: Node 0 DMA32 free:100848kB min:2656kB low:3320kB high:3984kB active_anon:1102268kB inactive_anon:375408kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1854168kB managed:1594440kB mlocked:0kB dirty:0kB writeback:0kB mapped:388kB shmem:384kB slab_reclaimable:1204kB slab_unreclaimable:2012kB kernel_stack:256kB pagetables:3640kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
2018-05-22 01:06:21 localhost kernel: lowmem_reserve[]: 0 0 24563 24563
2018-05-22 01:06:21 localhost kernel: Node 0 Normal free:41540kB min:41964kB low:52452kB high:62944kB active_anon:22659164kB inactive_anon:1514036kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:132120576kB managed:25153392kB mlocked:0kB dirty:0kB writeback:0kB mapped:260kB shmem:4904kB slab_reclaimable:20044kB slab_unreclaimable:41696kB kernel_stack:12784kB pagetables:53696kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:460 all_unreclaimable? yes
2018-05-22 01:06:21 localhost kernel: lowmem_reserve[]: 0 0 0 0
2018-05-22 01:06:21 localhost kernel: Node 1 Normal free:45444kB min:45460kB low:56824kB high:68188kB active_anon:24689068kB inactive_anon:1645292kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:134217728kB managed:27250304kB mlocked:0kB dirty:0kB writeback:0kB mapped:3044kB shmem:8852kB slab_reclaimable:13344kB slab_unreclaimable:39180kB kernel_stack:2272kB pagetables:56308kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:82 all_unreclaimable? no
2018-05-22 01:06:21 localhost kernel: lowmem_reserve[]: 0 0 0 0
2018-05-22 01:06:21 localhost kernel: Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 0*128kB 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (R) 3*4096kB (M) = 14728kB
2018-05-22 01:06:21 localhost kernel: Node 0 DMA32: 104*4kB (UE) 68*8kB (UEM) 29*16kB (UEM) 1171*32kB (UEM) 518*64kB (UEM) 99*128kB (EM) 23*256kB (UEM) 8*512kB (UEM) 6*1024kB (UEM) 0*2048kB 0*4096kB = 100848kB
2018-05-22 01:06:21 localhost kernel: Node 0 Normal: 276*4kB (UEM) 88*8kB (UEM) 55*16kB (UE) 72*32kB (UEM) 87*64kB (UEM) 71*128kB (UEM) 30*256kB (UEM) 8*512kB (UM) 10*1024kB (UM) 0*2048kB 0*4096kB = 41664kB
2018-05-22 01:06:21 localhost kernel: Node 1 Normal: 83*4kB (UEM) 88*8kB (UEM) 77*16kB (UEM) 71*32kB (UEM) 52*64kB (UEM) 35*128kB (UE) 27*256kB (UE) 13*512kB (UE) 17*1024kB (UM) 0*2048kB 0*4096kB = 43324kB
2018-05-22 01:06:21 localhost kernel: Node 0 hugepages_total=100 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
2018-05-22 01:06:21 localhost kernel: Node 1 hugepages_total=100 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
2018-05-22 01:06:21 localhost kernel: 39455 total pagecache pages
2018-05-22 01:06:21 localhost kernel: 35686 pages in swap cache
2018-05-22 01:06:21 localhost kernel: Swap cache stats: add 203844566, delete 203808880, find 44917268/61656884
2018-05-22 01:06:21 localhost kernel: Free swap  = 0kB
2018-05-22 01:06:21 localhost kernel: Total swap = 4194300kB
2018-05-22 01:06:21 localhost kernel: 67052113 pages RAM
2018-05-22 01:06:21 localhost kernel: 0 pages HighMem/MovableOnly
2018-05-22 01:06:21 localhost kernel: 53548605 pages reserved
2018-05-22 01:06:21 localhost kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
2018-05-22 01:06:21 localhost kernel: [ 1103]     0  1103    28624      377      60       50             0 systemd-journal
2018-05-22 01:06:21 localhost kernel: [ 1124]     0  1124    31668        0      29       96             0 lvmetad
2018-05-22 01:06:21 localhost kernel: [ 1131]     0  1131    10891        2      21      247         -1000 systemd-udevd
2018-05-22 01:06:21 localhost kernel: [ 1306]     0  1306    29185       43      26       78         -1000 auditd
2018-05-22 01:06:22 localhost kernel: [ 1328]     0  1328     6598       40      16       44             0 systemd-logind
2018-05-22 01:06:22 localhost kernel: [ 1330]    81  1330     6698       78      18       72          -900 dbus-daemon
2018-05-22 01:06:22 localhost kernel: [ 1347]     0  1347    31589       24      19      137             0 crond
2018-05-22 01:06:22 localhost kernel: [ 1354]     0  1354    23199        1      49      162             0 login
2018-05-22 01:06:22 localhost kernel: [ 1458]   997  1458   130880      103      52     2246             0 polkitd
2018-05-22 01:06:22 localhost kernel: [ 1461]     0  1461    13264        0      30      148             0 wpa_supplicant
2018-05-22 01:06:22 localhost kernel: [ 2006]     0  2006   138260      129      89     2535             0 tuned
2018-05-22 01:06:22 localhost kernel: [ 2007]     0  2007    20636        0      43      220         -1000 sshd
2018-05-22 01:06:22 localhost kernel: [ 3099]     0  3099    22781       19      43      238             0 master
2018-05-22 01:06:22 localhost kernel: [ 3151]    89  3151    22824       15      43      238             0 qmgr
2018-05-22 01:06:22 localhost kernel: [101525]    38 101525     7349       35      18      117             0 ntpd
2018-05-22 01:06:22 localhost kernel: [101526]     0 101526     7349       27      17      121             0 ntpd
2018-05-22 01:06:22 localhost kernel: [183605]     0 183605    28870        2      14      119             0 bash
2018-05-22 01:06:22 localhost kernel: [113888]     0 113888   182469       75     162      163             0 rsyslogd
2018-05-22 01:06:22 localhost kernel: [71337]   996 71337    21115        1      40      206             0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [71338]   996 71338    21116      425      41      188             0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [71339]   996 71339    21157      365      42      216             0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [71340]   996 71340    21157      310      42      214             0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [71341]   996 71341    21157      345      42      210             0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [71342]   996 71342    21149       40      40      203             0 zabbix_agentd
2018-05-22 01:06:22 localhost kernel: [ 8474]     0  8474     8037       28      20      131             0 ByNodemanagerWa
2018-05-22 01:06:22 localhost kernel: [32279]     0 32279 52532746      606      67     2927             0 ByNodemanagerSe
2018-05-22 01:06:22 localhost kernel: [32463]     0 32463 66557759 12936045   27027   832056             0 ByNodemanagerCl
2018-05-22 01:06:22 localhost kernel: [32565]     0 32565 52648096    19074     117     1063             0 ByNodemanagerXd
2018-05-22 01:06:22 localhost kernel: [63588]    89 63588    22807      251      44        0             0 pickup
2018-05-22 01:06:22 localhost kernel: [71804]   996 71804    28809       52      15        0             0 ntp_check.sh
2018-05-22 01:06:22 localhost kernel: [71805]   996 71805    28809       53      14        0             0 ntp_check.sh
2018-05-22 01:06:22 localhost kernel: [71806]   996 71806    28809       52      12        0             0 ntp_check.sh
2018-05-22 01:06:22 localhost kernel: [71807]   996 71807    28809       53      12        0             0 ntp_check.sh
2018-05-22 01:06:22 localhost kernel: [71809]   996 71809     6409      105      17        0             0 ntpq
2018-05-22 01:06:22 localhost kernel: [71810]   996 71810    28370       39      14        0             0 awk
2018-05-22 01:06:22 localhost kernel: [71811]   996 71811     6409      103      19        0             0 ntpq
2018-05-22 01:06:22 localhost kernel: [71812]   996 71812    28370       39      13        0             0 awk
2018-05-22 01:06:22 localhost kernel: [71813]   996 71813     2348       26       8        0             0 sh
2018-05-22 01:06:22 localhost kernel: Out of memory: Kill process 32463 (ByNodemanagerCl) score 199 or sacrifice child
2018-05-22 01:06:22 localhost kernel: Killed process 32463 (ByNodemanagerCl) total-vm:266231036kB, anon-rss:51743276kB, file-rss:888kB