I have a problem with my Java application on a Linux system (version 2.6.39.2) with 500MB RAM. Every hour my application takes 1-2MB resident memory more. The application starts with 40MB of resident memory used. After 15 days, OOM-killer kills my application (see attached log).
我在使用500MB RAM的Linux系统(2.6.39.2版本)上的Java应用程序有问题。我的应用程序每小时多占用1-2MB的驻留内存。应用程序从使用了40MB的驻留内存开始。15天后，OOM-killer杀死我的应用程序(见附录)。
More information:
更多信息:
- I haven’t set any memory limit (no Xms, Xmx, …) for my application.
- 我还没有为我的应用程序设置任何内存限制(没有Xms, Xmx，…)。
- I have made a heap dump when 396MB of resident memory where used but for Memory Analysis tool, I only use 7.8MB of memory and there were no special memory leaks.
- 当使用了396MB的常驻内存(但用于内存分析工具)时，我做了一个堆转储，我只使用了7.8MB的内存，并且没有特殊的内存泄漏。
What can I do to found the source of the problem or how to ensure that my application will not be killed?
我可以做什么来找到问题的根源，或者如何确保我的应用程序不会被终止?
Thanks in advance for your help. Best regards.
谢谢你的帮助。致以最亲切的问候。
EDIT: It is a standalone embedded application without server. The problem occurs on the deployed application on a machine (difficult to install analysis tools) but not on my development PC.
编辑:它是一个独立的嵌入式应用程序，没有服务器。问题出现在机器上部署的应用程序上(很难安装分析工具)，而不是在我的开发PC上。
Jun 7 06:15:41 HMI user.warn kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jun 7 06:15:42 HMI user.warn kernel: Pid: 650, comm: java Not tainted 2.6.39.2 #7
Jun 7 06:15:42 HMI user.warn kernel: Call Trace:
Jun 7 06:15:42 HMI user.warn kernel: [<c144c264>] ? printk+0xf/0x11
Jun 7 06:15:42 HMI user.warn kernel: [<c1074eb8>] dump_header.clone.8+0x58/0x150
Jun 7 06:15:42 HMI user.warn kernel: [<c11dd008>] ? ___ratelimit+0x78/0xf0
Jun 7 06:15:42 HMI user.warn kernel: [<c1075121>] oom_kill_process.clone.11+0x41/0x220
Jun 7 06:15:42 HMI user.warn kernel: [<c10376e8>] ? has_capability_noaudit+0x18/0x30
Jun 7 06:15:42 HMI user.warn kernel: [<c1074fd5>] ? oom_badness+0x25/0x130
Jun 7 06:15:42 HMI user.warn kernel: [<c10755ef>] out_of_memory+0x1bf/0x2a0
Jun 7 06:15:42 HMI user.warn kernel: [<c1078b4a>] __alloc_pages_nodemask+0x5aa/0x5c0
Jun 7 06:15:42 HMI user.warn kernel: [<c107a1cb>] __do_page_cache_readahead+0xdb/0x200
Jun 7 06:15:42 HMI user.warn kernel: [<c1072b00>] ? wait_on_page_bit+0x90/0xa0
Jun 7 06:15:42 HMI user.warn kernel: [<c107a539>] ra_submit+0x19/0x20
Jun 7 06:15:42 HMI user.warn kernel: [<c1074762>] filemap_fault+0x3d2/0x3e0
Jun 7 06:15:42 HMI user.warn kernel: [<c1086de9>] __do_fault+0x39/0x3a0
Jun 7 06:15:42 HMI user.warn kernel: [<c1074390>] ? __lock_page_or_retry+0x50/0x50
Jun 7 06:15:42 HMI user.warn kernel: [<c1089540>] handle_pte_fault+0x70/0x5b0
Jun 7 06:15:42 HMI user.warn kernel: [<c1089b0f>] handle_mm_fault+0x8f/0x100
Jun 7 06:15:42 HMI user.warn kernel: [<c101ff00>] do_page_fault+0x100/0x3d0
Jun 7 06:15:42 HMI user.warn kernel: [<c1053e0a>] ? sys_futex+0x5a/0x100
Jun 7 06:15:42 HMI user.warn kernel: [<c101fe00>] ? vmalloc_sync_all+0x110/0x110
Jun 7 06:15:42 HMI user.warn kernel: [<c144ed1a>] error_code+0x5a/0x60
Jun 7 06:15:42 HMI user.warn kernel: [<c1440000>] ? vsc_sata_init_one+0x1f6/0x278
Jun 7 06:15:42 HMI user.warn kernel: [<c101fe00>] ? vmalloc_sync_all+0x110/0x110
Jun 7 06:15:42 HMI user.warn kernel: Mem-Info:
Jun 7 06:15:42 HMI user.warn kernel: DMA per-cpu:
Jun 7 06:15:42 HMI user.warn kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jun 7 06:15:42 HMI user.warn kernel: Normal per-cpu:
Jun 7 06:15:42 HMI user.warn kernel: CPU 0: hi: 186, btch: 31 usd: 52
Jun 7 06:15:42 HMI user.warn kernel: active_anon:111882 inactive_anon:51 isolated_anon:0
Jun 7 06:15:42 HMI user.warn kernel: active_file:70 inactive_file:853 isolated_file:0
Jun 7 06:15:42 HMI user.warn kernel: unevictable:0 dirty:0 writeback:0 unstable:0
Jun 7 06:15:42 HMI user.warn kernel: free:1183 slab_reclaimable:307 slab_unreclaimable:1016
Jun 7 06:15:42 HMI user.warn kernel: mapped:149 shmem:122 pagetables:302 bounce:0
Jun 7 06:15:42 HMI user.warn kernel: DMA free:2028kB min:88kB low:108kB high:132kB active_anon:13696kB inactive_anon:0kB active_file:0kB inactive_file:72kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15800kB mlocked:0kB dirty:0kB writeback:0kB ma
Jun 7 06:15:42 HMI user.warn kernel: lowmem_reserve[]: 0 483 483 483
Jun 7 06:15:42 HMI user.warn kernel: Normal free:2704kB min:2768kB low:3460kB high:4152kB active_anon:433832kB inactive_anon:204kB active_file:280kB inactive_file:3340kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:495488kB mlocked:0kB dirty:0kB w
Jun 7 06:15:42 HMI user.warn kernel: lowmem_reserve[]: 0 0 0 0
Jun 7 06:15:42 HMI user.warn kernel: DMA: 393*4kB 7*8kB 3*16kB 1*32kB 1*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2028kB
Jun 7 06:15:42 HMI user.warn kernel: Normal: 676*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2704kB
Jun 7 06:15:42 HMI user.warn kernel: 1057 total pagecache pages
Jun 7 06:15:42 HMI user.warn kernel: 0 pages in swap cache
Jun 7 06:15:42 HMI user.warn kernel: Swap cache stats: add 0, delete 0, find 0/0
Jun 7 06:15:42 HMI user.warn kernel: Free swap = 0kB
Jun 7 06:15:42 HMI user.warn kernel: Total swap = 0kB
Jun 7 06:15:42 HMI user.warn kernel: 128928 pages RAM
Jun 7 06:15:42 HMI user.warn kernel: 0 pages HighMem
Jun 7 06:15:42 HMI user.warn kernel: 2847 pages reserved
Jun 7 06:15:42 HMI user.warn kernel: 476 pages shared
Jun 7 06:15:42 HMI user.warn kernel: 124258 pages non-shared
Jun 7 06:15:42 HMI user.info kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jun 7 06:15:42 HMI user.info kernel: [ 78] 0 78 535 77 0 -17 -1000 udevd
Jun 7 06:15:42 HMI user.info kernel: [ 314] 0 314 422 17 0 0 0 zeroconf
Jun 7 06:15:42 HMI user.info kernel: [ 320] 1 320 420 17 0 0 0 portmap
Jun 7 06:15:42 HMI user.info kernel: [ 349] 0 349 618 22 0 0 0 rc
Jun 7 06:15:42 HMI user.info kernel: [ 358] 42 358 585 38 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 362] 44 362 3139 184 0 0 0 hald
Jun 7 06:15:42 HMI user.info kernel: [ 363] 0 363 805 45 0 0 0 hald-runner
Jun 7 06:15:42 HMI user.info kernel: [ 374] 0 374 821 32 0 0 0 hald-addon-inpu
Jun 7 06:15:42 HMI user.info kernel: [ 386] 44 386 767 35 0 0 0 hald-addon-acpi
Jun 7 06:15:42 HMI user.info kernel: [ 401] 0 401 531 23 0 0 0 dropbear
Jun 7 06:15:42 HMI user.info kernel: [ 406] 0 406 425 22 0 0 0 acpid
Jun 7 06:15:42 HMI user.info kernel: [ 411] 0 411 617 16 0 0 0 syslogd
Jun 7 06:15:42 HMI user.info kernel: [ 413] 0 413 617 18 0 0 0 klogd
Jun 7 06:15:42 HMI user.info kernel: [ 422] 46 422 666 48 0 0 0 avahi-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 423] 46 423 666 42 0 0 0 avahi-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 431] 0 431 886 18 0 0 0 pure-ftpd
Jun 7 06:15:42 HMI user.info kernel: [ 442] 0 442 618 16 0 0 0 S97x11Dm
Jun 7 06:15:42 HMI user.info kernel: [ 443] 0 443 741 26 0 0 0 xinit
Jun 7 06:15:42 HMI user.info kernel: [ 445] 0 445 7850 4035 0 0 0 X
Jun 7 06:15:42 HMI user.info kernel: [ 452] 0 452 617 15 0 0 0 sh
Jun 7 06:15:42 HMI user.info kernel: [ 453] 0 453 2008 226 0 0 0 matchbox-window
Jun 7 06:15:42 HMI user.info kernel: [ 454] 0 454 2001 725 0 0 0 matchbox-deskto
Jun 7 06:15:42 HMI user.info kernel: [ 455] 0 455 1479 183 0 0 0 matchbox-panel
Jun 7 06:15:42 HMI user.info kernel: [ 472] 0 472 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 474] 0 474 154396 104395 0 0 0 java
Jun 7 06:15:42 HMI user.info kernel: [ 479] 0 479 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 484] 0 484 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 489] 0 489 585 36 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 496] 0 496 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 504] 0 504 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 509] 0 509 585 34 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 514] 0 514 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 519] 0 519 585 36 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 524] 0 524 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 529] 0 529 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 535] 0 535 585 34 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 540] 0 540 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 546] 0 546 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 552] 0 552 585 36 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 557] 0 557 585 36 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 562] 0 562 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 567] 0 567 585 34 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 572] 0 572 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 577] 0 577 585 36 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 582] 0 582 585 34 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 587] 0 587 585 34 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 592] 0 592 585 34 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 597] 0 597 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 602] 0 602 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 610] 0 610 585 34 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 616] 0 616 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 621] 0 621 585 34 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 626] 0 626 585 34 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 631] 0 631 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 636] 0 636 585 35 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 641] 0 641 585 34 0 0 0 dbus-daemon
Jun 7 06:15:42 HMI user.info kernel: [ 642] 0 642 1407 155 0 0 0 mb-applet-menu-
Jun 7 06:15:42 HMI user.info kernel: [ 649] 0 649 1397 139 0 0 0 mb-applet-clock
Jun 7 06:15:42 HMI user.info kernel: [ 837] 0 837 530 77 0 -17 -1000 udevd
Jun 7 06:15:42 HMI user.info kernel: [ 838] 0 838 532 78 0 -17 -1000 udevd
Jun 7 06:15:42 HMI user.err kernel: Out of memory: Kill process 474 (java) score 798 or sacrifice child
Jun 7 06:15:42 HMI user.err kernel: Killed process 474 (java) total-vm:617584kB, anon-rss:417580kB, file-rss:0kB
2 个解决方案
#1
2
So you're leaking memory and would like to know why and where.
所以你在泄露内存，想知道原因和地点。
A few things you can do
你可以做一些事情
- use garbage collection logging with all the verbose options
- 使用所有详细选项的垃圾收集日志记录
- use the jdk tools (jmap with the histo option shows you which classes have how many instances alive etc)
- 使用jdk工具(与histo选项一起的jmap向您显示哪些类具有多少实例等等)
- jkd tools #2 jvisualvm - on of the best free tools around
- jkd工具#2 jvisualvm -关于最好的免费工具
- there are free versions of commercial tools like appdynamics lite/ jxinsight opus
- 有免费版本的商业工具，如appdynamics lite/ jxinsight opus
- use -XX:+HeapDumpOnOutOfMemoryError and analyse the heap dump once you get the OOM
- 使用-XX:+HeapDumpOnOutOfMemoryError，并在获得OOM后分析堆转储
- a code review can't do harm either - most likely you're not letting go of information you don't need any more
- 代码评审也不会有什么害处——很可能您不会再放弃不需要的信息
#2
1
You probably have a memory leak. I suggest you to use a profiler tool (JProfiler is the best in my opinion) to find it. If your server doesn't have GUI you can set up the profiler remotely and use an agent on the Linux machine.
您可能有内存泄漏。我建议您使用profiler工具(我认为JProfiler是最好的)找到它。如果您的服务器没有GUI，您可以远程设置分析器并在Linux机器上使用代理。
#1
2
So you're leaking memory and would like to know why and where.
所以你在泄露内存，想知道原因和地点。
A few things you can do
你可以做一些事情
- use garbage collection logging with all the verbose options
- 使用所有详细选项的垃圾收集日志记录
- use the jdk tools (jmap with the histo option shows you which classes have how many instances alive etc)
- 使用jdk工具(与histo选项一起的jmap向您显示哪些类具有多少实例等等)
- jkd tools #2 jvisualvm - on of the best free tools around
- jkd工具#2 jvisualvm -关于最好的免费工具
- there are free versions of commercial tools like appdynamics lite/ jxinsight opus
- 有免费版本的商业工具，如appdynamics lite/ jxinsight opus
- use -XX:+HeapDumpOnOutOfMemoryError and analyse the heap dump once you get the OOM
- 使用-XX:+HeapDumpOnOutOfMemoryError，并在获得OOM后分析堆转储
- a code review can't do harm either - most likely you're not letting go of information you don't need any more
- 代码评审也不会有什么害处——很可能您不会再放弃不需要的信息
#2
1
You probably have a memory leak. I suggest you to use a profiler tool (JProfiler is the best in my opinion) to find it. If your server doesn't have GUI you can set up the profiler remotely and use an agent on the Linux machine.
您可能有内存泄漏。我建议您使用profiler工具(我认为JProfiler是最好的)找到它。如果您的服务器没有GUI，您可以远程设置分析器并在Linux机器上使用代理。