" . lang。OutOfMemoryError:无法创建新的本机线程”

We are getting "java.lang.OutOfMemoryError : unable to create new native Thread" on 8GB RAM VM after 32k threads (ps -eLF| grep -c java)

我们" . lang。OutOfMemoryError: 32k线程(ps -eLF| grep -c java)后，无法在8GB RAM VM上创建新的本机线程

However, "top" and "free -m" shows 50% free memory available. JDk is 64 bit and tried with both HotSpot and JRockit.Server has Linux 2.6.18

然而，“top”和“free -m”显示50%的可用内存。JDk 64位，并尝试使用HotSpot和JRockit。服务器Linux 2.6.18

We also tried OS stack size (ulimit -s) tweaking and max process(ulimit -u) limits, limit.conf increase but all in vain.

我们还尝试了OS stack size (ulimit -s)调整和max process(ulimit -u)限制，限制。一切都是徒劳的。

Also we tried almost all possible of heap size combinations, keeping it low, high etc.

我们还尝试了几乎所有可能的堆大小组合，保持低、高等。

The script we use to run application is

我们用来运行应用程序的脚本是。

/opt/jrockit-jdk1.6/bin/java -Xms512m -Xmx512m -Xss128k -jar JavaNatSimulator.jar /opt/tools/jnatclients/natSimulator.properties

Thanks for the reply.

谢谢你的回复。

We have tried editing /etc/security/limits.conf and ulimit but still that same

我们尝试过编辑/etc/security/limitsconf和ulimit不变

[root@jboss02 ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 72192
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 72192
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

11 个解决方案

#1

This is not a memory problem even though the exception name highly suggests so, but an operating system resource problem. You are running out of native threads, i.e. how many threads the operating system will allow your JVM to use.

这不是一个内存问题，尽管异常名非常明显，但这是一个操作系统资源问题。您正在耗尽本机线程，即操作系统将允许JVM使用多少线程。

This is an uncommon problem, because you rarely need that many. Do you have a lot of unconditional thread spawning where the threads should but doesn't finish?

这是一个不常见的问题，因为你很少需要那么多。您是否有很多无条件的线程生成，在这些线程应该但是没有完成的地方?

You might consider rewriting into using Callable/Runnables under the control of an Executor if at all possible. There are plenty of standard executors with various behavior which your code can easily control.

如果可能的话，您可以考虑将重写为在执行程序的控制下使用可调用/Runnables。有许多具有各种行为的标准执行者，您的代码可以轻松地控制它们。

(There are many reasons why the number of threads is limited, but they vary from operating system to operating system)

(线程数量有限的原因有很多，但它们因操作系统和操作系统而异)

#2

It's likely that your OS does not allow the number of threads you're trying to create, or you're hitting some limit in the JVM. Especially if it's such a round number as 32k, a limit of one kind or another is a very likely culprit.

很可能您的操作系统不允许创建线程的数量，或者您在JVM中遇到了一些限制。尤其是当它是一个整数，比如32k时，一种或另一种极限很可能就是罪魁祸首。

Are you sure you truly need 32k threads? Most modern languages have some kind of support for pools of reusable threads - I'm sure Java has something in place too (like ExecutorService, as user Jesper mentioned). Perhaps you could request threads from such a pool, instead of manually creating new ones.

你确定你真的需要32k的线程吗?大多数现代语言都对可重用线程池提供了某种支持——我确信Java也提供了一些支持(如用户Jesper提到的ExecutorService)。也许您可以从这样的池中请求线程，而不是手动创建新线程。

#3

I encountered same issue during the load test, the reason is because of JVM is unable to create a new Java thread further. Below is the JVM source code

我在负载测试中遇到了同样的问题，原因是JVM无法进一步创建新的Java线程。下面是JVM源代码

if (native_thread->osthread() == NULL) {    
// No one should hold a reference to the 'native_thread'.    
    delete native_thread;   
if (JvmtiExport::should_post_resource_exhausted()) {      
    JvmtiExport::post_resource_exhausted(        
        JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | 
        JVMTI_RESOURCE_EXHAUSTED_THREADS, 
        "unable to create new native thread");    
    } THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(), "unable to create new native thread");  
} Thread::start(native_thread);`

Root cause : JVM throws this exception when JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR (resources exhausted (means memory exhausted) ) or JVMTI_RESOURCE_EXHAUSTED_THREADS (Threads exhausted).

根原因:当jvmti_resource_empsted_oom_error(资源耗尽(意味着内存耗尽))或jvmti_resource_empsted_threads(线程耗尽)时，JVM会抛出这个异常。

In my case Jboss is creating too many threads , to serve the request, but all the threads are blocked . Because of this, JVM is exhausted with threads as well with memory (each thread holds memory , which is not released , because each thread is blocked).

在我的例子中，Jboss创建了太多的线程来处理请求，但是所有的线程都被阻塞了。因此，使用线程和内存也会耗尽JVM(每个线程都保存内存，但没有释放内存，因为每个线程都被阻塞)。

Analyzed the java thread dumps observed nearly 61K threads are blocked by one of our method, which is causing this issue . Below is the portion of Thread dump

分析了所观察到的java线程转储，我们的一个方法阻止了将近61K线程，这导致了这个问题。下面是线程转储的部分

"SimpleAsyncTaskExecutor-16562" #38070 prio=5 os_prio=0 tid=0x00007f9985440000 nid=0x2ca6 waiting for monitor entry [0x00007f9d58c2d000]
   java.lang.Thread.State: BLOCKED (on object monitor)

#4

I would recommend to also look at the Thread Stack Size and see if you get more threads created. The default Thread Stack Size for JRockit 1.5/1.6 is 1 MB for 64-bit VM on Linux OS. 32K threads will require a significant amount of physical and virtual memory to honor this requirement.

我还建议您查看线程堆栈的大小，看看是否有更多的线程被创建。JRockit 1.5/1.6的默认线程堆栈大小是Linux OS上64位VM的1 MB。32K线程将需要大量的物理和虚拟内存来满足这一需求。

Try to reduce the Stack Size to 512 KB as a starting point and see if it helps creating more threads for your application. I also recommend to explore horizontal scaling e.g. splitting your application processing across more physical or virtual machines.

尝试将堆栈大小减少到512kb作为起点，看看是否有助于为应用程序创建更多的线程。我还建议探索横向扩展，例如，将应用程序处理拆分到更多的物理或虚拟机。

When using a 64-bit VM, the true limit will depend on the OS physical and virtual memory availability and OS tuning parameters such as ulimitc. I also recommend the following article as a reference:

当使用64位VM时，真正的限制将取决于OS物理和虚拟内存的可用性以及OS调优参数(如ulimitc)。我还推荐以下文章作为参考:

OutOfMemoryError: unable to create new native thread – Problem Demystified

OutOfMemoryError:无法创建新的本机线程问题

#5

I had the same problem due to ghost processes that didn't show up when using top in bash. This prevented the JVM to spawn more threads.

我遇到了同样的问题，因为在bash中使用top时没有出现幽灵进程。这阻止了JVM生成更多的线程。

For me, it resolved when listing all java processes with jps (just execute jps in your shell) and killed them separately using the kill -9 pid bash command for each ghost process.

对我来说，当用jps列出所有java进程(只需在您的shell中执行jps)并使用kill -9 pid bash命令分别杀死它们时，问题得到了解决。

This might help in some scenarios.

这在某些情况下可能会有所帮助。

#6

If your Job is failing because of OutOfMemmory on nodes you can tweek your number of max maps and reducers and the JVM opts for each. mapred.child.java.opts (the default is 200Xmx) usually has to be increased based on your data nodes specific hardware.

如果您的工作因为节点上的OutOfMemmory而失败，您可以每周您的最大映射和减少器的数量，并且JVM为每一个选择。mapred.child.java。opts(默认值为200Xmx)通常需要根据特定的数据节点硬件来增加。

This link might be helpful... pls check

这个链接可能会有帮助……请检查

#7

your JBoss configuration has some issues, /opt/jrockit-jdk1.6/bin/java -Xms512m -Xmx512m Xms and Xmx are limiting your JBoss memory usage, to the configured value, so from the 8Gb you have the server is only ussing 512M + some extra for his own purpose, increase that number, remember to leave some free for the OS and other stuff running there and may be you get it running despite de unsavoury code. Fixing the code would be nice too, if you can.

JBoss配置有问题,/ opt / jrockit-jdk1.6 / bin / java -Xms512m -Xmx512m Xms和Xmx限制你JBoss内存使用,配置的价值,所以从8 gb的服务器只有我们512 m +一些额外的为了自己的目的,增加这个数字,记得留一些免费的操作系统和其他东西跑步,可能你把它尽管de令人讨厌的代码。如果可以的话，修改代码也不错。

#8

You have a chance to face the java.lang.OutOfMemoryError: Unable to create new native thread whenever the JVM asks for a new thread from the OS. Whenever the underlying OS cannot allocate a new native thread, this OutOfMemoryError will be thrown. The exact limit for native threads is very platform-dependent thus its recommend to find out those limits by running a test similar to the below link example. But, in general, the situation causing java.lang.OutOfMemoryError: Unable to create new native thread goes through the following phases:

您有机会面对java.lang。OutOfMemoryError:当JVM向OS请求新线程时，无法创建新的本机线程。每当底层OS无法分配新的本机线程时，就会抛出这个OutOfMemoryError。本地线程的确切限制非常依赖于平台，因此它建议通过运行类似于下面链接示例的测试来发现这些限制。但是，通常情况下，会导致java.lang。OutOfMemoryError:无法创建新的本机线程，需要经过以下阶段:

A new Java thread is requested by an application running inside the JVM
运行在JVM中的应用程序请求一个新的Java线程
JVM native code proxies the request to create a new native thread to the OS The OS tries to create a new native thread which requires memory to be allocated to the thread
JVM本机代码代理请求为OS创建一个新的本机线程，操作系统试图创建一个新的本机线程，该线程需要分配给线程的内存。
The OS will refuse native memory allocation either because the 32-bit Java process size has depleted its memory address space – e.g. (2-4) GB process size limit has been hit – or the virtual memory of the OS has been fully depleted
OS将拒绝本机内存分配，要么是因为32位Java进程大小耗尽了它的内存地址空间(例如，命中了(2-4)GB进程大小限制)，要么是因为OS的虚拟内存已经耗尽
The java.lang.OutOfMemoryError: Unable to create new native thread error is thrown.
. lang。OutOfMemoryError:无法创建新的本机线程错误。

Reference: https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread

参考:https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread

#9

I had this same issue and it turned out to be an improper usage of an java API. I was initializing a builder in a batch processing method that was that not supposed to be initiallized more than once.

我遇到了同样的问题，结果证明是java API使用不当。我正在批处理方法中初始化一个构建器，这个方法不应该被初始化多次。

Basically I was doing something like:

基本上我是这样做的:

for (batch in batches) {
    process_batch(batch)
}

def process_batch(batch) {
    var client = TransportClient.builder().build()
    client.processList(batch)
}

when I should have done this:

当我应该这样做的时候:

for (batch in batches) {
    var client = TransportClient.builder().build()
    process_batch(batch, client)
}

def process_batch(batch, client) {
    client.processList(batch)
}

#10

To find which processes are creating threads try:

要查找正在创建线程的进程，请尝试:

ps huH

I normally redirect output to a file and analysis the file offline (is thread count for each process is as expected or not)

我通常将输出重定向到文件，并对文件进行脱机分析(每个进程的线程数是否与预期相同)

#11

-1

First of all I wouldn't blame that much the OS/VM.. rather the developer who wrote the code that creates sooo many Threads. Basically somewhere in your code (or 3rd party) a lot of threads are created without control.

首先，我不会那么责怪操作系统/虚拟机。而是编写代码来创建sooo多个线程的开发人员。基本上，在您的代码(或第三方)中，很多线程是在没有控制的情况下创建的。

Carefully review the stacktraces/code and control the number of threads that get created. Normally your app shouldn't need a large amount of threads, if it does it's a different problem.

仔细检查堆栈跟踪/代码，并控制创建的线程数量。通常你的应用不需要太多的线程，如果需要，那就另当别论了。

#1