I'm currently working on a website that makes large use of cached data to avoid roundtrips. At startup we get a "large" graph (hundreds of thouthands of different kinds of objects). Those objects are retrieved over WCF and deserialized (we use protocol buffers for serialization) I'm using redgate's memory profiler to debug memory issues (the memory didn't seem to fit with how much memory we should need "after" we're done initializing and end up with this report
我目前正在开发一个大量使用缓存数据的网站,以避免往返。在启动时,我们得到一个“大”图形(数百个不同类型的对象的thouthands)。这些对象通过WCF检索并反序列化(我们使用协议缓冲区进行序列化)我正在使用redgate的内存分析器来调试内存问题(在我们完成之后,内存似乎与我们应该需要多少内存无关)初始化并以此报告结束
Now what we can gather from this report is that:
现在我们可以从这份报告中得到的结论是:
1) Most of the memory .NET allocated is free (it may have been rightfully allocated during deserialisation, but now that it's free, i'd like for it to return to the OS)
1)分配的大部分内存是免费的(它可能在反序列化期间被正确分配,但现在它是免费的,我希望它能够返回到操作系统)
2) Memory is fragmented (which is bad, as everytime i refresh the cash i need to redo the memory hungry deserialisation process and this, in turn creates large object that may throw an OutOfMemoryException due to fragmentation)
2)内存碎片化(这很糟糕,因为每当我刷新现金时我需要重做内存饥饿的反序列化过程,这反过来会创建大型对象,由于碎片而可能抛出OutOfMemoryException)
3) I have no clue why the space is fragmented, because when i look at the large object heap, there are only 30 instances, 15 object[] are directly attached to the GC and totally unrelated to me, 1 is a char array also attached directly to the GC Heap, the remaining 15 are mine but are not the cause of this as i get the same report if i comment them out in code.
3)我不知道为什么空间是碎片的,因为当我看到大对象堆时,只有30个实例,15个object []直接连接到GC并且与我完全无关,1也是char数组直接附加到GC堆,其余15个是我的,但不是原因,因为我得到相同的报告,如果我在代码中评论它们。
So my question is, what can i do to go further with this? I'm not really sure what to look for in debugging / tools as it seems my memory is fragmented, but not by me, and huge amounts of free spaces are allocated by .net , which i can't release.
所以我的问题是,我该怎么做才能更进一步呢?我不太确定在调试/工具中要找什么,因为看起来我的内存是碎片化的,但不是我,并且.net分配了大量的可用空间,我无法释放。
Also please make sure you understand the question well before answering, i'm not looking for a way to free memory within .net (GC.Collect), but to free memory that is already free in .net , to the system as well as to defragment said memory.
另外请确保你在回答之前理解这个问题,我不是在寻找一种方法来释放.net(GC.Collect)中的内存,而是释放已经在.net中释放的内存,以及系统以及对记忆进行碎片整理。
Note that a slow solution is fine, if it's possible to manually defragment the large heap i'd be all for it as i can call it at the end of RefreshCache and it's ok if it takes 1 or 2 second to run.
请注意,缓慢的解决方案很好,如果可以手动对大堆进行碎片整理,那么我可以在RefreshCache的末尾调用它,如果运行需要1或2秒,则可以。
Thanks for your help!
谢谢你的帮助!
A few notes i forgot: 1) The project is a .net 2.0 website, i get the same results running it in a .net 4 pool, idem if i run it in a .net 4 pool and convert it to .net 4 and recompile.
我忘记了一些注意事项:1)该项目是.net 2.0网站,我在.net 4池中运行相同的结果,如果我在.net 4池中运行它并将其转换为.net 4和重新编译。
2) These are results of a release build, so debug build can not be the issue.
2)这些是发布版本的结果,因此调试版本不是问题。
3) And this is probably quite important, i do not get these issues at all in the webdev server, only in IIS, in the webdev i get memory consumption rather close to my actual consumption (well more, but not 5-10X more!)
3)这可能非常重要,我在webdev服务器上根本没有得到这些问题,只在IIS中,在webdev中我的内存消耗量与我的实际消耗量相当接近(更多,但不是5-10倍! )
6 个解决方案
#1
6
Objects allocated on the large object heap (objects >= 85,000 bytes, normally arrays) are not compacted by the garbage collector. Microsoft decided that the cost of moving those objects around would be too high.
在大对象堆上分配的对象(对象> = 85,000字节,通常是数组)不会被垃圾收集器压缩。微软决定移动这些对象的成本太高。
The recommendation is to reuse large objects if possible to avoid fragmentation on the managed heap and the VM space.
建议尽可能重用大对象,以避免托管堆和VM空间上的碎片。
http://msdn.microsoft.com/en-us/magazine/cc534993.aspx
I'm assuming that your large objects are temporary byte arrays created by your deserialization library. If the library allows you to supply your own byte arrays, you could preallocate them at the start of the program and then reuse them.
我假设您的大对象是由反序列化库创建的临时字节数组。如果库允许您提供自己的字节数组,则可以在程序开始时预先分配它们,然后重用它们。
#2
5
I know this isn't the answer you'd like to hear, but you can't forcefully release the memory back to the OS. However, for what reason do you want to do so? .NET will free its heap back to the OS once you're running low on physical memory. But if there's an ample amount of free physical memory, .NET will keep its heap to make future allocation of objects faster. If you really wanted to force .NET to release its heap back to the OS, I suppose you could write a C program which just mallocs until it runs out of memory. This should cause the OS to signal .NET to free its unused portion of the heap.
我知道这不是你想听到的答案,但你不能强行将内存释放回操作系统。但是,你想要这样做的原因是什么?一旦物理内存不足,.NET就会将其堆释回操作系统。但是如果有足够的空闲物理内存,.NET将保留其堆,以便更快地分配对象。如果你真的想强制.NET将其堆释放回操作系统,我想你可以编写一个只有malloc的C程序,直到它耗尽内存。这应该导致操作系统发信号通知.NET释放其未使用的堆部分。
It's better that unused memory be reseved for .NET so that your application will have better allocation performance (since the runtime knows what memory is free and what isn't, allocation can just use the free memory without having to syscall into the OS to get more memory).
最好为.NET重新使用未使用的内存,以便您的应用程序具有更好的分配性能(因为运行时知道哪些内存是免费的,哪些不是,分配可以只使用空闲内存而无需系统调用进入操作系统更多的记忆)。
The garbage collector is in charge of defragmenting the heap. Every so often (usually during collection runs), it will move objects around the heap if it determines this needs to be done. (This is why C++/CLI has the pin_ptr construct for "pinning" objects).
垃圾收集器负责对堆进行碎片整理。每隔一段时间(通常在集合运行期间),如果确定需要完成,它将在堆周围移动对象。 (这就是为什么C ++ / CLI具有用于“固定”对象的pin_ptr构造的原因)。
Fragmentation usually isn't a big issue though with memory, since it provides fast random access.
碎片通常对内存来说不是一个大问题,因为它提供了快速的随机访问。
As for your OutOfMemoryException, I don't have a good answer for. Ordinarily I'd suspect that your old object graph isn't being collected (some object somewhere is holding a reference onto it, a "memory leak"). But since you're using a profiler, I don't know then.
至于你的OutOfMemoryException,我没有一个好的答案。通常我怀疑你的旧对象图没有被收集(某个对象某处有一个引用它,一个“内存泄漏”)。但是既然你正在使用一个分析器,我当时就不知道。
#3
2
Some testing and some C++ later, i've found the reason why i get so much free memory, it's because of IIS instancing the CLR via VM Hoarding (providing a dll to instantiate it without VM Hoarding takes up as much initial memory, but does release most of it as time goes which is the behavior i expect). So this does fix my reported memory issue, however i still get about 100mb free memory no matter what, and i still think this is due to fragmentation and fragments only being released at once, because the profiler still reports memory fragmentation. So not marking my own answer as an answer in hope someone can shed some light on this or direct me to tools that can either fix this or help me debug the root cause.
一些测试和一些C ++以后,我发现了为什么我得到这么多的可用内存,这是因为IIS通过VM Hoarding实例化CLR(提供一个dll实例化它而没有VM Hoarding占用尽可能多的初始内存,但是释放大部分时间,这是我期望的行为)。所以这确实解决了我报告的内存问题,但无论如何我仍然可以获得大约100mb的可用内存,我仍然认为这是由于碎片和片段只是立即发布,因为探查器仍然报告内存碎片。因此,不要将我自己的答案标记为答案,希望有人能够对此有所了解或指导我使用可以修复此问题或帮助我调试根本原因的工具。
#4
1
It's intriguing that it works differently on the WebDevServer as to IIS...
令人感兴趣的是它在WebDevServer和IIS上的工作方式不同......
Is it possible that IIS is using the server garbage-collector, and the WebDev server the workstation garbage collector? The method of garbage collection can affect fragmentation. It'll probably be set in your aspnet.config file. See: http://support.microsoft.com/kb/911716
IIS是否可能使用服务器垃圾收集器,WebDev服务器是否可能使用工作站垃圾收集器?垃圾收集的方法可能会影响碎片。它可能会在您的aspnet.config文件中设置。请参阅:http://support.microsoft.com/kb/911716
#5
1
As of .NET 4.5.1 you can set a one-time flag to compact LOH before issuing a call to GC collect, i.e.
从.NET 4.5.1开始,您可以在发出对GC collect的调用之前设置一次性标志来压缩LOH,即
Runtime.GCSettings.LargeObjectHeapCompactionMode = System.Runtime.GCLargeObjectHeapCompactionMode.CompactOnce; GC.Collect(); // This will cause the LOH to be compacted (once).
Runtime.GCSettings.LargeObjectHeapCompactionMode = System.Runtime.GCLargeObjectHeapCompactionMode.CompactOnce;所以GC.Collect(); //这将导致LOH被压缩(一次)。
#6
0
If you havent found your answer I think the following clues can help you :
如果您还没有找到答案,我认为以下线索可以帮助您:
Back to the basics : we sometimes forget that the objects can be explicitly set free, call explicitly the Dispose method of the objects (because you didnt mention it, I suppose you do an "object = null" instruction instead).
回到基础:我们有时会忘记对象可以显式设置*,显式调用对象的Dispose方法(因为你没有提到它,我想你做了一个“object = null”指令)。
Use the inherited method, you dont need to implement one, unless your class doesnt have it, which I doubt it.
使用继承的方法,你不需要实现一个,除非你的类没有它,我怀疑它。
MSDN Help states about this method :
MSDN帮助说明了这种方法:
... There is no performance benefit in implementing the Dispose method on types that use only managed resources (such as arrays) because they are automatically reclaimed by the garbage collector. Use the Dispose method primarily on managed objects that use native resources and on COM objects that are exposed to the .NET Framework. ...
...在仅使用托管资源(例如数组)的类型上实现Dispose方法没有性能优势,因为垃圾收集器会自动回收这些资源。主要对使用本机资源的托管对象和公开给.NET Framework的COM对象使用Dispose方法。 ...
Because it says that "they are automatically reclaimed by garbage collector" we can infer that when the method is called does the "releasing thing" (Again Im trying only to give you clues).
因为它说“它们被垃圾收集器自动回收”,我们可以推断,当调用该方法时,“释放的东西”(我再次尝试只是为了给你提供线索)。
Besides I found this interesting article (I suppose ... I didn read it ...completely) : Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework (http://msdn.microsoft.com/en-us/magazine/bb985010.aspx) which states the following in the "Forcing an Object to Clean Up" section :
除此之外我发现了这篇有趣的文章(我想......我没看过......完全):垃圾收集:Microsoft .NET Framework中的自动内存管理(http://msdn.microsoft.com/en-us/magazine /bb985010.aspx)在“强制清理对象”部分中声明以下内容:
..., it is also recommended that you add an additional method to the type that allows a user of the type to explicitly clean up the object when they want. By convention, this method should be called Close or Dispose ....
...,还建议您为该类型添加其他方法,以允许该类型的用户在需要时显式清理对象。按照惯例,这个方法应该叫做Close或Dispose ....
Maybe the answer lies in this article if you read it carefully or just keep investigating in this direction.
如果你仔细阅读或只是继续调查这个方向,也许答案就在于这篇文章。
#1
6
Objects allocated on the large object heap (objects >= 85,000 bytes, normally arrays) are not compacted by the garbage collector. Microsoft decided that the cost of moving those objects around would be too high.
在大对象堆上分配的对象(对象> = 85,000字节,通常是数组)不会被垃圾收集器压缩。微软决定移动这些对象的成本太高。
The recommendation is to reuse large objects if possible to avoid fragmentation on the managed heap and the VM space.
建议尽可能重用大对象,以避免托管堆和VM空间上的碎片。
http://msdn.microsoft.com/en-us/magazine/cc534993.aspx
I'm assuming that your large objects are temporary byte arrays created by your deserialization library. If the library allows you to supply your own byte arrays, you could preallocate them at the start of the program and then reuse them.
我假设您的大对象是由反序列化库创建的临时字节数组。如果库允许您提供自己的字节数组,则可以在程序开始时预先分配它们,然后重用它们。
#2
5
I know this isn't the answer you'd like to hear, but you can't forcefully release the memory back to the OS. However, for what reason do you want to do so? .NET will free its heap back to the OS once you're running low on physical memory. But if there's an ample amount of free physical memory, .NET will keep its heap to make future allocation of objects faster. If you really wanted to force .NET to release its heap back to the OS, I suppose you could write a C program which just mallocs until it runs out of memory. This should cause the OS to signal .NET to free its unused portion of the heap.
我知道这不是你想听到的答案,但你不能强行将内存释放回操作系统。但是,你想要这样做的原因是什么?一旦物理内存不足,.NET就会将其堆释回操作系统。但是如果有足够的空闲物理内存,.NET将保留其堆,以便更快地分配对象。如果你真的想强制.NET将其堆释放回操作系统,我想你可以编写一个只有malloc的C程序,直到它耗尽内存。这应该导致操作系统发信号通知.NET释放其未使用的堆部分。
It's better that unused memory be reseved for .NET so that your application will have better allocation performance (since the runtime knows what memory is free and what isn't, allocation can just use the free memory without having to syscall into the OS to get more memory).
最好为.NET重新使用未使用的内存,以便您的应用程序具有更好的分配性能(因为运行时知道哪些内存是免费的,哪些不是,分配可以只使用空闲内存而无需系统调用进入操作系统更多的记忆)。
The garbage collector is in charge of defragmenting the heap. Every so often (usually during collection runs), it will move objects around the heap if it determines this needs to be done. (This is why C++/CLI has the pin_ptr construct for "pinning" objects).
垃圾收集器负责对堆进行碎片整理。每隔一段时间(通常在集合运行期间),如果确定需要完成,它将在堆周围移动对象。 (这就是为什么C ++ / CLI具有用于“固定”对象的pin_ptr构造的原因)。
Fragmentation usually isn't a big issue though with memory, since it provides fast random access.
碎片通常对内存来说不是一个大问题,因为它提供了快速的随机访问。
As for your OutOfMemoryException, I don't have a good answer for. Ordinarily I'd suspect that your old object graph isn't being collected (some object somewhere is holding a reference onto it, a "memory leak"). But since you're using a profiler, I don't know then.
至于你的OutOfMemoryException,我没有一个好的答案。通常我怀疑你的旧对象图没有被收集(某个对象某处有一个引用它,一个“内存泄漏”)。但是既然你正在使用一个分析器,我当时就不知道。
#3
2
Some testing and some C++ later, i've found the reason why i get so much free memory, it's because of IIS instancing the CLR via VM Hoarding (providing a dll to instantiate it without VM Hoarding takes up as much initial memory, but does release most of it as time goes which is the behavior i expect). So this does fix my reported memory issue, however i still get about 100mb free memory no matter what, and i still think this is due to fragmentation and fragments only being released at once, because the profiler still reports memory fragmentation. So not marking my own answer as an answer in hope someone can shed some light on this or direct me to tools that can either fix this or help me debug the root cause.
一些测试和一些C ++以后,我发现了为什么我得到这么多的可用内存,这是因为IIS通过VM Hoarding实例化CLR(提供一个dll实例化它而没有VM Hoarding占用尽可能多的初始内存,但是释放大部分时间,这是我期望的行为)。所以这确实解决了我报告的内存问题,但无论如何我仍然可以获得大约100mb的可用内存,我仍然认为这是由于碎片和片段只是立即发布,因为探查器仍然报告内存碎片。因此,不要将我自己的答案标记为答案,希望有人能够对此有所了解或指导我使用可以修复此问题或帮助我调试根本原因的工具。
#4
1
It's intriguing that it works differently on the WebDevServer as to IIS...
令人感兴趣的是它在WebDevServer和IIS上的工作方式不同......
Is it possible that IIS is using the server garbage-collector, and the WebDev server the workstation garbage collector? The method of garbage collection can affect fragmentation. It'll probably be set in your aspnet.config file. See: http://support.microsoft.com/kb/911716
IIS是否可能使用服务器垃圾收集器,WebDev服务器是否可能使用工作站垃圾收集器?垃圾收集的方法可能会影响碎片。它可能会在您的aspnet.config文件中设置。请参阅:http://support.microsoft.com/kb/911716
#5
1
As of .NET 4.5.1 you can set a one-time flag to compact LOH before issuing a call to GC collect, i.e.
从.NET 4.5.1开始,您可以在发出对GC collect的调用之前设置一次性标志来压缩LOH,即
Runtime.GCSettings.LargeObjectHeapCompactionMode = System.Runtime.GCLargeObjectHeapCompactionMode.CompactOnce; GC.Collect(); // This will cause the LOH to be compacted (once).
Runtime.GCSettings.LargeObjectHeapCompactionMode = System.Runtime.GCLargeObjectHeapCompactionMode.CompactOnce;所以GC.Collect(); //这将导致LOH被压缩(一次)。
#6
0
If you havent found your answer I think the following clues can help you :
如果您还没有找到答案,我认为以下线索可以帮助您:
Back to the basics : we sometimes forget that the objects can be explicitly set free, call explicitly the Dispose method of the objects (because you didnt mention it, I suppose you do an "object = null" instruction instead).
回到基础:我们有时会忘记对象可以显式设置*,显式调用对象的Dispose方法(因为你没有提到它,我想你做了一个“object = null”指令)。
Use the inherited method, you dont need to implement one, unless your class doesnt have it, which I doubt it.
使用继承的方法,你不需要实现一个,除非你的类没有它,我怀疑它。
MSDN Help states about this method :
MSDN帮助说明了这种方法:
... There is no performance benefit in implementing the Dispose method on types that use only managed resources (such as arrays) because they are automatically reclaimed by the garbage collector. Use the Dispose method primarily on managed objects that use native resources and on COM objects that are exposed to the .NET Framework. ...
...在仅使用托管资源(例如数组)的类型上实现Dispose方法没有性能优势,因为垃圾收集器会自动回收这些资源。主要对使用本机资源的托管对象和公开给.NET Framework的COM对象使用Dispose方法。 ...
Because it says that "they are automatically reclaimed by garbage collector" we can infer that when the method is called does the "releasing thing" (Again Im trying only to give you clues).
因为它说“它们被垃圾收集器自动回收”,我们可以推断,当调用该方法时,“释放的东西”(我再次尝试只是为了给你提供线索)。
Besides I found this interesting article (I suppose ... I didn read it ...completely) : Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework (http://msdn.microsoft.com/en-us/magazine/bb985010.aspx) which states the following in the "Forcing an Object to Clean Up" section :
除此之外我发现了这篇有趣的文章(我想......我没看过......完全):垃圾收集:Microsoft .NET Framework中的自动内存管理(http://msdn.microsoft.com/en-us/magazine /bb985010.aspx)在“强制清理对象”部分中声明以下内容:
..., it is also recommended that you add an additional method to the type that allows a user of the type to explicitly clean up the object when they want. By convention, this method should be called Close or Dispose ....
...,还建议您为该类型添加其他方法,以允许该类型的用户在需要时显式清理对象。按照惯例,这个方法应该叫做Close或Dispose ....
Maybe the answer lies in this article if you read it carefully or just keep investigating in this direction.
如果你仔细阅读或只是继续调查这个方向,也许答案就在于这篇文章。