We have an application which imports objects from an XML. The XML is around 15 GB. The application invariably starts running out of memory. We tried to free memory in between operations but this has lead to degrading performance. i.e it takes more time to complete the import operation. The CPU utilization reaches 100%
我们有一个从XML导入对象的应用程序。XML大约是15gb。应用程序总是开始耗尽内存。我们试图在操作之间释放内存，但这导致性能下降。我。完成导入操作需要更多的时间。CPU利用率达到100%
The application is written in C++.
应用程序是用c++编写的。
Does the frequent call to free() will lead to performance issues?
频繁调用free()是否会导致性能问题?
Promoted from a comment by the OP: the parser being used in expat, which is a SAX parser with a very small footprint, and customisable memory management.
来自OP的注释:在expat中使用的解析器，这是一个占用空间很小的SAX解析器，以及可定制的内存管理。
9 个解决方案
#1
3
Have you tried resuing the memory and your classes as opposed to freeing and reallocating it? Constant allocation/deallocation cycles, especially if they are coupled with small (less than 4096 bytes) data fragments can lead to serious performance problems and memory address space fragmentation.
您是否尝试过重新调用内存和您的类，而不是释放和重新分配它?固定的分配/释放周期，特别是当它们与较小的(小于4096字节)数据片段耦合时，可能会导致严重的性能问题和内存地址空间碎片化。
#3
2
Profile the application during one of these bothersome loads, to see where it is spending most of its time.
在这些烦人的负载中配置应用程序，看看它在哪里花费了大部分时间。
I believe that free() can sometimes be costly, but that is of course very much dependent on the platform's implementation.
我认为free()有时会很昂贵，但这当然很大程度上取决于平台的实现。
Also, you don't say a lot about the lifelength of the objects loaded; if the XML is 15 GB, how much of that is kept around for each "object", once the markup is parsed and thrown away?
而且，你也不会说很多关于加载的对象的生命线;如果XML是15gb，那么每个“对象”都保留了多少，一旦标记被解析并扔掉?
It sounds sensible to process an input document of this size in a streaming fashion, i.e. not trying a DOM-approach which loads and builds the entire XML parse tree at once.
以流的方式处理这样大小的输入文档听起来是明智的，例如，不要尝试一次加载和构建整个XML解析树的dom方法。
#4
1
If you want to minimise your memory usage, took a look at How to read the XML data from a file by using Visual C++.
如果您想要最小化内存使用，请查看如何使用Visual c++从文件中读取XML数据。
#5
1
One thing that often helps is to use a lightweight low-overhead memory pool. If you combine this with "frame" allocation methods (ignoring any delete/free until you're all done with the data), you can get something that's ridiculously fast.
一个经常有用的东西是使用一个轻量级的低开销内存池。如果您将它与“frame”分配方法结合使用(在处理完所有数据之前忽略任何删除/释放)，您就可以获得非常快的结果。
We did this for an embedded system recently, mostly for performance reasons, but it saved a lot of memory as well.
我们最近为嵌入式系统做了这个，主要是出于性能原因，但它也节省了很多内存。
The trick was basically to allocate a big block -- slightly bigger than we'd need (you could allocate a chain of blocks if you like) -- and just keep returning a "current" pointer (bumping it up by allocSize, rounded up to maximum align requirement of 4 in our case, each time). This cut our overhead per alloc from on the order of 52-60 bytes down to <= 3 bytes. We also ignored "free" calls until we were all done parsing and then freed the whole block.
诀窍是分配一个大块,略大于我们需要(如果你喜欢你可以分配一个连锁块),继续返回一个“当前”指针(allocSize碰撞起来,围捕最大的对齐要求4在我们的例子中,每一次)。这将每个alloc的开销从52-60字节减少到<= 3字节。我们还忽略了“free”调用，直到我们完成解析，然后释放整个块。
If you're clever enough with your frame allocation you can save a lot of space and time. It might not get you all the way to your 15GiB, but it would be worth looking at how much space overhead you really have... My experience with DOM-based systems is that they use tons of small allocs, each with a relatively high overhead.
如果你足够聪明的话，你可以节省很多空间和时间。它可能不会把你带到你的15GiB，但它值得看看你真正有多少空间开销……我对基于dom的系统的经验是，它们使用大量的小分配器，每个分配器的开销都相对较高。
(If you have virtual memory, a large "block" might not even hurt that much, if your access at any given time is local to a page or three anyway...)
(如果您有虚拟内存，那么一个大的“块”可能不会造成太大的伤害，如果您在任何给定时间的访问都是本地到一页或三页…)
Obviously you have to keep the memory you actually need in the long run, but the parser's "scratch memory" becomes a lot more efficient this way.
显然，您必须长期保留实际需要的内存，但是解析器的“擦除内存”通过这种方式变得更加高效。
#6
0
We tried to free memory in between operations but this has lead to degrading performance .
我们试图在操作之间释放内存，但这会导致性能下降。
Does the frequent call to free() will lead to performance issues ?
频繁调用free()是否会导致性能问题?
Based on the evidence supplied, yes.
根据提供的证据，是的。
Since your already using expat, a SAX parser, what exactly are you freeing? If you can free it, why are you mallocing it in a loop in the first place?
既然您已经在使用SAX解析器expat，那么您要释放什么呢?如果你可以释放它，为什么你首先要在一个循环中对它进行定位?
#7
0
Maybe, it should say profiler.
也许，它应该说是轮廓线。
Also don't forget that work with heap is single-thread. I mean that if booth of your threads will allocate/free memory in ont time, one of them will waiting when first will done.
另外，不要忘记处理堆的工作是单线程的。我的意思是，如果你的线程的booth在没有时间分配/空闲内存，他们中的一个将在第一个完成的时候等待。
If you allocating and free memory for same objects, you could create pool of this object and do allocate/free once.
如果您为相同的对象分配和释放内存，您可以创建这个对象的池，并进行一次分配/释放。
#8
0
Try and find a way to profile your code.
尝试找到一种方式来配置代码。
If you have no profiler, try and organize your work so that you only have a few free() commands (instead of the many you suggest).
如果您没有分析器，那么尝试组织您的工作，以便您只有几个free()命令(而不是您建议的许多命令)。
It is very common to have to find the right balance between memory consumption and time efficiency.
在内存消耗和时间效率之间找到正确的平衡是很常见的。
#9
0
I did not try it myself, but have you heard of XMLLite, there's an MSDN artical introducing it. It's used by MS Office internally.
我自己没有尝试过，但是你听说过XMLLite吗，有一个MSDN的文章介绍它。这是微软办公室内部使用的。
#1
3
Have you tried resuing the memory and your classes as opposed to freeing and reallocating it? Constant allocation/deallocation cycles, especially if they are coupled with small (less than 4096 bytes) data fragments can lead to serious performance problems and memory address space fragmentation.
您是否尝试过重新调用内存和您的类，而不是释放和重新分配它?固定的分配/释放周期，特别是当它们与较小的(小于4096字节)数据片段耦合时，可能会导致严重的性能问题和内存地址空间碎片化。
#2
#3
2
Profile the application during one of these bothersome loads, to see where it is spending most of its time.
在这些烦人的负载中配置应用程序，看看它在哪里花费了大部分时间。
I believe that free() can sometimes be costly, but that is of course very much dependent on the platform's implementation.
我认为free()有时会很昂贵，但这当然很大程度上取决于平台的实现。
Also, you don't say a lot about the lifelength of the objects loaded; if the XML is 15 GB, how much of that is kept around for each "object", once the markup is parsed and thrown away?
而且，你也不会说很多关于加载的对象的生命线;如果XML是15gb，那么每个“对象”都保留了多少，一旦标记被解析并扔掉?
It sounds sensible to process an input document of this size in a streaming fashion, i.e. not trying a DOM-approach which loads and builds the entire XML parse tree at once.
以流的方式处理这样大小的输入文档听起来是明智的，例如，不要尝试一次加载和构建整个XML解析树的dom方法。
#4
1
If you want to minimise your memory usage, took a look at How to read the XML data from a file by using Visual C++.
如果您想要最小化内存使用，请查看如何使用Visual c++从文件中读取XML数据。
#5
1
One thing that often helps is to use a lightweight low-overhead memory pool. If you combine this with "frame" allocation methods (ignoring any delete/free until you're all done with the data), you can get something that's ridiculously fast.
一个经常有用的东西是使用一个轻量级的低开销内存池。如果您将它与“frame”分配方法结合使用(在处理完所有数据之前忽略任何删除/释放)，您就可以获得非常快的结果。
We did this for an embedded system recently, mostly for performance reasons, but it saved a lot of memory as well.
我们最近为嵌入式系统做了这个，主要是出于性能原因，但它也节省了很多内存。
The trick was basically to allocate a big block -- slightly bigger than we'd need (you could allocate a chain of blocks if you like) -- and just keep returning a "current" pointer (bumping it up by allocSize, rounded up to maximum align requirement of 4 in our case, each time). This cut our overhead per alloc from on the order of 52-60 bytes down to <= 3 bytes. We also ignored "free" calls until we were all done parsing and then freed the whole block.
诀窍是分配一个大块,略大于我们需要(如果你喜欢你可以分配一个连锁块),继续返回一个“当前”指针(allocSize碰撞起来,围捕最大的对齐要求4在我们的例子中,每一次)。这将每个alloc的开销从52-60字节减少到<= 3字节。我们还忽略了“free”调用，直到我们完成解析，然后释放整个块。
If you're clever enough with your frame allocation you can save a lot of space and time. It might not get you all the way to your 15GiB, but it would be worth looking at how much space overhead you really have... My experience with DOM-based systems is that they use tons of small allocs, each with a relatively high overhead.
如果你足够聪明的话，你可以节省很多空间和时间。它可能不会把你带到你的15GiB，但它值得看看你真正有多少空间开销……我对基于dom的系统的经验是，它们使用大量的小分配器，每个分配器的开销都相对较高。
(If you have virtual memory, a large "block" might not even hurt that much, if your access at any given time is local to a page or three anyway...)
(如果您有虚拟内存，那么一个大的“块”可能不会造成太大的伤害，如果您在任何给定时间的访问都是本地到一页或三页…)
Obviously you have to keep the memory you actually need in the long run, but the parser's "scratch memory" becomes a lot more efficient this way.
显然，您必须长期保留实际需要的内存，但是解析器的“擦除内存”通过这种方式变得更加高效。
#6
0
We tried to free memory in between operations but this has lead to degrading performance .
我们试图在操作之间释放内存，但这会导致性能下降。
Does the frequent call to free() will lead to performance issues ?
频繁调用free()是否会导致性能问题?
Based on the evidence supplied, yes.
根据提供的证据，是的。
Since your already using expat, a SAX parser, what exactly are you freeing? If you can free it, why are you mallocing it in a loop in the first place?
既然您已经在使用SAX解析器expat，那么您要释放什么呢?如果你可以释放它，为什么你首先要在一个循环中对它进行定位?
#7
0
Maybe, it should say profiler.
也许，它应该说是轮廓线。
Also don't forget that work with heap is single-thread. I mean that if booth of your threads will allocate/free memory in ont time, one of them will waiting when first will done.
另外，不要忘记处理堆的工作是单线程的。我的意思是，如果你的线程的booth在没有时间分配/空闲内存，他们中的一个将在第一个完成的时候等待。
If you allocating and free memory for same objects, you could create pool of this object and do allocate/free once.
如果您为相同的对象分配和释放内存，您可以创建这个对象的池，并进行一次分配/释放。
#8
0
Try and find a way to profile your code.
尝试找到一种方式来配置代码。
If you have no profiler, try and organize your work so that you only have a few free() commands (instead of the many you suggest).
如果您没有分析器，那么尝试组织您的工作，以便您只有几个free()命令(而不是您建议的许多命令)。
It is very common to have to find the right balance between memory consumption and time efficiency.
在内存消耗和时间效率之间找到正确的平衡是很常见的。
#9
0
I did not try it myself, but have you heard of XMLLite, there's an MSDN artical introducing it. It's used by MS Office internally.
我自己没有尝试过，但是你听说过XMLLite吗，有一个MSDN的文章介绍它。这是微软办公室内部使用的。